Crazy to not see a word about benchmarking/evaluation of the kernels.
IMO at this time when all performant GPU kernels are generated/hill-climbed, its the largest source of error we see.
Should be kernel run with a hot or cold cache in practice? Should your kernel be bitwise equivalent to a ref? What error margins should your kernel have to ref? How many times are you running the kernel to avoid thermal noise? Are you running on separate GPUs to avoid overfitting the current hardware? Can you cudagraph-compiling your kernel to remove launch overhead? Are you A/B B/A interleaving your kernel/reference runs to avoid running thermal bias?
As someone who loves being able to press a button to receive matmuls tuned to the 3090 by his feet, are there rules of thumb to work out how to set thresholds for evals?
Do I want to set a particular threshold on any individual op (or a layer for a given transformer)? e.g "for this test set, 99% of l2norms being within 1e-4 error of the golden reference kernel"
Or do I want to just shortcut all that and go with task based evals? eg "word error no worse than 1% greater than the reference speech-to-text model"?
I'm assuming the latter is the way to go in the first instance?
It depends on if you do training or inference and also at what scale.
For your task with the 3090, and I assume inference, I would like you suggest do end-to-end on target task.
Sometimes for example, a normalization layer can cancel out/fix an error from a previous layer, then it makes no sense to spend flops to save the numerics.
This is kind of a wild west in inference serving, you sometimes change exp to tanh, sometimes go full 4bit, sometimes remove full layers because you still perform fine on the end-task.
Numerics and correctness are more important for training kernels and backwards passes
Thanks! Interestingly I find that coding agents really don't now how to quantize a model to a given loose target. "Make it faster and accurate" is seemingly in distribution, "Make it _even_ faster and worse" is a good way to get Opus 5.5 to say "I tried nothing and I'm all out of ideas"
Crazy to not see a word about benchmarking/evaluation of the kernels.
IMO at this time when all performant GPU kernels are generated/hill-climbed, its the largest source of error we see.
Should be kernel run with a hot or cold cache in practice? Should your kernel be bitwise equivalent to a ref? What error margins should your kernel have to ref? How many times are you running the kernel to avoid thermal noise? Are you running on separate GPUs to avoid overfitting the current hardware? Can you cudagraph-compiling your kernel to remove launch overhead? Are you A/B B/A interleaving your kernel/reference runs to avoid running thermal bias?
As someone who loves being able to press a button to receive matmuls tuned to the 3090 by his feet, are there rules of thumb to work out how to set thresholds for evals?
Do I want to set a particular threshold on any individual op (or a layer for a given transformer)? e.g "for this test set, 99% of l2norms being within 1e-4 error of the golden reference kernel"
Or do I want to just shortcut all that and go with task based evals? eg "word error no worse than 1% greater than the reference speech-to-text model"?
I'm assuming the latter is the way to go in the first instance?
It depends on if you do training or inference and also at what scale.
For your task with the 3090, and I assume inference, I would like you suggest do end-to-end on target task.
Sometimes for example, a normalization layer can cancel out/fix an error from a previous layer, then it makes no sense to spend flops to save the numerics.
This is kind of a wild west in inference serving, you sometimes change exp to tanh, sometimes go full 4bit, sometimes remove full layers because you still perform fine on the end-task.
Numerics and correctness are more important for training kernels and backwards passes
Thanks! Interestingly I find that coding agents really don't now how to quantize a model to a given loose target. "Make it faster and accurate" is seemingly in distribution, "Make it _even_ faster and worse" is a good way to get Opus 5.5 to say "I tried nothing and I'm all out of ideas"
It’s not crazy at all, the blog post is clearly about human-guided optimization with LLMs rather than hillclimbing with an autonomous agent.
All of the above is as-relevant in human-guided optimization.
Yes, but they have nothing to do with GPUs and kernels. Those are general performance standards that would apply to any effort regardless of target.
Tell me more about cuda-graphs and async on CPU ;)
Sure, that might be worth discussing, I agree
The writing here is so clear. Do you use Llms to write?
My guess is that the LLMs helped with the diagrams but the writing is mostly or completely from a human. It’s quite good (and has a few typos :P)
The page design is the standard Claude design, what do you expect?
> On the H200s used for most of this work, the GPU's main memory is HBM, or high-bandwidth memory. Ironically, this is the slowest memory on the chip.
Nit: HBM is off chip, unlike shared memory and registers
Someone needs to write a book on this
Programming Massively Parallel Processors is a classic (written by NVIDIA/CUDA engineers themselves), if you haven’t checked it out already
I really liked AI Systems Performance Engineering by Chris Fregly