You can do even a bit better if you're willing to use intrinsics. In particular this kind of operation is well-suited for compress-type operations, available as a first-class operation in at least AVX512, SVE and RVV; you can also emulate them reasonably quickly on NEON and AVX2.
Here's an example, building on the OP's work:
pub fn filter_compress(input: &[f64], threshold: f64) -> Vec<f64> {
use std::arch::x86_64::*;
let mut out = vec![0.0; input.len()];
let mut n = 0usize;
let (head, tail) = input.as_chunks::<8>();
for chunk in head {
unsafe {
let p = _mm512_loadu_pd(chunk.as_ptr());
let m = _mm512_cmpnle_pd_mask(p, _mm512_set1_pd(threshold));
let compress = _mm512_maskz_compress_pd(m, p);
_mm512_storeu_pd(out.as_mut_ptr().wrapping_add(n), compress);
n += m.count_ones() as usize;
}
}
for &x in tail {
out[n] = x;
n += (x > threshold) as usize;
}
out.truncate(n);
out
}
For me it's about 25% less time than the branchless version with 1,000,000 elements, and 60% less with 10,000 elements where memory bandwidth effects are less relevant.
Nice! I saw the code and thought, I bet there’s a way to do some SIMD here… never touched intrinsics in Rust before so I really appreciate you writing it up!
Memory access patterns are usually the main reason for low performance and should be optimized first because it's a great low hanging fruit (and usually the bigger problem is latency, not bandwidth).
And don't think of waiting for memory as making CPU optimizations irrelevant, but instead as an oppurtinity to hide more CPU operations in the remaining 'memory access gaps' (e.g. the CPU won't simply stop working when waiting for data to be loaded from memory, it can continue with other things that don't depend on that data).
Good question. I personally doubt that the compress instruction is easy to coax compilers into generating, as there are many edge cases to consider.
For example, you'll notice here that we perform a full vector store of 8 elements unconditionally, even if only a few of the elements are active. This is safe, though, because the output buffer is as large as the input buffer, and we're chunking by 8, so we'll never trash memory past the end; but this is a tricky analysis. Performance-wise, we rely on the CPU's store buffer to make these overlapping stores cheap.
Instead, you might think that you could just store the elements which are actually active, using a masked store. In fact there is also an intrinsic for this purpose (_mm512_mask_compressstoreu_pd), but it is extremely slow on some CPUs, namely Zen 4, so it's dangerous to use unless you know exactly what CPU you're using. (In my testing, there also seems to be some weird hazard on Zen 5 where multiple memory-destination compress instructions to nearby, even non-overlapping, addresses are serialized. But I haven't looked closer at this.)
I think WUFFS "iterate loops" might help. WUFFS requires that processing a chunk of N items has code to process one at a time, which means it'll work for any N. However you can optionally provide specialisations for doing K at a time and the compiler is responsible for carving the input up as appropriate so e.g. N = K + K + 1 + 1 + 1 your K-at-a-time code runs twice, the extras are handled 1-at-a-time.
So this divides up the problem, the compiler can vectorize your 8-at-a-time code without needing to handle edge cases where N isn't a multiple of 8, and if a later pass notices we actually never end up using those edge cases they're dead code, if it doesn't they're just a rarely-taken branch once.
I believe the bounds check, in particular, is devastating for autovectorization. There are ways around it, but it requires additional code in safe rust.
Edit: actually looks like autovectorization is in play here [1]. Doesn't look like the bounds check gets in the way at all.
Compress patterns aren't recognized by any open-source compiler autovectorizer as far as I'm aware of. (I think intel's proprietary C/C++ compiler can?)
Once you have a mask of the positions you want to compress, you can generate a shuffle index vector from that mask to place the desired elements in the low part of the vector. You can expand the mask into nibble-sized indices using pext/pdep and some magic constants, then expand those nibble-sized indices into a vector of indices to use as the shuffle indices.
Yes, that's one approach. Another reasonable approach is to get out a mask from the comparison using `vmovmskpd` and use that to look up a shuffle constant, since there are only 16 possibilities. This also works well on NEON, although I wonder there whether it'd make more sense to find the shuffle dynamically rather than loading it.
I'm apparently not good at spotting it. I was put off by the overly dramatic presentation. It gets tiring that the author apparently finds this more exciting than I do, and writes like it's enthralling. I just assumed it was an excess of enthusiasm or the first experience with this kind of thing. If it's AI, I'm way behind the game noticing it.
There are lots of indicators in this text, and this breathless presentation is very much how modern LLMs present things.
Claude also loves to describe things as being "real", particularly saying "X is real".
In this case,
> The reallocations were real, but they were never the bottleneck.
There was never any indication or setup in the text that they weren't real, but it's how it justifies wasted effort, it insists that some phenomenon it corrected but failed to solve the problem "was real".
Another giveaway are nonsensical analogies:
> The predictor is like a barista who starts making your usual order the moment you walk in. If you are a regular, this is fantastic: the coffee is ready when you reach the counter. If you order something random every day, the barista keeps pouring drinks into the sink.
If you order "something random every day", then you don't have a usual order for them to be making, it's an analogy that doesn't work.
And of course, the smoking gun is:
> The smoking gun
It probably won't be a good indicator forever as it has been noticed so much, but it's a particular favourite of the current generation of anthropic models.
"The smoking gun" is right there in the text ;) (but also things like "Same million floats. Same threshold. Same function."). Don't know if other models have that same specific style, but it looks very 'claude-y'.
I wouldn't be surprised though if (especially) non-native speakers unconsciously start adopting the Claude writing style when they stare all day long at Claude generated text at work.
I would assume non-native speakers talk to Claude in their own language. Now I'm curious if Claude's weird quirks of speech are unique in each language or if they carry over!
I'm German but still do all my computing in English (partly out of habbit, but also because Germanized technical text is usually painful to work with because it's full of anglizisms (is that actually an English word?)).
One nice thing about English is you can just make up words with plausible etymological roots in Latin/French or Old English and often people will know what you mean. In this case though it would probably be spelled "Anglicism".
> Don't know if other models have that same specific style
An awful lot of the open-weight models also talk in the Claude-y style. Not sure if an artefact of distilling anthropic models, or just a preponderance of slop in the training set...
> I was put off by the overly dramatic presentation. It gets tiring that the author apparently finds this more exciting than I do, and writes like it's enthralling.
That's one of the main tells that AI wrote this. All the stylistic tics that people usually point out combine to make the writing seem more important than it is.
The main problem with the article is that the idea to influence backend code generation decisions via specific highlevel code constructs is mostly just mystical bullshit (some compilers do detect specific patterns - usually for bit twiddling hacks, but not on a basic level like control flow optimization).
Using a highlevel language construct like "y += (x > 0) as usize;" doesn't "switch on" branchless code just because the source code looks branchless, compilers are not that dumb anymore.
E.g. I bet that writing
if (x > 0) {
y += 1;
}
...generates the exact same code after optimization, otherwise I would consider that an LLVM bug.
The only reliable way is to mostly bypass the optimizer via simd intrinsics, or drop down to assembler, everything else is just cargo culting.
(fwiw I can't shake the feeling now that the article is recycled, I'm pretty sure I saw those exact same code examples in another "branchless" blog post, but maybe for a different language - because the next question was ineviatably "then why is the code using "if" slower? answer: because it also behaves differently). Or maybe I'm just having a strong dejavu ;)
Click on their blog index page, see posts going back to early 2010s and use the same writing style. He must have been time travelling and using AI all this time!
I had a look at their blog page out of curiosity, not that you can prove much from the purported dates and text on a blog, which could be edited at any time.
A page shows up in the Wayback Machine when it indexes it for the first time. The website itself goes back a lot if you look at the time. Here is a wayback machine link of a blog post from 2015 which got archived in 2017 for the first time (https://web.archive.org/web/20170607071648/http://greyblake....). Again, if you want to make an intelligent argument, at least know what you're talking about.
I think he's asked it to write in his specific style, or possibly he has edited parts of it to his style, or maybe used AI to generate the initial draft.
Something like that anyway. There are some very clear AI tells (smoking guns if you like), but most of it does not read like the prose AI produces by default.
Author if you are here I am curious about your writing process, and why you didn't remove the obvious AI tells.
I always got voted down when I posted the evaluation of the parent articles I got from my Ouija board. I just want to help people understand whether they should just reject bad articles, without having to bother reading them.
I'm moving on to evaluating articles with a modified lie detector test and tarot cards, I'm sure that'll help my credibility and give my public rejections more authority.
Do you have evidence Pangram is unreliable? There are independent evaluations [1, 2] showing it works, and it's getting used more and more scientific papers. Have you used it or evaluated it yourself? What do you think these other evaluators are doing or getting wrong?
I don't doubt that those detectors are generally correct. Pangram seems to be particularly accurate. I see independent evaluations ranging from 97% accurate to over 99%. Frankly, I somewhat doubt those numbers, but I do agree that LLM usage can be fairly accurately detected.
But procedurally, there are huge issues involved with automated tools used to harm other people. You are one of the 0.5% percent of people whose article was flagged as LLM-generated when it wasn't, one of the false positives. What do you do? Argue? The accusers will claim that you're 99.5% likely to be lying.
It's the same issue we have with automated customer service, automated insurance claims, and so forth. It is usually correct, and terrifically unjust when it fails... at which point there is no recourse. In a perverse sense, its accuracy can be a drawback, because if the false positive rate low enough, nobody is going to believe you when you're falsely accused. And people will be falsely accused.
I think it's ironic that it seems like it capitalizes on the same flaw that most LLM-posting does... "Chat GPT is usually right, I'm going with it." You shouldn't post an LLM article without independently validating its claims, so that there is a responsible person in the loop. The same is true for rejections and accusations, but more so, because they're more damaging.
Interesting topic but why destroy your own credibility and reputation by shoveling llm-assisted slop to us here at hn?
The post should be flagged, and in general, i wish hn would adopt a no-tolerance policy to enhanced posting like this.
So what if the original text, if it existed in a human written form at all, had weird textual quirks and prose issues the author wished to hide. That texture's what makes humans interesting to engage with in the first place.
Great explanation of why a branchless approach results in such a speed up. I've never really had to deal with performance optimization at this level. Generally it's probably best not to get too involved letting the CPU black box do its thing.
I do wonder, would the performance characteristics of branchless vs branching be consistent across different CPUs/architectures? If you had a CPU that wasn't trying to be fancy with branch prediction, would the regular algo be faster?
CPUs aren't black boxes. They are actually much better documented than almost all the software that runs on them.
If you want to treat the CPU as a black box, trust me you do not want to use a CPU with out a branch predictor, your slow code will run like molasses frozen in antarctica.
The regular algo will be lightyears slower on any CPU that does not have a branch predictor.
I think the Pentium is more or less the first microprocessor with branch prediction. Certainly the most mainstream.
PowerPC 601 arrived at more or less the same time, and the Alpha 21064 was a year earlier. There were a few minicomputers and mainframes before that with branch predictors.
Arguably the 486 could have done with a branch predictor (even a single entry loop predictor would have helped), and maybe the 386 too. But microcoded CISC designs didn't benefit much from predictors because they have multiple cycles to work it out.
And RISC cpus were in their "branch delay slots are awesome" phase throughout most of the 80s. With a bit of trickery (very simple branch conditions and a 2 phase clock), your classic 5-stage MIPS design can fully hide all branches with just a single branch delay slot, so they were a little slow to adopt predictors.
I get the impression that CPU designers in the 80s and early 90s massively underestimated just how beneficial even a small predictor can be.
Pentium was the first CPU with a branch predictor that many people could afford to buy.
Before dynamic branch prediction, where the prediction for every branch is updated after each branch execution, depending on its history, static branch prediction had been used for decades, since around 1960, typically using the rule that forwards branches are unlikely to be taken, but backwards branches are likely to be taken. An alternative was to have an instruction bit where the compiler stored its prediction about the probability of a branch being taken.
Dynamic branch predictors began to be used since the mid seventies.
I do not remember now if any notable monolithic CPU had a dynamic branch predictor before Pentium, but prior multi-chip CPUs certainly existed.
> Before dynamic branch prediction... static branch prediction had been used for decades.
I'm not sure that's true. At least not the "predict backwards branches are taken" part.
Many RISC cpus did kind of have "predict forwards as not taken", but really thats just speculative execution with the complete absence of any branch prediction at all. IMO "All branches are not taken" is not a prediction.
Actual Static Branch prediction is something that seems to have shown up at the exact same time as dynamic branch prediction (ie Pentium and PowerPC 601). Seems to be more of a "well do speculative execution now, what do we do when there is no dynamic prediction?" thing.
Maybe there is a multi-chip CPU out there that has proper static prediction but not dynamic? But I checked the likely candidate (The RS/6000 aka POWER1) and it doesn't have any prediction at all (just a hole where the static prediction bits will go later). Probably because static prediction requires support for speculative execution, which it doesn't do.
> I get the impression that CPU designers in the 80s and early 90s massively underestimated just how beneficial even a small predictor can be.
It's got a lot to do with how cpu clock speeds were getting way faster, but ram wasn't. That's what makes deeper pipelines attractive, and if you give a cpu a deeper pipeline, it's gonna want a good branch predictor.
I'm more thinking about how MIPS were quite late to branch predictors.
They were shipping the high-performance R4000 and R4400 with 8 stage pipelines and no branch predictors.
They could have really done with a branch predictor, each branch took three cycles (and the branch delay slot could fill only one instruction, often a useless NOP).
The Pentium only had a 5 stage pipeline and massively benefited from its branch predictor. IBM was slapping branch predictors on PowerPC designs with 4 stage integer pipelines. You simply don't need a long pipeline to justify the benefits of a branch predictor.
Cortex M0 and microprocessors generally do not. Cortex M3’s looks nothing like the branch prediction you think of when you think consumer or server CPU. Basically branch prediction requires extra power so it’s excluded or greatly simplified in low power use cases.
I'm not sure how your intuition can be that off, if you don't have a branch predictor then any branching code is going to be even slower than it already is, favouring branchless code even more for obvious reasons.
I say this as someone who is interested in a special type of processor architecture that has no branch prediction at all and would need a branchless subset of Rust to meaningfully program it at high performance.
Why no branch predictor at all? Even a brain-dead one that predicts all branches always/never taken is going to provide some benefit, it's not as if the processor can do anything else while it's waiting.
Or am I missing something?
I note the hazard 3 on the pi Pico rp2350 only predicts a branch if it's the last branch and was taken, ie a single loop. Which seems weird to me, so I'm probably lacking understanding somewhere.
Another recent story from github about case folding as part of code search, the simple version of the code had a couple of ifs, and the branchless version was actually slower.
They have a stupendously fast version and it is also branchless, but it just required more than branchless alone.
I'm fuzzy on the details but I think one of the ifs was an early exit, and without that the loop does a memory assignment on every byte instead of skipping most.
The really fast version was also vectorized. The branchless makes it possible to vectorize, but it was the vectorization that actually made it fast.
This problem is called stream compaction and there is a wealth of research on it. The best methods use prefix scan. They first efficiently compute the index in the output array of each element that satisfies the predicate and then they gather them in one linear operation.
Also, I can tell that you are a good writer. You didn't need the LLM to "polish" your text.
There is clearly LLM-prose involved, but it's pretty well done. Here's one example: "The reallocations were real, but they were never the bottleneck."
LLMs love this pattern. Whether one put it into this text, or the author soaked it up and now used it himself, who knows. But it is one of the few things in the post that gives me the ick.
And then, there's the verbosity.
If I had to guess, an LLM was involved, but the author did a good job with manual writing and editing, too.
Because what makes stream compaction challenging is the loop carried dependency: the location you write to in a given iteration depends on the locations you wrote to in previous iterations. By first creating a lookup table of source -> destination locations you remove the dependency. Then you can apply extremely efficient parallel methods.
I like how we have pretty much established how branchless coding is superior to branched coding.
However I wonder if the compiler itself could recognize these patterns and turn branches into branchless instead, rather than making the code harder to read? as removing if conditions of course have a readability impact on the code.
Branchless coding is superior to branched coding whenever the branches are more or less random, which happens frequently when checking some properties of input numbers, like their sign or whether they fall inside certain intervals, or when sorting an array that comes in random order.
When a branch alternative will be taken much more frequently than the other, then branched coding with an "if" becomes superior.
So neither is better in general than the other, whenever the program must choose between alternatives, you must think about whether one is more likely than the other, or if both have similar probabilities.
For instance, when sorting an array, the optimal algorithm is not the same when you expect the input array to have a random order and when you expect it to be already almost sorted.
This is true, but the example given yesterday showed that even if branches can be very well predicted (e.g. processing UTF-8 text which is 99.9999% ASCII), branchless code can result in speedup by making autovectorisation possible.
If the branchless code didn't transform to vector instructions, it would be strictly slower. But if it does, it allows the CPU to work on 16 bytes at a time instead of 1 at a time.
For sorting, conveniently we always definitely need to look at all the elements at least once anyway, so although even the early introspective sorts from the end of last century aren't designed this way both the Timsort and a modern sort like a PDQ sort will end up making that decision early.
"Oh, this was mostly already sorted, done"
If you meant exactly rather than almost then you can still squeak a small win from having an algorithm which is optimised for this case but the vast bulk of your runtime is eaten by the unavoidable work of checking. "Don't check" is faster but then you're not a sort algorithm at all.
Based on the title, I knew the issue as soon as I looked at the first table. Still, great primer for those who don't know about such CPU shenanigans, and I did appreciate the solution, since I knew high level how to solve it, but didn't come up with an actual piece of code before the author presented theirs.
I didn't know about branch prediction or pipelined CPUs back when I was profiling the code I wrote - honestly it probably would have helped.
Worth noting that as written the "trick" results in memory usage proportional to the size of the input rather than the output. If the filter rejects most of the input the difference could be quite noticeable.
I disagree in the sense that you can rewrite the code to use the trick and also not allocate in advance. Nothing about the trick requires you to allocate up front: before writing to out[n] you can extend the vector if it’s out of bounds. Or, after incrementing n, do out.push(0).
that's a really clever trick to write to out[n] multiple times but only move the index after the logical condition is true thus ending up with the correct values in out
> The predictor is like a barista who starts making your usual order the moment you walk in. If you are a regular, this is fantastic: the coffee is ready when you reach the counter. If you order something random every day, the barista keeps pouring drinks into the sink.
I laughed out loud reading this. Interesting writeup. I wonder what kinds of tricks like this exist for computation graph compilers like JAX.
Branchless code can indeed sometimes be slower than conventional one, but in this particular case, the article comes to the wrong conclusion. At a 1% kept, the branchless version is slower because it pays the cost of zero-initializing 8 MB of memory when allocating the Vec. This can be easily demonstrated by comparing it with a version that allocates uninitialized memory.
Did you read the article? The author was trying to even out the test cases and successfully did so. He stated that the idiomatic code was faster for the 1 % as you can see by quote below:
The worst case became almost 4 times faster. And look how flat the branchless column is: the running time does not depend on the data anymore, exactly as we wanted.
Notice the price we paid though. At 1% kept the idiomatic version wins, because an almost always correctly predicted branch is nearly free, while the branchless version always pays for one million writes. Branchless code is not faster in general: it trades the best case for the worst case.
Thanks for sharing, optimisations like these are what keeps the fun in programming. I have been optimising my JSONLogic evaluator in rust and used arena allocator and preallocation tricks that gave me good jump in tuning. Let me see if branchless programming techniques can get any further in my case
I don't know if an optimization is allowed to "invent" a write, but I would be surprised if an optimizer goes that far because I have to believe that the number of cases where more writes improve performance are pretty slim.
Keep in mind that this is a synthetic benchmark. The task is to remove outliers and for that the roughly 99% correct guesses of the branch predictor is perfectly fine.
This is a common pattern a compiler should recognise and optimise into an efficient data and control flow. So much for a sufficiently smart compiler. shrug
Nice post!
You can do even a bit better if you're willing to use intrinsics. In particular this kind of operation is well-suited for compress-type operations, available as a first-class operation in at least AVX512, SVE and RVV; you can also emulate them reasonably quickly on NEON and AVX2.
Here's an example, building on the OP's work:
For me it's about 25% less time than the branchless version with 1,000,000 elements, and 60% less with 10,000 elements where memory bandwidth effects are less relevant.Nice! I saw the code and thought, I bet there’s a way to do some SIMD here… never touched intrinsics in Rust before so I really appreciate you writing it up!
See the following pdf for example on how to do this with SSSE3 (pages 104-133) or even SSE2 (pages 151-173)
https://deplinenoise.files.wordpress.com/2015/03/gdc2015_afr...
Great link, nice find!
This is interesting. So at a certain scale, CPU optimization becomes irrelevant because you're just waiting for new data to come in?
Memory access patterns are usually the main reason for low performance and should be optimized first because it's a great low hanging fruit (and usually the bigger problem is latency, not bandwidth).
And don't think of waiting for memory as making CPU optimizations irrelevant, but instead as an oppurtinity to hide more CPU operations in the remaining 'memory access gaps' (e.g. the CPU won't simply stop working when waiting for data to be loaded from memory, it can continue with other things that don't depend on that data).
I heard there's a way to arrange code such that the compiler can autobectorize easier. I wonder if there's a way to do that here?
Would probably have to pass `-C target-cpu=native` to cargo so that llvm is allowed to use AVX512.
Good question. I personally doubt that the compress instruction is easy to coax compilers into generating, as there are many edge cases to consider.
For example, you'll notice here that we perform a full vector store of 8 elements unconditionally, even if only a few of the elements are active. This is safe, though, because the output buffer is as large as the input buffer, and we're chunking by 8, so we'll never trash memory past the end; but this is a tricky analysis. Performance-wise, we rely on the CPU's store buffer to make these overlapping stores cheap.
Instead, you might think that you could just store the elements which are actually active, using a masked store. In fact there is also an intrinsic for this purpose (_mm512_mask_compressstoreu_pd), but it is extremely slow on some CPUs, namely Zen 4, so it's dangerous to use unless you know exactly what CPU you're using. (In my testing, there also seems to be some weird hazard on Zen 5 where multiple memory-destination compress instructions to nearby, even non-overlapping, addresses are serialized. But I haven't looked closer at this.)
I think WUFFS "iterate loops" might help. WUFFS requires that processing a chunk of N items has code to process one at a time, which means it'll work for any N. However you can optionally provide specialisations for doing K at a time and the compiler is responsible for carving the input up as appropriate so e.g. N = K + K + 1 + 1 + 1 your K-at-a-time code runs twice, the extras are handled 1-at-a-time.
So this divides up the problem, the compiler can vectorize your 8-at-a-time code without needing to handle edge cases where N isn't a multiple of 8, and if a later pass notices we actually never end up using those edge cases they're dead code, if it doesn't they're just a rarely-taken branch once.
I believe the bounds check, in particular, is devastating for autovectorization. There are ways around it, but it requires additional code in safe rust.
Edit: actually looks like autovectorization is in play here [1]. Doesn't look like the bounds check gets in the way at all.
[1] https://godbolt.org/z/af4qGba5o
There's no autovectorization there; scalar f64-s just are always stored in xmm registers. And the bounds check is still there.
Compress patterns aren't recognized by any open-source compiler autovectorizer as far as I'm aware of. (I think intel's proprietary C/C++ compiler can?)
Thank you for sharing this. How would you emulate this kind of operation on avx2?
Once you have a mask of the positions you want to compress, you can generate a shuffle index vector from that mask to place the desired elements in the low part of the vector. You can expand the mask into nibble-sized indices using pext/pdep and some magic constants, then expand those nibble-sized indices into a vector of indices to use as the shuffle indices.
Yes, that's one approach. Another reasonable approach is to get out a mask from the comparison using `vmovmskpd` and use that to look up a shuffle constant, since there are only 16 possibilities. This also works well on NEON, although I wonder there whether it'd make more sense to find the shuffle dynamically rather than loading it.
Keep in mind it's UB to be:
> Executing code compiled with target features that the current thread of execution does not support
I.e. calling AVX512 on Neon architecture.
You need to wrap it in target attributes to even dream of it being safe.
This particular UB is not one of the subtle cases. You will almost certainly get illegal instruction signals if you mess this up.
Is rust UB different from C UB? C UB must be avoided at all costs even if you think you know the actual behavior.
It is not.
I'd say unwrap isn't a subtle case either. Yet people see examples and think - "This is the way!"
[dead]
This article is 100% AI written. The data was interesting, the commentary overly verbose and hard to gain useful insights from.
I'm apparently not good at spotting it. I was put off by the overly dramatic presentation. It gets tiring that the author apparently finds this more exciting than I do, and writes like it's enthralling. I just assumed it was an excess of enthusiasm or the first experience with this kind of thing. If it's AI, I'm way behind the game noticing it.
There are lots of indicators in this text, and this breathless presentation is very much how modern LLMs present things.
Claude also loves to describe things as being "real", particularly saying "X is real".
In this case,
> The reallocations were real, but they were never the bottleneck.
There was never any indication or setup in the text that they weren't real, but it's how it justifies wasted effort, it insists that some phenomenon it corrected but failed to solve the problem "was real".
Another giveaway are nonsensical analogies:
> The predictor is like a barista who starts making your usual order the moment you walk in. If you are a regular, this is fantastic: the coffee is ready when you reach the counter. If you order something random every day, the barista keeps pouring drinks into the sink.
If you order "something random every day", then you don't have a usual order for them to be making, it's an analogy that doesn't work.
And of course, the smoking gun is:
> The smoking gun
It probably won't be a good indicator forever as it has been noticed so much, but it's a particular favourite of the current generation of anthropic models.
"The smoking gun" is right there in the text ;) (but also things like "Same million floats. Same threshold. Same function."). Don't know if other models have that same specific style, but it looks very 'claude-y'.
I wouldn't be surprised though if (especially) non-native speakers unconsciously start adopting the Claude writing style when they stare all day long at Claude generated text at work.
Not sure if it was added later, but there's an even more obvious sign: the top of the blog post literally says that an LLM was used to write it.
I would assume non-native speakers talk to Claude in their own language. Now I'm curious if Claude's weird quirks of speech are unique in each language or if they carry over!
I'm German but still do all my computing in English (partly out of habbit, but also because Germanized technical text is usually painful to work with because it's full of anglizisms (is that actually an English word?)).
One nice thing about English is you can just make up words with plausible etymological roots in Latin/French or Old English and often people will know what you mean. In this case though it would probably be spelled "Anglicism".
German language: hold my beer ;)
Hey, this beer glass is cold, lemme get my Handschuh.
This isn't a necessarily true assumption. I my social circles of non native English speakers, most of us use English to talk with models.
I switch languages randomly.
What do Claude's tics look like in your other language? Do they carry over or does each language get its own weird rhetorical flourishes?
> Don't know if other models have that same specific style
An awful lot of the open-weight models also talk in the Claude-y style. Not sure if an artefact of distilling anthropic models, or just a preponderance of slop in the training set...
> I was put off by the overly dramatic presentation. It gets tiring that the author apparently finds this more exciting than I do, and writes like it's enthralling.
That's one of the main tells that AI wrote this. All the stylistic tics that people usually point out combine to make the writing seem more important than it is.
It was fun to read and insightful for me, not too artificial, and not too verbose.
I'm glad my internal AI detector doesn't win over my curiosity to learn.
The main problem with the article is that the idea to influence backend code generation decisions via specific highlevel code constructs is mostly just mystical bullshit (some compilers do detect specific patterns - usually for bit twiddling hacks, but not on a basic level like control flow optimization).
Using a highlevel language construct like "y += (x > 0) as usize;" doesn't "switch on" branchless code just because the source code looks branchless, compilers are not that dumb anymore.
E.g. I bet that writing
...generates the exact same code after optimization, otherwise I would consider that an LLVM bug.The only reliable way is to mostly bypass the optimizer via simd intrinsics, or drop down to assembler, everything else is just cargo culting.
(fwiw I can't shake the feeling now that the article is recycled, I'm pretty sure I saw those exact same code examples in another "branchless" blog post, but maybe for a different language - because the next question was ineviatably "then why is the code using "if" slower? answer: because it also behaves differently). Or maybe I'm just having a strong dejavu ;)
PS:
> fwiw I can't shake the feeling now that the article is recycled
Ok, I remembered wrong. The article I remembered was this: https://tiki.li/blog/blqsort
HN link: https://news.ycombinator.com/item?id=48375445
It's peddling the exact same myth though.
And, on some architectures, y+=(x>0) is branchful!
Yep, “instinct“ and “smoking gun” are LLM favourites.
I find it really annoying when the LLM says “good instinct” as if I’m an animal barely able to think.
Click on their blog index page, see posts going back to early 2010s and use the same writing style. He must have been time travelling and using AI all this time!
I had a look at their blog page out of curiosity, not that you can prove much from the purported dates and text on a blog, which could be edited at any time.
The blog posts from 2010s are in a completely different style and written by a human: https://www.greyblake.com/blog/vim-preview-plugin/ https://www.greyblake.com/blog/how-to-install-firefox-icewea... https://www.greyblake.com/blog/unexpected-ruby-behaviour/ ...
This new blog post is clearly AI edited (probably 'improved' with AI), the old ones are not.
The first link you cited[0] shows up on the Wayback Machine[1] for the first time on 2022-05-16 - more than 11 years after it was purportedly written.
[0] https://www.greyblake.com/blog/vim-preview-plugin/
[1] https://web.archive.org/web/20220516225844/https://www.greyb...
A page shows up in the Wayback Machine when it indexes it for the first time. The website itself goes back a lot if you look at the time. Here is a wayback machine link of a blog post from 2015 which got archived in 2017 for the first time (https://web.archive.org/web/20170607071648/http://greyblake....). Again, if you want to make an intelligent argument, at least know what you're talking about.
I think he's asked it to write in his specific style, or possibly he has edited parts of it to his style, or maybe used AI to generate the initial draft.
Something like that anyway. There are some very clear AI tells (smoking guns if you like), but most of it does not read like the prose AI produces by default.
Author if you are here I am curious about your writing process, and why you didn't remove the obvious AI tells.
> the same writing style
It's "fake corporate enthusiasm" style. LLMs were just trained in it.
I just scrolled through & read the code snippets. Interesting enough solution at end
idk why this is getting downvoted, I also got this sense, plugged it into Pangram and indeed, 80% AI-written score.
I guess that's fine, but after awhile I get a spidey-sense reading something that feels like a Claude session.
Sad to see you getting voted down. But I guess both the pro-AI crowd and anti-AI crowd hate Pangram.
I always got voted down when I posted the evaluation of the parent articles I got from my Ouija board. I just want to help people understand whether they should just reject bad articles, without having to bother reading them.
I'm moving on to evaluating articles with a modified lie detector test and tarot cards, I'm sure that'll help my credibility and give my public rejections more authority.
Do you have evidence Pangram is unreliable? There are independent evaluations [1, 2] showing it works, and it's getting used more and more scientific papers. Have you used it or evaluated it yourself? What do you think these other evaluators are doing or getting wrong?
1: https://bfi.uchicago.edu/insights/artificial-writing-and-aut... 2: https://arxiv.org/pdf/2501.15654
I don't doubt that those detectors are generally correct. Pangram seems to be particularly accurate. I see independent evaluations ranging from 97% accurate to over 99%. Frankly, I somewhat doubt those numbers, but I do agree that LLM usage can be fairly accurately detected.
But procedurally, there are huge issues involved with automated tools used to harm other people. You are one of the 0.5% percent of people whose article was flagged as LLM-generated when it wasn't, one of the false positives. What do you do? Argue? The accusers will claim that you're 99.5% likely to be lying.
It's the same issue we have with automated customer service, automated insurance claims, and so forth. It is usually correct, and terrifically unjust when it fails... at which point there is no recourse. In a perverse sense, its accuracy can be a drawback, because if the false positive rate low enough, nobody is going to believe you when you're falsely accused. And people will be falsely accused.
I think it's ironic that it seems like it capitalizes on the same flaw that most LLM-posting does... "Chat GPT is usually right, I'm going with it." You shouldn't post an LLM article without independently validating its claims, so that there is a responsible person in the loop. The same is true for rejections and accusations, but more so, because they're more damaging.
Hold on, let's not move the goalposts yet. Do you still consider Pangram on par with a "Ouija board" or "tarot cards"?
Sad little world we live in tbh.
because it's not much better than an RNG?
What data supports that conclusion about Pangram?
A good part of the article feels like it was written by Claude indeed:
- "The reallocations were real, but they were never the bottleneck."
- "Note that the villain is not the branch itself. It is the branch that [..]"
- "Same million floats. Same threshold. Same function."
- "Notice the price we paid though."
[flagged]
Agree.
Interesting topic but why destroy your own credibility and reputation by shoveling llm-assisted slop to us here at hn?
The post should be flagged, and in general, i wish hn would adopt a no-tolerance policy to enhanced posting like this.
So what if the original text, if it existed in a human written form at all, had weird textual quirks and prose issues the author wished to hide. That texture's what makes humans interesting to engage with in the first place.
I think I need to build “Hacker News Except All Arguments About Whether Or Not Something Is AI-Written Are Filtered Out”
A much clearer article from yesterday on making casefolding 15x faster by removing an if:
https://github.blog/engineering/architecture-optimization/do...
Discussion: https://news.ycombinator.com/item?id=49127983
Great explanation of why a branchless approach results in such a speed up. I've never really had to deal with performance optimization at this level. Generally it's probably best not to get too involved letting the CPU black box do its thing.
I do wonder, would the performance characteristics of branchless vs branching be consistent across different CPUs/architectures? If you had a CPU that wasn't trying to be fancy with branch prediction, would the regular algo be faster?
CPUs aren't black boxes. They are actually much better documented than almost all the software that runs on them.
If you want to treat the CPU as a black box, trust me you do not want to use a CPU with out a branch predictor, your slow code will run like molasses frozen in antarctica.
The regular algo will be lightyears slower on any CPU that does not have a branch predictor.
Virtually every CPU has branch prediction, going back to at least the original Pentium (1993), maybe earlier.
If you're running on a very old CPU, yes, the regular algo should be faster.
I think the Pentium is more or less the first microprocessor with branch prediction. Certainly the most mainstream.
PowerPC 601 arrived at more or less the same time, and the Alpha 21064 was a year earlier. There were a few minicomputers and mainframes before that with branch predictors.
Arguably the 486 could have done with a branch predictor (even a single entry loop predictor would have helped), and maybe the 386 too. But microcoded CISC designs didn't benefit much from predictors because they have multiple cycles to work it out.
And RISC cpus were in their "branch delay slots are awesome" phase throughout most of the 80s. With a bit of trickery (very simple branch conditions and a 2 phase clock), your classic 5-stage MIPS design can fully hide all branches with just a single branch delay slot, so they were a little slow to adopt predictors.
I get the impression that CPU designers in the 80s and early 90s massively underestimated just how beneficial even a small predictor can be.
Pentium was the first CPU with a branch predictor that many people could afford to buy.
Before dynamic branch prediction, where the prediction for every branch is updated after each branch execution, depending on its history, static branch prediction had been used for decades, since around 1960, typically using the rule that forwards branches are unlikely to be taken, but backwards branches are likely to be taken. An alternative was to have an instruction bit where the compiler stored its prediction about the probability of a branch being taken.
Dynamic branch predictors began to be used since the mid seventies.
I do not remember now if any notable monolithic CPU had a dynamic branch predictor before Pentium, but prior multi-chip CPUs certainly existed.
> Before dynamic branch prediction... static branch prediction had been used for decades.
I'm not sure that's true. At least not the "predict backwards branches are taken" part.
Many RISC cpus did kind of have "predict forwards as not taken", but really thats just speculative execution with the complete absence of any branch prediction at all. IMO "All branches are not taken" is not a prediction.
Actual Static Branch prediction is something that seems to have shown up at the exact same time as dynamic branch prediction (ie Pentium and PowerPC 601). Seems to be more of a "well do speculative execution now, what do we do when there is no dynamic prediction?" thing.
Maybe there is a multi-chip CPU out there that has proper static prediction but not dynamic? But I checked the likely candidate (The RS/6000 aka POWER1) and it doesn't have any prediction at all (just a hole where the static prediction bits will go later). Probably because static prediction requires support for speculative execution, which it doesn't do.
> I get the impression that CPU designers in the 80s and early 90s massively underestimated just how beneficial even a small predictor can be.
It's got a lot to do with how cpu clock speeds were getting way faster, but ram wasn't. That's what makes deeper pipelines attractive, and if you give a cpu a deeper pipeline, it's gonna want a good branch predictor.
I'm more thinking about how MIPS were quite late to branch predictors.
They were shipping the high-performance R4000 and R4400 with 8 stage pipelines and no branch predictors.
They could have really done with a branch predictor, each branch took three cycles (and the branch delay slot could fill only one instruction, often a useless NOP).
The Pentium only had a 5 stage pipeline and massively benefited from its branch predictor. IBM was slapping branch predictors on PowerPC designs with 4 stage integer pipelines. You simply don't need a long pipeline to justify the benefits of a branch predictor.
Cortex M0 and microprocessors generally do not. Cortex M3’s looks nothing like the branch prediction you think of when you think consumer or server CPU. Basically branch prediction requires extra power so it’s excluded or greatly simplified in low power use cases.
I'm not sure how your intuition can be that off, if you don't have a branch predictor then any branching code is going to be even slower than it already is, favouring branchless code even more for obvious reasons.
I say this as someone who is interested in a special type of processor architecture that has no branch prediction at all and would need a branchless subset of Rust to meaningfully program it at high performance.
Why no branch predictor at all? Even a brain-dead one that predicts all branches always/never taken is going to provide some benefit, it's not as if the processor can do anything else while it's waiting.
Or am I missing something?
I note the hazard 3 on the pi Pico rp2350 only predicts a branch if it's the last branch and was taken, ie a single loop. Which seems weird to me, so I'm probably lacking understanding somewhere.
Another recent story from github about case folding as part of code search, the simple version of the code had a couple of ifs, and the branchless version was actually slower.
They have a stupendously fast version and it is also branchless, but it just required more than branchless alone.
I'm fuzzy on the details but I think one of the ifs was an early exit, and without that the loop does a memory assignment on every byte instead of skipping most.
The really fast version was also vectorized. The branchless makes it possible to vectorize, but it was the vectorization that actually made it fast.
I really hope all these guns give up smoking sometime soon...
claude sends its regards
The annoyance is real, and honestly? It's worth sitting with it.
indeed
This problem is called stream compaction and there is a wealth of research on it. The best methods use prefix scan. They first efficiently compute the index in the output array of each element that satisfies the predicate and then they gather them in one linear operation.
Also, I can tell that you are a good writer. You didn't need the LLM to "polish" your text.
There is clearly LLM-prose involved, but it's pretty well done. Here's one example: "The reallocations were real, but they were never the bottleneck."
LLMs love this pattern. Whether one put it into this text, or the author soaked it up and now used it himself, who knows. But it is one of the few things in the post that gives me the ick.
And then, there's the verbosity.
If I had to guess, an LLM was involved, but the author did a good job with manual writing and editing, too.
If your objects are large, I can see why you would compute indices first. But why do that for floats?
Because what makes stream compaction challenging is the loop carried dependency: the location you write to in a given iteration depends on the locations you wrote to in previous iterations. By first creating a lookup table of source -> destination locations you remove the dependency. Then you can apply extremely efficient parallel methods.
I like how we have pretty much established how branchless coding is superior to branched coding. However I wonder if the compiler itself could recognize these patterns and turn branches into branchless instead, rather than making the code harder to read? as removing if conditions of course have a readability impact on the code.
Branchless coding is superior to branched coding whenever the branches are more or less random, which happens frequently when checking some properties of input numbers, like their sign or whether they fall inside certain intervals, or when sorting an array that comes in random order.
When a branch alternative will be taken much more frequently than the other, then branched coding with an "if" becomes superior.
So neither is better in general than the other, whenever the program must choose between alternatives, you must think about whether one is more likely than the other, or if both have similar probabilities.
For instance, when sorting an array, the optimal algorithm is not the same when you expect the input array to have a random order and when you expect it to be already almost sorted.
This is true, but the example given yesterday showed that even if branches can be very well predicted (e.g. processing UTF-8 text which is 99.9999% ASCII), branchless code can result in speedup by making autovectorisation possible.
If the branchless code didn't transform to vector instructions, it would be strictly slower. But if it does, it allows the CPU to work on 16 bytes at a time instead of 1 at a time.
https://github.blog/engineering/architecture-optimization/do...
For sorting, conveniently we always definitely need to look at all the elements at least once anyway, so although even the early introspective sorts from the end of last century aren't designed this way both the Timsort and a modern sort like a PDQ sort will end up making that decision early.
"Oh, this was mostly already sorted, done"
If you meant exactly rather than almost then you can still squeak a small win from having an algorithm which is optimised for this case but the vast bulk of your runtime is eaten by the unavoidable work of checking. "Don't check" is faster but then you're not a sort algorithm at all.
Based on the title, I knew the issue as soon as I looked at the first table. Still, great primer for those who don't know about such CPU shenanigans, and I did appreciate the solution, since I knew high level how to solve it, but didn't come up with an actual piece of code before the author presented theirs.
I didn't know about branch prediction or pipelined CPUs back when I was profiling the code I wrote - honestly it probably would have helped.
I've written something like this in C - including resulting assembler code for ARM and x86.
https://easylang.online/blog/branchless
I've been doing leetcode in Janet in a (sometimes) tacit (variabless), branchless way:
[dead]
Worth noting that as written the "trick" results in memory usage proportional to the size of the input rather than the output. If the filter rejects most of the input the difference could be quite noticeable.
I disagree in the sense that you can rewrite the code to use the trick and also not allocate in advance. Nothing about the trick requires you to allocate up front: before writing to out[n] you can extend the vector if it’s out of bounds. Or, after incrementing n, do out.push(0).
You should try writing it out. Doing it without introducing another unpredictable branch is harder than it looks.
I discussed this with a coworker earlier this week and the best they were able to come up with was
which works but is ugly af imo.Yep realized this after that my second solution (push a 0 if n is incremented) has the same branch prediction problem.
I think yours works. Alternatively in the loop:
In this case the if will be predicted well because it only triggers log(N) times, given how the std lib extends vectors.that's a really clever trick to write to out[n] multiple times but only move the index after the logical condition is true thus ending up with the correct values in out
> The predictor is like a barista who starts making your usual order the moment you walk in. If you are a regular, this is fantastic: the coffee is ready when you reach the counter. If you order something random every day, the barista keeps pouring drinks into the sink.
I laughed out loud reading this. Interesting writeup. I wonder what kinds of tricks like this exist for computation graph compilers like JAX.
Branchless code can indeed sometimes be slower than conventional one, but in this particular case, the article comes to the wrong conclusion. At a 1% kept, the branchless version is slower because it pays the cost of zero-initializing 8 MB of memory when allocating the Vec. This can be easily demonstrated by comparing it with a version that allocates uninitialized memory.
Did you read the article? The author was trying to even out the test cases and successfully did so. He stated that the idiomatic code was faster for the 1 % as you can see by quote below:
Thanks for sharing, optimisations like these are what keeps the fun in programming. I have been optimising my JSONLogic evaluator in rust and used arena allocator and preallocation tricks that gave me good jump in tuning. Let me see if branchless programming techniques can get any further in my case
https://news.ycombinator.com/item?id=48035568
> The reallocations were real, but they were never the bottleneck.
Why do people not write their own blogposts on their own?!
Branchless optimization is such a clean way to squeeze out maximum performance for hot loops. Love this approach.
Even though you used AI to help you with the article, knowing that performance trick is really cool.
"the smoking gun"... Mr. AI.
This is good technical content, but it's obvious that an AI wrote it.
Would PGO figure this out?
Generally most forms of PGO does not try to capture number of mispredicted branches (which isn't the same as how often a branch is taken).
They could.
but.... running PGO is just too much pain.
We can't do it "incrementally", can we? How about combining with LTO?
edit: I was thinking profiling individual module on a test driver and link them after PGO
I don't know if an optimization is allowed to "invent" a write, but I would be surprised if an optimizer goes that far because I have to believe that the number of cases where more writes improve performance are pretty slim.
a simple perf stat should show that the "Keep 50% of random data" case will have insanely more branch mis-predictions that the others.
Keep in mind that this is a synthetic benchmark. The task is to remove outliers and for that the roughly 99% correct guesses of the branch predictor is perfectly fine.
"A branch is cheap. A mispredicted branch is not."
oh hell nah
This is a common pattern a compiler should recognise and optimise into an efficient data and control flow. So much for a sufficiently smart compiler. shrug
[flagged]
[dead]
[dead]
[dead]
[flagged]