When doing web development I will occasionally connect my local code base to a remote SQL server via SSH.
This adds enough latency to be noticeable and I’ve found pages that were “OK” in prod that were unbearable in my local environment. Most of the time it was N+1 queries. Sometimes it was a cache that wasn’t working as intended. Sometimes it simply was a feature that “looked cool” but offered no value.
I’m not sure if there is a proxy that would do this locally but I’ve found it invaluable.
I do networked game development on Windows and I've found the clumsy program to be very valuable to simulate adverse network conditions. You can set it up to simulate arbitrary network latency, packet loss and so forth.
I’m not sure if you’re saying the latency was introduced in client <-> server hops or server <-> db hops, but chrome dev tools (I’m sure all browsers too) can simulate different network conditions with a few clicks! Useful for something similar to what you’ve said, but in the end I think you meant server <-> db latency is what you want to inject
> I’m not sure if there is a proxy that would do this locally but I’ve found it invaluable.
If you're on Linux, you can use iptables to randomly drop a fraction of packets to simulate bad connections - even for localhost. The TCP retransmits will induce a tunable latency. You have to be careful with this on a remote host or you may find yourself locked out, unless you can reboot out of band
macOS has embedded feature that allows slowing down network calls. Similar to the IP/netfilter suite Linux has. PF handles most of the magic behind the scenes.
Even better because there is an UI to configure it. You just need to download Xcode additional tools package, there is a NetworkConditioner.prefpane inside. Install that and various settings will show up in the regular System Settings/Preferences...
In my opinion, NOP and MOV, which are recommended in TFA for slowing down, are the worst possible choices.
The authors have tested a rather obsolete CPU, with a 10-year-old Skylake microarchitecture, but more recent Intel/AMD CPUs have special optimizations for both NOP and MOV, executing them at the renaming stage, well before the normal execution units, so they may appear to have been executed in zero time.
For slowing down, one could use something really slow, like integer division. If that would interfere with the desired register usage, other reliable choices would be add with carry or perhaps complement carry flag. If it is not desired to modify the flags, one can use a RORX instruction for multiple bit rotation (available since Haswell, but not in older Atom CPUs), or one could execute BSWAP (available since 1989, therefore it exists in all 64-bit CPUs, including any Atom).
They care about exact slowdown factors, and they don’t like the instructions that use any of the CPU’s execution resources because the program’s real work interferes with the slowdown factor making it harder to control.
The NOPs in effect use up a small fraction of the instruction decode bandwidth, and if they insert enough NOPs they can reduce the number of real instructions that are issued per cycle and slow down the program with a fine degree of precision.
Many recent CPUs eliminate NOPs without executing them, so using NOP may work on a CPU but be ineffective on another.
A double BSWAP is equivalent with a NOP, but no existing CPU does any special optimization for it and it is very unlikely that any future CPU would do something special with it, as it is mostly obsolete (nowadays MOVBE is preferable instead of it).
NOP cannot ensure a certain slow-down factor, except on exactly the same model of CPU.
> Many recent CPUs eliminate NOPs without executing them
The elimination will still take some amount of time, and the smaller this amount is, the better, because it allows dialing in the slowdown more precisely. Of course how many NOPs you need for a particular basic block needs to be measured on the CPU you're profiling on, but that's also the case if you use a more consistently slow instruction, since the other instructions won't necessarily maintain a consistent speed relative to it.
That's not necessarily true. Many modern CPUs have uop caches, and it's reasonable to assume implementations will eliminate NOPs before placing lines in the uop cache. A hit on the uop cache loses any slowdown you hoped to achieve since now the NOPs are neither fetched nor decoded.
That's an interesting question, because the uop cache is filled early in the pipeline, but zero-idioms are only detected around the rename stage.
They might be able to skip a plain 0x90, but something like mov rax, rax would still emit a uop for the cache, before being eliminated later in rename. So at best it would be a fairly limited optimization.
It's also nice because rename is a very predictable choke point, no matter what the rest of the frontend and the backend are busy doing.
There's a whole niche in cryptography called verifiable delay functions (VDFs) if you want a big rabbit hole to go down.
The idea is that these are unavoidably slow and preferably non-parallelizable to compute one way, but fast or near instantaneous to verify the result. Examples include the Weslowski VDFs based on similar math to RSA, MIMC, and the use of zero knowledge proofs to provide proof of slow computations.
For this purpose though, computation should be minimized to avoid spilling registers and having more complicated performance side effects than just the intended direct delay.
From what I understand if you really want to mess with some performance stuff just throw a random AVX or even better AVX512 workload in the critical path of all cores. Could be something really simple, like expanding out a normal cmov, jmp statement, or just regular integer addition to be needlessly AVX involved. Depending on the CPU this can apparently downclock the core, change the power delivery, and potentially result in pipeline stalls on other cores on the same die.
Not true. To downclock the core you will need to have a sustained AVX workload that spends the power budget, and probably over many many cores and not a single one, and even then it's going to be undeterministic.
RDTSC or RDTSCP, like also CPUID, work certainly very well for slowing down a program.
However, like integer division, they may clobber registers that the program wants to use for other purposes.
For great slow-downs when the clobbered registers do not matter, I think that CPUID is the best, as it serializes the execution and it has a long execution time on all CPUs.
For small slow-downs I think that BSWAP is a good choice, as it modifies only 1 arbitrary register without affecting the flags, and it also is a less usual instruction so it is unlikely that it will ever receive special optimizations, like NOP and MOV.
However, multiple BSWAPs must be used, to occupy all available execution ports, otherwise if there is any execution port not occupied by the rest of the program the BSWAP may be executed concurrently, not requiring any extra time.
The purpose of the Turbo button on PCs was indeed to slow it down, originally to the 4.77 MHz of the original 8088, for compatibility. For marketing reasons it was designated “turbo” instead of something indicating a slow-down. Turbo mode was always the default, since that was the standard speed of the respective CPU.
> we can estimate whether an optimization is beneficial before implementing it. Coz simulates it by slowing down all other program parts. The part we think might be optimizable stays at the same speed it was before, but is now virtually sped up, which allows us to see whether it gives enough of a benefit to justify a perhaps lengthy optimization project.
Really interesting idea. I suppose the underlying assumption is that speeding up a function might reveal a new performance floor that is much higher than the speedup would suggest. Spending time to halve a function's time only to discover that overall processing time only decreased slightly because another function is now blocking would indeed be quite bad.
Not sure how this handles I/O, or if it is enough to simply delay the results of I/O functions to simulate decreased bandwidth or increased latency.
When I had to do this, I just incremented a register.
xor rcx,rcx
loop:
inc rcx
cmp rcx, -1
jne loop
Something like that roughly. Except, I didn't count all the way to -1, just roughly half way through the 64bit space. It stalls for a couple of minutes.
Now, I wasn't doing this for stressing profilers or anything fancy like that. I was looking into malware delayed execution techniques. Sleep() and other "Dear system, I'm gonna nap for a bit, wake up later" routines have the downside of being api or system service calls, and detecting branching decisions after sleeping is a tell-tale red flag for anti-malware systems.
Sure, you can detect techniques like this as well, but I figured I could insert idempotent instructions in the loop's inner basic block that makes it look like it's crunching data for legitimate reasons instead. It just needs to delay execution long enough to fool sandbox automated analysis. I've thought about delaying/frustrating human analysis too, and I'm sure better minds than mine have thought of better solutions, but making this part of an unpacking routine that relies on the computed value to decrypt malicious code seems to be the obvious thing to do. Which again, I'm sure it's been done, but doing it yourself and figuring out the anti-anti-analysis techniques is fun.
Kind of weird that NOP actually slows down the pipeline, as I'd think that would be the easiest thing to optimize out of the pipeline, unless instruction fetch is one of the main limiting factors. Is it architecturally defined that NOP will slow down execution?
I think it would be easy, but still not worth the transistors. Think of it: what programs contain lots of NOPs? Who, desiring to write a fast program, sprinkles their code with NOPs?
It’s not worth optimizing for situations that do not occur in practice.
The transistors used to detect register clearing using XOR foo,foo, on the other hand, are worth it, as lots of code has that instruction, and removing the data dependency (the instruction technically uses the contents of the foo register, but its result is independent of its value) can speed up code a lot.
On CPUs with variable instruction length, like the Intel/AMD CPUs, many programs have a lot of NOPs, which are inserted by the compiler for instruction alignment.
However those NOPs are seldom executed frequently, because most are outside of loop bodies. Nevertheless, there are cases when NOPs may be located inside big loops, in order to align some branch targets to cache line boundaries.
That is why many recent Intel/AMD CPUs have special hardware for accelerating NOP execution, which may eliminate the NOPs before reaching the execution units.
It depends on the CPU. On some CPUs a NOP might take the same time as an ADD and it might have the same throughput per clock cycle as ADD.
However, there are CPUs among the Intel/AMD CPUs that can execute up to a certain number of consecutive NOPs in zero time, i.e. they are removed from the instruction stream before reaching the execution units.
In general, no instruction set architecture specifies the time needed to execute an instruction. For every specific CPU model you must search its manual to find the latency and throughput for the instruction of interest, including for NOPs.
Some CPUs, like the Intel/AMD CPUs, have multiple encodings for NOP, with different lengths in order to facilitate instruction alignment. In that case the execution time may be not the same for all kinds of NOPs.
I think so, as in "make sure all other stuff has run before calling the NOP finished". Otherwise, it would just skip past it and it would have no effect if placed in a loop, so it would be eating memory for no use at all.
Kinda related: sending a SIGSTOP to your datastore is a nice way to discover one of the failure modes of your architecture. (Use SIGCONT to make the server continue.)
After a recent upgrade, I've been pondering how to spot test optimizations on my workstation, which is now much faster than the average target system. (Keeping a slow computer around and continually moving my code to it would be effective, but inconvenient.)
My first idea is to taskset the code to a particular CPU core, and see if linux will let me put that core in a low frequency mode. Has anyone here done this on AMD hardware? If so, were the available frequencies slow enough to be useful for this purpose?
Taking this approach seems to be effective at surfacing issues that otherwise wouldn't show up in regular testing. I could see this being useful if it was standardized to help identify issues before they hit production.
Causal profiling is something I infrequently use but I have been able to apply it outside of micro benchmarks in an orchestration system.
If you have a business process that is complex and owned by many different groups then causal profiling can be a good tool for dealing with the complexity. For large workflows in particular this can be powerful as the orchestration owner can experiment without much coordination (other than making sure the groups know/agree that some processing might be delayed).
I fondly remember using a program called MoSlow to slow down older DOS games that were used to using every cycle they could get, and so were unplayable when I got a Pentium computer. I joked about how I wish I could find the corresponding "MoFast" program.
When doing web development I will occasionally connect my local code base to a remote SQL server via SSH.
This adds enough latency to be noticeable and I’ve found pages that were “OK” in prod that were unbearable in my local environment. Most of the time it was N+1 queries. Sometimes it was a cache that wasn’t working as intended. Sometimes it simply was a feature that “looked cool” but offered no value.
I’m not sure if there is a proxy that would do this locally but I’ve found it invaluable.
I'm a big fan of Toxiproxy for these kinds of things:
https://github.com/Shopify/toxiproxy
I do networked game development on Windows and I've found the clumsy program to be very valuable to simulate adverse network conditions. You can set it up to simulate arbitrary network latency, packet loss and so forth.
https://jagt.github.io/clumsy/
You can use the tc command from the netem package for those wondering how to achieve this on Linux. https://man7.org/linux/man-pages/man8/tc-netem.8.html
I've used `tc` about three times in the last 15 years. Every time I have to relearn it.
+1 for clumsy On Windows I've also used Heavy Load, Netlimiter for more fine-grained control and Microsoft's Driver Verifier.
This looks interesting. I'll check it out, thank you!
I’m not sure if you’re saying the latency was introduced in client <-> server hops or server <-> db hops, but chrome dev tools (I’m sure all browsers too) can simulate different network conditions with a few clicks! Useful for something similar to what you’ve said, but in the end I think you meant server <-> db latency is what you want to inject
This was the first thing I thought of. Dev Tools in any browser have a boatload of great stuff like this.
> I’m not sure if there is a proxy that would do this locally but I’ve found it invaluable.
If you're on Linux, you can use iptables to randomly drop a fraction of packets to simulate bad connections - even for localhost. The TCP retransmits will induce a tunable latency. You have to be careful with this on a remote host or you may find yourself locked out, unless you can reboot out of band
On FreeBSD I have used https://man.freebsd.org/cgi/man.cgi?dummynet for this exact thing.
macOS has embedded feature that allows slowing down network calls. Similar to the IP/netfilter suite Linux has. PF handles most of the magic behind the scenes.
Even better because there is an UI to configure it. You just need to download Xcode additional tools package, there is a NetworkConditioner.prefpane inside. Install that and various settings will show up in the regular System Settings/Preferences...
I do this with a Makefile that calls “kubectl port-forward.”
In my opinion, NOP and MOV, which are recommended in TFA for slowing down, are the worst possible choices.
The authors have tested a rather obsolete CPU, with a 10-year-old Skylake microarchitecture, but more recent Intel/AMD CPUs have special optimizations for both NOP and MOV, executing them at the renaming stage, well before the normal execution units, so they may appear to have been executed in zero time.
For slowing down, one could use something really slow, like integer division. If that would interfere with the desired register usage, other reliable choices would be add with carry or perhaps complement carry flag. If it is not desired to modify the flags, one can use a RORX instruction for multiple bit rotation (available since Haswell, but not in older Atom CPUs), or one could execute BSWAP (available since 1989, therefore it exists in all 64-bit CPUs, including any Atom).
They care about exact slowdown factors, and they don’t like the instructions that use any of the CPU’s execution resources because the program’s real work interferes with the slowdown factor making it harder to control.
The NOPs in effect use up a small fraction of the instruction decode bandwidth, and if they insert enough NOPs they can reduce the number of real instructions that are issued per cycle and slow down the program with a fine degree of precision.
Many recent CPUs eliminate NOPs without executing them, so using NOP may work on a CPU but be ineffective on another.
A double BSWAP is equivalent with a NOP, but no existing CPU does any special optimization for it and it is very unlikely that any future CPU would do something special with it, as it is mostly obsolete (nowadays MOVBE is preferable instead of it).
NOP cannot ensure a certain slow-down factor, except on exactly the same model of CPU.
> Many recent CPUs eliminate NOPs without executing them
The elimination will still take some amount of time, and the smaller this amount is, the better, because it allows dialing in the slowdown more precisely. Of course how many NOPs you need for a particular basic block needs to be measured on the CPU you're profiling on, but that's also the case if you use a more consistently slow instruction, since the other instructions won't necessarily maintain a consistent speed relative to it.
That's not necessarily true. Many modern CPUs have uop caches, and it's reasonable to assume implementations will eliminate NOPs before placing lines in the uop cache. A hit on the uop cache loses any slowdown you hoped to achieve since now the NOPs are neither fetched nor decoded.
That's an interesting question, because the uop cache is filled early in the pipeline, but zero-idioms are only detected around the rename stage.
They might be able to skip a plain 0x90, but something like mov rax, rax would still emit a uop for the cache, before being eliminated later in rename. So at best it would be a fairly limited optimization.
It's also nice because rename is a very predictable choke point, no matter what the rest of the frontend and the backend are busy doing.
Instead of arguing about it, you people could have run an experiment and written a paper.
Even modern CPUs still have to read the NOPs to skip them, and that's what he was talking about. Decode bandwidth.
There's a whole niche in cryptography called verifiable delay functions (VDFs) if you want a big rabbit hole to go down.
The idea is that these are unavoidably slow and preferably non-parallelizable to compute one way, but fast or near instantaneous to verify the result. Examples include the Weslowski VDFs based on similar math to RSA, MIMC, and the use of zero knowledge proofs to provide proof of slow computations.
For this purpose though, computation should be minimized to avoid spilling registers and having more complicated performance side effects than just the intended direct delay.
Those sound like the space-hard and time-hard hashes, like used for password checking
They're a relative of that, but have stronger linearity and asymmetrical verification time guarantees AFAIK.
Curious, is this what PoW (proof of work) captchas use?
From what I understand if you really want to mess with some performance stuff just throw a random AVX or even better AVX512 workload in the critical path of all cores. Could be something really simple, like expanding out a normal cmov, jmp statement, or just regular integer addition to be needlessly AVX involved. Depending on the CPU this can apparently downclock the core, change the power delivery, and potentially result in pipeline stalls on other cores on the same die.
Not true. To downclock the core you will need to have a sustained AVX workload that spends the power budget, and probably over many many cores and not a single one, and even then it's going to be undeterministic.
Skylake has the exact same optimisations.
RDTSC(P) is pretty slow. I wonder if that would work.
RDTSC or RDTSCP, like also CPUID, work certainly very well for slowing down a program.
However, like integer division, they may clobber registers that the program wants to use for other purposes.
For great slow-downs when the clobbered registers do not matter, I think that CPUID is the best, as it serializes the execution and it has a long execution time on all CPUs.
For small slow-downs I think that BSWAP is a good choice, as it modifies only 1 arbitrary register without affecting the flags, and it also is a less usual instruction so it is unlikely that it will ever receive special optimizations, like NOP and MOV.
However, multiple BSWAPs must be used, to occupy all available execution ports, otherwise if there is any execution port not occupied by the rest of the program the BSWAP may be executed concurrently, not requiring any extra time.
For the Commodore 64 there was a product called the C64 Snail which could slow it down.
Later on the early PCs we had a Turbo Button, but since everyone had it in Turbo mode all the time it essentially was a way to slow down the machine.
EDIT: Found an image of what I remember as the "C64 Snail". It is called "BREMSE 64" (which is German for brake) in the image.
https://retroport.de/wp-content/uploads/2018/10/bremse64_rex...
The purpose of the Turbo button on PCs was indeed to slow it down, originally to the 4.77 MHz of the original 8088, for compatibility. For marketing reasons it was designated “turbo” instead of something indicating a slow-down. Turbo mode was always the default, since that was the standard speed of the respective CPU.
Wow thanks, this makes so much sense! I spent way too many brain cycles wondering about that button when I was a teenager.
Interesting.
I had the impression, the turbo button was created to slow down new PCs, so they could run old software that relied heavily on CPU speed.
Yes, originally it was added to slow down faster XTs to exactly the 4.77 MHz of the original IBM XT.
With the AT it usually slowed down to some arbitrary frequency and it was more like a gimmick.
Afaik it was to interface with bit banged hardware that couldn't communicate at the faster rates. One would turn off turbo to print.
That is why I think it made sense when it switched to exactly 4.77 MHz but the arbitrary frequencies introduced later were just a toy.
> we can estimate whether an optimization is beneficial before implementing it. Coz simulates it by slowing down all other program parts. The part we think might be optimizable stays at the same speed it was before, but is now virtually sped up, which allows us to see whether it gives enough of a benefit to justify a perhaps lengthy optimization project.
Really interesting idea. I suppose the underlying assumption is that speeding up a function might reveal a new performance floor that is much higher than the speedup would suggest. Spending time to halve a function's time only to discover that overall processing time only decreased slightly because another function is now blocking would indeed be quite bad.
Not sure how this handles I/O, or if it is enough to simply delay the results of I/O functions to simulate decreased bandwidth or increased latency.
When I had to do this, I just incremented a register.
xor rcx,rcx
loop:
inc rcx
cmp rcx, -1
jne loop
Something like that roughly. Except, I didn't count all the way to -1, just roughly half way through the 64bit space. It stalls for a couple of minutes.
Now, I wasn't doing this for stressing profilers or anything fancy like that. I was looking into malware delayed execution techniques. Sleep() and other "Dear system, I'm gonna nap for a bit, wake up later" routines have the downside of being api or system service calls, and detecting branching decisions after sleeping is a tell-tale red flag for anti-malware systems.
Sure, you can detect techniques like this as well, but I figured I could insert idempotent instructions in the loop's inner basic block that makes it look like it's crunching data for legitimate reasons instead. It just needs to delay execution long enough to fool sandbox automated analysis. I've thought about delaying/frustrating human analysis too, and I'm sure better minds than mine have thought of better solutions, but making this part of an unpacking routine that relies on the computed value to decrypt malicious code seems to be the obvious thing to do. Which again, I'm sure it's been done, but doing it yourself and figuring out the anti-anti-analysis techniques is fun.
Kind of weird that NOP actually slows down the pipeline, as I'd think that would be the easiest thing to optimize out of the pipeline, unless instruction fetch is one of the main limiting factors. Is it architecturally defined that NOP will slow down execution?
I think it would be easy, but still not worth the transistors. Think of it: what programs contain lots of NOPs? Who, desiring to write a fast program, sprinkles their code with NOPs?
It’s not worth optimizing for situations that do not occur in practice.
The transistors used to detect register clearing using XOR foo,foo, on the other hand, are worth it, as lots of code has that instruction, and removing the data dependency (the instruction technically uses the contents of the foo register, but its result is independent of its value) can speed up code a lot.
On CPUs with variable instruction length, like the Intel/AMD CPUs, many programs have a lot of NOPs, which are inserted by the compiler for instruction alignment.
However those NOPs are seldom executed frequently, because most are outside of loop bodies. Nevertheless, there are cases when NOPs may be located inside big loops, in order to align some branch targets to cache line boundaries.
That is why many recent Intel/AMD CPUs have special hardware for accelerating NOP execution, which may eliminate the NOPs before reaching the execution units.
MIPS required NOP insertion for data hazards and the branch delay slot if a productive instruction couldn't be placed in the slot.
It depends on the CPU. On some CPUs a NOP might take the same time as an ADD and it might have the same throughput per clock cycle as ADD.
However, there are CPUs among the Intel/AMD CPUs that can execute up to a certain number of consecutive NOPs in zero time, i.e. they are removed from the instruction stream before reaching the execution units.
In general, no instruction set architecture specifies the time needed to execute an instruction. For every specific CPU model you must search its manual to find the latency and throughput for the instruction of interest, including for NOPs.
Some CPUs, like the Intel/AMD CPUs, have multiple encodings for NOP, with different lengths in order to facilitate instruction alignment. In that case the execution time may be not the same for all kinds of NOPs.
I think so, as in "make sure all other stuff has run before calling the NOP finished". Otherwise, it would just skip past it and it would have no effect if placed in a loop, so it would be eating memory for no use at all.
> I think so, as in "make sure all other stuff has run before calling the NOP finished".
Is this related to speculative execution? The high level description sounds like NOP works as sync points.
Eating memory alone may have the desired effect. The memory bandwidth of a cpu is not infinite.
Yeah, just decode. But that's nice because the effect is independent of the backend's state.
Kinda related: sending a SIGSTOP to your datastore is a nice way to discover one of the failure modes of your architecture. (Use SIGCONT to make the server continue.)
After a recent upgrade, I've been pondering how to spot test optimizations on my workstation, which is now much faster than the average target system. (Keeping a slow computer around and continually moving my code to it would be effective, but inconvenient.)
My first idea is to taskset the code to a particular CPU core, and see if linux will let me put that core in a low frequency mode. Has anyone here done this on AMD hardware? If so, were the available frequencies slow enough to be useful for this purpose?
Bring back the turbo button!
https://en.wikipedia.org/wiki/Turbo_button#Purpose
Interesting article OP. There was a similar post on Hacker News recently about using this approach on a Postgres database:
- Making Postgres slower (https://news.ycombinator.com/item?id=44704736)
Taking this approach seems to be effective at surfacing issues that otherwise wouldn't show up in regular testing. I could see this being useful if it was standardized to help identify issues before they hit production.
Causal profiling is something I infrequently use but I have been able to apply it outside of micro benchmarks in an orchestration system.
If you have a business process that is complex and owned by many different groups then causal profiling can be a good tool for dealing with the complexity. For large workflows in particular this can be powerful as the orchestration owner can experiment without much coordination (other than making sure the groups know/agree that some processing might be delayed).
slowing down processors also make them low power. https://github.com/hatonthecat/Solar-Kernel/blob/main/semi-a...
I fondly remember using a program called MoSlow to slow down older DOS games that were used to using every cycle they could get, and so were unplayable when I got a Pentium computer. I joked about how I wish I could find the corresponding "MoFast" program.
How: Just cross-compile it to Windows and run it there. Obviously with a virus scanner. Immediate 10x slowdown.
Why: 100x more users instantly
Well shucks that's simple you just gotta go where you want the slowdown and do something like
for i = 1 to 100
nextof course with modern cpus you gotta adjust your i to be something more reasonable, like
for i = 1 to 1000000000
works every time