OK, I have a fun one. I was a rookie back then, it was 2007, and the cloud was only beginning to take off. My company had a physical server room.
Automated CI was not quite as big back then as it is now; we had some in-house CI-ish facilities, one of which was buildbot: a small Python script that continuously built all of our projects one after another, reporting successes/failures in a simple UI.
One day I was puzzled to discover that buildbot hung trying to build a tiny Java applet. Just a handful of .java files. But it had been stuck for more than four hours at that point.
I ssh’d into buildbot and looked at what exactly the hung process was doing. It was not javac. It was jarsigner.
Then it dawned on me. jarsigner needed some crypto-safe randomness, it was trying to get it by default from /dev/random, not from /dev/urandom, and the machine ran out of entropy!
I ticketed a change, but as an interim measure, I went to the server room, switched the console to buildbot’s server, and moved the mouse vigorously for some seconds. It proceeded.
Hit something similar a fair time ago, and installed havenged¹ to stop it repeating. Less necessary now as the kernel has its own similar (jitter based) internal entropy source, and /dev/random has been made non-blocking anyway by adding a little extra compute to “stir the pot” (like /dev/urandom did anyway, but less cryptographically safely so).
Hey, creator here. I’ll look through these today to see what links are dead and try and grab archived ones. Will work on getting archives of the rest of them as well.
OK, I have a fun one. I was a rookie back then, it was 2007, and the cloud was only beginning to take off. My company had a physical server room.
Automated CI was not quite as big back then as it is now; we had some in-house CI-ish facilities, one of which was buildbot: a small Python script that continuously built all of our projects one after another, reporting successes/failures in a simple UI.
One day I was puzzled to discover that buildbot hung trying to build a tiny Java applet. Just a handful of .java files. But it had been stuck for more than four hours at that point.
I ssh’d into buildbot and looked at what exactly the hung process was doing. It was not javac. It was jarsigner.
Then it dawned on me. jarsigner needed some crypto-safe randomness, it was trying to get it by default from /dev/random, not from /dev/urandom, and the machine ran out of entropy!
I ticketed a change, but as an interim measure, I went to the server room, switched the console to buildbot’s server, and moved the mouse vigorously for some seconds. It proceeded.
Hit something similar a fair time ago, and installed havenged¹ to stop it repeating. Less necessary now as the kernel has its own similar (jitter based) internal entropy source, and /dev/random has been made non-blocking anyway by adding a little extra compute to “stir the pot” (like /dev/urandom did anyway, but less cryptographically safely so).
----
[1] https://manpages.debian.org/bookworm/haveged/haveged.8.en.ht...
Came across this list in the comments of this discussion, lots of great stories here! https://news.ycombinator.com/item?id=44466030
My personal list of stories of this kind includes:
- Car allergic to vanilla ice cream: https://www.cs.cmu.edu/~wkw/humour/carproblems.txt
- Can't log in when standing up: https://www.reddit.com/r/talesfromtechsupport/comments/3v52p...
- OpenOffice won't print on Tuesdays: https://bugs.launchpad.net/ubuntu/+source/cupsys/+bug/255161...
- Wi-Fi only works when it's raining: https://predr.ag/blog/wifi-only-works-when-its-raining/
I’ll take a look at these and add some I don’t have!
Hey this is my site! Thanks for sharing!
It’s been a bit since I’ve added stories. I’ll need to add some more with this renewed interest.
Let me know if have a good story!
Super fun to read, but many stories are deadlinks, maybe they are archived? It would be a god feature to links to archives and/or archive when posted.
Hey, creator here. I’ll look through these today to see what links are dead and try and grab archived ones. Will work on getting archives of the rest of them as well.
https://beza1e1.tuxen.de/lore/ is the original idea
Very cool!
Looks like a cleaner version of The Daily WTF: https://thedailywtf.com
I do sometimes find good stories on The Daily WTF, but their focus is a bit different from 500 Mile Email.
Yes. Sorry. I wasn't clear. That's what I meant by "cleaner." Good job!