FoundationDB is awesome testing wise as they have deterministic simulation testing [1] that can simulate distributed and operating system failures.
> We wanted FoundationDB to survive failures of machines, networks, disks, clocks, racks, data centers, file systems, etc., so we created a simulation framework closely tied to Flow. By replacing physical interfaces with shims, replacing the main epoll-based run loop with a time-based simulation, and running multiple logical processes as concurrent Flow Actors, Simulation is able to conduct a deterministic simulation of an entire FoundationDB cluster within a single-thread! Even better, we are able to execute this simulation in a deterministic way, enabling us to reproduce problems and add instrumentation ex post facto. This incredible capability enabled us to build FoundationDB exclusively in simulation for the first 18 months and ensure exceptional fault tolerance long before it sent its first real network packet. For a database with as strong a contract as the FoundationDB, testing is crucial, and over the years we have run the equivalent of a trillion CPU-hours of simulated stress testing.
Pretty cool. For it to scale they are building their own deterministic hypervisor too [0], but also a new distributed database to support their workloads more efficiently [1].
Type-safe message-passing is such a wonderful programming paradigm - and not just for distributed applications. I remember using QNX back in the 1990s. One of its fabulous features was a C message passing library allowing you to send arbitrary binary structs from one process to another. In the context of realtime software development, you often find yourself having one process that watches for events from a certain device, modify the information somehow, and then pass it on to another process that ends up doing something else. The message-passing idiom was far superior to what was available in Linux at the time (pipes and whatnot) because you were able to work with C structs. It was not strictly type safe (as is the case with FoundationDB’s library), but for the 1990s it was pretty great.
I remnber that ASN.1 does sth similar. You'd give a ASN.1 notation to a language generator (aka producing C) and not have to worry about parsing the actual structure anymore!
Unfortunately Thrift is a dead (AKA "Apache") project and it doesn't seem like anyone since has tried to do this. It probably didn't help that there are so many gaps in that support matrix. I think "Google have made a thing! Let's blindly use it!" also helped contribute to its downfall, despite Thrift being better than Protobuf (it even supports required fields!).
Actually I just took a look at the Thrift repo and there are a surprising number of commits from a couple of people consistently, so maybe it's not quite as dead as I thought. You never hear about people picking it for new projects though.
FB maintains a distinct version of Thrift from the one they gave to Apache. fbthrift is far from dead as it's actively used across FB. However in typical FB fashion it's not supported for external use, making it open source in name (license) only.
As an interesting historical note, Thrift was inspired by Protobuf.
Very true. ASN.1 is mostly not a great fit, yet has been the choice for everything to do with certificates and telecommunication protocols (even the newer ones like 5G for things like RRC AND NGAP) Mostly for bit-level support and especially long-term stability.
* and looking back in time ASN.1 has definetly proven its LTS.
actually never heard of thrift until today, thanks for the insight :)
Honestly, first time I've seen someone praising Thrift in a long time.
Wanted to do unspeakable and evil things to people responsible to choosing it as well as its authors last time I worked on a project that used Thrift extensively.
I'm always hearing about FoundationDB but not much about who uses it. I know Deno and obviously Apple is using it. Who else? I'd love to hear some stories about it.
My company (Matterport, YC Winter '12) uses it to store metadata about 3d models. I really don't have that much to say about it because it's not my primary area of focus, and besides that, has been extremely reliable and hands-off, administration-wise. I particularly love that you can change redundancy modes on the fly, for example those listed here[1], and FDB will automatically re-arrange data to your liking, all without downtime. It handles offline/missing or replacing nodes quite well, and I credit my coworker's great efforts to make it work on top of Kubernetes for making our lives so much easier.
At s2.dev (a serverless datastore for real-time streaming data), we started with DynamoDB for our metadata store, but our access patterns kept running into per-partition throughput limits. We switched to FoundationDB, and it’s been great so far.
There might be a good reason for the lack of stories. FoundationDB runs critical infrastructure I work on, but I never actually have to think about it.
I've never spent less time thinking about a data store that I use daily.
The strangest thing about Flow is that its compiler is implemented in C#. So if you decide to use it in your C++ codebase, you now have a C#/.Net dependency, at least at build time.
It’s also funny because it’s a small, incomplete, incompatible subset of c++… seems like a perfect LLVM / clang rewriter case too, it would be easy to convert and be pure c++. Hell even a clang plugin to put the compile time into one process wouldn’t be awful. But i wonder looking at the rewrites if there’s not a terribly janky way to not need a compiler, if at some runtime cost of contextual control flow info.
At first glance, it looks like Rust's channels with a polymorphic type -- when you receive from a channel, you do match and write branches for each variant of the type.
But I wonder if this can be a better abstraction than async. (And whether I can build something like this in existing Rust.)
It doesn't. It's "promise" based, not "communicating sequential processes". Erlang has more preemptive scheduling, a "thread" can be preempted at any time, here you can only be synchronized when you wait for result. It is called "actor-based", because only functions tagged as "actor" can call waiting functions.
This is more node.js-like communication than erlang.
By they looks of it they changed the word 'async' to 'actor' because they thought it was cool not because it actually uses the actor pattern. Which to me seems to be namespace pollution.
If I were designing it today rather than in... 2008?, I would use the terms 'async' and 'await' because they are a lingua franca now. And for a modern audience that already knows what promises are it probably makes sense to start the explanation with that part. But the thing as a whole was intended to build lightweight asynchronously communicating sequential processes with private state that can be run locally or in a distributed way transparently, restarted on failure, etc. I don't think the choice of terms was obviously a crime at the time.
They build channels on top of these "promises" and "futures" and this made them square into communicating sequential processes category. Also, you can look at promise-future pair as a single-element channel, again, it's CSP.
BTW, Erlang does not implement CSP fully. Its' interprocess communication is TCP based in general case and because of this is faulty.
It is not TCP based. In Erlang processes have mailboxes. But they don't have promises, you send a message and wait for response with timeout or do something else. And TCP is only used between nodes (vm instances). But you can use any communication channel (UDP, unix sockets, tls, serial port, some other process doing funny things).
> Its' interprocess communication is TCP based in general case and because of this is faulty.
What? It's faulty because of TCP? No, in Erlang it is assumed that communication can be faulty for a lot of reasons, so you have to program to deal with that and the standard library gives you tools to deal with this.
That was acknowledged by Erlang's developers before 2012. I remember that ICFP 2012 presentation about Cloud Haskell mentioned that "Erlang 2.0" apparently acknowledged TCP unreliability and tried to work around.
FoundationDB is awesome testing wise as they have deterministic simulation testing [1] that can simulate distributed and operating system failures.
> We wanted FoundationDB to survive failures of machines, networks, disks, clocks, racks, data centers, file systems, etc., so we created a simulation framework closely tied to Flow. By replacing physical interfaces with shims, replacing the main epoll-based run loop with a time-based simulation, and running multiple logical processes as concurrent Flow Actors, Simulation is able to conduct a deterministic simulation of an entire FoundationDB cluster within a single-thread! Even better, we are able to execute this simulation in a deterministic way, enabling us to reproduce problems and add instrumentation ex post facto. This incredible capability enabled us to build FoundationDB exclusively in simulation for the first 18 months and ensure exceptional fault tolerance long before it sent its first real network packet. For a database with as strong a contract as the FoundationDB, testing is crucial, and over the years we have run the equivalent of a trillion CPU-hours of simulated stress testing.
[1]https://pierrezemb.fr/posts/notes-about-foundationdb/#simula...
And they went on to build Antithesis to deliver the same capabilities to other systems, pretty cool stuff!
[1]: https://antithesis.com/company/backstory/
Pretty cool. For it to scale they are building their own deterministic hypervisor too [0], but also a new distributed database to support their workloads more efficiently [1].
[0] https://antithesis.com/blog/deterministic_hypervisor
[1] https://antithesis.com/blog/2025/testing_pangolin
Type-safe message-passing is such a wonderful programming paradigm - and not just for distributed applications. I remember using QNX back in the 1990s. One of its fabulous features was a C message passing library allowing you to send arbitrary binary structs from one process to another. In the context of realtime software development, you often find yourself having one process that watches for events from a certain device, modify the information somehow, and then pass it on to another process that ends up doing something else. The message-passing idiom was far superior to what was available in Linux at the time (pipes and whatnot) because you were able to work with C structs. It was not strictly type safe (as is the case with FoundationDB’s library), but for the 1990s it was pretty great.
I remnber that ASN.1 does sth similar. You'd give a ASN.1 notation to a language generator (aka producing C) and not have to worry about parsing the actual structure anymore!
Literally every schema-based serialisation format does this. ASN.1 is a pretty terrible option.
The best system for this I've ever used was Thrift, which properly abstracts data formats, transports and so on.
https://thrift.apache.org/docs/Languages.html
Unfortunately Thrift is a dead (AKA "Apache") project and it doesn't seem like anyone since has tried to do this. It probably didn't help that there are so many gaps in that support matrix. I think "Google have made a thing! Let's blindly use it!" also helped contribute to its downfall, despite Thrift being better than Protobuf (it even supports required fields!).
Actually I just took a look at the Thrift repo and there are a surprising number of commits from a couple of people consistently, so maybe it's not quite as dead as I thought. You never hear about people picking it for new projects though.
FB maintains a distinct version of Thrift from the one they gave to Apache. fbthrift is far from dead as it's actively used across FB. However in typical FB fashion it's not supported for external use, making it open source in name (license) only.
As an interesting historical note, Thrift was inspired by Protobuf.
Very true. ASN.1 is mostly not a great fit, yet has been the choice for everything to do with certificates and telecommunication protocols (even the newer ones like 5G for things like RRC AND NGAP) Mostly for bit-level support and especially long-term stability. * and looking back in time ASN.1 has definetly proven its LTS.
actually never heard of thrift until today, thanks for the insight :)
Honestly, first time I've seen someone praising Thrift in a long time.
Wanted to do unspeakable and evil things to people responsible to choosing it as well as its authors last time I worked on a project that used Thrift extensively.
How come? I haven't used it for like a decade but I remember it being good.
Lot of network issues coming from Thrift RPC runtime apparently not handling anything well.
I recall threatening I'll rewrite everything with ONC-RPC out of pure pettiness and wish to see the network stack not go crazy.
Reinventing QNX will be revolutionary for decades to come.
I'm always hearing about FoundationDB but not much about who uses it. I know Deno and obviously Apple is using it. Who else? I'd love to hear some stories about it.
Snowflake uses it: https://www.snowflake.com/en/blog/how-foundationdb-powers-sn...
Tigris uses it: https://www.tigrisdata.com/blog/building-a-database-using-fo...
A good collection of papers, blog posts, talks, etc.: https://github.com/FoundationDB/awesome-foundationdb
https://discovery.hgdata.com/product/foundationdb
This "Who is hiring" post for Tesla mentions FoundationDB [0].
Firebolt [1] uses it.
FoundationDB is used at Datadog [2].
[0] https://news.ycombinator.com/item?id=26306170
[1] https://www.firebolt.io/blog/decomposing-firebolt-transactio...
[2] https://news.ycombinator.com/item?id=36576775
Snowflake article from 2018, I wonder if it's still true
The article is pretty vague, but nothing in it expired until at least 2023 :V
Yes. They hire engineers specifically to work on it.
My company (Matterport, YC Winter '12) uses it to store metadata about 3d models. I really don't have that much to say about it because it's not my primary area of focus, and besides that, has been extremely reliable and hands-off, administration-wise. I particularly love that you can change redundancy modes on the fly, for example those listed here[1], and FDB will automatically re-arrange data to your liking, all without downtime. It handles offline/missing or replacing nodes quite well, and I credit my coworker's great efforts to make it work on top of Kubernetes for making our lives so much easier.
1: https://apple.github.io/foundationdb/configuration.html#choo...
At s2.dev (a serverless datastore for real-time streaming data), we started with DynamoDB for our metadata store, but our access patterns kept running into per-partition throughput limits. We switched to FoundationDB, and it’s been great so far.
Snowflake uses it as primary database for their metadata. https://www.snowflake.com/en/blog/how-foundationdb-powers-sn...
There might be a good reason for the lack of stories. FoundationDB runs critical infrastructure I work on, but I never actually have to think about it.
I've never spent less time thinking about a data store that I use daily.
Griffin Bank UK uses it for our entire system (https://griffin.com)
Apple uses it for iMessage I believe.
From what I've heard El Toro uses it to keep track of the billions of data points it harvests from the world every minute.
It's legacy technology. MongoDB is basically the same thing under the hood, and more "standard".
MongoDB is from 2009, while FoundationDB 2013, so wouldn't the notion of legacy be the reverse of what you wrote?
"Legacy" isn't about age, it's about adoption speed.
("Legacy" products have a negative growth rate.)
The strangest thing about Flow is that its compiler is implemented in C#. So if you decide to use it in your C++ codebase, you now have a C#/.Net dependency, at least at build time.
It’s also funny because it’s a small, incomplete, incompatible subset of c++… seems like a perfect LLVM / clang rewriter case too, it would be easy to convert and be pure c++. Hell even a clang plugin to put the compile time into one process wouldn’t be awful. But i wonder looking at the rewrites if there’s not a terribly janky way to not need a compiler, if at some runtime cost of contextual control flow info.
Not even that, this should more or less be directly translateable to C++ 20 coroutines.
Also of course years older than them.
I wonder why that decision was made. I know why I, a C# developer, would make that decision, but why Apple?
The original developers (before Apple bought the company) used Visual Studio on Windows
This entire codebase was acquired by apple in a state of substantial completion and since then relatively little has changed.
Someone knew C# and was good at parsers, would be my guess. It could have just as easily been Scala or something else.
At first glance, it looks like Rust's channels with a polymorphic type -- when you receive from a channel, you do match and write branches for each variant of the type.
But I wonder if this can be a better abstraction than async. (And whether I can build something like this in existing Rust.)
how does this compare to the inbox and supervisor model of erlang/elixir?
It doesn't. It's "promise" based, not "communicating sequential processes". Erlang has more preemptive scheduling, a "thread" can be preempted at any time, here you can only be synchronized when you wait for result. It is called "actor-based", because only functions tagged as "actor" can call waiting functions.
This is more node.js-like communication than erlang.
By they looks of it they changed the word 'async' to 'actor' because they thought it was cool not because it actually uses the actor pattern. Which to me seems to be namespace pollution.
If I were designing it today rather than in... 2008?, I would use the terms 'async' and 'await' because they are a lingua franca now. And for a modern audience that already knows what promises are it probably makes sense to start the explanation with that part. But the thing as a whole was intended to build lightweight asynchronously communicating sequential processes with private state that can be run locally or in a distributed way transparently, restarted on failure, etc. I don't think the choice of terms was obviously a crime at the time.
Unfounded guess, they probably didn't want to bump into the new C++ keywords for async/await.
They build channels on top of these "promises" and "futures" and this made them square into communicating sequential processes category. Also, you can look at promise-future pair as a single-element channel, again, it's CSP.
BTW, Erlang does not implement CSP fully. Its' interprocess communication is TCP based in general case and because of this is faulty.
It is not TCP based. In Erlang processes have mailboxes. But they don't have promises, you send a message and wait for response with timeout or do something else. And TCP is only used between nodes (vm instances). But you can use any communication channel (UDP, unix sockets, tls, serial port, some other process doing funny things).
> Its' interprocess communication is TCP based in general case and because of this is faulty.
What? It's faulty because of TCP? No, in Erlang it is assumed that communication can be faulty for a lot of reasons, so you have to program to deal with that and the standard library gives you tools to deal with this.
There is no such thing as "Communicating Sequential Processes with faulty channels and processes." I tried to find something like that, fruitlessly.
This means that Erlang does not implement CSP, it implements something else.
Again, general case of communication between Erlang processes includes communication between processes on different machines.
> BTW, Erlang does not implement CSP fully.
Specific evidence?
> Its' interprocess communication is TCP based in general case
No, it is not. Only between machines is that true.
> and because of this is faulty.
LOL, no. Why are you rolling with "speaking a whole lot of BS based on ignorance" today?
On the other hand, I now understand that one impediment to Elixir adoption is apparently "people repeating a lot of bullshit misinformation about it"
The general case includes same-VM processes, different VM processes and also different VMs on different machines.
TCP is unreliable: https://networkengineering.stackexchange.com/questions/55581...That was acknowledged by Erlang's developers before 2012. I remember that ICFP 2012 presentation about Cloud Haskell mentioned that "Erlang 2.0" apparently acknowledged TCP unreliability and tried to work around.
Here, page 31 on: https://wiki.haskell.org/wikiupload/4/46/Hiw2012-duncan-cout...
Erlang circa 2012 was even less reliable than TCP on which its interprocess communication was based.
Namely, TCP allows for any prefix of messages m1,m2,m3... to be received. But Erlang circa 2012 allowed for m1,m3... received, dropping m2.
It may be not case today, but it was case about ten years ago.
Ok a related note, how does it compare to SeaStar?
iirc there was a ticket/doc about FoundationDB deprecating usage of this and moving to C++ coroutines.
Maybe this? https://forums.foundationdb.org/t/swift-or-c-20-coroutine-wh...
How did they come up with such an original and unique name? Apple does it again.
FoundationDB was originally a startup, purchased by Apple in 2015.