Built in replicated database and filesystem is pretty rad. Fly leaves this as an exercise for the user.
You should license the backend to enterprises that want a cheaper/lighter weight option to kubernetes for the surge of agentically coded internal apps.
Thanks! I tried to make the sandbox-eval stdin interface as simple as possible so you could easily plug it anywhere. The executable docs run an actual full shell in a VM, since they're so fast and cheap to spin up- I hope the example made it clear how to use this feature!
This looks awesome! I really love the clean and minimalist website design too.
I've been excited about using (lib)krun as a drop in replacement for crun/runc for container engines, to make them run qemu microvms. I'm exploring what it could look like if every process of an OS ran like that. I envy you, would love to work on something like this!
Hey this is amazing! I see the sqlite db is replicated but how does it handle multiple instances as writers? Curious if the underlying sqlite engine is syzy or if there's other restrictions to be aware of
Thanks! Yes, the write-replication part is syzy, which hooks into the standard unmodified sqlite engine that reads and writes normal sqlite db files locally - syzy just keeps multiple instances in sync by capturing writes and applying them elsewhere.
It handles multiple writers by capturing changes as logical operations, then resolving conflicts in a globally-deterministic order based on their contents based on a CRDT model.
There are some limitations (see https://github.com/wjordan/syzy/blob/main/sqlite/docs/LIMITA...), mostly around primary keys (they have to be non-NULL) and UNIQUE constraints (there's some subtle differences in behavior between nullable and NOT NULL uniqueness). But in general, the goal is for all of SQLite, DML and DDL, to just work normally!
Pretty much! It's not dynamic memory hotplug or resizing, just dynamic billing. Usage is based on the actual RSS used by the app so you can think of the VM's defined size as just a limit, for both CPU and RAM. This means you don't have to spend time profiling and right-sizing instances to optimize costs, just make your application fast and efficient and don't worry about it!
On the HN frontpage is Superlogical (I have no affiliation) right now, and I used the shell provided by the Kedge News demo to `ssh superlogical.jobs` from my browser on a phone on an instance started seconds earlier. :)
I didn't immediately spot the durability contract for volumes -- local NVMe like Fly, or sitting on some underlying system like EBS?
> I didn't immediately spot the durability contract for volumes -- local NVMe like Fly, or sitting on some underlying system like EBS?
Persistent volumes use local NVMe that continuously syncs to object storage, so a similar architecture to Fly Sprites. It automatically recovers from host failures (with on-demand lazy restore), and you can manually migrate stable members to another region if needed.
I've positioned persistent volumes as an escape hatch for when you need an existing database app running in a VM, or when performance requirements outgrow the built-in replicated SQLite.
Gotcha, interesting. I would love it to death if there were a way to run some CLI, maybe on the system, that gave you information on the current replication state of the disk!
Two questions about the fork boundary. The snapshot is taken once when the process calls listen, so every later instance is a clone of the same frozen memory image, and anything seeded during init is identical across every clone in every region. That is exactly what the Linux vmgenid driver exists for, and it is why Lambda SnapStart had to ship explicit reseeding guidance. I could not find entropy, RNG or reseed anywhere in the runtime, performance or security docs, so is there a post-restore reseed hook, and does it fire before the first accept?
The other one is syzy. CRDT.md says eviction is operator-driven, that PRUNING.md is a design-stage contract and not yet code, and that a replica lagging beyond the hold window may transiently observe the departing duplicate, while the product headline offers scale to zero when idle, which mints offline replicas continuously. Who calls syzy_evict in the managed product?
These are both later-stage detail questions for when this project is further hardened for production workloads, but here's where each of these currently stands:
- Yes, there is a post-restore kernel reseed hook that happens after fork, and just before the trapped listen is continued. It doesn't use the vmgenid driver, it just calls the RNDRESEEDCRNG ioctl, so it would not be compatible with applications that depend on vmgenid notifications to reseed userspace PRNGs.
- In Syzy, the full pruning implementation is deferred for now; at the managed product's current small scale, it can simply eat the tiny overhead costs of un-collected garbage, while the user is only billed for the logical storage space consumed.
Built in replicated database and filesystem is pretty rad. Fly leaves this as an exercise for the user.
You should license the backend to enterprises that want a cheaper/lighter weight option to kubernetes for the surge of agentically coded internal apps.
https://kedge.dev/docs/sandboxes
Caught my eye with this one. I'll be taking a closer look as this is one of those "I'd rather let someone else do it." type infrastructure bits :)
Thanks! I tried to make the sandbox-eval stdin interface as simple as possible so you could easily plug it anywhere. The executable docs run an actual full shell in a VM, since they're so fast and cheap to spin up- I hope the example made it clear how to use this feature!
This is also extremely interesting to me.
An instant, ephemeral cloud sandbox is pretty handy.
This looks awesome! I really love the clean and minimalist website design too.
I've been excited about using (lib)krun as a drop in replacement for crun/runc for container engines, to make them run qemu microvms. I'm exploring what it could look like if every process of an OS ran like that. I envy you, would love to work on something like this!
Hey this is amazing! I see the sqlite db is replicated but how does it handle multiple instances as writers? Curious if the underlying sqlite engine is syzy or if there's other restrictions to be aware of
Thanks! Yes, the write-replication part is syzy, which hooks into the standard unmodified sqlite engine that reads and writes normal sqlite db files locally - syzy just keeps multiple instances in sync by capturing writes and applying them elsewhere.
It handles multiple writers by capturing changes as logical operations, then resolving conflicts in a globally-deterministic order based on their contents based on a CRDT model.
There are some limitations (see https://github.com/wjordan/syzy/blob/main/sqlite/docs/LIMITA...), mostly around primary keys (they have to be non-NULL) and UNIQUE constraints (there's some subtle differences in behavior between nullable and NOT NULL uniqueness). But in general, the goal is for all of SQLite, DML and DDL, to just work normally!
> Billed per-second on actual resources consumed, … memory tracks active resident pages
Wait, are you saying it has dynamic vertical scaling of RAM?
Pretty much! It's not dynamic memory hotplug or resizing, just dynamic billing. Usage is based on the actual RSS used by the app so you can think of the VM's defined size as just a limit, for both CPU and RAM. This means you don't have to spend time profiling and right-sizing instances to optimize costs, just make your application fast and efficient and don't worry about it!
this is super cool, bookmarking to give it a whirl
> As a demo, I made a Hacker News clone with story submission, votes, comments and auth in about 60 lines of Markdown, plus CSS
although why would you use it for evil
> although why would you use it for evil
I couldn't help it! Ben Johnson recently wrote a Litestream post touching on this topic [1]:
> multiple-writer distributed SQLite databases are the Lament Configuration and we are not explorers over great vistas of pain
I offer my sincerest apologies if this database release may have inadvertently opened the gates of Hell.
[1] https://fly.io/blog/litestream-writable-vfs/
This is phenomenal, great work!
On the HN frontpage is Superlogical (I have no affiliation) right now, and I used the shell provided by the Kedge News demo to `ssh superlogical.jobs` from my browser on a phone on an instance started seconds earlier. :)
I didn't immediately spot the durability contract for volumes -- local NVMe like Fly, or sitting on some underlying system like EBS?
Thanks for the kind words!
> I didn't immediately spot the durability contract for volumes -- local NVMe like Fly, or sitting on some underlying system like EBS?
Persistent volumes use local NVMe that continuously syncs to object storage, so a similar architecture to Fly Sprites. It automatically recovers from host failures (with on-demand lazy restore), and you can manually migrate stable members to another region if needed.
I've positioned persistent volumes as an escape hatch for when you need an existing database app running in a VM, or when performance requirements outgrow the built-in replicated SQLite.
Gotcha, interesting. I would love it to death if there were a way to run some CLI, maybe on the system, that gave you information on the current replication state of the disk!
Two questions about the fork boundary. The snapshot is taken once when the process calls listen, so every later instance is a clone of the same frozen memory image, and anything seeded during init is identical across every clone in every region. That is exactly what the Linux vmgenid driver exists for, and it is why Lambda SnapStart had to ship explicit reseeding guidance. I could not find entropy, RNG or reseed anywhere in the runtime, performance or security docs, so is there a post-restore reseed hook, and does it fire before the first accept?
The other one is syzy. CRDT.md says eviction is operator-driven, that PRUNING.md is a design-stage contract and not yet code, and that a replica lagging beyond the hold window may transiently observe the departing duplicate, while the product headline offers scale to zero when idle, which mints offline replicas continuously. Who calls syzy_evict in the managed product?
These are both later-stage detail questions for when this project is further hardened for production workloads, but here's where each of these currently stands:
- Yes, there is a post-restore kernel reseed hook that happens after fork, and just before the trapped listen is continued. It doesn't use the vmgenid driver, it just calls the RNDRESEEDCRNG ioctl, so it would not be compatible with applications that depend on vmgenid notifications to reseed userspace PRNGs.
- In Syzy, the full pruning implementation is deferred for now; at the managed product's current small scale, it can simply eat the tiny overhead costs of un-collected garbage, while the user is only billed for the logical storage space consumed.
Fun fact: we also built our cloud platform based on Fly.io’s ideas, but we supercharged it with a custom Rust filesystem.
This platform powers full-stack applications for business users, including Playcode.ai.
For more information about our cloud platform, visit: https://playcode.io/cloud