I wonder why it took so much time among entire team to actually look into the source code. That's probably third thing I would do after encountering that kind of issue. I would even look for a word "gigabyte" considering it is well documented, because bizarre things should be documented.
The worst thing is, it's also documented in the SQLite's documentation. The file format is documented, and it describes the existence of this lock-page located at 1 GiB mark. How they've managed to implement support for SQLite DB file format without reading the file format description is, frankly, beyond me.
Curious if having the automated tests do db operations on a persistent db would catch this.
It would do inserts/deletes/updates/create_table/alter_column etc. randomly for a long time, continue even if an operation fails unless there is a crash. This way the db could grow without bound and queries can get very slow
It only catches it if you re-load the DB into sqlite after every write by turso. At which point you'd flag pretty quickly that it was the next write after 1 GB that messed things up
Could also make switching an operation. So it randomly switches between implementations and keeps using it until the next switch. And can keep running two instances so it is always kind of cross checking
This. This exact bloody attitude of people who claim "we want to write a compatible system" but can not be bothered to read — not even the source code — the freaking docs, freely available, is why we can't have any decently compatible with each other systems nowadays. There, look: [0], it's explicitly documented right on the SQLite's "Database File Format" page, and has been for a decade at least. How the fuck did you manage to write a clone of SQLite without reading the description of its file format? Like, seriously, how?
I don't even understand the point in rewriting something that is rock solid. And on top of that you guys are asking for full compatibility. This is just nonsense.
> To the point that we have a challenge [...]: if anyone can find a bug that leads to data corruption *and improve our simulator to catch it next time*, we will pay you a cash prize.
Uhh, you should be paying people if you get them to work for you.
I wonder why it took so much time among entire team to actually look into the source code. That's probably third thing I would do after encountering that kind of issue. I would even look for a word "gigabyte" considering it is well documented, because bizarre things should be documented.
Yeah. Is their developer super human because he...reads the source code?? This should be obvious.
Now I would be genuinely worried about what other issues are lurking in their rewrite.
Previously in this saga: https://turso.tech/blog/a-deep-look-into-our-new-massive-mul...
They broke SQLite compatibility even before they started writing in Rust: https://news.ycombinator.com/item?id=42386894
The worst thing is, it's also documented in the SQLite's documentation. The file format is documented, and it describes the existence of this lock-page located at 1 GiB mark. How they've managed to implement support for SQLite DB file format without reading the file format description is, frankly, beyond me.
Curious if having the automated tests do db operations on a persistent db would catch this.
It would do inserts/deletes/updates/create_table/alter_column etc. randomly for a long time, continue even if an operation fails unless there is a crash. This way the db could grow without bound and queries can get very slow
It only catches it if you re-load the DB into sqlite after every write by turso. At which point you'd flag pretty quickly that it was the next write after 1 GB that messed things up
Could also make switching an operation. So it randomly switches between implementations and keeps using it until the next switch. And can keep running two instances so it is always kind of cross checking
This. This exact bloody attitude of people who claim "we want to write a compatible system" but can not be bothered to read — not even the source code — the freaking docs, freely available, is why we can't have any decently compatible with each other systems nowadays. There, look: [0], it's explicitly documented right on the SQLite's "Database File Format" page, and has been for a decade at least. How the fuck did you manage to write a clone of SQLite without reading the description of its file format? Like, seriously, how?
[0] https://sqlite.org/fileformat2.html#the_lock_byte_page
We live in a SOCIETY!!!
I don't even understand the point in rewriting something that is rock solid. And on top of that you guys are asking for full compatibility. This is just nonsense.
I think it's a valid question, but it's better to assume they had their reasons and try to understand why before drawing conclusions.
> To the point that we have a challenge [...]: if anyone can find a bug that leads to data corruption *and improve our simulator to catch it next time*, we will pay you a cash prize.
Uhh, you should be paying people if you get them to work for you.