Nice writeup. I work on native macOS utilities and have dealt with file monitoring quite a bit. kqueue is reliable but the per-file descriptor requirement can get tricky when you need to watch large directory trees.
For anyone looking at this space, FSEvents is the higher-level alternative Apple provides. It watches directory-level changes without needing a file descriptor per file, which scales better for broad monitoring. But kqueue gives you more granular control, like detecting attribute changes or renames specifically, which FSEvents sometimes lumps together.
In practice I've found a combination works well: FSEvents for broad directory watching to catch that something changed, then kqueue for targeted monitoring of specific files you care about.
There are benchmarks in there, and comparisons with all the other watchers out there I could find at the time I made it (see "Comparison with Similar Projects" at the very bottom of the readme).
It's a pretty long list. Tons of watchers out there with different design philosophies and shapes of problems they solve.
There are a lot of caveats to the filesystem monitoring APIs provided by kernels. Some projects (like facebook's watchman) take that as a kind of antagonism, and decide to fight back with layers and layers of fallbacks and distrust and rescans. That projects basically only makes sense as a daemon.
Other programs and libraries try to take that complexity and tame it by being super-focused on one platform or providing a lot of configuration options.
Some provide debouncing logic. This particular feature comes up from time to time, I believe, both for practical reasons and because over-reported events from the kernel subsystems, especially for some arcane events like moving a file across mount points can trigger a flurry of hard-to-associate events for the same path.
If you want to avoid dealing with under-documented filesystem event subsystems, you can also just make your own with ebpf. Especially for security-oriented systems, you'll find that the only (nearly) perfectly accurate filesystem event subsystem you can make, is the one you make from the ground up.
Love it. Are there any benchmarks available on how fast this is from file change, kernel event received, event dispatched? I would expect it to be really quick.
Nice writeup. I work on native macOS utilities and have dealt with file monitoring quite a bit. kqueue is reliable but the per-file descriptor requirement can get tricky when you need to watch large directory trees.
For anyone looking at this space, FSEvents is the higher-level alternative Apple provides. It watches directory-level changes without needing a file descriptor per file, which scales better for broad monitoring. But kqueue gives you more granular control, like detecting attribute changes or renames specifically, which FSEvents sometimes lumps together.
In practice I've found a combination works well: FSEvents for broad directory watching to catch that something changed, then kqueue for targeted monitoring of specific files you care about.
Since there are comments below asking about benchmarks, others noting or expressing interest in other projects, I'll drop the one I made: https://github.com/e-dant/watcher (and also an ebpf experiment https://github.com/e-dant/bpf-fs-events)
There are benchmarks in there, and comparisons with all the other watchers out there I could find at the time I made it (see "Comparison with Similar Projects" at the very bottom of the readme).
It's a pretty long list. Tons of watchers out there with different design philosophies and shapes of problems they solve.
There are a lot of caveats to the filesystem monitoring APIs provided by kernels. Some projects (like facebook's watchman) take that as a kind of antagonism, and decide to fight back with layers and layers of fallbacks and distrust and rescans. That projects basically only makes sense as a daemon.
Other programs and libraries try to take that complexity and tame it by being super-focused on one platform or providing a lot of configuration options.
Some provide debouncing logic. This particular feature comes up from time to time, I believe, both for practical reasons and because over-reported events from the kernel subsystems, especially for some arcane events like moving a file across mount points can trigger a flurry of hard-to-associate events for the same path.
If you want to avoid dealing with under-documented filesystem event subsystems, you can also just make your own with ebpf. Especially for security-oriented systems, you'll find that the only (nearly) perfectly accurate filesystem event subsystem you can make, is the one you make from the ground up.
I love small personal projects like this, especially their writeups. I always have a hard time learning unless I am actually using it for something.
For more software in this domain see also the excellent entr https://eradman.com/entrproject/
kqueue is quite portable and works across all the BSDs.
The OpenBSD documentation for it is top notch, as usual. No idea about the rest (I suspect they’ve all converged at this point).
Kqueue originated in FreeBSD, and like most all of FreeBSD, it is very well documented.
Love it. Are there any benchmarks available on how fast this is from file change, kernel event received, event dispatched? I would expect it to be really quick.
Would be neat if POSIX was reasonably kept up to date. Just saying.