Our team recently migrated a subset of Hugging Face Hub repositories (~6% of total download traffic) from LFS to a chunk-based system called Xet. Xet uses chunk-level deduplication to send only the bytes that actually change between file versions.
This is a post about how we got to this point and what the day of/days after the initial migration looked like as we dove into every nook and cranny of the infrastructure.
Our team recently migrated a subset of Hugging Face Hub repositories (~6% of total download traffic) from LFS to a chunk-based system called Xet. Xet uses chunk-level deduplication to send only the bytes that actually change between file versions.
This is a post about how we got to this point and what the day of/days after the initial migration looked like as we dove into every nook and cranny of the infrastructure.