(ass. sports.txt politics.txt and business.txt are text docs pertaining from the sports, politics and business domains, respectively, and have equal size)
The test file belongs to the topic with the smallest size *.gz file.
Witten's group at Waikato uni were perhaps the first to work on this.
Also check out the Hutter prize if you are interested in this.
That requires you to decide what a "word" is, which is not trivial (if you think that ignoring punctuation gets you to a clean "letters surrounded by spaces" you will get lots of issues with various Asian languages)
Also some languages have a lot of prefixes and suffixes on their verbs or even nouns, which dilutes your list of 1000 words by just adding the same common words over and over again with different suffixes designating grammatical tense, grammatical gender, etc.
The gzip version sounds more general and more obviously correct
Instead of deciding on words, maybe you can break them up into smaller, subword parts – let's call them "quantums". And these quantums can be the units the quantumizer works on to operate on inputs and outputs. We can then use them to build Expansive Dictionary Models, or EDMs. I suppose we'd need a software library to mak working on this easier, think something speedy, fast, hot, like fire: we can call it PHPFlame...
Byte Pair Encoding [1] will be different for different languages. Application of the per-language BPEs to the input text will produce encodings with different lengths.
It naturally takes care of common prefixes and suffixes.
It is easy and fast to apply using radix tree or with finite automata. Even without radix tree, it is possible to have processing speed in the range of hundredths of thousands of bytes per second.
FSST is based on a fixed size (255 items) dictionary of high frequency variable length strings/substrings (learned from the corpus) encoded as one byte.
For anyone wanting an introductory text for information theory & that explores some of these connections & applications, it's worth checking out the late David MacKay's 2003 textbook Information Theory, Inference & Learning Algorithms https://www.inference.org.uk/itila/
By looking at mutual information from different authors on the same topic vs same author on different topics. As I recall, it convincingly disproved the hypothesis.
You might also want the topic files to be compressed against each other to get a baseline matrix and then multiply any results by the inverse, assuming equal priors on the topics.
Nitpick: Doing it exactly like this is flawed because you let the compressibility of your references taint the result; what you would prefer is the compressed size of testfile given sports.txt/... as a dictionary without accounting for the compressed size of that, no?
While I also like 7z a lot and use it all the time, it's really not a better format that RAR. Higher compression ratios? Sure, marginally. But for archival RAR is still the better choice as it supports recovery records, which 7z still does not. I also found it more difficult to restore data from a partial 7z file (i.e. trailer data is missing) compared to an incomplete RAR file.
Yeah, zstd is awesome. I built a webapp that uses it via wasm and the decompression speed is incredible, so much so that I store everything in zstd and decompress it on the app load. My wasm binary also does advanced search and tag insertion and stuff in addition to zstd but it's only 38kb. I wish zstd was supported natively by web browsers. The 8kb implementation of zstd is only half as slow as wasm, so even that is still viable.
zstd is a good default for e.g. filesystem compression, but if you're making an archive file, presumably you're looking for higher compression and LZMA would be a better fit.
On a more realistic note: few years back, I've added zstd compression to our log subsystem (hand written direct buffers, native code, in-process, java). For the same CPU utilization if provides twice dense compression compared to regular [-6] gzip (the topic in the title). Zstd is =much= faster on decompression as well, and it this case - unparalleledly better as it uses twice less disk.
zstd is 'silicon valley' (the tv show) - life imitates fiction, except entirely open source
There is 2 softwares I will buy for myself if I still use Windows: WinRAR and Internet Download Manager. In fact, I will buy them in the future regardless the Windows usage.
I read GP more as "sure they're profitable, but they didn't exactly have many expense to cover so it's not a very useful comparison to make at this point" than doubt in the profitability itself.
E.g. when SpaceX started designing the Falcon 9 the losses were astronomical (pun intended) but a car dealership running more of a profit at the same time doesn't have much to say about which business is doing better.
Culonavirus (GGP from the original? Dunno how to refer to that)? Yeah - you could make a good guess of that even just by looking at the username they chose :D. Doesn't mean wolfi1, shezi, or myself can't have a serious discussion stemming from it as well.
Who knows though - maybe I missed that German company financials are also funny :p. Wouldn't be the first time something flew well over my head.
The original author and also creator of the RAR format, Eugene Roshal, is Russian. I think he moved to the US way back in the day, but not sure about that. He has been very private with his life.
The company doing the software distribution, is located in Berlin. The Managing Directors for that company seem to have Turkish names, but I don't know if they're Turkish.
BTW, Looking for some info I just found a website [0], clearly AI generated (but not necessarily meaning the content is false) claiming Eugene Roshal had severe kidney failure this past month, and he's waiting for surgery. They're asking for donations. There are some names on who's theoretically behind it [1] but they don't link to any LinkedIn profile or personal site. I can't find any other references. The BTC wallet they're using for donations hasn't seen any traffic ever. BE WARY, SMELLS FISHY.
WHOIS information for the domain shows it's registered in the Philippines. The name of the owner does not show up in any of the about section people. So I think those are fake.
In addition to what the other comments said ("WHOIS information for the domain shows it's registered in the Philippines"[0] etc), the "hospital bed" picture is blatantly AI-generated. If you zoom on his face, you can see that it looks extremely unnatural.
I suspect that someone's running a script that looks for public (-ish, Eugene has a Wikipedia page at least) figures without active social media presence and creates fake donation websites with AI-generated texts and pictures.
If my suspicion is correct, this is one of the most evil scams I can imagine.
A donation site that takes only bitcoin as payment is very suspicious, surely there are alot of services like GoFundMe who would accept him if it was genuine. I think this is a fake scam
edit: can you write in brackets after the URL that it may be scam. I think alot of people may just go to the URL without reading your warning
give it a normal text prompt, and it
continues that prompt by searching
for the byte sequences that compress
best.
One moment, how are we supposed to know how well that search was done? There is no way to search a meaningful part of the search space.
So the result only gives us some lower bound of how well gzip works as a "plausibility tester" of a continuation of a text. The space of possible sequences is many orders of magnitude larger than what was searched. So there might be sequences in there that compress much better.
The text mentions beamsearch, but I don't see a discussion about how well beamsearch performs in finding the global optima when it comes to gzip compressibility of a text?
That's a fair question. Suppose we have a way to find a byte sequence x that globally minimises len(gzip(context + prompt + x)) over all sequences x of length n. Here + denotes string concatenation.
It's unclear if this is very useful.
The reason it may not be very useful is that one of Deflate's ingredients is a pass that replaces repeated substrings with backreferences to the earlier occurrence in the plaintext input stream.
E.g. suppose we want to find an n=200 byte sequence x that minimises len(gzip(context+prompt+x)).
If there exists any 200 byte sequence y such that prompt+y is a substring of context, then Deflate can encode prompt+y as a backreference to that earlier sequence - it needs to store a match-length & a distance-length, encoded using its Huffman trees. This candidate solution y may not be a global minima to our stated objective function, but if not, it's probably going to be a very good near-optimal approximate solution.
Taking a step back, repeating huge chunks of the input context produces something that's great for minimising compressed output size but doesn't seem particularly helpful as a generative model.
edit:
Yep, I tried it out by running an experiment. Searching for the prompt in the context & then copying the following text as the solution produces solutions that are much better, in the sense of minimising the compressed output length, than beam search, while also being unhelpful as a generative tool.
With the same example as the blog post:
context: first 30,000 bytes of tinyshakespeare.txt
prompt: 'MENENIUS:\n'
Let x denote a solution, x is a string of length 200.
Let L(x) denote len(gzip(context+prompt+x)), our objective function
Let's call the proposed search method of searching for the prompt in the input rfind (after python's str.rfind).
Then we have
search method soln soln length feasible? objective value search time (wall clock, s)
------------- ---- ----------- --------- --------------- ---------------------------
emptystring "" 0 no 13,023 0.04s
gzipt beam search see blog post 200 yes 13,051 11.93s
rfind see below 200 yes 13,026 0.04s
So 'rfind' is finding a solution that does a better job of minimising the objective function -- it only takes 3 bytes more to encode than the infeasible emptystring solution, and costs 25 fewer bytes than the solution found by the beam search implemented by gzipt per the blog post.
Here's the solution 'generated' by rfind copying and pasting from the input context, starting from the rightmost occurrence of "MENENIUS:"
MENENIUS:
O, true-bred!
First Senator:
Your company to the Capitol; where, I know,
Our greatest friends attend us.
TITUS:
COMINIUS:
Noble Marcius!
First Senator:
MARCIUS:
Nay, let them follow:
The Volsces
Here's the code for 'rfind' - our complete 'generative algorithm':
def find_candidate_solution_from_context(context, prompt, length):
n = len(context)
i = context.rfind(prompt, 0, n-length)
if i < 0:
return b''
i += len(prompt)
return context[i:i+length]
Can hook it into gzipt.py by adding this line after out is defined, but before the beam search begins
out += find_candidate_solution_from_context(corpus_window, prompt, length)
Read to the end: they aren't actually looking for the best-compressing output, because this quickly devolves into aaaaaaaaaaa. They keep a sliding window over a small portion of recent text and use that.
Basically I think the entire premise falls apart due to that choice--they forced an interesting-looking outcome by adjusting the algorithm until gzip started picking random slabs of letters instead of ever-larger repeating runs.
Compressing something and next token prediction are very related. Once you understand the connection, things like ts_zip and hutter prize make a lot more sense.
This is fun, but historically people have gone a bit overboard with saying that models like this, or n-gram language models, are anywhere close to large neural network models. There is certainly a connection though.
Yes, but it is a useful insight that both methods try to solve the same mathematical problem. It's better than thinking of LLMs as magic.
When you say "cross-entropy loss" people without stats background go to Wikipedia, take a glance, and adjust their mental model to "inscrutable magic".
Thinking of the main difference as the trade-off in how much CPU, memory and storage is allowed is not really wrong.
The part that is wrong is to think of gzip as a method that might reach similar complexity or generalization. And more importantly, to ignore the advanced way how training data gets curated or generated for (instructed, chain-of-thought) LLMs. But even then. The mental model that the LLM's goal is text compression is not wrong. The question to ask next is what kind of text it is expecting to compress.
Yep agree, good details, and this doesn't contradict my point above, about people going "overboard" with the comparison.
I do think when making these comparisons, it is worth emphasising that neural nets are really different. E.g. I used to see people equating LLMs to n-gram models, etc. which is overly simplistic, (especially in the early days when the models weren't as good).
Hutter prize does have speed restrictions. If it did not, LLMs would win even with counting the size of the model (which is the most reasonable choice imo) as per the main benchmark: https://www.mattmahoney.net/dc/text.html.
zip2zip paper by Geng et al. also exploited LZ/LZW and made the rounds a while back; novel approach that uses zip content as output compression adapter
Would this work with video compression? Video codecs encode a lot of meaning; they use motion vectors to track the movement of objects on screen, for example.
I've been pondering on something related: can an LLM be a chat?
Some models are reproducible, in that the same prompt will generate the same output. Say that we could wire up such a model to generate some code.
In that case, we could create a prompt that generates, say, an entire codebase, or a large piece of text. The prompt (or really, the tokens) would then be the compressed version of the codebase or the text.
I am not talking about an "AI agent", but really a model that we call in a reproducible manner. Preferably one call, with one prompt. An agent could just run `git clone` to "decompress" a codebase, which conflates the idea of compression. If that were compression, then the "compressed version of the git kernel" would be a single line of text: `git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...`. I am really talking about having an LLM re-generate text based on a prompt.
Does that make sense? I can imagine that this is highly impractical and inefficient. But would this count as "compression" at all?
A large language model itself (the network) give you the probabilities for the next token given some prefix of tokens so far. You can use arithmetic coding to go from these probabilities to a deterministic compression / decompression algorithm.
When you use an LLM to generate text, you sample from that probability distribution. You can use a true random sample. Or you can make it trivially deterministic by using a seeded pseudo-random-number-generator or you just pick the highest probability each time. But that's all a red herring; really, what you want is arithmetic coding.
>An LLM is just as deterministic as any other computer program. For identical inputs (which includes the PRNG seed) it produces identical outputs.
This is not really true in practice because of multi-threading and out-of-order execution. Mathematically equivalent orderings of operations are not equivalent when dealing with floating point values, so most practical LLM implementations end up being non-deterministic.
"An LLM is just as deterministic as any other computer program" is not refuted by pointing out hardware limitations that would affect any other computer program implemented at similar scale (weather forecasts, or even just computing the average of a large stream of sensor readings).
I'm not really trying to 'refute' the original statement. Certainly, an LLM is just doing some calculations that can be done deterministically in principle. However, I think it's worth pointing out that there are practical barriers to doing those particular calculations both deterministically and efficiently. People who worry about LLM output not being reproducible aren't necessarily misunderstanding what an LLM is doing; they are responding to a real feature of most practical LLM implementations.
Sounds like an interesting way for future OS included apps to be distributed.
Like when you click the Calculator button on your android, it wouldn't actually exist yet, your click actually prompts it into existence. But naively that has problems because you don't want a different UI every time. There's something to your idea.
It makes a lot of sense. I thought about it in the context of pull requests or change sets: if the text-to-code process is reliable, why don't you give me prompts instead of code? Code becomes just an intermediate representation.
I think language itself is compression, so the arxiv paper tracks for me.
Viz. if Language is compression (of thought / culture / the tacit je ne sait quois of being-to-being communication etc.), then definitionally, Language Modelling must also be Compression.
Except, language is an arbitrarily lossy compressor, who's "compression-prediction equivalence" is indeterminate and unstable, because Language co-evolves constantly; both as a function of or response to culture, as well as an influencer of culture.
So, the subjective-objective goodness of Language Models (of any kind of language) would be, at best, upper-bounded by the compression-prediction equivalence of the Languages corpus itself. And that is assuming the language corpus is perfect in every way---it captures all knowledge expressible by language and it is always in-sync with live evolution of all language expression and evolution (i.e. LLM training is not a batch job, but a real-time present continuous process).
For example, to my layperson eyes, the mathematical language of proofs actively weeds out ambiguity of subjective interpretation. Ideally, a proof ought to lead to the exact same conclusion on every single reading by any reader who can follow the steps. A proof also holds only if the rest of the formal, explicit, inviolable, internally-consistent set of axioms and results holds.
So it stands to reason that mathematical prose of proofs, being optimised as mechanical procedure of taking an open question to a deterministically closed solution, has better odds of approximating the tacit aspects of mathematical derivation.
Which makes an LLM able to construct a mathematical proof, which is mind-melting to say the least.
However, I wonder, can LLMs dream of mathematical sheep?
Language is compression of a sort. A dictionary is a decompressor. You look up one word, and you may get a paragraph about its meaning. An encyclopedia can be thought of as roughly the same with more detail for some nouns.
It makes a lot of sense why dictionary-based compression is named the way it is. A shorter symbol is used to store information that would take more symbols in the uncompressed corpus, if the shorter symbol hadn't been assigned to represent it. That's in a way just what an actual dictionary on your English professor's shelf does. The big difference is your compressor is coining new short symbols all the time.
R. Hendricks, D. Chugtai, and J. Dunn, "Lossless compression via optimized middle-out bitstream processing," Pied Piper Inc., Palo Alto, CA, Tech. Rep. 42, Apr. 2014.
I was curious to see how this would work with bzip2 and zstd. The source is public at https://github.com/nathanrs/gzipt, and I asked MiMo-V2.6-Flash to fork and modify it in a straightforward way. The answer is that bzip2 produces sequences that don't resemble human language:
Zstandard produces whitespace with the occasional letter thrown in. To quote MiMo: "As you can see, zstd does not speak Shakespeare. ... zstd encodes a run of one repeated byte as a near-free run-length sequence, and space and newline are the cheapest literals in the corpus: ten newlines cost about the same to append ten bytes of genuine corpus text and less than nonsense does."
I did. I read the code to make sure the quality of MiMo's work matched mine for a quick experiment, though not that the code was free from subtle bugs.
This was the main change for bzip2:
@@ -33,19 +34,16 @@ def candidate_lengths(
level: int = 9,
pool: ThreadPoolExecutor | None = None,
) -> list[int]:
- """Compressed length of ``context + seq`` for each seq, sharing the context.
+ """Compressed length of ``context + seq`` for each seq.
- Compresses ``context`` once into a ``compressobj``, then clones its encoder
- state per candidate and feeds only that candidate. Identical to
- ``len(zlib.compress(context + seq, level))`` for each seq, but the expensive
- match search over ``context`` happens a single time.
+ Unlike ``zlib``'s ``compressobj``, Python's ``BZ2Compressor`` cannot be
+ snapshotted mid-stream, and bzip2's move-to-front + Huffman stages see the
+ whole block, so every candidate recompresses the full context. Threads
+ still scale because ``bz2`` releases the GIL.
"""
- base = zlib.compressobj(level)
- head = len(base.compress(context))
def length_for(seq: bytes) -> int:
- clone = base.copy()
- return head + len(clone.compress(seq) + clone.flush(zlib.Z_FINISH))
+ return len(bz2.compress(context + seq, level))
if pool is not None:
return list(pool.map(length_for, sequences))
Is the fact that the original did [compress base]+[compress seq] rather than [compress [bytes + seq]] not important?
(I honestly don’t know is gzip does something different when presented with two chunks as opposed to one, or, if it does, if bz2 has equivalent behaviour - but the difference in the code did stand out to me, and it does seem related to ‘extending the token sequence’)
This difference doesn't matter because of how zlib works. At least by default, zlib divides the input data into its own blocks independent of the caller. If you don't feed it enough data to complete a block, it waits until you feed it more or finish the stream.
At temperature zero, this outputs the same sample as commit 3734bf6, the most recent commit upstream:
MENENIUS:
'Though all at once cannq
MARCIUS:
I'll fight
'Though all at once cannq
MARCIUannq
MARCIUS:
I'll fight
'Though
AUFIDIUS:
If I fly, Marci
AUFIDIUS:
If I fly, Marci
AUFID
AUFIDIUS:
If
If I fly
A seemingly simple sentence like "I had lunch" has enormous amount of information compressed inside it.
The word lunch is a compressed form of "having food at noon" while "noon" in turn is a compressed form of "Sun's position against Earth's rotation" and so on and so forth.
Every sentence has layers of compressed sentences. How many layers one chooses to decompress is up to the person.
Will Wright once described the same connection from the opposite direction: compression as procedural content generation.
In the 2023 discussion of "Demoscene accepted as UNESCO cultural heritage in The Netherlands" I posted a transcript from a video of Will Wright discussing the demo scene:
>You can take any piece of content in the game, and imagine an algorithmic solution to it. Or also, you know, a way that the player could customize that object of thing.
>There's this group in Europe called the Demoscene that make these very elaborate demos for a computer that fit into very tiny little memory blocks, you know like 64K of memory, and you run the thing, and in fact it algorithmically generates about 100 megabytes worth of data, you know these rich 3D environment, generated music, generated wave files, generated animation.
>And they're developing techniques to generate, you know, huge amounts of interesting data, with very very simple, elegant, compression algorithms.
>And this is a skill that game developers used to have, back in the 8-bit days. That was the only ways to do a game like Karateka(?), was to find all these little tips and tricks to compress things and generate them algorithmically.
>But since the CD-ROM came out, and very cheap hard drives, storage is cheap, so basically we've lost that skill set, and now we attack all those problems with brute force. I think we've lost something by dropping that skill set.
[...] Here's a simple low-tech pre-LLM example that shows the equivalence of compression and procedural content generation:
Take a huge text file of HN postings, and compress it with gzip or compress or some other robust compression algorithm. The better the algorithm, the more the output will look like random noise. Then slice the compressed file in half, and replace the second half with random numbers. Then uncompress it. You'll find that at the point you sliced it, it keeps on writing out almost plausible text for a while, consisting of highly probably snippets of commonly encountered words and phrases, then goes downhill towards incoherence. It's not as coherent or confident as an LLM, but the point is to show how low the bar is for using compression for procedural content generation.
LLMs are essentially a form of compression of the world's knowledge or whatever they're trained on, not just word frequencies or pixel patterns, but also concepts and ideas. [...]
Yay, another mostly AI authored piece with vibe-coded aesthetics.
Some will say that I should 'judge the idea, not the form'.
But if the author didn't find enough strength to write alone a short ~700 words summary about his work, it means he himself isn't that interested or enthusiastic about it. Why should others bother then? Particularly since low-effort like that signals possibility the whole work is superficial and derivative.
Your claim that suspected AI assistance proves the author isn't interested -- and therefore that the work is probably superficial -- is unsupported.
The article presents a working experiment, explains why naive decoding fails, describes the beam-search fix, and links the code.
Dismissing all that with presumptuous personal speculation and banal boilerplate drive-by anti-AI snark adds absolutely nothing to the conversation -- and that is intrinsically poor form.
You couldn't even find enough strength to criticize anything beyond the form, while your own form is lackluster.
Ironically, an LLM could have written your comment and improved its form without losing anything distinctive.
> I'm judging the ideas in your post, not its form.
and simultaneously you write that 'my form is lackluster' and that 'an LLM could have written your comment and improved its form without losing anything distinctive'. We are having ourselves a small contradiction, aren't we.
Be my guest, enjoy chatbot writing and drowning in slop. But don't encroach upon my freedom to protest it.
Ironic, isn't it? Yet you continue your lackluster form devoid any criticism of ideas. All you can criticise is form, yet my criticism of your ideas still stands:
Your claim that suspected AI assistance proves the author isn't interested -- and therefore that the work is probably superficial -- is unsupported.
So I'm encroaching but you're only protesting, huh? I also have the freedom to ironically protest the poor form of your inability to criticize ideas, as well as your poorly formulated unsupportable ideas.
Say you have been working on a multi year project writing code and putting in a lot of effort.
If you then use AI to write a product page documentation and proof read it for correctness, would that constitute to signaling that the whole effort is superficial?
Some work may be left to AI while you focus on the more important aspects of the work.
Surprisingly people have got it completely backwards where they want AI to generate code and humans to write documentation.
Not without attention or something approximating it.
The fact that gzip is relatively fast should be your first clue that something important is missing.
Gzip is great at predicting the next token for one very specific narrative. LLMs can predict next tokens for entire universes of narratives. Searching for the correct next token across this space scales ~quadratically with the input size. Gzip scales linearly. I can gzip a one terabyte file. Imagine feeding that much into an LLM. These are wildly different animals that happen to overlap in a very small way. Equating compression to intelligence looks increasingly silly to me.
If we must compare language models to compression, they are much more like jpeg and mp3 than they are gzip and flac. I can go fuck with a jpeg file pretty severely at the bitstream level and still have something resembling performance on the other side. Gzip cannot remotely approach this.
> Gzip scales linearly. I can gzip a one terabyte file.
In part because gzip only has a 32KiB window size, and I think it'd be at least quadratic within that window if you were going for optimal compression.
I'll concede the window part, but Gzip runs within the physical confines of a single cpu core and is typically entirely resident in local caches. The point is not just the quadratic scaling but also what it scales with.
Show me an LLM that can run at 300 megabytes per second. Even dedicated ASICs with weights burned in will never move this fast.
> if LLMs are used as compressors, how well is that expected to work
Quite well. This project[1], by Fabrice Bellard of ffmpeg fame, is quite old in AI years and uses an ancient LLM, but still beats xz by a solid margin.
A challenge as I understand it is reproducibility.
Normal LLM runtimes aren't typically fully reproducible even with same random seeds for distribution sampling, due to floating-point numbers, batching and such.
Though averaging over many runs could alleviate that I suppose.
While it would measure some aspects of intelligence, I'd argue it fails to capture other, more creative aspects.
I think not. it's true that a large language model compresses knowledge and allows us to decompress knowledge. but gzip compresses data and not knowledge. with a LM you can decompress various forms of knowledge from the same data. gzip is a 1 to 1 kind of decompression where as an LM is a 1 to infinity kind of decompression.
Yes: you can classify a test file by topic with gzip as follows:
(ass. sports.txt politics.txt and business.txt are text docs pertaining from the sports, politics and business domains, respectively, and have equal size)The test file belongs to the topic with the smallest size *.gz file.
Witten's group at Waikato uni were perhaps the first to work on this.
Also check out the Hutter prize if you are interested in this.
Back in the day - maybe two decades ago - I implemented language detection like this.
I seeded gzip compressors’ dictionaries with Wikipedia articles in different languages.
I would then try to use said dictionaries on any random text, and the one that was best able to compress it, was the correct language.
Absolutely totally not the best approach, but very fast and super simple to implement.
Or maybe make a list of the most used 1000 words in each language. And see which list has the most occurrences.
That requires you to decide what a "word" is, which is not trivial (if you think that ignoring punctuation gets you to a clean "letters surrounded by spaces" you will get lots of issues with various Asian languages)
Also some languages have a lot of prefixes and suffixes on their verbs or even nouns, which dilutes your list of 1000 words by just adding the same common words over and over again with different suffixes designating grammatical tense, grammatical gender, etc.
The gzip version sounds more general and more obviously correct
Instead of deciding on words, maybe you can break them up into smaller, subword parts – let's call them "quantums". And these quantums can be the units the quantumizer works on to operate on inputs and outputs. We can then use them to build Expansive Dictionary Models, or EDMs. I suppose we'd need a software library to mak working on this easier, think something speedy, fast, hot, like fire: we can call it PHPFlame...
I'm guessing this is an allusion to reinventing something that already exists, but do you mind explaining what that is to me, since I don't know?
I suspect they're talking about LLM tokens.
It is describing essentially how tokenisation is done for LLMs (Large Language Model → Expansive Dictionary Model).
You mean Quantumnisation ;)
Even German has issues with "letters surrounded by spaces". They love to smash words together to get one giant word.
https://www.youtube.com/watch?v=m8niIHChc1Y
Byte Pair Encoding [1] will be different for different languages. Application of the per-language BPEs to the input text will produce encodings with different lengths.
[1] https://en.wikipedia.org/wiki/Byte-pair_encoding
It naturally takes care of common prefixes and suffixes.
It is easy and fast to apply using radix tree or with finite automata. Even without radix tree, it is possible to have processing speed in the range of hundredths of thousands of bytes per second.
You can just use 2-grams of Unicode code points.
Just collect common bi- and trigrams and a naive bayesian classifier.
See: Fast Static Symbol Table: https://github.com/duckdb/duckdb/pull/4366
FSST is based on a fixed size (255 items) dictionary of high frequency variable length strings/substrings (learned from the corpus) encoded as one byte.
Sounds like the best approach. :)
There are some deep connections between machine learning, compression, and cryptography with information theory as a common thread.
Also, I’ve never seen “ass.” Used to shorten “aside” — I typically use N.B. but perhaps only for important ones.
For anyone wanting an introductory text for information theory & that explores some of these connections & applications, it's worth checking out the late David MacKay's 2003 textbook Information Theory, Inference & Learning Algorithms https://www.inference.org.uk/itila/
The "ass." more likely stands for "assuming".
Yeah, I read it as "assuming" too.
3Blue1Brown has a good video about this: https://www.youtube.com/watch?v=l6DKRf-fAAM
aka Normalized compression distance (NCD). Its close cousin: Normalized Google distance (NGD) is also super interesting!
https://en.wikipedia.org/wiki/Normalized_compression_distanc...
We used a similar technique for a class project (N decades ago) to test this:
https://en.wikipedia.org/wiki/Baconian_theory_of_Shakespeare...
By looking at mutual information from different authors on the same topic vs same author on different topics. As I recall, it convincingly disproved the hypothesis.
You might also want the topic files to be compressed against each other to get a baseline matrix and then multiply any results by the inverse, assuming equal priors on the topics.
Nitpick: Doing it exactly like this is flawed because you let the compressibility of your references taint the result; what you would prefer is the compressed size of testfile given sports.txt/... as a dictionary without accounting for the compressed size of that, no?
Really interesting approach though.
I seem to remember it being shown for character recognition via JBIG. Maybe in Managing Gigabytes?
There would be some overlap with business sports analogies - eg team huddle.
we were doing this in Qualcomm 20 years ago
Is pigz faster?
This tracks perfectly with Winrar being more profitable than OpenAI... coincidence? I think not!
winrar is profitable? sure? well, on the other hand, they sure don't make losses
They are a German GmbH and must publicly state their financials: https://www.northdata.de/win%C2%B7rar%20GmbH,%20Berlin/Amtsg...
Looks pretty profitable to me.
They're one of the few companies that actually manage to sell "boxed software" (i.e. has not changed much in years but new customers keep buying it)
That said, Windows users should use 7-Zip. Better compression format, unpacks more kinds of archives
While I also like 7z a lot and use it all the time, it's really not a better format that RAR. Higher compression ratios? Sure, marginally. But for archival RAR is still the better choice as it supports recovery records, which 7z still does not. I also found it more difficult to restore data from a partial 7z file (i.e. trailer data is missing) compared to an incomplete RAR file.
> Windows users should use 7-Zip
Please no - no native zstd support. NanaZip is the better option (it's a different build of 7-zip) and it's available at windows store.
> Better compression format, unpacks more kinds of archives
winrar has supported zstd for 5 years[0]
In short - Everyone should be using zstd, and 7-zip does not support it.
[0]: https://www.win-rar.com/singlenewsview.html?&L=0&tx_ttnews%5...
Yeah, zstd is awesome. I built a webapp that uses it via wasm and the decompression speed is incredible, so much so that I store everything in zstd and decompress it on the app load. My wasm binary also does advanced search and tag insertion and stuff in addition to zstd but it's only 38kb. I wish zstd was supported natively by web browsers. The 8kb implementation of zstd is only half as slow as wasm, so even that is still viable.
Lzma. I did a simple test, same payload repeated with a gap. Lzma ruled it.
|gap |gzip |bz2 |lzma | |---------|----------|-----|--------| |0 |2.7% |18.9%|*0.9%*| |8 KB |2.4% |17.8%|0.9% | |*40 KB*|*94.4%* |17.7%|0.4% | |1 MB |*104.3%*|19.7%|*0.9%*|
HN doesn’t support Markdown tables. You can prepend two spaces to show it in monospace though.
I don’t see zstd in your comparison?
dont know the point of the table, but here it is:
zstd is a good default for e.g. filesystem compression, but if you're making an archive file, presumably you're looking for higher compression and LZMA would be a better fit.
>> Windows users should use 7-Zip
7z is now built into W11 right click so that or zip is what will be used by default anyway.
Last I checked windows handles compressed files in 4 byte or 4kb or something chunks and is incredibly slow to compress or decompress a lot of files.
> Everyone should be using zstd
Why?
It's just the best general purpose compression algorithm, in terms of compression ratio to CPU used, for the vast majority of use cases
It's just this good.
On a more realistic note: few years back, I've added zstd compression to our log subsystem (hand written direct buffers, native code, in-process, java). For the same CPU utilization if provides twice dense compression compared to regular [-6] gzip (the topic in the title). Zstd is =much= faster on decompression as well, and it this case - unparalleledly better as it uses twice less disk.
zstd is 'silicon valley' (the tv show) - life imitates fiction, except entirely open source
but worse filebrowser
Huh. I guess the warez kids grew up and have money, now?
There is 2 softwares I will buy for myself if I still use Windows: WinRAR and Internet Download Manager. In fact, I will buy them in the future regardless the Windows usage.
I read GP more as "sure they're profitable, but they didn't exactly have many expense to cover so it's not a very useful comparison to make at this point" than doubt in the profitability itself.
E.g. when SpaceX started designing the Falcon 9 the losses were astronomical (pun intended) but a car dealership running more of a profit at the same time doesn't have much to say about which business is doing better.
they're is obviously joking
Culonavirus (GGP from the original? Dunno how to refer to that)? Yeah - you could make a good guess of that even just by looking at the username they chose :D. Doesn't mean wolfi1, shezi, or myself can't have a serious discussion stemming from it as well.
Who knows though - maybe I missed that German company financials are also funny :p. Wouldn't be the first time something flew well over my head.
That’s surprising. Seems there is a niche for everything.
You need to visit Japan...
TIL they're Turkish
The original author and also creator of the RAR format, Eugene Roshal, is Russian. I think he moved to the US way back in the day, but not sure about that. He has been very private with his life.
The company doing the software distribution, is located in Berlin. The Managing Directors for that company seem to have Turkish names, but I don't know if they're Turkish.
BTW, Looking for some info I just found a website [0], clearly AI generated (but not necessarily meaning the content is false) claiming Eugene Roshal had severe kidney failure this past month, and he's waiting for surgery. They're asking for donations. There are some names on who's theoretically behind it [1] but they don't link to any LinkedIn profile or personal site. I can't find any other references. The BTC wallet they're using for donations hasn't seen any traffic ever. BE WARY, SMELLS FISHY.
--
WHOIS information for the domain shows it's registered in the Philippines. The name of the owner does not show up in any of the about section people. So I think those are fake.
In addition to what the other comments said ("WHOIS information for the domain shows it's registered in the Philippines"[0] etc), the "hospital bed" picture is blatantly AI-generated. If you zoom on his face, you can see that it looks extremely unnatural.
I suspect that someone's running a script that looks for public (-ish, Eugene has a Wikipedia page at least) figures without active social media presence and creates fake donation websites with AI-generated texts and pictures.
If my suspicion is correct, this is one of the most evil scams I can imagine.
[0] https://news.ycombinator.com/item?id=49801838
A donation site that takes only bitcoin as payment is very suspicious, surely there are alot of services like GoFundMe who would accept him if it was genuine. I think this is a fake scam
edit: can you write in brackets after the URL that it may be scam. I think alot of people may just go to the URL without reading your warning
3blue1brown did a series on this topic: https://www.youtube.com/watch?v=l6DKRf-fAAM https://www.youtube.com/watch?v=GlYgs6v2YfU (i think one more is yet to release)
Wonderful link. Just commenting to add: in the first video, Grant mentioned it would be a trilogy series.
So the result only gives us some lower bound of how well gzip works as a "plausibility tester" of a continuation of a text. The space of possible sequences is many orders of magnitude larger than what was searched. So there might be sequences in there that compress much better.
The text mentions beamsearch, but I don't see a discussion about how well beamsearch performs in finding the global optima when it comes to gzip compressibility of a text?
That's a fair question. Suppose we have a way to find a byte sequence x that globally minimises len(gzip(context + prompt + x)) over all sequences x of length n. Here + denotes string concatenation.
It's unclear if this is very useful.
The reason it may not be very useful is that one of Deflate's ingredients is a pass that replaces repeated substrings with backreferences to the earlier occurrence in the plaintext input stream.
E.g. suppose we want to find an n=200 byte sequence x that minimises len(gzip(context+prompt+x)).
If there exists any 200 byte sequence y such that prompt+y is a substring of context, then Deflate can encode prompt+y as a backreference to that earlier sequence - it needs to store a match-length & a distance-length, encoded using its Huffman trees. This candidate solution y may not be a global minima to our stated objective function, but if not, it's probably going to be a very good near-optimal approximate solution.
Taking a step back, repeating huge chunks of the input context produces something that's great for minimising compressed output size but doesn't seem particularly helpful as a generative model.
edit:
Yep, I tried it out by running an experiment. Searching for the prompt in the context & then copying the following text as the solution produces solutions that are much better, in the sense of minimising the compressed output length, than beam search, while also being unhelpful as a generative tool.
With the same example as the blog post:
Let x denote a solution, x is a string of length 200.Let L(x) denote len(gzip(context+prompt+x)), our objective function
Let's call the proposed search method of searching for the prompt in the input rfind (after python's str.rfind).
Then we have
So 'rfind' is finding a solution that does a better job of minimising the objective function -- it only takes 3 bytes more to encode than the infeasible emptystring solution, and costs 25 fewer bytes than the solution found by the beam search implemented by gzipt per the blog post.Here's the solution 'generated' by rfind copying and pasting from the input context, starting from the rightmost occurrence of "MENENIUS:"
Here's the code for 'rfind' - our complete 'generative algorithm': Can hook it into gzipt.py by adding this line after out is defined, but before the beam search beginsRead to the end: they aren't actually looking for the best-compressing output, because this quickly devolves into aaaaaaaaaaa. They keep a sliding window over a small portion of recent text and use that.
Basically I think the entire premise falls apart due to that choice--they forced an interesting-looking outcome by adjusting the algorithm until gzip started picking random slabs of letters instead of ever-larger repeating runs.
I had actually thought of doing this, but didn't for this exact reason. I knew I would have to fudge things to make it anything interesting.
Meh. That’s nothing compared to the amount of curation and tuning the LLMs are coerced with.
No--even a really tiny, underpowered model like GPT-2 with no system prompt produces coherent (though not necessarily desirable or correct) responses.
Compressing something and next token prediction are very related. Once you understand the connection, things like ts_zip and hutter prize make a lot more sense.
https://bellard.org/ts_zip/ https://corecursive.com/the-hutter-prize/
This is fun, but historically people have gone a bit overboard with saying that models like this, or n-gram language models, are anywhere close to large neural network models. There is certainly a connection though.
Yes, but it is a useful insight that both methods try to solve the same mathematical problem. It's better than thinking of LLMs as magic.
When you say "cross-entropy loss" people without stats background go to Wikipedia, take a glance, and adjust their mental model to "inscrutable magic".
Thinking of the main difference as the trade-off in how much CPU, memory and storage is allowed is not really wrong.
The part that is wrong is to think of gzip as a method that might reach similar complexity or generalization. And more importantly, to ignore the advanced way how training data gets curated or generated for (instructed, chain-of-thought) LLMs. But even then. The mental model that the LLM's goal is text compression is not wrong. The question to ask next is what kind of text it is expecting to compress.
Yep agree, good details, and this doesn't contradict my point above, about people going "overboard" with the comparison.
I do think when making these comparisons, it is worth emphasising that neural nets are really different. E.g. I used to see people equating LLMs to n-gram models, etc. which is overly simplistic, (especially in the early days when the models weren't as good).
Reminded me Google's earlier paper, "Language Modeling Is Compression" https://arxiv.org/abs/2309.10668
Learning being a compression is also recently proposed as Gibbs compression proposition.
See Gibbs randomness-compression proposition https://arxiv.org/abs/2505.23869v5
There is an older paper that also looks at gzip for ML: https://arxiv.org/abs/2212.09410
Video from tsoding where he implements the algorithm: https://www.youtube.com/watch?v=9n39SbRPXKQ
I'm more interested in the converse question: how well does an LLM perform as a compressor, compared to gzip (ignoring its insanely lower speed)?
Top contestant in the Hutter Prize uses a neural network for compression. So fair to say, LLMs would perform pretty well compared to gzip.
Even ignoring speed per GP, the Hutter Prize's metric includes the size of the decompressor. LLMs would be disqualified for being larger than 1GB.
Hutter prize does have speed restrictions. If it did not, LLMs would win even with counting the size of the model (which is the most reasonable choice imo) as per the main benchmark: https://www.mattmahoney.net/dc/text.html.
And the hutter prize disallows GPU's. If you allow use of a powerful GPU, you can do quite a bit better.
hallucinations are lossy compression artefacts
can they be considered to have compressed the entirety of their training data into their weights?
If you consider them lossy compression, then yes.
The goal would be to find the minimum model that, with a fixed seed, would exactly reproduce your text.
So I could hide information in a model basically
Much better
lossless vs lossy is the question.
How lossy? Because I can lossy compress anything into 0 bits.
0 bits is just a very small bloom filter.
zip2zip paper by Geng et al. also exploited LZ/LZW and made the rounds a while back; novel approach that uses zip content as output compression adapter
https://arxiv.org/abs/2506.01084
Would this work with video compression? Video codecs encode a lot of meaning; they use motion vectors to track the movement of objects on screen, for example.
I've been pondering on something related: can an LLM be a chat?
Some models are reproducible, in that the same prompt will generate the same output. Say that we could wire up such a model to generate some code.
In that case, we could create a prompt that generates, say, an entire codebase, or a large piece of text. The prompt (or really, the tokens) would then be the compressed version of the codebase or the text.
I am not talking about an "AI agent", but really a model that we call in a reproducible manner. Preferably one call, with one prompt. An agent could just run `git clone` to "decompress" a codebase, which conflates the idea of compression. If that were compression, then the "compressed version of the git kernel" would be a single line of text: `git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...`. I am really talking about having an LLM re-generate text based on a prompt.
Does that make sense? I can imagine that this is highly impractical and inefficient. But would this count as "compression" at all?
You're describing https://bellard.org/ts_zip/ ("Text Compression using Large Language Models") ?
Man,did that page load fast. It made me realize how slow the rest the (my) web is.
You sound a bit confused.
A large language model itself (the network) give you the probabilities for the next token given some prefix of tokens so far. You can use arithmetic coding to go from these probabilities to a deterministic compression / decompression algorithm.
When you use an LLM to generate text, you sample from that probability distribution. You can use a true random sample. Or you can make it trivially deterministic by using a seeded pseudo-random-number-generator or you just pick the highest probability each time. But that's all a red herring; really, what you want is arithmetic coding.
https://en.wikipedia.org/wiki/Arithmetic_coding
>I've been pondering on something related: can an LLM be a chat?
A chat?
>I am not talking about an "AI agent", but really a model that we call in a reproducible manner.
An LLM is just as deterministic as any other computer program. For identical inputs (which includes the PRNG seed) it produces identical outputs.
>compressed version of the git kernel
The git kernel, got it.
>But would this count as "compression" at all?
Yes. The decompressor is several tens of gigabytes though.
>An LLM is just as deterministic as any other computer program. For identical inputs (which includes the PRNG seed) it produces identical outputs.
This is not really true in practice because of multi-threading and out-of-order execution. Mathematically equivalent orderings of operations are not equivalent when dealing with floating point values, so most practical LLM implementations end up being non-deterministic.
"An LLM is just as deterministic as any other computer program" is not refuted by pointing out hardware limitations that would affect any other computer program implemented at similar scale (weather forecasts, or even just computing the average of a large stream of sensor readings).
I'm not really trying to 'refute' the original statement. Certainly, an LLM is just doing some calculations that can be done deterministically in principle. However, I think it's worth pointing out that there are practical barriers to doing those particular calculations both deterministically and efficiently. People who worry about LLM output not being reproducible aren't necessarily misunderstanding what an LLM is doing; they are responding to a real feature of most practical LLM implementations.
Sounds like an interesting way for future OS included apps to be distributed.
Like when you click the Calculator button on your android, it wouldn't actually exist yet, your click actually prompts it into existence. But naively that has problems because you don't want a different UI every time. There's something to your idea.
Someone did it!
https://youtu.be/7NfyZhV1dKM?is=YOUXCHuFUiPdlD0p
This was tried many times in the past for images, even before LLMs.
https://imalogic.com/blog/2024/06/03/image-compression-decom...
It makes a lot of sense. I thought about it in the context of pull requests or change sets: if the text-to-code process is reliable, why don't you give me prompts instead of code? Code becomes just an intermediate representation.
Previous discussions of that specific page https://news.ycombinator.com/item?id=48557691
I think language itself is compression, so the arxiv paper tracks for me.
Viz. if Language is compression (of thought / culture / the tacit je ne sait quois of being-to-being communication etc.), then definitionally, Language Modelling must also be Compression.
Except, language is an arbitrarily lossy compressor, who's "compression-prediction equivalence" is indeterminate and unstable, because Language co-evolves constantly; both as a function of or response to culture, as well as an influencer of culture.
So, the subjective-objective goodness of Language Models (of any kind of language) would be, at best, upper-bounded by the compression-prediction equivalence of the Languages corpus itself. And that is assuming the language corpus is perfect in every way---it captures all knowledge expressible by language and it is always in-sync with live evolution of all language expression and evolution (i.e. LLM training is not a batch job, but a real-time present continuous process).
For example, to my layperson eyes, the mathematical language of proofs actively weeds out ambiguity of subjective interpretation. Ideally, a proof ought to lead to the exact same conclusion on every single reading by any reader who can follow the steps. A proof also holds only if the rest of the formal, explicit, inviolable, internally-consistent set of axioms and results holds.
So it stands to reason that mathematical prose of proofs, being optimised as mechanical procedure of taking an open question to a deterministically closed solution, has better odds of approximating the tacit aspects of mathematical derivation.
Which makes an LLM able to construct a mathematical proof, which is mind-melting to say the least.
However, I wonder, can LLMs dream of mathematical sheep?
Language is compression of a sort. A dictionary is a decompressor. You look up one word, and you may get a paragraph about its meaning. An encyclopedia can be thought of as roughly the same with more detail for some nouns.
It makes a lot of sense why dictionary-based compression is named the way it is. A shorter symbol is used to store information that would take more symbols in the uncompressed corpus, if the shorter symbol hadn't been assigned to represent it. That's in a way just what an actual dictionary on your English professor's shelf does. The big difference is your compressor is coining new short symbols all the time.
I'd say the bigger difference is the lack of homophones in a compressor.
Languages add new words constantly, albeit slower than a computer does compressing a new file.
My goodness, gzip is intelligent! Enjoy your job while you have it; put on some hip hop music and practice your pivoting.
“Solving” mnist with gzip: https://jakobs.dev/solving-mnist-with-gzip/
Fun topic but generated article text and then not even actually using gzip? Rubs me in a weird way.
R. Hendricks, D. Chugtai, and J. Dunn, "Lossless compression via optimized middle-out bitstream processing," Pied Piper Inc., Palo Alto, CA, Tech. Rep. 42, Apr. 2014.
Interesting approach, I wonder how this could be used as a classifier. :)
Check out "Text classification with Python 3.14's zstd module" (https://maxhalford.github.io/blog/text-classification-zstd/, https://news.ycombinator.com/item?id=46942864). I wanted to link it somewhere in the comments. :-)
I've used LZO as a spam classifier on chat. Spam tends to be very content-less and repetitive...
I was curious to see how this would work with bzip2 and zstd. The source is public at https://github.com/nathanrs/gzipt, and I asked MiMo-V2.6-Flash to fork and modify it in a straightforward way. The answer is that bzip2 produces sequences that don't resemble human language:
Line breaks added. This looks roughly optimized for the most repetitive Burrows-Wheeler transform (https://en.wikipedia.org/wiki/Burrows%E2%80%93Wheeler_transf...). Why are they runs of alternating symbols and not one symbol?Zstandard produces whitespace with the occasional letter thrown in. To quote MiMo: "As you can see, zstd does not speak Shakespeare. ... zstd encodes a run of one repeated byte as a near-free run-length sequence, and space and newline are the cheapest literals in the corpus: ten newlines cost about the same to append ten bytes of genuine corpus text and less than nonsense does."
Did you check MiMo correctly performed this unfamiliar task before posting this comment?
I did. I read the code to make sure the quality of MiMo's work matched mine for a quick experiment, though not that the code was free from subtle bugs.
This was the main change for bzip2:
Is the fact that the original did [compress base]+[compress seq] rather than [compress [bytes + seq]] not important?
(I honestly don’t know is gzip does something different when presented with two chunks as opposed to one, or, if it does, if bz2 has equivalent behaviour - but the difference in the code did stand out to me, and it does seem related to ‘extending the token sequence’)
This difference doesn't matter because of how zlib works. At least by default, zlib divides the input data into its own blocks independent of the caller. If you don't feed it enough data to complete a block, it waits until you feed it more or finish the stream.
We can test it by going back to zlib:
At temperature zero, this outputs the same sample as commit 3734bf6, the most recent commit upstream: I also tried LZMA for good measure: The sample at temperature zero: This is followed by a lot of whitespace.python-lz4 gives you all newlines after the prompt. I tried debugging it, and the compressed length of different candidate seqs is the same.
So, you had an AI write code you don't understand, then posted output you don't understand in a comment on the internet for other humans to read?
Good article.
Compression is a property of language.
A seemingly simple sentence like "I had lunch" has enormous amount of information compressed inside it.
The word lunch is a compressed form of "having food at noon" while "noon" in turn is a compressed form of "Sun's position against Earth's rotation" and so on and so forth.
Every sentence has layers of compressed sentences. How many layers one chooses to decompress is up to the person.
Can JPEG be a vision model?
See also the Hutter Prize [1]. Compression is a subset of Intelligence.
[1] https://en.wikipedia.org/wiki/Hutter_Prize
Only if it can run DOOM.
Yes.
3blue did a series of videos on this
https://www.youtube.com/watch?v=l6DKRf-fAAM
Will Wright once described the same connection from the opposite direction: compression as procedural content generation.
In the 2023 discussion of "Demoscene accepted as UNESCO cultural heritage in The Netherlands" I posted a transcript from a video of Will Wright discussing the demo scene:
https://news.ycombinator.com/item?id=36599415
Will Wright Discusses the Demoscene:
https://www.youtube.com/watch?v=m7iuFVmTJus
>You can take any piece of content in the game, and imagine an algorithmic solution to it. Or also, you know, a way that the player could customize that object of thing.
>There's this group in Europe called the Demoscene that make these very elaborate demos for a computer that fit into very tiny little memory blocks, you know like 64K of memory, and you run the thing, and in fact it algorithmically generates about 100 megabytes worth of data, you know these rich 3D environment, generated music, generated wave files, generated animation.
>And they're developing techniques to generate, you know, huge amounts of interesting data, with very very simple, elegant, compression algorithms.
>And this is a skill that game developers used to have, back in the 8-bit days. That was the only ways to do a game like Karateka(?), was to find all these little tips and tricks to compress things and generate them algorithmically.
>But since the CD-ROM came out, and very cheap hard drives, storage is cheap, so basically we've lost that skill set, and now we attack all those problems with brute force. I think we've lost something by dropping that skill set.
[...]
https://news.ycombinator.com/item?id=36613058
[...] Here's a simple low-tech pre-LLM example that shows the equivalence of compression and procedural content generation:
Take a huge text file of HN postings, and compress it with gzip or compress or some other robust compression algorithm. The better the algorithm, the more the output will look like random noise. Then slice the compressed file in half, and replace the second half with random numbers. Then uncompress it. You'll find that at the point you sliced it, it keeps on writing out almost plausible text for a while, consisting of highly probably snippets of commonly encountered words and phrases, then goes downhill towards incoherence. It's not as coherent or confident as an LLM, but the point is to show how low the bar is for using compression for procedural content generation.
LLMs are essentially a form of compression of the world's knowledge or whatever they're trained on, not just word frequencies or pixel patterns, but also concepts and ideas. [...]
Yay, another mostly AI authored piece with vibe-coded aesthetics.
Some will say that I should 'judge the idea, not the form'.
But if the author didn't find enough strength to write alone a short ~700 words summary about his work, it means he himself isn't that interested or enthusiastic about it. Why should others bother then? Particularly since low-effort like that signals possibility the whole work is superficial and derivative.
I thought it was interesting.
so did I, that's why I clicked.
I'm judging the ideas in your post, not its form.
Your claim that suspected AI assistance proves the author isn't interested -- and therefore that the work is probably superficial -- is unsupported.
The article presents a working experiment, explains why naive decoding fails, describes the beam-search fix, and links the code.
Dismissing all that with presumptuous personal speculation and banal boilerplate drive-by anti-AI snark adds absolutely nothing to the conversation -- and that is intrinsically poor form.
You couldn't even find enough strength to criticize anything beyond the form, while your own form is lackluster.
Ironically, an LLM could have written your comment and improved its form without losing anything distinctive.
> I'm judging the ideas in your post, not its form.
and simultaneously you write that 'my form is lackluster' and that 'an LLM could have written your comment and improved its form without losing anything distinctive'. We are having ourselves a small contradiction, aren't we.
Be my guest, enjoy chatbot writing and drowning in slop. But don't encroach upon my freedom to protest it.
Ironic, isn't it? Yet you continue your lackluster form devoid any criticism of ideas. All you can criticise is form, yet my criticism of your ideas still stands:
Your claim that suspected AI assistance proves the author isn't interested -- and therefore that the work is probably superficial -- is unsupported.
So I'm encroaching but you're only protesting, huh? I also have the freedom to ironically protest the poor form of your inability to criticize ideas, as well as your poorly formulated unsupportable ideas.
> Ironic, isn't it? Yet you continue your lackluster form devoid any criticism of ideas.
Pointing out someone's contradiction is now being 'critical of form'? And someone's contradicting themselves is 'ironic'?
Now I'm not even sure you know the meaning of words you use. EOT from me.
Say you have been working on a multi year project writing code and putting in a lot of effort.
If you then use AI to write a product page documentation and proof read it for correctness, would that constitute to signaling that the whole effort is superficial?
Some work may be left to AI while you focus on the more important aspects of the work.
Surprisingly people have got it completely backwards where they want AI to generate code and humans to write documentation.
Not without attention or something approximating it.
The fact that gzip is relatively fast should be your first clue that something important is missing.
Gzip is great at predicting the next token for one very specific narrative. LLMs can predict next tokens for entire universes of narratives. Searching for the correct next token across this space scales ~quadratically with the input size. Gzip scales linearly. I can gzip a one terabyte file. Imagine feeding that much into an LLM. These are wildly different animals that happen to overlap in a very small way. Equating compression to intelligence looks increasingly silly to me.
If we must compare language models to compression, they are much more like jpeg and mp3 than they are gzip and flac. I can go fuck with a jpeg file pretty severely at the bitstream level and still have something resembling performance on the other side. Gzip cannot remotely approach this.
> Gzip scales linearly. I can gzip a one terabyte file.
In part because gzip only has a 32KiB window size, and I think it'd be at least quadratic within that window if you were going for optimal compression.
I'll concede the window part, but Gzip runs within the physical confines of a single cpu core and is typically entirely resident in local caches. The point is not just the quadratic scaling but also what it scales with.
Show me an LLM that can run at 300 megabytes per second. Even dedicated ASICs with weights burned in will never move this fast.
It won't move as fast as gzip provided the same resources, but to say that LLM's can never run at 300mbps on any hardware is ... a bold bet.
Match-finding does not need to be quadratic. However, truly optimal gzip block splitting is very slow, indeed.
I agree, but also equating LLMs with intelligence is wrong.
Perhaps a better question is if LLMs are used as compressors, how well is that expected to work.
> if LLMs are used as compressors, how well is that expected to work
Quite well. This project[1], by Fabrice Bellard of ffmpeg fame, is quite old in AI years and uses an ancient LLM, but still beats xz by a solid margin.
[1]: https://bellard.org/ts_zip/
Makes me wonder if compression ratio can be used as a measure for intelligence. Any benchmarks using it?
> Any benchmarks using it?
A challenge as I understand it is reproducibility.
Normal LLM runtimes aren't typically fully reproducible even with same random seeds for distribution sampling, due to floating-point numbers, batching and such.
Though averaging over many runs could alleviate that I suppose.
While it would measure some aspects of intelligence, I'd argue it fails to capture other, more creative aspects.
Extremely well, aside from speed.
> aside from speed.
And energy consumption.
I wonder if, on an information theoretical level, compression radiates heat, and decompression absorbs it?
I think not. it's true that a large language model compresses knowledge and allows us to decompress knowledge. but gzip compresses data and not knowledge. with a LM you can decompress various forms of knowledge from the same data. gzip is a 1 to 1 kind of decompression where as an LM is a 1 to infinity kind of decompression.