In Gemini at least, if you look at how they process PDFs, they do an OCR and then feed the text + image to the model, without charging you for the text tokens (I believe).
So my guess is that Claude’s backend is doing the same — so this hack is probably more of a loophole in token accounting that might get closed if Claude is doing what Gemini does
This is really fascinating to me. I was reading this article and originally agreed with you, "I mean, under the covers it's got to be converting to text tokens at some point, so there is no way it's actually cheaper for Claude itself to execute."
But then there is a comment below talking about how DeepSeek was able to get a huge improvement in compression by using visual tokens, https://news.ycombinator.com/item?id=48777848. I don't fully understand all of the underlying technical details so I am still fundamentally baffled about how going the OCR route could actually result in overall electricity/computational savings.
LLMs have a very bloated in-memory representation for text, on the order of megabytes of KV cache per byte of text. Meanwhile, for images a lossy representation is considered acceptable and it only takes up maybe a kilobyte of KV cache per byte of image. So if you can render your text into a hundred bytes of image per byte of text and then lossily expand it into 100 kB of KV cache per byte of text, you come out ahead!
Whether such lossy compression is acceptable for your use case is up to you.
I don't think it's that bad, if I recall correctly it's about 8 kilobytes per token, and a token can be 3-4 characters so you're talking ~2 kilobytes per character.
An image token I recall is something like 16x16, so you get 32 bytes of overhead per pixel. And a character is minimally like 20 pixels including the whitespace, so you've jumped from 4 characters per token to maybe 12.
So 3x savings... which actually maps pretty closely to 60% savings.
It's not quite as bad as the parent made it out to be, the largest I've seen is 32kB per token (where sometimes, a token represents a byte, but usually it represents more than one.)
It's forced by the nature of how LLMs use vector embeddings for language.
Basically, a single token in a LLM is represented as a n-element vector, where n is the "hidden dimension", also known as model dimension. In order for the model to be smart, the hidden dimension needs to be large, on the order of 2^16 on top-tier models. Elements of this vector are typically quantized to 2-byte floats, or sometimes smaller. Every possible fact is embedded as a direction in this very many dimensional vector space, and a token is related to a fact if the vector representing that token points into a similar direction as that fact. You can do vector math about these things, famously for most trained models, if you find the vector embedding for king, man, woman and queen, and calculate king - man + woman, the result is very close to queen.
(Does that mean that there are 2^16 possible different kinds facts about things in this model? No, because high-dimensional geometry is very unintuitively powerful. The facts are not axis-aligned, and they don't need to be perfectly non-orthogonal. This matters, because the numbers of individual vectors you can fit into a single 2^16 dimensional space that are orthogonal with each other (all angles 90degrees) is of course 2^16. But, if you allow for almost orthogonal vectors, the number is larger than the amount of atoms in the universe. If this sounds wacky, for people with a CS background it can help to think it working a bit like a bloom filter, in that collisions are possible. Although in actuality they are theoretical, because 2^16 is a very large number.)
“I mean, under the covers it's got to be converting to text tokens at some point”
Multi-modal models do actually natively tokenize images, though. So it doesn’t have to be converted to text for it to work. They may do it anyway for accuracy, but it’s not at all required.
Effectively an image is scaled to a standard size, rasterized / cut up, and each cut is assigned a separate token, much in the same way text is tokenized. Train the model on this as well and you’ll end up having a model that can understand images.
I am trying to get rough summaries of long PDFs of scanned pages of text. At first I was doing OCR and passing the (tens of thousands of) characters into the LLM, which works, but it's expensive.
I asked Gemini how to save costs and it said just send in all the images of the pages instead. Instinctively, as a developer, it's hard to fathom how sending 200 images is cheaper than sending the text, but it definitely works.
>This is really fascinating to me. I was reading this article and originally agreed with you, "I mean, under the covers it's got to be converting to text tokens at some point, so there is no way it's actually cheaper for Claude itself to execute."
It'd be weird if they were doing this, since it would mean the context window size was a lie and that the API would presumably reject requests whose expanded form went over the 1m limit. For someone using pxpipe with an effective context compression of 90% in some instances, it'd hit the limit at barely 100k.
That was DeepSeek OCR, not a DeepSeek lineage LLM. If the idea is introduced in LLMs, then you're right. But Gemini is not doing that, not yet. This is something I literally discussed with Claude last week, but took its word for it.
Deep seek OCR is an LLM, just one trained/post-trained specifically for OCR.
Exact details of text to image compression ratios are of course extremely dependent on the model architecture, training data, training objectives, etc., so there's probably not too much justification for generalizing to all models
Then you are paying for the electricity. It's not physically possible to do more computation & not use more energy b/c every arithmetic operation requires a minimum amount of energy so more operations = more energy.
Not necessarily. See the paper See "DeepSeek-OCR: Contexts Optical Compression" [1]
One option, when an image is fed into an LLM, is to divide it into tiles, then those tiles pass through a 'vision encoder' neural network to make 'vision tokens' which are then input into the LLM much like text tokens are. Obviously you train the vision encoder and LLM to understand one another. This is known as an 'end-to-end OCR model'.
And it turns out, once you've trained a model to do this, you can vary the number of 'vision tokens' used to represent a given text document by scaling an image of a document up or down, and see what happens. You also get a load of other parameters like patch size and vision encoder complexity and so on.
Turns out it works really well; in some tests they used 90% fewer input tokens, but still got 97% output performance.
I tried the same thing last year (with openai models), back then it worked to reduce prompt tokens, but you needed way more completion tokens, ultimately more expensive (and slower)
https://pagewatch.ai/blog/post/llm-text-as-image-tokens/
It’s so painful to read the LLM-compressed explanations. I can’t exactly identify what it is, but it’s an immediate tell and literally requires twice the effort to comprehend.
For example:
> Honest caveat, visible in the clip: the pxpipe arm answered the count first and needed one follow-up nudge to also print the ledger balance in the requested one-line format; the plain arm followed the format on the first try. Legibility is solved on Fable — single-reply format compliance is the remaining rough edge.
If I reread this four times, I can sort of interpolate what happened, but it’s mostly pointless and confusing information.
In my experience all models do this to an extent, but Claude seems to be the worst at this. GPT 5.5 is a bit more terse but seems to compress more valuable information.
My thought on this is that LLMs probably mimic writing patterns and structures from quality resources. But they don't construct a plausible thought hierarchy like an average human does, so their train of thought turns into a rollercoaster of thought. So the order of information is for humans completely out of order.
My guess is that it's a known problem, which steered the frontier models into bullet point preference.
To be fair, as you can see in the clip, the two models handled the prompt slightly differently. The pxpipe variant gave the right count initially but needed a quick follow-up to output the ledger balance in a single line. The standard model, on the other hand, nailed the formatting on its first try. We've completely solved readability here on Fable; our only real hurdle left is getting the models to follow formatting constraints perfectly on the very first reply.
Of course, this was just rewritten by another LLM.
Yeah, but as you say, this is another LLM rewriting it. The amount of noncontextual information is nauseating and destroys the point of a README (in my humble opinion).
A human might have written a disclaimer like this:
> When not using Fable, pxpipe may require additional follow-ups to precisely follow your formatting instructions.
This kind of garbled information dump is very inconsiderate of the reader, and all good writing is considerate of the audience consuming it.
To be fair to the rewriting LLM, it was given only that single paragraph, with no other context and asked only to make it more comprehensible. But your point still stands. Here's what it said about _your_ rewrite:
That human rewrite is excellent. It ruthlessly cuts out the "narrative" of the test case (the transaction counts, the video clip, the "first try vs second try" details) and extracts the only piece of information an actual user reading a README cares about: what to expect when using the software.
Which suggests some ideas that should have been included in the prompt, to get closer to your ideal rewrite.
Concise responses come from people with writing skills who can get to the point without adding more words to sound smart.
"Elegant prose instantiated through remarkably tailored execution of written word may allow an author's desired intention to flow in a certain way to achieve a precise effect whilst simultaneously allowing said author to sound of much higher mind and thought to the reader." - I probably butchered it but my point is that AI slop seems to be the average of the outputs.
Images that look similar to others because they're average of all the current outputs. Same with music and video. We're noticing when something is AI because it has this signature that's average to other outputs.
Original content is crafted even though inspired by other works.
We're at a weird point where AI is capable but constrained.
As compute increases and AI becomes more personalised I feel the current implosion will explode again into variety.
dead giveaway for me that something someone made and wants to share = they dont understand what they put together enough to like speak to it with some level of authority.
people can make some really useful stuff with AI especially when its a domain they're already an expert in, and it would go a long ways for them to just sit and explain that 1. they used AI to help 2. their own words to explain what the heck they put together, especially if they can speak to some of the limitations AI has working with it. just goes a long way to demonstrate this guys stuff is worth tinkering with because he has a good grasp on what was created
for 99% of the stuff out there now people are literally operating in domains they don't understand at all, i just close my tab when i see the damn vibe coded readmes
Step back and think about it another way - ask which scenario is more likely:
Some random person discovered a 60% across the board gain in all LLMs, using an extremely simple trick that none of the labs noticed in all these years. That trick being to rasterize 8bit characters into 8x8 pixels in a big image. 60% in a market worth trillions of dollars.
or
Anthropic's marketing team arbitrarily prices tokens to drive growth, according to vibes and feelings, and didn't think they needed to price images on par with text in their rush to burn cash & drive growth. Some folks take advantage of the trick during the first few days of the model's availability before Anthopic corrects their pricing, to align more proportionally with actual compute costs.
Nah, optical compression is a thing. You see it in a lot of different areas in ML. In this case, the "trick" has been known for a while, and belongs to a whole world of compression research. But I think where you're maybe getting mixed up is in where that 60% gain is coming from.
It's not a 60% percent reduction in cost for 100% of the same output. If you have a model and input text A, and you fix the seed etc. and run Text A through the model as text tokens and as compressed image tokens, you will not get identical outputs. You're specifically reducing the number of tensors needed to represent your input, which saves you on raw compute, but also by definition gives you less room to represent the information in your input. It's lossy, in other words.
Put another way, if you're using a model like Fable because you need the absolute frontier of capability and cheaper models cannot solve your tasks, then there is a very real chance that a compression strategy like this drops Fable's accuracy such that it's no longer suitable for your task. Which defeats the point of you paying for the most expensive model in the first place.
So, it's cool research. Might be useful for some people. Probably isn't something that has incredible utility in real use cases.
To me compression implies smaller size? However new line chars seems to be removed in the pic so I guess it could be expressed in fewer bytes than the original text with further compression ...
The size is indeed smaller, because text tokens and image tokens are embedded as vectors of the same size, but text tokens typically only cover a few characters, while image tokens typically cover many pixels, so many that you can fit more characters in there. So the same text takes up fewer tokens as an image, and hence requires less time and memory to process.
You could also imagine models where text tokens cover many characters and image tokens just a few pixels, which would invert the relationship, but this is typically suboptimal for the applications people have in mind when they train a model.
Lots of researchers have done just this! There's a really rich history of research + lots of contemporary work on different encoding/representation strategies. This might be interesting to you: https://sbert.net/
What makes the DeepSeek-OCR and related results exciting to some researchers is less about the fact that you could devise a tokenization scheme that has fewer tokens, and more about how well it works.
> Some random person discovered a 60% across the board gain in all LLMs, using an extremely simple trick that none of the labs noticed in all these years of multi-trillion dollar growth
DeepSeek published a pretty well circulated paper on exactly this many months ago. It just hasn’t been attempted and shared publicly, asa retrofit, AFAIK.
Also, it’s no free lunch, the readme indicates that this “use images” hack is lossy and reduces success rates alongside the reduced cost. Most labs would focus on success increases regardless of price.
If the trick were genuinely useful, and was well circulated months ago, the resource-starved inference providers would have squeezed this trick dry already, instead of wasting 60% of their tokens, waiting for users to implement it themselves in 5 minutes of effort.
That's like saying quantization isn't real because the frontier labs aren't using it in their production inference.
This is a lossy process, it produces worse results. It might be worth it for some situations, but applying it to everything would just be making your SOTA model worse
The "trick" is well documented in their Deepseek-OCR paper, that builds on plenty of other work. It's just not simple to just switch a commonly used LLM architecture to a new one, but I don't doubt most frontier labs are already experimenting with it. This by itself a very active field of research.
Isn't this just quantization with extra steps? Can converting the text to an image really be a better way to lossily compress it? (Not that I have any idea what I'm talking about on this topic.)
No, quantization is applied to model weights or the KV cache (the model activations of all past tokens), and is just storing everything with lower precision (carefully, so that it doesn't hurt performance much).
Sending an image of text instead of text reduces the number of input tokens, but they're still being processed by the model at the same precision. This probably also hurts performance in some way – the question is by how much.
I also have no idea what I'm talking about, but to me this seems closer to the "caveman mode" that some people use to compress info into fewer tokens. Going through the image tokenizer allows you to leave the source text untouched while still gaining (some of?) the benefits
I think you missed the part where this is a lossy technique that reduces performance.
The image trick reduces context because it’s lossy. The README says you can’t use it for anything needing exact recall. It produces a gist of the input.
You could achieve something similar by using a small, cheap model to pre-summarize information for the expensive LLM. This is what many people do already and it’s a much better way to do it for most situations.
This has been known since VLMs were a thing, that more information can be encoded visually and token efficiency is increased. But it came with performance issues (more hallucinations, etc).
Also I don't think you realize how much dumb stuff is still left on the table. That the market is worth trillions is quite irrelevant here given the dynamism of the field.
Alternative 1 isn’t all that unlikely given Opus 4.8 couldn’t do this. So it’s a recently possible hack. Not something LLM corps have been blindsided by for years. I also strongly recommend RTFA in this case, namely ”The honest part, read before relying on it”
I kinda wonder if it's extracting usable context from 2D proximity between lines? Normal text input wouldn't have that kind of information (though it could, and it's arguably just a lookahead/behind of N characters on average).
A token is probably not a single char, and an image is probably decomposed into tokens as well (and god knows how many tokens an image is decomposed into) which probably map to similar float-hungry vectors. Your counterargument could use a bit more flesh.
And we're talking about images of texts, not images that represent complex imagery such as a very detailed scene or what have you.
People really need to read their cites and not just the summaries.
The paper notes two things:
1) While the compression ratio for visual text is better than it is for regular text, but the absolute space required is still higher for the images. OPs were talking about the space required, not the ratio.
2) The results of the OCR must still be fed into a text-based LLM for linguistic processing. Otherwise, all you have achieved is turning an image into a bunch of text.
huh, what if the image encoding is 8 bits per R, G, B values of the pixel, then one can encode the same amount of text in less pixel dimensions (3 letters would need 1 pixel instead of three 12x12 pixels)
The top line can be the OCR-able instruction on how to decode the rest of the image, and the rest of the image would be random-looking colourful palette. It might not even need to use 8 bits per character, since ANSI is 7 bits/character.
Download the .bmp file and open it with 'edit' on windows or 'nano' on linux. You should see text. The bytes that precede the text are padding and a header that roughly 'tells how to interpret the data', in the sense that it describes the dimensions of the 'image'.
There's python code that shows how the file was generated, note that it just writes a string to the file and then wraps it with a .bmp header so that it can be interpreted by image viewing programs.
If you open that image with an image viewer you will see an image with colorful pixels. Whether you send it as an image or as text, the only difference lies in what you declare the file to be. For example if you change the extension, operating systems will "open" it with the appropriate program by default, but the data is the same, there's no difference between ASCII text and the encoding you describe where each subpixel describes one character, it's the same thing.
Your idea about using 7 bits instead of 8bits is more space efficient, but you'd need to decompress it so that the text is byte aligned. In terms of costs, you are not charged by the bandwidth consumed in transit, but by the computational cost, which wouldn't change as the text would need to be expanded to 8bits per char, (and then converted to the same token anyways)
Again don't want to be mean but these are 1st year comp sci topics, not at all related to LLMs. Hence why I recommend studying fundamentals more than papers that look like bleeding edge LLM research.
That's not what is happening. Claude isn't charging for the tokens it generates from the OCR on its side, but it's still processing the same number of tokens as if you had sent the text, just with the extra step of OCR on Claude's side. This is 100% a loophole that's burning extra resources.
Not really. They arent actually using more resources this way either. This might be a fundamental inefficiency thats being removed
It kinda makes sense too. Because while people do read code word by word, we often "glance over" it and do roughly pattern recognition on it to know what it does. Only homing in on something when we need to answer a specific question. I think humans kinda naturally do this exploit anyway
So, just be careful with this, it very likely is switching to other less capable model hence the cost reduction. So looks like Fable but isn’t. So you are doing extra work when you could just switch the model back to opus 4.8 instead.
Saw a Tweet a while ago from someone (maybe Carmack, maybe Geohot, maybe Karpathy?) wondering if images were just the better option.
Since then I've been using images with very simply worded prompts whenever I'm informing an agent of what is happening. Sometimes no text in the prompt at all.
It has been very very effective.
That being said, this isn't really what Karpathy was talking about. But it got me thinking a bit, and that got me to a much nicer workflow.
I'm sorry, but this is retarded. It works, and it's clever, but but it's clearly a workaround for a pricing failure. Much like the bounty on poisonous snakes leading to people taking up snake-breeding, this just exploits and promotes waste. I think ultimately blame falls on Anthropic for the poor pricing system the enables such arbitrage. But I'm also disgusted by the inevitable tide of people exploiting this until its fixed, and creating an entirely unnecessary extra tide of digital junk.
Tangentially related: I don't think OCR is the right term and I am generally vocal about that. But seeing this unquestioned here, I am wondering if I am the one who is wrong here. Is it ok to call this OCR? To me ocr means text in the end, not visual tokens.
OCR means optical character recognition. The terms do not require a direct transcription, but that is mostly what OCR meant in the past. If you’re using an LLM’s vision capability to pass in text and the LLM actually understands it, then I would say that it recognized the characters, hence OCR seems okay to use.
But characters only exist when we ask the model about this and it does it best to do this projection if asked. Vision model are richer than that. It "understands" visually a document. If it was only about characters, then there will be no way it beats the traditional pipelines of image->text->extractions or obtain the kind of results we see in this article. Vision models are more than characters recognition and OCR term don't do it justice IMHO.
seems really dumb and like it would need to violate basic information theory to work?
input tokens are cheaper than output tokens. seems like it would maybe reduce input tokens at the expense of many more output tokens if you're actually triggering OCR via thinking?
It's far from being a foolish idea, and it seems to me that the project correctly documents its own limitations. It's rare to see a README list so precisely where the tool falls short.
Then, yes, input tokens are cheaper than output. But when it comes to find ways to get reduce costs, you have to explore all the options.
Documents are processed as tokens as well, unless its bitmap is ocr'd.
Images tho are natively compatible with Multi-Modal LLMs, so theres no image->text translation layer in between.
It's that the unit of cost is different (e.g. "visual token" vs text token)
it is funny because nobody ever bother points out that they overcharge you for text input token price.
sure it was pretty resource intensity a few years before, but with turbo quant, sparse attention and various techniques, plus the advancing of hardware (dedicated prefill machine, memory pool for kv caching) the cost should be drastically reduced, and yet they still keep the same cost formula.
I can't help but laugh whenever someone proudly share how many billion input tokens they spent in their code sections and how much they saved with the subscription, meanwhile it is pretty much just electricity cost for the providers.
The image is still getting run through OCR and turned back into text before being fed into the LLM. There is no efficiency gain here, rather we have learned that Anthropic is applying a discount to text fed in via OCR.
I don't think this is what is happening, IMO. The models can genuinely "read" the text off the images, but usually at a less-than-perfect ratio, and it uses less tokens for the model on visual input than it does actually using OCR to convert them into text and then sending that in. I do not think there is any intermediate stage where they are applying a free OCR in this situation. (I realize that happens in some situations)
In Gemini at least, if you look at how they process PDFs, they do an OCR and then feed the text + image to the model, without charging you for the text tokens (I believe).
So my guess is that Claude’s backend is doing the same — so this hack is probably more of a loophole in token accounting that might get closed if Claude is doing what Gemini does
This is really fascinating to me. I was reading this article and originally agreed with you, "I mean, under the covers it's got to be converting to text tokens at some point, so there is no way it's actually cheaper for Claude itself to execute."
But then there is a comment below talking about how DeepSeek was able to get a huge improvement in compression by using visual tokens, https://news.ycombinator.com/item?id=48777848. I don't fully understand all of the underlying technical details so I am still fundamentally baffled about how going the OCR route could actually result in overall electricity/computational savings.
LLMs have a very bloated in-memory representation for text, on the order of megabytes of KV cache per byte of text. Meanwhile, for images a lossy representation is considered acceptable and it only takes up maybe a kilobyte of KV cache per byte of image. So if you can render your text into a hundred bytes of image per byte of text and then lossily expand it into 100 kB of KV cache per byte of text, you come out ahead!
Whether such lossy compression is acceptable for your use case is up to you.
I don't think it's that bad, if I recall correctly it's about 8 kilobytes per token, and a token can be 3-4 characters so you're talking ~2 kilobytes per character.
An image token I recall is something like 16x16, so you get 32 bytes of overhead per pixel. And a character is minimally like 20 pixels including the whitespace, so you've jumped from 4 characters per token to maybe 12.
So 3x savings... which actually maps pretty closely to 60% savings.
Can we get some refs for this number? If true it sounds like poor design.
It's not quite as bad as the parent made it out to be, the largest I've seen is 32kB per token (where sometimes, a token represents a byte, but usually it represents more than one.)
It's forced by the nature of how LLMs use vector embeddings for language.
Basically, a single token in a LLM is represented as a n-element vector, where n is the "hidden dimension", also known as model dimension. In order for the model to be smart, the hidden dimension needs to be large, on the order of 2^16 on top-tier models. Elements of this vector are typically quantized to 2-byte floats, or sometimes smaller. Every possible fact is embedded as a direction in this very many dimensional vector space, and a token is related to a fact if the vector representing that token points into a similar direction as that fact. You can do vector math about these things, famously for most trained models, if you find the vector embedding for king, man, woman and queen, and calculate king - man + woman, the result is very close to queen.
(Does that mean that there are 2^16 possible different kinds facts about things in this model? No, because high-dimensional geometry is very unintuitively powerful. The facts are not axis-aligned, and they don't need to be perfectly non-orthogonal. This matters, because the numbers of individual vectors you can fit into a single 2^16 dimensional space that are orthogonal with each other (all angles 90degrees) is of course 2^16. But, if you allow for almost orthogonal vectors, the number is larger than the amount of atoms in the universe. If this sounds wacky, for people with a CS background it can help to think it working a bit like a bloom filter, in that collisions are possible. Although in actuality they are theoretical, because 2^16 is a very large number.)
“I mean, under the covers it's got to be converting to text tokens at some point”
Multi-modal models do actually natively tokenize images, though. So it doesn’t have to be converted to text for it to work. They may do it anyway for accuracy, but it’s not at all required.
Effectively an image is scaled to a standard size, rasterized / cut up, and each cut is assigned a separate token, much in the same way text is tokenized. Train the model on this as well and you’ll end up having a model that can understand images.
I am trying to get rough summaries of long PDFs of scanned pages of text. At first I was doing OCR and passing the (tens of thousands of) characters into the LLM, which works, but it's expensive.
I asked Gemini how to save costs and it said just send in all the images of the pages instead. Instinctively, as a developer, it's hard to fathom how sending 200 images is cheaper than sending the text, but it definitely works.
>This is really fascinating to me. I was reading this article and originally agreed with you, "I mean, under the covers it's got to be converting to text tokens at some point, so there is no way it's actually cheaper for Claude itself to execute."
It'd be weird if they were doing this, since it would mean the context window size was a lie and that the API would presumably reject requests whose expanded form went over the 1m limit. For someone using pxpipe with an effective context compression of 90% in some instances, it'd hit the limit at barely 100k.
That was DeepSeek OCR, not a DeepSeek lineage LLM. If the idea is introduced in LLMs, then you're right. But Gemini is not doing that, not yet. This is something I literally discussed with Claude last week, but took its word for it.
Deep seek OCR is an LLM, just one trained/post-trained specifically for OCR.
Exact details of text to image compression ratios are of course extremely dependent on the model architecture, training data, training objectives, etc., so there's probably not too much justification for generalizing to all models
Baidu released a faster OCR model as well: https://github.com/baidu/Unlimited-OCR
It wouldn’t, they’re subsidizing it for training.
Edit: didn’t realize this occurred on local models(!!),
this is smarter https://news.ycombinator.com/item?id=48779884
can't explain with subsidies a model you host yourself (like deepseek)
Then you are paying for the electricity. It's not physically possible to do more computation & not use more energy b/c every arithmetic operation requires a minimum amount of energy so more operations = more energy.
Not necessarily. See the paper See "DeepSeek-OCR: Contexts Optical Compression" [1]
One option, when an image is fed into an LLM, is to divide it into tiles, then those tiles pass through a 'vision encoder' neural network to make 'vision tokens' which are then input into the LLM much like text tokens are. Obviously you train the vision encoder and LLM to understand one another. This is known as an 'end-to-end OCR model'.
And it turns out, once you've trained a model to do this, you can vary the number of 'vision tokens' used to represent a given text document by scaling an image of a document up or down, and see what happens. You also get a load of other parameters like patch size and vision encoder complexity and so on.
Turns out it works really well; in some tests they used 90% fewer input tokens, but still got 97% output performance.
[1] https://arxiv.org/abs/2510.18234
Claude Science has a tool to extract the PDF but not sure if it's OCR'ing it.
I tried the same thing last year (with openai models), back then it worked to reduce prompt tokens, but you needed way more completion tokens, ultimately more expensive (and slower) https://pagewatch.ai/blog/post/llm-text-as-image-tokens/
Ahhh my eyes the vibe coded readme
It’s so painful to read the LLM-compressed explanations. I can’t exactly identify what it is, but it’s an immediate tell and literally requires twice the effort to comprehend.
For example:
> Honest caveat, visible in the clip: the pxpipe arm answered the count first and needed one follow-up nudge to also print the ledger balance in the requested one-line format; the plain arm followed the format on the first try. Legibility is solved on Fable — single-reply format compliance is the remaining rough edge.
If I reread this four times, I can sort of interpolate what happened, but it’s mostly pointless and confusing information.
In my experience all models do this to an extent, but Claude seems to be the worst at this. GPT 5.5 is a bit more terse but seems to compress more valuable information.
My thought on this is that LLMs probably mimic writing patterns and structures from quality resources. But they don't construct a plausible thought hierarchy like an average human does, so their train of thought turns into a rollercoaster of thought. So the order of information is for humans completely out of order.
My guess is that it's a known problem, which steered the frontier models into bullet point preference.
Here's one rewrite that would have helped:
To be fair, as you can see in the clip, the two models handled the prompt slightly differently. The pxpipe variant gave the right count initially but needed a quick follow-up to output the ledger balance in a single line. The standard model, on the other hand, nailed the formatting on its first try. We've completely solved readability here on Fable; our only real hurdle left is getting the models to follow formatting constraints perfectly on the very first reply.
Of course, this was just rewritten by another LLM.
Yeah, but as you say, this is another LLM rewriting it. The amount of noncontextual information is nauseating and destroys the point of a README (in my humble opinion).
A human might have written a disclaimer like this:
> When not using Fable, pxpipe may require additional follow-ups to precisely follow your formatting instructions.
This kind of garbled information dump is very inconsiderate of the reader, and all good writing is considerate of the audience consuming it.
To be fair to the rewriting LLM, it was given only that single paragraph, with no other context and asked only to make it more comprehensible. But your point still stands. Here's what it said about _your_ rewrite:
That human rewrite is excellent. It ruthlessly cuts out the "narrative" of the test case (the transaction counts, the video clip, the "first try vs second try" details) and extracts the only piece of information an actual user reading a README cares about: what to expect when using the software.
Which suggests some ideas that should have been included in the prompt, to get closer to your ideal rewrite.
[dead]
Reads a little bit better, but still reads like a writer getting paid by the word, which I guess is fitting.
Or maybe: like a copywriter, paid by the dopamine hit
the foamy hedging makes me ill and hurts my eyes
It's about information density.
Most LLMs by default seem to write both text and code with low information density.
You can kind of get around it by prompting them to optimize for compactness, but most just let it run with a more generic prompt.
Concise responses come from people with writing skills who can get to the point without adding more words to sound smart.
"Elegant prose instantiated through remarkably tailored execution of written word may allow an author's desired intention to flow in a certain way to achieve a precise effect whilst simultaneously allowing said author to sound of much higher mind and thought to the reader." - I probably butchered it but my point is that AI slop seems to be the average of the outputs.
Images that look similar to others because they're average of all the current outputs. Same with music and video. We're noticing when something is AI because it has this signature that's average to other outputs.
Original content is crafted even though inspired by other works.
We're at a weird point where AI is capable but constrained.
As compute increases and AI becomes more personalised I feel the current implosion will explode again into variety.
[dead]
What, you don't like your caveats to be honest?
dead giveaway for me that something someone made and wants to share = they dont understand what they put together enough to like speak to it with some level of authority.
people can make some really useful stuff with AI especially when its a domain they're already an expert in, and it would go a long ways for them to just sit and explain that 1. they used AI to help 2. their own words to explain what the heck they put together, especially if they can speak to some of the limitations AI has working with it. just goes a long way to demonstrate this guys stuff is worth tinkering with because he has a good grasp on what was created
for 99% of the stuff out there now people are literally operating in domains they don't understand at all, i just close my tab when i see the damn vibe coded readmes
This seems like a pricing hack that burns resources, that when the loophole gets closed the price of OCR will have to rise?
It’s not a loophole, it just happens that encoding information as optical tokens is much more efficient than text.
Step back and think about it another way - ask which scenario is more likely:
Some random person discovered a 60% across the board gain in all LLMs, using an extremely simple trick that none of the labs noticed in all these years. That trick being to rasterize 8bit characters into 8x8 pixels in a big image. 60% in a market worth trillions of dollars.
or
Anthropic's marketing team arbitrarily prices tokens to drive growth, according to vibes and feelings, and didn't think they needed to price images on par with text in their rush to burn cash & drive growth. Some folks take advantage of the trick during the first few days of the model's availability before Anthopic corrects their pricing, to align more proportionally with actual compute costs.
Nah, optical compression is a thing. You see it in a lot of different areas in ML. In this case, the "trick" has been known for a while, and belongs to a whole world of compression research. But I think where you're maybe getting mixed up is in where that 60% gain is coming from.
It's not a 60% percent reduction in cost for 100% of the same output. If you have a model and input text A, and you fix the seed etc. and run Text A through the model as text tokens and as compressed image tokens, you will not get identical outputs. You're specifically reducing the number of tensors needed to represent your input, which saves you on raw compute, but also by definition gives you less room to represent the information in your input. It's lossy, in other words.
Put another way, if you're using a model like Fable because you need the absolute frontier of capability and cheaper models cannot solve your tasks, then there is a very real chance that a compression strategy like this drops Fable's accuracy such that it's no longer suitable for your task. Which defeats the point of you paying for the most expensive model in the first place.
So, it's cool research. Might be useful for some people. Probably isn't something that has incredible utility in real use cases.
> a compression strategy
To me compression implies smaller size? However new line chars seems to be removed in the pic so I guess it could be expressed in fewer bytes than the original text with further compression ...
The size is indeed smaller, because text tokens and image tokens are embedded as vectors of the same size, but text tokens typically only cover a few characters, while image tokens typically cover many pixels, so many that you can fit more characters in there. So the same text takes up fewer tokens as an image, and hence requires less time and memory to process.
You could also imagine models where text tokens cover many characters and image tokens just a few pixels, which would invert the relationship, but this is typically suboptimal for the applications people have in mind when they train a model.
So split the difference and start encoding input at the words or phrases level?
Lots of researchers have done just this! There's a really rich history of research + lots of contemporary work on different encoding/representation strategies. This might be interesting to you: https://sbert.net/
What makes the DeepSeek-OCR and related results exciting to some researchers is less about the fact that you could devise a tokenization scheme that has fewer tokens, and more about how well it works.
> Some random person discovered a 60% across the board gain in all LLMs, using an extremely simple trick that none of the labs noticed in all these years of multi-trillion dollar growth
DeepSeek published a pretty well circulated paper on exactly this many months ago. It just hasn’t been attempted and shared publicly, asa retrofit, AFAIK.
Also, it’s no free lunch, the readme indicates that this “use images” hack is lossy and reduces success rates alongside the reduced cost. Most labs would focus on success increases regardless of price.
If the trick were genuinely useful, and was well circulated months ago, the resource-starved inference providers would have squeezed this trick dry already, instead of wasting 60% of their tokens, waiting for users to implement it themselves in 5 minutes of effort.
That's like saying quantization isn't real because the frontier labs aren't using it in their production inference.
This is a lossy process, it produces worse results. It might be worth it for some situations, but applying it to everything would just be making your SOTA model worse
The "trick" is well documented in their Deepseek-OCR paper, that builds on plenty of other work. It's just not simple to just switch a commonly used LLM architecture to a new one, but I don't doubt most frontier labs are already experimenting with it. This by itself a very active field of research.
Isn't this just quantization with extra steps? Can converting the text to an image really be a better way to lossily compress it? (Not that I have any idea what I'm talking about on this topic.)
No, quantization is applied to model weights or the KV cache (the model activations of all past tokens), and is just storing everything with lower precision (carefully, so that it doesn't hurt performance much).
Sending an image of text instead of text reduces the number of input tokens, but they're still being processed by the model at the same precision. This probably also hurts performance in some way – the question is by how much.
I also have no idea what I'm talking about, but to me this seems closer to the "caveman mode" that some people use to compress info into fewer tokens. Going through the image tokenizer allows you to leave the source text untouched while still gaining (some of?) the benefits
[flagged]
An economist walks past a hundred dollar bill on the ground because someone would've picked it up already if it were real.
I think you missed the part where this is a lossy technique that reduces performance.
The image trick reduces context because it’s lossy. The README says you can’t use it for anything needing exact recall. It produces a gist of the input.
You could achieve something similar by using a small, cheap model to pre-summarize information for the expensive LLM. This is what many people do already and it’s a much better way to do it for most situations.
This has been known since VLMs were a thing, that more information can be encoded visually and token efficiency is increased. But it came with performance issues (more hallucinations, etc).
Also I don't think you realize how much dumb stuff is still left on the table. That the market is worth trillions is quite irrelevant here given the dynamism of the field.
Alternative 1 isn’t all that unlikely given Opus 4.8 couldn’t do this. So it’s a recently possible hack. Not something LLM corps have been blindsided by for years. I also strongly recommend RTFA in this case, namely ”The honest part, read before relying on it”
Of course it isn't
A text encoding uses 8bits per character on average, tokenization further compresses that
An image font would be 25 bits if 5x5, and most fonts are 12 pixels high
Of course it isn't efficient, this is a pricing inefficiency and a hack to exploit it (even the author describes it as an exploit)
You are wrong.
Text tokens are high-dimensional vectors, not 8 bits per character. Every token has a deep embedding, e.g. 1024 float values per text token.
DeepSeek-OCR proved 10x+ compression from visual embedding of text, which was a groundbreaking result. [1]
Very cool to see OP's project hacking on this principle. It's still not lossless, as noted in the github, but is a promising research direction.
[1] https://github.com/deepseek-ai/DeepSeek-OCR/blob/main/DeepSe...
I kinda wonder if it's extracting usable context from 2D proximity between lines? Normal text input wouldn't have that kind of information (though it could, and it's arguably just a lookahead/behind of N characters on average).
A token is probably not a single char, and an image is probably decomposed into tokens as well (and god knows how many tokens an image is decomposed into) which probably map to similar float-hungry vectors. Your counterargument could use a bit more flesh.
And we're talking about images of texts, not images that represent complex imagery such as a very detailed scene or what have you.
People really need to read their cites and not just the summaries.
The paper notes two things:
1) While the compression ratio for visual text is better than it is for regular text, but the absolute space required is still higher for the images. OPs were talking about the space required, not the ratio.
2) The results of the OCR must still be fed into a text-based LLM for linguistic processing. Otherwise, all you have achieved is turning an image into a bunch of text.
Well, then we could presumably also add lossy compression to texts, without passing through images first
>Text tokens are high-dimensional vectors,
You are conflating tokens with embeddings.
Tokens fit in a single word, modern gpt uses a vocabulary with 200k possible values, which would fit into 18 bits.
Have a good one
huh, what if the image encoding is 8 bits per R, G, B values of the pixel, then one can encode the same amount of text in less pixel dimensions (3 letters would need 1 pixel instead of three 12x12 pixels)
The top line can be the OCR-able instruction on how to decode the rest of the image, and the rest of the image would be random-looking colourful palette. It might not even need to use 8 bits per character, since ANSI is 7 bits/character.
then it's no longer an image, as the one in the github repo, you would be encoding the text as characters and sending it as an image.
You can achieve this by changing the extension of an image file from .bmp to .txt
Guys, not to be mean, but maybe chill with the state of the art research and go back to studying fundamentals.
If you want to be a know-it-all poseur, at least back it up with data.
https://chatgpt.com/share/6a4a7ea3-65e8-83e9-8871-5041d97bb9...
Download the .bmp file and open it with 'edit' on windows or 'nano' on linux. You should see text. The bytes that precede the text are padding and a header that roughly 'tells how to interpret the data', in the sense that it describes the dimensions of the 'image'.
There's python code that shows how the file was generated, note that it just writes a string to the file and then wraps it with a .bmp header so that it can be interpreted by image viewing programs.
If you open that image with an image viewer you will see an image with colorful pixels. Whether you send it as an image or as text, the only difference lies in what you declare the file to be. For example if you change the extension, operating systems will "open" it with the appropriate program by default, but the data is the same, there's no difference between ASCII text and the encoding you describe where each subpixel describes one character, it's the same thing.
Your idea about using 7 bits instead of 8bits is more space efficient, but you'd need to decompress it so that the text is byte aligned. In terms of costs, you are not charged by the bandwidth consumed in transit, but by the computational cost, which wouldn't change as the text would need to be expanded to 8bits per char, (and then converted to the same token anyways)
Again don't want to be mean but these are 1st year comp sci topics, not at all related to LLMs. Hence why I recommend studying fundamentals more than papers that look like bleeding edge LLM research.
[dead]
Truly a picture is worth a thousand words.
That's not what is happening. Claude isn't charging for the tokens it generates from the OCR on its side, but it's still processing the same number of tokens as if you had sent the text, just with the extra step of OCR on Claude's side. This is 100% a loophole that's burning extra resources.
There is no OCR, in the traditional sense, involved.
Anyone else laugh out loud when they read this? Like, okay so NO, that's entirely impossible. What's really going on?
> encoding information as optical tokens
Educate me: what is an "optical token" when dealing with LLMs?
Not really. They arent actually using more resources this way either. This might be a fundamental inefficiency thats being removed
It kinda makes sense too. Because while people do read code word by word, we often "glance over" it and do roughly pattern recognition on it to know what it does. Only homing in on something when we need to answer a specific question. I think humans kinda naturally do this exploit anyway
Related: https://blog.can.ac/2026/06/10/snapcompact/
there's also a DeepSeek whitepaper on this technique https://www.seangoedecke.com/text-tokens-as-image-tokens
I think Oh-My-Pi (OMP.sh) uses images for context compactificaton. OMP is built on top of Pi coding agent.
So, just be careful with this, it very likely is switching to other less capable model hence the cost reduction. So looks like Fable but isn’t. So you are doing extra work when you could just switch the model back to opus 4.8 instead.
Isn't this basically what DeepSeek came up with https://github.com/deepseek-ai/DeepSeek-OCR
Saw a Tweet a while ago from someone (maybe Carmack, maybe Geohot, maybe Karpathy?) wondering if images were just the better option.
Since then I've been using images with very simply worded prompts whenever I'm informing an agent of what is happening. Sometimes no text in the prompt at all.
It has been very very effective.
That being said, this isn't really what Karpathy was talking about. But it got me thinking a bit, and that got me to a much nicer workflow.
[dead]
I'm sorry, but this is retarded. It works, and it's clever, but but it's clearly a workaround for a pricing failure. Much like the bounty on poisonous snakes leading to people taking up snake-breeding, this just exploits and promotes waste. I think ultimately blame falls on Anthropic for the poor pricing system the enables such arbitrage. But I'm also disgusted by the inevitable tide of people exploiting this until its fixed, and creating an entirely unnecessary extra tide of digital junk.
Tangentially related: I don't think OCR is the right term and I am generally vocal about that. But seeing this unquestioned here, I am wondering if I am the one who is wrong here. Is it ok to call this OCR? To me ocr means text in the end, not visual tokens.
It's not. OCR is not what the vision model is doing here. We're used to using OCR as a verb but it's more accurate to say the model "visioned" it.
Also, some models still do OCR and it's usually way more expensive that way.
OCR means optical character recognition. The terms do not require a direct transcription, but that is mostly what OCR meant in the past. If you’re using an LLM’s vision capability to pass in text and the LLM actually understands it, then I would say that it recognized the characters, hence OCR seems okay to use.
But characters only exist when we ask the model about this and it does it best to do this projection if asked. Vision model are richer than that. It "understands" visually a document. If it was only about characters, then there will be no way it beats the traditional pipelines of image->text->extractions or obtain the kind of results we see in this article. Vision models are more than characters recognition and OCR term don't do it justice IMHO.
So if I OCR a document, edit it, and print it, OCR didn't happen?
seems really dumb and like it would need to violate basic information theory to work?
input tokens are cheaper than output tokens. seems like it would maybe reduce input tokens at the expense of many more output tokens if you're actually triggering OCR via thinking?
It's far from being a foolish idea, and it seems to me that the project correctly documents its own limitations. It's rare to see a README list so precisely where the tool falls short. Then, yes, input tokens are cheaper than output. But when it comes to find ways to get reduce costs, you have to explore all the options.
Reminds me of caveman: https://news.ycombinator.com/item?id=47647455
What about: "Read this document online : [URL]" and you add your text/context to an online document?
Would that reduce the number of tokens used too?
Documents are processed as tokens as well, unless its bitmap is ocr'd.
Images tho are natively compatible with Multi-Modal LLMs, so theres no image->text translation layer in between. It's that the unit of cost is different (e.g. "visual token" vs text token)
I see. I was thinking that it might be different if the document wasn't provided by you directly, but instead if the LLM fetched it itself online.
This probably works with PDF parsing as well I’m sure, even if it’s just from not having to parse pdf format alone.
That is hilarious and an amazing find.
Binary compression unpacked by OCR? This is the stuff of nightmares. So cursed, and yet...
it is funny because nobody ever bother points out that they overcharge you for text input token price.
sure it was pretty resource intensity a few years before, but with turbo quant, sparse attention and various techniques, plus the advancing of hardware (dedicated prefill machine, memory pool for kv caching) the cost should be drastically reduced, and yet they still keep the same cost formula.
I can't help but laugh whenever someone proudly share how many billion input tokens they spent in their code sections and how much they saved with the subscription, meanwhile it is pretty much just electricity cost for the providers.
Interesting approach, though that readme really needs a rewrite by a human...
I want to see more text-free foundation models
If the model has to be able to output text it needs to be able to read text tho
a pictures worth a thousand tokens
Are we really re-discovering that compressed binary formats are more efficient data representations?
No?
The image is still getting run through OCR and turned back into text before being fed into the LLM. There is no efficiency gain here, rather we have learned that Anthropic is applying a discount to text fed in via OCR.
I don't think this is what is happening, IMO. The models can genuinely "read" the text off the images, but usually at a less-than-perfect ratio, and it uses less tokens for the model on visual input than it does actually using OCR to convert them into text and then sending that in. I do not think there is any intermediate stage where they are applying a free OCR in this situation. (I realize that happens in some situations)
a picture paints a thousand words
No words.
I cant get past that LLM intense slop text in the Github repo
[flagged]
[dead]
[flagged]
[flagged]
[flagged]