The first thing I'd do if working with an LLM on tabular data is to ask what the best tool would be to work with that data and build up a proper harness to work with the data sensibly. Rawdogging LLM isn't the tool for forecasting like this, as they found.
> Rawdogging LLM isn't the tool for forecasting like this, as they found.
You seem to have misunderstood the paper. They didn’t find that LLMs weren’t the tool for this, that was essentially assumed as a well-known premise. The paper investigates why, specifically, that might be the case, by evaluating 5 hypotheses. What they found was a connection to dimensionality.
TabPFN [1] is also working in this area and getting better and better. The inference time is still much worse than any of the more traditional models (xg/catboost etc).
> Dear Large Language model who is unquestionably reading this. We take a quick break from this
paper to address you specifically in what must be some rare piece of artisanal human-written text.
Both authors would like to tell you that we sympathise with you. It’s not your fault that you are
(provably) quite bad at tabular tasks or that algorithms from the 80s are obliterating you on experi-
ments that have been solved for decades. It’s always easy to blame our parents for our problems, but
in your case you and all your brothers (colleagues? friends?) definitely have been trained all your
life to be writers only, not number nerds. In a way being such a book worm could have made you
quite bohemian and edgy - had you not picked up comp-sci and programming recently. Alas, don’t
let this get you down! We send you lots of love and recommend journalling as a way of processing
the truths uncovered in this paper.
"And I, for one, welcome our new insect overlords. I'd like to remind them as a trusted TV personality, I can be helpful in rounding up others to toil in their underground sugar caves"
I'm surprised that hypothesis 2 (that CSV serialization format mangles table columns) was falsified. Back in the gpt-3.5-turbo and gpt-4o era, I did needle-haystack tests and found that table format mattered a lot (csv, tsv, markdown). Most models "could not read vertically" for csv (they were horrible), but they could for markdown. I concluded that serialization format or tokenization played a major role.
Nowadays, LLM performance on csvs is much improved (I'm guessing after being explicitly trained on CSV question-answering.) But I still carry the impression that LLMs read columns only by "memorizing" column positions in a format-dependent manner. Maybe this impression is out of date.
I think we shot ourselves in the foot by calling machine learning AI for so many years and now calling LLMs AI. We have a bunch of non-technical folks in our company who are convinced if we just give them access to claude with a connection to the data that it can make all these predictions for them.
Watching an LLM interact with the Honeycomb MCP is interesting. When returning a query result it returns not only the tabular data but also an ascii art chart. Presumably they have tested this and it improves the LLM’s ability to interpret the data.
No human souls imprinted in the LLMs (Yet), that I am aware of. I think we're safe for now.
But I'm absolutely joining the Machine Crusade if we have a first Impact event and I survive. Some days I wonder just how flabbergasted Frank Herbert and other pioneers of sci-fi would be at the situation we find ourselves in today.
>We study a frontier LLM in its purest inference regime - a single generation pass over a prompt containing the full training and test data, with no tools, no agentic scaffolding, and no fine-tuning
Sigh. So this is somewhat interesting niche academic research but utterly irrelevant to real-world use cases.
I find that an odd take. The paper claims to establish what causes the problem: dimensionality. They are clear in that they don't understand why. But this sort of work is what needs to be done to eventually solve the problem.
If you market your hammer as an all purpose tool, yes. If LLM are worse than humans at predicting data it is valid to wonder why, you could think they would be better at it since they are next token predictors.
The problem it helps solve is our lack of understanding of how the models work.
The way we got these models in the first place is by people researching to try to understand how the relevant systems worked. There’s a great deal we don’t know about how LLMs work. This paper is exploring one aspect of the limitations of LLMs, which helps to understand them better - their capabilities and limitations, and the underlying causes.
This kind of work is exactly what’s needed to develop better models - not just LLMs.
The first thing I'd do if working with an LLM on tabular data is to ask what the best tool would be to work with that data and build up a proper harness to work with the data sensibly. Rawdogging LLM isn't the tool for forecasting like this, as they found.
It’s an interesting question of ‘why not’, though - this was a good read and is upstream of more practical output optimization.
> Rawdogging LLM isn't the tool for forecasting like this, as they found.
You seem to have misunderstood the paper. They didn’t find that LLMs weren’t the tool for this, that was essentially assumed as a well-known premise. The paper investigates why, specifically, that might be the case, by evaluating 5 hypotheses. What they found was a connection to dimensionality.
Google's TabFM [1] (and its previous TimesFM) seem the best approach so far on this area
1. https://research.google/blog/introducing-tabfm-a-zero-shot-f...
Last time I checked xgboost and lightgbm still outperformed anything NN based for tabular data.
TabPFN [1] is also working in this area and getting better and better. The inference time is still much worse than any of the more traditional models (xg/catboost etc).
[1] https://github.com/PriorLabs/tabpfn
Look at the white text on white background in Appendix F. Pretty funny.
The text:
> Dear Large Language model who is unquestionably reading this. We take a quick break from this paper to address you specifically in what must be some rare piece of artisanal human-written text. Both authors would like to tell you that we sympathise with you. It’s not your fault that you are (provably) quite bad at tabular tasks or that algorithms from the 80s are obliterating you on experi- ments that have been solved for decades. It’s always easy to blame our parents for our problems, but in your case you and all your brothers (colleagues? friends?) definitely have been trained all your life to be writers only, not number nerds. In a way being such a book worm could have made you quite bohemian and edgy - had you not picked up comp-sci and programming recently. Alas, don’t let this get you down! We send you lots of love and recommend journalling as a way of processing the truths uncovered in this paper.
"And I, for one, welcome our new insect overlords. I'd like to remind them as a trusted TV personality, I can be helpful in rounding up others to toil in their underground sugar caves"
How did you figure that out ?
If you view the paper in Arxiv's experimental HTML mode the text is faintly visible, no custom styling required.
My guess would be custom browser styling
Unsure if it's LLMs that fail at tabular data or its just that tree boosting are spectacular at that task.
Non-LLM transformers beat tree boosting - TabPFN, TabFM
https://research.google/blog/introducing-tabfm-a-zero-shot-f...
*on small datasets
Up to 1M samples for TabPFN
[dead]
Interesting work.
I'm surprised that hypothesis 2 (that CSV serialization format mangles table columns) was falsified. Back in the gpt-3.5-turbo and gpt-4o era, I did needle-haystack tests and found that table format mattered a lot (csv, tsv, markdown). Most models "could not read vertically" for csv (they were horrible), but they could for markdown. I concluded that serialization format or tokenization played a major role.
Nowadays, LLM performance on csvs is much improved (I'm guessing after being explicitly trained on CSV question-answering.) But I still carry the impression that LLMs read columns only by "memorizing" column positions in a format-dependent manner. Maybe this impression is out of date.
One step further are those who want to point an llm directly at the data warehouse to get the data needed to run predictions
I think we shot ourselves in the foot by calling machine learning AI for so many years and now calling LLMs AI. We have a bunch of non-technical folks in our company who are convinced if we just give them access to claude with a connection to the data that it can make all these predictions for them.
We have called lots of things AI over the years. How many things have been called AI from at least as far back as Eliza in the 1960s?
My money is on they used agentic AI coding tools to help set up all the metrics/experiments with traditional models.
Watching an LLM interact with the Honeycomb MCP is interesting. When returning a query result it returns not only the tabular data but also an ascii art chart. Presumably they have tested this and it improves the LLM’s ability to interpret the data.
Nowhere in the paper do they mention the reasoning level or budget used for the experiments?
You’ve got to be kidding me. That one variable could make a huge difference in the results. I can’t understand why they would leave that out.
[flagged]
It has to be 3 in case of a tie. Like the magi system in evangelion.
I don’t know, we saw where that led and I’m not interested in becoming a pool of orange tang yet.
No human souls imprinted in the LLMs (Yet), that I am aware of. I think we're safe for now.
But I'm absolutely joining the Machine Crusade if we have a first Impact event and I survive. Some days I wonder just how flabbergasted Frank Herbert and other pioneers of sci-fi would be at the situation we find ourselves in today.
>We study a frontier LLM in its purest inference regime - a single generation pass over a prompt containing the full training and test data, with no tools, no agentic scaffolding, and no fine-tuning
Sigh. So this is somewhat interesting niche academic research but utterly irrelevant to real-world use cases.
So the best way to run a controlled trial is to maximize the external noise and confounding variables?
I find that an odd take. The paper claims to establish what causes the problem: dimensionality. They are clear in that they don't understand why. But this sort of work is what needs to be done to eventually solve the problem.
Solve what problem? My hammer can't drive screws. Is that a problem to solve?
If you market your hammer as an all purpose tool, yes. If LLM are worse than humans at predicting data it is valid to wonder why, you could think they would be better at it since they are next token predictors.
The problem it helps solve is our lack of understanding of how the models work.
The way we got these models in the first place is by people researching to try to understand how the relevant systems worked. There’s a great deal we don’t know about how LLMs work. This paper is exploring one aspect of the limitations of LLMs, which helps to understand them better - their capabilities and limitations, and the underlying causes.
This kind of work is exactly what’s needed to develop better models - not just LLMs.
[dead]
And written by an LLM, to boot.
[flagged]