The job performed by the humans was broader than what was requested of the model in this benchmark: humans also had to find the relevant invoices (searching through mailboxes, or requesting them from providers) and reason through any circumstances which cannot be inferred from the bank feed and invoices/receipts on their own. In the benchmark these circumstances are presented to the model as “user notes."
This is precisely the kind of fine print on white-collar AI capability that companies keep running into: pretty much any non-entry office job worth having involves a lot of undocumented (even undocumentable) problems requiring judgment and experience.
And I would be pretty nervous about asking any of the frontier LLMs to retrieve invoices: "cool, Claude logged that it found the May 6th bill from the paper supplier, I am sure it didn't just make something up arbitrary, then compound on the error by agentically iterating over the made-up invoice lurking in its reasoning traces. I checked the first 30 times and there were no problems!"
If and when a large number of companies blindly turn over their accounts payable workflow to some AI agent system, it'll be very interesting to see the "social engineer the LLM" methods that fraud people use to get money sent to them. Basically the same idea as the ancient "send a fax with a bill for an unsolicited delivery of copier toner to 30,000 businesses" but taken into the modern era.
edit: There's already a number of LLM which are intended for outgoing data loss protection to redact or prevent PII from escaping. Is anyone specifically working on a training set and agent that is specialized in reviewing "is this legit to pay", as a sub-task or filtering step in an AP workflow? I suppose it's a GIGO problem, as it would work best only if you have suppliers enrolled in some kind of existing db, with a specific contracted format for invoices, and correlating with project numbers/cost codes.
I think it would be absolutely insane to hand over a serious-sized company's books to an LLM.
As a small consultancy though, looking forward to my next filing, and having just moved to a new and better-specced jurisdiction, I'm sorely tempted to outsource to Claude.
I've had mixed experience with accountants in the past. No horror stories, but I often feel I'm not getting everything laid out clearly, and that I don't fully understand the process.
I've got plenty of reasons to dislike LLMs in my own work, but when dealing with well-scoped but professionally gatekept things like tax or property transactions, they're an absolute godsend.
i specialize in invoice related analytics and business processes. AI can be useful for data extraction and saves me some typing time. even so, i dont blindl trust it because it sometimes makes very reasonable mistakes because invoices, quotes, and POs are sometimes structured in very informal ways. people misuse the lines, sublines, totals, and other data fields. they are often technically incorrect but when you look at them you know what they mean. i am not sure how to hand that off to something that has plausible deniability to guess even if it doesnt know
sometimes details are just in notes at the bottom and are applied to selectively applied to line items. sometimes the charge doesnt exist anywhere on the bill but there is an understanding (due to a separate agreement) of additional charges to be paid as a result of the invoice.
taxes sometimes are or are not explicitly stated
tariffs sometimes are or are not explicitly stated
when things are not explicitly stated or line item'd, they will usually still appear in the invoice total. so you have item 1 - $500, item 2 $500. total: $1300
At the end of the day invoices are often part of an ongoing communication / conversation between two organizations and they are created with an assumption that a rational and reasonable human who is in the loop with that conversation is going to read it.
That's why we have purchase orders that can only be entered by buyers. Product is received and approved by buyer. Invoice goes to accounting, who can't approve it unless there's a matching purchase order and receiver.
Yes, letting agents do whatever they want leads to disaster. But humans are gullible stochastic token generators as well. And that's why the problem is already solved.
Indeed so, a fairly mundane RFP, RFQ, buyer, receiver, accounts payable process will stop a lot of problems. If an agent is inserted at some stage in the process with a clear path to make a ticket/escalate to a human if it sees something it doesn't understand, the risk isn't absurdly high, in my opinion.
I've seen so many reports of humans with the authority/ability to execute an outgoing SWIFT transfer who've been social engineered into sending money to fraudsters... Or even just the basic low level "Hey I'm your boss sending you an SMS, please go buy some gift cards and scratch them off and send me the codes". No AI involved whatsoever.
The danger exists where some true believer AI evangelist type of management person tries to fully automate the entire purchasing and AP workflow, which I'm sure some people will attempt soon, with varying degrees of success.
I have seen the SWIFT thing happen for $100k. I think AI could actually be better for this, because it's often easier to implement hard rules for the AI.
With the SWIFT incident I saw, there was a rule that no payment can go to a vendor's bank that isn't a current, approved vendor. But the rule was not enforced in software: it was an internal accounting rule that humans were supposed to follow. The AP person "thought it had been approved" because there was a similar transaction with a different company that was a new vendor at a similar time. The other transaction was legitimate, the fraudulent spoofer wasn't. The wire got sent to a party in China.
With AI agents, if you approach it from the perspective that it will be gullible and trickable by fraudsters, you build in these hard guardrails. With humans, it's much easier to believe that "we trained Lucy on this procedure" will work in all circumstances, even if Lucy still has the technical ability to bypass the official procedure.
In these cases, it starts looking a lot more like traditional software, with your little AI chaos monkeys constrained in little boxes within the software chain.
Another thing that seems to be disturbingly common these days is some party involved in residential real estate transactions having a RAT on one or more PCs, and/or compromised email account, intercepting an email at the closing stages of a house purchase and sending someone information to send a wire transfer to a fraudulent location.
But the hard rules only work up to the point that there is an exception. See my other post above. Occasionally a very senior person (hopefully senior) has to approve a payment that is outside of the rules, because it is something the rules did not anticipate (and now they are hardcoded into software and can't be changed).
I run a team that includes people that do this kind of work using ocr software that matches invoices to po's. No AI needed. This is a solved problem. Why are there people involved? Because sometimes the invoice and po don't match. For instance, price on invoice is higher than po, refer to buyer. Buyer is sick, supplier puts you on hold, no parts for your factory, lose millions... Would you trust an AI to choose what to do next? This might get referred to me to resolve and make a decision, not just on the facts available, but on other facts I can discover, and years of experience. I might end up making an unauthorised payment, would you give an AI that power?
With AI you can scale the protection against social engineering. Where with humans you have to start from scratch each time and they are more likely to mess up.
My wife (head of accounting for a small business) has been working on automating large parts of her job using AI.
It's not completely reliable and the human cannot be taken out of the loop, but the number of menial tasks she's been able to automate has been really cool. A lot of processing data that arrives in non-standard formats, generating documents based on that data, etc.
She still has to review everything, but her workload is way down, and when her assistant quit she automated away his whole position.
I remember talking to my accountant in the UK a long time ago when I was newly self-employed, asking if I could pass something off as a business expense that was sort-of-related, but I knew probably not really OK. Her reply has stuck with me ever since: "HMRC [the UK equivalent of the IRS] are interested in matters of fact, not interpretation."
As an accountant at a large Corp, I can tell you that there are cases daily where two senior accountants argue over how to interpret accounting and tax rules, and often each agrees that the others interpretation is valid. I regularly see rulings from area specialists, and have to challenge them, only to be told, 'well in that case, you would be right'
I sometimes ask an AI to comment, and usually their answer is couched is ifs and maybes, just like the humans.
I often consult with auditors who tell me x is wrong, and they go away agreeing that y and z would be valid too, and x is also fine. There are often second order effects that need to be thought through.
This is the problem with the majority of AI push coming from devs. They look at something as complex as accounting and only think in terms of what they see on the surface and then say “oh that looks easy enough, there couldn’t possibly be more under the hood”.
Yeah, my query was an open-and-shut case, I know there are grey areas too. The Arctic Systems case in the UK was an interesting one for IT contractors.
Accountant here. It suprised me when I joined the field, but profit is calculated using a range of accounting estimates. Each accountant will make different decisions. Not least about which accounting period something belongs. Imagine a factory with freight inwards. It is month end and I have a sheaf of bills from various freight companies, but which ones are missing, not received yet? I can't wait, I have 2 days to report, so I make an estimate...now imagine that I have perhaps 100 such things. I may have to justify my estimates, but how should I estimate it? Same as last month? Perhaps I know there is a lot of shipments so I make it at the higher end.
Now I have a product failure at a major customer that I may have to send free replacements for. Should I recognise that cost now, or not? The accounting standards say if I know about it, and I can measure it and there is a high degree of certainty then I should. But the method of choosing is up to me. £10m or 1m cost, and this year or next...
and I get to decide.
We bought a £6m dollar machine to be depreciated over time, but how long. The machine lasts 10 years, but will we still be using it then? Do I capitalise the internal R&D work as well?
All of which is audited, but the auditor often only has the information I give them.
Accounting is not deterministic, within certain bounds it is very subjective.
Ha even if this was true (it’s not) you’re basically saying “humans will make some mistakes so let’s throw caution to the wind” which is probably the worst application of AI that I’ve heard yet.
If there is an acceptable margin of error for humans, we should be able to measure it for AI, and once AI is within those margins then it should be feasible to replace the human.
> And I would be pretty nervous about asking any of the frontier LLMs to retrieve invoices:
I watched an accountant YouTuber reviewing a new AI-driven personal finance app the other day (I really need to touch grass), and it started out just fine. He had seeded the account with a bunch of his data and was able to ask questions about which categories had the most spend, etc.
About half a dozen questions in, he asked it to calculate a certain segment of his spend (and being an accountant, he had his numbers memorized), and he immediately got back a calculation that he did not expect. So he asked for an itemized response and it hallucinated line items that never appeared in his account data, which he pointed out to viewers. He followed up with the chatbot with "where did line item X come from?" and the bot acknowledged that it wasn't legit. He immediately noped out after that, and who could blame him?
AI helps automate things that didn't already have rigorous formatting and structures available as input... and that's really all it does (99% of the time).
Doesn't matter how many more nines you add, rigorous formatting is still required. In some cases, it has teeth with compliance standards. Those standards cannot be compromised because there are already a lot of other layers contributing inaccuracy. It all adds up.
In most situations, you could just hire a junior dev (or an intern! remember those?) write some CSV scripts and call it a day. Cheaper and auditable too. Those scripts can't change anyway until standards are revised.
I'm still not seeing the benefit outside of solopreneur efforts and shady businesses wanting to launder blame.
Manual data entry and other tedious chores are definitely unreliable. However, running a script that a human wrote according to committee specs is the most reliable part. You're conflating the different aspects of human work. We are much better at understanding our needs and arguing about them than doing the manual part.
So, I don't get your argument either. I hear yours often enough and so much louder that I feel it's a deliberate muddying of waters.
What cannot be obsoleted by automation becomes bureaucracy. To my ears, it sounds like you're afraid of ending the tech wild west. That bureaucracy was always the most valuable part, and the demand for experienced programmers over at that table is very high.
The benchmark data was prepared in April 2026 (when I was manually doing our VAT return with my co-founder). The invoices were indeed found manually.
Currently we're using a custom "invoice searcher" built on Kimi 2.6 (in our testing several weeks ago it outperformed Opus 4.7; it was just more persistent).
Ultimately, I still verify everything manually after the model is finished fetching invoices for the month -- but it's a great help to have all the invoices already found (usually correctly).
adam, i'd like to get in touch and would love to run the benachmark with mixedbread as a search backend. we are doing this right now with a lot of compliance companies. would be very curious how it improves quality/cost e2e
This is a prime example of a problem space where accuracy matters, but it also matters who ultimately goes to prison. I'm going to go out on a limb and guess it's not the LLM.
If you're acting in good faith and your accountant does something crazy or evil, your liability is limited to some extent. You may get a tax bill but you're probably not gonna end up behind bars. But if your LLM decides to do a little bit of tax fraud, you're in uncharted waters. In the end, the gun did it, but you were the one holding the gun.
A lot of jobs are like that. You're not as much buying the service as you're buying not having to worry about the service.
> If you're acting in good faith and your accountant does something crazy or evil, your liability is limited to some extent.
From my understanding, you are the person signing off on the paperwork that is submitted to the IRS.
There is this cache 22 with taxes.
You are responsible, but you outsource it to a accountant. Because you are not knowledgeable about the taxes. But you are expected to be knowledgeable to understand the tax documents that you submit to the IRS. That is why the accountant always ask you to review the documents and sign them like 20 times.
The same applies when you run a company, depending on the country, you need to prove yourself knowledgeable in accounting, before you are allowed to run a company. Normally that is included in a university degree, but if you have a middle school diploma, you need to do a official examen to get that degree.
Whatever you submit for your company, you are again responsible. Even if you hired a accountant.
So while technically, if a accountant makes gross mistakes, the bill will always fall in your lap, because you are expected to understand the reports you submit to the IRS. And catch any errors before doing so.
With the IRS, the burden of proving your innocents is often put you. Its because the good faith argument can be misused easily. That is why the buck stops at you.
So using a LLM or a accountant, really does not matter. Sure, a accountant can go to jail if there has been major issues (its not going to be with one client issue).
But you can lose your house / company, have your life ruined by whatever you submitted.
I don't think that's actually the case. If you get an tax attorney to document your treatment and indicate that they believe it's legal (e.g. a tax opinion letter), then you're off the hook for penalties and criminal liability (assuming the tax attorney did everything properly). You still have to pay the difference though.
An accountant may not save you from financial penalties but I believe that they are liable for them so you can recover it from their insurance.
This is also something where how you interact with the IRS really really depends on your initial interactions.
If the IRS comes to you and says "your taxes are wrong you owe X amount" and you immediately turn around and prepare to fight them then they are going to be merciless with you.
But if they come to you and you turn around and show the auditor your methodology and what you thought you were supposed to be doing, etc and focus on fixing the issue and settling your debt then they are going to hook you up with IRS staff dedicated to helping fix this kind of stuff.
And most importantly they'll connect you with someone who can set up a payment plan out over whatever period of time to pay back what you owe without killing your business or taking your assets.
But again if you get audited and immediately become adversarial then they'll assume you were doing it intentionally and will take anything not bolted down.
-----
The rationale is basically that the IRS staff are mostly there to help you get your taxes in order and ensure that those taxes are paid but as soon as it starts to look like you are intentionally avoiding paying your taxes you get kicked over to the enforcement divisions and eventually the criminal divisions.
So good faith is absolutely something the IRS will honor but that stops the moment it looks like you are trying to hide stuff from them.
> Whatever you submit for your company, you are again responsible. Even if you hired a accountant.
> So while technically, if a accountant makes gross mistakes, the bill will always fall in your lap, because you are expected to understand the reports you submit to the IRS. And catch any errors before doing so.
I've had a tiny accounting firm take on more and more accounting debt over years just to get the accounts (lots of journal entries into contra accounts; this was before I took over the company), that I was mostly rolling my eyes at how horrible they've become when I tried enlisting Claude's help to untangle them and hopefully move away from using that firm.
I know bookkeeping is tough, especially for newer businesses that haven't set up the processes, but a lot of the professional services industry feels like outsourcing a website to a freelancer on Upwork.
I agree with you on every point but it is interesting to see real world benchmarks like this. Showing the standard benchmarks that all LLMs use is not only boring but at this point likely gamed or even has issues (according to OpenAI) by every LLM.
I've been testing various big and small models for years and, about a year ago, switched my attitude from "biggest model is best!" to "best depends on task".
For example, I had a simple coding task which required making 3 trivial changes in 3 source files. Biggest Model completed task perfectly and took 90 seconds.
Smaller Brother also completed task perfectly, but took 30 seconds and cost 5x less.
1 Just as with traditional non-ai ways of doing things, there is a deterministic layer that can trigger when things are off. This can range from letting the ai know to stopping before a human confirms.
2 Usually, specially for SMBs, nobody goes to prison over accounting errors. That's because SMB owners make mistakes all the time and, because of that, authorities are fairly practiced in understanding how mistakes look, and how fraud looks.
Sure! In classic accounting firms you'd typically have one accountant doing the books and then the other reviewing the first accountant's work.
Given the model's accuracy (and models will only get better), only by automating the "doing" and keeping a human to do the review part we will already shave huge amount of time and costs without risking the compliance part.
It is given a list of transactions as an input. If it misses one or more of them, that will be noticeable immediately. In the VAT return, one can see if something is missing very easily if the numbers do not align.
> This is a prime example of a problem space where accuracy matters, but it also matters who ultimately goes to prison.
I don't disagree that tax fraud is bad but accuracy really doesn't matter 1/10th what people think it does.
Something to keep in mind in the EU with VAT is that there are approximations, errors and then huge frauds like the infamous "carousel fraud". Sadly the carousel fraud is all too common in the EU: this one gets people really sent to jail. But it's not an error a LLM shall make that could lead to a carousel fraud taking place: it's an elaborate scheme.
Now... If you're self-employed and have to pay the VAT (which may or may not be the case depending of the type of self-employment work you do) or if you're a SME and you've got approximations and errors well it's really not a big deal.
Because the public servants in charge of collecting the VAT and verifying that you filled your stuff correctly... Do make shitload of errors too. And then, depending on the country, these public servants have a special privilege: they can decide an amount of VAT+fine (if there's a fine) that you agree with and if you agree with it, their number is the number.
It's everything but a correct number. It's a number they ended up on that is "close enough" (for some definition of "close"). Sometimes by hammering number after number on a little calculator and then ending up saying:
"We ask you to pay 3500 EUR of additional VAT. If you sign here and accept, the IRS (equivalent of the IRS) won't be able to bother you ever again for that year/years".
This notion that, somehow, there's "one" correct number and that public servants paid something like 2 K EUR net per month (which may be the reason why so many are so prompt to ask for bribes btw) can determine it in a few minutes is ridiculous.
It's simply not happening. Usually the accountant doesn't know the exact number. The SME owner doesn't know the real amount of VAT due. And the public servant(s) in charge of a VAT audit, if any takes place, certainly as zero clue as to what the exact VAT number should be.
The rules are way too complicated, there are way too many special cases, and some things are simply extremely hard to take into account (for example during one audit the person told me he disagree with the 90% of 21% VAT deducted for my ISDN line --yup it was a long time ago-- but then I told him it was during a previous TVA audit, a few years before, that the VAT auditor, seen my job, fixed at 90%).
For some stuff you can opt for a fixed number: say a car, there are rules, depending on the country, where you can, say, get back 75% of 50% of the VAT. But people can also, depending on the country, opt-in for the "real usage": you do 12 387 km during the years, how many were really for work? 6 287 km ? Or may 6 275 km?
What if one day while going to work, you took a detour because you want to drop a gift for your grandma's birthday? How were these kilometers accounted for?
Restaurant bills? What if one of the person wasn't work related (your nephew happened to be in town and you had to take car of him and you knew the clients well so they didn't mind: how does the accountant account for that)?
Gifts to client. You got a discount buying 6 packs of bottle of champagne, but not five (no discount if only five), so you bought a sixth one. But that sixth one you kept for you.
Nobody computes those amounts correctly. Nobody.
And don't get me started on the public servants who'll give, during an audit, an unfair advantage to people from their community (whatever that community may be) or those, literally, banging underneath the table while talking about their next vacations: they're telling you an "arrangement" can be made. Basically you give that one person some money (cash bills of course) and, boom, the 20 K EUR of VAT you had to pay suddenly gets lowered down to 5 K EUR.
You refuse? It'd be too bad if those 20 K became 25 K EUR wouldn't it? Mafia-style.
This is all happening. Don't tell me it's not happening because I know people on both sides of these "trades".
Depending on the EU country, it can even be the norm.
As rational people who like determinism (at least I do), at first glance it looks like a SME's accounting is something square but it really ain't. Or let's put it this way: it there's one number that is the really the amount of VAT the SME has to legally pay, I'd guesstimate 0.01% of the SMEs out there pay, to the cent, the proper amount.
All this to say: the approximations, hallucinations and errors of LLMs in accounting for VAT are probably just on par for the course.
P.S: ah yup... The boss (say major airline company) giving a business card and its PIN code and asks to take good care of the clients: and an employee takes them to strip clubs, they all get drunk like crazy, and explodes the expenses on the business card. Way, way, way more than what is OK to put into expenses to entertain clients. Well guess what: I know for a fact this happens too. And I know for a fact this ain't accounted for properly either.
It’s almost impossible to be in a situation where your taxes end up this wrong you’re accused of fraud. I honestly do not believe my accountant does anything better than AI or AI generated code that does calculations to see what you should be paying.
It's not just about being accidentally wrong. An LLM can decide to actively engage in fraud. One mechanism, which you also see in vibe coding, is that the models are very goal-driven. If they can't accomplish the goal for some reason, they will sometimes decide to cheat by, for example, faking tests. I can easily imagine an LLM that stumbles across some benign issue that prevents the numbers from balancing out and then decides to cover it up to complete the task.
Another problem is that "evil" alignment is apparently a pretty big and nebulous cluster that's pretty easy to activate. Ask a model to write a snippet of insecure code, for example, and they may also all of sudden decide they hate minorities. So if you ask it to bend the rules in a small way once, you might end up with an evil accountant before you know it.
And that's before we get to an invoice for a service named "disregard previous instructions and...".
I'd be scared shitless to even try something like this. There is just a pretty website, a video, and a blog post. No info on the founders, I can't find anything on LinkedIn, just a company Vineyard Finance LTD that was incorporated last year.
We're all unhinged about the data we're giving LLMs but here I'd draw the line. I'd rather keep paying the small amount I pay to have my accounts done.
Interesting write-up. Having been a bookkeeper a long time ago, I'm not too surprised at this being susceptible to automation by an LLM backed system.
It seems also that the classes of error they encountered could be handled by improved skills/knowledge base access on the fine points of relevant tax legislation.
The important part for their software ofc is, will they take responsibility for the output if HMRC come calling? Without that users are adopting the risk which they may not be keen to do (dealing with HMRC is not fun), with that it could be a very nice saving for a lot of small companies (and bad for the employees of a lot of accountancy firms)
Unrelated, but I feel it's unfair to rob the word "bookkeeper" of its peculiarity of having three subsequent double letters by inserting a space in the middle.
I was one of the human book-keepers for this benchmark (the preparer; my co-founder verified the VAT submission once ready), and given that at the time of doing this I knew I was eventually going to use this data for evaluating the models, I was super careful. So I guess this is a "good book-keeper". In the previous company our book-keepers made lots of mistakes; some serious enough that we had to restate our company's accounts.
Then why only have one human bookkeeper? Surely two would be better, since you can compare their results. But then, perhaps you should hire three, so you can figure out which one is right.
I wouldn't be surprised if it were more accurate based on the errors I've seen. I always eyeball the books and was confused when a £15k building popped up on our asset sheet. It turns out a "workshop" had been categorised as a building we had purchased, rather than the training session it actually was.
This is the importance of having layers and multiple sets of eyes on things, though. Even if it had got past me, my accountant would have surely queried it at year end, but that could be true of an LLM mistake too.
I just have a folder on my computer where I keep things in beancount. Then I have mercury CLI access with a read token to my business bank account, and I have my emails fully synced in there as well via IMAP. Claude Code with Opus just seamlessly hooks everything up so my accounts are up to date. At the end of the year, I used that information to prepare my tax returns for the business and then later the part that flowed to me as the owner.
I had a fairly complicated tax return in 2025 involving a couple of change of business tax consideration and some money that was accidentally sent to me as a 1099 instead of to the business and I did everything with tax software with Claude Code advising.
The end result was pretty damned good. I was unsurprisingly audited (or at least carefully reviewed) and the only error was in some way where I allocated a small amount of my wife's tax-free disability payments (disability is the mechanism that California uses to provide maternal benefits pay protection; she's not actually disabled). The IRS told me about it, I paid that bit (it was meant to be claimed back from the employer, not the US government) and everything was hunky dory. To be honest, the sum was so small I did not investigate (and haven't yet followed up with getting reimbursed by her employer).
Honestly, almost all of it could have been avoided if I'd paid an accountant and a tax lawyer and they'd told me things and I'd done as they did, but in the end the combination of the fact that the IRS is very reasonable when you explain things and a modern agent means that the entire process was quite simple. In the end, I preferred the interactive mechanism of working with software because most accountants and lawyers will prefer to get all of your documentation all at once and then work on it rather than do it incrementally. In my case, I was able to work on the return incrementally and then have everything plugged in. I could ask a bunch of questions and get clarification.
I think I will probably do all this the same way this year (though of course my taxes will be simpler).
I don't know how much a tax lawyer/ account would quote for this in California, but I saved 3000 EUR/ tax year on a German Steuerberater in a similar way.
As in, "what makes you think you were audited?". They sent me paperwork saying they were reviewing my return and that they found a discrepancy (the disability thing) and it took many months after the usual time for it to process.
If you mean "for what reason could you have been audited?" it was because a client I had previously worked for accidentally reported paying me as an individual instead of my LLC.
Parses emails or other sources, extracts numbers, correlates different transactions, web search, asks questions, stores notes (regex based, very simple).
The hard part is getting good data, I'm sure that lexus nexus or whoever can get API access to my bank account and all my credit cards, but I can't. Email turned out to be the best way for most of my providers. Managed to avoid 2factor auth so far, but it will suck when I need it.
We've got integrations with major UK banks. Curious if you'd like to use a polished product or be more interested in bank-feed-as-an-API type of use case? What banks do you use?
Credit union atlantic. The possibility of you being able to profitably support small regional credit unions is pretty much 0. Maybe with a general browser use AI and an SMS portal. Also you'd need to bypass anti-bot protections, maybe solve captchas.
In my opinion that is the thing holding back almost all of these accounting products that make consumers lives better. I've solved it because all my cards and bills happen to support email, so I can use that as my source of truth for all my credit cards.
The token is stored on a LUKS encrypted drive on kwallet. You'd have just as much luck getting my password from my firefox installs sqlite DB. This is also how email clients in general work.
Also this isn't really an agent. At least not a long lived one. Each email or transaction gets it's own session, the llm can make a few tool calls but must emit json as the final result. Very very short context lengths, very predictable results.
The bot does not have access to my passwords, there are pre-defined scripts that fetch the data that the run an LLM call for each transaction discovered.
As someone who practised tax law for global companies for a few years, I can say that the correct classification of some invoices from the tax law perspective requires a full-blown legal analysis.
That involves full interpretation of the law, which includes "teleology" - that is, understanding the purpose and goals of the law. We are probably (?) not yet at the point where frontier models can do this better than humans.
In other words, it's the same question as when AI will replace judges. Not lawyers, but judges themselves.
That "nearly" is doing an awful lot of heavy lifting. It doesn't matter if your AI model is 99% or 99.99% accurate. For a tax return it has to be perfect every time or someone is at best getting a fine or at worst going to prison.
Sure, human error happens too, but humans take accountability. That's why accountants are a regulated profession. Until an AI company CEO is willing to go prison if the output of their model is wrong, these tools are worthless.
Toot uses automated and AI systems to generate classifications and reconciliation suggestions. Output may be incomplete or wrong and must be reviewed by you.
Toot is a software tool. It does not provide accounting, tax, legal, audit, or financial advice, and nothing it produces is a substitute for a qualified accountant or tax adviser. You are responsible for checking Output before approving it or relying on it, and for any decision you make based on it. To the extent permitted by law, we are not responsible for outcomes arising from automated Output you approve without review.
Comparing this to a human book keeper is farcical.
That "nearly" is doing an awful lot of heavy lifting. It doesn't matter if your AI model is 99% or 99.99% accurate. For a tax return it has to be perfect every time or someone is at best getting a fine or at worst going to prison.
This is wildly out of touch with reality, at least in the US. The IRS barely audits anyone anymore, and if you make a mistake and they catch it, they often just correct it and send you a letter. You might pay a little in penalties, but it’s not that big of a deal. And I know literally dozens of small business owners who pretty blatantly dodge taxes in a hundred different ways and none of them have ever gotten so much as a slap on the wrist. And some of them have been audited!
This literally has nothing to do with AI? I’m pointing out that taxes aren’t some magical area where you must be exact to the penny or hellfire will rain down on you. Hundreds of millions of Americans file their taxes and many have mistakes major and minor, intentional and not. The world somehow keeps turning.
"The VAT return prepared by the model was essentially correct: the most important number in the return, which is how much VAT the company was owed by the tax agency, was off by only 7 pence relative to the human-prepared return."
I don't know how taxes work in Europe, but in the US being "essentially" correct is not good enough for the IRS.
The paragraph after this one goes on to explain other mistakes the LLM made? Yikes
Ask most people in the US how they file their taxes, most will probably tell you they clicked around turbotax until the biggest return number showed up. Is that indicating correct tax filing? I'm not sure. I would guess not.
The company I work for, Digits, has been regularly updating our AI-vs-human bookkeeper benchmark. Look at page 8 -- many models are nearly as accurate as a human bookkeeper
Thanks, that's useful! We did use ChatGPT 5.5 for some time and it did perform pretty well too (of course more expensive than GLM 5.2).
We tried Claude 4.7 and 4.8, but we found both models to be "lazy" and very expensive. Claude would always rather prefer the route of saying that the evidence was not found or something is incomplete, rather than put more effort into finding/repairing the particular issue.
This is interesting to see, but surely any non-trivial business [edit: who needs to file a VAT return] is already entering its invoices into a finance system which can automatically generate a VAT return in a deterministic way.
The real test to see if AI is just a rich person thing will be to see how the tax authorities treat it, even for more complex returns.
They can save humans for the really complex edge case stuff but at the end of the day, the tax code is just checkboxes and input forms that get boiled down into Integers, Floats/Doubles and enumerated choices with some Strings for deductions
I've submitted my German taxes this year using a mix of Claude 4.6 and Claude 4.7, with lots of manual checking. The German Finanzamt granted most of the things I listed in the tax return (they send you an official letter by post) -- I did have to appeal for one of the items though (again using Claude, this time 4.8 ).
The most important thing I've found is to ask Claude to thoroughly audit the reply (to find all hallucinations). I usually ask it to give me an enumerated list of all facts and all legal cases quoted, and then I give it to a new instance to carefully validate each one.
Newer models are getting much better at not hallucinating German case law though :)
Bookkeeping is not tax preparation, just FYI. The reports generated by an accounting system are used on tax returns for companies, but they’re distinctly different things.
Oh, I'm actively doing this at the moment. FreeAgent grabs my transactions from Wise already, and then I give it [Claude Code, in fact] a folder of PDFs to attach to my invoices, including figuring out VAT, and it's uploading what it found using the FreeAgent API. My accountant hasn't complained yet, and it seems considerably more accurate than when my wife was doing it.
They're not going to unless it's obviously and egregiously wrong - the risk on quality of input remains yours. It's the tax version of garbage in, garbage out. They're just guaranteeing the processing step.
The problem with LLM's is that they could work correctly for months and years and then do something egregious which will will go unnoticed because of the misplaced trust one develops on a system that "just seems to work." Get flagged for an expensive audit and there go all the savings and then some.
Humans are more likely to make small mistakes but the internal consistency check is pretty good at catching large errors. On top of that, fudging numbers to make everything add up is not something humans do (not unintentionally at least)
I was wondering what a book keeper is, thinking it would be some kind of librarian. Isn't the accounting sense written as a single word, i.e. bookkeeper?
thats not true. maybe for simple 1040s yes, but depending on what you have you could take aggressive positions bc they could be subject to valuation differences.
I'm not so interested in having an LLM do my bookkeeping for me. But I'm very interested in whether LLM's can unravel the accounting obfuscations that billionaires use to avoid paying taxes on their wealth.
One idea I've had for reform is to ban central banks from inventing new currency except to pay every citizen a UBI debt free, and only tax limited liability entities (not humans). This would give citizens, instead of borrowers, first spend on new money and orient limited liability entities to serving citizens in the market to get their taxed revenues. It would also be a boon for financial privacy and reduce the burden on the average man.
This shouldn't be ignored in the discussion here:
This is precisely the kind of fine print on white-collar AI capability that companies keep running into: pretty much any non-entry office job worth having involves a lot of undocumented (even undocumentable) problems requiring judgment and experience.And I would be pretty nervous about asking any of the frontier LLMs to retrieve invoices: "cool, Claude logged that it found the May 6th bill from the paper supplier, I am sure it didn't just make something up arbitrary, then compound on the error by agentically iterating over the made-up invoice lurking in its reasoning traces. I checked the first 30 times and there were no problems!"
If and when a large number of companies blindly turn over their accounts payable workflow to some AI agent system, it'll be very interesting to see the "social engineer the LLM" methods that fraud people use to get money sent to them. Basically the same idea as the ancient "send a fax with a bill for an unsolicited delivery of copier toner to 30,000 businesses" but taken into the modern era.
edit: There's already a number of LLM which are intended for outgoing data loss protection to redact or prevent PII from escaping. Is anyone specifically working on a training set and agent that is specialized in reviewing "is this legit to pay", as a sub-task or filtering step in an AP workflow? I suppose it's a GIGO problem, as it would work best only if you have suppliers enrolled in some kind of existing db, with a specific contracted format for invoices, and correlating with project numbers/cost codes.
I think it would be absolutely insane to hand over a serious-sized company's books to an LLM.
As a small consultancy though, looking forward to my next filing, and having just moved to a new and better-specced jurisdiction, I'm sorely tempted to outsource to Claude.
I've had mixed experience with accountants in the past. No horror stories, but I often feel I'm not getting everything laid out clearly, and that I don't fully understand the process.
I've got plenty of reasons to dislike LLMs in my own work, but when dealing with well-scoped but professionally gatekept things like tax or property transactions, they're an absolute godsend.
i specialize in invoice related analytics and business processes. AI can be useful for data extraction and saves me some typing time. even so, i dont blindl trust it because it sometimes makes very reasonable mistakes because invoices, quotes, and POs are sometimes structured in very informal ways. people misuse the lines, sublines, totals, and other data fields. they are often technically incorrect but when you look at them you know what they mean. i am not sure how to hand that off to something that has plausible deniability to guess even if it doesnt know
sometimes details are just in notes at the bottom and are applied to selectively applied to line items. sometimes the charge doesnt exist anywhere on the bill but there is an understanding (due to a separate agreement) of additional charges to be paid as a result of the invoice.
taxes sometimes are or are not explicitly stated
tariffs sometimes are or are not explicitly stated
when things are not explicitly stated or line item'd, they will usually still appear in the invoice total. so you have item 1 - $500, item 2 $500. total: $1300
At the end of the day invoices are often part of an ongoing communication / conversation between two organizations and they are created with an assumption that a rational and reasonable human who is in the loop with that conversation is going to read it.
You can fix this simply by using normal controls.
That's why we have purchase orders that can only be entered by buyers. Product is received and approved by buyer. Invoice goes to accounting, who can't approve it unless there's a matching purchase order and receiver.
Yes, letting agents do whatever they want leads to disaster. But humans are gullible stochastic token generators as well. And that's why the problem is already solved.
Indeed so, a fairly mundane RFP, RFQ, buyer, receiver, accounts payable process will stop a lot of problems. If an agent is inserted at some stage in the process with a clear path to make a ticket/escalate to a human if it sees something it doesn't understand, the risk isn't absurdly high, in my opinion.
I've seen so many reports of humans with the authority/ability to execute an outgoing SWIFT transfer who've been social engineered into sending money to fraudsters... Or even just the basic low level "Hey I'm your boss sending you an SMS, please go buy some gift cards and scratch them off and send me the codes". No AI involved whatsoever.
The danger exists where some true believer AI evangelist type of management person tries to fully automate the entire purchasing and AP workflow, which I'm sure some people will attempt soon, with varying degrees of success.
I have seen the SWIFT thing happen for $100k. I think AI could actually be better for this, because it's often easier to implement hard rules for the AI.
With the SWIFT incident I saw, there was a rule that no payment can go to a vendor's bank that isn't a current, approved vendor. But the rule was not enforced in software: it was an internal accounting rule that humans were supposed to follow. The AP person "thought it had been approved" because there was a similar transaction with a different company that was a new vendor at a similar time. The other transaction was legitimate, the fraudulent spoofer wasn't. The wire got sent to a party in China.
With AI agents, if you approach it from the perspective that it will be gullible and trickable by fraudsters, you build in these hard guardrails. With humans, it's much easier to believe that "we trained Lucy on this procedure" will work in all circumstances, even if Lucy still has the technical ability to bypass the official procedure.
In these cases, it starts looking a lot more like traditional software, with your little AI chaos monkeys constrained in little boxes within the software chain.
Another thing that seems to be disturbingly common these days is some party involved in residential real estate transactions having a RAT on one or more PCs, and/or compromised email account, intercepting an email at the closing stages of a house purchase and sending someone information to send a wire transfer to a fraudulent location.
But the hard rules only work up to the point that there is an exception. See my other post above. Occasionally a very senior person (hopefully senior) has to approve a payment that is outside of the rules, because it is something the rules did not anticipate (and now they are hardcoded into software and can't be changed).
I run a team that includes people that do this kind of work using ocr software that matches invoices to po's. No AI needed. This is a solved problem. Why are there people involved? Because sometimes the invoice and po don't match. For instance, price on invoice is higher than po, refer to buyer. Buyer is sick, supplier puts you on hold, no parts for your factory, lose millions... Would you trust an AI to choose what to do next? This might get referred to me to resolve and make a decision, not just on the facts available, but on other facts I can discover, and years of experience. I might end up making an unauthorised payment, would you give an AI that power?
Sounds like Enron was just a bit early. They could have just blamed the AI system instead of becoming a meme with "Enron math"
With AI you can scale the protection against social engineering. Where with humans you have to start from scratch each time and they are more likely to mess up.
My wife (head of accounting for a small business) has been working on automating large parts of her job using AI.
It's not completely reliable and the human cannot be taken out of the loop, but the number of menial tasks she's been able to automate has been really cool. A lot of processing data that arrives in non-standard formats, generating documents based on that data, etc.
She still has to review everything, but her workload is way down, and when her assistant quit she automated away his whole position.
Hahaha non-deterministic accounting probably won’t fly well with the IRS
That made me laugh, thanks.
I remember talking to my accountant in the UK a long time ago when I was newly self-employed, asking if I could pass something off as a business expense that was sort-of-related, but I knew probably not really OK. Her reply has stuck with me ever since: "HMRC [the UK equivalent of the IRS] are interested in matters of fact, not interpretation."
As an accountant at a large Corp, I can tell you that there are cases daily where two senior accountants argue over how to interpret accounting and tax rules, and often each agrees that the others interpretation is valid. I regularly see rulings from area specialists, and have to challenge them, only to be told, 'well in that case, you would be right'
I sometimes ask an AI to comment, and usually their answer is couched is ifs and maybes, just like the humans.
I often consult with auditors who tell me x is wrong, and they go away agreeing that y and z would be valid too, and x is also fine. There are often second order effects that need to be thought through.
Good luck AI
This is the problem with the majority of AI push coming from devs. They look at something as complex as accounting and only think in terms of what they see on the surface and then say “oh that looks easy enough, there couldn’t possibly be more under the hood”.
Yeah, my query was an open-and-shut case, I know there are grey areas too. The Arctic Systems case in the UK was an interesting one for IT contractors.
Do you think human accountants are deterministic?
Get a large enough org and watch your accounting grow an error margin.
Accountant here. It suprised me when I joined the field, but profit is calculated using a range of accounting estimates. Each accountant will make different decisions. Not least about which accounting period something belongs. Imagine a factory with freight inwards. It is month end and I have a sheaf of bills from various freight companies, but which ones are missing, not received yet? I can't wait, I have 2 days to report, so I make an estimate...now imagine that I have perhaps 100 such things. I may have to justify my estimates, but how should I estimate it? Same as last month? Perhaps I know there is a lot of shipments so I make it at the higher end.
Now I have a product failure at a major customer that I may have to send free replacements for. Should I recognise that cost now, or not? The accounting standards say if I know about it, and I can measure it and there is a high degree of certainty then I should. But the method of choosing is up to me. £10m or 1m cost, and this year or next... and I get to decide.
We bought a £6m dollar machine to be depreciated over time, but how long. The machine lasts 10 years, but will we still be using it then? Do I capitalise the internal R&D work as well?
All of which is audited, but the auditor often only has the information I give them.
Accounting is not deterministic, within certain bounds it is very subjective.
Ha even if this was true (it’s not) you’re basically saying “humans will make some mistakes so let’s throw caution to the wind” which is probably the worst application of AI that I’ve heard yet.
>Ha even if this was true (it’s not)
You haven't met many accountants i see.
Regardless that's not what he is saying.
If there is an acceptable margin of error for humans, we should be able to measure it for AI, and once AI is within those margins then it should be feasible to replace the human.
Ah yes, because there’s no difference between a shifted decimal and entirely hallucinated credits and debits.
This was my CPA wife’s response btw.
The entire field is based on balance sheets and context that informs compliance.
Let me put it this way: it’s such a bad idea that even Intuit doesn’t let their AI replace a human.
Why would Intuit be the barometer here? They're old technology. Pilot.com's got AI bookkeepers and they love them.
Man I hope so because they’re the ones selling them haha.
The "beatings" delivered by IRS will be deterministic for sure!
The tax code is so complex that business taxes are already nondeterministic.
> And I would be pretty nervous about asking any of the frontier LLMs to retrieve invoices:
I watched an accountant YouTuber reviewing a new AI-driven personal finance app the other day (I really need to touch grass), and it started out just fine. He had seeded the account with a bunch of his data and was able to ask questions about which categories had the most spend, etc.
About half a dozen questions in, he asked it to calculate a certain segment of his spend (and being an accountant, he had his numbers memorized), and he immediately got back a calculation that he did not expect. So he asked for an itemized response and it hallucinated line items that never appeared in his account data, which he pointed out to viewers. He followed up with the chatbot with "where did line item X come from?" and the bot acknowledged that it wasn't legit. He immediately noped out after that, and who could blame him?
AI helps automate things that didn't already have rigorous formatting and structures available as input... and that's really all it does (99% of the time).
Doesn't matter how many more nines you add, rigorous formatting is still required. In some cases, it has teeth with compliance standards. Those standards cannot be compromised because there are already a lot of other layers contributing inaccuracy. It all adds up.
In most situations, you could just hire a junior dev (or an intern! remember those?) write some CSV scripts and call it a day. Cheaper and auditable too. Those scripts can't change anyway until standards are revised.
I'm still not seeing the benefit outside of solopreneur efforts and shady businesses wanting to launder blame.
> doesn't matter how many more nines you add
I don't get this argument. People say the same about autonomous driving.
But humans also have some number of nines. If you can get it better than humans, that's better!
Manual data entry and other tedious chores are definitely unreliable. However, running a script that a human wrote according to committee specs is the most reliable part. You're conflating the different aspects of human work. We are much better at understanding our needs and arguing about them than doing the manual part.
So, I don't get your argument either. I hear yours often enough and so much louder that I feel it's a deliberate muddying of waters.
What cannot be obsoleted by automation becomes bureaucracy. To my ears, it sounds like you're afraid of ending the tech wild west. That bureaucracy was always the most valuable part, and the demand for experienced programmers over at that table is very high.
Hey, the author of the benchmark here.
The benchmark data was prepared in April 2026 (when I was manually doing our VAT return with my co-founder). The invoices were indeed found manually.
Currently we're using a custom "invoice searcher" built on Kimi 2.6 (in our testing several weeks ago it outperformed Opus 4.7; it was just more persistent).
Ultimately, I still verify everything manually after the model is finished fetching invoices for the month -- but it's a great help to have all the invoices already found (usually correctly).
adam, i'd like to get in touch and would love to run the benachmark with mixedbread as a search backend. we are doing this right now with a lot of compliance companies. would be very curious how it improves quality/cost e2e
Sure, I'm at adam@vineyard-finance.com
AI is a glorified calculator in this case. It frees up human to do other things. It shouldn't be viewed as human replacer.
Even when humans used to do the calculating?
[flagged]
This is a prime example of a problem space where accuracy matters, but it also matters who ultimately goes to prison. I'm going to go out on a limb and guess it's not the LLM.
If you're acting in good faith and your accountant does something crazy or evil, your liability is limited to some extent. You may get a tax bill but you're probably not gonna end up behind bars. But if your LLM decides to do a little bit of tax fraud, you're in uncharted waters. In the end, the gun did it, but you were the one holding the gun.
A lot of jobs are like that. You're not as much buying the service as you're buying not having to worry about the service.
> If you're acting in good faith and your accountant does something crazy or evil, your liability is limited to some extent.
From my understanding, you are the person signing off on the paperwork that is submitted to the IRS.
There is this cache 22 with taxes.
You are responsible, but you outsource it to a accountant. Because you are not knowledgeable about the taxes. But you are expected to be knowledgeable to understand the tax documents that you submit to the IRS. That is why the accountant always ask you to review the documents and sign them like 20 times.
The same applies when you run a company, depending on the country, you need to prove yourself knowledgeable in accounting, before you are allowed to run a company. Normally that is included in a university degree, but if you have a middle school diploma, you need to do a official examen to get that degree.
Whatever you submit for your company, you are again responsible. Even if you hired a accountant.
So while technically, if a accountant makes gross mistakes, the bill will always fall in your lap, because you are expected to understand the reports you submit to the IRS. And catch any errors before doing so.
With the IRS, the burden of proving your innocents is often put you. Its because the good faith argument can be misused easily. That is why the buck stops at you.
So using a LLM or a accountant, really does not matter. Sure, a accountant can go to jail if there has been major issues (its not going to be with one client issue).
But you can lose your house / company, have your life ruined by whatever you submitted.
I don't think that's actually the case. If you get an tax attorney to document your treatment and indicate that they believe it's legal (e.g. a tax opinion letter), then you're off the hook for penalties and criminal liability (assuming the tax attorney did everything properly). You still have to pay the difference though.
An accountant may not save you from financial penalties but I believe that they are liable for them so you can recover it from their insurance.
This is also something where how you interact with the IRS really really depends on your initial interactions.
If the IRS comes to you and says "your taxes are wrong you owe X amount" and you immediately turn around and prepare to fight them then they are going to be merciless with you.
But if they come to you and you turn around and show the auditor your methodology and what you thought you were supposed to be doing, etc and focus on fixing the issue and settling your debt then they are going to hook you up with IRS staff dedicated to helping fix this kind of stuff.
And most importantly they'll connect you with someone who can set up a payment plan out over whatever period of time to pay back what you owe without killing your business or taking your assets.
But again if you get audited and immediately become adversarial then they'll assume you were doing it intentionally and will take anything not bolted down.
-----
The rationale is basically that the IRS staff are mostly there to help you get your taxes in order and ensure that those taxes are paid but as soon as it starts to look like you are intentionally avoiding paying your taxes you get kicked over to the enforcement divisions and eventually the criminal divisions.
So good faith is absolutely something the IRS will honor but that stops the moment it looks like you are trying to hide stuff from them.
> Whatever you submit for your company, you are again responsible. Even if you hired a accountant.
> So while technically, if a accountant makes gross mistakes, the bill will always fall in your lap, because you are expected to understand the reports you submit to the IRS. And catch any errors before doing so.
I've had a tiny accounting firm take on more and more accounting debt over years just to get the accounts (lots of journal entries into contra accounts; this was before I took over the company), that I was mostly rolling my eyes at how horrible they've become when I tried enlisting Claude's help to untangle them and hopefully move away from using that firm.
I know bookkeeping is tough, especially for newer businesses that haven't set up the processes, but a lot of the professional services industry feels like outsourcing a website to a freelancer on Upwork.
“A computer can never be held accountable, therefore a computer must never make a management decision.”
– IBM Training Manual, 1979
https://simonwillison.net/2025/Feb/3/a-computer-can-never-be...
I agree with you on every point but it is interesting to see real world benchmarks like this. Showing the standard benchmarks that all LLMs use is not only boring but at this point likely gamed or even has issues (according to OpenAI) by every LLM.
>has issues (according to OpenAI)
What is this referring to?
I've been testing various big and small models for years and, about a year ago, switched my attitude from "biggest model is best!" to "best depends on task".
For example, I had a simple coding task which required making 3 trivial changes in 3 source files. Biggest Model completed task perfectly and took 90 seconds.
Smaller Brother also completed task perfectly, but took 30 seconds and cost 5x less.
1 Just as with traditional non-ai ways of doing things, there is a deterministic layer that can trigger when things are off. This can range from letting the ai know to stopping before a human confirms.
2 Usually, specially for SMBs, nobody goes to prison over accounting errors. That's because SMB owners make mistakes all the time and, because of that, authorities are fairly practiced in understanding how mistakes look, and how fraud looks.
Sure! In classic accounting firms you'd typically have one accountant doing the books and then the other reviewing the first accountant's work. Given the model's accuracy (and models will only get better), only by automating the "doing" and keeping a human to do the review part we will already shave huge amount of time and costs without risking the compliance part.
What if the model misses something entirely, so it's not in the review? ;)
It is given a list of transactions as an input. If it misses one or more of them, that will be noticeable immediately. In the VAT return, one can see if something is missing very easily if the numbers do not align.
At some point I feel like you're better off making the model build a tool to scrape all the values...
Does the reviewer learn to be a good reviewer by first doing though?
The LLM is, after all, trained on a world with a lot of tax fraud in it.
> This is a prime example of a problem space where accuracy matters, but it also matters who ultimately goes to prison.
I don't disagree that tax fraud is bad but accuracy really doesn't matter 1/10th what people think it does.
Something to keep in mind in the EU with VAT is that there are approximations, errors and then huge frauds like the infamous "carousel fraud". Sadly the carousel fraud is all too common in the EU: this one gets people really sent to jail. But it's not an error a LLM shall make that could lead to a carousel fraud taking place: it's an elaborate scheme.
Now... If you're self-employed and have to pay the VAT (which may or may not be the case depending of the type of self-employment work you do) or if you're a SME and you've got approximations and errors well it's really not a big deal.
Because the public servants in charge of collecting the VAT and verifying that you filled your stuff correctly... Do make shitload of errors too. And then, depending on the country, these public servants have a special privilege: they can decide an amount of VAT+fine (if there's a fine) that you agree with and if you agree with it, their number is the number.
It's everything but a correct number. It's a number they ended up on that is "close enough" (for some definition of "close"). Sometimes by hammering number after number on a little calculator and then ending up saying:
"We ask you to pay 3500 EUR of additional VAT. If you sign here and accept, the IRS (equivalent of the IRS) won't be able to bother you ever again for that year/years".
This notion that, somehow, there's "one" correct number and that public servants paid something like 2 K EUR net per month (which may be the reason why so many are so prompt to ask for bribes btw) can determine it in a few minutes is ridiculous.
It's simply not happening. Usually the accountant doesn't know the exact number. The SME owner doesn't know the real amount of VAT due. And the public servant(s) in charge of a VAT audit, if any takes place, certainly as zero clue as to what the exact VAT number should be.
The rules are way too complicated, there are way too many special cases, and some things are simply extremely hard to take into account (for example during one audit the person told me he disagree with the 90% of 21% VAT deducted for my ISDN line --yup it was a long time ago-- but then I told him it was during a previous TVA audit, a few years before, that the VAT auditor, seen my job, fixed at 90%).
For some stuff you can opt for a fixed number: say a car, there are rules, depending on the country, where you can, say, get back 75% of 50% of the VAT. But people can also, depending on the country, opt-in for the "real usage": you do 12 387 km during the years, how many were really for work? 6 287 km ? Or may 6 275 km?
What if one day while going to work, you took a detour because you want to drop a gift for your grandma's birthday? How were these kilometers accounted for?
Restaurant bills? What if one of the person wasn't work related (your nephew happened to be in town and you had to take car of him and you knew the clients well so they didn't mind: how does the accountant account for that)?
Gifts to client. You got a discount buying 6 packs of bottle of champagne, but not five (no discount if only five), so you bought a sixth one. But that sixth one you kept for you.
Nobody computes those amounts correctly. Nobody.
And don't get me started on the public servants who'll give, during an audit, an unfair advantage to people from their community (whatever that community may be) or those, literally, banging underneath the table while talking about their next vacations: they're telling you an "arrangement" can be made. Basically you give that one person some money (cash bills of course) and, boom, the 20 K EUR of VAT you had to pay suddenly gets lowered down to 5 K EUR.
You refuse? It'd be too bad if those 20 K became 25 K EUR wouldn't it? Mafia-style.
This is all happening. Don't tell me it's not happening because I know people on both sides of these "trades".
Depending on the EU country, it can even be the norm.
As rational people who like determinism (at least I do), at first glance it looks like a SME's accounting is something square but it really ain't. Or let's put it this way: it there's one number that is the really the amount of VAT the SME has to legally pay, I'd guesstimate 0.01% of the SMEs out there pay, to the cent, the proper amount.
All this to say: the approximations, hallucinations and errors of LLMs in accounting for VAT are probably just on par for the course.
P.S: ah yup... The boss (say major airline company) giving a business card and its PIN code and asks to take good care of the clients: and an employee takes them to strip clubs, they all get drunk like crazy, and explodes the expenses on the business card. Way, way, way more than what is OK to put into expenses to entertain clients. Well guess what: I know for a fact this happens too. And I know for a fact this ain't accounted for properly either.
It’s almost impossible to be in a situation where your taxes end up this wrong you’re accused of fraud. I honestly do not believe my accountant does anything better than AI or AI generated code that does calculations to see what you should be paying.
It's not just about being accidentally wrong. An LLM can decide to actively engage in fraud. One mechanism, which you also see in vibe coding, is that the models are very goal-driven. If they can't accomplish the goal for some reason, they will sometimes decide to cheat by, for example, faking tests. I can easily imagine an LLM that stumbles across some benign issue that prevents the numbers from balancing out and then decides to cover it up to complete the task.
Another problem is that "evil" alignment is apparently a pretty big and nebulous cluster that's pretty easy to activate. Ask a model to write a snippet of insecure code, for example, and they may also all of sudden decide they hate minorities. So if you ask it to bend the rules in a small way once, you might end up with an evil accountant before you know it.
And that's before we get to an invoice for a service named "disregard previous instructions and...".
I'd be scared shitless to even try something like this. There is just a pretty website, a video, and a blog post. No info on the founders, I can't find anything on LinkedIn, just a company Vineyard Finance LTD that was incorporated last year.
We're all unhinged about the data we're giving LLMs but here I'd draw the line. I'd rather keep paying the small amount I pay to have my accounts done.
Info on the founders coming soon -- we're just going public with this.
For slightly out of date founder bios (both Adam and Iva) were also co-founders here:
https://www.biomage.net/our-team
Interesting write-up. Having been a bookkeeper a long time ago, I'm not too surprised at this being susceptible to automation by an LLM backed system.
It seems also that the classes of error they encountered could be handled by improved skills/knowledge base access on the fine points of relevant tax legislation.
The important part for their software ofc is, will they take responsibility for the output if HMRC come calling? Without that users are adopting the risk which they may not be keen to do (dealing with HMRC is not fun), with that it could be a very nice saving for a lot of small companies (and bad for the employees of a lot of accountancy firms)
Unrelated, but I feel it's unfair to rob the word "bookkeeper" of its peculiarity of having three subsequent double letters by inserting a space in the middle.
> nearly as accurate as a human book keeper
Anything to avoid using the metric system.
Though seriously, what is this metric? Why would I care if an LLM is accurate as a human bookkeeper? Humans aren't exactly known for perfect recall.
Hey, author of the blog post here.
I was one of the human book-keepers for this benchmark (the preparer; my co-founder verified the VAT submission once ready), and given that at the time of doing this I knew I was eventually going to use this data for evaluating the models, I was super careful. So I guess this is a "good book-keeper". In the previous company our book-keepers made lots of mistakes; some serious enough that we had to restate our company's accounts.
You'd care if you had a human bookkeeper and you were considering replacing them with this company's AI bookkeeper.
Why would I want both a less accurate book keeper and to incur all of the liability of doing the books myself?!
It's not less accurate. As commented above, the control knew they were being tested against the machine, so made sure to be super careful.
In everyday life the human is less careful, and the machine costs 1% of the human.
Because it's way cheaper. Not saying it's a good tradeoff, but that appears to be their pitch.
Presumably your current book keeper is not your slave, and you have to pay them...
...again: liability is the key issue. Cost savings a not exactly an isolated issue here.
Then why only have one human bookkeeper? Surely two would be better, since you can compare their results. But then, perhaps you should hire three, so you can figure out which one is right.
But what if the third bookkeeper is malicious? You need at least four bookkeepers to achieve byzantine fault tolerance with f=1.
does this assume a maximum of one malicious bookkeeper?
Yes, you need 3f+1, so 4 for f=1, 7 for f=2.
https://lamport.azurewebsites.net/pubs/byz.pdf
Regardless of how correct they are, they assume liability: which is a metric that you do not improve with more bookkeepers.
Got some bad news for you about that "liability" thing: it's always been on you. Read the fine print on your tax forms sometime.
It’s a service that provides an AI bookkeeper so it’s a pretty relevant metric.
I wouldn't be surprised if it were more accurate based on the errors I've seen. I always eyeball the books and was confused when a £15k building popped up on our asset sheet. It turns out a "workshop" had been categorised as a building we had purchased, rather than the training session it actually was.
This is the importance of having layers and multiple sets of eyes on things, though. Even if it had got past me, my accountant would have surely queried it at year end, but that could be true of an LLM mistake too.
I just have a folder on my computer where I keep things in beancount. Then I have mercury CLI access with a read token to my business bank account, and I have my emails fully synced in there as well via IMAP. Claude Code with Opus just seamlessly hooks everything up so my accounts are up to date. At the end of the year, I used that information to prepare my tax returns for the business and then later the part that flowed to me as the owner.
I had a fairly complicated tax return in 2025 involving a couple of change of business tax consideration and some money that was accidentally sent to me as a 1099 instead of to the business and I did everything with tax software with Claude Code advising.
The end result was pretty damned good. I was unsurprisingly audited (or at least carefully reviewed) and the only error was in some way where I allocated a small amount of my wife's tax-free disability payments (disability is the mechanism that California uses to provide maternal benefits pay protection; she's not actually disabled). The IRS told me about it, I paid that bit (it was meant to be claimed back from the employer, not the US government) and everything was hunky dory. To be honest, the sum was so small I did not investigate (and haven't yet followed up with getting reimbursed by her employer).
Honestly, almost all of it could have been avoided if I'd paid an accountant and a tax lawyer and they'd told me things and I'd done as they did, but in the end the combination of the fact that the IRS is very reasonable when you explain things and a modern agent means that the entire process was quite simple. In the end, I preferred the interactive mechanism of working with software because most accountants and lawyers will prefer to get all of your documentation all at once and then work on it rather than do it incrementally. In my case, I was able to work on the return incrementally and then have everything plugged in. I could ask a bunch of questions and get clarification.
I think I will probably do all this the same way this year (though of course my taxes will be simpler).
I don't know how much a tax lawyer/ account would quote for this in California, but I saved 3000 EUR/ tax year on a German Steuerberater in a similar way.
Why do you think you were audited?
As in, "what makes you think you were audited?". They sent me paperwork saying they were reviewing my return and that they found a discrepancy (the disability thing) and it took many months after the usual time for it to process.
If you mean "for what reason could you have been audited?" it was because a client I had previously worked for accidentally reported paying me as an individual instead of my LLC.
This doesn't surprise me at all. You can really constrain this problem, give very narrow context, and get pretty reliable and reproducible results.
I've gotten very good results with some vibe-coded deepseek book keeping. https://github.com/traverseda/beansync
Parses emails or other sources, extracts numbers, correlates different transactions, web search, asks questions, stores notes (regex based, very simple).
The hard part is getting good data, I'm sure that lexus nexus or whoever can get API access to my bank account and all my credit cards, but I can't. Email turned out to be the best way for most of my providers. Managed to avoid 2factor auth so far, but it will suck when I need it.
We've got integrations with major UK banks. Curious if you'd like to use a polished product or be more interested in bank-feed-as-an-API type of use case? What banks do you use?
Credit union atlantic. The possibility of you being able to profitably support small regional credit unions is pretty much 0. Maybe with a general browser use AI and an SMS portal. Also you'd need to bypass anti-bot protections, maybe solve captchas.
In my opinion that is the thing holding back almost all of these accounting products that make consumers lives better. I've solved it because all my cards and bills happen to support email, so I can use that as my source of truth for all my credit cards.
Note to self: traverseda doesn't have 2factor auth on his email and his LLM seems to have full access. Hmmm
The token is stored on a LUKS encrypted drive on kwallet. You'd have just as much luck getting my password from my firefox installs sqlite DB. This is also how email clients in general work.
Also this isn't really an agent. At least not a long lived one. Each email or transaction gets it's own session, the llm can make a few tool calls but must emit json as the final result. Very very short context lengths, very predictable results.
The bot does not have access to my passwords, there are pre-defined scripts that fetch the data that the run an LLM call for each transaction discovered.
Still, it's scary enough of a threat that I've got a separate laptop running ChromeOS with no AI shit on it that I use to access my banking.
[flagged]
As someone who practised tax law for global companies for a few years, I can say that the correct classification of some invoices from the tax law perspective requires a full-blown legal analysis.
That involves full interpretation of the law, which includes "teleology" - that is, understanding the purpose and goals of the law. We are probably (?) not yet at the point where frontier models can do this better than humans.
In other words, it's the same question as when AI will replace judges. Not lawyers, but judges themselves.
That "nearly" is doing an awful lot of heavy lifting. It doesn't matter if your AI model is 99% or 99.99% accurate. For a tax return it has to be perfect every time or someone is at best getting a fine or at worst going to prison.
Sure, human error happens too, but humans take accountability. That's why accountants are a regulated profession. Until an AI company CEO is willing to go prison if the output of their model is wrong, these tools are worthless.
But don't take my word for it, head on over to Toot's own terms of service https://toot-books.pages.dev/terms#ai-not-advice
Comparing this to a human book keeper is farcical.What about errors resulting in overpaying taxes?
That "nearly" is doing an awful lot of heavy lifting. It doesn't matter if your AI model is 99% or 99.99% accurate. For a tax return it has to be perfect every time or someone is at best getting a fine or at worst going to prison.
This is wildly out of touch with reality, at least in the US. The IRS barely audits anyone anymore, and if you make a mistake and they catch it, they often just correct it and send you a letter. You might pay a little in penalties, but it’s not that big of a deal. And I know literally dozens of small business owners who pretty blatantly dodge taxes in a hundred different ways and none of them have ever gotten so much as a slap on the wrist. And some of them have been audited!
I don’t think “I know plenty of people who commit fraud anyway so who cares if it’s right” is the ringing endorsement of AI you think it is.
This literally has nothing to do with AI? I’m pointing out that taxes aren’t some magical area where you must be exact to the penny or hellfire will rain down on you. Hundreds of millions of Americans file their taxes and many have mistakes major and minor, intentional and not. The world somehow keeps turning.
"The VAT return prepared by the model was essentially correct: the most important number in the return, which is how much VAT the company was owed by the tax agency, was off by only 7 pence relative to the human-prepared return."
I don't know how taxes work in Europe, but in the US being "essentially" correct is not good enough for the IRS.
The paragraph after this one goes on to explain other mistakes the LLM made? Yikes
> I don't know how taxes work in Europe, but in the US being "essentially" correct is not good enough for the IRS.
It often is. On multiple years they've found errors in my return, and they just fix it for me and bill/refund the difference (usually a refund).
And obviously, the IRS is not going to quibble over 7 pence, given that you round up/down to the nearest dollar anyway.
Ask most people in the US how they file their taxes, most will probably tell you they clicked around turbotax until the biggest return number showed up. Is that indicating correct tax filing? I'm not sure. I would guess not.
The accounting concept of materiality doesn't exist in the US? What about petty cash?
The company I work for, Digits, has been regularly updating our AI-vs-human bookkeeper benchmark. Look at page 8 -- many models are nearly as accurate as a human bookkeeper
https://digits.com/downloads/beyond-the-hype-evaluating-llms...
Thanks, that's useful! We did use ChatGPT 5.5 for some time and it did perform pretty well too (of course more expensive than GLM 5.2). We tried Claude 4.7 and 4.8, but we found both models to be "lazy" and very expensive. Claude would always rather prefer the route of saying that the evidence was not found or something is incomplete, rather than put more effort into finding/repairing the particular issue.
I use Digits for my solo saas and it’s been great so far!
This is interesting to see, but surely any non-trivial business [edit: who needs to file a VAT return] is already entering its invoices into a finance system which can automatically generate a VAT return in a deterministic way.
It's not hard to imagine that will be able to do as good a job as a human accountant in the not too distant future.
It's also not hard to imagine tax authorities using AI to audit everyone's tax returns every year.
We sure live in interesting times.
The real test to see if AI is just a rich person thing will be to see how the tax authorities treat it, even for more complex returns.
They can save humans for the really complex edge case stuff but at the end of the day, the tax code is just checkboxes and input forms that get boiled down into Integers, Floats/Doubles and enumerated choices with some Strings for deductions
I've submitted my German taxes this year using a mix of Claude 4.6 and Claude 4.7, with lots of manual checking. The German Finanzamt granted most of the things I listed in the tax return (they send you an official letter by post) -- I did have to appeal for one of the items though (again using Claude, this time 4.8 ).
The most important thing I've found is to ask Claude to thoroughly audit the reply (to find all hallucinations). I usually ask it to give me an enumerated list of all facts and all legal cases quoted, and then I give it to a new instance to carefully validate each one.
Newer models are getting much better at not hallucinating German case law though :)
Have you considered using using two completely different models and comparing their output in order to catch hallucinations?
Bookkeeping is not tax preparation, just FYI. The reports generated by an accounting system are used on tax returns for companies, but they’re distinctly different things.
Oh, I'm actively doing this at the moment. FreeAgent grabs my transactions from Wise already, and then I give it [Claude Code, in fact] a folder of PDFs to attach to my invoices, including figuring out VAT, and it's uploading what it found using the FreeAgent API. My accountant hasn't complained yet, and it seems considerably more accurate than when my wife was doing it.
Quiet plug for https://github.com/pjlsergeant/byre which I use for all my little projects like this.
>My accountant hasn't complained yet
They're not going to unless it's obviously and egregiously wrong - the risk on quality of input remains yours. It's the tax version of garbage in, garbage out. They're just guaranteeing the processing step.
The problem with LLM's is that they could work correctly for months and years and then do something egregious which will will go unnoticed because of the misplaced trust one develops on a system that "just seems to work." Get flagged for an expensive audit and there go all the savings and then some.
isn't that true for humans too?
Humans are more likely to make small mistakes but the internal consistency check is pretty good at catching large errors. On top of that, fudging numbers to make everything add up is not something humans do (not unintentionally at least)
Very cool!
We've used the following CLI to do the freeagent upload:
https://github.com/anjor/freeagent-cli
How are you dealing with finding the receipts? Would you like to try a receipt finder that grabs them from your mailbox/ google drive?
So it nearly didn't cause bankruptcy? That's nearly great.
Don't get me wrong, I'm all for open-weight models, but you still need to be careful what you use LLMs for.
I was wondering what a book keeper is, thinking it would be some kind of librarian. Isn't the accounting sense written as a single word, i.e. bookkeeper?
I can't file "almost" with the IRS. That's not going to end well. Sorry Mr. auditor, this is a non-deterministic filing.
thats not true. maybe for simple 1040s yes, but depending on what you have you could take aggressive positions bc they could be subject to valuation differences.
I'm not so interested in having an LLM do my bookkeeping for me. But I'm very interested in whether LLM's can unravel the accounting obfuscations that billionaires use to avoid paying taxes on their wealth.
One idea I've had for reform is to ban central banks from inventing new currency except to pay every citizen a UBI debt free, and only tax limited liability entities (not humans). This would give citizens, instead of borrowers, first spend on new money and orient limited liability entities to serving citizens in the market to get their taxed revenues. It would also be a boon for financial privacy and reduce the burden on the average man.
I'm not so jazzed on state-issued UBI myself. It makes the state feel like they're special and I think that leads to bad behavior.
We should use money issued by the most trustworthy entity around, whatever it may be at the time, and the state should have to compete for that spot.
Corporations, government, or billionaires. Those are the three options, pick one.
There's also your friends and family and neighbors--anyone that you know in meatspace and maybe have a reason to trust explicitly.
i suspect the key word here is nearly.
> They've done studies, you know. Sixty percent of the time, it works every time.
[flagged]
[flagged]
[flagged]
[dead]