> it runs up to 1.7× faster than comparable MoEs on-device, with 184 tokens/s on an Apple M5 Max and up to 21K output tokens/s on a single NVIDIA B300 GPU
Wow, those are seriously impressive numbers for how intelligent this model seems to be!
One thing that this article doesn't clarify, and that I guess I really should look into, is what exactly AutomationBench tests for as compared to other benchmarks. The standout performance on AutomationBench is interesting to see, but I don't know what it really means because the model seems to score relatively close to other local models on every other benchmark.
I'd guess it's mostly just an artifact of the strength of their Agentic RL environments for those specific domains. If they had the same kind of access to massive amounts of institutional software knowledge, compute infrastructure, and hundreds of millions of users dropping them endless highly technical agent traces, there may have been similar disparities on the agentic benchmarks in other fields.
It's possible that they may have generally capped out for some reason... but, I'd rather be bullish on a new American AI lab making it to first place among western open-weight models that can run on a common consumer device. (Closest in size that's better across that range of benchmarks is Inkling-Small I think, at 276B params, unless I'm forgetting someone.)
I asked it to implement the simplest todo in ghostty codebase and it failed badly - kept going in circles fixing one compile error and introducing another. Qwen 3.8 was slower (40 tok/s compared to 75) but did get it done.
If Qwen had released a Qwen3.8 MoE of similar size, with a naive extrapolation its capabilities would roughly be similar to this model. So basically someone has caught up to Qwen (albeit Qwen from 1-2 months ago) in parameter efficiency.
I don't agree with that conclusion. I'm looking at the benchmarks page for LFM2.5-24B-A2B, and it appears to get a worse score on SWE-Bench-Verified (by a lot) when compared to Qwen 3.6 35B-A3B. It only clearly beats Qwen 3.6 35B on AutomationBench, ToolSandbox, and LiveClaw.
At least for my use cases, Qwen 3.6 35B-A3B appears better than this new release. An MoE for Qwen 3.8 would be even better than that!
Would you call Qwen 3.6 35B "similar size" with ~50% more parameters (both active and total) though? I believe what GP meant that if there was a Qwen 3.8 24B-A2B, it would roughly be as intelligent as this one.
yeah it's kind of the point. they're pedagogical tools. they give you the full recipe for building it from scratch or fine-tuning it beyond what it is.
but i'm prepared to be surprised, maybe they'll conjure up something standalone useful this time
>How many tokens/s do you need for a coding agent to feel like Claude or codex?
it's kind of a loaded question, this model gets a lot of tok/s but reasoning is forced on; the time-to-answer is really a combination of things like "how much does this thing want to over-reason" and "how much does this thing like tool use?" .
You are right, they are not exactly the same thing, just wanted to have a reference point
When running locally usually I have to trade off speed for a better model. I can run decently big models at about 2tk/s, or a pretty dumb one at 40-100tk/s
AFAIU prefill speed makes a bigger difference in agentic coding workflows as the harness likely spends more time reading than creating new code. Qwen in particular spends a lot of time reasoning so it does make a larger impact there
> it runs up to 1.7× faster than comparable MoEs on-device, with 184 tokens/s on an Apple M5 Max and up to 21K output tokens/s on a single NVIDIA B300 GPU
Wow, those are seriously impressive numbers for how intelligent this model seems to be!
One thing that this article doesn't clarify, and that I guess I really should look into, is what exactly AutomationBench tests for as compared to other benchmarks. The standout performance on AutomationBench is interesting to see, but I don't know what it really means because the model seems to score relatively close to other local models on every other benchmark.
I'd guess it's mostly just an artifact of the strength of their Agentic RL environments for those specific domains. If they had the same kind of access to massive amounts of institutional software knowledge, compute infrastructure, and hundreds of millions of users dropping them endless highly technical agent traces, there may have been similar disparities on the agentic benchmarks in other fields.
It's possible that they may have generally capped out for some reason... but, I'd rather be bullish on a new American AI lab making it to first place among western open-weight models that can run on a common consumer device. (Closest in size that's better across that range of benchmarks is Inkling-Small I think, at 276B params, unless I'm forgetting someone.)
I asked it to implement the simplest todo in ghostty codebase and it failed badly - kept going in circles fixing one compile error and introducing another. Qwen 3.8 was slower (40 tok/s compared to 75) but did get it done.
Have you tried Gemma-4 26BA4B? I feel like it is underrated for tasks like that, but maybe I've just been lucky with my use cases.
This seems like a big deal, no?
If Qwen had released a Qwen3.8 MoE of similar size, with a naive extrapolation its capabilities would roughly be similar to this model. So basically someone has caught up to Qwen (albeit Qwen from 1-2 months ago) in parameter efficiency.
I don't agree with that conclusion. I'm looking at the benchmarks page for LFM2.5-24B-A2B, and it appears to get a worse score on SWE-Bench-Verified (by a lot) when compared to Qwen 3.6 35B-A3B. It only clearly beats Qwen 3.6 35B on AutomationBench, ToolSandbox, and LiveClaw.
At least for my use cases, Qwen 3.6 35B-A3B appears better than this new release. An MoE for Qwen 3.8 would be even better than that!
Would you call Qwen 3.6 35B "similar size" with ~50% more parameters (both active and total) though? I believe what GP meant that if there was a Qwen 3.8 24B-A2B, it would roughly be as intelligent as this one.
Way more excited for Nvidia's A9B model coming out tomorrow.
Why? Their models are useful as a starting place for doing your own fine tuning, but they have never stood well on their own.
they always undertrain the model
yeah it's kind of the point. they're pedagogical tools. they give you the full recipe for building it from scratch or fine-tuning it beyond what it is.
but i'm prepared to be surprised, maybe they'll conjure up something standalone useful this time
They have an incentive to produce a more finished model to go with the launch of the RTX Spark laptops, I guess.
i'm gettin between 200-320 tok/s on a 5070ti with 131k context on llamacpp and no tuning.
very nice.
I can't speak to the intelligence of it yet.
these small agentic-focused models are a lot of fun to play with if you haven't yet.
Love seeing the effort Liquid is putting into local models. I’m looking forward to a lot of progress here in the next 5 years
> with 184 tokens/s on an Apple M5 Max
That seems pretty decent. How many tokens/s do you need for a coding agent to feel like Claude or codex?
>How many tokens/s do you need for a coding agent to feel like Claude or codex?
it's kind of a loaded question, this model gets a lot of tok/s but reasoning is forced on; the time-to-answer is really a combination of things like "how much does this thing want to over-reason" and "how much does this thing like tool use?" .
You are right, they are not exactly the same thing, just wanted to have a reference point
When running locally usually I have to trade off speed for a better model. I can run decently big models at about 2tk/s, or a pretty dumb one at 40-100tk/s
fwiw this thing puts out results very fast even on max reasoning.
the results i've gotten so far have been a little braindead though. still playing around. bad luck with traversing filesystems remotely so far.
it seems to want to make concluding decisions very fast regardless of reasoning levels.
AFAIU prefill speed makes a bigger difference in agentic coding workflows as the harness likely spends more time reading than creating new code. Qwen in particular spends a lot of time reasoning so it does make a larger impact there
Claude code is 60-80 tps, codex varies by model but mostly 40-50 tps.
I’d say only 65