In this case the "agent" definition they are using is the one from the https://github.com/openai/openai-agents-python Python library, which they are running in the browser via Pyodide and WASM.
That library defines an agent as a system prompt and optional tools - notable because many other common agent definitions have the tools as required, not optional.
> notable because many other common agent definitions have the tools as required, not optional.
This feels weird to me. I would think of an agent with no tools as the trivial case of a "null agent" or "empty agent".
It would be like saying you can't have a list with no elements because that's not a list at all... but an empty list is actually quite useful in many contexts. Which is why almost all implementations in all languages allow empty lists and add something distinct like a NonEmptyList to handle that specific case.
We looked at Pyodide and WASM along with other options like firecracker for our need of multi-step tasks that require running LLM generated code locally via Ollama etc. with some form of isolation than running it directly on our dev machines and figured it would be too much work with the various external libraries we have to install. The idea was to get code generated by a powerful remote LLM for general purpose stuff like video editing via ffmpeg, beautiful graphs generation via JS + chromium and stuff and execute it locally with all dependencies being installed before execution.
We built CodeRunner (https://github.com/BandarLabs/coderunner) on top of Apple Containers recently and have been using it for sometime. This works fine but still needs some improvement to work across very arbitrary prompts.
For the Gemini-cli integration, is the only difference between code runner with Gemini-cli, and gemini-cli itself, is that you are just using Gemini-cli in a container?
No, Gemini-cli still is on your local machine, when it generates some code based on your prompt, with Coderunner, the code runs inside a container (which is inside a new lightweight VM courtesy Apple and provides VM level isolation), installs libraries requested, executes the generated code inside it and returns the result back to Gemini-cli.
This is also not Gemini-cli specific and you could use the sandbox with any of the popular LLMs or even with your local ones.
When I saw the title, I thought this was running models in the browser. IMO that's way more interesting and you can do it with transformers.js and onnx runtime. You don't even need a gpu.
You never need WASM (or any other language, bytecode format, etc) to talk to LLMs. But WASM provides things people might like for agents, eg. strict sandboxing by default.
From a quick gander.
WASM is not to talk to the servers.
WASM can be utilized to run AI Agents to talk to local LLMs from a sandboxed environment through the browser.
For example in the next few years if Operating System companies and PC producers make small local models stock standards to improve the operating system functions and other services.
This local LLM engine layer can be used by browser applications too and that being done through WASM without having to write Javascript and using WASM sandboxed layer to safely expose the this system LLM Engine Layer.
They're using some python libraries like openai-agents so presumably it's to save on development efforts of calling/prompting/managing the HTTP endpoints. But yes this could just be done in regular JS in the browser, they'd have to write a lot of boilerplate for an ecosystem which is mainly Python.
I recently wrote some Javascript to automate clicking coupons. The website checks for non-human clicks using event.isTrusted. Firefox allowed me to bypass this by rewriting the JS to replace s/isTrusted/true, while Chrome Manifest V3 doesn't allow it. Anyway, Firefox might be the future of agents, due to its extensibility.
Mildly interesting article - I mean, you can already run a ton of libraries that talk to an inference backend. The only difference here is that the client-side code is in Python, which by itself doesn't make creating agents any simpler - I would argue that it complicates things a tone.
Also, connecting a model to a bunch of tools and dropping it into some kind of workflow is maybe 5% of the actual work. The rest is spent on observability, background tasks, queueing systems, multi-channel support for agents, user experience, etc., etc., etc.
Nobody talks about that part, because most of the content out there is just chasing trends - without much real-world experience running these systems or putting them in front of actual customers with real needs.
Agreed, regarding the other parts of the "LLM" stack, have a look at the, IMO best, LLM coordination / Observability platform TS library:
https://mastra.ai/
The frustrating thing about this is the limitation of using a browser. Agents should be long-running processes that exist external to a browser. The idea of using wasm is clever, but it feels like the entire browser environment needs to evolve because we're no longer dealing with just web pages. I think we are looking at a true evolution of the web now if this is the way it's going to go
> Agents should be long-running processes that exist external to a browser
Sure, but there are a ton of ways for doing that today. What this specific thing is addressing, is removing the requirement of "the dependency on extra tools and frameworks that need to be installed before the agents can be run".
This is trying to use the word agent to make it sound cool, but it doesn't make a case for why it's particularly about agents and not just basic level AI stuff.
> The agent code is nothing more than a Python script that relies on the openai-agents-python library to run an AI agent backed by an LLM served via an OpenAI-compatible API.
The openai-agents-python code is useful for writing agents but it is possible to use it to write code that isn't very agentic. None of the examples are very agentic.
i build an opensource mobile browser - we create ai agents (that run in the background) on the mobile browser. and build an extension framework on top so u can create these agents by publishing an extension.
we hook into the android workmanager framework and do some quirky things with tab handling to make this work. its harder to do this on mobile than on desktop.
bunch of people are trying to do interesting things like an automatic labubu purchase agent (on popmart) :D
lots of purchase related agents
Can you bypass the cors issue with a browser extension? I seem to recall CORS doesn't apply to extensions, or at least the part that isn't injected to the webpages.
CORS is mentioned on that page: NOTE: If you want to run tools that get information from some other server into your HTML page (e.g. the visit_webpage tool or the Ollama server itself), you need to make sure that CORS is enabled for those servers. For more information, refer to the troubleshooting section in our GitHub repository.
Of course. Here is a corrected version of your text that fixes the grammar and typos while keeping the colloquial tone:
I'd like to offer a less skeptical view on this, contrary to what I've read here so far. LLMs that act (a.k.a. agents) bring a whole lot of new security and privacy issues. If we were already heading toward a privacy dystopia (with trackers, centralized services, etc.), agents could take that to a whole new level.
That's why I can only cheer when I see a different path where agents are run locally (by the way, Hugging Face has already published a couple of spaces demonstrating that). As a plus, because they're small, their environmental footprint will also be smaller (although, admittedly, I can also see the Jevons Paradox possibly happening here too).
That’s a well-remembered video, but I don’t really think it fits with the original comment. What we’re seeing is more akin to already having the plane flying (badly, still crashing frequently and landing in the wrong country) and instead of making it more reliable, everyone is trying different wheels and paint colours.
We’re not really seeing any significant development with this. What LLMs need most desperately (and are far from getting) is reliability and not being convincing liars. Being able to query existing server models from your oven timer is a cool gimmick but not really transformative or advancing anything.
It’s like a reverse JFK Space Effort speech: “We choose to indiscriminately throw at the wall every single LLM-adjacent idea we can think of. Not because it is useful, but because it is easy and potentially profitable”.
In this case the "agent" definition they are using is the one from the https://github.com/openai/openai-agents-python Python library, which they are running in the browser via Pyodide and WASM.
That library defines an agent as a system prompt and optional tools - notable because many other common agent definitions have the tools as required, not optional.
That explains why their "hello world" demo just runs a single prompt: https://github.com/mozilla-ai/wasm-agents-blueprint/blob/mai...
> notable because many other common agent definitions have the tools as required, not optional.
This feels weird to me. I would think of an agent with no tools as the trivial case of a "null agent" or "empty agent".
It would be like saying you can't have a list with no elements because that's not a list at all... but an empty list is actually quite useful in many contexts. Which is why almost all implementations in all languages allow empty lists and add something distinct like a NonEmptyList to handle that specific case.
A lot of the agent/agentic definitions I see floating around are variants on "an LLM running tools in a loop".
Can't do that without tools!
We looked at Pyodide and WASM along with other options like firecracker for our need of multi-step tasks that require running LLM generated code locally via Ollama etc. with some form of isolation than running it directly on our dev machines and figured it would be too much work with the various external libraries we have to install. The idea was to get code generated by a powerful remote LLM for general purpose stuff like video editing via ffmpeg, beautiful graphs generation via JS + chromium and stuff and execute it locally with all dependencies being installed before execution.
We built CodeRunner (https://github.com/BandarLabs/coderunner) on top of Apple Containers recently and have been using it for sometime. This works fine but still needs some improvement to work across very arbitrary prompts.
For the Gemini-cli integration, is the only difference between code runner with Gemini-cli, and gemini-cli itself, is that you are just using Gemini-cli in a container?
No, Gemini-cli still is on your local machine, when it generates some code based on your prompt, with Coderunner, the code runs inside a container (which is inside a new lightweight VM courtesy Apple and provides VM level isolation), installs libraries requested, executes the generated code inside it and returns the result back to Gemini-cli.
This is also not Gemini-cli specific and you could use the sandbox with any of the popular LLMs or even with your local ones.
Thanks for explaining
When I saw the title, I thought this was running models in the browser. IMO that's way more interesting and you can do it with transformers.js and onnx runtime. You don't even need a gpu.
https://huggingface.co/spaces/webml-community/llama-3.2-webg...
I think you _do_ need a GPU. But it can work with an integrated one, no need for a discrete one.
I can't run it on Linux since WebGPU is not working for me...
It is yet not available on any browser on Linux, other than Android/Linux and ChromeOS.
It seems the only code that runs in the browser here is the code that talks to LLMs on servers.
Why would you need WASM for this?
You never need WASM (or any other language, bytecode format, etc) to talk to LLMs. But WASM provides things people might like for agents, eg. strict sandboxing by default.
From a quick gander. WASM is not to talk to the servers. WASM can be utilized to run AI Agents to talk to local LLMs from a sandboxed environment through the browser.
For example in the next few years if Operating System companies and PC producers make small local models stock standards to improve the operating system functions and other services. This local LLM engine layer can be used by browser applications too and that being done through WASM without having to write Javascript and using WASM sandboxed layer to safely expose the this system LLM Engine Layer.
No matter if the LLM is on the same machine or elsewhere, why would you need WASM to talk to it and not just JS?
They're using some python libraries like openai-agents so presumably it's to save on development efforts of calling/prompting/managing the HTTP endpoints. But yes this could just be done in regular JS in the browser, they'd have to write a lot of boilerplate for an ecosystem which is mainly Python.
> But yes this could just be done in regular JS in the browser, they'd have to write a lot of boilerplate for an ecosystem which is mainly Python.
Surely that's a prime use for AI?
I have a demo that runs llama3-{1,3,8}B in browser on cpu. It can be integrated with this thing in the future to be fully local
https://galqiwi.github.io/aqlm-rs
I recently wrote some Javascript to automate clicking coupons. The website checks for non-human clicks using event.isTrusted. Firefox allowed me to bypass this by rewriting the JS to replace s/isTrusted/true, while Chrome Manifest V3 doesn't allow it. Anyway, Firefox might be the future of agents, due to its extensibility.
Mildly interesting article - I mean, you can already run a ton of libraries that talk to an inference backend. The only difference here is that the client-side code is in Python, which by itself doesn't make creating agents any simpler - I would argue that it complicates things a tone.
Also, connecting a model to a bunch of tools and dropping it into some kind of workflow is maybe 5% of the actual work. The rest is spent on observability, background tasks, queueing systems, multi-channel support for agents, user experience, etc., etc., etc.
Nobody talks about that part, because most of the content out there is just chasing trends - without much real-world experience running these systems or putting them in front of actual customers with real needs.
Agreed, regarding the other parts of the "LLM" stack, have a look at the, IMO best, LLM coordination / Observability platform TS library: https://mastra.ai/
thanks, we are building chatbotkit.com / cbk.ai (not opensource)
The frustrating thing about this is the limitation of using a browser. Agents should be long-running processes that exist external to a browser. The idea of using wasm is clever, but it feels like the entire browser environment needs to evolve because we're no longer dealing with just web pages. I think we are looking at a true evolution of the web now if this is the way it's going to go
That's what the Component Model[0] is all about.
WASIp3[1] is gonna be awesome. Hopefully releasing later this year.
[0]: https://component-model.bytecodealliance.org/
[1]: https://wasi.dev/roadmap
> Agents should be long-running processes that exist external to a browser
Sure, but there are a ton of ways for doing that today. What this specific thing is addressing, is removing the requirement of "the dependency on extra tools and frameworks that need to be installed before the agents can be run".
When you say agents should be long running, which definition of "agent" are you talking about?
mcp.run is entirely based on wasm. Tools can run on our cloud or locally
This is trying to use the word agent to make it sound cool, but it doesn't make a case for why it's particularly about agents and not just basic level AI stuff.
> The agent code is nothing more than a Python script that relies on the openai-agents-python library to run an AI agent backed by an LLM served via an OpenAI-compatible API.
The openai-agents-python code is useful for writing agents but it is possible to use it to write code that isn't very agentic. None of the examples are very agentic.
i build an opensource mobile browser - we create ai agents (that run in the background) on the mobile browser. and build an extension framework on top so u can create these agents by publishing an extension.
we hook into the android workmanager framework and do some quirky things with tab handling to make this work. its harder to do this on mobile than on desktop.
bunch of people are trying to do interesting things like an automatic labubu purchase agent (on popmart) :D lots of purchase related agents
pull requests welcome ! https://github.com/wootzapp/wootz-browser/pull/334
Can you bypass the cors issue with a browser extension? I seem to recall CORS doesn't apply to extensions, or at least the part that isn't injected to the webpages.
CORS is mentioned on that page: NOTE: If you want to run tools that get information from some other server into your HTML page (e.g. the visit_webpage tool or the Ollama server itself), you need to make sure that CORS is enabled for those servers. For more information, refer to the troubleshooting section in our GitHub repository.
No mention of WebGPU...
Putting Python code in a string in a html file is a big no no for me. We should be passed this. It looks like going 20 years back in time.
Having to disable CORS restrictions is a bit meh, I understand why, but still.
hmm
Of course. Here is a corrected version of your text that fixes the grammar and typos while keeping the colloquial tone:
I'd like to offer a less skeptical view on this, contrary to what I've read here so far. LLMs that act (a.k.a. agents) bring a whole lot of new security and privacy issues. If we were already heading toward a privacy dystopia (with trackers, centralized services, etc.), agents could take that to a whole new level.
That's why I can only cheer when I see a different path where agents are run locally (by the way, Hugging Face has already published a couple of spaces demonstrating that). As a plus, because they're small, their environmental footprint will also be smaller (although, admittedly, I can also see the Jevons Paradox possibly happening here too).
I guess we're at the stage where every permutation of "AI Agents" and X (Where X is technologies & or spaces) must be tried and posted on HN.
We are in the early stage :)
https://www.youtube.com/watch?v=gN-ZktmjIfE
That’s a well-remembered video, but I don’t really think it fits with the original comment. What we’re seeing is more akin to already having the plane flying (badly, still crashing frequently and landing in the wrong country) and instead of making it more reliable, everyone is trying different wheels and paint colours.
We’re not really seeing any significant development with this. What LLMs need most desperately (and are far from getting) is reliability and not being convincing liars. Being able to query existing server models from your oven timer is a cool gimmick but not really transformative or advancing anything.
It’s like a reverse JFK Space Effort speech: “We choose to indiscriminately throw at the wall every single LLM-adjacent idea we can think of. Not because it is useful, but because it is easy and potentially profitable”.
Sometimes you gotta shake the tree to see what falls out.
Or pump the bubble
That's either the peak of inflated expectations or the slope of enlightenment.
Depending on what side you're on and "only time will tell."
Ship AI Agents as a web page :-)
An excellent idea to have those be AI generated and posted! I'll start:
https://chatgpt.com/share/68679564-6a44-8012-b1bd-25819bfbf0...