I think this is a good idea in general, but perhaps a bit too simple. It looks like this only works for static sites, right? It then performs a JS fetch to pull in the html code and then converts it (in a quick and dirty manner) to markdown.
I know this is pointing to the GH repo, but I’d love to know more about why the author chose to build it this way. I suspect it keeps costs low/free. But why CF workers? How much processing can you get done for free here?
I’m not sure how you could do much more in a CF worker, but this might be too simple to be useful on many sites.
Example: I had to pull in a docs site that was built for a project I’m working on. We wanted an LLM to be able to use the docs in their responses. However, the site was based on VitePress. I didn’t have access to the source markdown files, so I wrote an MCP fetcher that uses a dockerized headless chrome instance to load the page. I then pull the innerHTML directly from the processed DOM. It’s probably overkill, but an example of when this tool might not work.
But — if you have a static site, this tool could be a very simple way to configure MCP access. It’s a nice idea!
The simplicity is a feature. I avoided headless Chrome because standard fetch tools (and raw DOM dumps) pollute the context with navbars and scripts, wasting tokens.
This parser converts to clean Markdown for maximum density.
Also, by treating this as an MCP Resource rather than a Tool, the docs are pinned permanently instead of relying on the model to "decide" to fetch them.
Cloudflare Workers handle this perfectly for free (100k reqs/day) without the overhead of managing a dockerized browser instance.
I like the idea of exposing this as a resource. That’s a good idea so you don’t have to wait for a tool call. Is using a resource faster though? Doesn’t the LLM still have to make a request to the MCP server in both cases? Is the idea being that because it is pinned a priori, you’ve already retrieved and processed the HTML, so the response will be faster?
But I do think the lack of a JavaScript loader will be a problem for many sites. In my case, I still run the innerHTML through a Markdown converter to get rid of the extra cruft. You’re right that this helps a lot. Even better if you can choose which #id element to load. Wikipedia has a lot of extra info that surrounds the main article that even with MD conversion adds extra fluff. But without the JS loading, you’re still going to not be able to process a lot of sites in the wild.
Now, I would personally argue that’s an issue with those sites. I’m not a big fan of dynamic JS loaded pages. Sadly, I think that that ship has sailed…
I’m a bit confused because I don’t clearly understand the value this tool adds. Could you help me understand it?
From what I can see, if the content I want to enrich is static, the web fetch tool seems sufficient. Is this tool capable of extracting information from dynamic websites or sites behind login walls, or is it essentially the same as a web fetch tool that only works with static pages?
I see many of you asking about the differences between using this versus web_fetch. The main differences are the quality of the data and token usage.
1. Standard web_fetch tools usually dump raw HTML into the context (including navbars, scripts, and footer noise). This wastes a huge amount of tokens and distracts the model. toMCP runs the page through a readability parser and converts it to clean markdown before sending it to the AI.
2. Adding a website as an MCP Resource pins it as a permanent, read-only context, making it ideal for keeping documentation constantly available. This differs from the web_fetch tool, which is an on-demand action the AI only triggers when it decides to, meaning the data isn't permanently attached to your project.
I thought this is what the web_fetch tools already did? Tools are configured through MCP also, right? So why am I prepending a URL, and not just using the web_fetch tool that already works?
Does this skirt the robots.txt by chance? Not being to fetch any web page is really bugging me and I'm hoping to use a better web_fetch that isn't censored. I'm just going to copy/paste the content anyway.
I think the idea here is that the web_fetch is restricted to the target site. I might want to include my documentation in an MCP server (from docs.example.com), but that doesn’t mean I want the full web available.
I am not quite clear why this adds value over a simple web fetch tool which does not require configuration per site.
I think this is a good idea in general, but perhaps a bit too simple. It looks like this only works for static sites, right? It then performs a JS fetch to pull in the html code and then converts it (in a quick and dirty manner) to markdown.
I know this is pointing to the GH repo, but I’d love to know more about why the author chose to build it this way. I suspect it keeps costs low/free. But why CF workers? How much processing can you get done for free here?
I’m not sure how you could do much more in a CF worker, but this might be too simple to be useful on many sites.
Example: I had to pull in a docs site that was built for a project I’m working on. We wanted an LLM to be able to use the docs in their responses. However, the site was based on VitePress. I didn’t have access to the source markdown files, so I wrote an MCP fetcher that uses a dockerized headless chrome instance to load the page. I then pull the innerHTML directly from the processed DOM. It’s probably overkill, but an example of when this tool might not work.
But — if you have a static site, this tool could be a very simple way to configure MCP access. It’s a nice idea!
The simplicity is a feature. I avoided headless Chrome because standard fetch tools (and raw DOM dumps) pollute the context with navbars and scripts, wasting tokens. This parser converts to clean Markdown for maximum density.
Also, by treating this as an MCP Resource rather than a Tool, the docs are pinned permanently instead of relying on the model to "decide" to fetch them.
Cloudflare Workers handle this perfectly for free (100k reqs/day) without the overhead of managing a dockerized browser instance.
I like the idea of exposing this as a resource. That’s a good idea so you don’t have to wait for a tool call. Is using a resource faster though? Doesn’t the LLM still have to make a request to the MCP server in both cases? Is the idea being that because it is pinned a priori, you’ve already retrieved and processed the HTML, so the response will be faster?
But I do think the lack of a JavaScript loader will be a problem for many sites. In my case, I still run the innerHTML through a Markdown converter to get rid of the extra cruft. You’re right that this helps a lot. Even better if you can choose which #id element to load. Wikipedia has a lot of extra info that surrounds the main article that even with MD conversion adds extra fluff. But without the JS loading, you’re still going to not be able to process a lot of sites in the wild.
Now, I would personally argue that’s an issue with those sites. I’m not a big fan of dynamic JS loaded pages. Sadly, I think that that ship has sailed…
Who is tom and why is he copying?
lol I got this domain at 2am and didn't think through.
I’m a bit confused because I don’t clearly understand the value this tool adds. Could you help me understand it?
From what I can see, if the content I want to enrich is static, the web fetch tool seems sufficient. Is this tool capable of extracting information from dynamic websites or sites behind login walls, or is it essentially the same as a web fetch tool that only works with static pages?
I see many of you asking about the differences between using this versus web_fetch. The main differences are the quality of the data and token usage.
1. Standard web_fetch tools usually dump raw HTML into the context (including navbars, scripts, and footer noise). This wastes a huge amount of tokens and distracts the model. toMCP runs the page through a readability parser and converts it to clean markdown before sending it to the AI.
2. Adding a website as an MCP Resource pins it as a permanent, read-only context, making it ideal for keeping documentation constantly available. This differs from the web_fetch tool, which is an on-demand action the AI only triggers when it decides to, meaning the data isn't permanently attached to your project.
I thought this is what the web_fetch tools already did? Tools are configured through MCP also, right? So why am I prepending a URL, and not just using the web_fetch tool that already works?
Does this skirt the robots.txt by chance? Not being to fetch any web page is really bugging me and I'm hoping to use a better web_fetch that isn't censored. I'm just going to copy/paste the content anyway.
I think the idea here is that the web_fetch is restricted to the target site. I might want to include my documentation in an MCP server (from docs.example.com), but that doesn’t mean I want the full web available.
This is a clever solution to a real problem. I could use this for quick turn around from webpage kb to the mcp. Thanks for sharing.
Fun idea although I thought the industry is leaning towards using llms.txt.
Isn’t that for scraping? I think this is for injecting (or making that possible) to add an MCP front end to a site.
Different use cases, I think.
Cool :)