Are you using residential proxies? How do you handle websites that don't want to be scraped.
EG if I start passing in Linkedin pages what is your expectation of the result that people would see per profile.
EDIT:
Congrats on the launch seriously hard work, just wanting to understand your scraping stance more. I've worked with a lot of tools on this, didn't mean for my initial comment to be adversarial.
yes, we use residential proxies + all requests go are js-rendered, we maintain a caching layer which is 95%+ opted into by customers
it's all included in the credit price, great value compared to alternatives, the business model does rely on scale and our margin gets better the more requests we serve (esp infra cost for k8 + browser fleet)
to answer your e.g., yes public linkedin pages will work fine, anything behind a login we don't really support out of the box until we can figure out a safe way to do so, since that's where red lines are drawn
we step in whenever we see our service is hitting a website more than it should, this usually means reaching out to a customer for clarity on why they are not opting into the cache, we have alot of safeguards around fraud/spam and will let someone know if their request pattern looks like they're causing harm
I like the clarity, tone, and readability of your webpage. Also your FAQ is refreshing
> When Should I talk to sales?
> Talk to sales if you need high-volume pricing beyond 2M credits/month, custom rate limits, SSO / SAML, SCIM provisioning, an uptime SLA, annual invoicing, an MSA / DPA, or a dedicated support channel. Reach us at hello@context.dev or through the contact page.
Would that this were the norm everywhere, rather than (say) a sales rep from Datadog scraping my phone number from who knows where to ask about my company's needs after I sign up for a free account on a whim :)
happy you noticed that, i put quite a bit of love into the ui, i've found engineers care alot about polish & feel of the webapps they use, even for an api product
i'm an engineer by trade, and always hated things like forcing a sales call, or having hidden credit multipliers, i tried to build this with the same ethos i like for my own dependencies (shoutout axiom.co)
I was using Context back when it was still Brand.dev. I found it to be a great product- one of those rare APIs that immediately made a problem I had disappear. Had it in production within an hour of signing up
Agents need clean/current context from the web, and this is the best way I’ve found to give it to them. The internet is clearly moving in this direction: companies are starting to realize their sites need to be legible to agents. Some are already adapting but many haven’t yet. Context feels like an important part of that transition
Yahia is a great builder. His pace of expansion has been impressive, excited to see where he takes Context.
Unclear what difference exists against Firecrawl - their team has been shipping great features extremely quickly lately, and their core offerings have become really good.
I am interested in KnifeGeek though - looking for a good OTF (ultratech?)
There's alot of overlap at the moment, but we're diverging quite quickly.
If you want differences as of right now
- 1 credit = 1 scrape, no hidden credit multiplier
- we have world class brand data
- we're focused primarily on the infra use-case, rather than gtm & everything else
- anecdotally, customers have seen their error rates drop quite dramatically
In general it's a huge space, firecrawl is a wonderful company, it's fun to compete with them, planning to add more things soon which should make the differences clear
please do, and if you see anything off let me know, we've yet to lose a single "bakeoff", i normally ask customers to just have cc/codex run it so its somewhat unbiased
on the 2nd point, most industries are not zero-sum, and many of customers use multiple data providers in any case, so agree with you there
oh also on knifegeek, it'll be live again this weekend when i find a moment to fix the bug causing it to crash, i collect knives/watches so it was a super fun project to work on
we try to read robots.txt, it's definitely more efficient if agents just see what humans see, we run a custom browser rendering stack
in terms of llms.txt, we're not primarily an AI product (although some features do use LLMs), and speaking to friends who run products it seems to be not very helpful, even though we have one as well, i didn't see it move the needle much
even my own coding agents don't look at llms.txt when looking at our own website, so unsure of whether that standard will survive the test of time
im not sure the models do this, rather the tooling around them, however web search/extraction by the model providers is ridiculously expensive and quite slow, so going into production it makes sense to use a provider (like us)
yes but we do alot more that may not be clear at first glance, things like brand data, and for scraping handling pdf, ocr, docx, ppt, xlsx automatically
shipping a bunch of new things soon which should make it clearer, but as of today yeah
Great, another thing I have to block server side. Reminds me of the image leech protections that had to be in place because bandwidth was expensive. History doesn’t repeat but rhymes as they say.
Yep. It will be interesting to see if things move that way or not. AI scraping might get annoying but a lot of sites (eg e-commerce) have obvious incentives to allow AIs to scrape them for free (or even pay AI companies to prioritize them, like they already do with Google).
you're welcome to try it on our demo page (no signup needed), should handle everything just fine, yes we don't mention ips on the homepage
also, while it might seem expensive, we're cheaper than every other option out there, because there's no hidden credit multipliers. every single customer who uses us halved their bill + error rate
im not sure if grok is on here, i think that's an X thing
but if you were to ask me, we're more fully featured than cloudflare, and anecdotally a ton more reliable in terms of error rates. back when it was brand.dev, i actually tried to use cloudflare's apis and it was quite unreliable, so we built our own stack instead
Are you using residential proxies? How do you handle websites that don't want to be scraped.
EG if I start passing in Linkedin pages what is your expectation of the result that people would see per profile.
EDIT:
Congrats on the launch seriously hard work, just wanting to understand your scraping stance more. I've worked with a lot of tools on this, didn't mean for my initial comment to be adversarial.
hi, no worries at all, it came off perfectly fine
yes, we use residential proxies + all requests go are js-rendered, we maintain a caching layer which is 95%+ opted into by customers
it's all included in the credit price, great value compared to alternatives, the business model does rely on scale and our margin gets better the more requests we serve (esp infra cost for k8 + browser fleet)
to answer your e.g., yes public linkedin pages will work fine, anything behind a login we don't really support out of the box until we can figure out a safe way to do so, since that's where red lines are drawn
we step in whenever we see our service is hitting a website more than it should, this usually means reaching out to a customer for clarity on why they are not opting into the cache, we have alot of safeguards around fraud/spam and will let someone know if their request pattern looks like they're causing harm
I like the clarity, tone, and readability of your webpage. Also your FAQ is refreshing
> When Should I talk to sales? > Talk to sales if you need high-volume pricing beyond 2M credits/month, custom rate limits, SSO / SAML, SCIM provisioning, an uptime SLA, annual invoicing, an MSA / DPA, or a dedicated support channel. Reach us at hello@context.dev or through the contact page.
Would that this were the norm everywhere, rather than (say) a sales rep from Datadog scraping my phone number from who knows where to ask about my company's needs after I sign up for a free account on a whim :)
happy you noticed that, i put quite a bit of love into the ui, i've found engineers care alot about polish & feel of the webapps they use, even for an api product
i'm an engineer by trade, and always hated things like forcing a sales call, or having hidden credit multipliers, i tried to build this with the same ethos i like for my own dependencies (shoutout axiom.co)
I was using Context back when it was still Brand.dev. I found it to be a great product- one of those rare APIs that immediately made a problem I had disappear. Had it in production within an hour of signing up
Agents need clean/current context from the web, and this is the best way I’ve found to give it to them. The internet is clearly moving in this direction: companies are starting to realize their sites need to be legible to agents. Some are already adapting but many haven’t yet. Context feels like an important part of that transition
Yahia is a great builder. His pace of expansion has been impressive, excited to see where he takes Context.
oh wow, happy to see you here, we didn't post this anywhere so its nice to see a customer find us so quickly, thank you for the kind words.
brand data is a shockingly hard problem to get right
Unclear what difference exists against Firecrawl - their team has been shipping great features extremely quickly lately, and their core offerings have become really good.
I am interested in KnifeGeek though - looking for a good OTF (ultratech?)
There's alot of overlap at the moment, but we're diverging quite quickly.
If you want differences as of right now
- 1 credit = 1 scrape, no hidden credit multiplier - we have world class brand data - we're focused primarily on the infra use-case, rather than gtm & everything else - anecdotally, customers have seen their error rates drop quite dramatically
In general it's a huge space, firecrawl is a wonderful company, it's fun to compete with them, planning to add more things soon which should make the differences clear
Interesting - I'll be sure to benchmark it at some point. We've found the best results come from blending providers depending on the task anyways.
Thanks for the quick response - and always happy to see more competition in the space. Best of luck with future features!
please do, and if you see anything off let me know, we've yet to lose a single "bakeoff", i normally ask customers to just have cc/codex run it so its somewhat unbiased
on the 2nd point, most industries are not zero-sum, and many of customers use multiple data providers in any case, so agree with you there
thank you!
oh also on knifegeek, it'll be live again this weekend when i find a moment to fix the bug causing it to crash, i collect knives/watches so it was a super fun project to work on
Does this respect llms.txt and robots.txt, or have you found it more effective if agents see what humans see?
we try to read robots.txt, it's definitely more efficient if agents just see what humans see, we run a custom browser rendering stack
in terms of llms.txt, we're not primarily an AI product (although some features do use LLMs), and speaking to friends who run products it seems to be not very helpful, even though we have one as well, i didn't see it move the needle much
even my own coding agents don't look at llms.txt when looking at our own website, so unsure of whether that standard will survive the test of time
Awesome! Been great watching this product improve so quickly, can't wait for what's next :)
thank you, i try to ship updates often: https://docs.context.dev/changelog
love the design... congrats
will let my designer know, it's a dev-focused product, and we're all so finely tuned to avoid slop, so design & feel was really important
new frontier models do this already
im not sure the models do this, rather the tooling around them, however web search/extraction by the model providers is ridiculously expensive and quite slow, so going into production it makes sense to use a provider (like us)
So basically web scraping as a service with an API on top?
yes but we do alot more that may not be clear at first glance, things like brand data, and for scraping handling pdf, ocr, docx, ppt, xlsx automatically
shipping a bunch of new things soon which should make it clearer, but as of today yeah
Great, another thing I have to block server side. Reminds me of the image leech protections that had to be in place because bandwidth was expensive. History doesn’t repeat but rhymes as they say.
Maybe AI-service-blocklist-as-a-service could be a YC company.
Better business model would be some sort of micropayment setup - allow humans, but make scrapers pay a hundredth of a cent for access.
Is this similar? https://blog.cloudflare.com/monetization-gateway/
Yep. It will be interesting to see if things move that way or not. AI scraping might get annoying but a lot of sites (eg e-commerce) have obvious incentives to allow AIs to scrape them for free (or even pay AI companies to prioritize them, like they already do with Google).
Seems wildly expensive, furthermore not a single mention of "ip" on homepage? Not using rotating ip's, residential proxies?
AKA unusable for high value data.
you're welcome to try it on our demo page (no signup needed), should handle everything just fine, yes we don't mention ips on the homepage
also, while it might seem expensive, we're cheaper than every other option out there, because there's no hidden credit multipliers. every single customer who uses us halved their bill + error rate
Nice, @grok how does it compare to Cloudflare that also provides a REST endpoint for structured markdown data and screenshots?
im not sure if grok is on here, i think that's an X thing
but if you were to ask me, we're more fully featured than cloudflare, and anecdotally a ton more reliable in terms of error rates. back when it was brand.dev, i actually tried to use cloudflare's apis and it was quite unreliable, so we built our own stack instead