LLM bots are the easiest to spot. Rate limit them to 16kb/s and 1 request every 5 seconds. Give them a friendly multilingual WCAG status 429 page that says to slow down or status 444 and drop them into nftables ipsets if they continue to be aggressive. Ask your favorite LLM how to set this up. If your LLM gets session paused, ask Gemini or Grok. Tell it if you are behind a CDN. Enable logging of ssl curves, ciphers, etc... Show your LLM some examples from your access logs.
LLM bots make only a part of traffic and really easy to set up (and I think this will eventually throttled by OAI or Anthropic themselves).
But there are bad actors - bots that try every possible urls and traversals and cross scripting and so on… I see it in our access logs. This cannot be solved otherwise than some greylisting/blacklisting on ISP level. E.g. this is so huge that we have started to block all Chinese inbound traffic. I am sorry for that because Deepseek and other good actors are probably blocked as well but IP rotation made it impossible to filter bad ones only… (and I suppose that DS uses EU or US addresses as well so they shouldn’t be blocked in full).
eventually throttled by OAI or Anthropic themselves
Indeed, they use an edge SSL Bump proxy that is already bandwidth limited and has to speak real HTTPS.
I do not focus too much on IP addresses but rather the behavior that they can not hide. LLM's can help you spot this behavior. Good LLM vs. Bad LLM. Gemini knows a few tricks but you have to coax it out of them. It seems to not want to overload people with information. That said there are some good IP lists that can be used in blackhole routing tables, nginx geo tables, etc...
I get tens of thousands of IPs making 1 or 2 requests against pages on The Consensus (job search, code viewing, etc.). They're not super expensive pages. But I don't want to deal with the bot traffic (mostly because it messes up view counts I use to sell sponsorships). But then if I require an account (even a free one), valid readership is also going to drop with the new friction.
I could build something more complex to discover the patterns, or use Turnstile maybe. I'm not really sure at the moment.
I mind a public access website for a government entity and this is exactly the pattern I see. US residential IP make a couple of requests then never come back.
The bots may have layer 7 signatures that make them "easy" to block, but I'm dealing with COTS software with no ability to make changes to the stack. Even if I could by the time they're blocked the work is done and they're never coming back anyway.
I don't think I'm seeing the LLM bots in the scraping on this particular site. This site's data is highly sought-after by data brokers, legal services advertising, debt collectors, etc. I tend to think I'm seeing homespun scrapers. This site has been up for 20+ years and the scrapers seem to be well-adapted to using the HMTL as an ad-hoc API for the data. The difference in the last few years has been the availability of vast numbers of residential proxies to mask the traffic.
Bots give all the information required to block or rate limit them including those that try to look like a real browser (some things they just can't hide, even residential proxies via behavior.). Ask the LLM how to change your log format to gather all the fields your web server can see. Share the resulting updated logs with your favorite LLM. Ask it for generalized rules to block the bots whilst permitting real browsers. A tiny percentage of bots will still get through but the load becomes much more manageable. Tell your LLM you also wish to spot fake chrome via CH and bots via ssl curves. No need for JA3. Nginx can expose the important aspects of GREASE.
I know you can't be super specific because you'd just be enabling the companies who get around this (who, funnily, I also index on The Consensus), but I have spent a few rounds on this over the months and haven't yet been able to cut them out.
Some of the things that seemed like they'd cut out lazier bots/scripters involved browser fingerprinting that I didn't want to yet have to deal with disclosures or approval for. But also even some of these browser fingerprinting things were disable-able via flags when you start up Chrome or whatever.
I think what would work overall is to track all requests for a pattern of page access and start blocking other systems that are part of that pattern. But that also wouldn't account for like... landing on the front page of HN, which happens not infrequently.
All of this is aside from per-IP or per-user proof of work. Whenever someone lands on a more expensive page I make the browser do proof of work and store it in a cookie, unique to the IP or user. (Signed in users don't have to do proof of work). This is a perfect solution except for the IP rotation I'm talking about above.
edit: ah also, I don't do SSL termination so can't spot the SSL configuration. But Cloudflare does. And all of this is even after Cloudflare paid bot traffic systems are in place.
edit 2:
> tiny percentage of bots will still get through but the load becomes much more manageable.
The load is not the problem. It's only at most a few bot hits per second. I'm just having trouble removing bot attribution from view counts.
Aside from the things I suggested asking LLM's (they will take one down some fun rabbit holes) there are more crude methods such as forcing brotli or zstd on specific URLs accepting that older versions of Safari pre-2026 will get gibberish if forcing zstd. It's rude and paternal but botters can be rude and childish so these are the games we play.
I love sharing methods with the botters so they step up their game, thus forcing me to step up my game. Everything I use does not require javascript, cookies or authentication. I save authentication as a last resort which means I lost that round.
Botters are welcome and encouraged to test their setup on my silly hobby sites. Here's one of them: [1]
P.S. These are the simplest crude methods for the obvious bots, let your LLM to help spot the fakers. pre-2026 Safari will not work. Do not visit image boards with unpatched browsers.
What I still don't understand is who is doing it? I assume it's not the big companies like OpenAI or Google or Anthropic, right? So there are a bunch of small things out there doing the scraping and then they sell it to the big places? Or are these just smaller AI models doing the scraping and using it for themselves? Is there an underground ecosystem where people buy and sell this stuff? I understand why it's valuable to the AI companies I'm curious about how all this work, the economics of it all.
Can't speak for everyone. I'm very gently scraping a lot of libraries for public domain periodical scans. The scans often exist nowhere else, or not in the same quality. Sometimes I find a certain library has perhaps the only known remaining intact copy of a periodical or a specific page.
The problem with the libraries is that all these scans are behind walls, so while the texts are public domain nobody can easily access them. I'm trying to liberate them and re-upload them outside the walls.
Until recently a lot of libraries would allow you to remotely sign up for a library card, but in the last 12 months almost all of them have reverted to having to bring ID in person to the library to validate your account, which puts the brakes on things.
We're currently in the middle of an AI goldrush. There are probably hundreds of companies trying to do model pretraining. Anthropic and OpenAI and so on probably care about optics and implementing their crawlers properly. But there will be many firms few of us have heard of. I would guess some of them are Chinese, for example.
It's a canonical DOI on what was a download server. It says exactly which file is being requested. (I chose a random recent one.) At the very most, they should serve a static page with a bibtex entry, and a few other metadata fields, and a link to the PDF and whatever other formats are available.
Instead, we get piles of captchas, login-gate spam about how they're "open access" now and have some initiative to be LLM friendly.
Getting back to the problem of people doing stuff that requires access to library materials: What sort of server-side cost multiplier do you suppose this ACM page has vs. just serving a few S3-compatible URLs that point to a few mirrors? (Or, got forbid, Anna's Archive / libgen, since ACM is open access, and those sites would be doing nothing wrong by mirroring ACM content.)
The problem is unintentional DOS due to traffic can constitute a felony under the CFAA; the case in question being Pulte Homes, Inc. v. Laborers' International Union 2011.
After Pulte fired an employee represented by the union, LIUNA urged members to call and send email to the company, expressing their opinions. As a result of the increased traffic, the company's email system crashed. The Sixth Circuit ruled that the LIUNA's instruction to call and email "intentionally caused damage," reversing the lower court's decision.
Now thankfully we have Sandvig v. Barr, 2020 where the DC Federal Court ruled that the CFAA does not criminalize the violation of a website's terms of service, but imo all bets are off in terms of interpretation of the law in the wild west of AI.
In what world is 11k searches a lot? The story here is that AI bots are exposing University Libraries bad setups. I haven't heard any institution here in northern europe having any trouble.
Agreed, this just reads like the universities have no idea how to actually serve data effectively. Just bad architecture and lazy or incompetent administrators.
"Open Access" means supporting users. Now your users are also bots, and the people that deploy those bots. You need to be able to serve those users too.
LLM bots are the easiest to spot. Rate limit them to 16kb/s and 1 request every 5 seconds. Give them a friendly multilingual WCAG status 429 page that says to slow down or status 444 and drop them into nftables ipsets if they continue to be aggressive. Ask your favorite LLM how to set this up. If your LLM gets session paused, ask Gemini or Grok. Tell it if you are behind a CDN. Enable logging of ssl curves, ciphers, etc... Show your LLM some examples from your access logs.
LLM bots make only a part of traffic and really easy to set up (and I think this will eventually throttled by OAI or Anthropic themselves).
But there are bad actors - bots that try every possible urls and traversals and cross scripting and so on… I see it in our access logs. This cannot be solved otherwise than some greylisting/blacklisting on ISP level. E.g. this is so huge that we have started to block all Chinese inbound traffic. I am sorry for that because Deepseek and other good actors are probably blocked as well but IP rotation made it impossible to filter bad ones only… (and I suppose that DS uses EU or US addresses as well so they shouldn’t be blocked in full).
eventually throttled by OAI or Anthropic themselves
Indeed, they use an edge SSL Bump proxy that is already bandwidth limited and has to speak real HTTPS.
I do not focus too much on IP addresses but rather the behavior that they can not hide. LLM's can help you spot this behavior. Good LLM vs. Bad LLM. Gemini knows a few tricks but you have to coax it out of them. It seems to not want to overload people with information. That said there are some good IP lists that can be used in blackhole routing tables, nginx geo tables, etc...
I get tens of thousands of IPs making 1 or 2 requests against pages on The Consensus (job search, code viewing, etc.). They're not super expensive pages. But I don't want to deal with the bot traffic (mostly because it messes up view counts I use to sell sponsorships). But then if I require an account (even a free one), valid readership is also going to drop with the new friction.
I could build something more complex to discover the patterns, or use Turnstile maybe. I'm not really sure at the moment.
I mind a public access website for a government entity and this is exactly the pattern I see. US residential IP make a couple of requests then never come back.
The bots may have layer 7 signatures that make them "easy" to block, but I'm dealing with COTS software with no ability to make changes to the stack. Even if I could by the time they're blocked the work is done and they're never coming back anyway.
I don't think I'm seeing the LLM bots in the scraping on this particular site. This site's data is highly sought-after by data brokers, legal services advertising, debt collectors, etc. I tend to think I'm seeing homespun scrapers. This site has been up for 20+ years and the scrapers seem to be well-adapted to using the HMTL as an ad-hoc API for the data. The difference in the last few years has been the availability of vast numbers of residential proxies to mask the traffic.
Bots give all the information required to block or rate limit them including those that try to look like a real browser (some things they just can't hide, even residential proxies via behavior.). Ask the LLM how to change your log format to gather all the fields your web server can see. Share the resulting updated logs with your favorite LLM. Ask it for generalized rules to block the bots whilst permitting real browsers. A tiny percentage of bots will still get through but the load becomes much more manageable. Tell your LLM you also wish to spot fake chrome via CH and bots via ssl curves. No need for JA3. Nginx can expose the important aspects of GREASE.
I know you can't be super specific because you'd just be enabling the companies who get around this (who, funnily, I also index on The Consensus), but I have spent a few rounds on this over the months and haven't yet been able to cut them out.
Some of the things that seemed like they'd cut out lazier bots/scripters involved browser fingerprinting that I didn't want to yet have to deal with disclosures or approval for. But also even some of these browser fingerprinting things were disable-able via flags when you start up Chrome or whatever.
I think what would work overall is to track all requests for a pattern of page access and start blocking other systems that are part of that pattern. But that also wouldn't account for like... landing on the front page of HN, which happens not infrequently.
All of this is aside from per-IP or per-user proof of work. Whenever someone lands on a more expensive page I make the browser do proof of work and store it in a cookie, unique to the IP or user. (Signed in users don't have to do proof of work). This is a perfect solution except for the IP rotation I'm talking about above.
edit: ah also, I don't do SSL termination so can't spot the SSL configuration. But Cloudflare does. And all of this is even after Cloudflare paid bot traffic systems are in place.
edit 2:
> tiny percentage of bots will still get through but the load becomes much more manageable.
The load is not the problem. It's only at most a few bot hits per second. I'm just having trouble removing bot attribution from view counts.
Aside from the things I suggested asking LLM's (they will take one down some fun rabbit holes) there are more crude methods such as forcing brotli or zstd on specific URLs accepting that older versions of Safari pre-2026 will get gibberish if forcing zstd. It's rude and paternal but botters can be rude and childish so these are the games we play.
I love sharing methods with the botters so they step up their game, thus forcing me to step up my game. Everything I use does not require javascript, cookies or authentication. I save authentication as a last resort which means I lost that round.
Botters are welcome and encouraged to test their setup on my silly hobby sites. Here's one of them: [1]
[1] - https://altchan.org/sh/33
P.S. These are the simplest crude methods for the obvious bots, let your LLM to help spot the fakers. pre-2026 Safari will not work. Do not visit image boards with unpatched browsers.
seems like open web is dead which is sad
i heard hcaptcha is good
Static web pages might be back for a bit while this gets figured out
What I still don't understand is who is doing it? I assume it's not the big companies like OpenAI or Google or Anthropic, right? So there are a bunch of small things out there doing the scraping and then they sell it to the big places? Or are these just smaller AI models doing the scraping and using it for themselves? Is there an underground ecosystem where people buy and sell this stuff? I understand why it's valuable to the AI companies I'm curious about how all this work, the economics of it all.
Can't speak for everyone. I'm very gently scraping a lot of libraries for public domain periodical scans. The scans often exist nowhere else, or not in the same quality. Sometimes I find a certain library has perhaps the only known remaining intact copy of a periodical or a specific page.
The problem with the libraries is that all these scans are behind walls, so while the texts are public domain nobody can easily access them. I'm trying to liberate them and re-upload them outside the walls.
Until recently a lot of libraries would allow you to remotely sign up for a library card, but in the last 12 months almost all of them have reverted to having to bring ID in person to the library to validate your account, which puts the brakes on things.
We're currently in the middle of an AI goldrush. There are probably hundreds of companies trying to do model pretraining. Anthropic and OpenAI and so on probably care about optics and implementing their crawlers properly. But there will be many firms few of us have heard of. I would guess some of them are Chinese, for example.
11 THOUSAND searches in one day? What's the problem?
If anything, libraries should be becoming more AI friendly, not less. Instead, we get this sort of bullshit from the ACM:
https://dl.acm.org/doi/10.1145/3832783.3834511
It's a canonical DOI on what was a download server. It says exactly which file is being requested. (I chose a random recent one.) At the very most, they should serve a static page with a bibtex entry, and a few other metadata fields, and a link to the PDF and whatever other formats are available.
Instead, we get piles of captchas, login-gate spam about how they're "open access" now and have some initiative to be LLM friendly.
Getting back to the problem of people doing stuff that requires access to library materials: What sort of server-side cost multiplier do you suppose this ACM page has vs. just serving a few S3-compatible URLs that point to a few mirrors? (Or, got forbid, Anna's Archive / libgen, since ACM is open access, and those sites would be doing nothing wrong by mirroring ACM content.)
The problem is unintentional DOS due to traffic can constitute a felony under the CFAA; the case in question being Pulte Homes, Inc. v. Laborers' International Union 2011.
After Pulte fired an employee represented by the union, LIUNA urged members to call and send email to the company, expressing their opinions. As a result of the increased traffic, the company's email system crashed. The Sixth Circuit ruled that the LIUNA's instruction to call and email "intentionally caused damage," reversing the lower court's decision.
Now thankfully we have Sandvig v. Barr, 2020 where the DC Federal Court ruled that the CFAA does not criminalize the violation of a website's terms of service, but imo all bets are off in terms of interpretation of the law in the wild west of AI.
In what world is 11k searches a lot? The story here is that AI bots are exposing University Libraries bad setups. I haven't heard any institution here in northern europe having any trouble.
Agreed, this just reads like the universities have no idea how to actually serve data effectively. Just bad architecture and lazy or incompetent administrators.
It was 11,000 searches on just one topic (Finnish music). Not sure how many topics.
Get with the times, university libraries.
"Open Access" means supporting users. Now your users are also bots, and the people that deploy those bots. You need to be able to serve those users too.
I wonder if Internet 2: The sequel is infested as well.
https://en.wikipedia.org/wiki/Internet2