Every server with port 80/443 open has thousands of hits a day from random boxes looking for wordpress login pages. The only new thing is that they're pretending to be a different type of annoying bot. There's a new layer of sophistication and subterfuge, but it's the same junk traffic we've always dealt with.
Think about how many webmaster and business owners' egos are stroked by all the traffic they are getting, when in actuality they are often just serving thousands of bots.
Humans never visit but I have some websites with ancient cms's. When I got bored playing with them I download the html, change the extensions to php and replace the site with a static copy. People then tirelessly try to hack it since they have laundry lists of known vulnerabilities. (Now that I think about it I regret not making the admin area public for added nostalgia.)
Could be because of number of pages you have. At work we have several million public content pages, so a few badly behaving bots can already do a lot of damage.
For my personal website it’s 10x more bots but I barely notice because it’s a few pages.
I recently brought up a website on a never-before-seen .com domain. Within about 10 mins of bringing it up with a SSL certificate, Anthropic came knocking on the door requesting the front page. (Almost certainty due to them watching the Public Certificate Transparency logs)
I am always surprised that its considered legal.
IMO that's the same as going on the street door by door and checking if one is left open to steal everything inside the house...
That sort of vulnerability scanning is at best legally dubious, and almost certainly illegal under CFAA and similar state statues when there's clear criminal intent. That's why the 2022 DOJ guidance regarding non-prosecution good faith security research was such a big deal at the time.
> IMO that's the same as going on the street door by door and checking if one is left open to steal everything inside the house...
From experience: this does happen regularly in some neighborhoods of some cities in the US, and even that isn't always an enforcement priority. So lack of enforcement on the internet, where most the perpetrators probably aren't even in a jurisdiction with an extradition treaty, isn't exactly surprising.
This is so true. Every junior sysadmin I have trained over the years (including myself) has had a “are we being attacked?!” moment when tasked with WAF report analysis, monitoring fail2ban logs, etc.
Monitoring WAN traffic really gets the paranoia juices flowing.
Most servers with port 25565 open get hits from either Minecraft griefer bots, or from a bot that looks for that port and warns anyone on that server about the risks of leaving that port open. It doesn't take a huge scale operation to spam every IPv4 address in the world, there are only 2^32 of them, and even then many of those addresses are reserved.
People or their agents must be accidentally committing or publishing their repository level secrets and configs with enough regularity that it’s worth scanning.
There are a few novel ones but I’ve been seeing most of them in my logs for longer than generative AI has existed. This isn’t remotely new, the vector is just getting bigger.
On average about 100 (TCP) requests hit my home router per minute doing various probing and scanning. Lots of checking for the telnet port obviously. Sometimes you can see a swarm of entirely different IPs scanning the full port range (probing the ports one-by-one).
You'll see a lot of deepfield, censys-scanner, visionheight.com, shadowserver.io, etc., but also the usual suspects of Chinese or Russian IPs.
With OpenWRT I use something like this: `tcpdump -i pppoe-wan 'inbound and tcp[tcpflags] & (tcp-syn|tcp-ack) == tcp-syn'`, or alternatively `tcpdump -i pppoe-wan 'inbound and tcp[tcpflags] & (tcp-syn|tcp-ack) == tcp-syn and not port 44000'`, if we have some torrent client running (e.g. here at port 44000) which would mess up the result. I'm not sure it's the best way to handle this, but it's definitely enlightening what bounces off on the router.
The easiest way to deal with the usual suspects is to just block the entire countries network range(s). There really is no reason they should be connecting to your home router anyway, and you lose nothing from blocking them.
Sure their packets will still hit your router, but if they are dropped immediately at least you're not wasting a syn-ack on them.
I, temporarly, banned some ip range.
I didn't find a source for pinpointing countries; though I am interested. Could you point me to some sources which, deterministically, resolve to some countries? To my knowledge you can not reliably identify countries by ip since this would be dependent on DNS servers. Though I am just a application programmer!
Your router doesn't care about their DNS settings. IP addresses are very easy to tie back to countries. The reason they say it's not reliable is because it's trivial to spoof the country, but even so, a lot of attackers don't even bother. It's sort of like the Nigerian prince scam calls: if you're wise enough to block Russia, you're not worth their time.
Your firewall vendor should supply you with country lists, just select the known bad ones and drop their traffic. If you have a consumer grade router, you will probably have to configure the blocklists manually.
I have ubiquity UniFi for this reason (amongst others). OpenWRT is a good choice as well. Most home router software is such junk, might as well leave the door open..
Many of those user-agents listed are often faked. Look up which ASN owns their IP. If I block most VPS providers most of the faked bots vanish. There are still some running from residential and phones using hijacked code (readers that are not really just readers but really multipurpose proxies). On that note, do not trust the linked source code but rather decompile the live code your phone is running and have AI analyze it.
Is there an easy way to block any requests originating from VPS etc instead of residential/commercial IP from legitimate users ? I know cloudflare does a few things but I really want to figure out a way to block any request say at nginx or caddy (reverse proxy) from reaching origin servers if they are not from an IP that is not a VPS etc.
There are several methods. [1] The most aggressive method-02 and method-03 on my document will block VPS and some data-centers but that also means it will block some legit users that are on a VPN. Most VPNs transit a data-center. If experimenting with these methods use a test server that you do not care about and set up a dummy site and ask people in your circle of friends to test it. I have to step away for a bit but if you have questions I will try to answer.
I second this. When I have tested blocking VPS/data-centers to my silly blog there were about a dozen people on HN [1] that could not view my site out of the roughly ~17,000 (not counting bots) that could. It's not a big number but those are real people and they count.
I am going to move full blocking to a test node that people can play with but I have to finish working with Claude to revise someones repo is is no longer maintained because one does not simply put an anonymous chan board on the great wide open internets without some critical thinking.
Yes but it’s not cheap. Maxmind and ipinfo etc sell a tier that tells you this information, then you can 403 based on it.
But the price is nuts like $40,000 a year.
I work for IPinfo. We offer IPinfo Lite for free. With a little bit of time in identifying the ASNs, you can implement a decent way to block a good number of bots fairly easily using the free data alone.
You can block entire ASNs. If you are frustrated with bots, blocking Tencent's entire IP address space would have very few downsides.
If you have fail2ban or NGINX logs, you can use our CLI to summarize those IPs and identify the ASNs you want to block. But before you block entire ASNs, make sure they are not classified as "ISP" type. For that, visit our website's ASN page first.
Yeah, that's exactly what these visits are: faked user agents that fail IP verification or Web Bot Auth. What's interesting is the surge across so many websites in the last week.
There are many possibilities but one of them could be some new vuln was released and they are looking for it. That would require looking at the URL's they are requesting. Botters run their own purpose built campaigns. Do you also have a summary of URL's requested by unique counts?
Same for the origin IP address. The fiber leaving your country is tapped, and those people can inject packets with any origin IP that they want. Your ISP has no way to check if their peer actually received a certain packet from a certain country or not.
From a technical perspective, all this "china/russia" attribution is built on a quite shaky foundation. As a sysadmin you'd never know if it would be the British crown attacking your European company instead.
Not minimizing nation state cyber crime here, but the packet goes through many hands with different incentives.
Problem here is there are not single fibers attaching (most) countries, but a bunch of them. If you control both the ingress and egress for some particular users it's possible, but if you don't then your probing packing may end up back in China with a lot of evidence of backscatter.
Fake Googlebot visits are #1 in website logs I've been working on. At the beginning I was fighting with them using Cloudflare ASN block rules or their managed Bot Fight mode but it appeared to be not only pointless, but also harmful for my websites.
Bot Fight mode randomly started blocking real Bing / Google / OpenAI crawlers what wasted crawling budget and discouraged crawlers to revisit updated pages.
Sometimes it's better to not fight with bots actively but harden environment and only react for the worst offenders.
Just in case any of the authors read HN, I'm getting a pretty crazy rendering bug on this page, where a bunch of the contents are redrawing up and down by a few pixels. It seemed to go away with resizing the width a few times, but I didn't look into it too hard. My page width was probably small on first draw. Incredibly distracting though and hard to read with the text moving. Using latest chrome, and it occurred on more than one page refresh. I didn't dig in beyond that though.
I'm seeing the same thing on Firefox on Linux. It almost looks like the page scroll is jiggling up and down a tiny amount constantly when it's supposed to be stationary.
Why would you voluntarily pretend to be a AI bot, when those have already a much higher chance of being blocked? Seems holly unproductive.
Best hypothesis I can come up with is to somehow make the AI companies look bad, but they seem to be doing an excellent job at that themselves already by scraping everyone hundreds of times per hour over and over.
Because businesses dont want them blocked, that would be a very stupid thing for most of them to do given its becoming a vital traffic source now that people are using chatbots instead of google.
Definitely, the point here though is there is a stat sig surge in the chart in the last week, across thousands of websites. At least a surge in this particular spoofing pattern.
This is a random sample of completely unrelated websites, which indicates that the total scale is much larger. This is not saying that it is difficult to make thousands of requests.
Is this your company? If it is, your cheerleading makes you very hard to trust. If it’s not, I’m sure that everyone gets the point - you adore everything about this research and can’t see any possible problems.
I think this is more of a "if you left your AI tools exposed someone is looking for them" change. Hacking someone else's agents sounds like a great way to spend less on your own tokens.
This kind of stuff is getting pretty wild, even for using something like Cloudflare it seems like a good idea to have another layer behind it that's non-cloudflare for when vulnerabilities are discovered.
Every server with port 80/443 open has thousands of hits a day from random boxes looking for wordpress login pages. The only new thing is that they're pretending to be a different type of annoying bot. There's a new layer of sophistication and subterfuge, but it's the same junk traffic we've always dealt with.
Think about how many webmaster and business owners' egos are stroked by all the traffic they are getting, when in actuality they are often just serving thousands of bots.
I was insulted recently reading about the apparently thousands of hits per second the rest of you guys are getting. Even the bots are shunning me!
Humans never visit but I have some websites with ancient cms's. When I got bored playing with them I download the html, change the extensions to php and replace the site with a static copy. People then tirelessly try to hack it since they have laundry lists of known vulnerabilities. (Now that I think about it I regret not making the admin area public for added nostalgia.)
Could be because of number of pages you have. At work we have several million public content pages, so a few badly behaving bots can already do a lot of damage.
For my personal website it’s 10x more bots but I barely notice because it’s a few pages.
Get a letsencrypt cert, I always get a massive wave of vulnerability probers after renewing.
Because your certificate shows up in the global chain, which triggers all kinds of automated things including bots
I don't even have a domain name and I'm getting lots of hits.
I recently brought up a website on a never-before-seen .com domain. Within about 10 mins of bringing it up with a SSL certificate, Anthropic came knocking on the door requesting the front page. (Almost certainty due to them watching the Public Certificate Transparency logs)
Old and busted: serverless
New hotness: DNS-less
imagine having a KPI tied to cash bonuses based on that...
I am always surprised that its considered legal. IMO that's the same as going on the street door by door and checking if one is left open to steal everything inside the house...
That sort of vulnerability scanning is at best legally dubious, and almost certainly illegal under CFAA and similar state statues when there's clear criminal intent. That's why the 2022 DOJ guidance regarding non-prosecution good faith security research was such a big deal at the time.
> IMO that's the same as going on the street door by door and checking if one is left open to steal everything inside the house...
From experience: this does happen regularly in some neighborhoods of some cities in the US, and even that isn't always an enforcement priority. So lack of enforcement on the internet, where most the perpetrators probably aren't even in a jurisdiction with an extradition treaty, isn't exactly surprising.
Unauthorized access is not legal, it's just not enforced in the slightest.
Illegal and actively enforced are different things. Report it to the hosting company originating the abuse and they usually don't care.
Opening port 80 and realizing the world is an anarchic warzone is a canonic rite of initiation for otherwise innocent backend devs and sysadmins.
This is so true. Every junior sysadmin I have trained over the years (including myself) has had a “are we being attacked?!” moment when tasked with WAF report analysis, monitoring fail2ban logs, etc.
Monitoring WAN traffic really gets the paranoia juices flowing.
Most servers with port 25565 open get hits from either Minecraft griefer bots, or from a bot that looks for that port and warns anyone on that server about the risks of leaving that port open. It doesn't take a huge scale operation to spam every IPv4 address in the world, there are only 2^32 of them, and even then many of those addresses are reserved.
Another interesting thing here is the paths they're targeting, many are for newish AI coding tools
People or their agents must be accidentally committing or publishing their repository level secrets and configs with enough regularity that it’s worth scanning.
Totally. I'm sure this campaign was inspired by sloppy vibe coding
There are a few novel ones but I’ve been seeing most of them in my logs for longer than generative AI has existed. This isn’t remotely new, the vector is just getting bigger.
On average about 100 (TCP) requests hit my home router per minute doing various probing and scanning. Lots of checking for the telnet port obviously. Sometimes you can see a swarm of entirely different IPs scanning the full port range (probing the ports one-by-one).
You'll see a lot of deepfield, censys-scanner, visionheight.com, shadowserver.io, etc., but also the usual suspects of Chinese or Russian IPs.
With OpenWRT I use something like this: `tcpdump -i pppoe-wan 'inbound and tcp[tcpflags] & (tcp-syn|tcp-ack) == tcp-syn'`, or alternatively `tcpdump -i pppoe-wan 'inbound and tcp[tcpflags] & (tcp-syn|tcp-ack) == tcp-syn and not port 44000'`, if we have some torrent client running (e.g. here at port 44000) which would mess up the result. I'm not sure it's the best way to handle this, but it's definitely enlightening what bounces off on the router.
The easiest way to deal with the usual suspects is to just block the entire countries network range(s). There really is no reason they should be connecting to your home router anyway, and you lose nothing from blocking them.
Sure their packets will still hit your router, but if they are dropped immediately at least you're not wasting a syn-ack on them.
I, temporarly, banned some ip range. I didn't find a source for pinpointing countries; though I am interested. Could you point me to some sources which, deterministically, resolve to some countries? To my knowledge you can not reliably identify countries by ip since this would be dependent on DNS servers. Though I am just a application programmer!
Thanks in advance.
Your router doesn't care about their DNS settings. IP addresses are very easy to tie back to countries. The reason they say it's not reliable is because it's trivial to spoof the country, but even so, a lot of attackers don't even bother. It's sort of like the Nigerian prince scam calls: if you're wise enough to block Russia, you're not worth their time.
Your firewall vendor should supply you with country lists, just select the known bad ones and drop their traffic. If you have a consumer grade router, you will probably have to configure the blocklists manually.
I wonder if adding the US is now sensible.
Getting it directly from IANA is always the best approach.
Here is a "simplified" version in various formats.
https://github.com/HotCakeX/Official-IANA-IP-blocks
Routers got such a thing build in nowadays, just gotta enable it (not the ones from your ISP of course)
I have ubiquity UniFi for this reason (amongst others). OpenWRT is a good choice as well. Most home router software is such junk, might as well leave the door open..
This is one reason I don’t mind that I’m behind CGNAT.
That's the only advantage of CGNAT I can think of. :D
(You still could get poked from the other users' hosts behind the ISP's NAT, of course.)
I can corroborate visionheight and shadowserver from my firewall logs.
Many of those user-agents listed are often faked. Look up which ASN owns their IP. If I block most VPS providers most of the faked bots vanish. There are still some running from residential and phones using hijacked code (readers that are not really just readers but really multipurpose proxies). On that note, do not trust the linked source code but rather decompile the live code your phone is running and have AI analyze it.
Is there an easy way to block any requests originating from VPS etc instead of residential/commercial IP from legitimate users ? I know cloudflare does a few things but I really want to figure out a way to block any request say at nginx or caddy (reverse proxy) from reaching origin servers if they are not from an IP that is not a VPS etc.
I run an open source honeypot that collects these botnet scans and produces blocklists.
Blocklist download and configuration: https://knock-knock.net/blocklist
Honeypot dashboard, where you can see attempted attacks in realtime: http://knock-knock.net
API: http://knock-knock.net/api
There are several methods. [1] The most aggressive method-02 and method-03 on my document will block VPS and some data-centers but that also means it will block some legit users that are on a VPN. Most VPNs transit a data-center. If experimenting with these methods use a test server that you do not care about and set up a dummy site and ask people in your circle of friends to test it. I have to step away for a bit but if you have questions I will try to answer.
[1] - https://nochan.net/b/Internet-Crap/20260606-How-To-Block-Som...
Your site does not currently seem to be reachable / responsive when I try to reach it from a US Comcast IPv4 address - you are not advertising IPv6.
Edit / Update: It was Apple's Private browsing mode that causes it not to respond. I can now see it when this is disabled.
Per your link...
> block http 1.1, real users only use 2.0
Chrome on android and Firefox on linux both appear to use 1.1 still...
> /commercial IP from legitimate users
No, because legitimate users do not just use residential and "commercial" IPs. Like me, right now
You are the 0.001%
Much more than 0.001% of people care about their privacy or (the larger portion) do not have unfiltered access to the internet.
I second this. When I have tested blocking VPS/data-centers to my silly blog there were about a dozen people on HN [1] that could not view my site out of the roughly ~17,000 (not counting bots) that could. It's not a big number but those are real people and they count.
I am going to move full blocking to a test node that people can play with but I have to finish working with Claude to revise someones repo is is no longer maintained because one does not simply put an anonymous chan board on the great wide open internets without some critical thinking.
[1] - https://news.ycombinator.com/item?id=49060945
Yes but it’s not cheap. Maxmind and ipinfo etc sell a tier that tells you this information, then you can 403 based on it. But the price is nuts like $40,000 a year.
Open source works just fine for this - take a look at geoacumen-country
https://github.com/geoacumen/geoacumen-country
I work for IPinfo. We offer IPinfo Lite for free. With a little bit of time in identifying the ASNs, you can implement a decent way to block a good number of bots fairly easily using the free data alone.
Good to know, thank you. Would you do this by fully blocking particular ASNs? Or something more granular?
You can block entire ASNs. If you are frustrated with bots, blocking Tencent's entire IP address space would have very few downsides.
If you have fail2ban or NGINX logs, you can use our CLI to summarize those IPs and identify the ASNs you want to block. But before you block entire ASNs, make sure they are not classified as "ISP" type. For that, visit our website's ASN page first.
I have quite a few community posts around this approach. https://community.ipinfo.io/
If you have raw logs, you can send them to me as well, and I can review them and provide some guidance.
You don't need to spend anywhere near $40k a year to get that info... You don't even need to spend $1
What's your strategy?
Focsec.com IP database for offline use (datacenters, VPNs, proxies, bots) runs around $1k/month for internal-only use.
Yeah, that's exactly what these visits are: faked user agents that fail IP verification or Web Bot Auth. What's interesting is the surge across so many websites in the last week.
There are many possibilities but one of them could be some new vuln was released and they are looking for it. That would require looking at the URL's they are requesting. Botters run their own purpose built campaigns. Do you also have a summary of URL's requested by unique counts?
Looks like many of the paths relate to AI coding tools. There are some examples below the chart
You keep repeating this about a small minority of the tools that were posted.
Same for the origin IP address. The fiber leaving your country is tapped, and those people can inject packets with any origin IP that they want. Your ISP has no way to check if their peer actually received a certain packet from a certain country or not.
From a technical perspective, all this "china/russia" attribution is built on a quite shaky foundation. As a sysadmin you'd never know if it would be the British crown attacking your European company instead.
Not minimizing nation state cyber crime here, but the packet goes through many hands with different incentives.
Problem here is there are not single fibers attaching (most) countries, but a bunch of them. If you control both the ingress and egress for some particular users it's possible, but if you don't then your probing packing may end up back in China with a lot of evidence of backscatter.
I'd be surprised if there is a single route from EU to non-EU countries which does not pass through British control.
Fake Googlebot visits are #1 in website logs I've been working on. At the beginning I was fighting with them using Cloudflare ASN block rules or their managed Bot Fight mode but it appeared to be not only pointless, but also harmful for my websites. Bot Fight mode randomly started blocking real Bing / Google / OpenAI crawlers what wasted crawling budget and discouraged crawlers to revisit updated pages.
Sometimes it's better to not fight with bots actively but harden environment and only react for the worst offenders.
How about them apples... ai bots use faked browser user agents, so people start pretending to be ai instead...
(Insert spider man meme)
Just in case any of the authors read HN, I'm getting a pretty crazy rendering bug on this page, where a bunch of the contents are redrawing up and down by a few pixels. It seemed to go away with resizing the width a few times, but I didn't look into it too hard. My page width was probably small on first draw. Incredibly distracting though and hard to read with the text moving. Using latest chrome, and it occurred on more than one page refresh. I didn't dig in beyond that though.
I'm seeing the same thing on Firefox on Linux. It almost looks like the page scroll is jiggling up and down a tiny amount constantly when it's supposed to be stationary.
I recently blogged about some Cloudflare Workers I developed to combat this type of traffic: https://code.backwater.systems/blog/#2026-06-29T23:40:00.000...
Using a normal page per blog entry would go a long way to making your site more indexable, readable, shareable and seo-able. (Good article btw).
Why would you voluntarily pretend to be a AI bot, when those have already a much higher chance of being blocked? Seems holly unproductive.
Best hypothesis I can come up with is to somehow make the AI companies look bad, but they seem to be doing an excellent job at that themselves already by scraping everyone hundreds of times per hour over and over.
Because businesses dont want them blocked, that would be a very stupid thing for most of them to do given its becoming a vital traffic source now that people are using chatbots instead of google.
Most websites don't have an incentive to block AI bots to their main sites. Think businesses, government and community websites, nonprofits, etc.
Interesting thought: what if the idea of an open internet is over.
What if we're now moving into a world of strictly KYC. The same way "The Facebook" generated massive revenue by creating a KYC world.
Man, this "someone" guy sounds like a real jerk!
Someone is always running mass vulnerability scans. That's a "water is wet" state of the Internet.
Definitely, the point here though is there is a stat sig surge in the chart in the last week, across thousands of websites. At least a surge in this particular spoofing pattern.
It's still not really anything special. Thousands isn't even large scale.
Any random bozo can trigger that.
This is a random sample of completely unrelated websites, which indicates that the total scale is much larger. This is not saying that it is difficult to make thousands of requests.
Is this your company? If it is, your cheerleading makes you very hard to trust. If it’s not, I’m sure that everyone gets the point - you adore everything about this research and can’t see any possible problems.
Not asking for trust, just sharing the data/math
You forgot the "Yes, that's my company" part in your reply (https://ghking.co)
That doesn't change the data/math brother
It's still good etiquette to disclose affiliation in online discussions.
Fair enough
I think this is more of a "if you left your AI tools exposed someone is looking for them" change. Hacking someone else's agents sounds like a great way to spend less on your own tokens.
Nah these are just the typical vuln scans that try all kinda basic shit, looking for anything commonly used AI tools included.
This kind of stuff is getting pretty wild, even for using something like Cloudflare it seems like a good idea to have another layer behind it that's non-cloudflare for when vulnerabilities are discovered.
What, like some kind of firewall?