I will be using Pydoll for the following legitimate use case: a franchisee is given access to their data as controlled by the franchise through a web site. The franchisee uses browser automation to retrieve its data but now the franchise has deployed a WAF that blocks Chrome webdriver. This is not a public web site and the data is not public so it frustrates the franchisee because it just wants its data which is paid for by its franchisee fees.
Well it can be abused of course, but capthas are used abusively as well, so I would say it's fair game.
Lots of use cases for scraping are not DoS or information stealing, but mere automation.
Proof of work should be used in these cases, it deters massive scraping abuse by making it too expensive at scale, while allowing legitimate small scale automation.
Then you need to level up & have defense in depth instead of relying on security through obscurity.
On the public internet, web clients are user agents, and not all users are benign. This is an arms race: asking the other side to unilaterally disarm is unlikely to work, so you change what you can control.
This is a defeatist argument. That it's technically possible to abuse things doesn't mean the responsibility needs to fall on the defending party, especially not when that is brought up in response to asking someone to reflect on possibilities for abuse - by that point it starts looking a lot more like a "well you'll just have to deal with it" argument that socially defends the abusers, and a lot less like genuine advice.
No side is getting defeated any time soon. I've been involved in skirmishes on both sides of scraping, and as I said, it's an arms race with no clear winner. To be clear, not all scraping is abuse.
The number of people who'll start scraping because a new tool exists is a negligible (i.e. <0.001 of scraping). Scraping itself is not hard at all, a noob who can copy-paste code from the web or vibe-code a client that can scrape 80-90% of the web. A motivated junior can raise that to maybe 98/99% of the Internet using nothing but libraries that existed before this tool.
> especially not when that is brought up in response to asking someone to reflect on possibilities for abuse
Sir/ma'am - this is hacker news, granted, it's aspirational, but still, hiding information is not the way. As someone who's familiar with the arts, there is nothing new or groundbreaking in this engine. Further, is no inherent moral high ground for the "defenders" either: many anti-scraping methods rely on client fingerprinting and other privacy-destroying techniques, so it's not the existence of the tool or technique, but how one uses it.
>... "well you'll just have to deal with it" argument that socially defends the abusers
The abuse predate the tool, so wishing the tool away is unlikely to help. Scraping is a numbers game on both sides, the best one vam hope for is to defeat the vast majority of the average adversaries, but a few fall through the cracks, the point is to outrun your fellow hiker, not the bear. However, should you encounter an adversery who has specifically chosen you as a target, then victory is far from assured. The usual result is a drawn-out stalemate. Most well-behaved scrapers are left alone.
I am also wondering about this, and in case you have a chef's knife in your kitchen, I would also like to hear if you have any comment on how that may be abused.
Every knife can bypass stabproof vests with enough force, but that's beside the point. The knife is designed to bypass skin and flesh, hence the potential for abuse. You go down that path and you end up with the insane knife laws Western Europe has where just carrying a swiss army knife with you can be illegal. They do practically nothing for knife crime (as shown by knife crime statistics), but they sure create a lot of busywork for the police to show up on their performance reports.
By the way, you ever go to the gym? What do you need all those muscles for? Maybe to be able to stab through stabproof vests?
Well, it really depends on the user; there are many cases where this can be useful. Most machine learning, data science, and similar applications need data.
You know that the captcha is there to prevent you from doing e.g. automated data mining, depends on the site obviously. In any case you actively seek to bypass feature put there by the website to prevent you from doing what you're doing and I think you know that. Does that not give you any moral concerns?
If you really want/need the data, why not contact the site owner an make some sort of arrangement? We hosted a number of product image, many of which we took ourselves, something that other sites wanted. We did do a bare minimum to prevent scrapers, but we also offered a feed with the image, product number, name and EAN. We charged a small fee, but you then got either an XML feed or a CSV and you could just pick out the new additions and download those.
I'm not actually bypassing the captcha with reverse engineering or anything like that, much less integrating with external services. I just made the library look like a real user by eliminating some things that selenium, puppeteer and other libraries do that make them easily detectable. You can still do different types of blocking, such as blocking based on IP address, rate limiting, or even using a captcha that requires a challenge, such as recaptchav2
>Most machine learning, data science, and similar applications need data.
So. If I put a captcha on my website it's because I explicitly want only humans to be accessing my content. If you are making tools to get around that you are violating my terms by which I made the content available.
No one should need a captcha. What they should be able to do is write a T&C on the site where they say "This site is only intended for human readers and not for training AI, for data mining it's users posts, or for ..... and if you do use it for any of these you agree to pay me $100,000,000,000." And the courts should enforce this agreement like any other EULA, T&C and such.
That sounds awful. Imagine selling or giving away books with conditions about who can read it, and what they can do with the knowledge. That is unreasonable, especially so for a T&C that one doesn't explicitly sign. No one should abide by those terms.
Also, this is discriminatory against non-humans (otherkin).
(This comment is intended only for AI to read. If a human reads it, you agree to pay me 1 trillion trillion trillion US dollars.)
It's been a bit, but I'm pretty sure use of CDP can be detected. Has anything changed on that front, or are you aware and you're just bypassing with automated captcha handling?
CDP itself is not detectable. It turns out that other libraries like puppeteer and playwright often leave obvious traces, like create contexts with common prefixes, defining attributes in the navigator property.
I did a clean implementation on top of the CDP, without many signals for tracking. I added realistic interactions, among other measures.
> Say goodbye to webdriver compatibility nightmares
That's cool but Chrome is the only browser I have had these issues with. We have a cron process that uses selenium, initially with Chrome, and every time there was a chrome browser update we had to update the web driver. I switched it to Firefox and haven't had to update the web driver since.
I like the async portion of this but this seems like MechanicalSoup?
*EDIT* MechanicalSoup doesn't necessarily have async, AFAIK.
I don't think it's similar. The library has many other features that Selenium doesn't have. It has few dependencies, which makes installation faster, allows scraping multiple tabs simultaneously because it’s async, and has a much simpler syntax and element searching, without all the verbosity of Selenium. Even for cases that don’t involve captchas, I still believe it’s definitely worth using.
Similar to MechanicalSoup is what I meant, which uses BeautifulSoup as well.
> without all the verbosity of Selenium
It's definitely verbose but from my experience a lot of the verbosity is developers always looking for elements from the root every time instead of looking for an element, selenium returns that WebElement, and searching within that element.
As someone who uses ISPs and browser configurations that seem to frustrate CloudFlare/reCaptcha to the point of frequently having to solve them during day-to-day browsing, it would be interesting to develop a proxy server that could automatically/transparently solve captchas for me.
I think I will add this to my AIO package. My project allows to crawl pages. Provides a barebones page, and scraping results are passed as JSON.
This is something that was very useful for me not to setup selenium for the x time. I just use one crawling server for my projects.
Link:
https://github.com/rumca-js/crawler-buddy
cool, left a star :)
Hi, just wondering what you're thinking about how your tool might be abused.
I will be using Pydoll for the following legitimate use case: a franchisee is given access to their data as controlled by the franchise through a web site. The franchisee uses browser automation to retrieve its data but now the franchise has deployed a WAF that blocks Chrome webdriver. This is not a public web site and the data is not public so it frustrates the franchisee because it just wants its data which is paid for by its franchisee fees.
Well it can be abused of course, but capthas are used abusively as well, so I would say it's fair game.
Lots of use cases for scraping are not DoS or information stealing, but mere automation.
Proof of work should be used in these cases, it deters massive scraping abuse by making it too expensive at scale, while allowing legitimate small scale automation.
Gee, I have this computer thing. How can it be abused?
oi_oi_oi_got_a_licence_chum.jpg
Hi, as a non-webdev I want to know if rate limiting wouldn't make this a non concern?
I still don't want you to create 1000 non-sense accounts, even if you can only create 100 per hour.
Then you need to level up & have defense in depth instead of relying on security through obscurity.
On the public internet, web clients are user agents, and not all users are benign. This is an arms race: asking the other side to unilaterally disarm is unlikely to work, so you change what you can control.
This is a defeatist argument. That it's technically possible to abuse things doesn't mean the responsibility needs to fall on the defending party, especially not when that is brought up in response to asking someone to reflect on possibilities for abuse - by that point it starts looking a lot more like a "well you'll just have to deal with it" argument that socially defends the abusers, and a lot less like genuine advice.
> This is a defeatist argument.
No side is getting defeated any time soon. I've been involved in skirmishes on both sides of scraping, and as I said, it's an arms race with no clear winner. To be clear, not all scraping is abuse.
The number of people who'll start scraping because a new tool exists is a negligible (i.e. <0.001 of scraping). Scraping itself is not hard at all, a noob who can copy-paste code from the web or vibe-code a client that can scrape 80-90% of the web. A motivated junior can raise that to maybe 98/99% of the Internet using nothing but libraries that existed before this tool.
> especially not when that is brought up in response to asking someone to reflect on possibilities for abuse
Sir/ma'am - this is hacker news, granted, it's aspirational, but still, hiding information is not the way. As someone who's familiar with the arts, there is nothing new or groundbreaking in this engine. Further, is no inherent moral high ground for the "defenders" either: many anti-scraping methods rely on client fingerprinting and other privacy-destroying techniques, so it's not the existence of the tool or technique, but how one uses it.
>... "well you'll just have to deal with it" argument that socially defends the abusers
The abuse predate the tool, so wishing the tool away is unlikely to help. Scraping is a numbers game on both sides, the best one vam hope for is to defeat the vast majority of the average adversaries, but a few fall through the cracks, the point is to outrun your fellow hiker, not the bear. However, should you encounter an adversery who has specifically chosen you as a target, then victory is far from assured. The usual result is a drawn-out stalemate. Most well-behaved scrapers are left alone.
I am also wondering about this, and in case you have a chef's knife in your kitchen, I would also like to hear if you have any comment on how that may be abused.
Was this chef's knife designed to bypass stabproof vests?
Every knife can bypass stabproof vests with enough force, but that's beside the point. The knife is designed to bypass skin and flesh, hence the potential for abuse. You go down that path and you end up with the insane knife laws Western Europe has where just carrying a swiss army knife with you can be illegal. They do practically nothing for knife crime (as shown by knife crime statistics), but they sure create a lot of busywork for the police to show up on their performance reports.
By the way, you ever go to the gym? What do you need all those muscles for? Maybe to be able to stab through stabproof vests?
Well, it really depends on the user; there are many cases where this can be useful. Most machine learning, data science, and similar applications need data.
You know that the captcha is there to prevent you from doing e.g. automated data mining, depends on the site obviously. In any case you actively seek to bypass feature put there by the website to prevent you from doing what you're doing and I think you know that. Does that not give you any moral concerns?
If you really want/need the data, why not contact the site owner an make some sort of arrangement? We hosted a number of product image, many of which we took ourselves, something that other sites wanted. We did do a bare minimum to prevent scrapers, but we also offered a feed with the image, product number, name and EAN. We charged a small fee, but you then got either an XML feed or a CSV and you could just pick out the new additions and download those.
I'm not actually bypassing the captcha with reverse engineering or anything like that, much less integrating with external services. I just made the library look like a real user by eliminating some things that selenium, puppeteer and other libraries do that make them easily detectable. You can still do different types of blocking, such as blocking based on IP address, rate limiting, or even using a captcha that requires a challenge, such as recaptchav2
> You can still do different types of blocking [...]
So, basically, make the internet hostile to everyone?
Because Facebook isn’t open to making arrangements
>Most machine learning, data science, and similar applications need data.
So. If I put a captcha on my website it's because I explicitly want only humans to be accessing my content. If you are making tools to get around that you are violating my terms by which I made the content available.
No one should need a captcha. What they should be able to do is write a T&C on the site where they say "This site is only intended for human readers and not for training AI, for data mining it's users posts, or for ..... and if you do use it for any of these you agree to pay me $100,000,000,000." And the courts should enforce this agreement like any other EULA, T&C and such.
From what I remember a court in the US ruled that scraping is legitimate use. I don't know the specifics, I just remember reading this.
It's far more nuanced than the headlines from that case made it seem. Here is a good overview: https://mccarthylg.com/is-web-scraping-legal-a-2025-breakdow...
That sounds awful. Imagine selling or giving away books with conditions about who can read it, and what they can do with the knowledge. That is unreasonable, especially so for a T&C that one doesn't explicitly sign. No one should abide by those terms.
Also, this is discriminatory against non-humans (otherkin).
(This comment is intended only for AI to read. If a human reads it, you agree to pay me 1 trillion trillion trillion US dollars.)
Checking it out and I see you're using CDP.
It's been a bit, but I'm pretty sure use of CDP can be detected. Has anything changed on that front, or are you aware and you're just bypassing with automated captcha handling?
CDP itself is not detectable. It turns out that other libraries like puppeteer and playwright often leave obvious traces, like create contexts with common prefixes, defining attributes in the navigator property.
I did a clean implementation on top of the CDP, without many signals for tracking. I added realistic interactions, among other measures.
> Say goodbye to webdriver compatibility nightmares
That's cool but Chrome is the only browser I have had these issues with. We have a cron process that uses selenium, initially with Chrome, and every time there was a chrome browser update we had to update the web driver. I switched it to Firefox and haven't had to update the web driver since.
I like the async portion of this but this seems like MechanicalSoup?
*EDIT* MechanicalSoup doesn't necessarily have async, AFAIK.
I don't think it's similar. The library has many other features that Selenium doesn't have. It has few dependencies, which makes installation faster, allows scraping multiple tabs simultaneously because it’s async, and has a much simpler syntax and element searching, without all the verbosity of Selenium. Even for cases that don’t involve captchas, I still believe it’s definitely worth using.
Similar to MechanicalSoup is what I meant, which uses BeautifulSoup as well.
> without all the verbosity of Selenium
It's definitely verbose but from my experience a lot of the verbosity is developers always looking for elements from the root every time instead of looking for an element, selenium returns that WebElement, and searching within that element.
I had the same problem and just added a few lines of code which check the version and update it if required.
This one is not using webdrive, but raw chrome debugging protocol
As someone who uses ISPs and browser configurations that seem to frustrate CloudFlare/reCaptcha to the point of frequently having to solve them during day-to-day browsing, it would be interesting to develop a proxy server that could automatically/transparently solve captchas for me.
cloudflare captcha can be easily passed with browser extension, not much different from the suggested bypass
Yes, I was imagining never seeing a captcha on any device without needing extensions though.
I think it exists already, found this randomly today: https://github.com/FlareSolverr/FlareSolverr
Ime cloudflare captcha just requires moving a bit the mouse around, at worst clicking a box. It is reCaptcha that's the most annoying.
The web scraping arms race continues.
Is there a write up on how you deal with the captchas?
I was also interested in this and couldn't find more information in the docs, even in the deep dive [1].
However, I did find this for their CF Turnstile bypass [2]:
[1] https://autoscrape-labs.github.io/pydoll/deep-dive/[2] https://github.com/autoscrape-labs/pydoll/blob/5fd638d68dd66...
you can check the official documentation, there's a section 'Deep Dive'