New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
Just red through docs, looks pretty interesting thabnks.
You just described what a dedicated mobile proxy does lol. I use similar stack from voidmob.
I don't use proxies or that familiar; my first reaction was based on "it's encapsulating, therefore it has packet overhead".
Quick search from Google, Cloudflare and haproxy says yes.
You should definitely revisit proxy idea, IPs matter a lot.
Yeah, i get where the idea comes from. There's no encapsulation going on with common proxy protocols though. As far as something like a SSL proxy can be called protocol when it's basically just the client opening a connection, sending "CONNECT host:port\n\n" and after that everything gets directly passed on to the final target.
Wut? I have IP's, why would I want to share someone else's?
I think I only saw that in reference to SOCKS5 with "expensive setup". Sounds a bit dramatic for some extra free bits.
Yeah, SOCKS is a little more clever. I wouldn't be surprised if it supported reusing the connection, which would mean it needs at least some kind of escape sequence to establish a control channel. In any case it's a bit more than just a dumb passthrough.
You see, when you are doing things that violate others' terms of service and are equivalent to DoSing them, you don't want to use your real phone IP
You're most likely getting fingerprinted by leaking some information by the libraries you use. Nine out of ten times it comes down to that.
Depending on the tier of anti-bot protection the site you're scraping is using, you can often times bypass completely the JS challenge by using really high quality IPs and actually mimicking correctly the TLS fingerprint, but the IP has to be truly clean as hell, else Akamai escalates to a JS challenge. Same happens after a certain level of concurrency from the same IP. You can start at curl_cffi, but it doesn't to JA3 nor JA4 fingerprinting correctly. If you search thoroughly, you'll find at least two libraries that do so, both with Python wrappers.
You can also check if they have an APK. If they do have an android application, it's trivial to decompile it and reverse engineer the mobile endpoints, more often than not, they will have less protection than the web app, or lack it completely. Once you go that path, you'll also see a lot of things that the devs probably forgot, e.g. testing endpoints that work. Do not scrape through those, its legality is grey almost black and if that doesn't convince you, their uptime is often just a few days or hours per week, so not worth it even from that standpoint. For actual usable endpoints, try tls-client, and it might just work if you use the OkHTTP TLS fingerprint.
But I'd say that overall using a stealth playwright is the way to go, as someone mentioned cloakbrowser is good, you can also use patchright or camoufox. Cloakbrowser is probably the most prod-ready out of all of them.
Little extra: you can fingerprint the request itself, if you search for the original blog post about curl_cffi you'll see how you fingerprint, at a very basic level. The same idea happens when you use a browser, although it's often through obfuscated VMs such as the ones from Akamai/DataDome/Imperva. If you're technical enough, you can reverse engineer it and improve the way you use these libraries. They're really fun to analyze! You can pretty quickly find some of the parameters they take into consideration to determine if you're a bot or not, e.g. the entropy in your mouse movements, which is why a lot of the stealth browsers have a humanize feature which is just a bezel movement of the mouse, almost erratic.
This always amuses me particularly if the site is paying for akamai because they provide a SDK for every mobile platform that interfaces as a native version of its JS challenge/sensor_data nonsense, included if you are paying for the web version. Like why leave the back door open
Fun fact: Even the exact math library you use is fingerprintable (due to inexact rounding of transcendental numbers) and gives away not just your OS, but sometimes OS version or even distro.
Yup CloakBrowser is pretty cool, testing it out now. I read their github few days ago, following this guide now:
https://voidmob.com/blog/cloakbrowser-mobile-proxy-scrape-without-detection
So you're breaking the web, right? THAT'S GREAT!
THANKS A BUNCH YOU CREEP, here's your award
YOU ARE DOING SOME HECKUVA JOB, CONGRATULATIONS!
I use port forwarded VPN for torrents. Other than that, I don't need a proxy.
Such VPN will get banned almost at any site lol
What's wrong with you? He says he's only using it for torrents and you blabber on about websites...
This sounds shady. I'm in. Why not setup LET SIM farm in Balkan triangle, get Albania, Serbia and Kosovo sim cards and abuse their tourist plans as VPN
If done right i bet there would be some actual money to make on this. I just fear people would complain about their IPs not getting extra preferential treatment due to location. At a certain scale it would probably also get the NAT IPs of the respective providers blacklisted (i kinda doubt they are much more fancy with their setups than mobile providers in central/west Europe).
anyone knows a source for used android phones?
thabnks. to you too
Cloudflare are black hats. I support anything that opposes Cloudflare.
Strongly depends on your threat model. If you are only trying to avoid being automatically blocked, then you can use your own IP as long as you have a way to change it so that you're not stuck on a blocked IP.
You would only need to avoid your own IP if you're worried about legal action, at which point it's firmly into black hat territory. Note that violating someone's ToS is, in most cases (and I am not a lawyer) not illegal or immoral.
The fact that they are keeping people's websites from being overrun by garbage bots somehow makes them blackhats? See, i'm not a CF fan either but in that case they are just one of the few (semi-)solutions people have available.
Edit: If these options start failing completely we'll see an even bigger and faster push towards massive amounts of KYC and TE vendor locks. Undermining them because mimimi-i-don't-like-CF is as shortsighted as it gets.
The fact that CF are conducting a DoS against my internet connection, blocking me from accessing websites, makes them blackhats.
99% of websites that use CF don't need it.
Yes, they do that and the reason is to a good part shit like this thread.
Sure. That's why even the Debian wiki moved behind a challange reading "AI is breaking the web". There's no problem. Just buy a big enough cluster to tank the stupid scrapers hammering your server 24/7.
I mean, sure, no one likes hydrants but if you have an even semi legitimate looking setup i can't imagine them to be that big of a problem. Hell, i'm surfing the web with a datacenter IP and i'm hardly getting bothered at all. If that changes it's either the reputation of the ASN i'm in having gone down or shit like this thread having forced them into getting even more aggressive to keep the bots out.
And that's just the start of it. There's a point where no more aggressive is left and that's where shit like phone captchas (the garbage that forces you to scan a QR code using your locked down phone logged into a Google account) are going to take over. Comparatively CF isn't really that bad. The whole push towards trusted environments opens a ton of options that make CF seem like a great service.
As i've said, i don't like them either but the alternatives that'll take their place if they fail are just way, way worse.
Edit: Shit, even bandcamp as notoriously wide open as they were is beginning to implement bot blocks (that'll sadly? fail - i'm not entirely sure how to feel as them not failing means my downloader is toast too...) because generative AI morons are scraping them to hell and back.
Seems counterintuitive for every torrent website to advertise VPN services and then block them.
Oh wait, he's full of shit.
Sounds snowflakey.
Repairman shop, lol. I think it starts from $10-20 depending on the condition
For phone farm?