New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Best mobile proxy for Cloudflare / Akamai bypass?
Sup guys, currently running a scraping project that extracts data from high anti-bot protected sites. Tried resi and ISP IPs, mobile rotating as well, but my script is still getting detected and failing to retrieve info.
I don't need thousands of IPs, I need a few good quality mobile IPs with per req rotation. Currently looking at dedicated proxies from IPRoyal, Voidmob and Proxidize. Any other recommendations or hidden gems?
Key requirement is that it has to be a genuine mobile carrier IP from a real device, preferably with switchable p0f as well. No KYC or crypto payment would be great as well.
Budget is not a problem, I just want to solve this.
Thanks
Comments
Are you sure it's your proxies and not just your browser/whatever being fingerprinted?
Yes, I am very sure. The proxies it self is a problem, it requires genuine connection with low fraud score and no LINUX or Datacenter fingerprint.
Cf fingerprinting gone bonkers lately:
2captcha no longer works reliably.
Not only proxies. Hardware. And linux is completely legit. Running linux mint on optiplex locally. iGPU works.
Welcome to blackhatworldtalk
Resi proxies and Linux fingerprints are not trusted by systems at all nowadays, especially scraping environments I run. What I want to achieve is human like connection that is nearly indistinguishable form other normal users/visitors.
How the fuck would you fingerprint a SSL (or socks) proxy? It does nothing but passing on the raw TCP stream. At best you could fingerprint the network stack and like others said Linux on desktops is still very much working fine.
I scrape sites that are highly protected against bots using CloakBrowser:from Data Center IPs. I'd recommend trying it before paying more for residential proxies: https://github.com/CloakHQ/cloakbrowser
Thanks for helping some skid fuck up the web even further. Much appreciated.
Yes I do fingerprint on TCP/IP usually iOS or Mac OS and finish it off with antidetect browser. That's my plan now, seems to work.
I see. Interesting approach of fingerprinting a proxy. Noted.
why not just buy an unlimited sim in in EU country (Croatia 30eur for 5G) and pay someone a few euro to "colo" an android phone? I do this for few people that left to work in the West but had to keep their sim cards, its just a stack of redmi phones that call back VPN to the owners server and then the tunnel is reversed.
They have overhead that causes fragmentation or lower mtu.
SSL proxies or what do you mean? There isn't any overhead during transmission since once the target is chosen during the initial connection process there isn't any need for further information and the stream is just pushed through 1:1.
Sure, if there's some kind of unfortunate buffering strategy going on it might still stick out one way or the other but generally proxies lack the property of VPNs where every packet needs some kind of extra header.
Do you know where to find an up-to-date analysis of the type of fingerprinting Cloudflare does? I don't care about bypassing it, but I'm interested in what it does from a privacy/anonymity point of view.
These type of things are probably a) heavily guarded by the people monetizing bypass solutions or b) outdated very shortly after becoming public due to Cloudflare being forced to keep their filters working. It's pretty much security by obscurity after all. Once you know what's relevant it's probably not that hard to give your client those specific characteristics. At this point having some kind of semi working detection mechanism can probably safely be expected to use as much information as possible though.
Nah, the heavy details (like exactly what APIs are checked and what values increase spam score) are definitely heavily-guarded by the people reverse engineering it, but the general idea isn't. The general deals are something anyone with a browser and a decent understanding of JS debugging could find out in a few hours, but it wouldn't help anyone bypass it.
One reason Tor browser is hit so hard by Cloudflare with captchas is because it does its best to disable or normalize any fingerprintable characteristics.
I've mulled over this for a bit too at some point (somewhat inspired by the GPU detection above). I guess if there's something that's easy to do in WebGL but very hard to emulate on a CPU it might be something that's worth implementing (i haven't really bothered to go deeper, so maybe i'm 100% on the wrong path here).
Kind of in the same spirit it would likely be a good idea to grab information that's a result from actual rendering (pixel coordinates of specific elements for example) that isn't overly easy to calculate in a headless bot just parsing html sources.
In any case just checking results of some JS API reporting the name of a given card seems kinda pointless. SSL will likely play a role too but that's probably to be expected anyways.
Yeah those hardware-unique features are scary. The same can be done with audio via AudioContext fingerprinting. In some cases, each and every computer produces completely unique results that don't depend on your OS or browser.
Exactly how canvas fingerprinting works!
Yeah, that's not exactly what i meant here though. I was thinking more of (extremely simplified but i guess you get the idea) transmitting a couple images to the client, blending them together using some kind of complex GLSL shader and reporting back the result. I'm not sure how easy/far it's possible avoid having the shader emulated on the CPU but as long as that's reasonably nasty it might be something that's at least somewhat effective in hampering emulation (needs an actual GFX card to begin with, needs context initialization, OpenGL code, ...). Images could practically obviously also be a bunch of vertex geometry or really whatever the CPU hates the most.
I think there are WebGL fingerprinting techniques that do that as well. Not just blending, but various other operations. The exact CPU and GPU combination, and sometimes even the GPU's current load, can actually have an impact.
Not exactly what you want, but there's this thread on the Pale Moon forums where the lead dev and users report Cloudflare-specific bugs, and TL;DR after this post the Cloudflare turnstiles and challenge pages start working both because of some newly implemented JS features in Pale Moon's Goanna engine + Cloudflare reaching out to the browsers devs and improving things on their side. So my assumption is that Cloudflare is probably doing this directly with the other main browser developers as well to ensure the challenge works by providing enough entropy Cloudflare "trusts" to distinguish users. They use JA4+ fingerprints in other products for bot management (maybe in the challenges as well) which do some TLS-level fingerprinting but the browser-level stuff is undocumented.
I think the eventual long-term move for them is to switch to doing nothing but hardware-level checking since users hate challenge pages. Google tried it with Web Environment Integrity, now Cloudflare seems to be looking into it again with PACT...
Yeah, there's a lot of information involved where it's likely that no obvious synthetic approach for generation exists. If CF (or really any kind of detection system) could generate some input where they are able to predict the expected output (or at least some kind of range) they'd basically have a blackbox for generating values that prove that there was an actual GFX card involved in generating them.
One of the significantly increased spam score I noticed is if you have a weird resolution or low power.
if you are launching a 640x480, 800x600 window or have 1 core (navigator.hardwareConcurrency) it is nearly guaranteed that you are a bot, because you can't use excuse of very old hardware - will not be able to do HTTP3 or HTTP2 or modern TLS handshake, you are running a VM, or otherwise
Amusingly, Cloudflare is sometimes more lenient with Tor browser than with a connection that is particularly suspicious and appears to be attempting to look legitimate.
What worries me is the push for remote hardware attestation, since, if it's done without privacy in mind, would effectively ban anyone using older hardware or anonymity. If direct anonymous attestation (a feature TPMs have) is used, it could prove that a connection has a real TPM but it won't prove who actually has it, but I don't trust policymakers to actually prioritize privacy.
curl-impersonate is worth to take a look
That's unlikely to actually fool Cloudflare. It just changes things like HTTP headers and TLS settings.
Take a look at Camoufox. It's a stealth Firefox browser that bypasses Cloudflare Captchas so it should be possible to port some of the techniques to the Tor Browser. It's not as good as it used to be, but there must still be some takeaways from it: https://camoufox.com/
Just setup home server within your home ISP. The best solution. IP changes with each router restart and no game with endless proxy bans.