Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Residential IP - Cloudflare Protection
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Residential IP - Cloudflare Protection

I'm developing a Telegram bot that executes a Selenium script when I trigger it through Telegram. Unfortunately, it's currently not functioning on my server due to Cloudflare's protection, which blocks traffic originating from the server's IP address.

I've already configured the user-agent header to resemble a valid browser, and the script runs smoothly when executed on a laptop. However, I need to find a hosting solution that offers high availability, allowing me to run the bot without encountering the Cloudflare blockage issue.

Comments

  • emghemgh Member
    edited May 2023

    Blocking is normally a scale based on several factors, you might get away without getting a residential IP (expensive)

    First, try using this as the driver: https://github.com/ultrafunkamsterdam/undetected-chromedriver

    Edit/addon: And DONT run headless

    If that’s not enough, keep the driver and combine it with a non-spammy VPN with a free tier, try a few ones (Windscribe for starters)

    If not, detect the Cloudflare block, change IPs, restart driver and try again. This can be done several times if time isn’t a big issue

    Finally, if not, what kind of block?

    If we’re talking captcha, you can get away with an AI-based captcha solver for basically free

    Edit/addon 2: For this, I recommend https://truecaptcha.org/. Very easy to work with their API and $0.33 USD for 1000 captchas is hard to beat..

    Thanked by 1Void
  • FatGrizzlyFatGrizzly Member, Host Rep

    Try using cloudflare warp+, might help.

    Not sure

    Thanked by 1host4cheap
  • Does it need to be Selenium? Cloudscraper works well if you can just work using HTTP requests.

    Thanked by 1emgh
  • emghemgh Member

    @CyberneticTitan said:
    Does it need to be Selenium? Cloudscraper works well if you can just work using HTTP requests.

    Nice find! I didn’t know about this module

    But as I said, it also depends on what he means by blocked too

    If they require a captcha for example, this won’t do him any good

  • VoidVoid Member

    Cloudflare clearance token might work, if the api or whatever you are querying supports it. Or try one of the VPS sold by @Calin as it has the cheapest residential IP and cloudflare friendly. I was playing around you.com api and had the same issue and it worked alright using his VPS.

  • FatGrizzlyFatGrizzly Member, Host Rep

    And chances are, sometimes using privacy pass extension can help

  • @CyberneticTitan said:
    Does it need to be Selenium? Cloudscraper works well if you can just work using HTTP requests.

    Tried that and didnt work. Pasted output here https://pastebin.com/ZAE8Sj9a

  • @host4cheap said:

    @CyberneticTitan said:
    Does it need to be Selenium? Cloudscraper works well if you can just work using HTTP requests.

    Tried that and didnt work. Pasted output here https://pastebin.com/ZAE8Sj9a

    Kinda interested in giving this a shot. Can you PM me an address or a similar website that doesn't work?

  • emghemgh Member

    @host4cheap said:

    @CyberneticTitan said:
    Does it need to be Selenium? Cloudscraper works well if you can just work using HTTP requests.

    Tried that and didnt work. Pasted output here https://pastebin.com/ZAE8Sj9a

    Did you try my solutions?

  • edited May 2023

    After some digging around and testing on several of my machines, residential, VPN, and VPS, the website OP is interested is using firewall rules to block non-residential traffic. Cloudscraper and other Cloudflare anti-bot methods won't work here because it's not blocking bot traffic.

    Aside from running the script on your own home network, you may be interested in buying residential proxies. Cost is usually less than $10/GB.

    Or this scraping platform, which gives you $5 free in credits per month, and also has access to residential proxies: https://apify.com/pricing

    Thanked by 1host4cheap
  • inlandinland Member

    @host4cheap said: I'm developing a Telegram bot that executes a Selenium script when I trigger it through Telegram

    .

    @host4cheap said: I've already configured the user-agent header to resemble a valid browser

    .

    @emgh said: Did you try my solutions?

    He's not even using Selenium in the final script. His pastebin clearly shows python-requests request.

  • ObelousObelous Member

    @emgh said: Edit/addon 2: For this, I recommend https://truecaptcha.org/. Very easy to work with their API and $0.33 USD for 1000 captchas is hard to beat..

    Do they only do image captchas? And I'd probably go with capmonster.cloud or 2captcha instead

  • emghemgh Member

    @Obelous said: 2captcha

    Sure, pay 3x more without any benefit.

  • bgerardbgerard Member

    @emgh said:

    @Obelous said: 2captcha

    Sure, pay 3x more without any benefit.

    It looks like truecaptcha doesn't support recaptcha, hcaptcha, turnstile etc. Is that correct?

  • ObelousObelous Member

    @emgh said: Sure, pay 3x more without any benefit.

    More well-known, and solves more than just text on images (+ capmonster.cloud does these too for the same price as truecaptcha, and faster.)

  • inlandinland Member

    @CyberneticTitan said: After some digging around and testing on several of my machines, residential, VPN, and VPS, the website OP is interested is using firewall rules to block non-residential traffic. Cloudscraper and other Cloudflare anti-bot methods won't work here because it's not blocking bot traffic.

    OP forgot to redact the link from the pastebin. I was able to access the website via Tor after couple of attempts. This is probably just a country-level block.

    Thanked by 1emgh
  • emghemgh Member
    edited May 2023

    @bgerard said:

    @emgh said:

    @Obelous said: 2captcha

    Sure, pay 3x more without any benefit.

    It looks like truecaptcha doesn't support recaptcha, hcaptcha, turnstile etc. Is that correct?

    Only image to text, obviously only use it for that.

    @Obelous said: More well-known

    Makes sense if you're choosing a bank, not if you're literally choosing a captcha solver API lol

    If you want to pay 3x for nother because the brand is recognizable, you've came to the wrong forum

    @Obelous said: capmonster.cloud

    Now that's interesting

  • ObelousObelous Member

    @emgh said: not if you're literally choosing a captcha solver API lol

    Except for the fact you recharge into a balance, sure maybe it's not a huge deal when you're depositing a few dollars but I'd still pick something more well-known

  • @FatGrizzly said:
    Try using cloudflare warp+, might help.

    Not sure

    Thank you so much. This works!!!

    Installed warp on the VPS with this and it works
    https://github.com/fscarmen/warp

  • emghemgh Member

    @Obelous said:

    @emgh said: not if you're literally choosing a captcha solver API lol

    Except for the fact you recharge into a balance, sure maybe it's not a huge deal when you're depositing a few dollars but I'd still pick something more well-known

    Yes, paying 3x is better than eventually losing 2 bucks

  • host4cheaphost4cheap Member
    edited June 2023

    Looking for some guidance in running the wireguard with warp config from https://github.com/ViRb3/wgcf as sidecar

    Basically using the wireguard as proxy/passthru for traffic

    version: '3.7'
    services:
      wireguard_cloudflare_warp_vpn:
        image: linuxserver/wireguard
        container_name: wireguard_cloudflare_warp_vpn
        restart: always
        networks:
          - wireguard_cloudflare_warp_vpn
        volumes:
          - './wireguard_config:/config'
          - '/lib/modules:/lib/modules:ro'
        environment:
          - PUID=1000
          - PGID=1000
        cap_add:
          - NET_ADMIN
          - SYS_MODULE
        sysctls:
          - net.ipv4.conf.all.src_valid_mark=1
          - net.ipv4.conf.all.rp_filter=2
        ports:
          - 51820:51820/udp
    
      my-telegram-bot:
        image: my-telegram-bot
        container_name: mytv-telegram-bot
        restart: unless-stopped
        network_mode: 'service:wireguard_cloudflare_warp_vpn'
        depends_on:
          - wireguard_cloudflare_warp_vpn
    
    networks:
      wireguard_cloudflare_warp_vpn:
        driver: bridge
    

    This doesnt work when I have both containers configured as above.
    It DOES WORK if I have only wireguard_cloudflare_warp_vpn container

  • alt_alt_ Member

    Cloudflare recently launched their public browser rendering and isolation services. To get the page content instead of a screenshot, please refer to this guide and make the necessary adjustments to the configuration file. Hope you find this helpful.
    https://developers.cloudflare.com/browser-rendering/get-started/screenshots/

Sign In or Register to comment.