All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Automatically generated IP blocklists of various types
I thought I would share this here as there are many people here who may find lists of network abusers and common bots to be useful in their projects.
We (at X4B) recently decided to undertake a project to modernise our blocklist generation. As part of that work we opted to make the vast majority of our lists open source.
Now you can (from GitHub) fetch lists of many kinds that are:
- CI generated (Github Actions via daily schedule)
- consistently formatted (all lists are newline seperated CIDRs)
Lists currently include:
- TOR Exit Nodes
- VPN / server networks
- Search Engines
- StopForumSpam
- Uptimerobot
- Paypal
- Cloudflare
- Some extras
In case you don't know you use the raw button to get a consistent link to add to pfsense and similar software
Pull requests and requests for additional lists are welcome.
If you find these useful please do let me know. I've got a few other ideas for other ideas for similar open source (Open source GeoIP anyone?) work and justification for the effort would be grand.
Also thanks @Meganitrospeed for his work on the VPN list. In many ways it was his request that inspired this specific approach.
Comments
Thanks for this!
What would be the reason for blocking UptimeRobot, PayPal, and Cloudflare?
Any blacklist can also be a whitelist (or exception list).
The Paypal list is particularly unique by the way. Web scraping is required for that one
Interesting, why anyone want to block PayPal?
You block them before they block you.
I outsmarted your outsmarting
duckduckgo ip list ?
Instead of blocking you could use the ip's for whitelisting.
Spoofed DDoS is one. By spoofing into those ranges, the attack might get through some filters. For example, OVH is known to have global whitelist for some well known ip ranges.
I've added a feature request to the internal issue. I don't have any data on their crawler currently however. I'd need to first build a reliable detecter (we generally require IP validation from an authorative source e.g ns1 for the company).
I'll look into it in the future however.
https://help.duckduckgo.com/duckduckgo-help-pages/results/duckduckbot/
It would be nice to have the iplist's in nginx format aswell ?
Thanks for the hard work!
Add VPN like NordVPN and others. You can simple enumerate all subdomains to get ip.
Edit: https://ipinfo.io/AS136787#blocks also need to be blocked
it would also be great to get a list with ip without classes, only addresses like not 1.1.1.0/24 but 1.1.1.0, .1.1.1.1, 1.1.1.2, 1.1.1.3 etc.
I know, but then I have to convert the list myself, then upload it somewhere to only import addresses to devices which do not support ip addresses with classes
Great idea!
Here's a couple for you..
Shodan idiots.
All those friggin' census/so-called research sites!
Ugly to scrape but not a horrible idea.
Honestly I'm torn between submitting the data from detection and fragile web scraping.
For now I'll say. PR welcome. See lists_paypal for an example of scraping.
For large frequently updating lists like lists_vpn that would be too much I think. Part of the reason I'm going with subnets is to keep the resource usage for GitHub reasonable.
There are plenty of easy to work with tools for making /32 IP Lists from a CIDR List. I also believe that's within the capability of most people, but if there is a good argument to be made I'm happy to hear it.
Not a bad idea. awk could do it quite easily. PRs accross the repos welcome.
PR welcome. see this commit for your provided ASN for an example.
https://github.com/X4BNet/lists_vpn/commit/3824a35f9be687c37d3fc7ad24a0ffd6029b65e4
You can simple enumerate all subdomains to get ip.
Processing RDNS db's is not particularly fast and the DB's I've seen are massive to download. Not something that lends itself to CI processing.
While there are sql dbs some people make of all RDNS results for the IPv4 space they arent easy to process in reasonable CI time.
PR for CI scripts processing RDNS is welcome if it's:
a) Necessary to cover a particular case
b) Reasonable to process (time)
c) Data sourced from a trustworthy source
I mean you could use a CI script to automate that...
If you have a valid case PR quality CI scripts and make a compelling case. Of course you only need to make the case if you want to merge upstream, else you can run your fork. Ain't git great.
@Hotmarer for note the lists_vpn repository currently covers 140,385,323 IPs. Ballpark estimation of an individual IP list says that would be an at-least 2GB. Running the risk of not even being commitable to Github...
Commit the scripts and have Github actions upload to the releases function?
Personally I think that would fall under rule one of free services (e.g Github's free open source repositories). Don't be a Dick.
Anyway
Not to mention it wouldnt be much use, you would be swapping the need to process the list for the need to form download URLs for the release assets.
All you have to do is query subdomains. No needed RDNS here.
alX.nordvpn.com - Albanian servers
You're right, I didn't think it through.