New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
@black
Will you be releasing the source for how it checks? I understand about not releasing the database but it would be nice to see how it works.
How about making no parameter=checking the requesting client's IP?
Unfortunately no. I've discussed some dynamic checks used in this thread because it'd be somewhat hard (or at least costly) for a VPN server to mask these features. However, not all dynamic checks are hard to disguise so if I release the source code, they can potentially reduce the score of their IP address.
For a given IP, it checks the static list (manually banned IPs), dynamic list (list of tor nodes and stuff), and a cache list (which is dumped every 48 hours). If the IP is not on any of those lists, it'll query one of 3 dynamic check servers that is online, which returns 4 values. Each of these values are assigned a probability between 0-1, and the entire dynamic check system is modelled as a reliability system in a parallel configuration, which spits out a reliability value between 0-1 of the system. This is exactly how the code works (minus some sanity checks and stuff).
Done.
My home IP gets the same score as one of my VPSs (0.6).
Found some concurrency issues with the code, should be fixed now (hopefully).
I'll wait the 48 hours for the cache to clear.
Now my home ISP scores a 0.55 and the VPS that was scoring a 0.6 now gets a 1 so that's fixed.
I get 0.55 (home ISP)? How do you get 0?
It returns 1 for most of our address space, which is kind of strange since most of this address space is used by local residential ISPs.
Just checked again and got 1 for a vps, and .55 for my DSL line (down from .6 and .9 earlier).
I PMed you about the home ISP issue.
One of the dynamic checks tries to look for people running servers / VPNs on their home network. Your home network should not respond to outside requests.
I remember looking at this AS. According to bgp.he.net, the company's looking glass was for a VPS website. I went and looked at the description of each announced IP block by that AS and didn't ban the ones that said wifi, or ISP in it. Hopefully as the project develops and/or I get more complaints of being falsely classified as a proxy, it'll become more accurate.
Nice script. Working good.
I'm not running a web server on port 80 or a VPN server on my home network, but my IP is pingable and I have some webcams on non-standard ports (> 10,000).
Maybe configure your firewall to drop packets that do not connect directly to your webcams. Most people don't run public / private VPNs on port 80 so it's not something specific I look for.
Thanks.
@black can the 0-1 be converted to a percentage (i.e. 0.1 = 10%, 0.5 = 50%, 0.95 = 95%, etc...) or does the formula you use not scale like that? Maxmind uses a 0-4 scale for their proxy detection but unfortunately it does not convert to percentage very well (1.0 = 30%, 2.0 = 60%, 3.0 = 90%, 4.0 = 120%???). Basically I'm utilizing your site in a new script I'm writing where it can grade a new order based on the output of your formula and I would like to have a scale to work with.
You don't have to post your formula but do each of the checks you do have a specific weight or are they treated equal? Like if one of the checks is more reliable than the others, will failing that check automatically give the IP a score above 0.5? I'm not considering using the score for anything automated but I would like to get an idea of how many other anti-fraud steps to put in place and which ones will hold more weight when reviewing new accounts.
Thanks.
The output will always be between 0 - 1, in percent. so 1 = 100% and 0 = 0%, 0.60 = 60%. It will never return values outside of that range unless there's an error in processing the request. The results mean, I'm x percent sure this is a proxy. If it's 50%, then you can treat it like a coin flip.
The 4 dynamic checks are not weighted the same because, for example, a domain hosted on that IP is a stronger indication than a domain hosted on the same /24 as that IP. The formula is exactly the formula for reliability in a parallel system. Each dynamic check is assigned a reliability (a value between 0 and 1), then it computes the total reliability of the system. It's computed as R_total = 1- ((1 - R1) × (1 - R2) × (1 - R3) × (1 - R4)). Each R will only have a probability value > 0 if that particular dynamic check thinks it's a proxy. For example, if R1 and R2 thinks it's a proxy, R3, R4 doesn't, and R1 is given a reliability of 0.6 and R2 is given a reliability of 0.65, then the total reliability of the system is 1-( (1-.6) ( 1-.65)), giving a total reliability of 0.86.
As this system is still in early beta, I wouldn't use it for anything more than a warn notification of sorts.
Thanks for explaining that for me. I've decided to use the results to color code new accounts (0 = white, 1 = red, and anything in between will be a shade of white or red depending on the level) so at a quick glance, I can see which accounts will need closer examination.
In case anybody does was to integrate this into your system, I'm using the following code which works nicely (in addition to the score being displayed inside user accounts): http://pastebin.com/KSwSpRXg
Like @black said though, it's probably not a good idea to use this in production on anything business related, but for a free web host I think it's good enough.
Looks good @KuJoe, thanks for writing it up.
Sorry for the bump. I just wanted to inform those who use this service that even in a distributed system across 13 servers, some of the websites I pull information from are blocking my IPs (at least temporarily) because there's been ~20k lookups in the past week or so. Therefore, results may be inaccurate when it comes to dynamic checks. However, if you get a value of "1" it's still fairly reliable in a sense, since these IPs are explicitly banned. However, if you get a value below that, it may be inaccurate.
I have a few ideas to remedy the situation which will be implemented the near future.
@black if you need some IPs that you can cycle through across the US let me know and I'll be happy to help out (I'll throw in some free IPs for each VPS you order, even if it's just our 64MB ones).
@KuJoe Thanks for the offer. I'll throw it in the list of options I have.
I think my best chance is to make it smarter. To reduce the number of dynamic checks, I'll try to do some preliminary checks on that IP address. If the preliminary checks are OK and they're on a known ISP, I'll just return 0 instead of running a full set of dynamic checks. It should still be fairly accurate.
My home IP got 0.55 :P
Home IP also getting 0.55 on Internode.
0.55 residential Comcast
0 on home IP on Internode
If your IPs get temp bannend by so much requests then you should consider Tor
Edit: words were cutt off e.e
I am indeed, reconsidering, by making it more efficient
1 on Fasthosts, 0 on TalkTalk.
That's correct, right?
On another note, things are partially working again.
Todo:
-Add one or two more dynamic checks to make results more "accurate"