All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Prager-IT/Stefan Prager absence, And a new DoS vector brought to you by Hetzner.
I would like to preface this by saying that I almost never write "bad reviews," because they are almost always the opinion of the vocal minority, and not generally a reliable representation of overall standing with the public. But today I'm going to make an exception, as I have granted these two companies far more leniency than they deserve to address concerns.
If you're interested in a full, more detailed writeup than this forum post, here's the entire article on our portal. (publicly accessible)
We had been a Prager-IT client since September of 2018, and to my knowledge, we always had a great working relationship with Stefan Prager. He would personally handle support requests over Skype when I reached out to him, was never concerned with our customer base, their hosting intents (mostly gaming related), nor was he ever concerned about not receiving his payment on time.
With that said, we have a very strict policy against abuse, and in the past 3 years and 10 months that our Prager-IT account was active, and we were leasing IPs from them, we got a total of 3 abuse reports. (one erroneous/bogus) - details of those abuse reports are below.
| Incident Date & Time | Incident Description | Incident Resolution Response Time |
|---|---|---|
| December 20, 2019 - 12:04 PM EST | Spam compliant, single /32 | 22 Minutes - Resolved at 12:26 PM EST |
| September 23, 2021 - 10:34 AM EST | Bittorrent activity, single /32 | 12 Minutes - Resolved at 10:46 AM EST |
| July 18, 2022 - 11:26 AM EST | Erroneous (see below) | N/A |
It should also be known that Prager-IT implemented an automatic nullrouting system sometime in 2020. Which is fine, we had no issue with such a system as we rarely ever get abuse reports, and when we do, they are generally resolved within an hour, or at the very least, mitigated within an hour awaiting response from our customer.
But for the final abuse report above, an erroneous, and automated Hetzner abuse report was sent to Prager-IT, which was the result of a malicious actor spoofing our leased IP addresses into thousands of different networks to launch a denial of service attack at one of our customers. As a result, the automated nullrouting system at Prager-IT blackholed the victim address of the DDoS attack.
Normally this wouldn't be cause for concern, we would just contact Stefan and have the blackhole listed, explain what happened, and that would prevent it from occurring again the future. Except this time, Stefan Prager is nowhere to be found, it has been 33 days, I have phoned his office numbers, contacted him on Skype, his listed cell phone number, and via his ticket system. Currently, I do not even know that Stefan Prager is alive, as there has been no indication from him or anyone representing themselves as an employee of Prager-IT to even respond to my inquiries.
We have already replaced the IP space we leased from Prager-IT once we couldn't get ahold of him within a few days, but service has also been terminated automatically on his end. (his support ticket system is not detecting responses to the automated nullroutng system) It would appear that Prager-IT is on the brink of collapse, at least from my view. There does not appear to be anyone at the helm of this company, so if you have IP space you lease from Prager-IT, you should seek a replacement immediately, as all it takes is one abuse report for you to lose your subnets, as there is nobody managing their support system, it's all automated and not detecting responses made to abuse complaints, and then auto terminating as a result.
With that said, I have no personal issues with Stefan, I hope that he is in good health, but since it has been 33 days without any communication with his company, or himself, for all I know, he could be dead.
Now let's address the plague on the internet that is Hetzner Online. Hetzner has developed an automated abuse reporting system, it detects network scans, DDoS attacks, etc. and sends an abuse complaint to the subnet owners.
Except they forgot one crucial facet of the internet, things are not always as they appear. They did not take into account that UDP packets can come from a spoofed source IP. So malicious actors are spoofing victim IPs into Hetzner's network, and then Hetzner will automatically send an abuse report to the victim IP's subnet owner/abuse contact.
I reached out to Hetzner on July 20th, 2022, to see if they could address this issue. The "senior network engineer" I spoke with does not even understand that packets/source IPs can be spoofed, nor do they even know who designed the automated abuse system. So I reached out to the account that is active both here on LET, and on WHT, who appears to be Katie/Lea, depending on the day, and they said they would respond as soon as possible. That's been a month, which is more than enough time for them to address a critical misstep on their behalf.
Automated abuse reporting is fine, and generally a net positive, but when you have a completely inept company like Hetzner, it results in a new form of DoS. Where the actor doesn't even need to flood your network, they can just spoof your IPs and get abuse reports sent to your ISP.
If anyone has any questions, feel free.
TL;DR: Hetzner's automated abuse reporting is now being used as a DoS vector. And Stefan Prager of Prager-IT has disappeared, so if you're leasing IP space there, make sure you're prepared to move, as their automation for abuse reporting has broken down, and no longer detects a response from you on their ticket system, so you cannot resolve abuse complaints no matter how many responses you make on their WHMCS.
Comments
Damn that sounds super rough (the Prager going fully MIA more so)
Out of curiosity, you mentioned there is an implemented "automatic null route" from the Prager side to IPs that you are announcing? does he automatically mark your leased /24 (or greater) as invalid on a RPKI and remove an IRR route?
If so, thats a super hostile way to handle abuse for a /32 taking down a full /24 that is leased out
Edit: Kodus on your great resolution times and lack of inbound abuse reports you mentioned
His automated system will just blackhole the /32, and it will give you a notice that you must respond to the abuse complaint within like 72 hours.
The bigger issue is, his automated system is not detecting the ticket replies on his WHMCS anymore, so no matter how many times you respond to the abuse ticket, it will never be marked as resolved by the automated system, and after 72 hours, you get a termination notice regarding your entire IP space.
So instead of just one customer and their /32 being offline, now the whole /24 is offline because of a single abuse report and his poorly functioning automated abuse system.
All of that is compounded by the fact that you cannot reach anyone at his company to resolve it, including himself.
Shitty situation indeed.
Not sure if I am being stupid, or just dont understand, how can he control your BGP/Routing when you are leasing IP addresses and routing them via your network and your upstreams?
The only way I thought this was possible was to invalidate your ROA and remove IRR routes but that would affect a full /24 at a time.
Is it possible Hetzner is getting reports of potential abuse or threatening IP addresses from a third party? Can you say or share a bit more detail about the specific erroneous report? Maybe a redacted copy of it? Can you share at least what protocol or details of the purported attack were?
Relaying a report seems fine, but it would seem the alleged abuse can and should be validated with traffic or port statistics before applying a null route. Otherwise, their automated null routing system is going to be forever at risk of false positives and potentially responsible for unnecessary outages.
This is probably a question for someone far more proficient with BGP than myself, but I do know he maintains the larger /22 in which my /24 was residing.
We simply had the /24 announced by Psychz Networks with an LOA Stefan provided.
Here's an example of the email from Prager that gets sent regarding an offending /32.
And a bit off-topic, but if you google the IP in that screenshot above, you'll see that it was a very popular game server a month ago when this occurred. (read: magnet for DDoS attacks)
Hetzner's issue is their automated abuse system doesn't take into account that IPs can be spoofed.
So for example, one of the most common DDoS attacks is DNS reflection/amplification, this is spoofing the source IP (a victim) into public facing DNS servers, and then those servers respond with an amplified response to the real IP, causing a denial of service.
Hetzner detects the victim IP in the above scenario as conducting a network scan, and therefore sends an abuse report to the victim's ISP.
Holy shit.
Even requesting user details. Thats kind of aggresive they handle it.
That reminds me of cases where police is asking about customer details.
There is no other way i see as well. Thought of the same. And it very sounds like this.
Other then his automated system will just send automated email to Swiftnode Upstream requesting a /32 blackhole.
So next time upstreams update filters, the prefix will go down.
Crucify him! Crucify him! He's a heretic who doesn't blindly love and praise Hetzner!
@Swiftnode
Do you happen to know (roughly) how many people work at Prager IT? I'm asking because that info might help to understand the situation.
(just speculating - not sure if this is a realistic option) Prager could theoretically get transit from a number of T1s and announce the less specific /22 (for example) and simply push a blackhole community for a /32 within that to said T1s and hope they overlap with the client's upstreams resulting in a blackhole? (the /32 would be still partially routed via non transit routes though)
Regardless how its done... this is a super hostile way to handle IP leasing and honestly, sounds scary for any business relaying on (what sounds like) a really trigger happy system.
Doesn't sound like I can, but if I can do anything for you don't be shy. More than anything I just feel like I've lived that day more than a few times and the kindness of a friend quite often helped me out of it. It's that moment when you drop everything and say "This problem is too big for me" and it would be humbling if it wasn't too exhausting to be anything else.
No mucking with any RPKI validation / ROA, or the IRR is probably necessary. If these are Prager prefixes and/or they are doing the route announcements already, they can easily announce a more specific null route internally or to the upstream provider.
A RTBH system implementation should act on those intended null routes before RPKI validation, so they can be handled accordingly.
From what people have posted on the LIR thread here, it seems like a one man band.
Though even if that is true, it's been 33 days. And I've reached out in every way, across every method of contact, including his cell.
A one man band not responding for a couple days, or even a week, could be understandable. Not responding for over a month? That sounds indicative of a larger issue.
Especially considering in the past, He used to always be responsive on Skype, within minutes/hours.
That's about what I thought, a one man operation, probably with one or two more persons in admin and support.
IMO the most probable explanation for what you experienced is something "innocent" like e.g. he made a short vacation (couple of days) but had an accident or he suddenly got quite ill. The way you describe the man, based on your experience, seems to suggest that as long as he's able to he'd never just "vanish", so some kind of accident or serious health issue seems to be the best hypothesis.
But then, him highly likely being "innocent" doesn't change your situation ... I understand that.
Absolutely, Stefan was always super friendly, I don't have anything personally against him, and his service was fine for nearly four years up until this incident, unfortunately this one incident is bad enough to result in not feeling comfortable relying on his services again.
I do find it odd that nobody else is reporting the same lack of response though, since Stefan has quite a pool of IP space that he leases out, most of which I assume is leased by providers here and on WHT.
Appreciate it.
Thought I'd also pitch in on this topic - my experience with Stefan was also pretty decent way back in 2018. All the way up to about 2020 when he started nullrouting entire /24s for what was a plain and simple email spam report to one of our VPS clients. As per his request, we restricted their services and in other words sorted out the abuse report.
We reported that back to Stefan and minutes later, another two subnets that we used to rent from him suddenly get terminated, even though they've been paid for 3 months in advance. While the responses seem mostly automated, I'm almost convinced he at least reviews them manually because I'd always get a variation of a similar sounding message, sometimes with a typo here and there.
To my knowledge, Stefan is the only person working at Prager-IT and became increasingly hostage towards his customers a couple of years back. While I understand that his services have and will attract abusers, blackholing services that aren't otherwise affiliated with the abuser just doesn't seem to be a good thing. In addition to that, he started asking for our customer' information and I quote "full name of your customer, your customer's IP address, their email, residing address and method of payment" to which we didn't comply for obvious reasons and that reply then let to complete termination without refund or communication from his end other than something alone the lines of "I don't want you as a customer". We ended up charging back for the remaining months of service since we were unable to get any reasonable response from him.
In my honest opinion, he turned into a complete control freak and lost his grip on reality.
Woah, the abuse email is requesting an unnecessary amount of information from someone who is supposed to be a trusted partner. Also, requiring to terminate a customer over a single P2P abuse seems unreasonable and not a discretion which corresponds to the IP space owner to enforce.
About Hetzner abuse, they have always been heavy handed, both in handling it and sending notices. The relevant issue here is not them sending fake abuse, plenty of other organizations do too - the issue is your upstream acting on it.
I contacted Stefan about a month ago to request an additional subnet but never heard anything back. He’d always been excellent in previous dealings but it sounds like I may have had a lucky escape.
My last contact with Stefan was on 27 of July and he responded within minutes. You have sended him a message before the 27 of July?
I've sent him over a dozen messages since July 18, 2022.
On his whmcs support system, on his Skype, called every number on his website, called both numbers on his Skype, emailed, etc.
All have gone unanswered, and as I stated above, Stefan used to always respond within hours, it's been 34 days now since this incident started, and he hasn't responded at all.
I thought he was always extremely strict on any sort abuse? There was some complaints on WHT from at least a few chinese based users that got terminated w/ minimal/no warning because someone torrented, scanned, or got reflected ddos off them. Something like that.
I always figured that was his main point. If you're running a very clean shop/personal use that'll never, ever, have a complaint, then his pricing is very attractive.
If you use his blocks but run a hosting company on them, you'll likely have a bad time.
Francisco
A year or so after we began leasing a /24 from him, we got our first abuse report on a single /32 for spam. It was resolved in 22 minutes. I explained to Stefan over Skype what happened, and how we handled signup/initial fraud screenings for customers.
He responded with "Not a problem, it happens." - he marked the ticket resolved, and we moved on.
So it doesn't seem like he was always this way, but you're absolutely correct, I am finding more and more complaints regarding his overbearing ways of handling abuse reports.
It would appear so, since any hosting company is inevitably going to end up getting some abuse reports. You can run the absolute cleanest network, screen every customer thoroughly, and still end up with a customer whose device is compromised, and as a result get abuse reports.
And to be fair, we openly stated our usage for the IP blocks was to provide gaming services, websites, etc. Stefan didn't have any issue with it at the time we originally signed up, perhaps his practices/acceptable uses have changed over the years.
Still, it would have been better had he just reached out to us and said, "after your contract is up, we do not wish to renew," rather than just radio silence for over a month.
But all's well that ends well, I guess.
Sorry for necro, but is Stefan Prager back?
I'm curious personally.
Seems like he is still somewhat active as latest (negative)review is pretty recent.
https://www.trustpilot.com/review/www.prager-it.com
I'm not sure, it seems like the complaints are basically the same as mine. Those people are getting the same autoresponse from Stefan's automated abuse system.
I haven't had any contact with him, all my skype messages and phone calls have gone unanswered and unseen.
I also still have monitoring on the range he "reclaimed" from us. And it has never came back online at any other ASN.