All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Kiwix servers terminated by Hetzner
Hetzner again cancels servers without any explanation. I suspect it’s due to traffic usage, similar to previous cases.
This was posted on Kiwix mastodon account:
https://mastodon.social/@kiwix/113622081750449356
Murphy's law states that if things can go wrong, they will. Ideally in the worst possible way.
For us, that meant having our servers disconnected at 00:00 on a Sunday.
Our main storage backend became entirely unreachable. For the average user that meant not being able to access the library and download files, and for us that meant not being able to connect to it and see what was wrong.
Turns out that Hetzner has decided to cancel our account and terminate all servers. There was no warning (yes, we checked our spam folder), and nobody could be reached before Monday morning.When reached, they could not explain the reason for the cancellation:
Them: - We sent you an email.
Us : -We did not receive it, can you please resend?
Them: - We don't have it
Us: ಠ_ಠ
In the meantime, all servers had been wiped already so no way to retrieve our data.If you are looking for a bad case of the Mondays, well, that was one.
Luckily we have mirrors and these were not affected. We grabbed a new machine somewhere else (Scaleway ; if we name-and-shame the one we might as well name-and-greet the other) and immediately started re-importing our data to our new Master server.All in all, it still took about 48 hours to get these 8-ish TB back online.
If there is any silver lining to this, it is that we could see a few points of vulnerabilities as well as our ability to turn things around in a reasonably quick manner (here be kudos for the two heroes who manage our infra).Learning were made, and we will see in the coming weeks/months how we can implement new safegards within our resource constraints.
/END
Comments
Will @host_c become a sponsor of Kiwix?
Everytime i see Hetzner recently its always negative...they are going downhill rapidly
Hetzner seem to increasingly turn into Hahahatzner. What a ridiculous bunch of losers.
Proud to not be a Hahahatzner customer
I agree as far as their reputation is concerned. They are taking actions without warning and usually when and if they give a reason, it's some technicality. Having said that, the product itself is quiet solid.
The product is solid (100% uptime for me for several years) but since last year I'm buying VPS from different providers to diversify after reading countless such stories.
Agreed. I used to have a dedi with them but not anymore due to the exact reason mentioned here.
Just dont abuse their traffic? Then you dont will have problems with them.
I have around 12 server since 2+ years and total not over 200TB per month.
slowly moving from hetzner to ovh (Waiting for ks deal to steal) to make all the switch. they have raised the price and cut all the bandwidth.
Well done, good boy! Just stay tightly within the regime's line and you'll be safe. Unless of course, you do not even know the regime's line, in which case you might be informed before they cut off your head, or, as we've seen plenty often by now, you do not even get to know why they chop your head off ...
How much bandwidth is abuse? What if it's 201TB and all 12 of your servers are deleted tomorrow?
You're assuming that they were banned due to bandwidth rather than the questionable content. Previous times Hetzner have cancelled people for excessive bandwidth, they've told people and given a month to leave rather than being immediately cancelled.
If you knew where the line was, they wouldn't be able to arbitrarily move it when convenient.
BUT, they talk to you.
Basically CF "security" comes around and says, "reduce your bw usage or there is the door or pay per TB".
I hope there's some life still left in this account @Hetzner_OL Katie are you still there? knock knock
That's what I meant. If they were just terminated with no warning or explanation, it probably wasn't just due to excessive bandwidth.
The post seems like it intentionally didn't want the reason to be stated. The little exchange? How about asking "what was the reason?" And "what did the email say?" to answer the most important question.
Their response appears to intentionally obfuscate this into "they cancelled us for NO reason whatsoever and won't tell us why" with "they didn't give us notice!".
the proper thing to do when someone abuses bandwidth is to LIMIT THEIR BANDWIDTH, not WIPE THE SERVER. completely irresponsible and inexcusable behavior from what is supposed to be a serious company
I moved everything out of Hetzner for few years, never looked back
I do have to say I've recently had really bad experiences with Hetzner - I've been with them for over 5 years.
I emailed their support because I needed to update my credit card, but couldn't due to a CORS error on their portal (read, it's a bug on their side).
I only got a canned answer saying I should use the portal... obviously didn't read my email.
So I paid $25 for a long distance charge on my phone to talk to someone in billing directly and got nothing but attitude. I was eventually transferred to their internal IT department, who nicely transferred me back to billing. I was then informed they can't help me over the phone (had to be email).
Ultimately, going back and fourth over email and phone, they ended up blaming me for their system issues, saying it must be my fault (because no one else has called in to complain) or that is was my bank's fault (Capital One, only the 3rd largest credit card issuer).
Hetzner really needs to get their shit together...
@Hetzner_OL maybe wanna give out a statement?
You gonna end up on some peoples shit list.
Imagine having people running to OVH because of this.
Would be like loosing WW2 all over again.
I had the exact same experience with OVH a few years ago after they updated their billing system. I explained to them that all my other providers and every other online store I use worked fine, and showed them in Chrome inspector that it was very clearly their system that was was failing a 3D Secure request before it got to the point that it would have redirected to my banks site, and they still refused to fix it and blamed my bank.
I eventually had to get a new card with a different bank just to pay my OVH bill.
Same actually, it was a CORS error with 3D Secure - which I also told them. Their portal was rejecting the request because their server was miss-configured - before the bank redirect.
Really makes me wonder what Hetzner has done to make their support so frustrating to work with....
I was trying to help them fix their service, so they could take more money from me.
To provide machines for prices that are almost less that what it cost in electricity to power them, there's got to be a catch.
Hi there, as some untrue news is making its way about this case: There was a notice of termination via email with a deadline in accordance with our T&C, on 30 October 2024. Our team has already been in contact with this customer several times and we also have the transmission protocol of the communication. You can all rest assured that we do not close accounts randomly. There is always a specific and legitimate reason for doing so, such as abuse of our services, not following our terms and conditions, etc. So please make sure you comply with our T&C: https://www.hetzner.com/legal/terms-and-conditions/.
So they lied, I see.
Kiwix:
Probably the whole "too successful issue". When companies grow, at the start there is a major focus on customer gain and keeping customers. Over time as they grow, the whole support starts to become more expensive as more customers means more support = more personnel.
So then you start to see the cost cutting behavior, where they start rejecting less profitable customers or potential risks (before given the people a change to prove they are not a risk customer), and at the same time, you also see the cost cutting on the support people.
This behavior always seem to coincides with price increases. And there have been plenty of examples of that. AX entry systems are now 30% more expensive, Auctions have gone from 23 "lowest" to 40 "lowest", auction storage servers have now dead stops (where they are listed but will never count down), they did not in the past, the whole electricity increase (and "we promise to lower later", never did), the normal storage servers increases, the lack of improvements for years on their VPS prices/storage/memory (vs competition that is now better deals). I can go on as its a loooong list.
Its not a surprise they cut back on communication. Fixing a issue for a low paying customer may cost them more money to fix it, then to just get rid of that customer. I am betting there has been some internal management change in the last few years, because that tends to coincides with those changes.
They did not (if we check the original post)... Hetzner claims they send a mail, OP stated before they never got any termination mail from Hetzner (even checked their Spam Folder).
This is the issue with terminations, its a BIG deal for the customer but its threated like just any other mail (what it is not).
You expect to see more follow-up from Hetzner (even if its just to legally cover their behinds), because email is NOT a secure (and legal) way to terminate one sided a contract. Its that pesky "proof" that you really informed somebody, that is legally a issue.
This is why so many companies insist that if you terminate, it need to be done via their own secure portal or via actual physical post.
But companies can not spare the cost to be sure that customers are actually informed (even if its to legally cover their own behinds), because they assume that most customers will never sue them (as they are small, non-company customers). You can be sure that if they wanted to terminate a bigger company account, there will have been more follow-up or talks to fix the issue they had.
So now one customer lost his data, luckily they had backups and again Hetzner their bad termination policy creates negative vibes for them. I mean, @Hetzner_OL there is no reason to have a bit better follow-up and communication, so when you terminate a customer, you have more clear "we did X, Y, Z to inform the customer of their termination", not a simple "There was a notice of termination via email "...
A Email is not a secure way to ensure a proper termination!
@Hetzner_OL ... all you need do to is blacklist a persons IP, their server goes down, they contact you, and voila, now you have official proof of the termination before the timeline and their data is gone. Most providers have a X days retention policy as a lot of customers ignore terminations because spam filters, or thinking its not real, or its a scam. So by having a better process in place, you avoid all this public drama and have something to point at. But who am i to tell you guys to do better.
They used Hetzner for 3 months only
https://securitytrails.com/domain/library.kiwix.org/history/a
You may be surprised how much controversial political stuff is on Wikipedia. Mentionings political dissidents, some event in the 1989 in China, etc... you get the idea. And that is just one country i can think... Plenty more countries that see information in wikipedia that they rather not want to see (even some western allied countries).
So when a state organization gets a bit frisky that people can access some political sensitive stuff beyond ... Not saying that this is the reason but claiming Wikipedia is not a hot potato if you offer it to download bypassing regional firewalls/filters.