Smallweb LON shared hosting, boom!!!
Dear friend,
At 6:03PM UK time on 25th July 2021 I received an email stating that the server and licenses for London were suspended.
Shortly after, I was alerted that the service was infact terminated in error by the infrastructure provider's WHMCS installation as opposed to being suspended.
Because of this, I am sorry to tell you that catastrophic data loss has occurred in London. With the server terminated, no live data remains.
An emergency re-installation of DirectAdmin on a new server began at 7:46PM once the license was re-activated and the server is back online in a "skeleton" mode which currently does not include the likes of CloudLinux and Softaculous.
I am working non-stop to secure and restore any of our available backups. If you have your own backup, please open a ticket and I can re-create your account.
I am beyond sorry for this damaging event.
I will be communicating with you further as I gather more information to provide.
Michael,
SmallWeb.net
Except for my backup months ago, all files are gone.
BOOM!
Comments
Its okay you can still get your files..
@ezeth
When I read the title I thought he was talking about me as well
Thankfully he's not haha
So, WHCMS went Taliban, does that mean, it has a bug, that may exterminate other stuff too, should I be worried?
Rebell of the Machines? Already this Year?
Backup months ago? That is bigger issue looks like
@SmallWeb All the best mate
WHMCS automated involucration, how nice it is?
P.S. I have @VirMach freebie that will auto-involucrate after 120 days.
Yet I still run production stuff on there, because my app is fully redundant.
Yikes. This is the second time SmallWeb has had catastrophic data loss in a year (LAX1 was the other iirc), and I can't recommend them to anyone else even though they're otherwise a prem service.
Yesterday, I was thinking of having a service with them! Thankfully, I did not.
I don’t get this……. Directadmin has a simple backup that gives offsite backups daily…..
I am also not sure how WHMCS kills a server….
All the best to @SmallWeb . Hope this gets resolved soon. When something like this happens to super-budget hosts (like many are on LE) I always wonder if the trouble was worth it for the pennies they charge.
Good luck with the restore, mate!
I thought initially this all sounded a bit sus, but the upstream provider is indeed taking blame for the issue. Just FYI - I confirmed the initial statement is true and this wasn't an issue caused by SmallWeb themselves.
Thank you, I appreciate that but please don't believe that I am guilt free. As the "driver of the car", I have a responsibility to ensure things like this don't happen, and if they do, that they are resolved completely and although I did not flick the off switch I am the one responsible for flicking it back on.
Thanks mate, you're a star.
I say with 99% certainty you should not be concerned in your general use of WHMCS (In this regard)
Cheers mate
I am used to going to bed at 9AM, but these last two nights it's not been out of choice. I'm exhausted, but this is about the customers and not me. So with that said, I will paste the 2nd update email below:
Who's the Upstream Provider?
Do they have a back door to delete your data?
Michael is a good guy, but this was unprofessional.
I've lost all of my websites and current data (20+ projects). There was an e-mail send out issue I was facing, it was fixed but I was told Michael is running his own dedicated server with DA instance in this location. I fail to understand how a dedicated server gets deleted via WHMCS to the point of full data wipe out, especially since this location "should not be "a re-seller" instance on his side. It's rare to see providers using WHMCS for dedicated servers and having such level of automation.
If that really is the case, you should seriously switch to another dedi provider, but I feel like it isn't the whole story. You should also inform us who is the provider.
For me, there was nothing to restore the backups from. While I have some manual backups, they aren't all up to date, but a full dedi termination is the last thing I find possible in this story.
I seriously hope you re-think the strategy, your upstream providers and put automated backups in place from an admin level. It doesn't seem that backups you have (where applicable), are within the relevant scope (over a month old) as well. So those who got recoveries, got recovered to quite outdated versions. Might be wrong on this, but since there isn't my backup available, I doubt those are recent at all.
I am a long term customer, and wasn't one of those July accounts,yet full data loss occured.
Nonetheless, I wish Michael luck to recover from this, from personal perspective, I hope he can put this case to rest and move on, from business perspective, this shouldn't even be a possibility.
I hear you. This is why I mentioned earlier that it is especially during events like this where I wonder whether the trouble is worth the few pennies Michael or similar LE providers earn from these offers. I agree a lot went wrong and while your comment has been reasonable and not emotionally driven, often enough I see people expecting diamonds for pennies. While it is true that the customer should be able to expect to get what they pay for as outlined by the product description (in this case daily backups), I feel offering backups as part of the product often backfires for small hosts.
This extraction from a german provider's ToS has been automatically translated with DeepL. The Provider I am referring to is what I would consider a premium shared hosting provider in Germany and probably the go-to solution I'd recommend people without a doubt. They charge significantly more than @SmallWeb and yet they do still obligate their clients to take their own backups waiving any responsibility. Now, with this certain provider I would not doubt for a second that they'd have disaster recovery backups in place to restore whole servers and/or specific accounts, even though they are not obligated to. With them you also wont get access to the backups they take without contacting support (no Jetbackups or similar). What I am saying is that Clients should be able to expect to get what they paid for but also not expect too much from budget hosts. To Michael, perhaps removing daily backups could save some trouble.
Guys,
We were the upstream provider. Without getting much into details or violating our customers privacy, we had some billing agreement setup with the client and some service extensions.
Problem was, a lot of the communications were done over discord chat and not helpdesk. Part of me lost track as I have personally been busy with many things (as you may have seen based on my forum interactions or the lack of it lately).
Unfortunately a wrong flag was placed and that resulted in service getting termed since whmcs didn't update the service renewal but chose to simply termination.
This was a mutual error. Neither Michale nor I saw it coming. This is also the main reason why I don't offer discord/dm support since things can get lost easily.
None the less,on our end, we have learnt from the mistake and some of the things will require double approval moving onwards. One from me and one from Roy.
We apologies to everyone for the mishap.
Props for the transparency and coming out yourself to say this rather than putting Michael in the position to spill the name. Mistakes were made, shit happens. Wishing you guys all the best and I am sure everyone has learnt something from this, at the end of the day.
It's your fault for failing to secure a disaster recovery plan, with either up-to-date off-site backups or a method to quickly re-deploy your website to another server.
The provider can burn down the data center or deadpool OfflineServers style at any time, without any advance warning, and you must be prepared for that.
I setup cron job to backup my site weekly, if the provider got something like this, all data gone, I will deploy my backup to other service immediately and never use them anymore. So far, all my provider is good, never got problem.
I don't agree with your view, but you are entitled to it as much as I am entitled to mine. I put my trust into the provider to provide the service and stability, so I do not need to worry about it myself, which is why I don"t host it locally. In this specific case I cannot fault myself as a buyer, because this was miscommunication on the provider's side and could easily be avoided.
Refering to OVH, I don't see the correlation. OVH was able to recover a lot of data from customers disks and re-deploy dedi's as well, although the circumstances were much harsher, involving fire.
Your ideology is that the customer is at fault for not taking daily backups. I partially agree, that clients are responsible for their backups (I took my portion of the blame for not taking daily backups btw), but in my opinion, so are providers at providing stable services. I'd say generally the ratio is at least 70/30, 70 being on the provider, 30 on the client, and negligence on the provider side, generally causes a bigger problem.
We can go back and forth in discussion, but I believe we won't reach a common ground in our views. I've stayed as transparent as possible,this will be my last comment on this matter as well, because my opinion at the end of the day doesn't do any good for each parties ( me or the provider)
You're welcome to take this approach and will probably end up fine most of the time, but just know that there's a very high risk of you getting burned again.
This is just a warning from someone who may've said the same thing a few years back.
So you're actually blaming a Low end provider that doesn't guarantee any user data backups whatsoever for...
checks notes
...not having backups?
Now I know that @SmallWeb has some backups, but it's not weekly or monthly backups as you might expect from mainstream providers. And these providers even put this in their USP's.
The server going down is on the provider. But not having recent backups while hosting with a provider that doesn't offer that is totally on you. Ideology or not.
An update for affected customers:
To clarify comments made by @debaser and @Pax
Backups are advertised, but the Terms of Service state they include no warranty or guarantee of their date, accuracy, and integrity. Does this make one person right and one person wrong? No. It is much easier to throw blame at a customer for not taking backups on a thread about a provider that isn’t myself. But being behind the reality, I find it extremely hard to place blame on the customers, who were just casually going about their life until this event occurred. There is responsibility that is on my end and I would like to apologise to all of those involved.
Looking at this incident, I believe I have failed in the following ways:
I will continue to be regretful for this incidence throughout my life and if I can do anything for any affected users then please reach out and I will be there.