New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
Probably more than 50% of their servers are from Scaleway (Online.net) which still advertises them on main website "OneProvider Selection of outsourcing providers certified by Scaleway with privileged access to Scaleway's tools and resources to help you manage your servers."
I wonder if they'll takeover customers and made some offer for "refugees" to continue using servers if oneprovider doesn't come back with website and panel.
Got this email from Oneprovider a few minutes ago
Also got it.
.
So it seems that they at least restored mail database
I have one server with them in the Paris location (so online.net), of which I'm happy about. However, such a long outage smells bad and not communicating properly leads to all sort of speculations: don't they have recent backups? Have they tested a restore? Worse, could they have been hacked?
If this continues further I will have to look for alternatives: can anyone suggest a provider in Europe offering a dedicated server with 32GB RAM and at least 2x4TB HDD? CPU is not that important, budget around 30-35 EUR/month, should offer the ability to install from my own uploaded ISO.
Maybe get a Hetzner auction? Not sure their recently price. You can open a ticket ask for IPMI.
Someone on OneProvider FB posted this link: https://oneprovider.com/billing - it is clean WHMCS but any action (login, password reset etc) triggers errors.
It seems like they somehow lost their previous OnePanel and trying to setup another software?
No, thats the one always used. Check the announcements, last is 12. march and goes years back.
Their control panel was called onepanel, looked completely different, and was available on panel.op-net.com
I have now successfully logged into this WHMCS, server list (with some old ones that I don't have from years) and invoices are here but there's no way to control servers (reboot, reinstall etc), create onecloud instance etc. It really seems that they are starting from scratch with this.
Exactly like i said before it seems they lost the backup i guess !
Wow, such a magnitude of a disaster from this large a provider is really rare. I mean - good that it appears they did not lose any customer data, but the fact that they had to setup their billing/client front-end from scratch with what seems to be vanilla WHMCS is really peculiar.
It's possible that OnePanel was WHMCS in the backend through the WHMCS API. While going through my history with them, all of the support tickets I've opened has had the default WHMCS ticket template, although it is unconfirmed and it's possible that they just copied off of WHMCS.
Damn still going on didn't expect that
Onepanel and Oneprovider website are finally back and working.
https://panel.op-net.com/announcement/552
Dear customers,
I would like to begin this statement by thanking you for your immense patience and understanding during our longer than desired absence.
In events such as we have just traversed, it is my strong belief that that the only correct course of action here is one of full transparency and accountability.
As such, it is essential that we go over the events of the last few days, and what has led to them. Before we delve into the technical points however, let me address a few recurring points of concern.
First and foremost, we want to re-emphasize that the website downtime was NOT the result of a cybersecurity incident. As previously stated, your personal information and data are safe and have not been compromised in any way.
In light of this downtime, we understand the importance of ensuring our customers have sufficient time to access their account, settle any outstanding invoices, and replenish account funds if necessary.
As of writing, a 72 hours grace period is in place for all payment related concerns. This grace period may be extended if deemed necessary.
No service, dedicated or virtual will be suspended during this period, regardless of your invoices current payment status.
This should give you plenty of time to ensure that any due payment can be made, and that your account funds can be replenished. A notice will be sent out prior to ending the grace period.
Furthermore, all pending orders that have been delayed due to our downtime will benefit from an additional 72 hours renewal date (on top of the usual renewal date adjustment to the date of delivery).
Thirdly, due compensation will be provided on a case by case basis for all customers who were unable to obtain assistance from our support team. They are actively focusing on all pending and urgent support requests that we were unable to get to during this period, while our accounting and delivery teams have resumed and are expediting all pending deliveries and refunds.
Now for the post mortem.
-Following a planned performance upgrade of our platforms hardware on February 29th, incompatibilities between older software and more recent hardware have led to unexpected complications
-On March 13th, at 15:09 EST, an attempt at mitigating those incompatibilities resulted in what appeared to be corruption on some of our infrastructures file system. A few database tables were also affected.
-As a security precaution, we went into emergency maintenance mode to prevent any further damage from potentially occurring.
-Early attempts to fallback on our primary backup servers unfortunately failed, as those had become corrupted by the same material upgrade.
-Secondary backups were then successfully rolled back to shortly after.
-Out of an abundance of caution, we have chosen to extend our down time in order to run complete integrity tests on every single file and database of our infrastructure, and have thoroughly eliminated the unlikely possibility of a malicious event.
I am happy to report that those integrity tests have all been 99.5% successful, and no permanent data loss has occurred. A few incongruities do remain and it is possible that some customers encounter odd but harmless bugs in the coming days such as missing log entries, or missing spec details on a given service. Rest assured that our team is actively ironing out those remaining quirks.
We have also taken this time to accelerate our roadmap on increasing protections so that such events do not occur again. Multiple preventative measures that were already planned have now been completed.
While the duration of the incident was certainly unusual, I am proud of the work that my team has accomplished in the last few days. This was indeed a true test of our resilience, one that every company faces sooner or later, and I could not be more satisfied of the resolve, dedication, and competence with which it was met. We will continue learning from this incident in the coming days, but it is one that we have surmounted, and will, in consequence, grow from.
On behalf of my team and I, we would like to sincerely apologize for any inconvenience this downtime may have caused you. As always, we appreciate your patience and understanding as we worked diligently to resolve the issue. We remain committed to continuously improving our systems and processes to prevent similar incidents in the future.
If you have any further questions or concerns, please do not hesitate to reach out to our support team.
Sincerely,
Charles-René Paquet - CEO
no one is free from this
Looks like their backups were on the same storage service as the live data... At least they had "secondary backups" to fall back to.
It would have been better if they'd been more transparent since the beginning about the actual nature of the problem, there wouldn't have been any speculation and no loss of trust in their operation.