New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
The website went down because of a filesystem issue. We had to spend a lot of time being careful as we did not want to lose data. We used a backup that caused one day of orders to disappear from WHMCS. This was manually restored. Nothing was lost.
We have thousands of active customers per team member. We have monitoring setup on all servers and notice it immediately, but one of the thousands of customers can post on this thread quicker than we can check and verify the issue and post it on the network issue page, which we do, with the exception of Frankfurt yesterday due to our website downtime.
We're trying very hard to expand support. We've doubled support hours in the past month and cut down average response times on Monday through Friday. We plan on having better support on weekends as well, very soon. Our response times definitely drop significantly during outages, due to redirecting our attention to that as well as receiving immense amount of inquiries from customers, informing us their service is down and asking for updates.
Just an update on everything for people that haven't read through the thread or our network issue page.
Issue #1 where our website went down has been resolved. There was an approximate 12 hour downtime due to database issues after a problem with our filesystem. Only our website was affected.
Issue #2a where Frankfurt location had a brief network outage. This occurred around the same time our site went down, but it is unrelated.
Issue #2b where Frankfurt had another network outage. Datacenter confirmed this outage quickly and resolved it. This outage affected about 5% of our customers. Uptime for Frankfurt currently stands at 99.9% - which is lower than what we'd prefer it to be (100%) but it's not significantly low. This was a carrier-wide outage. Below is a message from the DC @ 11:58AM, approximately one hour after outage.
Don't forget that we have scheduled maintenance from June 27 to June 30. Check our network issue page for more information. These are planned downtimes to upgrade our nodes.
(edit) Double post
whmcs database restored from yesterday backup, but no data lost? dont make any sense.
Allow me to clarify. We minimized data loss. Then we manually entered lost invoice and service data and synced it back with SolusVM - so no data was permanently lost there. The only thing that is permanently lost is the tickets created within those 24 hours. In addition, many portions of the database was successfully recovered so it was not lost, such as transaction information and clients. In addition the SolusVM DB was unaffected and all services unaffected. No customer data on VMs was lost.
So you're correct if you only quote the part where I said no data was lost instead of including the context around it. I apologize for the confusion. Clearly there was some data lost that was restored.
I was speaking about the orders disappearing and being manually restored, with no loss there.
Start backing up hourly instead of daily, next time this won't happen again.
Their client area goes down often from what I can see, it happened yesterday and today as well.
I know it's an old topic but it might be worth mentioning for future reference.