Welcome to LowEndHelpDesk @wxservers. Your call is important to us, please wait while i attempt to connect you to an agent...
Your patience is appreciated, unfortunately all agents are currently busy, you will be forwarded to our voice mail box and one of our agents will attempt to call you back as soon as possible.
ATHK said: Yes solus is down, my VPS is still up though.
Mine's the other way round - VPS down, Solus up. Solus showing status of box as 'unknown' and won't let me restart or do anything else. Only in NYC location, my Amsterdam box is running fine.
We are investigating a potential RAID card failure on NYC3 at this time, the chances of data loss is highly possible. We are awaiting on-site tech to reseat our card and drives in the hope the configuration can be imported back.
We are investigating a potential RAID card failure on NYC3 at this time, the chances of data loss is highly possible. We are awaiting on-site tech to reseat our card and drives in the hope the configuration can be imported back.
INIZ Management
Ouch, hope the data is safe if not then I hope y'all had backups!
Welcome to LowEndHelpDesk wxservers. Your call is important to us, please wait while i attempt to connect you to an agent...
Your patience is appreciated, unfortunately all agents are currently busy, you will be forwarded to our voice mail box and one of our agents will attempt to call you back as soon as possible.
Please read what I said I am asking if anyone is experiencing it too. I am NOT looking for an agent etc. use your head
@Spirit said:
We are investigating a potential RAID card failure on NYC3 at this time, the chances of data loss is highly possible. We are awaiting on-site tech to reseat our card and drives in the hope the configuration can be imported back. INIZ Management
Yeah I just received now too. Thanks for the info.
We are still attempting to recover the /vz partition with clients data, however if you have your own backups then please open a ticket to NOC by emailing [email protected] to get setup with a new VPS on another node in NYC.
>
At the moment we have no ETA and are working as fast as we can whilst ensuring we can salvage any data as possible.
Update 3:
We are still waiting for FSCK to complete however there is a lot of data being pushed into lost+found, this would mean data loss has occured and we will be unlikely able to recover majority of the data. At this point the easiest way to be back online is on a new VM.
>
We apologise for the inconvenience this may have caused to our users on the node, we have spent countless hours and have replaced our RAID card with our spare and replaced all SATA cables to no avail.
>
Please await a final update upon completion of the FSCK which is taking time due to the amount of data.
We have made all attempts to the recover any data and have been unable to do so, fsck has recently been completed and the majority of the files have been dumped into lost+found which makes it near impossible to recover any data with the amount of files in the folder. As such we will be creating new VMs and emailing you within the next 24 hours, as this is a manual process please bare with us.
Clients will be placed back on NYC3 which has new RAID card, new SATA cables and new HDDs and SSDs (for caching) in place from our spares.
Management is currently reviewing todays events and making changes to the current non-existent backup procedure, more details on this will be sent in due time which will be a global change. Credits and further details will be sent in the coming week once all clients are backup and running.
Important thing for me was "current non-existent backup procedure"...
shantanuthatte said: Important thing for me was "current non-existent backup procedure"...
That is completely correct, who does not advertise backups or at least managed services, does not have backup procedures, the customer is supposed to keep backups if his data is important, either on free space as many people offer or at another provider to make sure you still have it if your main host goes down.
There is no need to increase prices with backups when most people dont need them because they know better when to save data, how many copies to keep, how to setup redundancy and replication and for what data in particular.
Saving OS files is useless, the backup the provider has will probably be outdated already, you know best what to save and when, so, you should take care of your backups, at most, the provider will offer you free ftp space as an option, if you care for your data. If not, nobody else will and you should never expect it.
We have a saying here, God gives you everything, but does not put it in your bag. Help yourself before expecting others to jump through hoops for you.
@Maounique said:
That is completely correct, who does not advertise backups or at least managed services, does not have backup procedures, the customer is supposed to keep backups if his data is important, either on free space as many people offer or at another provider to make sure you still have it if your main host goes down.
There is no need to increase prices with backups when most people dont need them because they know better when to save data, how many copies to keep, how to setup redundancy and replication and for what data in particular.
Saving OS files is useless, the backup the provider has will probably be outdated already, you know best what to save and when, so, you should take care of your backups, at most, the provider will offer you free ftp space as an option, if you care for your data. If not, nobody else will and you should never expect it.
We have a saying here, God gives you everything, but does not put it in your bag. Help yourself before expecting others to jump through hoops for you.
Serious? If I'm buying a RAID 10 VPS I expect NO dataloss...Regarding to you, everyone should buy single HDD servers and backup 100GB productive websites every day or what? Maybe Hourly? Where? Traffic? But yes, it is easier to say it's the costumers fault...
RAID=/=Backup.
Even so, RAID cards can fail and corrupt data, we had an instance where half of the RAID failed suddenly (those drives dropping out for unknown reasons) leading to dataloss. While we had a backup, was 12-14 hours old.
Who cares for their data makes back-ups, who does not, it means they have nothing important or unique there and a simple blanked VPS will be at most a minor annoyance.
You know this, i know this and most reasonable people know this.
But there will always be people who will cry and curse because their precious data is gone from their precious $10/year VPS and they didn't have backups. And of course they lost thousands because of this ;-)
Comments
Yes solus is down, my VPS is still up though.
Have you raised a ticket?
Yep done.
Welcome to LowEndHelpDesk @wxservers. Your call is important to us, please wait while i attempt to connect you to an agent...
Your patience is appreciated, unfortunately all agents are currently busy, you will be forwarded to our voice mail box and one of our agents will attempt to call you back as soon as possible.
It's up and running at the moment. But in the statistics within SolusVM I see a gap about 4 hours ago. The last 3 hours it is OK.
But the "outages" keep continuing, is it just coincidence? Every provider has some bad luck now and again.
Mine's the other way round - VPS down, Solus up. Solus showing status of box as 'unknown' and won't let me restart or do anything else. Only in NYC location, my Amsterdam box is running fine.
I just got:
555 views
Ouch, hope the data is safe if not then I hope y'all had backups!
Checked and for some reason my rsync cron hasn't run since 2nd October... Oh dear.
Please read what I said I am asking if anyone is experiencing it too. I am NOT looking for an agent etc. use your head
Yeah I just received now too. Thanks for the info.
GVH, 123systems and now INIZ. Will these be on dead pool list?
+1.
Great, just checked and my VPS with INIZ is also down.
Mine's online still. But I've put in a request for cancellation.
Mines been up for 125 days.
And so it begins. My stable horse starts limping...
Well to be fair it's a reported raid card failure. Happens all to often unfortunately
Update 2:
>
Update 3:
>
>
Still down ...
And another update:
Important thing for me was "current non-existent backup procedure"...
yeah but no reply from Kayako or etc..
That is completely correct, who does not advertise backups or at least managed services, does not have backup procedures, the customer is supposed to keep backups if his data is important, either on free space as many people offer or at another provider to make sure you still have it if your main host goes down.
There is no need to increase prices with backups when most people dont need them because they know better when to save data, how many copies to keep, how to setup redundancy and replication and for what data in particular.
Saving OS files is useless, the backup the provider has will probably be outdated already, you know best what to save and when, so, you should take care of your backups, at most, the provider will offer you free ftp space as an option, if you care for your data. If not, nobody else will and you should never expect it.
We have a saying here, God gives you everything, but does not put it in your bag. Help yourself before expecting others to jump through hoops for you.
There is no need to increase prices with backups when most people dont need them because they know better when to save data, how many copies to keep, how to setup redundancy and replication and for what data in particular.
Saving OS files is useless, the backup the provider has will probably be outdated already, you know best what to save and when, so, you should take care of your backups, at most, the provider will offer you free ftp space as an option, if you care for your data. If not, nobody else will and you should never expect it.
We have a saying here, God gives you everything, but does not put it in your bag. Help yourself before expecting others to jump through hoops for you.
Serious? If I'm buying a RAID 10 VPS I expect NO dataloss...Regarding to you, everyone should buy single HDD servers and backup 100GB productive websites every day or what? Maybe Hourly? Where? Traffic? But yes, it is easier to say it's the costumers fault...
RAID=/=Backup.
Even so, RAID cards can fail and corrupt data, we had an instance where half of the RAID failed suddenly (those drives dropping out for unknown reasons) leading to dataloss. While we had a backup, was 12-14 hours old.
Who cares for their data makes back-ups, who does not, it means they have nothing important or unique there and a simple blanked VPS will be at most a minor annoyance.
+1.
A customer should be dtaking backups regardless of if the host does or not..
It's your data your responsibility.
You know this, i know this and most reasonable people know this.
But there will always be people who will cry and curse because their precious data is gone from their precious $10/year VPS and they didn't have backups. And of course they lost thousands because of this ;-)
Absolutely. No question about it. Also, don't forget to backup your backups and check if your backups are working once in a while.