Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


GreenCloudVPS Node Down & all Data lost
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

GreenCloudVPS Node Down & all Data lost

I have a VPS from GreenCloudVPS's Node DC2-SG2 in Singapore. Recently, I received an email telling that they found a bad memory & there will be a scheduled maintenance for briefly 1 hour. Then, it went offline & after some moments, they announced that they found issue with Raid Array & trying to re-build it which will take around 8-10 hours. After that I forgot about it as the VPS was being used by My Dev Team.

Suddenly, I received an email with New KVM VPS information & also found that they extended the service period.

I have backups of the data. But, many might not have. My concern is, if Raid can't save you, then what is the importance of making Raid ? Is there any statictics that in how many incidences someone got the advantage of Raid ? Also, what's the way which the VPS Providers can adopt to recover data loss in any situation ?

Thanked by 2alinull BasToTheMax

Comments

  • NeoonNeoon Community Contributor, Veteran

    This can happen, keep backups.
    Enjoy your 6 months free service.

  • dosaidosai Member

    Raid is not backup.

  • @dosai said:
    Raid is not backup.

    It is not a backup. I agree. But then what's the advantage of doing RAID?

  • I think this is very good behaviour of Greencloud. They seem to have informed you correctly, and, without asking for it, they've generously extended your service period.

    As others say as well: RAID is not backup. I don't know what RAID configuration they used, but even if it wasn't RAID0, you can always have a failure in a way that your storage is lost. RAID with some kind of redundancy can prevent a lot of scenarios, but not every one.

  • Mahfuz_SS_EHLMahfuz_SS_EHL Host Rep, Veteran

    @NDTN might share some insights so that we can know more about the incident & get something new to learn.

  • @rafathossain said:

    @dosai said:
    Raid is not backup.

    It is not a backup. I agree. But then what's the advantage of doing RAID?

    Some protection against disk failure, increase in IO and storage space, nothing more

  • @rafathossain said:

    @dosai said:
    Raid is not backup.

    It is not a backup. I agree. But then what's the advantage of doing RAID?

    Its for some resilancy and some performance depending on arrangement. If the machine was provisioned with just a bunch of disks with no RAID or just a simple RAID0 - you would've likely experienced more issues and all would be catastrophic.
    But as with everything in life, somethings just don't work out as planned and catastrophically fails. Free 6mo's of free service is good compensation.

  • host_chost_c Member, Patron Provider

    @Calypso said:
    I think this is very good behaviour of Greencloud. They seem to have informed you correctly, and, without asking for it, they've generously extended your service period.

    As others say as well: RAID is not backup. I don't know what RAID configuration they used, but even if it wasn't RAID0, you can always have a failure in a way that your storage is lost. RAID with some kind of redundancy can prevent a lot of scenarios, but not every one.

    This is true, even raid 10 has a week spot, if you loose the same 2 drives belonging to the same mirror, you loose all the pool, even if it is al 10,16,20 drive pool.

    GreenCloudVPS handled this like a champ, giving you 6 months of free VPS.

    If this is something that bothers you, next time get something that has a backup plan included.

    Thanked by 2MikeA BasToTheMax
  • MikeAMikeA Member, Patron Provider

    6 month extension is very generous. Shit happens.

  • @rafathossain said:
    It is not a backup. I agree. But then what's the advantage of doing RAID?

    At the risk of being obvious, everyone who ever uses RAID should at least read this:

    https://en.wikipedia.org/wiki/Standard_RAID_levels

    Then, figure out why you might want to use a particular RAID level, and then you'll have your list of advantages for your own use cases. YMMV in all cases.

    For me, I use RAID 1, 5, and 10 for different use-cases when needed. The primary benefits are all listed in the article above.

    In ALL cases, I have actual backups, of course. Again, "RAID is not backup," as the mantra goes. Say it 10 times. Put it on a post-it on your computer monitor.

    GreenCloud did not have a backup in this case. Nor did they promise they would. You, as a user, should have your backup/failure plan, and assume a level of risk.

  • host_chost_c Member, Patron Provider

    @rafathossain said: My concern is, if Raid can't save you, then what is the importance of making Raid ?

    It is extrenly important, it will give you performance ( IOPS ) and data redundancy depending on the raid type used, and number of drives.

    A typical SATA HDD 7200 RPM drive will have like 140-160 IOPS/SEC
    A typical SAS HDD 7200 RPM drive will have like 160-200 IOPS/SEC
    SSD's wil start at ~5000 to 10000 IOPS/SEC

    Now, if I want large capacity and higher number of IOPS I have to 'bond' the drives together, and this is a must.

    My options are quite limited:

    RAID 5 - 1 drive failure, would not touch this, not even with a stick!

    RAID 6 - 2 drive failure, this is something that is recommended for backup servers, and typically not more than 12 drives.

    RAID10 - can endure 50% of drives being loss, as long as you do not loose the same 2 drives / mirror. This is definitely the way to go for VPS/VM, but you loose 50% of available capacity, as each pair of 2 drives is mirrored.

    Of all, RAID 10 has the highest performance, Raid 5/6 will have the write performance of a single drive, the reads will be good, as all drives participate in reading, at raid 10 all drives participate at reading, at writing you will have the speed all drives / 2.

    How many do you think will burn 50% of the 10/12 TB drives in a raid 10 array with today prices?

    Get a backup plan.

    Thanked by 1Calin
  • jlet88jlet88 Member
    edited March 2023

    @host_c said:
    RAID 5 - 1 drive failure, would not touch this, not even with a stick!

    RAID 6 - 2 drive failure, this is something that is recommended for backup servers, and typically not more than 12 drives.

    RAID10 - can endure 50% of drives being loss, as long as you do not loose the same 2 drives / mirror. This is definitely the way to go for VPS/VM, but you loose 50% of available capacity, as each pair of 2 drives is mirrored.

    I'm not a hosting provider, so I use RAID 5 for certain use cases and it's great. However, if I were a hosting provider, I agree, I would definitely NOT use it.

    And agreed about RAID 10. My preference when I have enough drives.

    Get a backup plan.

    200% agree. Typical 3-2-1 approach is a good starting place for everyone to consider.

    https://www.crashplan.com/en-us/business/resources/3-2-1-backup-method/

    Thanked by 1host_c
  • host_chost_c Member, Patron Provider

    @jlet88 said: I'm not a hosting provider, so I use RAID 5 for certain use cases and it's great. However, if I were a hosting provider, I agree, I would definitely NOT use it.

    I do not wish to spam this tread, but again, even for personal files, i **highly ****recommend **a raid 6 or raid 10, even if it is from 4 drives.

    I personally lost a lot of important data to me on a RAID5 array.

  • ralfralf Member

    The biggest problem is that disks from the same batch tend to fail around the same time. Maybe it's different with SSD, but that was certainly the case with HDDs. Maybe not immediately, but often within a few days or weeks of each other.

    Many years back when I used to manage lots of servers, we'd always have disks from multiple batches in stock for this reason, and every RAID would be built with 3 or 4 disks ordered at least 2 or 3 months apart from each other.

    RAID is not a backup, it just gives you a bigger window to notice a failing disk and hopefully repair the array before another failure happens. Repairing the array is typically quite tough on disks too, so there's an increased risk of failure during that time, and usually when the array is most vulnerable unless you have multiple redundant disks.

    The other main advantage of RAID is that when a disk fails you can often hot swap the disk without loss of service other than reduced performance due to the background repair.

    It's NEVER intended to be a replacement for a backup. You should have MULTIPLE backups as well.

    Thanked by 1host_c
  • @host_c said:

    @jlet88 said: I'm not a hosting provider, so I use RAID 5 for certain use cases and it's great. However, if I were a hosting provider, I agree, I would definitely NOT use it.

    I do not wish to spam this tread, but again, even for personal files, i **highly ****recommend **a raid 6 or raid 10, even if it is from 4 drives.

    I personally lost a lot of important data to me on a RAID5 array.

    Gotcha, I know the risks, I was pretty careful to indicate in my posts that it's great for certain use cases, and that in ALL cases, I have actual backups, and strongly suggest backups of course. And I certainly would not do it as a hosting provider, which I'm not one. I thought I was pretty clear in my posts, but maybe not. The point of my post was NOT push RAID 5 on anyone. This is a sidebar distraction. But to clarify, I was suggesting people become acquainted with the different RAID levels and make their own choices, understand the levels of risk, and as I also said, YMMV.

    As for you losing data on RAID5, that sucks, been there, I've lost data on every kind of RAID configuration, and what matters more than anything is what backups I had. Obviously different RAID levels are more or less resistant to failure. Just understand the risk for each situation.

    Small practical example, at the risk of further distracting from the thread, if I have a total of ONLY 3 drives or bays or ports to use, and nothing else, no other options, which has happened to me more than once, then in that situation, I would NEVER run RAID 6 on that, it's impossible. You need 4 drives. So that's an obvious one. I would also NEVER run RAID 5 on it, since there's no local backup, UNLESS I was very comfortable with the risk of losing everything. So for me, in that case, I would probably run RAID 1, and use the third drive as a backup. Not very efficient, but I'd still have a local backup, which is better than no backup at all.

    The same logic can be applied for whatever number of drives you have, whatever resources you have, from little guy up to huge cloud provider. Just understand the risks, benefits. But I'm not pushing a specific level of RAID. And I thought I was clear about that, but I could have been more clear for sure.

    Back to the regular discussion.

  • TrKTrK Member

    Every lowend server is required to have standalone backups just in case, i remeber i loosing my data once and after that, i don't even believe any backup solution that comes with the VPS, always i mean always have a remote backup available in hand, i am doing my backups with s3 that just have me spend my days without worries. Enjoy your 6 month service extension.

    Thanked by 1jlet88
  • This is why I love providers like Vultr/DigitalOcean/Hetzner for production. They make it really easy to set up and it's cheap.

  • @TrK said: i don't even believe any backup solution that comes with the VPS, always i mean always have a remote backup available in hand,

    I would never trust solely on the providers backups alone. You never know if the provider will deadpool, their backup systems fail, or the provider has had enough of you and boots you out the door.

    Having an additional remote backup sites is always a fine plan on keeping data loss to a minimum.
    Relying on RAID is reckless. Relying on the providers backup is careless. Having no backup just means the data was worthless.

    Thanked by 1nick_
  • NDTNNDTN Member, Patron Provider, Top Host

    @Mahfuz_SS_EHL said:
    @NDTN might share some insights so that we can know more about the incident & get something new to learn.

    An email should be sent to affected customers on that node already. We do not use RAID-0, RAID-5 or RAID-6. On that node we have 8x drives: 2x drives in RAID-1 for the OS and 6x drives in RAID-10 for the VMs.
    A scheduled maintenance took place to replace a bad RAM stick, the OS booted fine after that but the RAID-10 corrupted, disappeared and cannot be mounted back, all 8x drives are healthy. It's a really rare case that happened the first time in our 10 years in business, looked around and it might be related to both software/hardware issues. We have reprovisoned the node and tested all scenarios to prevent it from happening again. We only have 2 nodes with this hardware and setup, VMs on the other nodes will be migrated then we will apply the fix.

  • nick_nick_ Member

    Was it node KVMSG22?

  • NDTNNDTN Member, Patron Provider, Top Host

    @nick_ said:
    Was it node KVMSG22?

    It was node NVMeDC2SG2. Node KVMSG22 also had a RAM replacement scheduled, all went fine.

    Thanked by 2nick_ ariq01
  • Nothing on their network status page:
    https://status.greencloudvps.com/

    Seems their status page only monitors the datacenter/network, not individual server issues/maintenence.

    Thanked by 2loay ariq01
  • @host_c said:

    @jlet88 said: I'm not a hosting provider, so I use RAID 5 for certain use cases and it's great. However, if I were a hosting provider, I agree, I would definitely NOT use it.

    I do not wish to spam this tread, but again, even for personal files, i **highly ****recommend **a raid 6 or raid 10, even if it is from 4 drives.

    I personally lost a lot of important data to me on a RAID5 array.

    Then you didn't obey rule #1: RAID is not backup. Always backup. And if it's really really really important data: double backup. Triple backup.

    Every RAID type, even RAID0, has it's usecase. It depends on choices made by the user, the characteristics the user wants and the risks the user wants to take.

  • crunchbitscrunchbits Member, Patron Provider, Top Host

    @host_c said:

    @jlet88 said: I'm not a hosting provider, so I use RAID 5 for certain use cases and it's great. However, if I were a hosting provider, I agree, I would definitely NOT use it.

    I do not wish to spam this tread, but again, even for personal files, i **highly ****recommend **a raid 6 or raid 10, even if it is from 4 drives.

    I personally lost a lot of important data to me on a RAID5 array.

    Depending on use-case, thats about all we deploy in production. RAID1, 6, or 10. It's just a buffer, and we monitor the RAID arrays closely. As soon as we see something drop, it's top priority to hot-swap and immediately rebuild. Then everyone sweats for ~1-12 hours.

    @rafathossain as others have said it's not a backup, but it is used to buy you some time to potentially rebuild/recover by removing a SPOF (single point of failure: 1 disk down = all data lost). It also adds performance benefits and scaling benefits.

    It sounds like GreenCloud had a different issue regarding the entire array which is rare but absolutely does happen--which is why RAID is not a replacement for backups. It is just an extra tool in your belt against data loss. This is why we include backup slots for all our VMs which go to a separate cluster. It's not fast disks or pretty, but they are RAID10, can be automatically scheduled, and just remove an excuse for a customer to have data loss. The ideal situation is:
    1. Host uses RAID(1/6/10) on the node
    2. User makes religious backups using host-provided resources
    3. User also backs data up locally or to another separate provider

    Personally, I think @NDTN handled it professionally if they kept you informed and gave half a year of free service. It's really not something you ever want to see as a service provider, but it is just a potential risk of business as you get more TB's under management.

    Thanked by 1SeederKun
  • TeoMTeoM Member

    I see quite clearly the provider in the debt even if it would only be a drop in the bucket and you had 1 backup before 1 week, from me on magnetic tapes would be enough. It can't be too much to ask for a disaster backup.

  • JounouchiJounouchi Member
    edited March 2023

    I am one of the victim in this incident.
    As mentioned by the team, they sent a notification email to inform on the activity on 10 March, but I don't receive any email from my end so no backup being taken.
    I have asked another victim and he also didn't receive any planned maintenance email.
    The first email I received is on 14/3 1.42PM (UTC+8) which I raised the ticket after the server down.

    Lesson learned: Do backup always.

  • bdlbdl Member
    edited March 2023

    @Jounouchi said: I am one of the victim in this incident.

    ...

    I have asked another victim

    ... a bit dramatic, don't you think?

    You had no backup plan and got bitten - I'm sure you'll take backups regularly from now on...

  • ralfralf Member

    @TeoM said:
    I see quite clearly the provider in the debt even if it would only be a drop in the bucket and you had 1 backup before 1 week, from me on magnetic tapes would be enough. It can't be too much to ask for a disaster backup.

    For many people having a 3 day-old backup automatically re-installed over a live system can be a worse outcome - they'll have missing data for that period, maybe not informed and if they have taken their own backups and want to restore from that, they then have to worry about new data that's come in and sort out divergent databases etc.

    Do your own backups. No ifs or buts. Even if your provider offers free backups, if you don't backup your data yourself you have no idea what's being backed up, how often and whether the restore process actually works.

    It's not like there's much of a reason to not do it. Get another "storage VPS" from somewhere for a couple of dollars a month (e.g. I have a 2TB hosthatch and 1TB interservermike just for backups) and set up borg. It's easy. I have a backup script like this on every machine that's run by cron:

    export BORG_PASSPHRASE='....'
    export BORG_REPO=ssh://borg@....
    borg create --compression lz4 --exclude-caches --exclude '/home/*/.cache/*' --exclude '/var/tmp/*' --exclude /tmp --exclude /proc --exclude /dev --exclude /sys --exclude /run -x ::'{hostname}-{now:%Y%m%d-%H%M}' /home /
    borg prune --keep-daily=7 --keep-weekly=4 --keep-monthly=12 ::
    

    Very simple to set up, it typically takes me about 5 minutes per new VM to set up backups to 4 different locations. FWIW, I use a different passphrase on every VM/backup combination which isn't strictly required but means that if one of my servers AND my one of my backup servers were ever compromised, they still only have data from the original compromised machine.

    I also run them all in --append-only so that someone on a compromised machine can't delete any backups and then manually run borg compact $REPO on the server every few months on the couple of backup locations where the disk is somewhat small. For the locations where I don't ever plan to do that, I also don't bother running the prune.

    Thanked by 3bdl hapkido Jounouchi
Sign In or Register to comment.