NodeDeploy

birdie25

Shit's hit the fan, yo.

http://nodedeploy.com/

dominicl

Looks like it @birdie25

gsrdgrdghd

Maybe tag @PhilND but i'm sure hes busy right now fixing stuff.

Jack

@gsrdgrdghd said: Maybe tag @PhilND but i'm sure hes busy right now fixing stuff.

There LEB offer has a lot of info.

PhilND

@Everyone

The hardware raid controller on aura completely nuked it's self and the entire raid array, taking the disks with it, hetzner are pending a shipment of raid controllers in the morning. Fortunately no data was lost as we have measures in place to recover from such instances, it's just a pain to have to shuffle 1.5tb of data around overnight on a very very tight time scale.

Rest assured we're getting things ironed out and new procedures put in place to stop un announced things like this happening again, e.g complete downtime of our services, websites etc. Phoenix and Zeus are unaffected however, we realise this is unacceptable and we're willing to offer compensation and discuss things with clients who feel mistreated, if you need direct contact, ding an email to support@nodedeploy.com

Thanks for your patience.

Phil

ShardHost

@PhilND said: The hardware raid controller on aura completely nuked it's self and the entire raid array, taking the disks with it, hetzner are pending a shipment of raid controllers in the morning. Fortunately no data was lost as we have measures in place to recover from such instances, it's just a pain to have to shuffle 1.5tb of data around overnight on a very very tight time scale.

Rest assured we're getting things ironed out and new procedures put in place to stop un announced things like this happening again, e.g complete downtime of our services, websites etc. Phoenix and Zeus are unaffected however, we realise this is unacceptable and we're willing to offer compensation and discuss things with clients who feel mistreated, if you need direct contact, ding an email to support@nodedeploy.com

Thanks for your patience.

Phil

All the best :)

PhilND

@ShardHost Thanks! Teething problems.. always do suck the most! Things were running smooth as aswell. We'll get it sorted :)

ShardHost

Did the disks die during a rebuild or did the controller dump trash onto the disks?

BBU/No BBU?

Jack

.@PhilND Good Luck hopefully no data is lost!

PhilND

@ShardHost

Dumped aload of crap onto the disks, was then getting I/O errors from 2/4 drives in the array, thought it would be best for them to pull them all and stick some more in.

This particular model doesn't have BBU on it (though it does support it, PITA :()

@Jack We're verifying everything we have on our backup servers, everything looks good to go, waiting on raid controller before we push it all over. :)

Jack

@PhilND what are you using to make the backups? Just SolusVM or ?

Jacob

Thats just a disaster waiting to happen, Controller with no BBU, Oh well....

PhilND

@Jack Rsync to an external server :-)

jarland

It happens. Hindsight is always 20/20. Good luck and hope you guys get everything straightened out.

ErawanArifNugroho

Everyone can have problem, so, good luck, and wish you all the best :)

Satellite

This is why you keep your website on a different server, amateur mistake :P

jarland

@Satellite No one wants to think about hardware failure, it hurts ;)

For real though I have my website on my shared hosting server. That's in my list for July. It was a dumb thing to do, but at the time it didn't even occur to me.

PhilND

@jarland said: but at the time it didn't even occur to me.

This. It'll all be resolved in due course!

GetKVM_Ash

Yep, school boy error. But best of luck @PhilND getting stuff sorted.

For reference, we host our main site with @prometeus and <3 it so far.

Jack

@GetKVM_Ash said: For reference, we host our main site with @prometeus and <3 it so far.

@GetKVM_Ash I got a KVM with them to do the same thing haha.

HalfEatenPie

@prometeus you're the host of hosting companies :P

PAD

Never host your WHMCS on your own shared hosting node, so easy to hack its unbelivable. So many vulnerabilities. Lesson learnt though right? :D GL getting everything back up.

prometeus

@HalfEatenPie said: @prometeus you're the host of hosting companies :P

Well, when I realized that there are more hosts than clients I adapted our business :P

Maounique

@prometeus said: Well, when I realized that there are more hosts than clients I adapted our business :P

Hahah :P Well, looks like most hosts are customers too for things like DNS, monitoring, even main site host :) M

PhilND

@PAD It was actually on a VPS, not our shared node.

GetKVM_Ash

@PAD said: Never host your WHMCS on your own shared hosting node, so easy to hack its unbelivable. So many vulnerabilities. Lesson learnt though right? :D GL getting everything back up.

What lol. A shared hosting server is no more vulnerable than a VPS or dedicated server :S

Well, at least if your with a reputable company.

Corey

I think it's funny that 'providers' here will rent a VPS from another provider to host their main website and talk to someone else about making a 'school boy error'. I could see renting a VPS from another provider for redundancy on your own site - but why the hell would you put your main site in the hands of another VPS provider here?

GetKVM_Ash

@Corey said: I think it's funny that 'providers' here will rent a VPS from another provider to host their main website and talk to someone else about making a 'school boy error'. I could see renting a VPS from another provider for redundancy on your own site - but why the hell would you put your main site in the hands of another VPS provider here?

Dude, you could have just tagged me if you wanted to bitch.

PAD

@GetKVM_Ash

Yes it is, huge difference - you have configuration, control, and privacy.

I would never put my own site hosting confidential information alongside my customers.

@Corey exactly, same aspect.

PAD

@GetKVM_Ash I don't think he was trying to start an argument, but he has a valid point. Its idiotic to me and would put me off hosting with you. I just can't see it making any sense.

GetKVM_Ash

@Corey said: I think it's funny that 'providers' here will rent a VPS from another provider to host their main website and talk to someone else about making a 'school boy error'. I could see renting a VPS from another provider for redundancy on your own site - but why the hell would you put your main site in the hands of another VPS provider here?

Okay, let me elaborate.

Its safer because its outside my own network and thus our site will remain online in cases where our other services may be down.

And i don't see why i wouldn't put my site in the hands of another reputable provider, even more so one that advertises here. Obviously if anything shady went off, it would all come out here?

@PAD said: I would never put my own site hosting confidential information alongside my customers.

Its not alongside my customers.