New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
@Francisco
@Jack
@Francisco,
I understand
Glad to hear from you. Even if it's just an "I am alive".
Was there a DoS in NY or did you just fubar something? I know you have complicated routing and very simple to do something wrong and set a cascade effect. As it reads sounds like SJ and NY both were under attack as the move was about to happen.
I respect the crazy move you planned with such a skeleton crew. Can't believe three of you pulled that move off. That's a couple thousand pounds of gear likely and that's serious workout just heavy lifting.
Me, I am all for bolt cutters to fingers of DoS idiots. Or a few rear hoof kicks to the head.
Yep, notice I mentioned the amount of autonulls we pushed
Floods were anywhere from 500Mbit to a few gbit.
There was a bug that didn't catch all of them and that's when things kind of threw up, but I adjusted the limits and it took care of every flood NY had.
Francisco
@Francisco:
The MySQL® server is currently offline.
Hope it back soon
workin' on it
Francisco
Thanks @Francisco!
I noticed the autonulls, but reading it seemed a bit dancey and room for some interpretation.
It was a combo problem. Cool, thanks for being honest / candid. That's why folks like you and your company.
Care to comment about the buyvmstatus.com page? Seeing bunch of nodes (18 I think) with current downtime on there. Beginning to think there might be some bugs / issues there.
Interesting, my OpenVZ VPS was still down, i went in stalling and booted id, now it's up.
So if you still have a VPS down, check the node status on buyvmstatus.com, if the node is UP - go in stallion and tell it to boot your VPS.
Is there an ETA on the debian repo mirrors?
Any word on the nodes that are still down? Are they broken, and you don't have the parts? Working on it etc? It doesn't seem right to have a bunch of nodes down with no explanation 23 hours on, and over 12 hours after everything was supposed to be back online. Both my servers are on down nodes, 31 and 55. I have backups, and need to know whether to move on.
In the other post:
20:11 <Aldryic[m]> Alright, give me like 15 minutes to his the IPMIs and get the VPSes up
@ihatetonyy. Thanks for the info. Forgot?! Palm plant on head. But your time stamp in the message is 20:11 and isn't it 21.53. Am I off on timezones? That is a lot longer than 15 mins.
True -- the post was a while ago. Drop by IRC or submit a ticket?
Woah what happened to the IRC? I suddenly need a registered nick?
@ihatetonyy 55 just came up, Yeah! So all I need now is 31. And the strange thing is is that my VPS was up within 2 minutes of it coming back on line.
I've been watching (mostly) and asking on IRC for about 8 hours. I paid attention to the emails and I figured a few hours was OK, but I was getting nervous after most of the nodes went online, but then no more.
@manma. I don't know why, but I was not disconnected I've been using the IP address for 8 hours or so. Pwnyexpress said it was a NETSPLIT. I had to look it up. After it, there was only 5 of us in the room.
Is the offloaded MySQL up?
@jcaleb
Apparently it needs some hands on work, and it's what they'll be doing when they get to the DC in the morning.
A few minutes ago my vps was up, now I can less worries.
But still waiting info from sonicvps whics also moving node with buyvm.
thanks @BK_
@jcaleb IRC is quiet now, and the last I could see, offloaded MySQL was not up.
Perhaps I will just put mysql on my vps instead of offloading, so i can use asap
Mine is down
Not a problem.
2 of mine are down still unfortunately aswell, but eh.
Just woke up and working on the awk side of things.
Some of the nodes likely crappy due to some RAM getting popped out during travel.
the mysql process won't start on addons01 and is giving cryptic errors as to what's up. I'm thinking it's innodb related but I won't know till I head to the DC. I'm about to give Anthony a ring and see what he comes up with.
We're workin' at it. It was a ton ton ton of work to work to pull off.
Francisco
My vps work smooth like butter, very good ping from my desk and yet coming under current management; well done buyvm!
Guess my KVM is on one of the nodes that isn't online yet... Tried booting it through the C/P - said 'successful', but nothing happens...
Will check again in a few hours - hope they get everything fixed. Getting nodes that won't boot after driving & moving for hours must be a real pain in the *** ...
Respect for those guys !
They boot, just a few things got bumped
Node08 was working just fine when i put the crash cart on it, just wasn't wanting to ping out. I thought I had it but nope. KVM06 is likely the same. I know its raid card got bumped and it was an easy enough fix.
Francisco
Supermicro éh ?
(just kiddin' - have tons of them at work, good stuff for the price)
For those with KVM instances that won't boot there was some advice given in IRC that proved helpful to me:
Make sure you don't have any ISO images mounted. If you don't have one, try mounting one and then unmounting it again, then try to initiate a boot.
(This helped me get two instances online.)
SQL is all fixed up.
For some reason our percona is crapping out with userstat=1. We aren't using any sort of replication yet so this doesn't make sense.
The box is back online either way. I'll be doing a quick reboot once i'm in the DC to upgrade the percona build.
Francisco