New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
Haha, I laughed a bit too much at this
haha!
I guess the Twitter updates plan went out the window, heh.
Looks like they just started racking and stacking. https://twitter.com/Aldryic
Glad to hear that
One of my KVM already up...
All three of my OVZ boxes are up, but I think theres some weirdness with routing/latency somewhere. I can telnet into my mail servers mail port and get the server banner instantly from my home machine, but from my box in KC it takes about 45 seconds for the banner to appear (same treatment from mxtoolbox, it reports about 45 seconds to do an open relay test).
Hopefully its just due to so many nodes/containers being brought up at once and it will fix itself.
yeah, ns2.buyvm.net is still down too...
The node with my VPS has been up for 2+ hours, but it will still not respond to SSH to the IP address. I had no idea that OpenVZ based systems took so long to come on line. Live and learn I guess.
One of my VPSs is up. Now I'm waiting for the other two.
Every one of their node but like 38 nodes are online.
omg my site is down for like 3 days now. no way to contact them?
A quick report for my VMs:
SJ-kvm04 - UP
SJ-kvm05 - UP
SJ-node14 - DOWN
SJ-node21 - DOWN
SJ-node23 - DOWN
SJ-node25 - UP
SJ-node28 - DOWN
SJ-node32 - DOWN
SJ-node38 - DOWN
NY-node01 - UP
For now, all you can do is wait patiently
is sj-node54 down?
Haha
SJ-kvm05 - UP
But manage.buyvm.net is down. I dutifully powered my KVM off on Friday...no way to power it back on until they restore Stallion.
Man, this move didn't go at all like one would expect.
Thought they'd be updating on Twitter. Stopped bothering to check for updates since they all but stopped last night. Haven't checked today.
The NY outage was a big black eye. Seems like @Francisco is the magic man behind the curtain and when untethered, well things will happen.
Shame this stuff happened. Lots of foolish children with their DoS attacks. Waiting for DoS attackers in general to get isolated, charged criminally and thrown in prison for a few years of timeout.
Hopefully this incident is noticed by other providers and they get automated and distributed network protection processes in place that can be ran by a team. Time to make such attacks a non issue.
@Jack, yeah not much on Aldryic's twitter either.
I like the BuyVM guys, all sh!t aside. Remain impressed, but this move was a big monster to chomp.
Lesson #1, timeline. 9AM estimated from the email. We are 9PM now.
Lesson #2, time is vital. Food and drinks should be in tow and at the datacenter. Denny's = sleepy. Done that stop over in probably 10 different US states while doing similar types of work. Now I bring food / order into facilities or tell folks to bring their lunchboxes.
Lesson #3, stuff always breaks. Bring spares, documentation, etc.
Lesson #4, 36 hours awake time can kill high performing folks. Some delay due to tired minds. Team needs to be big enough and allow overlapping shifts. There is most likely going to be lingering breakage for remainder of the week, usually is.
Lesson #5, being high profile with multiple targets and putting yourself in mobile situation means targets will get smashed (DoS). That's why the team that knows how to survive sheds the issues and a super owner person like @Francisco doesn't have a temper induced heart attack on the highway in the dessert.
All said, 24 hours downtime, that's really respectable.
I expected more communication out of the crew. I would have only watched along since not impacted, but the NY downtime got me up and out of bed and moving stuff all over and mucking up my DNS records to deal with.
Have to agree with this, although I'm not mad at them. I was expecting ~24 hours downtime anyways.
Laff.
Here we go...?
Things can go wrong. At this point it can't be done much except waiting. Give them time to fix issues and set things up and this is it. We know those guys and we can be sure that they are working on it and they will explain in details what went wrong once they will set things up and find time for this. Just sit down and relax.. or something.
My VPSs came up about 4.5 hours ago but when I logged into them just now the SSH was going a bit slow. @Jack reminded me to check resolvers and I changed them to Google and Level3 (8.8.8.8 and 4.2.2.2) and it's back to normal for me.
My vps in back online, but mysql offload server is still offline. Totally, my site is still down
Hope it will be back soon.
@ihatetonny
Hold your horses here...
The flood was planned to happen. It didn't 'just' happen. It targetted lots of our internal IP's (KVM nodes, router IP's, etc) and happened minutes before the power down in SJ. This wasn't a coincidence because the same flood has been wacking at SJ's door since we got the router online.
Now, the only reason NY had an issue was the thresholds were copy/pasted from SJ a long time ago and I wasn't thinking. SJ has more transit than NY so it was a screw up on my part. There was something like 50+ nulls applied over the period of the night but were all handled quickly and automatically.
In SJ we had 6 people helping unloading while in Vegas there was only Karen, Aldryic, & myself. We had to drive, unload, install rails, configure the network, rack up the servers, handle all the switches, etc, etc, etc.
On just 3 people.
Sure, I would have liked it if we met the 10 AM target but once we got there and saw how annoying supermicro's new rails work, we knew it was going to be missed.
Now, ontop of that only 2 of us had phones that work in the US and Karen's was tied up being a dedicated GPS/tomtom. T-Mobile is pretty terrible so Aldryic didn't have service most of the time.
I truely do apologize that we didn't give lots of details but we were so burned out from the drive and then having to carry all that gear in.
I'm patching whatever last things broke in the move but there aren't many. Resolvers are fixed and were suffering a binding issue.
Francisco
The big boss is exhausted but i'm busy bringing awk back our way
Shouldn't be long at all.
Francisco