All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
WebNX / GorillaServers Ogden, UT Datacenter Outage - Generator Fire 🔥
Today's edition of "Do you have a DR plan? Have you tested it lately?" is brought to you by WebNX. Also, lessons on "how NOT to handle outage communication".
WebNX, which I believe is connected to or provides services for GorillaServers, has been down since approximately 2:45PM PST (21:45 GMT) yesterday (Easter Sunday) - so about 14 hrs from the time I'm writing this.
Communication has been horrible (nearly non-existent) - their website, billing/ticket system, email servers, etc went down because they were hosted in the same location. After they got their website back up, they shut off live chat.
They've now updated their site to highlight a post they originally made on Facebook (and nowhere else) 12 hours ago:
Hello Everyone,
Now that we have a better understanding of what happened we would like to give everyone an update.
One of our old generators that have worked for years and was recently load tested had a mechanical failure and caught fire resulting in power being cut to our core routers and fire suppression system controlling the fire. Unfortunately, the fire department opted to cut power to the rest of the building as a precaution even though the power systems were independent. We are currently waiting for an emergency inspector to arrive to give the all-clear so we can bring most of the servers in Ogden back online. Some servers will have an extended outage as they may require rebuilds due to some water damage. Those builds have a high probability that data is intact.
We would like to thank you for your patience and know that we are doing everything we can to get everyone back online.
If you have a Server at our LA location and need help please email us at [email protected]
Everyone getting OVH'd out here. I will give them credit for having some kind of fire suppression, even if it means our Ryzen boxes could be a little soggy.
WebHostingTalk outage thread has some other details, same with the comment section of the Facebook post they made.
It sounds like they're still waiting for a city(?) inspector to give the go-ahead to turn on most of the DC, which is probably hard to get someone out for on a holiday.
Hoping they get things back online somewhat quickly and not too many people were in the splash zone portion of datacenter.
We (at my work) made the call to kick off our DR plans after 4 hours (being Easter we had more time to wait, since we're closed anyways). Shout out to @Francisco for digging up some slice capacity for us, we're balancing things between that and some Vultr VMs.
We have core data, however a few services are down due to complexity, announced IPs, etc - so we'll be putting together a better (redundant) setup, and basically spending more money to double up important things that are harder to restore.
Comments
https://www.lowendtalk.com/discussion/170830/2021-second-fire-webnx#latest
Ah, sorry, that was buried down page 1 and someone bumped it after I posted. My bad. Mods can merge.
So, where can I rent a high-capacity storage server at the moment?
How many TB do you need and what is your budget?
Budget around $500, for storage : 60-90TB with 2Gbit/s unmetered bandwidth and friendly network speed to Asia country ( mainly china) at least comparable with ogden’s speed.
Let's make this a rescue thread! @Harambe, make it happen!
You can contact us for a custom quote.