New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Racknerd Los Angeles (Multacom) offline
Racknerd's been down in Los Angeles for over two hours now. According to their status page, the issue stems from cooling problems at their data center provider, Multacom.
If I’m counting right, this is the third long outage at this site in the past year. Considering Multacom (under Edge Centers) also owns Quadranet - which hasn't exactly had a smooth ride lately - it’s hard not to wonder if we’re starting to see a bigger pattern here.
Comments
cooling problem?
Oh my god, the official Multacom website won't even open
Hi @cu_olly -- appreciate you bringing this up. Here’s the official status page link for reference: https://status.racknerd.com/incident/3112
To quickly summarize -- this is stemming from a cooling issue at the facility (Multacom), which has caused disruptions to some core network infrastructure. Based on what’s been communicated to us, it appears to be a building-level chiller failure, affecting not just Multacom but other tenants in the Aon Center as well.
The building engineers are on-site actively investigating, and with regards to Multacom, they’ve been very responsive and transparent to us throughout. They’re also working directly with their own external cooling vendors to help bring things back to normal as soon as possible.
We’ll continue keeping our status page updated in real-time, and appreciate everyone’s patience while this gets sorted.
Hope it will be resolved soon🤞
Fingers crossed everyone gets back online as soon as possible
regards
Nodeseek prediction is at least a week.
Back online. Uptimerobot says 3:20 hrs down. Anyhow good job guys to bring it back online.
Like this?
Hi @cu_olly -- while not the most ideal solution, as a temporary measure, the portable A/C units did help bridge the gap and allowed us to get service restored for a good number of physical devices 1-2 hours sooner than we otherwise could have. I can totally respect that Multacom made an effort towards doing everything within their control, while waiting on the building (Aon Center) to restore/remedy the main chillers.
That said, the building chillers have now been fully restored -- and with that, the temporary units are no longer needed. Cooling conditions inside the facility are stabilizing well.
The good news is that service is restored and back to normal for over 99%+ of our footprint in Los Angeles DC-02 -- just 3x physical nodes are still offline. While the rest of the affected nodes came back online without issue, these 3 specific nodes didn’t power back on and will need to be investigated physically by a datacenter technician.
Dustin, the issue isn’t simply that the response was inadequate; it was both insufficient and untimely. This marks the second such failure within a year, doesn’t it? Proactive measures should have been implemented after the first disruption to ensure operational continuity in the event of a recurrence. A retrospective look at Multacom and Aon’s history in this location reveals a recurring pattern of infrastructure failure, particularly around power and chiller systems. This wasn’t unforeseeable.
It’s like living in a seismically active region and choosing not to install emergency ladders, reinforce buildings, or revise safety codes, despite the well-known risk of earthquakes. At that point, the issue isn’t unpredictability, it’s negligence. We’re not dealing with a freak accident; we’re witnessing a systemic failure to act where action was clearly warranted.
The solution is move to other data center I think.
man you haven't had fun til you've been one of the noc monkeys moving chillers around because the ac is out. woohoo. i do not miss that tier of work.
I'll order when racknerd open antarctica DC or submerge their DC into the sea (like Microsoft) to avoid any future cooling problem /s
Hi @cu_olly — We understand the importance of uptime, and also equally share the same frustration when such rare events happen. We truly don’t want to experience them, but just as with any provider, it’s always a possible risk. One thing for sure, we’re always prepared, and here for our valued customers Delivering prompt updates, providing solutions, and assistance is what you can expect from RackNerd.
To be clear we put our customers first, and have a strong reputation of delivering performance, quality service and excellent pricing. With that being said, I’m not sure if you’re familiar with DTLA datacenters, but almost all providers rely on building infrastructure to a degree, and are subject to CAM. It’s very common, in short, you’re not always in control of everything, despite your redundancy preparations and implementation. Events can and will happen, it’s the nature of the industry. Such as today’s - whereby building redundancy for their chillers failed Multacom and other tenants, and as such failed downstream customers such as RackNerd.
Rest assured, events such as today always call for additional measures to be explored or introduced, with the goal in mind to prevent reoccurring events. Multacom has done right by us, and they’ve yet to let us down in conversations we’ve had that aim towards expansions, and stability. We believe in transparency, and have visited the facilities many times, have met with management (including the past/new) multiple times, and have good dialogue. This ultimately benefits RackNerd’s customers. On a similar topic, for customers open to other LA locations of ours, etc, we’re open to having direct conversations too. Anything we can do to help, we always will. We also have another presence in our Los Angeles DC-03 location which is based out of an entirely different facility altogether (1200 W 7th).
We still have work to do because of this event and will work through it, but in the meantime we sincerely apologize to all affected valued customers of RackNerd.
You aren't kidding! Despite being in sales for a majority of my career, I dabbled and helped support a lot, including in situations as you described at previous employers. We also, once stepped in and assisted a competitor (on a different floor) who experienced a massive water leak, with the hope for minimal damage to their customers. In short, the undesired happens.
Thank You for being so understanding, during these difficult times @Intelpentiumm As always, Thank You for trusting us with your business, and we hope to restore confidence in RackNerd's services.
A status page that actually shows real status. Ah the little things that make you happy.