Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In Register

Quick Links


Shells Virtual Desktop
BMail.ag - Secure Email Service
Server.net
CPLicense.net
VPS Server
Buy VPN
Vultr
VMs for AI
HostDare
HostDare
ReliableSite White-Label Dedicated Hosting for Resellers
InterServer VPS
BMail.ag - Secure Email Service
Best VPN
High-Performance Bare Metal Server Solutions
Karvl.com
Server Mania Cloud Hosting
DataWagon Hosting
AlphaVPS Hosting
Evoxt.com
Clouvider
VPS Hosting with NVMe
Residential IPs in the US & 4G Mobile Proxies in EU & US with Unlimited Bandwidth
ReliableSite White-Label Dedicated Hosting for Resellers
Rabisu - Hosting Solutions
Shells Virtual Desktop

Categories

In this Discussion

Home › Providers › Outages
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Fire in electrical room of Dedicated.com in New York

1235

Comments

  • FranciscoFrancisco Top Host, Host Rep, Veteran

    @MikePT said:

    @edunog77 said:
    Nice @toast555 It would be amazing. And if you can tell us what to see there, that would be great. This whole situation, in addition to being stressful, generates a breach of trust. The installations are fantastic but I'm leaving this DC.

    Olá! From Portugal :)

    Incidents like this one may happen to any DC. You shouldn't leave them just because this happened. The fire marshal has to be sure they comply to the safety rules, but again, this may happen to any other DC l.

    2 fire issues with the same DC vendor in 2 months is a pretty shit track record though.

    Francisco

  • MikePTMikePT Veteran

    @Francisco said:

    @MikePT said:

    @edunog77 said:
    Nice @toast555 It would be amazing. And if you can tell us what to see there, that would be great. This whole situation, in addition to being stressful, generates a breach of trust. The installations are fantastic but I'm leaving this DC.

    Olá! From Portugal :)

    Incidents like this one may happen to any DC. You shouldn't leave them just because this happened. The fire marshal has to be sure they comply to the safety rules, but again, this may happen to any other DC l.

    2 fire issues with the same DC vendor in 2 months is a pretty shit track record though.

    Francisco

    I missed the story about the first fire then... In that case, yep. But could also be bad luck.

  • FranciscoFrancisco Top Host, Host Rep, Veteran

    @MikePT said:

    @Francisco said:

    @MikePT said:

    @edunog77 said:
    Nice @toast555 It would be amazing. And if you can tell us what to see there, that would be great. This whole situation, in addition to being stressful, generates a breach of trust. The installations are fantastic but I'm leaving this DC.

    Olá! From Portugal :)

    Incidents like this one may happen to any DC. You shouldn't leave them just because this happened. The fire marshal has to be sure they comply to the safety rules, but again, this may happen to any other DC l.

    2 fire issues with the same DC vendor in 2 months is a pretty shit track record though.

    Francisco

    I missed the story about the first fire then... In that case, yep. But could also be bad luck.

    Evocative had a fire in LAX that knocked out a row or two of cabs.

    Francisco

    Thanked by 1PureVoltage
  • edunog77edunog77 Member

    Without a doubt I agree with you @MikePT but in the case in question I no longer have enough confidence to believe that there is no technical problem. So far they have not explained what actually happened.

    Thanked by 1MikePT
  • MikePTMikePT Veteran

    @Francisco said:

    @MikePT said:

    @Francisco said:

    @MikePT said:

    @edunog77 said:
    Nice @toast555 It would be amazing. And if you can tell us what to see there, that would be great. This whole situation, in addition to being stressful, generates a breach of trust. The installations are fantastic but I'm leaving this DC.

    Olá! From Portugal :)

    Incidents like this one may happen to any DC. You shouldn't leave them just because this happened. The fire marshal has to be sure they comply to the safety rules, but again, this may happen to any other DC l.

    2 fire issues with the same DC vendor in 2 months is a pretty shit track record though.

    Francisco

    I missed the story about the first fire then... In that case, yep. But could also be bad luck.

    Evocative had a fire in LAX that knocked out a row or two of cabs.

    Francisco

    Daaamn.

    @edunog77 said:
    Without a doubt I agree with you @MikePT but in the case in question I no longer have enough confidence to believe that there is no technical problem. So far they have not explained what actually happened.

    I totally get it.

  • edunog77edunog77 Member
    edited July 2023

    New Update:

    DC Update:

    Our onsite team is currently bringing our UPS Systems online. We now have UPS-4 and UPS-R online. While bringing up UPS-1, we ran into an issue and we are unable to bring it online. We have engaged with our vendor and are finding a workaround to deliver power downstream to customer cabinets.

Due to the issue with UPS-1, we will not be automatically powering up all customer cabinets. We will be reaching out to you to let you know if you are part of the group of customers on unprotected/non-UPS backed utility power so that you can make the decision whether to energize your cabinets at that time.

We are currently at 50% for completion toward bringing the site back online and the revised ETA for bringing up the critical infrastructure systems is approximately 3 hours. The current time frame for when clients will be able to come back onsite is approximately 10:30PM EDT.

We are currently sourcing materials to bring our fire system fully online and do not have an ETA for completion. Because of this and fire marshal compliance we will only be allowed to have supervised escorted customer access when we finish bringing up the critical infrastructure systems. We will have additional personnel onsite to assist us with this escort policy.
  • edunog77edunog77 Member

    @toast555

    revised ETA for bringing up the critical infrastructure systems is approximately 3 hours. The current time frame for when clients will be able to come back onsite is approximately 10:30PM EDT.

  • PureVoltagePureVoltage Member, Patron Provider

    Insert "It's the final countdown"

    Hopefully for the sake of others this timeline is right.

  • edunog77edunog77 Member

    I hope Kamatera is not on UPS1. Or if you are authorizing a reconnection as specified.

  • srch07srch07 Member

    @aj_potc said:

    @moodwriter said:
    Would be nice to have somebody to setup live stream from some building nearby to check whats going on realtime.

    I agree. I would love to see this famous UPS room that supposedly needed all of the cleaning.

    Don't forget the famous room needed 36 hours of cleaning, additional man power, and fire marshal was pissed over it asking for more cleaning.

  • EthernetServersEthernetServers Member, Patron Provider

    Our gear is back online.

    Some nodes are having file system issues. Investigating.

    Thanked by 2alilet titus
  • jon617jon617 Veteran
    edited July 2023

    My VPS at nfoservers came back online around 7pm EDT. Looks like Evocative is getting things turned back on now.

  • titustitus Member
    edited July 2023

    My VPS's with DediPath and EthernetServers got back online. :)
    Edit: VPS from VirMach also online.

  • EthernetServersEthernetServers Member, Patron Provider

    We've repaired XFS damage on our cPanel platform, which is now back online.

    FSCK's are running on Piano and Fife.

  • conceptconcept Member

    It's interesting to see the amount of hosts all using the same datacenter. I wonder if there will be expansion to other datacenters in same area. I've noticed that Coresite and Equinix both have datacenters right next to INAP/Evocative one.

    Thanked by 1yoursunny
  • basskillinbasskillin Member

    20min humm

  • edunog77edunog77 Member

    KAMATER IS ON

  • aliletalilet Member

    VirMach is back online!

  • aj_potcaj_potc Member

    @concept said:
    It's interesting to see the amount of hosts all using the same datacenter. I wonder if there will be expansion to other datacenters in same area. I've noticed that Coresite and Equinix both have datacenters right next to INAP/Evocative one.

    Yes, it is interesting. Equinix has four datacenters in the area, Evoque operates one across the street from Evocative, and there are a few more smaller ones to the west, all in the same general industrial area. I wonder which providers would have been affected if any of those others went down...

  • aliletalilet Member

    According to Hetrix Tools, the downtime was 2 days, 2 hours, 49 minutes.

  • emghemgh Member, Megathread Squad

    @alilet said:
    According to Hetrix Tools, the downtime was 2 days, 2 hours, 49 minutes.

    Average timespan of their technicians after this just went down by 13 years

    Thanked by 1CheepCluck
  • edunog77edunog77 Member

    anyone recommends a redundancy service?

  • SirFoxySirFoxy Member
    edited July 2023

    @concept said:
    It's interesting to see the amount of hosts all using the same datacenter. I wonder if there will be expansion to other datacenters in same area. I've noticed that Coresite and Equinix both have datacenters right next to INAP/Evocative one.

    Wonder which one is the cheap one?

    Thanked by 3PureVoltage alilet fluffernutter
  • PureVoltagePureVoltage Member, Patron Provider

    @SirFoxy said:

    @concept said:
    It's interesting to see the amount of hosts all using the same datacenter. I wonder if there will be expansion to other datacenters in same area. I've noticed that Coresite and Equinix both have datacenters right next to INAP/Evocative one.

    Wonder which one is the cheap one?

    Reason we didn't go there.

    Thanked by 1SirFoxy
  • DennyDenny Member
    edited July 2023

    Ethernetservers online, Hawkhost still down.

  • hosthatchhosthatch Patron Provider, Top Host, Veteran

    It's almost as if people who use Psychz (you should really see their DC to believe it, it beats out any Equinix facility /s) or the people who should be getting "who has had the most power outages in the last 5 years" award are now sharing expert opinions on this unfortunate event.

    Thanked by 1fluffernutter
  • tdaddysimitdaddysimi Member

    I rent space from dedipath and I am still offline :( 2 days 7 hours, 58 minutes, and counting. I get you can't plan for the situation buts its disheartening that power is restored and I'm still down. I have a mikrotik router that should immediately wake up once power is restored, so I am guessing its the power going into my rack or something along that. I'm hoping their staff finds something tomorrow morning, as i'm states away.

  • tdaddysimitdaddysimi Member

    Thank you all for the updates btw. This thread has been a fantastic resource to follow.

    Thanked by 2cocotte67 Zeniic
  • chipchip Member
    edited July 2023

    @alilet said:
    According to Hetrix Tools, the downtime was 2 days, 2 hours, 49 minutes.

    I've got 2 days 4 hours so we are all around the same downtime

    This is just bad luck but I think ethernet servers and dedicated.com (I think there are other providers in this thread too) did the best they could under the circumstances to keep their customers informed .... hey even OVH had a fire a few years ago and I think it did substantial damage from what I can remember

  • mansoormansoor Member

    @Denny said:
    Ethernetservers online, Hawkhost still down.

    Hawk host is back online. As their status said, first the DC had to be powered on, then co-located servers could be powered on and then Hawkhost had to spin up their whole cloud environment.

Sign In or Register to comment.