Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Delimiter down?
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Delimiter down?

anton000anton000 Member
edited March 2017 in Outages

Is anyone currently experiencing issues with delimiter in atlanta?
Seems like my servers are unreachable and doing a trace route to my ip doesn't even reach delimiters network.

Comments

  • pbgbenpbgben Member, Host Rep

    Who

    Thanked by 1inthecloudblog
  • BG32BG32 Member

    Yup they down. ilo down too. Grab the pitch forks

  • Same here. I'm getting reports from routers way outside the Delimiter network that my IP is unreachable, so I'm guessing it's a problem with a bad BGP announcement.

    http://delimiterstatus.com/ reports problems with Atlanta's networking too, but no further details

  • @DeftNerd said:
    Same here. I'm getting reports from routers way outside the Delimiter network that my IP is unreachable, so I'm guessing it's a problem with a bad BGP announcement.

    http://delimiterstatus.com/ reports problems with Atlanta's networking too, but no further details

    Gotta be BGP. Can't get anywhere near their network. Hopefully its not another Power Outage type situation...

    Thanked by 1joshb
  • raindog308raindog308 Administrator, Veteran

    I still have at least half of that case of popcorn left.

    In fact, BRB...

  • BG32BG32 Member

    We are investigating packet loss into our Atlanta datacentre.

    i think it may be more than pl.....

  • netomxnetomx Moderator, Veteran

    I am greateful I get off from them months ago

  • It's back up

  • yup back up now. whew

  • @anton000 was it a network or power outage?

    Thanked by 1elgs
  • My server was power cycled, back up for 26 minutes. MySql is now refusing to start... yippie.

  • BG32BG32 Member

    My network still down

  • @BG32 said:
    My network still down

    Mine too

  • Anyone want to jump in for a refugee offer? @VortexMagnus?

  • WSSWSS Member

  • Ofcourse it was power related... what else would it be at Delimiter:

    -- 05:25 GMT+0 We’ve been waiting for the official RFO from the building management.

    So far we know that the utility feed into the building was hit by lightning at an above ground point. Georgia Power suffered a power outage to the building. Our UPS protected the supply for the first 7 minutes at which point the building managed generators failed to come online.

    The generator failure caused a power outage on floors 7, 8 and 17 (our floor) and had to be manually started by building management; then later transitioned back to utility power.

    Originally this started as packet loss as we saw loss of service on some of our upstreams transit connections. It turns out the reason for the packet loss was that they had already lost power to their routers. Two of our providers remained online until such time as our UPS had depleted and we subsequently went offline.

    The building management had serviced bank of generators providing power to our floor last week without incident. Prior to this they had serviced their other bank of generators and caused disruption to power on other floors in the building.

    We have requested a detailed RFO from the building owners as well as their contractors.

  • FoulFoul Member

    We had a major power outage here last night in Georgia it was statewide for majority.

    Severe thunderstorm and it was really bad lol. Trees fell down and such. Swanee(another power company here) had outages everywhere.

  • Do I understand it correctly and they had two seperate generator arrays that failed? Or did they only have one?

    How many arrays do DC's normally have? Just a question I ask myself.

  • raindog308raindog308 Administrator, Veteran

    MagicalTrain said: Do I understand it correctly and they had two seperate generator arrays that failed? Or did they only have one?

    I think the first one was in a DC they owned.

  • miamiconsultantmiamiconsultant Member, Patron Provider

    ThracianDog said: Anyone want to jump in for a refugee offer? @VortexMagnus?

    **** (you know who) was somehow able to change refugee offers from 'hey let me help those poor customers out' into something ugly that resembled kicking people when they were down.

    MagicalTrain said: Do I understand it correctly and they had two seperate generator arrays that failed? Or did they only have one?

    How many arrays do DC's normally have? Just a question I ask myself.

    "Building managed generators" = "hopefully the shared generators for the building work"

    Check with your DC to make sure they either own the building they are in or they are the master tenant and test and operate the switchboards and generators.

    Thanked by 1MagicalTrain
  • @miamiconsultant said:
    "Building managed generators" = "hopefully the shared generators for the building work"

    This line just makes me unsure if they had two separate generator arrays:

    they had serviced their other bank of generators

    And I wasnt particularly interested in any specific DC. Just wondering how many generators are standard in one.

  • miamiconsultantmiamiconsultant Member, Patron Provider

    MagicalTrain said: Just wondering how many generators are standard in one.

    Some buildings with 4 or 5 utility feeds near a couple power plants are fine for decades with a single genset, or maybe no gensets in a few special locations.

    Most buildings will want an array or 2 or more gensets for redundancy.

    This is one of those things though, where 'we have generator power' can look a lot different when you ask your provider to provide specs, diagrams and test reports.

    Thanked by 1MagicalTrain
  • raindog308raindog308 Administrator, Veteran
    edited March 2017

    MagicalTrain said: And I wasnt particularly interested in any specific DC. Just wondering how many generators are standard in one.

    https://en.wikipedia.org/wiki/Data_center#Data_Center_Levels_and_Tiers

    Datacenters are rated by tiers, 1 to 4. Tier 4 (the best) means everything is redundant, and they do mean everything - power, cooling, lights, security lights in the parking lot, datacenter staff, alerting systems, feeds from multiple different power and network providers, etc.

  • ClouviderClouvider Member, Patron Provider
    edited March 2017

    And a link from two separate power grids operated by two separate companies. So unfair this can't be done in the UK (National Grid) :(

    Thanked by 2vimalware netomx
  • J1021J1021 Member

    @Clouvider said:
    And a link from two separate power grids operated by two separate companies. So unfair this can't be done in the UK (National Grid) :(

    The need for multiple utility feeds is hyped anyway - a solid generator system should negate the need for multiple utility feeds.

  • lazytlazyt Member

    Maybe it's just coming from a medical background. I've came to have no faith in generators.

    It's totally amazing how many ways they can fail.

  • WSSWSS Member

    @kcaj said:

    @Clouvider said:

    The need for multiple utility feeds is hyped anyway - a solid generator system should negate the need for multiple utility feeds.

    I've seen this fail time, and time again. Hell, in 4 datacenters I was in (and only two I worked at).

  • J1021J1021 Member

    @WSS said:

    @kcaj said:

    @Clouvider said:

    The need for multiple utility feeds is hyped anyway - a solid generator system should negate the need for multiple utility feeds.

    I've seen this fail time, and time again. Hell, in 4 datacenters I was in (and only two I worked at).

    Then you didn't have a solid generator system..

  • pbgbenpbgben Member, Host Rep

    @Clouvider said:
    And a link from two separate power grids operated by two separate companies. So unfair this can't be done in the UK (National Grid) :(

    Not strictly true. But soo hard to achieve its financially unviable.

  • WSSWSS Member

    @kcaj said:

    @WSS said:

    @kcaj said:

    @Clouvider said:

    The need for multiple utility feeds is hyped anyway - a solid generator system should negate the need for multiple utility feeds.

    I've seen this fail time, and time again. Hell, in 4 datacenters I was in (and only two I worked at).

    Then you didn't have a solid generator system..

Sign In or Register to comment.