Delimiter down?

anton000

Is anyone currently experiencing issues with delimiter in atlanta?
Seems like my servers are unreachable and doing a trace route to my ip doesn't even reach delimiters network.

pbgben

Who

BG32

Yup they down. ilo down too. Grab the pitch forks

DeftNerd

Same here. I'm getting reports from routers way outside the Delimiter network that my IP is unreachable, so I'm guessing it's a problem with a bad BGP announcement.

http://delimiterstatus.com/ reports problems with Atlanta's networking too, but no further details

kingjj

@DeftNerd said:
Same here. I'm getting reports from routers way outside the Delimiter network that my IP is unreachable, so I'm guessing it's a problem with a bad BGP announcement.

http://delimiterstatus.com/ reports problems with Atlanta's networking too, but no further details

Gotta be BGP. Can't get anywhere near their network. Hopefully its not another Power Outage type situation...

raindog308

I still have at least half of that case of popcorn left.

In fact, BRB...

BG32

We are investigating packet loss into our Atlanta datacentre.

i think it may be more than pl.....

netomx

I am greateful I get off from them months ago

ThracianDog

It's back up

anton000

yup back up now. whew

ThracianDog

@anton000 was it a network or power outage?

kingjj

My server was power cycled, back up for 26 minutes. MySql is now refusing to start... yippie.

BG32

My network still down

DeftNerd

@BG32 said:
My network still down

Mine too

ThracianDog

Anyone want to jump in for a refugee offer? @VortexMagnus?

WSS

kingjj

Ofcourse it was power related... what else would it be at Delimiter:

-- 05:25 GMT+0 We’ve been waiting for the official RFO from the building management.

So far we know that the utility feed into the building was hit by lightning at an above ground point. Georgia Power suffered a power outage to the building. Our UPS protected the supply for the first 7 minutes at which point the building managed generators failed to come online.

The generator failure caused a power outage on floors 7, 8 and 17 (our floor) and had to be manually started by building management; then later transitioned back to utility power.

Originally this started as packet loss as we saw loss of service on some of our upstreams transit connections. It turns out the reason for the packet loss was that they had already lost power to their routers. Two of our providers remained online until such time as our UPS had depleted and we subsequently went offline.

The building management had serviced bank of generators providing power to our floor last week without incident. Prior to this they had serviced their other bank of generators and caused disruption to power on other floors in the building.

We have requested a detailed RFO from the building owners as well as their contractors.

Foul

We had a major power outage here last night in Georgia it was statewide for majority.

Severe thunderstorm and it was really bad lol. Trees fell down and such. Swanee(another power company here) had outages everywhere.

MagicalTrain

Do I understand it correctly and they had two seperate generator arrays that failed? Or did they only have one?

How many arrays do DC's normally have? Just a question I ask myself.

raindog308

MagicalTrain said: Do I understand it correctly and they had two seperate generator arrays that failed? Or did they only have one?

I think the first one was in a DC they owned.

miamiconsultant

ThracianDog said: Anyone want to jump in for a refugee offer? @VortexMagnus?

**** (you know who) was somehow able to change refugee offers from 'hey let me help those poor customers out' into something ugly that resembled kicking people when they were down.

MagicalTrain said: Do I understand it correctly and they had two seperate generator arrays that failed? Or did they only have one?

How many arrays do DC's normally have? Just a question I ask myself.

"Building managed generators" = "hopefully the shared generators for the building work"

Check with your DC to make sure they either own the building they are in or they are the master tenant and test and operate the switchboards and generators.

MagicalTrain

@miamiconsultant said:
"Building managed generators" = "hopefully the shared generators for the building work"

This line just makes me unsure if they had two separate generator arrays:

they had serviced their other bank of generators

And I wasnt particularly interested in any specific DC. Just wondering how many generators are standard in one.

miamiconsultant

MagicalTrain said: Just wondering how many generators are standard in one.

Some buildings with 4 or 5 utility feeds near a couple power plants are fine for decades with a single genset, or maybe no gensets in a few special locations.

Most buildings will want an array or 2 or more gensets for redundancy.

This is one of those things though, where 'we have generator power' can look a lot different when you ask your provider to provide specs, diagrams and test reports.

raindog308

MagicalTrain said: And I wasnt particularly interested in any specific DC. Just wondering how many generators are standard in one.

https://en.wikipedia.org/wiki/Data_center#Data_Center_Levels_and_Tiers

Datacenters are rated by tiers, 1 to 4. Tier 4 (the best) means everything is redundant, and they do mean everything - power, cooling, lights, security lights in the parking lot, datacenter staff, alerting systems, feeds from multiple different power and network providers, etc.

Clouvider

And a link from two separate power grids operated by two separate companies. So unfair this can't be done in the UK (National Grid)

J1021

@Clouvider said:
And a link from two separate power grids operated by two separate companies. So unfair this can't be done in the UK (National Grid)

The need for multiple utility feeds is hyped anyway - a solid generator system should negate the need for multiple utility feeds.

lazyt

Maybe it's just coming from a medical background. I've came to have no faith in generators.

It's totally amazing how many ways they can fail.

WSS

@kcaj said:

@Clouvider said:

The need for multiple utility feeds is hyped anyway - a solid generator system should negate the need for multiple utility feeds.

I've seen this fail time, and time again. Hell, in 4 datacenters I was in (and only two I worked at).

J1021

@WSS said:

@kcaj said:

@Clouvider said:

The need for multiple utility feeds is hyped anyway - a solid generator system should negate the need for multiple utility feeds.

I've seen this fail time, and time again. Hell, in 4 datacenters I was in (and only two I worked at).

Then you didn't have a solid generator system..

pbgben

@Clouvider said:
And a link from two separate power grids operated by two separate companies. So unfair this can't be done in the UK (National Grid)

Not strictly true. But soo hard to achieve its financially unviable.

WSS

@kcaj said:

@WSS said:

@kcaj said:

@Clouvider said:

The need for multiple utility feeds is hyped anyway - a solid generator system should negate the need for multiple utility feeds.

I've seen this fail time, and time again. Hell, in 4 datacenters I was in (and only two I worked at).

Then you didn't have a solid generator system..