New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
@Nick_A Phoenix or bust?
Good quotable there @unused
Unsure if Fran knew about the power expansion. Unsure how much anyone gets informed of these potential hazards.
This will end or at least be reduced when Fran gets redundant power. Not that I know, but assumed here that he's not wired for A+B power.
The A+B power was knocked off by the original electrician blamed incident. This time it looks like A leg only was knocked offline. So, one outage would have been prevented (today).
Lots of other folks ran to FIberhub lately. Waiting to hear them called out too
But I won't as we know the issue, the facility, Fiberhub.
Rob needs to take a timeout and stop whatever and regroup and reteam.
What?
Asians might be affected
@Nick_A Reference to this: http://buyvm.net/vegasorbust.txt
We were considering Fiberhub and were in talks, but have put it on hold due to these issues.
That should not happen.
Even me at home have UPS and after a power failure I have the beeping that wakes me up.
They are supposed to have generators and backup circuits coming from them to all power cabinets so only a few racks go offline and only if they have no redundant B leg.
The fact that all the facility went offline is a bad sign and could be because they are a budget DC and dont have all the bells and whistles of premium ones, but power is one of the basics IMO.
Even me at home have UPS and after a power failure I have the beeping that wakes me up.
Oh come on man, I was obviously joking. Of course, even I have a UPS that beeps and annoys the hell out of me when the power goes out.
@KernelSander, I've been in facilities before where a row or two just went power down. Facility didn't notice for a bit.
Comes down to monitoring.
But I will say, has to be mighty freaking eerie to have a complete facility power down unexpected.
Even if mine doesnt or I can turn it off, across the street is some IT consultant which has an alarm that goes off when power fails and is like a big ambulance, cant sleep even with double windows with heat/sound insulation.
They posted an RFO here:
http://www.fiberhub.com/wp-content/uploads/2013/02/FiberhubRFO2-14-13.pdf
TL;DR - Todays issue was fully related to the problems caused on the 12th. True A+B power will be available within 120 days.
With that being said, I'm currently in the process of pricing out some ATS units and we'll just get A+B power feeds. This will let them do maint on either leg of our power without issue.
There will be an increase in our own monthly operating costs but nothing to the client.
http://www.newegg.com/Product/Product.aspx?Item=N82E16812228693&Tpk=PDU ATS
Seem to be pretty decent units, though I'd want the ones with 10 outlets on them.
I'm not happy but neither are they. 3 years of perfect history and then this caused by human error. It happens but in the end we'll be better off than we have ever been in any of our locations. Due to power density rules in coresite & FMT, we never had the option of A+B power w/o having to literally fill our racks to half what we normally do.
Customers are welcome to claim an SLA and we'll just give anyone in good standing a month free.
Francisco
FYI as well, they have UPS', generators as well as a very solid power grid. The issue was at the UPS itself which cut power from the generator/outside source. The only way around this is A+B.
They're fixing or have fixed the issue and we're also putting our money where our mouth is on the A+B.
Francisco
@Francisco I like TL, they did have issues with in various units shipped a few years ago, would just die out after 12-16mo. Can't place any links to this but from what I recall it was a manufacturing error and resolved.
It happens
I know EGI had a couple regular APC strips that crapped out.
Francisco
@pubcrawler may just have an orgy after that
I love APC. I am yet to have an issue with them while previous brands i tried (true, cheap) gave up after max 2 years.
Will it be okay to Aldryic? Do we just ticket? Thanks!
Yep.
Francisco
Yeah, is back up now
I'm still considering moving stuff away from BuyVM though. This is starting to become a trend =(
@pubcrawler said: I'd certainly be replacing the electrician if not the company he/she works for at this point.
I would tend to be considering the same thing. At the same time, bringing in a new vendor who doesn't have the years of experience working with your infrastructure can sometimes be problematic in its own way.
Aside from this, I work in the telecom sector . . . work possibly affecting critical power systems is typically not done in the middle of the day. A public DC might be a bit different as "day" can mean a variety of things depending on where the customers are, but I imagine the majority of FHs customers are domestic.
They do seem to have a clear plan in their RFO. Assuming that gets implemented as outlined, there is future reassurance.
This being said, I don't know all of the details -- so it's hard to make any definite judgments, but on the surface it seems a bit odd.
The work they were doing shouldn't have triggered anything from what Rob has been saying.
Todays issue was a regression from the issue earlier in the week.
Either way, ATS' will be had assuming there isn't some retarded crap our cage restricts. Ryan from coloat mentioned that A+B doesn't count against density readings so lets see what Rob confirms.
Fran
I can understand that, but on the same token (in my experience) "shouldn't have" often means someone didn't cross their Ts or dot their Is during planning. I guess the bottom line is that s**t happens. I can't say I've never created a "shouldn't have" situation that could have been prevented.
A+B just means a B power feed is provided to the racks. You have and use the same amount of power, but balanced between the A and B feed. You don't get 1/2 as much power or any less than what you paid for.
E.g. if you have 20A with 16A usable on A, you still get the 16A usable on A+B, but it might be 8A on A and 8A on B but max is 16A on A or 16A on B at one time.
too harsh..
Wouldn't the ats pdu be the spof?
.
Yes, this is why it's better to get dual PSU per server and then do A+B, but if this isn't available on the servers, ATS would be the most cost benefit solution.
It's true, but they are typically designed/simplified to the point they much more reliable.
I'm not sure if @francisco has dual- network drops for example on both external+internal network - if not the switch is another SPOF.
We don't have dual brocades but the brocade will move onto an ATS at some point.
Dual psu's are an option we are also considering but it would require a lot more work.
I have no doubts Rob will have this resolved it just sucks it happens within the first month :P
It's not just that - any maintenance will be on ALL the servers attached to the ATS PDU and not just per server.