New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
@Jacob
If you are burning through BBU(s) you should look at temperature issues within the server. High temps will burn out a BBU in no time,
Primary VPN is on a KVM.
I wish you luck, hope everything gets started again okay.
...particularly when their data centers are typically hundreds/thousands of miles away and they are doing everything via remote hands.
If I was to run a VPS business, I suppose I'd want to have my production nodes, plus a spare for easy failover, plus a locker full of parts that typically fail (disks, cards, memory sticks, etc.)
I suspect a lot of providers run more, ah, "lean".
I suppose that's one of the advantages of renting dedicated servers.
@RyanD Yeah, there is no problems with the cooling though since the data floor is pretty cool, they're all under warranty so no worries on that side but it's starting to get annoying now.
Drives are running at 25-35C, not the greatest, but definitely not the worst.
@Jacob maybe just somebody is supplying you with fake chinese BBUs
@rds100 Pretty sure they say "Made in China" on them, I doubt they're dodgy, or atleast I hope not. :-P
Have you put any temp probes and checked the inside temp?
My thoughts are the same. Thanks @FRCorey for keeping everyone in the loop through this message thread, and through all the emails you've been sending me.
We've managed to restore our website after a 4 hour fsck process recovered the boot partition the LVM partitions were not corrupted. I'm still working with the vendor on the raid cards were going to try a CMOS reset to see if that has any impact, and they're on the phone with LSI to see what they can do to help.
@FRCorey Any updates on kenobi yet? Other then what is on facebook I mean.
:thumbsup:
I strongly think the cards might be still alive.
No luck so far with the CMOS, I have 2 new cards being overnighted, just hoping this blizzard does not pan out UPS are a bunch of chickens when it comes to weather.
So Assuming we get the cards tomorrow we should be up, if the weather prevents delay unfortunately it's going to be monday before I see these cards. Unfortunately as well none of the other ISP's here and the datacenter carry cachecade compatible cards, but they've been pretty supportive of my issue.
At least we have our website back so that's a bit of good news and we've made sure it's doing backups again, but I'm going to run it across 2 servers as soon as we have time to do it.
Still working out Kenobi's problems it's in a fsck process the /vz mount appears to have some corruption that's causing a KP so once that's finished hopefully those customers will be up and running. At least Kenobi's raid card is functioning that was the 3rd server, but were lucky it's not an issue with the card.
@FRCorey Appreciate the updates so far.
But there's just one unanswered question: If Pike and Breck are down, how come Torrey is also down? It's been 16 hours since the shutdown with no updates for that node.
Sorry if this is a stupid question, but how can a power outage so easily kill a RAID Card? I don't understand how pulling power to a card would kill it. Sure, it may cause corruption of some data, but killing the card?
Corey,
You mention Pike and Breck. What about Torrey?
My container on Torrey is offline and the panel wont start it.
Can you post a complete update on what is still down please?
thanks,
PHT
Breck - KVM - Failed Raid Card
Kenobi - OpenVZ - might be fixable just not sure.
Yes - but perhaps it's bigger than this. I have a KVM on Breck and a KVM on Torrey. Both have been down the whole time.
These cards are a bit funny to what I've read...
They don't have a BBU built in, however some models had an alternative to the BBU, which 'securely' and 'safely' stored data onboard that was being x'fered that was physical and could just pick up where it left off if power dropped.
I hope you can get this sorted, such bad timing, goes to show the damage a Power Outage Can do.
Nothing was wrong with Torrey, it was hung on a vps that would not terminate during last night's shutdown but looked like it was running so a reboot was all that it needed.
@SimpleNode
All of our cards have capacitor backed batteries they last for hundreds of days with no charge.
The power dropped hit and dropped probably because of the nature of the failure that's enough to do what it did, trust me several other providers in that building were also having issues, even similar ones with other vendor's cards. At least we will have spares after tomorrow and the RMA's show up.
All new servers ordered will all be the same, but will have a set of critical spares to cover them such as MB/Raid Cards, I have RAM Drives and Nic's, but none of that went bad.
@RyanD Thanks LSI had us try that and a few other things, but they're convinced the cards bricked themselves.
We'll the datacenter had 9260's but they're not models that do CacheCade, but we did try them in the slots to make sure the riser card was not bad etc.
You can also purchase CacheCade license separately.
That's why you don't use BBU. You use CacheVault. It doesn't run out. A bit more expensive, but safer and less hassles.
Any rough estimate when will pike be up?
When will this be fixed?
From reading this thread, the answer probably is when UPS delivers their new cards.
Just got a 'welcome' email for my Kenobi server. Was only running a name server... so would be quite a small backup. Fresh install. So... yeah.
@Kris Kenobi is up?
Yes - the previous data... not so much. Not sure what this means, and living in Denver - the snow storm is really kicking up.
Rule of Thumb: Never use UPS for vital merchandise. If it's Saturday Delivery, use FedEx.
UPS... It's drizzled before (distribution center was 15 minutes away) and had merchandise delayed for adverse weather conditions.
Any ETA for Breck?
@Kris my box in Kenobi is offline