New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
LoL. It wouldn't be much of a backup if it wasn't. It was running on the backup router for 6+ hours today without any issues and as soon as I go to rebuild the primary router (different IPs) it goes offline. I guess the DDOS attack isn't helping much either.
Why not just null the IP ? Even though it shouldn't be going down at all, it dosn't make sense.
Appears to be back.... Nevermind
It's on and off....
This has to be the worst downtime I've had from SD yet
Looks back up
Running BIND9 just fine on 128MB of ram. Solid 12MB memory usage (with about 50 domains, that is).
Also, cPanel DNS Only works fine on 128MB of RAM.
Was back up, then went back down for a few minutes. Is back up now.
KuJoe took networking classes from @Francisco :P
@KuJoe needs to look at who keeps attracting DDoS traffic to his network and sh!tcan them.
Then again, when in doubt have a hardcore network person on call to come deal with the fire. Seems like he got overwhelmed by piling network issues including configs. Easy as pie to happen.
The DDOS attack I mentioned before was just against our website and wasn't impacting our network at all, just made it harder to relay information to our clients (and the Twitter outage wasn't helping either).
The actual outage was caused by... well we still don't know. The problem was fixed by... well again we don't know. The problem would randomly fix itself and then break again without any intervention by us. Our DC claims they have made no changes on their end so it appears that BGP sessions decided to take MLK day off and go elsewhere.
Today we had a weird issue where our primary router nullrouted and IP that it should never need to nullroute and then it died (as in deleted the config file and no services would start up except for SSH). Our backup router took over like it was supposed to and was running find for 3-4 hours by itself. I logged into the primary router and tried to restore the config file that was deleted but restored an older config file and when VRRP switched back it had the wrong BGP info causing yet another outage, I quickly changed the VRRP priority so the backup router would take over again and it did. I went to rebuild the config from scratch on the primary router and while I was doing that the backup router lost it's BGP session with E Solutions. I contacted E Solutions and they were unable to ping the IPs we use for our BGP session so we changed these out and things went back online. I went to rebuild the primary router a second time (fresh Vyatta install) and before I could finish, the BGP sessions stayed up but our IPs stopped responding. After fighting with it for an hour at my desk I decided to drive to the data center and just unplug the primary router from the switches and network, but this did nothing. Our DDOS protected IPs stayed online the whole time as did our BGP sessions but it was not announcing our IPs to the outside world. After building the primary router a few more times (3 in total) it started flashing a hardware error. CPU1 died so I quickly replaced it. Powered it on and CPU2 was reporting bad also. I pulled the drives and the fiber card and put them into a spare OpenVZ node, the array wouldn't import so I built Vyatta again from scratch and got the new router online and working with a very basic config and no IPv6. All was well and everything was back online so I decided to look at the backup router, I unplugged the monitor and keyboard from the new router and all pings stopped. I tried to revive the new router without and luck so I brought the backup router back online so our DDOS Protected IPs (i.e. website) would be online still (hooray for Internal Error 500s from spoofed IPs!). I fight with both routers for another few hours and then decide to start trying different uplinks (we have 2 fiber and 2 copper). I try all of the fiber, no luck except for our DDOS Protected IPs. I try the primary copper uplink and nothing, I try the secondary copper uplink and pings return to normal. At this time our router is online but how it's online, why it's still online, and how long it will remain online is anybody's guess. We will be contacting E Solutions to find a permanent solution to all of this and hope to be done with all of our BGP issues once and for all!
On a positive note, our new OpenVZ node will be arriving in Denver today and we will be setting it up ASAP. Active clients will get first dibs on new VPSs and transfers.
This wall of text is the unofficial version, I will put a more professional and easier to read version on our site as an RFO tomorrow.
I hope you are aware that both NSes of floridawebhosting.net are down
LoL, it looks like our IPv6 static route is killing our BGP somehow. Fun times.
LoL, some backup router
, It sounded like BGP wasn't running.
I assure you it is.
Just notice our VPS are off-line again. I see that https://my.securedragon.net/clientarea.php is shutdown too.
https://twitter.com/SecureDragonLLC <- Updates will be posted here.
Kujoe, I understand. Best of luck with it all !!
https://twitter.com/SecureDragonLLC
Looks like they are dead in the water
http://stats.pingdom.com/f9bng3rw047t
@Jack ah
You been up all night dealing with this? Also pm me when you get everything back online as Buyvm has had way to much downtime and about time to move my master else where.
I took a 3 hour nap after rebuilding our router 2 more times. Waiting on the network techs for the data center to get back to me now.
it's up now, hope it lasts....
Thanks @KuJoe! However, IPv6 still doesn't seem to work.
At the risk of losing our whole network again, we are not enabling IPv6 at this time.
Uh, ok. Any ETA when IPv6 will be enabled again?
don't enable ip6, it's useless anyway...
You mean IPv6 itself is useless?
It is down right now, so?
IPv6 is not useless.
You have two routers, right?
Make router1 route ipv4, router2 route ipv6.