New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
Thanks boss. Can we use it now? Or do wait for reboot?
Go for it.
The reboot will be a couple minutes thing. I'll be onsite and i'll watch it through all of it
Francisco
Wow, all my VMs running like horses now! Thanks!
Thanks for the update, would explain why despite setting up a haproxy load balancer failover on buyvm NY side, I still had a few minutes of downtime - (most was 17 mins with some 2-4min minor downtimes reported).
One strange thing is my SJ now Las Vegas load balancer is reporting NY load balancer's public ip is down (when it's in fact up) and my NY load balancer is reporting LV load balancer public ip + backend LV vpses as down (when it's in fact up).
Anyone know of anything blocking LV and NY buyvm vpses from talking or connecting with each other on their public ips right now ?
LV load balancer status
http://i.imgur.com/fnAMdmdl.png
NY load balancer status
http://i.imgur.com/sFyg8bKl.png
Any idea when node 30 will be back up? This is insane, a 1 day 3 hour downtime...
The status page shows node14 is up, and the control panel shows my vps is running, but I'm not able to connect to it. Trying to ping it and run traceroutes from various locations goes nowhere. Routing problem I guess?
I'm heading to the DC in a few hours to figure out the last of the boxes.
PM me an IP david and i'll check into it. It's very much possible you got wacked with a nullroute during the earlier floods.
Im going to rest a few more hours before karen & ald get up.
Francisco
IPv6 isn't working. Connectivity between lv and ny is also foobar. rDNS = poop.
My VPS's (3) have been up for about 7-10 hours now. Filtered/Awk IP is still not working.
Do you have an ETA Francisco?
Double check that.
I fixed up awknet a few hours ago and frantech.ca loads without issue. My.frantech and buyvm require SQL though and pony is adjusting that all.
Francisco
Everything fine now, my vps online, sql offload online. Wating for restock, i need 1x256 Vegas
Thank you, seems working on my app now.
Jesus guys. Give the man some time to relax.
@Francisco, you're doing a great job. Also you probably have gotten no sleep.
I'll check into the LV<>NY thing.
RDNS should be all back in order. Sorry, the VM wasn't set to autoboot on our internal OVZ node for some retarded reason.
Francisco
Been pinging it for a few hours from a few different places, 100% ping loss atm. Traceroute:
2 te2-2.ccr01.san05.atlas.cogentco.com (38.122.216.1) 4.452 ms 4.685 ms 4.749 ms
3 te4-7.ccr01.san03.atlas.cogentco.com (154.54.83.101) 4.359 ms 4.366 ms 4.353 ms
4 te7-2.ccr01.san01.atlas.cogentco.com (154.54.82.65) 124.259 ms te8-2.ccr01.san01.atlas.cogentco.com (154.54.82.69) 124.301 ms 124.466 ms
5 te0-4-0-4.ccr21.lax01.atlas.cogentco.com (154.54.85.165) 4.592 ms 4.593 ms 4.586 ms
6 te0-6-0-4.ccr21.lax04.atlas.cogentco.com (154.54.84.118) 4.650 ms te0-6-0-1.ccr21.lax04.atlas.cogentco.com (154.54.1.10) 4.196 ms te0-6-0-4.ccr21.lax04.atlas.cogentco.com (154.54.84.118) 4.186 ms
7 38.122.146.18 (38.122.146.18) 3.850 ms 38.104.83.182 (38.104.83.182) 4.282 ms 38.122.146.18 (38.122.146.18) 4.332 ms
8 aiix-40ge.lax615.awknet.com (186.190.208.50) 4.518 ms 3.838 ms 3.949 ms
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
Jesus had a 3 day nap bro.
We're trying. I know it didn't go as planned but I'm proud to say we had no hardware get damaged or anything which is impressive for the distance travelled.
I'm at 60 - 70 hours w/o sleep which doesn't sound like a whole lot, but we've been doing nothing but physical & mental work since then.
I'll keep hammering any last bumps here/there
Francisco
PM me your IP and i'll check into it. I get the feeling you may have been nulled.
Francisco
Rob has said he's been working overttime to get us a V6 BGP session ASAP.
If he can't get it by Friday I'll just setup an HE BGP tunnel to hold us over till then. I hate the tunnels but it's better than nothing.
Francisco
Pm sent. Thanks!
Touche. Either way You can use some rest.
FYI,
Pony is far from a quiet sleeper. He snores so loud he'd wake the lost souls of billing audits past.
Francisco
Same room? Bow chica bow wow.
Ok I'm done. I'll show myself out.
IRC is still in NY? Must be the same issue. I can't connect to the IRC from my LV VPS.
@Francisco what do you get from your upstream, full table or defaultroute ?
I haven't checked my routing table but I told Rob I only needed a default route at this time.
Connecting to the IRC IP will be fucky since the routing entries are all shades of screwed up. Once I'm done the maint on addons01 today I'll migrate IRC back home.
Francisco
Everything's working perfectly on my end. Thanks for all the hard work @Francisco!
@Francisco seems you are not taking defaultroute (at least traceroutes to destination not in the global table seem to be dying before hitting the upstream). In this case to be able to see the NY networks you need either a static default or neighbor x.x.x.x allow-as in
BGP table : 31
BGP node : 832692
BGP route : 454529
I got an idea about what's screwing up
Our NY entries should be in the routing table as well.
Francisco
LAS<>NY connectivity has been addressed
Francisco