New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
PSA: Telia L.A outage
Just a quick public service announcement for those Sys Admins (or technically adept users) scratching their heads.
We have been seeing issues with Telia in L.A since 2:42 AM (39 minutes ago). Confirmed with three L.A Telia sessions / networks. This isn't effecting us (beyond re-routing traffic via other upstreams and PoPs), but it's affecting many other providers.
I see one session behaving out of the 4 I have tested, and that's Choopa L.A.
Thanked by 1Fries
Comments
What problems exactly are being reported?
@Clouvider Peer failure. This is most obvious with networks that are single-homed Telia (such as Voxility L.A for protected ranges) where there is currently no routes.
Voxility had a router failure at the same time.
@qps what @kcaj is echoing isn't Voxility it's Telia. Most people with Voxility protection are Singlehomed Telia meaning effectively they lost all their routes (this is why you multihome people).
It's pretty major since they are still down.
Quadranet was hit by this too https://www.webhostingtalk.com/showthread.php?t=1664693&p=9928122#post9928122
The Voxility NOC has told me that the issue is resolved, but it clearly is not since we are still seeing 60 to 90% packet loss.
@qps Voxility has resolved it via announcing other routes on routes that were previously restricted to Telia (I'm seeing heavy Comcast). Since Telia is most of their transit at DDoS levels there is likely some saturation.
There is heavy packet loss on their internal network between POPs. It could be a capacity issue.
Well you can't blame them since they probably have smaller pipes coming from Comcast but I feel like they probably have other providers they can split the load with but they're not using it, because it can't be marked solved if you get 80% packet lost and high ping.
@Edmond they have NTT in some locations (unsure if L.A) but I doubt they would take any attacks there. Too pricey.
Ya I figured other providers would get upset, but with all the traffic coming into Comcast, it's like a DDoS attack already to Comcast, aside from it's from legit users. I mean it's better than being down, but still you can hardy use it.
@Edmond Realistically it's the same everywhere when a big network in a popular region goes down. Right now Tinet is seeing alot more traffic from us, and could be exposed to bigger attacks (although I wouldn't expect issues since we have sane protection limits). If 300G normally goes over Telia in L.A thats 300G distributed elsewhere right now.
It would be interesting to know how other comcast services in L.A are fairing. Hopefully fine. I'd expect Voxility are handing out nulls like free lollipops too right now anyway (due to reduced capacity).
That 300G goes somewhere else right? Well providers aren't going to buy insanely large bandwidth pipes that are left empty because it cost money, so it's really overloaded now. Wonder what makes Comcast to take the huge DDoS traffic into their networks anyway as it's causing a big load on it.
These are one of the times where people should spend more money and get better protection from some other company that has backup peers like yours, can they even withstand 500G of DDoS anyway, it seems to be a unrealistic number.
@Edmond few single network regions can handle 500G unexpected anyway. Real 500G normally has to be distributed either by PoP or transit. No one (transit) just keeps that around for no reason.