Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Twelve99 / Telia Carrier: Huge congestion and routing issues since 24th or 26th of Nov
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Twelve99 / Telia Carrier: Huge congestion and routing issues since 24th or 26th of Nov

PulsedMediaPulsedMedia Member, Patron Provider
edited December 2021 in General

They are really badly congested, and i guess that has resulted them also doing some very wonky routings as of late as well. Not just congestion, but actually dropping so much packets that connections drop.

Talked with other businesses and network operators, and it is at least concerning the whole northern europe (sweden, norway, finland, estonia). Have not specifically studied how their network works within central europe, but considering the peculiar routings reported i guess not much better.

Twelve99 has been off from our mix for ~6months or so now because they are shaping speeds on per network basis (IP subnet, or per AS#). Couple days back there was a mistake and Twelve99 was risen back to standard BGP preference -- and so did drop 1/3rd of our bandwidth utilization too. Twelve99 is back on low priority now for us: https://pulsedmedia.com/clients/announcements.php?id=538

Not sure if Twelve99 will be on the mix at all anymore within a few months. When Cogent starts to look like premium top notch quality .... your network has a severe issues.

Funny enough, Cogent again called us (me, Aleksi) if we'd be interested. I asked how much they'd be willing to pay for us to peer with them, and the sales man quickly backed away lol

«13

Comments

  • Noticing the same thing on our end too, really disappointed in Telia.

  • PulsedMediaPulsedMedia Member, Patron Provider

    @LiliLabs said:
    Noticing the same thing on our end too, really disappointed in Telia.

    Yea they were once regarded as top notch. These days? Not at all.
    Before the first rumours of telia carrier being sold to an investment firm it was clear they had congestion issues and they refused to invest. Since then, they not only refuse to invest in their network BUT rather introduced bandwidth shaping.

    We first thought it was just us being targeted, but another operator confirmed they had the same issue too.

  • Routes to servers I have in Europe are fucked since last week and MTR shows max loss with one of the telia servers

  • yoursunnyyoursunny Member, IPv6 Advocate

    What are the major IPv4 transit providers, ranked from worst to best?
    I used to think Twelve99 is better than Cogent, and was avoiding provider/location when I see Cogent in traceroute.

  • @yoursunny said:
    What are the major IPv4 transit providers, ranked from worst to best?
    I used to think Twelve99 is better than Cogent, and was avoiding provider/location when I see Cogent in traceroute.

    Cogent has gotten a LOT better in recent times, still not too good but definitely not bad. I see no issues with it being in the blend, as long as it's not singlehomed.

    Thanked by 1pbx
  • @yoursunny said:
    What are the major IPv4 transit providers, ranked from worst to best?
    I used to think Twelve99 is better than Cogent, and was avoiding provider/location when I see Cogent in traceroute.

    NTT at the top?

  • NyrNyr Community Contributor, Veteran

    @yoursunny said:
    I used to think Twelve99 is better than Cogent, and was avoiding provider/location when I see Cogent in traceroute.

    Having Cogent in a blend can be generally a good idea for many providers. Being single homed to any carrier is not smart, but a diverse blend can include Cogent and be considered a good quality network.

    Also, Cogent is not that bad nowadays, better than some "premium" carriers in some regards.

  • @PulsedMedia said:
    Funny enough, Cogent again called us (me, Aleksi) if we'd be interested. I asked how much they'd be willing to pay for us to peer with them, and the sales man quickly backed away lol

    That's not how you convince them to fix their shit and win back your business. Just straight up say "I can't do business with you until you fix this, this and this".

  • @PulsedMedia said:
    They are really badly congested, and i guess that has resulted them also doing some very wonky routings as of late as well. Not just congestion, but actually dropping so much packets that connections drop.

    Talked with other businesses and network operators, and it is at least concerning the whole northern europe (sweden, norway, finland, estonia). Have not specifically studied how their network works within central europe, but considering the peculiar routings reported i guess not much better.

    Twelve99 has been off from our mix for ~6months or so now because they are shaping speeds on per network basis (IP subnet, or per AS#). Couple days back there was a mistake and Twelve99 was risen back to standard BGP preference -- and so did drop 1/3rd of our bandwidth utilization too. Twelve99 is back on low priority now for us: https://pulsedmedia.com/clients/announcements.php?id=538

    Not sure if Twelve99 will be on the mix at all anymore within a few months. When Cogent starts to look like premium top notch quality .... your network has a severe issues.

    Funny enough, Cogent again called us (me, Aleksi) if we'd be interested. I asked how much they'd be willing to pay for us to peer with them, and the sales man quickly backed away lol

    Downloading from pulsedmedia for more than a week has been very spotty, flutuates betweens 20mbps to 200mbps all the time, no matter what time, no matter if through ftp, web, proxied through cloudflare, with VPN or without VPN.

    I hope it becomes better, right now I rather seed with my own server at home, it takes ages to download stuff.

    Download to pulsedmedia servers seems to not suffer at all.

  • yoursunnyyoursunny Member, IPv6 Advocate

    @tr1cky said:

    Downloading from pulsedmedia for more than a week has been very spotty, flutuates betweens 20mbps to 200mbps all the time, no matter what time, no matter if through ftp, web, proxied through cloudflare, with VPN or without VPN.

    I stream from PulsedMedia on Android VLC player.
    Last week I can watch 720p content, via VPN in OVH Roubaix.
    This week I cannot watch anything.

    CBS and Fox have no new episodes until January.
    I canceled Amazon Prime last month.
    Maybe I'll have to settle with YouTube until Twelve99 is fixed.

    I hope it becomes better, right now I rather seed with my own server at home, it takes ages to download stuff.

    What if FBI knocks on your door?
    Streaming is not a problem because content is not downloaded but only cached.
    Having the file locally and you are going to jail.

  • If this ain't about pron - I don't get it. Theres a thing called Kodi and its best friend - real debrid. Its 2021 (almost 2022). Who downloads stuff nowadays?! Stream stream stream.

  • yoursunnyyoursunny Member, IPv6 Advocate

    @NobodyInteresting said:
    If this ain't about pron - I don't get it. Theres a thing called Kodi and its best friend - real debrid. Its 2021 (almost 2022). Who downloads stuff nowadays?! Stream stream stream.

    Yes we love Kodi.
    Just look at those triceps and traps!

    Cody on Big Brother

    However, the question remains:
    How to stream from PulsedMedia at 720p if network can only transfer 7Mbps?

    CBS content is available 240p, 360p, 480p, 720p, and 1080p.
    Torrented content has only one resolution.
    I wish it's 360p, but most come as 720p or 1080p.

  • Google, level3 those are the carriers I think of. Telia will solve their problems.

  • PulsedMediaPulsedMedia Member, Patron Provider

    @TimboJones said:

    @PulsedMedia said:
    Funny enough, Cogent again called us (me, Aleksi) if we'd be interested. I asked how much they'd be willing to pay for us to peer with them, and the sales man quickly backed away lol

    That's not how you convince them to fix their shit and win back your business. Just straight up say "I can't do business with you until you fix this, this and this".

    we have repeatedly and multiple times. Sales people keep changing there, and always when they got a new one, the new one calls. So receive 1-2 calls annually from there.

    You cannot get sick business culture fixed, the problem starts from the very very top with Cogent.

    Absolutely will not give them any money, and they'd have to cover the operating expenses to peer with them, and even then i might have to ask for money to peer with them as i am certain they find a way to screw with us at that point, one way or another.

    @LTniger said:
    Google, level3 those are the carriers I think of. Telia will solve their problems.

    Twelve99 has been having severe issues for ~3 years now. I would not hold my breath until they get their act together. Even if they fix the congestion, they still do major shaping.

    @yoursunny said: How to stream from PulsedMedia at 720p if network can only transfer 7Mbps?

    We routed around Twelve99 heavily already, next step is to simply drop them completely if things do not improve. You might want to open a ticket so we can look at your IP routing and add it to the "evidence pool".

    Let's just say that the situation with Twelve99 is now well, sternly worded communication level now.

    @tr1cky said: Downloading from pulsedmedia for more than a week has been very spotty, flutuates betweens 20mbps to 200mbps all the time, no matter what time, no matter if through ftp, web, proxied through cloudflare, with VPN or without VPN.

    That is probably load on that particular server, and not related to this. We have like 1ms latency to Cloudflare and route does not go anywhere near Twelve99 :)

  • TimboJonesTimboJones Member
    edited December 2021

    @NobodyInteresting said:
    If this ain't about pron - I don't get it. Theres a thing called Kodi and its best friend - real debrid. Its 2021 (almost 2022). Who downloads stuff nowadays?! Stream stream stream.

    Backwards. Why stream when you can download? You guys don't like smooth fast forward and rewind? What a waste of money and time dealing with share sites, file size caps and removed files.

  • @PulsedMedia opened a ticket! Been getting absolute shit speeds for the past two weeks. Hopefully you can help with a workaround temporarily.

  • @TimboJones said:

    @NobodyInteresting said:
    If this ain't about pron - I don't get it. Theres a thing called Kodi and its best friend - real debrid. Its 2021 (almost 2022). Who downloads stuff nowadays?! Stream stream stream.

    Backwards. Why stream when you can download? You guys don't like smooth fast forward and rewind? What a waste of money and time dealing with share sites, file size caps and removed files.

    Hence why I mentioned real debrid. No limit on size, super fast, and gives you enough links to make sure you can always watch.

    Thanked by 1TimboJones
  • stratagemstratagem Member, Host Rep

    Have noticed some issues here too. Lots & lots of packet loss at some of their PoPs

  • Even now Spotify stutters so bad for me. Quick MTR and there are packet losses after Telia. Have to use VPN just for it to streams properly.

  • ClouviderClouvider Member, Patron Provider

    Did not observe similar issues or heard similar complaints and we use Telia in all our locations globally.

    mtr both ways, please?

    Thanked by 1Edding
  • TamerciagaTamerciaga Member, Host Rep

    We did not encounter any issues so far with Telia, in the Netherlands. In fact, it has been quite premium so far, compared to their colleagues.

    @PulsedMedia said: Funny enough, Cogent again called us

    This is nothing new, they have been spamming us for a long time.

  • Is this issue from Telia only? Because, 3 weeks ago my latency to all my ISP's upstreams screwed. For example, but not limited to Cloudflare, I used to get routed through Marseille with total 40-44ms, now, this is what I get:

  • PulsedMediaPulsedMedia Member, Patron Provider

    @BarkingIron said:
    @PulsedMedia opened a ticket! Been getting absolute shit speeds for the past two weeks. Hopefully you can help with a workaround temporarily.

    Some work around was done already.

    That being said, there will be new options by end of January.

    @Clouvider said:
    Did not observe similar issues or heard similar complaints and we use Telia in all our locations globally.

    mtr both ways, please?

    Lucky you. Very lucky you.

    This also highlights how every route is different, and if you guys use Telia as your primary provider everywhere they might classify your connections differently too; ie. not shaped between your sites.

    @Tamerciaga said:
    We did not encounter any issues so far with Telia, in the Netherlands. In fact, it has been quite premium so far, compared to their colleagues.

    @PulsedMedia said: Funny enough, Cogent again called us

    This is nothing new, they have been spamming us for a long time.

    Yea we get 1-4 calls annually from them, always new sales person.

  • ClouviderClouvider Member, Patron Provider

    @PulsedMedia said:

    @BarkingIron said:
    @PulsedMedia opened a ticket! Been getting absolute shit speeds for the past two weeks. Hopefully you can help with a workaround temporarily.

    Some work around was done already.

    That being said, there will be new options by end of January.

    @Clouvider said:
    Did not observe similar issues or heard similar complaints and we use Telia in all our locations globally.

    mtr both ways, please?

    Lucky you. Very lucky you.

    This also highlights how every route is different, and if you guys use Telia as your primary provider everywhere they might classify your connections differently too; ie. not shaped between your sites.

    I think that’s extremely unlikely.

    Do you have an mtr both ways to show where the congestion actually is ?

  • edited December 2021

    @Clouvider said: Did not observe similar issues or heard similar complaints and we use Telia in all our locations globally.

    I vouch for this one. Currently using Clouvider due to having Telia in the mix, not seeing any network issues (heavy packet loss or bandwidth drop) from Asia <-> Europe (Frankfurt in this case).

    Maybe issue is within Europe only? This is smokeping to Singapore from Frankfurt last 7 days.

    Thanked by 1Clouvider
  • PulsedMediaPulsedMedia Member, Patron Provider

    This shows that the network does not route via a single point. As a provider @Clouvider should know this -- every single route is different.

    MTR: I cannot reveal customer IPs. Very few customers are willing to do proper MTR as well to the server -- and the direction for us is from server to customer.

    This is not affecting just us, but everyone in the nordics, not just dozens of companies but hundreds of companies.

    Certainly a route from singapore to frankfurt uses DIFFERENT route than say Finland to Norway, or Sweden to Frankfurt.

  • ClouviderClouvider Member, Patron Provider
    edited December 2021

    I’m aware how the Internet works :-).

    I see. Well that is not very transparent of you given you are blaming publicly a major global carrier.

    With no MTR to show I say there’s no congestion in Telia core and the issue lies elsewhere, either on A or Z end unless proven otherwise.

  • stratagemstratagem Member, Host Rep
    edited December 2021

    @clouvider here's a snippet from one we got sent. Customer -> Us - Paris was the one giving me issues.

    10.|-- ffm-bb1-link.ip.twelve99. 0.0% 10 225.4 225.5 224.8 226.2 0.5
    11.|-- prs-bb1-link.ip.twelve99. 90.0% 10 246.8 246.8 246.8 246.8 0.0
    12.|-- ldn-bb1-link.ip.twelve99. 20.0% 10 242.6 258.5 242.6 286.8 18.5
    13.|-- ldn-b9-link.ip.twelve99.n 10.0% 10 233.9 238.6 233.9 271.2 12.2

  • ClouviderClouvider Member, Patron Provider
    edited December 2021

    Snippet is not enough; traffic directed at the routers is rate limited. Also, mtr is needed both ways to semi-accurately pin point the issue.

Sign In or Register to comment.