New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
Going back the other way had less drastic figures, but still some loss in the same area. It is less problematic now than say yesterday at any rate.
Frankfurt <-> Paris:
Frankfurt <-> Helsinki:
and this is pinging their router directly.. sad that they do traffic shaping/prioritization for other customers..
do you have any smokeping to show? maybe it could be used for evidence to them..
Your snippet doesn't show any packet loss. What does hop 14, 15 etc. show?
Loss from hop 11 onwards to the endpoint. I haven’t checked again today but will run some more tests later and see where it’s at. It seemed better yesterday.
It's irrelevant, it may well be a return route or congestion at the source or destination. Without a full mtr both ways and in the absence of any other data it has to be assumed the issue is not with Telia / Twelve99.
Ahha. The MTRs has been submitted to the affected parties, AKA upstream. Customer, Telia.
You are asking to break privacy of customers publicly. Hope you are aware of that.
Yet you are publicly riding on a brand with no evidence, this is why I’m calling you out and asking for the most basic evidence.
Take me through your login how I’m doing that exactly?
I’m asking you to provide an mtr both ways to back up your claims. None of this will contain your personal Customer information.
Just so @Clouvider can be tagged as a troll i spent the time to find old data from just before routing was changed:
Customer speed issue solved by routing away from Telia and packet loss from first hop on Twelve99 all the way to customer:
This was repeated over dozens of customers. Issue went away when routes were not to Telia anymore.
The Telia issues were reported by a multitude of companies, to huge variety of end points.
Complete connectivity loss was also quite common, too many packet drops.
Is the troll happy now?
And yet you know for 100% certainty that we cannot disclose customer IPs publicly. That would violate their privacy.
Again, this is something you should know. It reveals the IP of the user at a given time and date, and that alone will not identify directly their identity, but can be used to find it out at the minimum the identity of who is paying the bills. At the very least it reveals what kind of services the user behind that IP is interested in, whether direct paying customer or just end user.
I am sorry you don't value your customers privacy the least. We value, and any information which could be utilized to reveal customer identity even potentially we will not publish publicly.
Regardless, since the issue is solved since routing away from Telia for not just us, but for so many others it is clear beyond shadow of the doubt to us Telia was congested once again. It does not matter do you believe it, you cannot fix it.
These days, most people that use Real Debrid use it for cached torrents rather than file sharing sites. Real Debrid download the torrent on their end, and you can then stream it directly from their servers. Popular torrents are all already cached and thus instantly available to stream.
As I said earlier, one way mtr is completely meaningless and proves a grand total of nothing, as a return route can be causing packet loss. The fact you seem to not understand that starts to reflect on their your skill or honesty.
I’m not trolling. I’m just curious whether the issues faced by your Customers are not perhaps of your own making, or their ISP making.
For one we use Telia and consider them top notch. By riding on their brand you ride on ours, and I won’t tolerate that, not without evidence.
No one asked you to provide a Customer IP, you can always obfuscate the last octet.
Either stop accusing some entity in public of wrong doing or be prepared to provide evidence in public too. Given some have been complaining about the connectivity to your service this begins to look awfully lot like trying to get an alibi through raising alarm bells on LET.
This is actually a direct Twelve99 customer:
and so forth and so forth and so forth.
Visible also on total transit usage, and immediately jumped back up once as much routed away from telia as possible.
Our link capacity at the time of these were around 40% utilization, zero errors on the links. Even lower than that from racks to router. Once fixed jumped straight to something like 80-85% for half an hour or so before settling to normal levels
Once again, you should know better. You need to look each direction individually. Yes, another direction can affect the another and does affect, but these links are not half-duplex, they are full-duplex and they are not identical routes typically.
If your issue is from server to user, and that side has an obvious issue you look into that. On the other direction, user to server your only course of action is to drop announces where possible if you need to reroute.
Not much better than trolling. You are insinuating that after decades we have no clue, nor does our upstream providers have any clue, nor does anyone we work with.
At the very least you are trying to gaslight or cause FUD.
You took it personally then? Telia has not been top notch for many years now, and you are probably blinded by the loyalty.
If we obfuscate the last bit, you will just ask for it. and going even all the way to last hop before customer can tell you the general area. There are ISPs which have last route on the local POP and they name them by the area.
Ah! There we go. I don't know what we have done to your company or why you feel so threatened by our existence, but maybe stop the gaslighting.
Meh.
You still haven’t sent the full mtr, both ways.
What are you hiding if Telia is so bad and at fault here?
Are you a Twelve99 network operator?
Meh, there’s no need for me to dig here further. Everyone can make their own mind up regarding the baseless claims posted in this thread 😉
From the partial MTRs you sent above, it's more likely that your upstream's Telia port is congested rather than Telia itself is congested. Your upstream is AS47605 and I don't see that you have direct peering with Telia if I'm not wrong.
Like what @Clouvider said, you need to send Full MTRs for both ways to check which parts could be problematic. Linode has a well-written article about how to read MTR, you can take some time to read it :-)
https://www.linode.com/docs/guides/diagnosing-network-issues-with-mtr/#verify-packet-loss
The Networking Guru has spoken.
This is Telia Paris from:
Path Networks AS396998 (Amsterdam):
@Clouvider 's (Frankfrut):
Two different locations, two different providders.. kinda confused where the issue is now..
@PulsedMedia I'm sorry to say but @Clouvider is simply right, without a proper MTR/tracert from both sides nobody can help you, even Telia can't. On top of that I don't think this is the place to get some sort of support/help.
You don't have to harm the privacy of your customer, simply take an IP from within your network and use a looking glass of the affected providers. If they don't have one ask their NOC to do an MTR.
So, what response did you get so far from Telia?
Telia is giving run around. They pretty much ignore any MTR reports, and apparently lack the means to check for congestion on their own network.
The last hop before Telia network shows 0% packet loss, <0.5ms average latency across many tests. In the same building, and definitively not the bottleneck. That hop has 2x100Gbps to Telia network, well underutilized as well.
This was reproduced by many others in Finland, not just us. Both directions were checked originally, the issue is always on those Telia hops. This was verified from other AS numbers, to other AS numbers by 3rd parties.
This post was not made to seek help, but to let others know. This was already handled by us by the time we did make the post.
Our customer base is very sensitive to network throughput
Will not reroute back to Telia and cause suffering for 10s of thousands -- hence i cannot even redo the MTR. Routing is already way different.
Haven't really noticed this as an end user of several providers that use Telia as an uplink (Clouvider, for instance).
Andwhen I do an mtr to Helsinki(1) I can't reproduce this. Sure, there is some packet loss, but only on routers that will be more busy with actually routing traffic, but not end to end.
This sounds a bit strange to me. One of the largest tier 1 networks in the world, wouldn't have the means to check for and act on congestion? Of course, could be that they just don't see it at their end either and the support engineer acted on that information. What happened when you escalated this through your account manager?
I mean, just derouting Telia seems a bit wasteful. I'm sure you're paying a high monthly base price for that 2x100 Gbps connection. It's better to get it sorted out, really.
1: To an IPv6 address in the AS of your current uplink that's only routed through Telia.
https://bgp.he.net/AS203003
thats your asn right ?
I was once banned for calling Clouvider a cunt and not apologizing to him, so this should give you perspective.
He isn't trolling and he isn't gaslighting you, just requesting proper evidence that is needed for the engineers to actually do something specific to address a specific problem. You haven't been getting effective responses because they haven't been getting proper traces.
You appear to be the only one gaslighting. You took such offense that you were inexperienced and then made the same comment about Telia, except that's far more laughable.
Read again.
The proper evidence has been given to the proper people, as in Twelve99. Situation is on-going and affecting a lot of different parties.
disclaimer; We did not make this particular MTR.
While that mtr looks bad, I still can't say if this is caused by Telia. While there's some packet loss on all Telia routers, the loss is much higher in the target network. Problems could also be caused there or in the reverse path. We can't know that without a reverse mtr.
Some other thoughts: I still don't see an end to end traceroute. Routers tend to reject icmp packets because they give priority to routing. It would be good to know if and what the packet loss on the target address is.
I noticed that you're not a direct customer at Telia either, which only makes it more complex. Looks like you're crusading against Telia but you have no means to check if this is really something in their network.
FWIW, I can't reproduce your mtr on a route that's ~90% the same (Stockholm as starting point, same AS as target). So if it's Telia, it's a pretty local problem.
Let's say that you are right and there is an issue with Telia.
You are not a customer of Telia, so not sure why you complain about them being unwilling to help you. Contact YOUR upstream.
Also, the privacy claims are ridiculous to the extreme. They will fly with your customers from reddit, but not here.
I'm not sure if you're aware @PulsedMedia and im not flaming/blaming you but the routes you've showed from now on, all those routers/hops in between the connection there is some form of ICMP limiting in place by Telia, so counting hops in between as packetloss can't be measured therefor. I've just tested from their LG (https://lg.telia.net/) to some other New Zealand and aussy hops from multiple routers and different destinations, i can't find anything out of the ordinairy..
There’s good chance the OP’s upstream simply has saturated port to Telia. Can’t say for sure without an return mtr and with this level of obfuscation. Notice the loss starting right after the last hop of the OP upstream. If I was to bet - I would bet this. No science here until an return MTR can be provided
Also, why would you even contact Telia? This is a case to discuss with your upstream. It’s between the upstream and Telia.
Yes they do, typically ~1Mbps limit on ICMP. hence end to end MTR.
Indeed, but we have very close relationship with the intermediary, and they've been working really hard with Telia on this. Finally TSIC acknowledged but it did require threat of involving lawyers.
The issue seems to be limited to Twelve99 (not Telia Carrier anymore btw!) hls. Which is kinda ironic that they just built a huge and really expensive DC here. We are considering building our next DC on the neighborhood btw.
Issues has been on going for better part of a year now, but previously our network was routed around, and after the worst days it's been reverted back to that config.
And once again; This issue does not concern just us, but hundreds of other businesses in Finland, some of which are local broadband ISPs.
Uhm we did and we do, and the sole reason for this reason is essentially PSA. We submit ours, and customer MTRs to our upstream, they validate them, then submit the chosen ones to TCIC.
Provide us means to acces customer network for MTRs without revealing specific customer identity then.
You should also check which niche we are in; Going a bit extreme on privacy is a good thing IMHO.
Indeed, all routers have ICMP limiters in place, otherwise their CPUs would drown in the ICMP requests.
Different route, different countries. These are not comparable therefore.
Albeit, more and more evidence points to the issues being isolated to nordics.
Not the case at all, not even close.
I don't know where this assumption came from. This thread was about PSA that there issues on Telia carrier. Really do not appreciate your constant insinuations neither.
We did not, we contacted our upstream, who confirmed the issue concerns all of their customers (weeks prior than it became issue for us), who in turn contacted Telia.
We have very close relationship with this upstream network operator.