New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Traceroute being slow
gsrdgrdghd
Member
I have an issue with traceroutes being very slow. Here are some examples:
time traceroute google.com traceroute to google.com (173.194.35.35), 30 hops max, 60 byte packets 1 194.14.179.252 (194.14.179.252) 0.082 ms 0.025 ms 0.022 ms 2 gw-cdlan-2.prometeus.cdlan.net (217.171.46.253) 0.347 ms 0.409 ms 0.454 ms 3 ibgp-gw-core-a.cdlan.net (217.171.32.129) 0.712 ms 0.783 ms 0.850 ms 4 google.mix-it.net (217.29.66.96) 0.243 ms 0.278 ms 0.226 ms 5 209.85.249.54 (209.85.249.54) 0.352 ms 0.280 ms 0.258 ms 6 209.85.241.67 (209.85.241.67) 0.598 ms 1.229 ms 1.261 ms 7 mil01s17-in-f3.1e100.net (173.194.35.35) 0.389 ms 0.385 ms 0.498 ms real 0m15.567s
time traceroute -n google.com traceroute to google.com (173.194.35.35), 30 hops max, 60 byte packets 1 194.14.179.252 0.041 ms 0.018 ms 0.015 ms 2 217.171.46.253 0.405 ms 0.454 ms 0.514 ms 3 217.171.32.129 0.365 ms 0.450 ms 0.684 ms 4 217.29.66.96 0.237 ms 0.238 ms 0.252 ms 5 209.85.249.54 0.302 ms 0.366 ms 0.273 ms 6 209.85.241.67 0.868 ms 0.966 ms 1.063 ms 7 173.194.35.35 0.297 ms 0.269 ms 0.307 ms real 0m0.005s
time traceroute6 google.com traceroute to google.com (2a00:1450:4002:802::1000) from xxx, 30 hops max, 16 byte packets 1 2a00:dcc0:eda:89::252 (2a00:dcc0:eda:89::252) 0.22 ms 0.367 ms 0.137 ms 2 gw6.prometeus.net (2a00:dcc0:eda:2::1) 0.467 ms 0.431 ms 0.411 ms 3 2001:b60:0:c100::1 (2001:b60:0:c100::1) 0.938 ms 0.953 ms 0.851 ms 4 google-v6.mix-it.net (2001:7f8:b:100:1d1:a5d1:5169:96) 5.072 ms 29.613 ms 29.646 ms 5 2001:4860::1:0:458d (2001:4860::1:0:458d) 2.155 ms 2.106 ms 1.992 ms 6 2001:4860:0:1::4c7 (2001:4860:0:1::4c7) 1.196 ms 1.35 ms 1.146 ms 7 2a00:1450:8000:17::8 (2a00:1450:8000:17::8) 0.94 ms 1.064 ms 1 ms real 0m0.912s
(I've also tested this with other domains)
As you can see the bottleneck seems to be the reverse DNS lookups of IPv4 addresses. Does anyone know how to fix this?
Comments
1) Who uses traceroute anymore, install 'mtr'.
2) to /etc/resolv.conf:
The strange thing: When mtr runs in report-mode (
mtr --report --report-cycles=3 google.com) it's also incredible slow but when i just runmtr google.comit's muuuuch fasterI already tried both the providers resolvers and Google's
me
For further investigation i've tcpdump'd my DNS traffic while doing a traceroute and loaded it into Wireshark:
As you can see there is a ~5sec delay always after a "No Such Domain" (=a hop without reverse DNS is it). This correlates with the traceroute always pausing for ~5 seconds right before display a hop without reverse DNS entry.
Now the question is: How can this be fixed?
@gsrdgrdghd,
Well, hard to say where the delay is. If it is in traceroute itself or the remote DNS server (Google). mtr seems to perform much better and most folks today use it exclusively. I use both and often traceroute since often default installed in distros I use.
I had a similar issue. My solution was to install dnsmasq (which I use often). Then run queries through it and turn on caching of negative records and bump the cache of records total to a big number 15k.
I also trimmed the /etc/resolv.conf entries down to a single entry for now. In my case I used 4.2.2.4 (Level 3).
For the most part, not seeing these delays. My delays often were on internal and first 6 hops which are outside my facility in upstream providers network and thus no reverse DNS.
If you look at the screenshot i think the remote DNS server can be ruled out. Every DNS request gets answered within <100ms. The problem is that my server simply doesn't send DNS queries for 5 seconds after a No such Domain.
I'll look into setting up dnsmasq, thanks for the suggestion.
traceroute -w 0.5 google.com
Does this help?
Sadly, no.
Try the following
netstat -a
then try
netstat -an
Is there a difference in speed?
Yes very much. -a is slower. Between these lines there is a huge delay:
Who's managing the reverse entries? Either way get them checked. Ensure the IPs have been delegated to the correct servers. This generally occurs when the IPs have been delegated to the wrong server then that server is taken offline.
The problem orrcurs with all IPs that don't have a reverse DNS entry set so it can't be caused by one misconfigured DNS server. Also the answers for the DNS queries actually arrive without any delay so the problem doesn't seem to involve anything network-related.
Yes, seems for some reason your resolver lib is ignoring the servfail messages and waiting more for an answer.
What's in your /etc/resolv.conf ? Perhaps it is asking other name servers too.
edit: i saw you wrote that you have only one entry in resolv.conf
Maybe install another traceroute alternative? Which traceroute version is this?
Here is mine:
And the traceroute:
Btw logging into SSH also takes some time if no reverse DNS entry is set for the IP i'm connecting from.
@gsrdgrdghd try leaving only the first entry (nameserver 8.8.8.8) and then try the traceroute again.
Doesn't help. Is there any easy way to get a call trace for the DNS lookups? That might help determining in which library/process the call hangs.
Unfortunately now. System calls can be seen with strace, but not library calls. It probably could be stepped with gdb, but i doubt there is symbol / debug information included and it would be much work for nothing.
What's your traceroute version? Is it the same as mine?
Modern traceroute for Linux, version 2.0.19, Dec 10 2012It's the default one that comes with Debian Sid.
done the same test from a Prometus VPS:
no problem using debian squeeze with
@marrco weird that you get routed to the US when tracing google.com, it resolved to an Italian IP for me.
Anyway i guess i don't have much choice but to reinstall my OS.
For this try adding the following to your
/etc/ssh/sshd_configand restart
sshd.Thanks, but i ended up reinstalling my VPS. Everything works fine now
@gsrdgrdghd reinstalled with the same OS, or different?
Ubuntu x64 instead of Debian x32
Well, Debian does the same thing @gsrdgrdghd.
I haven't fully isolate the traceroute and tracepath issue.
mtr seems to always work just dandy. That's what I am sticking to.
Mine seems to work fine on debian squeeze. Don't know if they broke something in later versions.
On my Debian wheezy x32 it runs fine.