Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Help tracking in-game network issues.
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Help tracking in-game network issues.

TheBoxTheBox Member
edited July 2021 in Help

Hey guys. I run game servers (RUST/CS/7 Days/Squad/Ark/Arma2)...

I have few dedicated servers at ReliableSite and few at OVH and others and I use BuyVM GRE Tunnel to protect my Game Servers (because they use path.net and they have the juicy anycast.

Randomly those servers at ReliableSite using BuyVM the network start freezing on our game servers. All players start lagging out. No packet loss. ICMP is fine. No high ping looking at "SmokePing" and WinMTR is fine.

I tracked both ends and they are fine (It seems for me). I changed MTU's and ReliableSite and BuyVM confirms that their end is working fine. So makes me think. What it could be?

What could cause in-game desync (1-5min) and stop randomly like nothing happenned?

Latency between each end is max 1-6ms. So Tunnel latency is fine.

I'm using Ping Plotter now sending UDP raw of 180 bytes every 1sec to see if I can track some sort of bad "routing" being applied or some UDP packet loss. My games use UDP if UDP has packet loss or has "Jitter" players would lag in-game.

This has been for a month and I tried everything and I come here at peace asking for help or any tips for tracking this.

Kind regards

P.S I don't want this to be a "war" thread competitors blaming other competitors etc...

I just want new views or what tools I can use to help trace this "ghost" issue...

If you come here just to blame the service providers I use don't waste your time, leave this discussion.

Comments

  • pikepike Veteran

    Sounds like the gameserver traffic is considered as an attack. Your best bet would be to have a complete network capture (wireshark) from one disconnect incident so you can report that to @Francisco

    Thanked by 1TheBox
  • TheBoxTheBox Member

    @pike said:
    Sounds like the gameserver traffic is considered as an attack. Your best bet would be to have a complete network capture (wireshark) from one disconnect incident so you can report that to @Francisco

    Thanks. I also want to point out that I also use Wireguard Tunneling. Port 51820.

    I was thinking it could be their network carriers or the routing changing since game servers lag and they only use UDP. If UDP is lost or delayed it causes the known desync in gaming world.

    On BuyVM I have 2 KVM acting as a Remote Protection MIA and NY and on ReliableSite I have NY dedicated servers same for MIA.

    Yesterday the whole network of my game servers including MIA and NY started lagging in-game at same time and they ended at same time and they use ReliableSite as a dedicated server.

    So my stuff is like this:

    RS NY > BUYVM NY > WORLD.

    RS MIA > BUYVM MIA > WORLD.

    Etc...

    Ones at OVH with BuyVM did not lag in-game.

    I also want to point out that Radic and Francisco' are amazing. They are helping me troubleshoot this and giving me tips. I'm not blaming their services. Their services are amazing. The only sole purpose of this post is to gather new "angles" of the situation. New people/minds sharing tips on what could be causing this :)

    Kind regards.

  • BinaryBinary Member, Host Rep
    edited July 2021

    Is that specific game supported by Path?
    Besides the fact that Path is pretty much known for constant packet loss and outages, they also have some kind of "automatic mitigation" going on for certain attack traffic.

    If you run an unsupported UDP app, and an attacker sends "TSource Engine Query" attacks, Path will consider your server as Source Engine - and drop all traffic that isn't source engine-related.

    Besides that, ReliableSite's network is congested 24/7 (what a surprise).

    Thanked by 1TheBox
  • TheBoxTheBox Member

    @Binary said:
    Is that specific game supported by Path?
    Besides the fact that Path is pretty much known for constant packet loss and outages, they also have some kind of "automatic mitigation" going on for certain attack traffic.

    If you run an unsupported UDP app, and an attacker sends "TSource Engine Query" attacks, Path will consider your server as Source Engine - and drop all traffic that isn't source engine-related.

    Besides that, ReliableSite's network is congested 24/7 (what a surprise).

    The game is not supported.

    I have many dedicated servers at ReliableSite using BuyVM remote protection. like 10 servers and all of them in-game lagged (desync) at same time and ended at same time randomly.

    I tracked with MTR no packet loss.

    SmokePing on RS and BuyVM no packetloss (agressive probing).

    Ping Plotter no packet loss. Few Jitters here and there. But not at the time where the desync was occurring.

    Bandwidth Graphs (No spikes) both on RS and BuyVM.

    I've talked with Radic and I know MIA has been ddosed hard few weeks ago and their network seems fine according to smokeping and their MTRS and their answers.

  • BinaryBinary Member, Host Rep
    edited July 2021

    @TheBox said:

    @Binary said:
    Is that specific game supported by Path?
    Besides the fact that Path is pretty much known for constant packet loss and outages, they also have some kind of "automatic mitigation" going on for certain attack traffic.

    If you run an unsupported UDP app, and an attacker sends "TSource Engine Query" attacks, Path will consider your server as Source Engine - and drop all traffic that isn't source engine-related.

    Besides that, ReliableSite's network is congested 24/7 (what a surprise).

    The game is not supported.

    I have many dedicated servers at ReliableSite using BuyVM remote protection. like 10 servers and all of them in-game lagged (desync) at same time and ended at same time randomly.

    I tracked with MTR no packet loss.

    SmokePing on RS and BuyVM no packetloss (agressive probing).

    Ping Plotter no packet loss. Few Jitters here and there. But not at the time where the desync was occurring.

    Bandwidth Graphs (No spikes) both on RS and BuyVM.

    I've talked with Radic and I know MIA has been ddosed hard few weeks ago and their network seems fine according to smokeping and their MTRS and their answers.

    The game not being supported could likely be the reason then.
    Path is ok for applications that are supported, but once you run an unsupported app, you're basically doomed by their "automated mitigation".

    I could be totally wrong however. Maybe check in with Zigi or someone from Path to see if your IP is being ddosed when the lag spikes happen?

  • TheBoxTheBox Member
    edited July 2021

    @Binary said:

    @TheBox said:

    @Binary said:
    Is that specific game supported by Path?
    Besides the fact that Path is pretty much known for constant packet loss and outages, they also have some kind of "automatic mitigation" going on for certain attack traffic.

    If you run an unsupported UDP app, and an attacker sends "TSource Engine Query" attacks, Path will consider your server as Source Engine - and drop all traffic that isn't source engine-related.

    Besides that, ReliableSite's network is congested 24/7 (what a surprise).

    The game is not supported.

    I have many dedicated servers at ReliableSite using BuyVM remote protection. like 10 servers and all of them in-game lagged (desync) at same time and ended at same time randomly.

    I tracked with MTR no packet loss.

    SmokePing on RS and BuyVM no packetloss (agressive probing).

    Ping Plotter no packet loss. Few Jitters here and there. But not at the time where the desync was occurring.

    Bandwidth Graphs (No spikes) both on RS and BuyVM.

    I've talked with Radic and I know MIA has been ddosed hard few weeks ago and their network seems fine according to smokeping and their MTRS and their answers.

    The game not being supported could likely be the reason then.
    Path is ok for applications that are supported, but once you run an unsupported app, you're basically doomed by their "automated mitigation".

    I could be totally wrong however. Maybe check in with Zigi or someone from Path to see if your IP is being ddosed when the lag spikes happen?

    I think it's not the reason. I've been using BuyVm (that has path protection) from 5 months and the first 4 months were fine. Last month this started going on.

    and as I stated previously it happens randomly on all my game servers.

    I'm tracking both ReliableSite and BuyVm and both ends are fine as their staff says.

    So I'm not sure what could be causing this.

    I have overall 3-8k players.

  • jordynegen11jordynegen11 Member
    edited July 2021

    You could try to switch to a GRE tunnel (just for testing). Mabe somewhere the Wireguard traffic is getting dropped by any kind of mitigation. Can be on both sides.

    We had a similar issue with the OVH mitigation few months back. Specific traffic over a wiredguard (UDP) tunnel was marked as "attack" resulted in random connection issues. Idk how RS their protection works in this case.

    To be sure, I think it's worth the shot.

    Thanked by 1TheBox
  • TheBoxTheBox Member

    @jordynegen11 said:
    You could try to switch to a GRE tunnel (just for testing). Mabe somewhere the Wireguard traffic is getting dropped by any kind of mitigation. Can be on both sides.

    We had a similar issue with the OVH mitigation few months back. Specific traffic over a wiredguard (UDP) tunnel was marked as "attack" resulted in random connection issues. Idk how RS their protection works in this case.

    Interesting. Maybe I should try a different wireguard port? I've read that sometimes ISP or networks block 51820. I've read that people once had this issue used 500UDP or 1024 or 443. Etc...

    Sadly I can't switch from wireguard due I use windows on my dedicated servers and I don't know how to do GRE from Windows > Linux.

  • jordynegen11jordynegen11 Member
    edited July 2021

    @TheBox said:

    @jordynegen11 said:
    You could try to switch to a GRE tunnel (just for testing). Mabe somewhere the Wireguard traffic is getting dropped by any kind of mitigation. Can be on both sides.

    We had a similar issue with the OVH mitigation few months back. Specific traffic over a wiredguard (UDP) tunnel was marked as "attack" resulted in random connection issues. Idk how RS their protection works in this case.

    Interesting. Maybe I should try a different wireguard port? I've read that sometimes ISP or networks block 51820. I've read that people once had this issue used 500UDP or 1024 or 443. Etc...

    Sadly I can't switch from wireguard due I use windows on my dedicated servers and I don't know how to do GRE from Windows > Linux.

    Ah that's a shame. Yeah if you use windows, it's a different story.

    I know from experience that UDP traffic can be affected by mitigation services. Most mitigation will check the UDP traffic by the amount of bandwidth and pre-defined patterns. Changing the port won't really help you I guess.

    For example wireguard will send a ton of encrypted UDP traffic to your server. That can easily be detected as an "attack". OVH did change the thresholds for us back then, and that fixed it.

    Thanked by 1TheBox
  • TheBoxTheBox Member

    @jordynegen11 said:

    @TheBox said:

    @jordynegen11 said:
    You could try to switch to a GRE tunnel (just for testing). Mabe somewhere the Wireguard traffic is getting dropped by any kind of mitigation. Can be on both sides.

    We had a similar issue with the OVH mitigation few months back. Specific traffic over a wiredguard (UDP) tunnel was marked as "attack" resulted in random connection issues. Idk how RS their protection works in this case.

    Interesting. Maybe I should try a different wireguard port? I've read that sometimes ISP or networks block 51820. I've read that people once had this issue used 500UDP or 1024 or 443. Etc...

    Sadly I can't switch from wireguard due I use windows on my dedicated servers and I don't know how to do GRE from Windows > Linux.

    Ah that's a shame. Yeah if you use windows, it's a different story.

    I know from experience that UDP traffic can be affected by mitigation services. Most mitigation will check the UDP traffic by the amount of bandwidth and pre-defined patterns. Changing the port won't really help you I guess.

    For example wireguard will send a ton of encrypted UDP traffic to your server. That can easily be detected as an "attack". OVH did change the thresholds for us back then, and that fixed it.

    I will try changing port just in case.

    Anyways I don't believe it's mitigation since the whole network is affected. For example I have 10 servers at RS that are using remote protection with BuyVM and these 10 servers lag in-game at same time randomly and stop randomly. No ICMP Packet loss. No MTR packet loss. Nothing.

    That's why I'm thinking which side is it? Mine or RS or BuyVM? They say their end is fine so I believe them.

  • TheBoxTheBox Member

    Plus any tools that can be used to help me find this ghost issues are appreciated 🙂

  • jordynegen11jordynegen11 Member
    edited July 2021

    @TheBox

    Hosting companies will always say that there is nothing wrong on their end (mostly). You should believe your own tests. Its hard to say which fault it is at the moment. But if it's working fine with OVH, then it's probably at the RS end.

    Anyways I don't believe it's mitigation since the whole network is affected. For example I > have 10 servers at RS that are using remote protection with BuyVM and these 10 servers > lag in-game at same time randomly and stop randomly. No ICMP Packet loss. No MTR > packet loss. Nothing.

    Again I don't know how RS handles their mitigation. But it's possible that when mitigation kicks in, this can limit connections from your buyvm IP on the entire network of RS.

    Another thing you can try is a UDP iPerf3 test. On the moment it starts lagging, start a iPerf3 server on your buyvm server and execute this command on the RS server:

    Incoming traffic
    iperf3 -c wireguard.endpoint.ip.here -R -u -b 1000M

    Outgoing traffic
    iperf3 -c wireguard.endpoint.ip.here -u -b 1000M

    Thanked by 1TheBox
  • TheBoxTheBox Member

    @jordynegen11 said:
    @TheBox

    Hosting companies will always say that there is nothing wrong on their end (mostly). You should believe your own tests. Its hard to say which fault it is at the moment. But if it's working fine with OVH, then it's probably at the RS end.

    Anyways I don't believe it's mitigation since the whole network is affected. For example I > have 10 servers at RS that are using remote protection with BuyVM and these 10 servers > lag in-game at same time randomly and stop randomly. No ICMP Packet loss. No MTR > packet loss. Nothing.

    Again I don't know how RS handles their mitigation. But it's possible that when mitigation kicks in, this can limit connections from your buyvm IP on the entire network of RS.

    Another thing you can try is a UDP iPerf3 test. On the moment it starts lagging, start a iPerf3 server on your buyvm server and execute this command on the RS server:

    Incoming traffic
    iperf3 -c wireguard.endpoint.ip.here -R -u -b 1000M

    Outgoing traffic
    iperf3 -c wireguard.endpoint.ip.here -u -b 1000M

    Thank you very much for your answer.

    I will make that iperf3 test once lag happens.

    Will this show if it's BuyVM or RS?

    Any other tool that I can use to help trace this? Or add more confluence.

    Thanks.

  • jordynegen11jordynegen11 Member
    edited July 2021

    It will confirm that> @TheBox said:

    @jordynegen11 said:
    @TheBox

    Hosting companies will always say that there is nothing wrong on their end (mostly). You should believe your own tests. Its hard to say which fault it is at the moment. But if it's working fine with OVH, then it's probably at the RS end.

    Anyways I don't believe it's mitigation since the whole network is affected. For example I > have 10 servers at RS that are using remote protection with BuyVM and these 10 servers > lag in-game at same time randomly and stop randomly. No ICMP Packet loss. No MTR > packet loss. Nothing.

    Again I don't know how RS handles their mitigation. But it's possible that when mitigation kicks in, this can limit connections from your buyvm IP on the entire network of RS.

    Another thing you can try is a UDP iPerf3 test. On the moment it starts lagging, start a iPerf3 server on your buyvm server and execute this command on the RS server:

    Incoming traffic
    iperf3 -c wireguard.endpoint.ip.here -R -u -b 1000M

    Outgoing traffic
    iperf3 -c wireguard.endpoint.ip.here -u -b 1000M

    Thank you very much for your answer.

    I will make that iperf3 test once lag happens.

    Will this show if it's BuyVM or RS?

    Any other tool that I can use to help trace this? Or add more confluence.

    Thanks.

    It will confirm that something is limiting or slowing down the UDP traffic or not. It will also show you if it's only incoming, outgoing or both directions. You can send the result to RS for further investigation and post them here :smiley:

    I don't know a test to find the one responsable. These things are hard to find. But like I said, If it's a no-issue at OVH, it's probably a RS problem.

  • TheBoxTheBox Member

    @jordynegen11 said:
    It will confirm that> @TheBox said:

    @jordynegen11 said:
    @TheBox

    Hosting companies will always say that there is nothing wrong on their end (mostly). You should believe your own tests. Its hard to say which fault it is at the moment. But if it's working fine with OVH, then it's probably at the RS end.

    Anyways I don't believe it's mitigation since the whole network is affected. For example I > have 10 servers at RS that are using remote protection with BuyVM and these 10 servers > lag in-game at same time randomly and stop randomly. No ICMP Packet loss. No MTR > packet loss. Nothing.

    Again I don't know how RS handles their mitigation. But it's possible that when mitigation kicks in, this can limit connections from your buyvm IP on the entire network of RS.

    Another thing you can try is a UDP iPerf3 test. On the moment it starts lagging, start a iPerf3 server on your buyvm server and execute this command on the RS server:

    Incoming traffic
    iperf3 -c wireguard.endpoint.ip.here -R -u -b 1000M

    Outgoing traffic
    iperf3 -c wireguard.endpoint.ip.here -u -b 1000M

    Thank you very much for your answer.

    I will make that iperf3 test once lag happens.

    Will this show if it's BuyVM or RS?

    Any other tool that I can use to help trace this? Or add more confluence.

    Thanks.

    It will confirm that something is limiting or slowing down the UDP traffic or not. It will also show you if it's only incoming, outgoing or both directions. You can send the result to RS for further investigation and post them here :smiley:

    I don't know a test to find the one responsable. These things are hard to find. But like I said, If it's a no-issue at OVH, it's probably a RS problem.

    Nice.

    Should I also make IPerf3 server on RS and test from BuyVM?

    Kind regards.

  • @TheBox said:

    @jordynegen11 said:
    It will confirm that> @TheBox said:

    @jordynegen11 said:
    @TheBox

    Hosting companies will always say that there is nothing wrong on their end (mostly). You should believe your own tests. Its hard to say which fault it is at the moment. But if it's working fine with OVH, then it's probably at the RS end.

    Anyways I don't believe it's mitigation since the whole network is affected. For example I > have 10 servers at RS that are using remote protection with BuyVM and these 10 servers > lag in-game at same time randomly and stop randomly. No ICMP Packet loss. No MTR > packet loss. Nothing.

    Again I don't know how RS handles their mitigation. But it's possible that when mitigation kicks in, this can limit connections from your buyvm IP on the entire network of RS.

    Another thing you can try is a UDP iPerf3 test. On the moment it starts lagging, start a iPerf3 server on your buyvm server and execute this command on the RS server:

    Incoming traffic
    iperf3 -c wireguard.endpoint.ip.here -R -u -b 1000M

    Outgoing traffic
    iperf3 -c wireguard.endpoint.ip.here -u -b 1000M

    Thank you very much for your answer.

    I will make that iperf3 test once lag happens.

    Will this show if it's BuyVM or RS?

    Any other tool that I can use to help trace this? Or add more confluence.

    Thanks.

    It will confirm that something is limiting or slowing down the UDP traffic or not. It will also show you if it's only incoming, outgoing or both directions. You can send the result to RS for further investigation and post them here :smiley:

    I don't know a test to find the one responsable. These things are hard to find. But like I said, If it's a no-issue at OVH, it's probably a RS problem.

    Nice.

    Should I also make IPerf3 server on RS and test from BuyVM?

    Kind regards.

    No you don't have to. But when you doing the outgoing test, please check the results on the buyVM server and not on the RS server.

  • For a bit of testing, you can also try OpenVPN TCP/UDP as well just to isolate where the issue is.

  • TheBoxTheBox Member

    @jordynegen11 said:

    @TheBox said:

    @jordynegen11 said:
    It will confirm that> @TheBox said:

    @jordynegen11 said:
    @TheBox

    Hosting companies will always say that there is nothing wrong on their end (mostly). You should believe your own tests. Its hard to say which fault it is at the moment. But if it's working fine with OVH, then it's probably at the RS end.

    Anyways I don't believe it's mitigation since the whole network is affected. For example I > have 10 servers at RS that are using remote protection with BuyVM and these 10 servers > lag in-game at same time randomly and stop randomly. No ICMP Packet loss. No MTR > packet loss. Nothing.

    Again I don't know how RS handles their mitigation. But it's possible that when mitigation kicks in, this can limit connections from your buyvm IP on the entire network of RS.

    Another thing you can try is a UDP iPerf3 test. On the moment it starts lagging, start a iPerf3 server on your buyvm server and execute this command on the RS server:

    Incoming traffic
    iperf3 -c wireguard.endpoint.ip.here -R -u -b 1000M

    Outgoing traffic
    iperf3 -c wireguard.endpoint.ip.here -u -b 1000M

    Thank you very much for your answer.

    I will make that iperf3 test once lag happens.

    Will this show if it's BuyVM or RS?

    Any other tool that I can use to help trace this? Or add more confluence.

    Thanks.

    It will confirm that something is limiting or slowing down the UDP traffic or not. It will also show you if it's only incoming, outgoing or both directions. You can send the result to RS for further investigation and post them here :smiley:

    I don't know a test to find the one responsable. These things are hard to find. But like I said, If it's a no-issue at OVH, it's probably a RS problem.

    Nice.

    Should I also make IPerf3 server on RS and test from BuyVM?

    Kind regards.

    No you don't have to. But when you doing the outgoing test, please check the results on the buyVM server and not on the RS server.

    Thanks Jordy for helping me. I will be back with results.

  • TheBoxTheBox Member
    edited July 2021

    @jordynegen11 said:

    @TheBox said:

    @jordynegen11 said:
    It will confirm that> @TheBox said:

    @jordynegen11 said:
    @TheBox

    Hosting companies will always say that there is nothing wrong on their end (mostly). You should believe your own tests. Its hard to say which fault it is at the moment. But if it's working fine with OVH, then it's probably at the RS end.

    Anyways I don't believe it's mitigation since the whole network is affected. For example I > have 10 servers at RS that are using remote protection with BuyVM and these 10 servers > lag in-game at same time randomly and stop randomly. No ICMP Packet loss. No MTR > packet loss. Nothing.

    Again I don't know how RS handles their mitigation. But it's possible that when mitigation kicks in, this can limit connections from your buyvm IP on the entire network of RS.

    Another thing you can try is a UDP iPerf3 test. On the moment it starts lagging, start a iPerf3 server on your buyvm server and execute this command on the RS server:

    Incoming traffic
    iperf3 -c wireguard.endpoint.ip.here -R -u -b 1000M

    Outgoing traffic
    iperf3 -c wireguard.endpoint.ip.here -u -b 1000M

    Thank you very much for your answer.

    I will make that iperf3 test once lag happens.

    Will this show if it's BuyVM or RS?

    Any other tool that I can use to help trace this? Or add more confluence.

    Thanks.

    It will confirm that something is limiting or slowing down the UDP traffic or not. It will also show you if it's only incoming, outgoing or both directions. You can send the result to RS for further investigation and post them here :smiley:

    I don't know a test to find the one responsable. These things are hard to find. But like I said, If it's a no-issue at OVH, it's probably a RS problem.

    Nice.

    Should I also make IPerf3 server on RS and test from BuyVM?

    Kind regards.

    No you don't have to. But when you doing the outgoing test, please check the results on the buyVM server and not on the RS server.

    I want to update this thread.

    Jordy was very helpul on helping me find the issues.

    We found this once all my game servers lagged at same time.

    This is a iperf3 UDP test.

    ```Accepted
    [  5] local 10.0.0.1 port 5201 connected to 10.0.0.2 port 49315
    [ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
    [  5]   0.00-1.00   sec  0.00 Bytes  0.00 bits/sec  0.000 ms  0/0 (0%)
    [  5]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec  0.000 ms  0/0 (0%)
    [  5]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec  0.000 ms  0/0 (0%)
    [  5]   3.00-4.00   sec  1008 KBytes  8.26 Mbits/sec  0.066 ms  4791/4917 (97%)
    [  5]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec  0.066 ms  0/0 (0%)
    [  5]   5.00-6.00   sec   264 KBytes  2.16 Mbits/sec  0.065 ms  2716/2749 (99%)
    [  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec  0.065 ms  0/0 (0%)
    [  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec  0.065 ms  0/0 (0%)
    [  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec  0.065 ms  0/0 (0%)
    [  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec  0.065 ms  0/0 (0%)
    [ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
    [  5]   0.00-10.04  sec  1.24 MBytes  1.04 Mbits/sec  0.065 ms  7507/7666 (98%)  receiver```
    

    This is ReliableSite >>>> BuyVM.

    BuyVM >>> ReliableSite is okay and never has Iperf3 issues.

    We will keep testing and I will talk to both ends.

  • did you have a solution for it @TheBox ? Because I've started getting the same problem

Sign In or Register to comment.