Need help troubleshooting my server

JohnRoe

@jsg said:
@JohnRoe

I lack experience with torrenting (I'm not interested), with Ubuntu (I don't like it), and with systemd (I absolutely avoid it) ... I'll risk a wild guess: some (many) of your peers (or however the systems connecting to your server are called in the torrrenting universe) have poorly configured NS records, in particular wrt PTR records -and- your system is configured to be picky about DNS.
Something like that might be the context in which the killer works, resource exhaustion, and to a degree that in the end even emergency logging doesn't work anymore (if systemd hampered systems have emergency logging at all, that is).

Which "resource" should I monitor? If it is RAM, I often checking htop and the RAM usage is barely 3GB from 16GB. I can't keep my eyes on the screen all the time tho, so maybe someone can suggest any tools that can record and have history that I can check on later.

JohnRoe

@AuroraZ said:

@JohnRoe said:

@uptime said:

@AnthonySmith said:
This is one of those situations where by it can be explained to you but it cannot be understood for you.

Just old fashioned slow leaning for you I am afraid.

@jsg said:
Btw, if I'm not mistaken Hetzner provides a remote console for their dedis that allow you to have kind of "direct" access which might be helpful when you experience problems agains.

@JohnRoe the rapid-fire nature of this forum can make it challenging to focus on the useful suggestions such as from @AuroraZ and the more general guidance quoted above. So a bit of frustration and impatience is going to be part of the equation sometimes - just have to roll with it and keep thinking logically.

not much more for me to suggest except please do add more details as you continue to figure things out - "thinking out loud" mode is a reasonable approach if that's where you are at now, okay! Good luck.

EDIT2:

For example ("thinking out loud" / thinking logically / incorporating suggestions)

Can't ping - why ?

Either:

• server is down or in a hard loop or out of RAM and busy swapping
• or network interface crashed but console access still works
• or provider null-routed your IP because IDS / DDoS protection triggered
• or ...?

so how to narrow down these possibilities?

• Check if console access works.
• Assume reboot would not fix null-route (but maybe inquire via ticket to confirm with provider)
• and so on ...

Take notes, test hypotheses, try to be methodical ...

and be prepared to discover something unexpected anyway!

Thanks. It happened again a few hours ago and I decided to give it a little time before restarting to see if I can ping it at least once, in-case this is bandwidth problem. I can't.

So I rebooted and quickly checking syslog and I found a part with many ^@ symbols, exact issue like this one. Before the symbols were a few lines of

Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.

I don't know if this DNS error is related or not. The log stopped at those ^@ characters and new entry was when the server booting after I restarted, around ~20mins of time gap.

So I believe the system was completely froze. I have stopped rtorrent and qbittorrent and only transmisison was running with 10mbps limit like suggested. I don't know how to proceed tracking this problem. I have reinstalled the server and hope the issue won't happen again. Anyone with suggestion on how to do something so I can trace the cause will be really appreciated!

Millions of people around the world have this error. It is systemd's problem not yours that is not why your server is locking up.

Hmmm. I have no idea then. I have reinstalled the server. So I am hoping the issue is gone now.

AuroraZ

Is this a Ubuntu system? If so show me your kernel version. Useuname -r

jsg

@JohnRoe said:
Which "resource" should I monitor? If it is RAM, I often checking htop and the RAM usage is barely 3GB from 16GB. I can't keep my eyes on the screen all the time tho, so maybe someone can suggest any tools that can record and have history that I can check on later.

Too little even basic info provided by you.

Edit: Reinstalling the server is very unlikely to solve your problem.

uptime

atop will keep history of resource use (including RAM) - I think default settings will log every 10 minutes

sudo apt install atop should do the trick on ubuntu or Debian

There is also a netatop module which can track network activity

AuroraZ

If it is Ubuntu and the kernel is not higher then 4.1.6 then update it. I am hearing the firmware or something may be a bit wonky on some machines.

JohnRoe

Sorry guys, got some life matter preventing me from visiting LET for a couple of days.

@AuroraZ said:
Is this a Ubuntu system? If so show me your kernel version. Useuname -r

If it is Ubuntu and the kernel is not higher then 4.1.6 then update it. I am hearing the firmware or something may be a bit wonky on some machines.

I have reinstalled the server, but it should be the same because I am still using Ubuntu server 18.04 with default kernel. Anyway, it's 4.15.0-52-generic

@uptime said:
atop will keep history of resource use (including RAM) - I think default settings will log every 10 minutes

sudo apt install atop should do the trick on ubuntu or Debian

There is also a netatop module which can track network activity

Thanks a lot

@jsg said:

@JohnRoe said:
Which "resource" should I monitor? If it is RAM, I often checking htop and the RAM usage is barely 3GB from 16GB. I can't keep my eyes on the screen all the time tho, so maybe someone can suggest any tools that can record and have history that I can check on later.

Too little even basic info provided by you.

Edit: Reinstalling the server is very unlikely to solve your problem.

Sorry about that. But seems that it did really solve my problem. No freeze since I reinstalled the server. I am even run the torrent clients mentioned again yesterday and ther is no slowdown or freeze.

JohnRoe

Thanks a lot guys for replying. After a couple days since I reinstalling the server, the issue didn't happen anymore. In before, it happen once or twice a day.

I hope this issue really went away tho. Thanks again

AuroraZ

@JohnRoe said:
Thanks a lot guys for replying. After a couple days since I reinstalling the server, the issue didn't happen anymore. In before, it happen once or twice a day.

I hope this issue really went away tho. Thanks again

Most likely it was something withe image you used. Then it got an update and fixed it. It happens sometimes and is something we all deal with on occasion. I did not mean to sound harsh or make you feel like I did not like you or anything, just when this kind of thing happens I have a tendency to get to point and fix it and not worry about anything else.

JohnRoe

@AuroraZ said:

@JohnRoe said:
Thanks a lot guys for replying. After a couple days since I reinstalling the server, the issue didn't happen anymore. In before, it happen once or twice a day.

I hope this issue really went away tho. Thanks again

Most likely it was something withe image you used. Then it got an update and fixed it. It happens sometimes and is something we all deal with on occasion. I did not mean to sound harsh or make you feel like I did not like you or anything, just when this kind of thing happens I have a tendency to get to point and fix it and not worry about anything else.

In this kind of thing, I totally understand. No problem, and thanks