New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Need help troubleshooting my server
Hi. My server got a weird issue. Sometimes it cannot be reached. But will be fixed after a reboot. It happened twice today and I have no idea why.
Just now, x11vnc somehow stopped and tmux sessions were missing too.
Anyone know can I track the cause? Any help will be appreciated.
I am using hetzner auction server, i7-3770 2x3TB 16GB.
Ubuntu 18.04 is installed.
Comments
If there's nothing important, I'll just do a clean re-install. You might want to share what you did to your server (used what script to install what, etc.), and share logs if there's any available which will help pinpoint the issue
Hello.MJJ.
nginx, php7.3-fpm, transmission, qbittorrent, rtorrent, ffmpeg, mediainfo, mkvtoolnix, some other programs I couldn't recall right now. Thats pretty much of what I have installed.
What's that?
It's the IATA code for Moki Airport, Papua New Guinea.
If this is KVM....
Have you been able to review the console via VNC when this occurs? One common issue I see is systems running out of memory, and when that occurs Linux will randomly kill processes such as SSHd.
You could try adding a significant amount of swap (you can create a file for swap, you don’t need a dedicated partition).
Does it stop responding to pings?
Err... okay.
Another lesson on Chinese
MJJ usually means someone without d*cks in Chinese. It is being used widely in Chinese forum, basically a way to call each other. However, there is no exact definition for this term as it can be used in both good and bad way.
It also means someone without any VPS. If you ever used Google Translate on some Chinese forum, you will see "Chicken" everywhere. "Chicken" can mean both d*cks and VPS when stacked. "Chick" is basically VPS and "Hen" is basically Dedicated server.
All the terms I explained above are informal.
Three torrent clients are eating your ram. When you reboot they calm down, but after a bit the ram usage climbs again. This causes your server to lock up hence your trouble.
Pick one and run no need for three of them.
Use the htop program to look at your ram usage and identify which program is the worst.
Just type htop and see it clearer.
@JohnRoe
It's probably not the help you are looking for but it's honest: with that vague and unspecific request largely two things will happen. (a) those with the know-how to help you won't waste their time, and (b) those who "help" won't (be capable to) really help.
This is probably the right answer, try this and report back @OP
I am playing with their API and see which will suite my PHP project.
They actually hold small amount of torrents just for testing except transmission which have more than 100 torrents.
In htop, only ~3GB RAM is used.
rtorrent = 0.8%, qBittorrent = 0.9%, Transmission = 1.2%
Also I can't see any way that these clients can make my server unreachable, which only a reboot can fix it. Can they really do that?
I feel dumb. I don't understand and can't see what it's relation with me.
Any suggestion on how getting help for this? I am here for help and to learn.
Since I posted this, my server has been running well. I can't expect when it will happen again or will it happen again.
Chances are you did not set bandwidth limits and they are going all out and eating it up. Also if the server is with hostsolutions then it is probably a network problem and nothing to do with you. A lot of people have been having all kinds of problems there.
If it does respond to ping then the server is either off, port is saturated, or the network is bunked some how.
Without knowledge of the server and/or port then I can not help you further.
Only transmission hold more than 100 torrents. The other 2 are just holding less than 5. If they are the cause, I don't think people can use them to handle thousands of torrents. Nonetheless, I'll take those suggestions. I will only let transmission running for now.
This is one of those situations where by it can be explained to you but it cannot be understood for you.
Just old fashioned slow leaning for you I am afraid.
I am using hetzner auction server. I am using the server with specs: i7-3770 2x3TB 16GB.
@FAT64 Thanks for another chapter in our wonderful Chinese Crash Course program
But, what makes an exceptionally cheap&good d*ck? Is it the
RAMsize or theballsdiskcapacity?If you have not throttled the damned torrent clients then they can eat the whole port and you will have nothing left to use. Last time I am going to say that. Even one torrent in an unthrottled torrent client can eat the whole port. The people running thousands of them have throttle damned clients so it will not happen.
FFS now I remember why I do not do tech repair or try to help anyone anymore. they don't listen and come up with shit off the top of their heads to justify it.
I will make this really easy for the @OP, ticket Hetzner and ask them, see where it gets you.
Info. Good and relevant info. That's the basis for good help. Without it you'll get funny guesses and joking.
Example: "2 x 3 TB" isn't worth much in terms of info. "I run on two paritions, one is x GB (root) and the other one (swap) is y GB" is much more useful. Throwing in the output of "free" and "df -h" is another example for useful info.
Btw, if I'm not mistaken Hetzner provides a remote console for their dedis that allow you to have kind of "direct" access which might be helpful when you experience problems agains.
Torrents in qBittorrent and rtorrent are stopped after they finished because like I said, I am playing with their API for my project. Transmission is not throttled. I am only using torrents from private tracker, not public. I am sorry if you got offended by my replies, that is not my intention. I am trying to be as clear as possible because only Transmission is active seeding and the other 2 only have a few torrents that are not running. I am trying to understand how they can make my server unreachable, not even ping.
I will contact hetzner directly if it happen again. thanks a lot for your help.
Then throttle it, and take 5 minutes to ask yourself why you believe pps hitting your port cares about your public or private tracker status.
Thanks. Would be better if someone knows what my issue would probably related to, can tell me what specific information they need in order to detect them. I am away from my home. I will update information as you ask later when I get in touch with my PC. Thanks a lot!
Done throttling it . At this point, I think @cociu server is better than my hetzner dedicated server because I used public tracker on his server back then without any issue. Public torrents are much greedy in term of bandwidth usage.
Still, I can't understand how torrenting a little amount of torrents can make my server totally unreachable until I rebooted it. If i can't access it, no peers can leech the torrents, so the bandwidth usage should go down and I should be able to access it again, right? But why the server can't be reached at all until i rebooted it?
I don't even know if the server is alive or dead because it was not even pingable. I am seeding 73 public torrents on my kimsufi 100mbps server and they sucking all bandwidth the server have and my website hosted on the same server still loads fine.
I am not trying to be clever whatsoever, but I am trying to find the logic behind the theory that those torrent clients caused my server went unreachable, not even ping. I would be grateful if someone can explain that to me.
For those who feel offended, I am sorry and thanks a lot for your help. That is not my intention at all.
@JohnRoe the rapid-fire nature of this forum can make it challenging to focus on the useful suggestions such as from @AuroraZ and the more general guidance quoted above. So a bit of frustration and impatience is going to be part of the equation sometimes - just have to roll with it and keep thinking logically.
not much more for me to suggest except please do add more details as you continue to figure things out - "thinking out loud" mode is a reasonable approach if that's where you are at now, okay! Good luck.
EDIT2:
For example ("thinking out loud" / thinking logically / incorporating suggestions)
Can't ping - why ?
Either:
so how to narrow down these possibilities?
Take notes, test hypotheses, try to be methodical ...
and be prepared to discover something unexpected anyway!
Thanks. It happened again a few hours ago and I decided to give it a little time before restarting to see if I can ping it at least once, in-case this is bandwidth problem. I can't.
So I rebooted and quickly checking syslog and I found a part with many
^@symbols, exact issue like this one. Before the symbols were a few lines ofServer returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.I don't know if this DNS error is related or not. The log stopped at those
^@characters and new entry was when the server booting after I restarted, around ~20mins of time gap.So I believe the system was completely froze. I have stopped rtorrent and qbittorrent and only transmisison was running with 10mbps limit like suggested. I don't know how to proceed tracking this problem. I have reinstalled the server and hope the issue won't happen again. Anyone with suggestion on how to do something so I can trace the cause will be really appreciated!
@JohnRoe
I lack experience with torrenting (I'm not interested), with Ubuntu (I don't like it), and with systemd (I absolutely avoid it) ... I'll risk a wild guess: some (many) of your peers (or however the systems connecting to your server are called in the torrrenting universe) have poorly configured NS records, in particular wrt PTR records -and- your system is configured to be picky about DNS.
Something like that might be the context in which the killer works, resource exhaustion, and to a degree that in the end even emergency logging doesn't work anymore (if systemd hampered systems have emergency logging at all, that is).
Millions of people around the world have this error. It is systemd's problem not yours that is not why your server is locking up.