Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Shells Virtual Desktop
BMail.ag - Secure Email Service
Server.net
CPLicense.net
VPS Server
Buy VPN
Vultr
VMs for AI
HostDare
ReliableSite White-Label Dedicated Hosting for Resellers
InterServer VPS
BMail.ag - Secure Email Service
Best VPN
High-Performance Bare Metal Server Solutions
Karvl.com
Server Mania Cloud Hosting
DataWagon Hosting
AlphaVPS Hosting
Evoxt.com
Clouvider
VPS Hosting with NVMe
Residential IPs in the US & 4G Mobile Proxies in EU & US with Unlimited Bandwidth
ReliableSite White-Label Dedicated Hosting for Resellers
Rabisu - Hosting Solutions
Shells Virtual Desktop
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

New Free VPS Location! Alexhost, Chisinau, Moldova via FOSSVPS!

135678

Comments

  • Not_OlesNot_Oles Member, Patron Provider

    @Aphelios said:

    @Not_Oles said:
    @Aphelios Can you please give me your ed25519 public key? Thanks!

    I have pm you my public key, and I’ll treasure this VPS,thanks

    Login details sent by PM. Thanks for joining! Welcome to the server! :)

    Thanked by 1Aphelios
  • Not_OlesNot_Oles Member, Patron Provider
    edited August 2025

    For the Alexhost Cisinau server, password login is disabled. Thus, clients' public ssh keys are necessary to enable clients to login.

    I hear that ed25519 ssh keys are considered safer to use than some other types of keys.

    Why are public keys called "public"?

    Why can anyone download a public key from a Github repository if the repository owner has set up ssh access?

    FOSSVPS clients who are concerned about the privacy of their public key can send their public key via PM.

    Best wishes and kindest regards!

  • StarnbergStarnberg Member
    edited August 2025

    Made some progress setting up the VPS kindly loaned to me, mostly considering and preparing options to expose some of the metadata it produces for others to see. E.g., started setting up a few data sources to expose.

    As one potential source, I set up a screensnap container for local graphical rendering of the NTP Pool stats of the server, but due to recent, and currently still ongoing, changes to various internals of the NTP Pool, it currently doesn't produce any output. I'll see whether I can track down, and fix the issue, and/or work with the Pool developers on this (currently, they have more essential items to still work out with the latest iteration of the monitoring system, and the screensnap feature to produce static images is just a convenience function).

    Based on someone else mentioning this in here, I also set up a HetrixTools agent, to eventually expose some basic VPS performance data.

    Also registered a free domain name, for now to expose the NTP service plus optional NTS. Will add another one in due course for the data exposure. Had things set up with one provider when I found that the service is slow/unreliable, so needed to start again with another one.

    Got Let's encrypt certificates for the existing names, working out the best way for the needed verification step (that is where the first DNS provider's name resolution being slow/unreliable turned out to be a blocker, and made me switch). I try to expose as little as possible for security reasons, so I tried the DNS-based verification first, but eventually settled on HTTP-based verification (as I am going to be exposing a web interface anyway). Maybe I'll revisit the DNS-based verification later on, couldn't get the relevant certbot plugin to work with the provider.

    Started to look into what front-end to use for the data exposure, including exploring options for SNI-based L4 routing. That would allow to share the relevant web ports with other people who might have a need but don't have a dedicated IPv4 address.

    Also set up a reverse tunnel to be able to access the VPS for management purposes without exposing the SSH port, which was seeing quite a bit of probing.

    NTP traffic looks good, interesting to see an initial IPv6/IPv4 traffic ratio very much in line with the general IPv6 adoption rate. Other zones have much lower IPv6 adoption rate for NTP pool traffic than their estimated overall adoption rate. Will observe further as traffic stabilizes a bit. The zone seems quite well served, so it won't be an issue at all to start to cool down well ahead of the currently scheduled end of the VPS.

    Thanked by 2Not_Oles BasToTheMax
  • My vps update log!

    Yesterday, I've played a bit with making my own docker images.
    I also tried sysbox for running docker-in-docker, but then resource usage reporting and limiting stopped working.

    Today, I've installed Incus (LXD fork) on my vps. I created my first container with debian 12.

    Tomorrow, I'm going to look into incus ACLs to block some ports for the containers. (Like mail ports, as they are not needed for testing)

  • Not_OlesNot_Oles Member, Patron Provider

    @Starnberg @BasToTheMax

    Thanks for your updates! Applause for your updates! :star:

    I think it's great to show everyone what you are doing and to create the opportunity for fun discussion and learning! <3

    @Starnberg I am curious about how much bandwidth you think your VPS might use in a 24 hour period. Perhaps less than one might think?

    @BasToTheMax Congrats on your first container! Everyone has to start at the beginning, and the first is often the most fun because you always get to learn something new!

  • @Not_Oles said: I am curious about how much bandwidth you think your VPS might use in a 24 hour period. Perhaps less than one might think?

    The average bitrate looks like it will be below, or at least not much above 1Mbit/s. That'll probably make for somewhat less than 20GB or so in+out per day. As I was working on stuff yesterday and today and not fully exercising the capacity yet, the picture is currently not complete. Tomorrow, Thursday, will be the first full day of somewhat stable traffic, so we'll have Thursday's data on Friday to give a first more complete impression as to what to expect going forward.

    And yes, based on own experience in some other zones, but especially also from reports throughout the pool's forum, I was expecting much worse. In contrast, if traffic rate remains at current levels (going a bit up as it stabilizes), it will be among the "best" zones I've encountered so far - totally the opposite of what I expected. I'm honestly very positively surprised.

    But my expectation likely was a bit biased as obviously, most reports are when stuff doesn't work, e.g., because a zone generates too much traffic for but the sturdiest servers to survive, creating a chicken and egg problem. Or when at least a noticeable number servers struggle. When things work very smoothly, there are obviously far fewer reports, but there are.

    While admittedly very likely over-optimistic, had the zone had a chicken and egg problem, the ambitious hope would have been that a beefy server like this VPS could have helped break that - though most likely, the time would have been too short for enough people to notice the relief, and add their own servers in time to reach a critical mass while this VPS was providing "cover".

  • Not_OlesNot_Oles Member, Patron Provider
    edited August 2025

    @Starnberg Yeah! Sounds excellent! Keep up your good work! :star:

    Thanked by 1Starnberg
  • I ran scripts to test disk I/O and network speed - performance looks great! Already got certificates installed and some pods deployed. My pc is pretty old with a mechanical hard drive, so local db(postgre) queries are really slow. But querying the db on this vps is 50ms faster! cpu and memory are plenty too. Really happy with it.

    Thanked by 1Not_Oles
  • some things on the London vps.
    Firstly, setting up environment such as setting ssh to public key login only, install fail2ban, nginx, sing-box for proxy, and some utilities which are common in Linux.
    As Tom said, the network is pretty good. watching / downloading iplayer is lightening fast which is greater than 500MB/s. I don't try to test the highest speed cause it's good for me now.
    There is an open source project named yt-dlp webui, so I installed it. as its document is outdated, there were some errors, I need to look up logs and check them through its code, and modify the configuration file, thanks god, finally it's done.

    thank you @Not_Oles and OnlyServers

    Thanked by 1Not_Oles
  • BasToTheMaxBasToTheMax Member
    edited August 2025

    @igctt said:
    There is an open source project named yt-dlp webui, so I installed it. as its document is outdated, there were some errors, I need to look up logs and check them through its code, and modify the configuration file,

    If it's an open source project, you could fix the errors in the documentation and make a pull request :wink:

    Also, I'm surprised that youtube isn't blocking the IP!

    Thanked by 1Not_Oles
  • Another vps update: :wink:

    Today:

    • Played a bit with firewall rules. Containers now have all outgoing mail ports blocked, to prevent spam and unauthorized mailing.
    • Added resource (ram/cpu) limits to containers. The storage limit doesn't seem to be working tho. I'll investigate this later.
    • Today or tomorrow, I'll also try to setup port forwarding (depends on how much time i have left today)

    Tomorrow:

    • I'll start with a (bash) script to automatically setup and install a new container!

    Maybe I could provide some small NAT LXC containers to the LET community!

    Thanked by 1Not_Oles
  • hello Santa I need Moldova to working VPN

  • Not_OlesNot_Oles Member, Patron Provider

    @BasToTheMax said: Maybe I could provide some small NAT LXC containers to the LET community!

    For context, maybe look at what @Neoon has been doing with microLXC.

    Thanked by 1BasToTheMax
  • StarnbergStarnberg Member
    edited August 2025

    A bit less to report than yesterday. Started by picking some local upstream NTP servers, replacing some of my generic, somewhat global default upstreams I had started out with. Installed trafficserver and started to configure it to expose the data that I think is worth sharing, aggregating multiple types of sources behind a common domain. As expected, it's not the big items anymore that allow fast progress, unlike yesterday, but more tedious nitty-gritty setting things up, and stuff I am less familiar with. E.g, added a certificate for TLS, enabled co-existence with lighttpd that will actually provide some content, started to tweak one source to give greater prominence to aspects I personally more relevant, and a few other, smaller things.

    After a rather cool period, it's gotten real hot over here (relative to what would be more typical for this location), so I will be spending less time in front of the screen the coming days, but plan to continue with lower intensity on some of the items I started, as mentioned above.

    Thanked by 1Not_Oles
  • Not_OlesNot_Oles Member, Patron Provider

    @Clanly said:
    hello Santa I need Moldova to working VPN

    How To Get Your Request Approved

    When somebody posts a request, and they have stuff like:

    • links to the sources of not obviously open source software they want to run,
    • maybe been around the LET community a while,
    • maybe a significant number of thanks on their profile,
    • maybe positive and helpful post and comment history,
    • can see their email on their LET profile,
    • polite a/k/a please and thank you,
    • no spammy posts,
    • no troll, nasty, or negative posts,
    • good jokes,
    • nice avatar,
    • nice signature,
    • maybe a link to their website and source code repo,
    • etc.

    This isn't a list of rules or requirements. Just a few ideas to consider when writing your request.

    More, or different stuff, might be okay. . . . Even brand new LETizens might be okay.

    The more effort you invest in your request, the more likely your request will be approved.

    Once you are in, it's lovely to post a little about what you are doing and how everything is going. Your contributions to higher quality discussion make it more likely that Providers will continue to donate servers and that FOSSVPS will continue giving away free VPSes to the community.

  • Not_OlesNot_Oles Member, Patron Provider

    Free Los Angeles VPS

    Been a few days now that we still have one available free Los Angeles VPS courtesy of @fmxm's kind donation <3 and @babywhale's work making the VPSes with BashVM. <3

    Here's a Yabs from a similar VPS on the same Node.

    Does anyone want this VPS?

    Best wishes!

    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    #              Yet-Another-Bench-Script              #
    #                     v2025-04-20                    #
    # https://github.com/masonr/yet-another-bench-script #
    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    
    Sat Jul 12 09:10:14 PM EDT 2025
    
    Basic System Information:
    ---------------------------------
    Uptime     : 0 days, 0 hours, 29 minutes
    Processor  : Intel(R) Xeon(R) CPU E3-1230 v6 @ 3.50GHz
    CPU cores  : 2 @ 3503.962 MHz
    AES-NI     : ✔ Enabled
    VM-x/AMD-V : ✔ Enabled
    RAM        : 1.9 GiB
    Swap       : 953.7 MiB
    Disk       : 72.8 GiB
    Distro     : Debian GNU/Linux 12 (bookworm)
    Kernel     : 6.1.0-37-amd64
    VM Type    : KVM
    IPv4/IPv6  : ✔ Online / ❌ Offline
    
    IPv4 Network Information:
    ---------------------------------
    ISP        : Cnservers LLC
    ASN        : AS40065 CNSERVERS LLC
    Host       : CloudRadium L.L.C
    Location   : Los Angeles, California (CA)
    Country    : United States
    
    fio Disk Speed Tests (Mixed R/W 50/50) (Partition /dev/vda1):
    ---------------------------------
    Block Size | 4k            (IOPS) | 64k           (IOPS)
      ------   | ---            ----  | ----           ---- 
    Read       | 142.18 MB/s  (35.5k) | 196.25 MB/s   (3.0k)
    Write      | 142.56 MB/s  (35.6k) | 197.28 MB/s   (3.0k)
    Total      | 284.74 MB/s  (71.1k) | 393.53 MB/s   (6.1k)
               |                      |                     
    Block Size | 512k          (IOPS) | 1m            (IOPS)
      ------   | ---            ----  | ----           ---- 
    Read       | 235.13 MB/s    (459) | 249.17 MB/s    (243)
    Write      | 247.63 MB/s    (483) | 265.77 MB/s    (259)
    Total      | 482.76 MB/s    (942) | 514.95 MB/s    (502)
    
    iperf3 Network Speed Tests (IPv4):
    ---------------------------------
    Provider        | Location (Link)           | Send Speed      | Recv Speed      | Ping           
    -----           | -----                     | ----            | ----            | ----           
    Clouvider       | London, UK (10G)          | 87.6 Mbits/sec  | 88.8 Mbits/sec  | 143 ms         
    Eranium         | Amsterdam, NL (100G)      | 88.7 Mbits/sec  | 89.3 Mbits/sec  | 151 ms         
    Uztelecom       | Tashkent, UZ (10G)        | 84.1 Mbits/sec  | 86.1 Mbits/sec  | 226 ms         
    Leaseweb        | Singapore, SG (10G)       | 84.4 Mbits/sec  | 88.4 Mbits/sec  | 182 ms         
    Clouvider       | Los Angeles, CA, US (10G) | 94.1 Mbits/sec  | 94.3 Mbits/sec  | 0.431 ms       
    Leaseweb        | NYC, NY, US (10G)         | 91.0 Mbits/sec  | 92.1 Mbits/sec  | 74.4 ms        
    Edgoo           | Sao Paulo, BR (1G)        | 83.8 Mbits/sec  | 85.7 Mbits/sec  | 212 ms         
    
    Geekbench 6 Benchmark Test:
    ---------------------------------
    Test            | Value                         
                    |                               
    Single Core     | 1446                          
    Multi Core      | 2549                          
    Full Test       | https://browser.geekbench.com/v6/cpu/12837563
    
    YABS completed in 11 min 32 sec3-manassas$ 
    
  • Not_OlesNot_Oles Member, Patron Provider

    @Starnberg

    Could you please check the CPU usage in your VPS? Thanks!

    PID 102721 seems to be your VPS.

    I am seeing:

    top - 20:47:59 up 7 days, 22:39,  2 users,  load average: 1.87, 1.63, 1.56
    Tasks: 458 total,   1 running, 457 sleeping,   0 stopped,   0 zombie
    %Cpu(s):  3.0 us,  3.6 sy,  0.0 ni, 93.4 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st 
    MiB Mem : 257874.7 total, 209633.0 free,  13504.5 used,  36557.1 buff/cache     
    MiB Swap:   2048.0 total,   2048.0 free,      0.0 used. 244370.2 avail Mem 
    
        PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND           
     102721 libvirt+  20   0   10.2g   3.8g  20800 S 108.0   1.5     30,22 qemu-system-x86   
     184210 libvirt+  20   0   10.2g   4.1g  20800 S  45.2   1.6      9,46 qemu-system-x86   
     141504 libvirt+  20   0 8442384   1.3g  21120 S   1.7   0.5  37:06.39 qemu-system-x86   
      80026 libvirt+  20   0   10.2g   3.5g  21120 S   0.7   1.4  66:15.94 qemu-system-x86   
     236519 root      20   0    9376   5440   3520 R   0.7   0.0   0:00.05 top               
        268 root      25   5       0      0      0 S   0.3   0.0  22:54.83 ksmd              
        764 root      rt   0  289104  25920   8000 S   0.3   0.0   0:53.17 multipathd        
     141518 root      20   0       0      0      0 S   0.3   0.0   0:45.31 kvm-pit/141504    
     236379 root      20   0   14972   9600   8320 S   0.3   0.0   0:00.18 sshd              
          1 root      20   0   22704  12480   8640 S   0.0   0.0   0:48.98 systemd           
          2 root      20   0       0      0      0 S   0.0   0.0   0:00.43 kthreadd          
          3 root      20   0       0      0      0 S   0.0   0.0   0:00.00 pool_workqueue_r+ 
          4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-rcu_g   
    
  • Not_OlesNot_Oles Member, Patron Provider

    @Starnberg

    Here's another top which is even higher. What's up?

    Tasks: 462 total,   1 running, 461 sleeping,   0 stopped,   0 zombie
    %Cpu(s):  3.9 us,  4.1 sy,  0.0 ni, 91.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st 
    MiB Mem : 257874.7 total, 209669.8 free,  13464.0 used,  36560.9 buff/cache     
    MiB Swap:   2048.0 total,   2048.0 free,      0.0 used. 244410.7 avail Mem 
    
        PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND           
     102721 libvirt+  20   0   10.2g   3.8g  20800 S 166.7   1.5     30,42 qemu-system-x86   
     237493 root      20   0    9376   5120   3200 R  16.7   0.0   0:00.03 top               
     141504 libvirt+  20   0 8442384   1.3g  21120 S   8.3   0.5  37:21.72 qemu-system-x86   
     184210 libvirt+  20   0   10.2g   4.1g  20800 S   8.3   1.6      9,58 qemu-system-x86   
          1 root      20   0   22704  12480   8640 S   0.0   0.0   0:49.12 systemd           
          2 root      20   0       0      0      0 S   0.0   0.0   0:00.43 kthreadd          
          3 root      20   0       0      0      0 S   0.0   0.0   0:00.00 pool_workqueue_r+ 
          4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-rcu_g   
          5 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-rcu_p   
          6 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-slub_   
          7 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-netns   
          9 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/0:0H-eve+ 
         12 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-mm_pe   
    root@alexhost:~# 
    
  • Not_OlesNot_Oles Member, Patron Provider

    @Starnberg Your VPS has been shut down. Something seems not quite right. . . . Ideas?

  • StarnbergStarnberg Member
    edited August 2025

    @Not_Oles said: Your VPS has been shut down.

    Ah, already noticed it, now I know why.

    @Not_Oles said: Something seems not quite right. . . .

    Agree.

    @Not_Oles said: Ideas?

    Hmm, not really, in the sense that I don't do anything that I know would, or even could, cause something like that... Or do anything that would jeopardize use of this kind gift.

    Some suspicions:

    • The NTP load in the zone could be much higher than originally seen (there can be quite significant variations that could have hidden the true situation in the zone so far, both occurring "naturally" (e.g., daily/weekly variations (e.g., with one setting, Singapore varies between ~1.5 Mbit/s during the night and 6+ Mbit/s during the day), or DNS cache for a large number of clients behind CGNAT being refreshed pointing to a new "victim", misbehaving clients, another big server going offline, ...), as well as due to issues on the Pool infrastructure side, e.g., the zones not being refreshed, or a DNS server getting stuck and continuously handing out the same set of IP addresses, rather than iterating through them). But chronyd is essentially single-threaded, so it shouldn't ever go higher than 100%. And with the packet rate required to drive chronyd's load that high, I'd suspect that something else outside the VPS would give first (and massively start dropping packets) before chronyd would even reach 100% CPU load (never seen it do that before).
    • I have no prior experience with the HetrixTools agent, so cannot vouch for its behavior (but perhaps unlikely).
    • The screensnap container is the only one that can cause higher loads, but usually only peaks, when rendering of an image is triggered. But I've once before seen something getting stuck, and old processes not getting reaped, and accumulating. But that was on a way slower and less powerful machine (Raspberry Pi), so it was overwhelmed processing requests, and they kept piling up until the machine started thrashing. That shouldn't be the case here, still my current best guess as being the culprit. Simply restarting it (or the whole VM) should clear that up, and I'll disable it once I can access the machine again.

    To be sure, I'd need to take a look. Unfortunately, the different timezones hamper communication, else I would most definitely have addressed this last night right away, but I didn't see your messages before signing off for the day, found them this morning only.

    Sorry for the hassle, nothing of what I did was intended to cause these issues, or should even cause that much load (I am even considering relinquishing some of the VPS' resources so that perhaps another small one could be carved out of them). I have a very similar setup on a much smaller device, without issues that would lead me to believe this would be a common occurrence.

    Thanked by 1Not_Oles
  • First of all, thank you to alexhost @Not_Oles for providing the free VPS. <3

    I'm a development engineer who primarily works on server-side applications, and sometimes I need to use CI/CD to deploy and maintain projects.

    Kubernetes is an excellent container orchestration and management platform that's becoming increasingly popular in enterprise development. While I'm not a professional DevOps engineer, I'm very interested in server and container technologies, studying the official documentation to understand its architecture and powerful features. I have a Windows laptop with low specs that can't run Linux through VMware, and previously I only used AWS free tier 1c1g EC2, which also couldn't handle k8s setup. Now with this VPS, I can build k8s following the official documentation and deploy applications - only through hands-on practice can you truly understand k8s's rich functionality.

    Below I'll share how I'm using this machine, hoping my experience can provide some small help to everyone.

    Reference: Official documentation https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/

    Core Steps:

    1. Disable swap and configure IPv4 forwarding
    2. Install Docker and cri-dockerd (you can also use containerd, but I want to deploy Portainer to monitor Docker, so I'm installing cri-dockerd)
    3. Install kubelet, kubeadm, and kubectl
    kubeadm init --pod-network-cidr=10.244.0.0/16 --cri-socket /var/run/containerd/containerd.sock --kubernetes-version=v1.29.0
    

    Since I only have one machine, my node serves as both control plane and worker node.

    1. Install CNI - I'm using Flannel, reference: https://github.com/flannel-io/flannel

    2. Deploy other plugins such as:

    3. Deploy ingress-nginx-controller and management dashboard - you can choose Rancher, but I recommend Kite, a lightweight management panel: https://github.com/zxh326/kite

    Now you can have fun!

    You can deploy infrastructure like GitLab, MySQL, PostgreSQL, Elasticsearch, Kafka - local development applications can connect to these databases.

    Recommended Useful Projects:

    A few days ago, my best friend purchased three VPSs and set up a k8s cluster running open source projects. I wanted to join his cluster, but there's a big challenge - clusters typically communicate within the same LAN. I found a solution using Tailscale networking:

    curl -fsSL https://tailscale.com/install.sh | sh
    tailscale up
    

    Follow the steps to successfully create a tunnel.

    First, modify Flannel's default configuration:

    wget https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
    

    Modify kube-flannel.yml to add --iface=tailscale0:

    containers:
      - args:
          - '--ip-masq'
          - '--kube-subnet-mgr'
          - '--iface=tailscale0'
    
    kubectl apply -f kube-flannel.yml
    

    On his control plane node, execute the command to get the join command:

    kubeadm token create --print-join-command
    

    Note: If your friend's cluster was initialized with the parameter --apiserver-advertise-address=<public ip>, you need to perform the following operation on my node:

    vim /etc/default/kubelet
    KUBELET_EXTRA_ARGS="--node-ip=100.104.55.20"  # Fill in Tailscale IP
    
    Thanked by 2Starnberg Not_Oles
  • @Starnberg said:

    @Not_Oles said: Your VPS has been shut down.

    Ah, already noticed it, now I know why.

    @Not_Oles said: Something seems not quite right. . . .

    Agree.

    @Not_Oles said: Ideas?

    Hmm, not really, in the sense that I don't do anything that I know would, or even could, cause something like that... Or do anything that would jeopardize use of this kind gift.

    Some suspicions:

    • The NTP load in the zone could be much higher than originally seen (there can be quite significant variations that could have hidden the true situation in the zone so far, both occurring "naturally" (e.g., daily/weekly variations (e.g., with one setting, Singapore varies between ~1.5 Mbit/s during the night and 6+ Mbit/s during the day), or DNS cache for a large number of clients behind CGNAT being refreshed pointing to a new "victim", misbehaving clients, another big server going offline, ...), as well as due to issues on the Pool infrastructure side, e.g., the zones not being refreshed, or a DNS server getting stuck and continuously handing out the same set of IP addresses, rather than iterating through them). But chronyd is essentially single-threaded, so it shouldn't ever go higher than 100%. And with the packet rate required to drive chronyd's load that high, I'd suspect that something else outside the VPS would give first (and massively start dropping packets) before chronyd would even reach 100% CPU load (never seen it do that before).
    • I have no prior experience with the HetrixTools agent, so cannot vouch for its behavior (but perhaps unlikely).
    • The screensnap container is the only one that can cause higher loads, but usually only peaks, when rendering of an image is triggered. But I've once before seen something getting stuck, and old processes not getting reaped, and accumulating. But that was on a way slower and less powerful machine (Raspberry Pi), so it was overwhelmed processing requests, and they kept piling up until the machine started thrashing. That shouldn't be the case here, still my current best guess as being the culprit. Simply restarting it (or the whole VM) should clear that up, and I'll disable it once I can access the machine again.

    To be sure, I'd need to take a look. Unfortunately, the different timezones hamper communication, else I would most definitely have addressed this last night right away, but I didn't see your messages before signing off for the day, found them this morning only.

    Sorry for the hassle, nothing of what I did was intended to cause these issues, or should even cause that much load (I am even considering relinquishing some of the VPS' resources so that perhaps another small one could be carved out of them). I have a very similar setup on a much smaller device, without issues that would lead me to believe this would be a common occurrence.

    Have you set these deployed processes to start automatically?
    If the issue persists after a reboot, it’s likely caught in an infinite loop, which could eventually force a full system reinstall.
    I’ve encountered similar high-load problems like this in my work before.

    Thanked by 1Not_Oles
  • Hello @Aphelios,

    Thanks for your thoughts, very appreciated.

    @Aphelios said: Have you set these deployed processes to start automatically?

    Yes.

    @Aphelios said: it’s likely caught in an infinite loop

    While almost anything is possible, I don't think that is the case here. I don't expect I botched anything that's supposed to happen during the startup process that might cause this. Slight concern always is that I lock myself out of a system, especially when I don't have direct access to a console, or to the virtual power button, myself. But that was not the issue, or an issue, yesterday, and I hope for that to still be true once the machine comes back up again.

    @Aphelios said: eventually force a full system reinstall

    That seems like a drastic measure. As long as the VM can be accessed, if need be after an externally triggered or even forced reboot, the issue can be analyzed, either through post-mortem, or by setting up more fine-grained monitoring to detect an impending issue before it happens, or at least have more detailed post-mortem info the next time around.

    But I am rather confident I'll be able to figure out what was going on, and fix it going forward, once I can access the VM again. Will obviously share my findings here, and strongly hope that a re-install can be avoided.

    Thanks again!

    Thanked by 1Not_Oles
  • Not_OlesNot_Oles Member, Patron Provider

    @Starnberg said:

    The screensnap container is the only one that can cause higher loads . . . That shouldn't be the case here, still my current best guess as being the culprit. Simply restarting it (or the whole VM) should clear that up, and I'll disable it once I can access the machine again.

    If a restart could be helpful, certainly we could try that.

    To be sure, I'd need to take a look. Unfortunately, the different timezones hamper communication, else I would most definitely have addressed this last night right away, but I didn't see your messages before signing off for the day, found them this morning only.

    Yes, of course.

    I will try a restart and report back here about what happens.

  • Not_OlesNot_Oles Member, Patron Provider

    @Starnberg

    I will leave your VPS on so you can go in and take a look.

    Please let us know what happens. Thanks!

    Best!

    Tom

    top - 15:00:03 up 8 days, 16:51,  2 users,  load average: 1.93, 1.00, 0.56
    Tasks: 462 total,   1 running, 461 sleeping,   0 stopped,   0 zombie
    %Cpu(s):  5.0 us,  5.3 sy,  0.0 ni, 89.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st 
    MiB Mem : 257874.7 total, 205245.3 free,  11995.0 used,  42454.4 buff/cache     
    MiB Swap:   2048.0 total,   2048.0 free,      0.0 used. 245879.7 avail Mem 
    
        PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND   
     273181 libvirt+  20   0 8465848   1.5g  20800 S 177.2   0.6   2:21.17 qemu-sys+ 
     184210 libvirt+  20   0   10.1g   4.1g  20800 S  58.4   1.6     19,01 qemu-sys+ 
     141504 libvirt+  20   0 8581652   1.3g  21120 S   3.0   0.5  52:35.90 qemu-sys+ 
     273335 root      20   0    9372   5440   3520 R   1.0   0.0   0:00.43 top       
          1 root      20   0   22704  12800   8640 S   0.3   0.0   0:55.27 systemd   
          2 root      20   0       0      0      0 S   0.3   0.0   0:00.48 kthreadd  
       1229 message+  20   0   10172   4800   4480 S   0.3   0.0   0:08.19 dbus-dae+ 
      80026 libvirt+  20   0   10.2g   3.5g  21120 S   0.3   1.4  76:34.85 qemu-sys+ 
     273194 root      20   0       0      0      0 S   0.3   0.0   0:00.05 kvm-pit/+ 
          3 root      20   0       0      0      0 S   0.0   0.0   0:00.00 pool_wor+ 
          4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/+ 
    
    Thanked by 1Starnberg
  • @BasToTheMax said:
    Maybe I could provide some small NAT LXC containers to the LET community!

    Hi everyone!

    To continue on my idea, I have a few questions for the community:

    • Which Linux distro would you like to see?
    • The node has 4 GB ram, 2 vCPU and 50 GB disk (about 40-45 GB available after OS + base ct images), provided by FOSSVPS! How many resources should each container get?
    • What would you do with the LXC container / what applications would you deploy?

    And lastly, I want to thank @Not_Oles for starting FOSSVPS and all the generous providers who offered a node!

    If my free nat lxc hosting grows, maybe one could donate a node for it too!

    Thanked by 1Not_Oles
  • StarnbergStarnberg Member
    edited August 2025

    @Not_Oles said: Please let us know what happens.

    Thank you for restarting the VPS, allowed me to take a look. And it is really puzzling, because for one, the internal view at least of the overall load is totally different from what you see on the outside:

    top - 15:26:12 up 29 min,  1 user,  load average: 0.06, 0.04, 0.05
    Tasks: 201 total,   1 running, 199 sleeping,   1 stopped,   0 zombie
    %Cpu(s):  0.3 us,  0.5 sy,  0.0 ni, 99.2 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st 
    MiB Mem :   3915.3 total,   2783.8 free,    570.6 used,    799.0 buff/cache     
    MiB Swap:   1024.0 total,   1024.0 free,      0.0 used.   3344.7 avail Mem 
    
        PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                                           
       1277 traffic+  20   0 1176928  60072  15744 S   3.3   1.5   1:00.52 [TS_MAIN]                                                                                                                                                         
        877 root      20   0   29332   6784   3840 S   0.7   0.2   0:00.40 [obfuscated for privacy]                                                                                                                                                          
      48585 root      20   0   12356   5760   3584 R   0.7   0.1   0:00.81 top                                                                                                                                                               
        859 root      20   0 1876272  47284  33280 S   0.3   1.2   0:03.03 containerd                                                                                                                                                        
        930 _chrony   -2   0   23372  23328  10240 S   0.3   0.6   0:06.45 chronyd                                                                                                                                                           
          1 root      20   0   22448  13312   9344 S   0.0   0.3   0:03.19 systemd                                                                                                                                                           
          2 root      20   0       0      0      0 S   0.0   0.0   0:00.01 kthreadd                                                                                                                                                          
          3 root      20   0       0      0      0 S   0.0   0.0   0:00.00 pool_workqueue_release                                                                                                                                            
          4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-rcu_g                                                                                                                                                   
          5 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-rcu_p                                                                                                                                                   
          6 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-slub_                                                                                                                                                   
          7 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-netns                                                                                                                                                   
         10 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/0:0H-events_highpri                                                                                                                                       
         11 root      20   0       0      0      0 I   0.0   0.0   0:00.03 kworker/u8:0-ext4-rsv-conversion                                                                                                                                  
         12 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-mm_pe                                                                                                                                                   
    [...]
    

    And for another, even the internal accounting inside the VM. [TS_MAIN] is the main trafficserver process. While the current CPU percentage seems low, the accumulated time seems rather high, the current "uptime" of the system itself is just 30-something minutes. Essentially, trafficserver is currently idling, not serving any pages right now (except some automatic reload of some test page by my browser), not even publicly reachable while I am still tinkering with the setup until I think it is secure enough for general access by the Internet population (once public, I expect not only the friendly people from the LET community wanting to take a look, less at the content rather than other aspects of the system...)

    My hypothesis would be that the way trafficserver handles its idle time, doing essentially nothing but internal stuff while waiting for external requests, is interacting badly with the virtualization layer.

    To test the hypothesis, I stopped trafficserver for now, @Not_Oles please let me know at your earliest convenience whether you see a difference now.

    I also faintly remember having seen something in the documentation about tweaking the main loop behavior, will try to find that again to see whether there might be an explanation that can be derived from that, and, e.g., a different setting/mode of operation for the main loop that is friendlier to the host system.

  • ApheliosAphelios Member
    edited August 2025

    @BasToTheMax
    Thanks for offering this service!
    I often write Linux-specific code — for example, using epoll.h, which only exists on Linux. On mac or win, the compiler and IDE will flag it as missing, so I need a real Linux environment to compile and test.

    I would prefer Debian 12 or Ubuntu 22.04 for the container, since they’re stable, lightweight, and have good package availability. I’d install gcc, g++, make, and other build tools, then set up SSH access and run VSCode Server or JetBrains Clion for remote coding and debugging from my local machine.

    For developing micro or small open-source projects, 0.5-1 vCPU and 0.5-1 GB RAM should be sufficient. For example, a lightweight web server project like https://github.com/oatpp/oatpp

    However, remote development always has some latency, so the experience isn’t as smooth as working locally. I’ve been considering buying a cost-effective Mac mini M4 with 32 GB RAM and 512 GB SSD to use as my main development machine, and use Docker Desktop for testing and debugging. o:)

    Thanked by 2BasToTheMax Not_Oles
  • Not_OlesNot_Oles Member, Patron Provider

    @Starnberg said: To test the hypothesis, I stopped trafficserver for now, @Not_Oles please let me know at your earliest convenience whether you see a difference now.

    Seems to be a big difference!

    top - 16:13:53 up 8 days, 18:05,  2 users,  load average: 0.39, 0.33, 0.36
    Tasks: 464 total,   1 running, 463 sleeping,   0 stopped,   0 zombie
    %Cpu(s):  1.4 us,  1.7 sy,  0.0 ni, 96.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st 
    MiB Mem : 257874.7 total, 205498.9 free,  11685.0 used,  42510.7 buff/cache     
    MiB Swap:   2048.0 total,   2048.0 free,      0.0 used. 246189.7 avail Mem 
    
        PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND   
     184210 libvirt+  20   0   10.1g   4.1g  20800 S  63.4   1.6     19,37 qemu-sys+ 
     273181 libvirt+  20   0 8623300   1.6g  20800 S   9.9   0.6  11:54.08 qemu-sys+ 
     275987 root      20   0    9372   5440   3520 R   1.0   0.0   0:00.27 top       
      80026 libvirt+  20   0   10.2g   3.5g  21120 S   0.7   1.4  76:59.98 qemu-sys+ 
     141504 libvirt+  20   0 8573456   1.3g  21120 S   0.7   0.5  53:07.95 qemu-sys+ 
        268 root      25   5       0      0      0 S   0.3   0.0  25:53.33 ksmd      
        728 root      19  -1  136740  89588  88948 S   0.3   0.0   1:15.36 systemd-+ 
     276000 root      20   0   12148   8320   7360 S   0.3   0.0   0:00.01 sshd      
          1 root      20   0   22704  12800   8640 S   0.0   0.0   0:55.64 systemd   
          2 root      20   0       0      0      0 S   0.0   0.0   0:00.48 kthreadd  
          3 root      20   0       0      0      0 S   0.0   0.0   0:00.00 pool_wor+ 
    
    Thanked by 1Starnberg
  • @Not_Oles said: Seems to be a big difference

    Thanks for checking, and letting me know!

    Hmm, not sure how to proceed. Will need to mull over this a bit. Nothing from trafficserver's documentation so far sprang out at me as suggesting it might be related to that. Need to dig a bit deeper, but also let it rest and mature a bit as well.

    Obvious way out would be to just switch to another reverse proxy, I'd just prefer to avoid that. Not on principle, I am always happy to learn, and to explore new things, so not a blocker. It would just mean a bit more effort, and related delay, to get sufficiently acquainted with another implementation which I would rather do without less "pressure" (and if only just perceived, or coming from myself) to deliver some outcome.

    By the way, if not too much trouble, is there a way I could get some sort of direct view myself at the host's live performance data? I might want to try a few things, and it might be easier, and both more efficient and effective, to not have to bother you all the time about taking a look and sharing the outcome.

    Will think about the situation, and come back once I have a clearer picture at to a potential direction forward.

    Thanks!

Sign In or Register to comment.