New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
A helpful gentleman asked me:
My reply:
Is my guess correct that it's not a network issue causing the high %CPU?
Here's a Yabs from inside the VPS that I just made. Its IPv4 is one from the newly assigned group. The results seem pretty good, especially considering that quite a few of the existing VMs are running 100% CPU. These results are -- if memory serves -- consistent with earlier Yabs when the node was not stressed.
(SUnet is the Swedish University computer NETwork, just used it as an example to get a bit of a longer route and to a rock solid node).
If I can assist in any way do not hesistate to reach out! I might take a while to answer from time to time but when I'm here I'm here :P
@zejjnt
Thanks very much! Now we know for sure (almost!
) that at least 4 VMs are working: my test VM, @FrankCastle, the new VM I just made, and yours. That's great!
Thanks especially for your offer to help!
Now that we know (probably!
) that I can make new VPSes with the newly assigned IPv4s, and now that we know several VMs survived the netplan adjustment, reversion, and the reboot -- the question on the table is why are so many (8, 9?) VMs running 100% CPU apparently without console or WAN connectivity (haven't tested more than @Starnberg's)?
So, restarting libvirtd didn't fix network connectivity on @Starnberg's VPS.
More ideas, please!
Thanks for looking into this as well!
I am currently still downloading the image, will be able to look into it tomorrow. In the meantime, please don't invest too much time into the issue with my VPS - unless there's others having issues as well. I'll try to figure out what's up with the image, and would propose to take it from there once I have a better understanding as to whether it's an issue with the image, or when I cannot find anything wrong with it.
Thanks!
Signing off for the day, have a good rest of the day whoever still has more hours to go until then!
OpenBSD fixes everything!
I haven't even turned off the 100% CPU VPSes. I thought maybe I might get lucky and find a fix.
The issue with starting over fresh is that freshness might leave the issue intact and hiding, ready to pop up again soon.
Where is the libvirt genius when we need him?
I want to try again to make a Debian VPS, which now might be working. I want to go through my VM creation procedure, and post it, so everybody can see everything and suggest improvements. It would be nice to setup, as @cmeerw kindly suggested, a unix socket interface for user accessible consoles.
But, yeah, it might be getting time to refresh! Unless someone has a better idea!
Excellent! Have a good sleep! Thanks for all your help and kindness!
Given the lack of good ideas about the high %CPU VPSes, and in the absence of objection, I propose to shut them down, perhaps as soon as tomorrow.
One thing we might try is reverting one of the high CPU VPSes to its fresh snapshot. Maybe they all will work again if reverted. Otherwise I guess they could be reinstalled.
In the future, we could make snapshots after users finish their basic installs. With "basic" snapshots we could revert to an installed state.
I made a second new Ubuntu 24.04 VPS with another of the newly assigned IPv4s. It works great and will be assigned to someone, along with more, soon. You can see a list of the requests at https://fossvps.org/list.html.
Thanks everyone!
Thanks Alexhost! 
Hello @Not_Oles
Debian 12
I want to use the free VPS to host my personal website (as alternatives to the one I am paying for).
Here is some of my past project:
This is my github profile: https://github.com/ShayNeeo
I will ensure my request will not be use for any business-related case.
I can pm ssh key
I can wait till there are mode VPSes available.
Thanks @alexhost, thanks Not_Oles!
@shayneeo
Hello! Nice to meet you!
Your application is accepted. I will add you to the list at https://fossvps.org/list.html.
Would you be okay with IPv6-only in Chisinau, or do you need IPv4? Thanks Alexhost!
I have been wanting to try making a few more IPv6-only VPSes in London. Maybe you might like IPv6-only in London? That server has a pretty good network. Thanks OnlyServers!
Here are a few additional non-FOSSVPS possibilities which might be of interest to you. Two dedicated servers: an E3-1240 v5 in Bucharest running NetBSD, and a dual E5-2683 V4 in Texas running Debian sid. Besides these dedis, I could share a couple of new, fast Ryzen 9 9950X VPSes in the US, one running self-built NetBSD-current, the other running Debian sid.
Please go ahead when you have time and email or PM your ssh public key.
Thanks for your request! Nice work on your projects!
I sent the public key to you.
And yeah I want to thanks OnlyServers too. Because I am likely to receive the VPS from OnlyServers.
Thank you for providing such a good services for developers around the Internet.
Hello!
At the moment there are 27 VPSes configured on the Chisinau server.
I don't know why, but 8 VPSes seemed to be running continuously at high %CPU following recent update and revert of the netplan. As far as I am aware, the remaining 19 VPSes are working fine.
All 8 of the overperforming VPSes have been shutdown. From here at LET, @Aphelios, @seoky, @rickey, @thenet, and @BreakaWD are down.
Do you gentlemen want me to attempt restoring your VPSes from your "fresh" snapshots? Attempting the restore will destroy whatever happened subsequent to the snapshot. Alternatively, we could rebuild your VPSes from scratch. Please let me know.
Or, maybe, someone has a better idea about how to adjust the overperforming VPSes?
Thanks! Sorry for the issue!
Thanks to Alexhost! 
Tom
Yes, I’ve noticed the VPS is unreachable. It’s likely the deployed application is failing and repeatedly restarting, which is driving CPU usage through the roof. Could you please rebuild the VPS for me? Thanks.
Hi @Not_Oles,
I've noticed that my VPS has been shut down, but weirdly on Hetrix it hasn't shown any high cpu usage, as seen here on the screenshot:
Could you please check what has exactly happened there? Also, could you please add my other key, ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBDoJsBKotHRqRQCSi+oj4bi9DHfcHuAtIMlgt4aPMhy [email protected] ? It’s fine to reset it to the snapshot, but I’d appreciate if you could let me know what happened to the original VPS.
Thanks a lot!
Thanks for the heads-up, Tom. A rebuild is totally fine for me. Appreciate you sorting it out!
Hi @Aphelios!
I rebuilt your VPS as requested. Yabs run inside your newly created VPS is below.
I double checked that I can get in to your newly recreated VPS via ssh with both IPv4 and IPv6.
The VPS has your ssh key so you should be able to get in with the same user name and IP addresses as before. As you know, there will be new host ssh keys, so you will have to adjust your known_hosts file.
Password login is disabled.
Please feel free to remove my keys and to change the password.
When you have a chance, please post here again to let us know whether you can get in okay and if everything seems working as expected.
When you have finished your basic install, please consider asking me to make a snapshot so that maybe you won't have to reinstall again in the hopefully unlikely event of more issues.
I hope you have fun on your VPS!
Thanks very much, as always! Best wishes!
Tom
Hi @rickey!
How should I check exactly what has happened with your VPS?
With one of the troubled VPSes, the client took a copy of the qcow2 image, the seed.iso, the libvirt xml file, and the backing image. He set everything up in a test environment and did a lot of detective work, but without any definitive result.
I have looked through everything shallowly with my limited technical skills. I haven't found the root cause.
I do not know why your VPS and several others were running high CPU. I wish I did know!
One thing I could do is give you also a copy of your qcow2 image, seed.iso, xml file, and the backing image. Then you would be able to go through and investigate.
Reverting to the snapshot would change the qcow2 image, so I have not reverted, wanting to save the original state or as much as I could in case you wanted to check it.
Additionally, none of the other stressed VPSes have successfully reverted to their snapshots. Not even one of the several I tried successfully reverted. After reverting, they go rapidly to very high CPU again.
For this current situation, based on several trials with different VPSes, reverting to snapshot seems not useful. So I now prefer to go directly to rebuilding from scratch. Of course it's no problem to add your additional ssh key.
But before I rebuild, do you want copies of your existing files?
Thanks very much! Sorry for the inconvenience!
Best!
Tom
@seoky Thanks for your message! You are the next rebuild. Maybe this evening, maybe tomorrow morning. I will post here when your VPS is again ready. Best wishes for sure! Tom
@seoky
Here is a Yabs from your newly rebuilt VPS. Looks like it might be okay.
Login should be the same as before. I double checked that your VPS reboots okay and that I can access via ssh on both IPv4 and IPv6.
My ssh keys are there together with your key. Please feel free to remove my keys.
Password login is disabled. No need for password for sudo. Default user password same as before. Please feel free to change the password.
When you have a chance, please post here to let us know if you get in okay and if everything seems okay to you. If you have trouble I will try to assist.
If you want, when you get your basic install done, please consider letting me know. We can shut down and make another snapshot.
Sorry about the inconvenience! Thanks for your patience and kindness!
Best!
Tom
Hi @Not_Oles,
rebuilding the VPS from scratch sounds like the most reasonable solution. The VPS hasn't had any important data, so I wouldn't want to waste yours and mine time on recovering the files.
Thank you for adding my second ssh key. Please go ahead with the rebuild.
I might have messed something up, I attempted an OS reinstall, and now the VPS is unreachable (no ping, no SSH). Would you be able to check or see what's going on?
@seoky Hello!
Your VPS is back to using high CPU.
So far, the VPS images have been made with qemu-img's "-b" backing store flag. This saves disk space. However, my guess is that backing store might break some internal OS reinstalls and perhaps also some Docker installs.
I might try rebuilding your VPS without the backing store. It might work better for you that way.
I will post again when the rebuild and testing are done.
Best wishes!
Tom
@seoky
Your VPS now is remade without the backing store. In other words, you have the full Ubuntu cloud image within your own qcow2.
Your keys and password remain the same as before.
Please let me know if you can get in and if everything seems okay.
If you want, when you get your basic install done, we can make a snapshot. If you have any questions or problems, I will try to help.
Have fun with your VPS!
Best wishes!
Tom
Hi @JohnDoherty!
Welcome to LET!
To answer your question about the "catch," please take a look at the Rules and Warnings section at https://fossvps.org/.
If you have more questions, please post again.
Best wishes!
Tom
why is your fish story hosted on google?
I arrived at this thread when looking a vps for a tiny non-profit org.
I have to say I find all this issue quite intriguing. My suspicion is at some driver code interaction with the network. Perhaps a spinlock which is never exited. That would explain that a different OS (OpenBSD). or even a different flavor of Linux not showing it.
Still, that doesn't explain why other VPS using the same template and host didn't suffer that.
Regarding debugging, you mention taking a copy of the qcow2 image, the seed.iso, the libvirt xml file, and the backing image. But I think it misses an important piece: a memory dump. The ideal scenario would be to trace what is being run which uses 100% CPU. Is it a userland process? A kernel thread? On which function?
Not that such debugging would be necessarily easy, though, specially if needing to replicate it on a different environment- But it seems more promising than trying to replicate the issue from just the disk. If the bug is triggered by something external (such as certain network packet), it may never be reproduced locally.
Your suggested scenario sounds like a good idea. It's above my current skill level, but I could try. Unfortunately, the 100% issue has not shown up on any VPS at all during my initial testing. Nor has any issue appeared on the VPS which I have used for testing since the original server install.
The assigned VPSes are private to the assignees, so I wouldn't go delving into their memory without permission.
Perhaps I should add that several clients have mentioned issues with Docker and also with scripts which internally reinstall to another OS.
Recently I have switched to making the VPSes without using the backing store. Now all the new Ubuntu VPSes work great. As you can see I have been posting Yabs from all the newly made VPSes.
If I see the 100% issue appear on a VPS which I am working on, then we can run memory testing.
Thanks very much for your helpful comments! I appreciate them very much!
Update
The FOSSVPS website has been updated to show the total of 47 previously assigned VPSes (up from the September 12 count of 40) and the total of 5 VPSes currently available for assignment.
The FOSSVPS Request List has been updated to move Assigned VPSes to the bottom so that the list of Pending and Accepted Requests at the top of the page is easier to read. If you have a Pending or an Accepted Request, please double check that your Request appears on the List.
Thanks everyone!
VPSes for @Aphelios and @seoky have been rebuilt. Can you guys get in to your new VPSes? How are they doing?
@rickey is next for rebuild.
@thenet and @BreakaWD Do you want me to rebuild your VPSes?
Thanks everyone!