Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Provider-Supplied Linux Templates and .iso Image Files for VPSs - Are They Secure? How Do You Know?
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Provider-Supplied Linux Templates and .iso Image Files for VPSs - Are They Secure? How Do You Know?

emgemg Veteran
edited November 2022 in General

Question #1:
How do YOU authenticate and validate provider-supplied operating system templates for proper configuration and good security? Are the templates secure? How do you know?

So far, all I have come up with is, "Trust the provider to get it right or install your own from an iso image." (True, you get that choice with KVM, but not with OpenVZ.)

Question #2:
Is there a way to compute a checksum on a mounted .iso file, where the provider mounts it for you?

In other words, how can you verify that the mounted .iso image is the same .iso file that you requested?

.

Personal note: I was ready to start this thread a week or two ago, but someone mooned LowEndTalk, which created a distraction. Yesterday, @dustinc created a new thread announcing that Racknerd had added new templates for the latest AlmaLinux, Rocky Linux, and CentOS, which reminded me.

Comments

  • bulbasaurbulbasaur Member
    edited November 2022

    @emg said: How do YOU authenticate and validate provider-supplied operating system templates for proper configuration and good security? Are the templates secure? How do you know?

    You could get the checksums of all files on the systems on a fresh install, and check them against virustotal. Though, this is a tedious enough process that even the most paranoid people would be discouraged from doing so. Not to mention the virustotal API is ratelimited, so you'd probably need a VT enterprise subscription.

    @emg said: Is there a way to compute a checksum on a mounted .iso file, where the provider mounts it for you?

    Again, you can't. You could use the verify checksums option that the ISO provides, but this is to protect against tampering, and someone could change the checksums if they're taking the trouble to add malware in the ISO. Perhaps the best way to address the issue of a provider mounting a malicious ISO could be to use netboot.xyz, though it simply delegates the problem of the netboot.xyz ISO not being malicious itself.

    Thanked by 1emg
  • Realistically you can't. Even if you verified every byte on your disk, if you're using a VPS, the provider could modify your memory, sniff keys, whatever whenever they wanted and you'd have absolutely no way of detecting it. Many templates also run qemu-guest-agent which allows the host to run arbitrary commands as root.

    You kind of just need to consider whether any provider is going to care enough about you to want to do that. If you're just one average Joe out of 10,000 other customers just running a small website, it's literally not worth anybody's time to even care what's on your machine (as long as it's legal and not going to cause them problems later down the line).

    TBH, even with a dedi, you have no idea what's really in the hardware. If someone cared enough about your data, they could provision you with a drive with backdoored firmware that allows them to extract data out, etc. Who knows even if your CPU is really running bare-metal or actually has a hypervisor layer hiding above that.

    If you're a huge enterprise customer with data that's actually important enough that someone would want to steal it, then you are probably only safe with on-site hosting. Even colocation somewhere that you think you can absolutely trust and where you have the only keys to the rack, if someone who was determined enough they could gain physical access to the server and do something to it.

    But as for the template on a VPS? shrug It's probably not a concern, the risk to their reputation if it was discovered they were installing malicious software would be enough to discourage them.

    Thanked by 3emg dev077 Void
  • mrs92mrs92 Member
    edited November 2022

    Q1: Scan all data with some AV. Counting number of security updates that weren't installed. Full-upgrade your OS if possible.
    eg. for debian
    apt list --upgradable 2>/dev/null | grep "\-security" | wc -l
    apt update && apt upgrade && apt full-upgrade

    Q2: Have no idea. Would better if go with the providers who have the upload iso feature.

  • @ralf said: But as for the template on a VPS? shrug It's probably not a concern, the risk to their reputation if it was discovered they were installing malicious software would be enough to discourage them.

    It's not always the provider :D

    https://lowendtalk.com/discussion/169685/vulnerability-in-solusvm-debian-10-template-debianuser-backdoor-default-user/p1

    Thanked by 2ralf dev077
  • bulbasaurbulbasaur Member
    edited November 2022

    @ralf said: Who knows even if your CPU is really running bare-metal or actually has a hypervisor layer hiding above that.

    Well, there's always ring -2 (system management mode) which is a layer running beneath ring 0 for system management functionality, but if there's a vulnerability or a backdoored processor, you can do anything you want from it.

    Some material, if you're interested:

    https://en.wikipedia.org/wiki/System_Management_Mode
    https://www.osfc.io/2022/talks/i-have-come-to-bury-the-bios-not-to-open-it-the-need-for-holistic-systems/
    (if anything, you should watch the first 5 minutes for the cool demo of the exploit).

  • are you sane? How do we know that?

  • rustelekomrustelekom Member, Patron Provider

    This was the reason why we don't use images to install VPS O/S. So we just run a script that installs O/C from its repositories. Of course, it takes longer, but the O/S installation is always fresh, and our hands do not touch it if the client is worried about it :)

    Thanked by 1crilla
  • ShakibShakib Member, Patron Provider

    Your provider can access your server and all of your data anytime they want to. There is no need for backdoor in the OS.

    There is no way to stop your provider from lurking in your closet.

    root password, SSH keys nothing matters. You have to make a choice and trust your provider with your data.

  • I always go with providers that I think are too lazy to be snooping around.

    Thanked by 3ralf chria838373 Tony40
  • If you can not trust template/iso from a provider, why do you trust to order in the first place?
    As @Shakib said, your provider can access your data because they have access to the hardware.
    If you really need their services but you can't trust them, find a provider that give a way to use your own iso (not their cached iso), and encrypt the disk during installation process.

  • FatGrizzlyFatGrizzly Member, Host Rep

    @Shakib said: root password, SSH keys nothing matters. You have to make a choice and trust your provider with your data.

    LUKS encryption wins

  • emgemg Veteran
    edited November 2022

    Great points from everyone. Thanks.

    I should have excluded what the provider may do after the installation.
    -> Everyone is correct that the VPS provider has absolute power and control over data in motion, data at rest, data in memory, and whatever the processor does. The provider can do anything they want, and detection can range from difficult to impossible. Even if you encrypt your data, the provider can get the keys from the running VPS.

    The truth is that I wanted to learn if anyone knows how to hash an ISO file when it is mounted as a disk image. In addition, I had hoped that someone had done work related "tripwire-like" hash checks for files in the templates, but had low expectations there.

  • @stevewatson301 said:

    @emg said: Is there a way to compute a checksum on a mounted .iso file, where the provider mounts it for you?

    Again, you can't.

    Why not? What's stopping him from reading the mounted cdrom device to .iso and compare? It should be the same as original source, there's no conversion from digital.

    Thanked by 1bulbasaur
  • jmgcaguiclajmgcaguicla Member
    edited November 2022

    @TimboJones said:

    @stevewatson301 said:

    @emg said: Is there a way to compute a checksum on a mounted .iso file, where the provider mounts it for you?

    Again, you can't.

    Why not? What's stopping him from reading the mounted cdrom device to .iso and compare? It should be the same as original source, there's no conversion from digital.

    You can literally sha256sum your copy of the ISO and the block device inside the VM

    $ sudo sha256sum  /dev/loop0
    60347cee8ede09b3db4d354f2b4b4f5f989761b1d0ccff06c7564561c1865578  /dev/loop0
    
    $ sha256sum nixos-minimal-22.05.4033.ebf65554b18-x86_64-linux.iso 
    60347cee8ede09b3db4d354f2b4b4f5f989761b1d0ccff06c7564561c1865578  nixos-minimal-22.05.4033.ebf65554b18-x86_64-linux.iso
    

    Unless of course they also rigged sha256sum to spit out fake hashes

    image

    Thanked by 1bulbasaur
  • rm_rm_ IPv6 Advocate, Veteran
    edited November 2022

    In Debian, thanks to the debsums utility, you can verify checksums of all the installed binary files, or run debsums -a to also check for any modified configs. Fooling that by replacing checksums requires at least some dedication and may not be easily possible. But this will not do anything to spot any extra files added to the system outside of the package manager.

  • denellumdenellum Member, Host Rep

    Rule number one for IT : Trust but verify

  • @jmgcaguicla said:
    Unless of course they also rigged sha256sum to spit out fake hashes

    That's possible. It'd be trivial by adding a block-device filter on the host (e.g. VM's disk image served from a fusefs mount) and "less trivial" by hacking the drive firmware to return one set of data when it detects a data access pattern that looks like a boot, and another set of data afterwards when someone is accessing the disc normally.

  • @ralf said:
    That's possible. It'd be trivial by adding a block-device filter on the host (e.g. VM's disk image served from a fusefs mount) and "less trivial" by hacking the drive firmware to return one set of data when it detects a data access pattern that looks like a boot, and another set of data afterwards when someone is accessing the disc normally.

    Sure, you're at the mercy of the node hosting your VM so everything could be hosed anyway.

    Pick a struggle threat model.

  • @TimboJones said: What's stopping him from reading the mounted cdrom device to .iso and compare?

    @jmgcaguicla said: You can literally sha256sum your copy of the ISO and the block device inside the VM

    Yes, this is correct. A compromised sha256sum utility was not in the scope of my message, and yet I assumed that while writing the second half.

Sign In or Register to comment.