Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Do you use full disk encryption (e.g., LUKS) on your server?
New on LowEndTalk? Please Register and read our Community Rules.

Do you use full disk encryption (e.g., LUKS) on your server?

mathenymatheny Member
edited December 2022 in General

As title, do you normally use full disk encryption (FDE) on your VPS?

It would be great to protect your data at rest (e.g., in provider's backup), or you can theoretically "erase" all your disk by just shredding the LUKS header. (ofc not 100% safe because malicious provider can still peep into your kernel memory from the hypervisor)

But some weaker CPU get overloaded just from I/O encryption/decryption. Also, the problem of bootstrapping is hard (how to automatically unlock the host at boot? AFAIK, there's no way to do that securely without some kind of physcial secure device).

I recently started to setup FDE on my servers and the results seem to be okay. My use case doesn't need heavy I/O so performance drops isn't noticable too. But I just used it for a short amount of time and don't have much experience with it.

Do you use FDE on your own servers? What's your setup and how do you unlock the disk at boot time? Also, any war stories about losing all the data because of header corruption or losing password? I'd love to hear your experiences with FDE :D

Poll
  1. Do you use FDE?62 votes
    1. Yeah
      25.81%
    2. Nah, too slow
        8.06%
    3. Nah, don't care
      51.61%
    4. 👁👁
      14.52%

Comments

  • let_rockslet_rocks Member
    edited December 2022

    To remotely unlock boot disk over SSH you can use dropbear SSH server.

    On VPS I do not see so much benefit in FDE.

    Thanked by 1matheny
  • What is missing from the poll is "Sometimes yeah, sometimes nah".

    That would fit my use. Sometimes yes, sometimes no.

    Take a look at the full disk encryption discussion that weaves in and out of this thread:
    https://lowendtalk.com/discussion/182987/you-got-a-new-server-now-what/p1

    I just set up an encrypted Debian and tomorrow will try installing and testing ssh-based authentication to unlock the server. I learned about it in that thread above.

    Thanked by 2matheny cheapdave
  • Depends on the service I'm running on that KVM, if it's something that would benefit from FDE, then sure, otherwise nope.

    Thanked by 1matheny
  • Is full disk encryption even possible with OpenVZ VPSs?

  • @emg said:
    Is full disk encryption even possible with OpenVZ VPSs?

    Maybe possible with fuse?

  • no one going to sneak into the DC and unplug your hard drive

  • @swagger said:
    no one going to sneak into the DC and unplug your hard drive

    Says the guy who is doing exactly that. ;-)

  • skorousskorous Member
    edited December 2022

    @matheny said: ... how do you unlock the disk at boot time? Also, any war stories about losing all the data because of header corruption or losing password?

    Network Bound Disk Encryption. Two servers in two different regions for redundancy running tang. I haven't had any problem with header corruption yet ( knock on wood ).

  • @skorous said: Network Bound Disk Encryption.

    this is interesting, are you running those traffic through some kind of secure network tunnel? or just normal HTTPS request should be okay

  • @matheny said:

    @skorous said: Network Bound Disk Encryption.

    this is interesting, are you running those traffic through some kind of secure network tunnel? or just normal HTTPS request should be okay

    Would've preferred a tunnel to handle it but as this happens before boot I wasn't able to set one up. It's just regular HTTP because everything is already encrypted during provisioning. I also rotate the keys every so often.

  • mathenymatheny Member
    edited December 2022

    @skorous said:

    @matheny said:

    @skorous said: Network Bound Disk Encryption.

    this is interesting, are you running those traffic through some kind of secure network tunnel? or just normal HTTPS request should be okay

    Would've preferred a tunnel to handle it but as this happens before boot I wasn't able to set one up. It's just regular HTTP because everything is already encrypted during provisioning. I also rotate the keys every so often.

    thanks, this looks very promising! Hopefully I can get to setup ansible or puppet to automate the key rotating process.

  • @matheny said:

    @skorous said:

    @matheny said:

    @skorous said: Network Bound Disk Encryption.

    this is interesting, are you running those traffic through some kind of secure network tunnel? or just normal HTTPS request should be okay

    Would've preferred a tunnel to handle it but as this happens before boot I wasn't able to set one up. It's just regular HTTP because everything is already encrypted during provisioning. I also rotate the keys every so often.

    thanks, this looks very promising! Hopefully I can get to setup ansible or puppet to automate the key rotating process.

    I haven't done it yet but it's just a half dozen shell commands so should be doable fairly easily.

    Thanked by 1matheny
  • @emg said:
    [...]
    Take a look at the full disk encryption discussion that weaves in and out of this thread:
    https://lowendtalk.com/discussion/182987/you-got-a-new-server-now-what/p1

    I just set up an encrypted Debian and tomorrow will try installing and testing ssh-based authentication to unlock the server. I learned about it in that thread above.

    I installed a Dropbear SSH server on the encrypted Debian. It was relatively easy and there were only minor issues, easily found and corrected. See this post for a brief write-up of what I learned. :

    https://lowendtalk.com/discussion/comment/3561296#Comment_3561296

  • On a VPS, to automatically enter the OS after reboot (without entering password to decrypt), it seems that the key has to be stored in the unencrypted /boot partition, making the encryption nonsense

  • @Chocoweb said:
    On a VPS, to automatically enter the OS after reboot (without entering password to decrypt), it seems that the key has to be stored in the unencrypted /boot partition, making the encryption nonsense

    It is not the encryption that is nonsense. It is the combination of automatic boot coupled with an encrypted boot partition.

    Your statement is the equivalent of:
    "To allow reentry through the locked door (without a key to unlock the door), it seems that the key must be hidden under the doormat, making the door lock nonsense."

    The thread discussion here is about full disk encryption. If you want automatic boot with encrypted data, you could leave the boot and operating system unencrypted, but encrypt the sensitive data separately. There are potential issues with this approach, but it may be the best solution for your needs.

  • @matheny said:
    What's your setup and how do you unlock the disk at boot time?

    I have a somewhat janky system, first strung together years ago, that I really should reassess, mostly for my backups. It isn't quite full system encryption as /boot and / are not encrypted. /tmp and swap get recreated with new keys on startup and main storage isn't mounted until I SSH in and mount that volume manually. I get alerts when the machines reboot, so I know that I need to go in and do that.

    I considered having them call out to another machine to get the keys, or have the other machine notice them having reset and connect that way round (this is slightly more secure, but both methods have vulnerabilities whereby someone else to obtain the keys from the automated response) to mount the secure volume, that way there is no delay waiting for me to do something manual but the keys are still not stored on the same box as the encrypted data. I've rarely been in a position when the reboot isn't due to me doing admin, so I'm present to mount the volumes needing keys anyway, so I never got around to doing that.

    Apparently there has been built-in support for remote keys for quite some time, search for the terms other have mentioned to find details of that.

    @matheny said: any war stories about losing all the data because of header corruption or losing password?

    I've not lost data due to corruption or key loss like that.

    It was something I worried about when I set it up first time around, but I'm fairly blasé about it now. Just have multiple independent copies of the data (good backup strategy anyway) and make sure you have the keys stored safely in multiple locations.

    Obviously greater care is needed if the data you are storing is particularly sensitive.

    Thanked by 1matheny
  • @emg said: I installed a Dropbear SSH server on the encrypted Debian. It was relatively easy and there were only minor issues, easily found and corrected.

    Which stops you from needing a console connection but it still doesn't automagically decrypt the disks on reboot, right?

    @MeAtExampleDotCom said: I considered having them call out to another machine to get the keys, or have the other machine notice them having reset and connect that way round (this is slightly more secure, but both methods have vulnerabilities whereby someone else to obtain the keys from the automated response)

    Obviously I'm not an expert but I didn't think that was a concern due to how it's encrypted and the way it's exchanged.

    Thanked by 1matheny
  • a_usernamea_username Member
    edited December 2022

    @matheny said: Also, any war stories about losing all the data because of header corruption or losing password?

    I am encrypting important servers (and my PC) with LUKS for years and this is the first time i hear about header corruption. Did a little googling and yeah, that's interesting - so thank you for bringing up this topic. Will immediately perform a backup of the luks header :smiley:

    But wouldn't a header corruption only occur if the hard drive is damaged / slowly dying (bad sectors, etc.)? Probably a RAID does help to prevent header corruption?

  • @skorous said: Network Bound Disk Encryption. Two servers in two different regions for redundancy running tang.

    For those willing to know a bit more about tang and clevis this DebConf video is pretty good:

    Thanked by 2matheny skorous
  • @a_username said: But wouldn't a header corruption only occur if the hard drive is damaged / slowly dying (bad sectors, etc.)? Probably a RAID does help to prevent header corruption?

    Yeah I think it would be rare, also LUKS 2 stores the header twice just in case. But this could also be a plus, if you want to destroy the data, you only need to overwrite the header instead of waiting for hours to overwrite the whole drive.

    Thanked by 1a_username
  • @skorous said:

    Which stops you from needing a console connection but it still doesn't automagically decrypt the disks on reboot, right?
    [...]

    Good question. You are correct. It does not automatically decrypt the disks on reboot. You must be available to enter the password on reboot, too.

    All that the Dropbear SSH server does is let you bypass the console and enter the unlock password/passphrase over a secure SSH connection.

    Thinking outside the box, off the top of my head:

    My first thoughts about surviving a reboot would be to run a "reboot" program on the system that would create a bootable loader in a RAM disk. The bootable loader would include a copy of the decryption key or passphrase. On "reboot", create the RAM disk, boot from the RAM disk, and automatically execute software on the RAM disk that uses the encryption key to unlock and boot the normal operating system. That scheme would work on a reboot, but not a cold boot. The decryption key is already in RAM on the running server, so I wonder what increased risk there would be. Honestly, I would want more time to do a proper security analysis. I am just sharing the idea. I doubt that I am the first to think of it.

  • mathenymatheny Member
    edited December 2022

    @emg said: The decryption key is already in RAM on the running server, so I wonder what increased risk there would be.

    Does that RAM content survive the reboot? The bootloader (BIOS/UEFI, or grub) may somehow clear/overwrite to RAM content?

    EDIT: it seems possible on qemu: https://security.stackexchange.com/a/181213, but we need to write kernel driver to read the data out, because Linux kernel zero out all the memory before handling to the userland.

  • jmgcaguiclajmgcaguicla Member
    edited December 2022

    The proper solution to prevent hosts yanking out keys from live/snapshot memory is to encrypt the guest memory, the general idea is to have somewhere/something (a secure enclave) that only the CPU can reach to store confidential data.

    Some of the big boys already have these, for example Google calls these Confidential VMs (iirc Azure calls them that as well) and are utilizing AMD's SEV feature.

    Thanked by 4emg matheny pbx that_guy
  • Nope. Always thought about it, but never had the time/motivation/need to properly learn all about it (I hate doing things when I don't know exactly how it works, and how I could troubleshoot problems).
    BTW: Has anyone ever used the TRESOR kernel patch to store encryption keys in CPU registers or CPU cache instead of RAM? Any comments, like drawbacks, problems, etc.?
    https://en.wikipedia.org/wiki/TRESOR

    Thanked by 1matheny
  • @MeAtExampleDotCom said: I considered having them call out to another machine to get the keys, or have the other machine notice them having reset and connect that way round (this is slightly more secure, but both methods have vulnerabilities whereby someone else to obtain the keys from the automated response)

    Obviously I'm not an expert but I didn't think that was a concern due to how it's encrypted and the way it's exchanged.

    The question is: how does the server handing out the keys know to trust the server requesting them? A signed request? The signing keys will need to be on the remote server so what we have there is storing-the-keys-on-the-same-machine with extra steps.

    For some (many?) threat models this (an adversary with access to the machine in its normal location (or an adversary able to fake being at that network address)) may not be a significant concern to the point where it is more important than the machine being down until manual intervention happens.

    I'm no expert either, though do have some professional security experience, so maybe there is something obvious I'm missing, happy to be reeducated if that is the case!

  • mathenymatheny Member
    edited December 2022

    @MeAtExampleDotCom said: The signing keys will need to be on the remote server so what we have there is storing-the-keys-on-the-same-machine with extra steps.

    I think it's still valuable in some cases, e.g., you can disable remote unlock when a physical breach was detected at a remote datacenter location.

    We can even go furthur and store the client identity on hardware (e.g., TPM) so that it can't be cloned at all.

  • From a personal perspective on a VPS, I wouldn't. I don't have any data that would require it.

    In a datacenter... Obviously, it depends on the usage, but as a matter of practice on production servers in a data center, nope, we never did nor was it a consideration. Availability is always the key and anything which could compromise that would have to have a good business reason. Your protection lies in front of the data or the sensitive data should be hashed.

  • mathenymatheny Member
    edited December 2022

    @Toonces said:
    From a personal perspective on a VPS, I wouldn't. I don't have any data that would require it.

    In a datacenter... Obviously, it depends on the usage, but as a matter of practice on production servers in a data center, nope, we never did nor was it a consideration. Availability is always the key and anything which could compromise that would have to have a good business reason. Your protection lies in front of the data or the sensitive data should be hashed.

    Yeah I agree with you, it is not really needed in a secured datacenter location. It might make more sense if you are colo-ing at a location not fully in your control. CloudFlare talked a little bit about how they use FDE on their CDN nodes https://blog.cloudflare.com/speeding-up-linux-disk-encryption/

  • jmgcaguiclajmgcaguicla Member
    edited December 2022

    @that_guy said:
    BTW: Has anyone ever used the TRESOR kernel patch to store encryption keys in CPU registers or CPU cache instead of RAM? Any comments, like drawbacks, problems, etc.?

    Exfiltration will be a wee bit harder, but ultimately the host can still see the keys.

    The method I described a few posts above takes this idea further by moving the trust anchor up to the CPU itself instead of the host by not allowing anyone (even the host) access to the keys.

    Don't get me wrong, this isn't a cure-all. There are still a myriad of ways the host can screw your VM over as it controls other parts of the virtualization process (e.g. disks).

    tl;dr: FDE on a shared environment (with a host you don't control) shouldn't be expected to protect oneself from someone/thing actively out to get you, it's just a way to deter casual garden peeking.

    Thanked by 2that_guy Peppery9
Sign In or Register to comment.