Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Making KVM reinstalls less painful? - Page 2
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Making KVM reinstalls less painful?

2»

Comments

  • @geekalot said: [various bits and pieces - snipped for readability]

    How do you boot the VPS (or dedis) though - I assume it'll wait at some stage for a password to decrypt the FS (also, are you using LUKS or something else - details please!) - so how do you automate a reboot to get back to running stage? Key file (without password) implies no real security.

    • Yes, using LUKS for encryption.
    • Passphrase provided manually over SSH during the boot process (using dropbear/busybox) ... see example link below.
    • No automation. Manual boot. Monitoring servers notify if system goes down, manually provide passphrase over SSH.
    • Only time I have configured "automatic"/keyfile was for low risk data ("linux ISO's") using a random 4K+ sequence from a USB drive ... really just for a proof of concept.

    Understood.

    Won't work for me (I like to have my VPSs reboot and come up without any intervention). Of course I don't have any dedis or security needs (stringent) at this point. Many times, the reboots are not even in my control. OpenVZ is generally far worse than KVM in this regard (and forget about LUKS with OpenVZ). If there's fuse support encfs can work (helps with the VPS-at-rest scenario) and that should satisfy most scenarios that are "solvable".

    The keyfile concept works as a deterrent but will not stop a determined snooper (esp. if they have control over the box).

    One other option to consider instead of interactive SSH/password entry is to have a script fetch the password via HTTP/S. The advantage (as compared to keyfile) is that you still retain some notion of control over the key material. So how this works is the script connects to a URL (add password auth if you'd like) which "returns" the key. The server side is remote (and managed by you) and has IP specific rules to return whatever. The nice aspect of this is you control the (https) server and can prevent key material being returned at will. This allows for auto starts as long as the IP is the same (add other factors you want via the boot script including mac address, UUID etc. that may help you detect change of location etc. - the IP itself should be a dead giveaway though). If you suspect compromise, don't make the boot automatic but fall back to interactive control which allows you to potentially check stuff before typing password through OOB stuff (since you have dedis). Once can do a few extra things this way but nothing really buys you any serious security - the owner of the machine really can get away with anything.

    Similarly even in your current setup where the machine stops requiring you to ssh in and type the password, it is at best a (good) deterrent. In theory, assuming I have access to the box, can replace your initrd scripts (which is of course unencrypted) with something that saves the password and then decrypts the volume. Once I have the password, you're toast even if you nuke the disk at that point (I could have made an image of the disk and I can use the password against the image without your even knowing or being able to do anything about it).

    • Wow, now you are really going deep lol. There is always a way around every security mechanism. I am simply trying to protect against the "average" opportunist attacker and when shipping servers, so I am willing to make some compromise on edge cases (based on what is important to me).

    In this case see the remote HTTP/S key distributor (see above) should work nicely for you - you lock it down when you have an "incident" but otherwise it at least allows for automated reboots without you having to jump through hoops.

    Unfortunately under the circumstances I really can't think of anything better to give you fool proof security unless you can sign the boot loader etc. and prevent it from being tampered with (at which point it self destructs type of thing).

    So in short, IMHO you're doing the best and I sadly have nothing concretely better to offer.

    Can't think of better options...

    Anyone has ideas, please share.

  • @geekalot problem with encrypting a VPS is all someone has to do is get access to the management platform and initiate a snapshot with the RAM contents included and BAM they've got the encryption key sitting there in the resulting dump.

    Not really sure how you'd counter that one.

  • @dragon2611 said:
    @geekalot problem with encrypting a VPS is all someone has to do is get access to the management platform and initiate a snapshot with the RAM contents included and BAM they've got the encryption key sitting there in the resulting dump.

    Not really sure how you'd counter that one.

    Understood, not a "bulletproof" situation by any means on a VPS. A bit more durable (IMHO) on host nodes (for example, when shipping pre-configured server).

  • @nullnothere said:

    @geekalot said: [various bits and pieces - snipped for readability]

    How do you boot the VPS (or dedis) though - I assume it'll wait at some stage for a password to decrypt the FS (also, are you using LUKS or something else - details please!) - so how do you automate a reboot to get back to running stage? Key file (without password) implies no real security.

    • Yes, using LUKS for encryption.
    • Passphrase provided manually over SSH during the boot process (using dropbear/busybox) ... see example link below.
    • No automation. Manual boot. Monitoring servers notify if system goes down, manually provide passphrase over SSH.
    • Only time I have configured "automatic"/keyfile was for low risk data ("linux ISO's") using a random 4K+ sequence from a USB drive ... really just for a proof of concept.

    Understood.

    Won't work for me (I like to have my VPSs reboot and come up without any intervention). Of course I don't have any dedis or security needs (stringent) at this point. Many times, the reboots are not even in my control. OpenVZ is generally far worse than KVM in this regard (and forget about LUKS with OpenVZ). If there's fuse support encfs can work (helps with the VPS-at-rest scenario) and that should satisfy most scenarios that are "solvable".

    The keyfile concept works as a deterrent but will not stop a determined snooper (esp. if they have control over the box).

    One other option to consider instead of interactive SSH/password entry is to have a script fetch the password via HTTP/S. The advantage (as compared to keyfile) is that you still retain some notion of control over the key material. So how this works is the script connects to a URL (add password auth if you'd like) which "returns" the key. The server side is remote (and managed by you) and has IP specific rules to return whatever. The nice aspect of this is you control the (https) server and can prevent key material being returned at will. This allows for auto starts as long as the IP is the same (add other factors you want via the boot script including mac address, UUID etc. that may help you detect change of location etc. - the IP itself should be a dead giveaway though). If you suspect compromise, don't make the boot automatic but fall back to interactive control which allows you to potentially check stuff before typing password through OOB stuff (since you have dedis). Once can do a few extra things this way but nothing really buys you any serious security - the owner of the machine really can get away with anything.

    Similarly even in your current setup where the machine stops requiring you to ssh in and type the password, it is at best a (good) deterrent. In theory, assuming I have access to the box, can replace your initrd scripts (which is of course unencrypted) with something that saves the password and then decrypts the volume. Once I have the password, you're toast even if you nuke the disk at that point (I could have made an image of the disk and I can use the password against the image without your even knowing or being able to do anything about it).

    • Wow, now you are really going deep lol. There is always a way around every security mechanism. I am simply trying to protect against the "average" opportunist attacker and when shipping servers, so I am willing to make some compromise on edge cases (based on what is important to me).

    In this case see the remote HTTP/S key distributor (see above) should work nicely for you - you lock it down when you have an "incident" but otherwise it at least allows for automated reboots without you having to jump through hoops.

    Unfortunately under the circumstances I really can't think of anything better to give you fool proof security unless you can sign the boot loader etc. and prevent it from being tampered with (at which point it self destructs type of thing).

    So in short, IMHO you're doing the best and I sadly have nothing concretely better to offer.

    Can't think of better options...

    Anyone has ideas, please share.

    @nullnothere, I like the way you think. Fetching the key via some remote means has been on my mind as well. And, of course, that process has to be super-protected from snooping also. I've been wanting to do a proof of concept with that as well ... just need time & opportunity.

    As @dragon2611 mentioned, I know there are definitely ways to get past all of this (especially on a VPS), but in IMHO it is better than doing nothing.

    Information Security is a big set of compromises -- how much you have/deploy depends on budget, skill, time, resources .... and the types of threats that you face.

    Cheers

  • letboxletbox Member, Patron Provider

    We do our own templates with Promox ! They are saving a lot of time.

  • I'm a little late to the party here but I just wanted to expand on the discussion from @geekalot and @raindog308 regarding iPXE.

    I've had great experiences with iPXE while trying to install custom templates/ISOs on KVM virtual servers. My current approach is install whatever linux template the provider gives me (even if it's outdated). Then I log into the VPS, download the iPXE ISO, and add it to GRUB so it boots automatically on the next restart.

    Once iPXE loads, it's set to chainload the script from http://justingoins.com/ipxe/debian.config
    That script starts the Debian installer and provides a preseed file (also online) that is configured to provide static IP information and automatically launches an SSH server to complete the installation. The preseed file even allows you to specify a rsyslog host so that you can log the boot process and remotely monitor progress.

    This technique isn't just limited to VPS applications. I've successfully used it to install a custom Debian distro on a Hetzner server where I had no remote KVM access.

    Thanked by 1geekalot
  • geekalotgeekalot Member
    edited November 2016

    @goinsj2010 said:
    I'm a little late to the party here but I just wanted to expand on the discussion from @geekalot and @raindog308 regarding iPXE.

    I've had great experiences with iPXE while trying to install custom templates/ISOs on KVM virtual servers. My current approach is install whatever linux template the provider gives me (even if it's outdated). Then I log into the VPS, download the iPXE ISO, and add it to GRUB so it boots automatically on the next restart.

    Once iPXE loads, it's set to chainload the script from http://justingoins.com/ipxe/debian.config
    That script starts the Debian installer and provides a preseed file (also online) that is configured to provide static IP information and automatically launches an SSH server to complete the installation. The preseed file even allows you to specify a rsyslog host so that you can log the boot process and remotely monitor progress.

    This technique isn't just limited to VPS applications. I've successfully used it to install a custom Debian distro on a Hetzner server where I had no remote KVM access.

    @goinsj2010, this is nice to know and confirms the proof of concept. Thanks for chiming in on this!

  • I use to use this:

    http://fai-project.org

    Which will auto install the OS with a default root password. It's very quick. But I don't use it now since I use openvz typically.

Sign In or Register to comment.