Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


You got a new server... now what? - Page 2
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

You got a new server... now what?

2»

Comments

  • @Liso said: Quick question, how do you install from ISO if the provider does not officially support it ?

    If there is boot in recovery you can install it from there.. but you need to have ram bigger than actual iso size (or 2x). I've managed to install Slackware 15.0 on KS-1 that way which is harder for that specific OS than lets say debian or almalinux.

  • pbxpbx Member
    edited December 2022

    @emg said: remote unlocking via SSH

    It seems to me that it's the best way: you don't have to trust the code of the "html console" you'd otherwise have to use. Trusting what's in the repos of the distro is in most cases a better choice. That being said FDE is often overkill for a VPS: if you encrypt the folder where your private data is stored, this can be enough in many cases.

    @emg said: Another threat is the provider itself.

    Another one is hardware thrown away. Provider deadpooling, hard drives being sold to a third party, hardware upgrade and everything ends up on the second hand market... all it takes is a clever guy tempted to check what these machines were used for for your 'private' data to become quite public.

    But again it depends what data is stored on the VPS. For most uses, encryption at the storage level is not needed, and might even worsen the system's security: with no FDE you can have the server reboot by itself when there is a kernel upgrade... (yeah you can use live patching but the best solutions aren't that cheap!).

    Thanked by 1emg
  • @sandoz said: Sooo you are also paranoid but you use a company that creates honeypot?

    Maybe he audited the code before launching the playbook!

  • emgemg Veteran

    All helpful and useful comments above. Thanks for the suggestions.

    I have a few new tricks to try: (a) Installing directly from the internet rather than a mounted .iso, (b) SSH connection to authenticate unlocking the full disk encryption, (c) ?...

  • Daniel15Daniel15 Veteran
    edited December 2022

    @darkimmortal said:

    @emg said:
    What I would like to know is:

    • How do you decrypt / unlock it? Are you using the provider's HTML VNC console (in the control panel) from your browser? Something else?

    Debian has a package that lets you unlock it over SSH dropbear-initramfs

    This is what I do too. There's a decent guide here: https://www.cyberciti.biz/security/how-to-unlock-luks-using-dropbear-ssh-keys-remotely-in-linux/

    It runs an SSH server very early in the boot process, before mounting the root file system (which is why it has to be embedded in the initramfs). Essentially all the SSH server does is let you enter the unlock password.

    Just make sure you configure it to run on a different port to your usual SSH server, otherwise you'll get errors about the server's identity changing (since it's a completely separate SSH server with different keys).

    I recently converted a dedicated server from one SSD to two SSDs in RAID1, adding encryption too, without having to completely reinstall. Essentially:

    1. Set up degraded RAID1 array (just one drive) with only the NEW SSD, using the entire SSD
    2. Set up encryption on top of RAID
    3. Set up LVM on top of encryption
    4. Boot from Clonezilla live CD and clone data from old drive to new drive
    5. Modify GRUB config on both old and new drive to mount new LVM as /boot and root filesystem
    6. Modify /etc/fstab to mount new LVM
    7. Install GRUB to new drive
    8. Boot into system using NEW drive
    9. Wipe old drive (delete old LVM, format)
    10. Add old drive to RAID array
  • rm_rm_ IPv6 Advocate, Veteran

    @pbx said: It seems to me that it's the best way: you don't have to trust the code of the "html console" you'd otherwise have to use.

    The unencrypted part can still be compromised: https://en.wikipedia.org/wiki/Evil_maid_attack
    E.g. the Dropbear instance could be backdoored to also send your entered password elsewhere. So do not store your Bitcoin wallet worth millions on this.

    Thanked by 1pbx
  • @rm_ said: The unencrypted part can still be compromised

    Sure; and if you chose FDE to limit this risk, /boot is still unencrypted... But well... key is in RAM anyway!

  • @plumberg said:
    Wow, thats so awesome. I would love to hear more about the scripts you have about notifications, if you dont mind.

    Absolutely!
    It isn't hugely complicated.

    For the root login notifications I added a line to .bashrc to run a curl call in the background to send me a Pushover notification. I'm going to expand it to send me an email similar to the below. Pretty simple stuff.

    I do it in.bashrc because I figure if someone is gaining root access to one of my servers, I have much larger problems anyway.

    For reboot notifications I have a simple systemd service that calls a bash script that sends me an email using curl and simple SMTP - kind of on topic here.

    @plumberg said:
    Also, not sounding like a moocher, would you mind sharing some snippets of your playbook?

    I don't mind at all - but I need to clean them up a little before I publish them.

    I've got one role that bootstraps, which I call from a simple bash script to pass the IP address, first time root password etc...
    The bootstrap role itself then does apt update+upgrade, installs sudo, creates the ansible management user + keys.
    I want to expand the simple bash script to add the bootstrapped hosts to my Ansible inventory - but not sure I can automate this part.

    I then run my primary playbook which depending on the "role" of the machine will install what is required (eg, DB, VPN, WEB, Docker, etc...) as well as what I call my "common" role which is effectively what I think all my instances should have installed - snippet below:

    ---
    - name: Apt install common packages
      apt:
        name:
          - curl
          - wget
          - tree
          - htop
          - ncdu
          - ca-certificates
          - zip
          - unzip
          - openssl
        state: present
        update_cache: true
    
    - name: Update Hostname
      hostname:
        name: "{{ hostn }}"
      when: (hostn is defined) and (hostn|length > 0)
    
    - name: Set Timezone
      timezone:
        name: "{{ tz }}"
      when: (tz is defined) and (tz|length > 0)
    
    - name: Create Power User
      user:
        name: "{{ user }}"
        password: "{{ pass }}"
        groups: sudo
        shell: /bin/bash
        update_password: always
    
    - name: Add Authorised Keys
      authorized_key: 
        user: "{{ user }}"
        key: "{{ ssh_key }}"
        state: present
    

    All my hosts are Debian or Ubuntu based.

    Notifications, firewall, fail2ban are all seperate roles, and so is the monitoring (Hetrix) which I can clean-up and publish on Github soon-ish.

    The dev-sec role is a great Ansible Galaxy role that anyone can audit on Github.

    @sandoz said:
    Sooo you are also paranoid but you use a company that creates honeypot?
    https://www.crowdsec.net/product/threat-intelligence

    Great way to be "paranoid", only god knows if your machine is also a "honeypot collecting data" or not.

    I don't really get what you're trying to say - but who said I was paranoid?
    Crowdsec behaves very simirlarly to fail2ban but with the added benefit of a crowdsourced threat list.

    Thanked by 2plumberg pbx
  • @tjn said:

    @plumberg said:
    Wow, thats so awesome. I would love to hear more about the scripts you have about notifications, if you dont mind.

    Absolutely!
    It isn't hugely complicated.

    For the root login notifications I added a line to .bashrc to run a curl call in the background to send me a Pushover notification. I'm going to expand it to send me an email similar to the below. Pretty simple stuff.

    I do it in.bashrc because I figure if someone is gaining root access to one of my servers, I have much larger problems anyway.

    For reboot notifications I have a simple systemd service that calls a bash script that sends me an email using curl and simple SMTP - kind of on topic here.

    @plumberg said:
    Also, not sounding like a moocher, would you mind sharing some snippets of your playbook?

    I don't mind at all - but I need to clean them up a little before I publish them.

    I've got one role that bootstraps, which I call from a simple bash script to pass the IP address, first time root password etc...
    The bootstrap role itself then does apt update+upgrade, installs sudo, creates the ansible management user + keys.
    I want to expand the simple bash script to add the bootstrapped hosts to my Ansible inventory - but not sure I can automate this part.

    I then run my primary playbook which depending on the "role" of the machine will install what is required (eg, DB, VPN, WEB, Docker, etc...) as well as what I call my "common" role which is effectively what I think all my instances should have installed - snippet below:

    ---
    - name: Apt install common packages
      apt:
        name:
          - curl
          - wget
          - tree
          - htop
          - ncdu
          - ca-certificates
          - zip
          - unzip
          - openssl
        state: present
        update_cache: true
    
    - name: Update Hostname
      hostname:
        name: "{{ hostn }}"
      when: (hostn is defined) and (hostn|length > 0)
    
    - name: Set Timezone
      timezone:
        name: "{{ tz }}"
      when: (tz is defined) and (tz|length > 0)
    
    - name: Create Power User
      user:
        name: "{{ user }}"
        password: "{{ pass }}"
        groups: sudo
        shell: /bin/bash
        update_password: always
    
    - name: Add Authorised Keys
      authorized_key: 
        user: "{{ user }}"
        key: "{{ ssh_key }}"
        state: present
    

    All my hosts are Debian or Ubuntu based.

    Notifications, firewall, fail2ban are all seperate roles, and so is the monitoring (Hetrix) which I can clean-up and publish on Github soon-ish.

    The dev-sec role is a great Ansible Galaxy role that anyone can audit on Github.

    @sandoz said:
    Sooo you are also paranoid but you use a company that creates honeypot?
    https://www.crowdsec.net/product/threat-intelligence

    Great way to be "paranoid", only god knows if your machine is also a "honeypot collecting data" or not.

    I don't really get what you're trying to say - but who said I was paranoid?
    Crowdsec behaves very simirlarly to fail2ban but with the added benefit of a crowdsourced threat list.

    This is smashing. I will take a look over the weekend and see how it all fits in

    I am a centos user. There is going to be lots of learning...

    BTW, what does bootstrap mean? Without first install how can you run that script? Maybe I am confused.

  • @tjn said:
    My flow is similar to yours but I've got an Ansible playbook that bootstraps my machines, creates relevant users, adds SSH keys, installs necessary (imo) programs, and hardens installations (https://github.com/dev-sec/ssh-baseline) + firewall + crowdsec or fail2ban, and adds monitoring.

    It's all well automated at this point.

    I also have a scripts to notify me if someone logs in as root, or if the server reboots.

    How are you going about full disk encryption?

    Do you need Ansible installed on the server before executing the task? I always use shell script to automate setup in new VPS, I heard Ansible is better suited for deployment task— so I'd like to give it a try.

  • @plumberg said:
    BTW, what does bootstrap mean? Without first install how can you run that script? Maybe I am confused.

    Textbook definition is:

    a technique of loading a program into a computer by means of a few initial instructions which enable the introduction of the rest of the program from an input device.

    It's basically running a simple script (or in this case a script that calls an Ansible playbook) that prepares your system in order for it to be managed or adopted into your environment.

    In this case, my bootstrap role just creates a management user that the rest of my playbook can then use to manage the host.

    @Liso said:
    Do you need Ansible installed on the server before executing the task? I always use shell script to automate setup in new VPS, I heard Ansible is better suited for deployment task— so I'd like to give it a try.

    No you need to have it installed on your own computer, don't need anything except SSH access on your server.

    Ansible is great for "configuration management of an immutable environment" - in that it will always reconfigure things to a particular baseline. Which is why ApisCP uses it.

    I used to use shell scripts as well, but Ansible is more flexible, especially when managing more than one host.

    If you want to try it, I posted some resources in an older comment - https://lowendtalk.com/discussion/comment/3348095#Comment_3348095

    Thanked by 1plumberg
  • @tjn said:

    @plumberg said:
    BTW, what does bootstrap mean? Without first install how can you run that script? Maybe I am confused.

    Textbook definition is:

    a technique of loading a program into a computer by means of a few initial instructions which enable the introduction of the rest of the program from an input device.

    It's basically running a simple script (or in this case a script that calls an Ansible playbook) that prepares your system in order for it to be managed or adopted into your environment.

    In this case, my bootstrap role just creates a management user that the rest of my playbook can then use to manage the host.

    @Liso said:
    Do you need Ansible installed on the server before executing the task? I always use shell script to automate setup in new VPS, I heard Ansible is better suited for deployment task— so I'd like to give it a try.

    No you need to have it installed on your own computer, don't need anything except SSH access on your server.

    Ansible is great for "configuration management of an immutable environment" - in that it will always reconfigure things to a particular baseline. Which is why ApisCP uses it.

    I used to use shell scripts as well, but Ansible is more flexible, especially when managing more than one host.

    If you want to try it, I posted some resources in an older comment - https://lowendtalk.com/discussion/comment/3348095#Comment_3348095

    Totally sold on the idea, thanks I'll try it asap. Shouldn't be to hard to convert my current script to playbook.

  • @yoursunny said:
    First, I enable the rescue system and look for secret push-up videos in the hard drive.
    After that, I reinstall with netboot.xyz, so that the system is clean and consistent.

    I keep setup commands of each server in text files.
    They are pasted manually into the console.
    When I acquire a new server, I duplicate the file of a similar server, and make modifications from there.

    I find a lot of consoles don't have working paste. It works most often for you?

  • yoursunnyyoursunny Member, IPv6 Advocate

    @TimboJones said:

    @yoursunny said:
    First, I enable the rescue system and look for secret push-up videos in the hard drive.
    After that, I reinstall with netboot.xyz, so that the system is clean and consistent.

    I keep setup commands of each server in text files.
    They are pasted manually into the console.
    When I acquire a new server, I duplicate the file of a similar server, and make modifications from there.

    I find a lot of consoles don't have working paste. It works most often for you?

    SSH has working paste.
    VNC does not.

    Since Debian 11 and Ubuntu 22, pasted commands appear highlighted and I must press ENTER key to execute.
    This can be reverted with command:

    echo 'set enable-bracketed-paste off' >> /etc/inputrc
    

    and then re-login.

    Thanked by 2TimboJones plumberg
  • @Daniel15 said:
    I recently converted a dedicated server from one SSD to two SSDs in RAID1, adding encryption too, without having to completely reinstall. Essentially:

    1. Set up degraded RAID1 array (just one drive) with only the NEW SSD, using the entire SSD
    2. Set up encryption on top of RAID
    3. Set up LVM on top of encryption
    4. Boot from Clonezilla live CD and clone data from old drive to new drive
    5. Modify GRUB config on both old and new drive to mount new LVM as /boot and root filesystem
    6. Modify /etc/fstab to mount new LVM
    7. Install GRUB to new drive
    8. Boot into system using NEW drive
    9. Wipe old drive (delete old LVM, format)
    10. Add old drive to RAID array

    This guy fucks.

    Thanked by 1Daniel15
  • @tjn said: I used to use shell scripts as well, but Ansible is more flexible, especially when managing more than one host.

    I've been meaning to switch to using Ansible for setting up servers. I set up new servers so infrequently these days though (I didn't get any new servers during Black Friday for example) that it hasn't been a pressing issue.

    I do use Ansible for the DNSTools probes, but the role is super basic: https://github.com/Daniel15/dnstools/blob/master/ansible/roles/dnstools-worker/tasks/main.yml

    @Liso said: Do you need Ansible installed on the server before executing the task?

    You literally just need Python and SSH on the server.

    Ansible is great because you basically describe the intended final state of the server, and Ansible does the work to get there, avoiding things that have already been done. For example, if you tell it to install a package but that package is already installed, it won't do anything and will just skip that step. Same thing if you ask it to add a line to a file (maybe something in sysctl.conf? maybe disabling PasswordAuthentication in sshd_config?) and the line is already there, it won't add it again.

    This makes it a lot easier to make it idempotent (meaning you can run it multiple times without messing anything up) compared to a shell script. If you add something new, you can run the playbook across all your servers to bring them all up-to-date :smile: Shell scripts tend to assume that the server is empty and the script has never been ran.

    Thanked by 1dev077
  • I have a text file where I keep all my commands which I run one by one. My usual steps are:

    1. Change SSH port
    2. Add new user and disable root login
    3. Enable SSH keys
    4. Install fail2ban
    5. Install ufw and block all incoming and outgoing except SSH, 80, 443, 53, 565 and 25 ports. May be one or two more ports which I am forgetting now.
    6. Install nginx
    7. Install PHP FPM 8.1
    8. Install mariadb
    9. Move website if required

    I always install from provider template and never use ISO but now I am thinking about full disk encryption after reading OP.

    Thanked by 2plumberg yoursunny
  • emgemg Veteran

    @Daniel15 said: This is what I do too. There's a decent guide here: https://www.cyberciti.biz/security/how-to-unlock-luks-using-dropbear-ssh-keys-remotely-in-linux/

    It runs an SSH server very early in the boot process, before mounting the root file system (which is why it has to be embedded in the initramfs). Essentially all the SSH server does is let you enter the unlock password.

    Just make sure you configure it to run on a different port to your usual SSH server, otherwise you'll get errors about the server's identity changing (since it's a completely separate SSH server with different keys).

    Thanks to @Daniel15 for this great hint. I followed the procedure in the link that Daniel provided, this one:
    https://www.cyberciti.biz/security/how-to-unlock-luks-using-dropbear-ssh-keys-remotely-in-linux/

    I setup a testbed with an encrypted Debian instance, installed the Dropbear SSH server and followed the configuration steps. For me, the setup was straightforward, but I have experience with SSH public key authentication, something that may be unfamiliar to beginners.

    Hint for beginners:
    If you have not used public key authentication with SSH before, it might be helpful to try it out first using an ordinary SSH server, rather than trying to troubleshoot it with a newly installed Dropbear SSH server at initial startup/decrypt time.

    For the first Dropbear setup, there was only one problem to troubleshoot, and it was easy to find and fix. In this case, the problem was that my chosen port appeared to be closed, but the server was still responding to pings (before the unlock). I ran an nmap scan and determined that port 22 was open, but my chosen port was not. Since the port choice was configured by setting the DROPBEAR_OPTIONS in /etc/dropbear-initramfs/config, it was easy to find the issue. Here were my mistakes, easily corrected:

    • Mistake 1: Did not see the quotation marks around the "DROPBEAR_OPTIONS=..." setting. The quotation marks are essential. I added them to the config file and rebooted the server, but that did not solve the issue. Why? See Mistake 2, below.
    • Mistake 2: Forgot to run "update-initramfs -u" or "update-initramfs -u -v" after correcting the DROPBEAR_OPTIONS setting. If you change anything about the Dropbear configuration, you must run update-initramfs before you reboot. It copies the configuration settings from the encrypted partition to the startup SSH server, the one you use to unlock the encryption. If you forget to run this, then your startup SSH server will be using old (or original default) settings.

    I hope this helps others.

    Thanked by 2Daniel15 abtdw
  • emgemg Veteran
    edited December 2022

    Thanks to @Daniel15 for posting the links and writing the tutorials on the site. I got the grub loader working as described here:
    https://lowendtalk.com/discussion/comment/3556642#Comment_3556642

    I followed the tutorial in this link:
    https://netboot.xyz/docs/booting/grub/

    I am now able to boot installers on my VPS over the internet from the netboot.xyz site. The menu of installers is extensive and easy to follow. I recommend you give it a try and take a look, even if you don't install anything today, it may be useful to you in the future.

    A Debian 11 (bullseye) installer is there for me and it works. Here are my new questions:

    -> How do I trust the netboot.xyz installers?

    Is there a way to run checksums to compare against known installers from the "official" websites?
    Is there a digital signature to check?
    Are we limited to "trust us"?

    Thanked by 1abtdw
  • @emperor said:

    @plumberg said: What's this serverstatus client?

    Its BoToX script for monitoring : https://github.com/BotoX/ServerStatus
    Sadly script its in python2 so wont work on alma9 and other os where python2 is not in repos. It will work in bash but in bash shows wrong values.

    A more modern version with more features is forked here: https://github.com/cppla/ServerStatus/

    It's in chinese however, but tried running it on a docker container with success, but would need to go through the code and translate it to english. Or does anyone here know of a fork with the update cppla-features but translated to english?

  • emgemg Veteran
    edited December 2022

    @emg said:
    Thanks to @Daniel15 for posting the links and writing the tutorials on the site. I got the grub loader working as described here:
    https://lowendtalk.com/discussion/comment/3556642#Comment_3556642

    I followed the tutorial in this link:
    https://netboot.xyz/docs/booting/grub/

    I am now able to boot installers on my VPS over the internet from the netboot.xyz site. The menu of installers is extensive and easy to follow. I recommend you give it a try and take a look, even if you don't install anything today, it may be useful to you in the future.

    A Debian 11 (bullseye) installer is there for me and it works. Here are my new questions:

    -> How do I trust the netboot.xyz installers?

    Is there a way to run checksums to compare against known installers from the "official" websites?
    Is there a digital signature to check?
    Are we limited to "trust us"?

    Since posting the above, I found some answers. The answer is: RTFM.

    There is a blog entry in the Docs on the netboot.xyz website. It describes how to self-host your own version of netboot.xyz on your own server. You can configure the menus and the boot files for yourself.

    (I admit, I am still struggling with the configuration for self-hosting, but it is one step closer.)

  • Daniel15Daniel15 Veteran
    edited December 2022

    @emg said: How do I trust the netboot.xyz installers?

    The scripts are all plain text files so you can just read them and see what they do.

    For Debian, it loads stuff directly from the official Debian servers:
    https://github.com/netbootxyz/netboot.xyz/blob/development/roles/netbootxyz/templates/menu/debian.ipxe.j2
    https://github.com/netbootxyz/netboot.xyz/blob/9302484e7f522a210edcb091d6faead617850448/roles/netbootxyz/defaults/main.yml#L192

    If you go to http://boot.netboot.xyz/debian.ipxe (which is the version used on the production web site), you can verify that it's using the same archive.debian.org server :)

    You can pretty easily make your own iPXE scripts if you want to. You can copy and paste their Debian script, put it on your own server, and it'd work. The netboot.xyz menu is just a script that downloads and runs other scripts depending on which option you pick.

    Thanked by 2emg ehab
  • I recently setup KS-LE-1 with zfs native encryption and mirror setup. The intird has sshd and .profile to prompt for key and import pool.
    Since the initrd and kernel is unauthenticated(and unencrypted but that's not a concern), theoretically it can be backdoored and i probably won't notice it. So there has to be some trust that the provider won't do such thing :).

    If you pay a lot maybe it's possible to get a full hardware trust chain going. But not feasible in low-end world :).

Sign In or Register to comment.