Securing servers

vitobotta

What do you usually do to secure your servers?

Besides

• using a non-standard port for SSH
• using a non-root user to SSH into the server
• disabling root login
• disabling password authentication
• requiring 2FA OTP code
• having a passphrase on the SSH key
• having fail2ban configured with no tolerance
• using https://github.com/chaifeng/ufw-docker to fix the Docker issue with ufw

What else would you do? Do you use something like Crowdec or similar? What else?

TrK

You can do the hardware key login with yubikey, i never tried that though.

cpsd

Those things + knock on port to allow my current ip

jugganuts

@TrK said:
You can do the hardware key login with yubikey, i never tried that though.

works great with my yubikey, but I only have one production machine I deam worth while to do it with.

TrK

@jugganuts said:

@TrK said:
You can do the hardware key login with yubikey, i never tried that though.

works great with my yubikey, but I only have one production machine I deam worth while to do it with.

Thanks for the update will try it soon on a test machine.

Merakith

• Lock SSH over Tailscale/Wireguard Instance. No one should be able to SSH without being on your network. You can then remove fail2ban as bots won't be able to hammer your SSH port and no need for a custom SSH port. Security by obscurity is not security.
• No one should be directly able to access your IP. Only allow traffic through an external WAF configured with strong ruleset. WAF should never be on the same server which you are protecting.
• Default deny in Firewall and only allow what you need.
• Use AppArmor to restrict services. For example https://www.secjuice.com/apparmor-say-goodbye-to-remote-command-execution/
• You can also make your CMS / control panel pages only accessible via Cloudflare Access.

This is enough in most cases.

serveradministrator

SSH on IPv6 only.

let_rocks

@serveradministrator said:
SSH on IPv6 only.

How do you manage if you’re on a IPv4-only network? Or don’t you have such scenario?

vitobotta

@TrK said:
You can do the hardware key login with yubikey, i never tried that though.

I use OTP, I don't think there is much difference other than OTP is kinda easier since I have it with me with the phone (Bitwarden) without having to carry the Yubikey.

@cpsd said:
Those things + knock on port to allow my current ip

Is it really necessary? It complicates things if you want to access from elsewhere and anyway with root/password auth disabled, OTP required and Fail2ban banning after just one failed attempt, do you really need to restrict access by IP?

@Merakith said:

• Lock SSH over Tailscale/Wireguard Instance. No one should be able to SSH without being on your network. You can then remove fail2ban as bots won't be able to hammer your SSH port and no need for a custom SSH port. Security by obscurity is not security.

I was planning to set this up https://github.com/WeeJeWel/wg-easy

But same question I asked @cpsd : is this really necessary considering what I am already doing with SSH? The problem with these solutions is that they make it a bit more difficult to access my servers from arbitrary locations.

• No one should be directly able to access your IP. Only allow traffic through an external WAF configured with strong ruleset. WAF should never be on the same server which you are protecting.

For anything exposed to the web I use Cloudflare's proxy

• Default deny in Firewall and only allow what you need.

Yep I am doing that, I forgot to add it to the list

• Use AppArmor to restrict services. For example https://www.secjuice.com/apparmor-say-goodbye-to-remote-command-execution/

Will that work if I run everything as Docker containers?

• You can also make your CMS / control panel pages only accessible via Cloudflare Access.

Not familiar with Access but will take a look, thanks!

vitobotta

@serveradministrator said:
SSH on IPv6 only.

What's the benefit?

harrison

@vitobotta said: using a non-root user to SSH into the server
disabling root login

Why? The idea is to login as non-privileged user and then login as root after connection?

Mumbly

@vitobotta said: What's the benefit?

To weed out all IPv4 scanners?
His /64 subnet have IPv6 addresses from potential 18,446,744,073,709,551,616 addresses pool. What will you scan here?

ralf

My Hetzner dedi (which is just for my company's new project) only allows SSH from 2 external IPs and anything that has a wireguard connection to it. I also have full disk encryption using the small ramfs with dropbear so you can ssh and unlock the LUKS volume to continue booting.

Apart from SSH, the only other ports open are HTTP and HTTPS which are DMASQ forwarded to a VM running haproxy, and can send IP traffic to the VMs that are the back-end servers, either on the same box or elsewhere, connected via wireguard and only allowing traffic on the one port used to talk to my web-app.

I have another dedi at Kimsufi that's not quite as locked down internally, as I never set up full disk encryption, but the only public ports are SSH and SMTP which is forwarded to a VM. There's a bunch of VMs on here, one per service, only accessible via wireguard.

For other services like public DNS, they are all on smaller VPS instances that talk to the primary DNS VM over wireguard. I also have a couple of redundant haproxy VPS instances that connect to the back-ends over wireguard, and only allowing traffic on the one port, and also a couple of VPS instances with trustworthy providers for additional redundant web back-ends, again they can only talk to the others over wireguard and only on certain ports.

ralf

@vitobotta said:

@serveradministrator said:
SSH on IPv6 only.

What's the benefit?

Only half the internet supports IPv6, so there's fewer attackers!

(But seriously, also if you have a decent IPv6 allocation, e.g. /80, you can choose a semi-random address in that space for SSH and nobody will guess it)

rcy026

@vitobotta said:

@serveradministrator said:
SSH on IPv6 only.

What's the benefit?

Do you know how many ipv6 addresses there are? Just doing random scans are basically useless, just pick a random ipv6 in your /64 or whatever and the chances of anybody finding it is close to zero. Plus, most of the scriptkiddies haven't figured out how to use ipv6 yet, so the noise is minimal compared to ip4.
Of course disabling root, fail2ban, ufw and all other countermeasures still apply, but on ipv6 you are battling against a few directed attempts, not the fucking shitstorm of crap you get on ip4.

And come on, it's 2023, if you are still not running ipv6 you are 20 years behind.

rcy026

@ralf said:
Only half the internet supports IPv6, so there's fewer attackers!

You mean only half the internet uses ipv6.
The vast majority of products released in the past 10 years supports ipv6 just fine, most even defaults to ipv6, so it is not a question of support.

stoned

Many way to skin this cat, but essentially you want to implement the Principal of Least Privilege.

Non standard ssh port only thwarts low level scanners, not targeted attacks, but it still helps.

Fail2ban helps.

Mostly, just lock down everything other than 80/443 if you're serving that, else, keep everything closed and only your selected IPs should be able to access.

Create an IPSET group and put all your home, server, and whatever IPs you will access your server with. Allow only those IPs in this IPSET group to access anything. If anyone not in your IPSET group should nmap every port, they should get back 0 open ports, unless you have a public service, which needs to be opened.

Use iptables conntrack to do rate limiting

# Allowed local port range, allow what you want
net.ipv4.ip_local_port_range = 2000 65535

# Protect Against TCP Time-Wait
net.ipv4.tcp_rfc1337 = 1

# Decrease the time default value for tcp_fin_timeout connection
net.ipv4.tcp_fin_timeout = 15

# Decrease the time default value for connections to keep alive
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_keepalive_intvl = 15

net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.default.secure_redirects = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.all.log_martians = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.all.accept_redirects = 0

net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_recv=45

# Ignore all ICMP ECHO and TIMESTAMP requests sent to it via broadcast/multicast
net.ipv4.icmp_echo_ignore_broadcasts = 1

# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0

# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0

net.ipv4.tcp_syncookies and net.ipv4.tcp_synack_retries = 3 to avoid A TCP SYN flood attack DOS denial of service

Addresses of mmap base, heap, stack and VDSO page are randomized kernel.randomize_va_space=2

Ignore bad ICMP errors net.ipv4.icmp_ignore_bogus_error_responses=1

Protects against creating or following links under certain conditions
fs.protected_hardlinks=1
fs.protected_symlinks=1

Linux can limit the amount of various system resources available to a user process. These limitations include how many files a process can have open, how large of a file the user can create, and how much memory can be used by the different components of the process such as the stack, data and text segments. ulimit is the command used to accomplish this.

For the ulimits to persists across reboots you would need to set the ulimit values in the configuration file /etc/security/limits.conf.

DP

@vitobotta said: But same question I asked @cpsd : is this really necessary considering what I am already doing with SSH? The problem with these solutions is that they make it a bit more difficult to access my servers from arbitrary locations.

From a security's perspective, isn't that good?

If it's difficult for you, it'll be a lot harder for anyone else

vitobotta

@harrison said:

@vitobotta said: using a non-root user to SSH into the server
disabling root login

Why? The idea is to login as non-privileged user and then login as root after connection?

Basically yes. 'root' would be the default bots try to brute force, while a custom username may not be as easy to guess

rick2610

Only allow SSH from static office/home IPs. If there is no static IP use a cheap VPS and install VPN on it.

cochon

@vitobotta said:

• using a non-standard port for SSH
• having fail2ban configured with no tolerance

I find these of little use these days, brute forcing is usually low and slow from many IPs, rarely enough to trigger Fail2Ban.

• disabling password authentication

This seems to be the main influencer along with ChallengeResponseAuthentication no. With passwords or challenges enabled you get a lot of tries, with both disabled barely enough to be an annoyance. I guess scripts these days take more notice of 'Authentications that can continue: publickey' and go elsewhere.

• having a passphrase on the SSH key

Not sure that really needed to be on the list

As others have mentioned, can't beat the comfort factor of the SSH key being in a YubiKey where it can't be exfiltrated even if you use it on a compromised PC.

Hiding the connection behind another port or a hard to find IPv6 or firewalls is fine for reducing log file bloat if that's really a problem, but not as a security measure, a classic example of 'security through obscurity' if so.

Brute forcing is literally impossible with [only] SSH keys enabled, and not worth losing sleep over, thus time would be better spent worrying about the client side security than the server.

ralf

@rcy026 said:

@ralf said:
Only half the internet supports IPv6, so there's fewer attackers!

You mean only half the internet uses ipv6.

Actually, no, I meant supports.

I have quite a few VPS that are IPv4 only. Quite a few big name sites are IPv4 only, e.g. github. Even my home internet got downgraded to IPv4 only when I upgraded from ADSL to fibre.

That IPv6 usage is around 40% globally, isn't just because people have it and choose not to use it... it's mostly because they can't actually use it.

And anyway, even if my point wasn't true, it was also clearly marked with a to show it was only intended to be a joke.

serveradministrator

@let_rocks said:

@serveradministrator said:
SSH on IPv6 only.

How do you manage if you’re on a IPv4-only network? Or don’t you have such scenario?

I have no other option if I am not connected via IPv6, I just don't have any access to the server. All the other answers are correct, there are 0 attacks since I moved to IPv6 only SSH, good luck scanning my SSH server.

SocksAreComfortable

All this talk about IPv6 support on servers -- is my home stuff gonna be operating on IPv6 in any significant fashion? Or does it just keep using NAT from now until the forseeable future?

serveradministrator

@SocksAreComfortable said:
All this talk about IPv6 support on servers -- is my home stuff gonna be operating on IPv6 in any significant fashion? Or does it just keep using NAT from now until the forseeable future?

My home network is fully IPv6 ready, including servers for the last 7-8 years.

SocksAreComfortable

@serveradministrator said:

@SocksAreComfortable said:
All this talk about IPv6 support on servers -- is my home stuff gonna be operating on IPv6 in any significant fashion? Or does it just keep using NAT from now until the forseeable future?

My home network is fully IPv6 ready, including servers for the last 7-8 years.

So if you go to a local IP, instead of going to a 192.168.x.x or a 10.x.x.x. you go to a...long-but-unique address? Does that allow for more nuanced control of external access?

serveradministrator

I go to fd88::5 instead of 192.168.1.5 and so on.

cochon

@SocksAreComfortable said:
All this talk about IPv6 support on servers -- is my home stuff gonna be operating on IPv6 in any significant fashion? Or does it just keep using NAT from now until the forseeable future?

If you have IPv6 from your ISP then most mainstream desktops Windows/Mac/Linux will use that in preference. Where the service only supports IPv4 that will be used as a fallback via NAT locally as before. A lot of major sites, services and CDN's support IPv6 these days.

eris

Besides

• using a non-standard port for SSH
  Don't care

• using a non-root user to SSH into the server
  Yes Only for production servers

• disabling root login
  Yes Only for production servers

• disabling password authentication
  Yes always + My own templates already disabled by default

• requiring 2FA OTP code
  Don't care

• having a passphrase on the SSH key
  Don't care

• having fail2ban configured with no tolerance
  using https://github.com/chaifeng/ufw-docker to fix the Docker issue with ufw

• Don't care

• For some servers I have an iptables active + whitelist my ip addresses and VPN / VPS server

SocksAreComfortable

@cochon said:

@SocksAreComfortable said:
All this talk about IPv6 support on servers -- is my home stuff gonna be operating on IPv6 in any significant fashion? Or does it just keep using NAT from now until the forseeable future?

If you have IPv6 from your ISP then most mainstream desktops Windows/Mac/Linux will use that in preference. Where the service only supports IPv4 that will be used as a fallback via NAT locally as before. A lot of major sites, services and CDN's support IPv6 these days.

Seems my ISP (Comcast) is not offering that here at least, sadly:

No IPv6 address detected You appear to be able to browse the IPv4 Internet only. You will not be able to reach IPv6-only sites. To ensure the best Internet performance and connectivity, ask your ISP about native IPv6. Your DNS server (possibly run by your ISP) appears to have IPv6 Internet access. Your readiness score 0/10 for your IPv6 stability and readiness, when publishers are forced to go IPv6 only

So aside from the DNS servers (I use the 1.1.1.1 from Cloudflare)....bah

SocksAreComfortable

@serveradministrator said:
I go to fd88::5 instead of 192.168.1.5 and so on.

Ah that's right you can use the shorthand version...I was worried about remembering those long addresses, hah. Glad to know it works well for local access!