After you finished installing and rolling out all your services in part 1,2,3 you should now invest a bit more time in hardening those services and containers. Why did we not do that beforehand? Because the next few steps have the potential to break any of your services from working.
I usually recommend to get to a working setup first and then attempt to secure it afterwards. I mainly used Ubuntu minimal 20.04 but most of those steps also work for any other distro.
Before you attempt any of those steps make sure you have a working snapshot you can revert back to in case you screw up at any point.
More basic stuff
- Install UFW
UFW is a simple Firewall that creates iptables-rules. Install ufw with
apt install ufw
and then allow specific ports such as SSH, HTTPS and so on with
ufw allow 22 ufw allow 443
When you are done you can enable ufw with
- Change SSH Port in /etc/ssh/sshd_config
This one is very simple. Change the default SSH-Port to something higher up like 22334 for example. This does not really help with security but will stop bots and simple attacks from reaching your SSH-Server.
- Disable root-access via ssh
Another simple one. In /etc/ssh/sshd_config set PermitRootLogin to no to disallow root-logon via ssh.
- Secure shared memory (tmpfs)
This one should normally not lead to any issues but a typo here or there could brick your system, so make sure that you have a working snapshot.
Open up the file /etc/fstab and look for the line starting with "tmpfs" (if not found simply add it yourself) and modify it to look like the following:
tmpfs /dev/shm tmpfs defaults,noexec,nosuid 0 0
Afterwards reboot and check if everything is still working.
The next few lines disallow spoofing, enable syncookies and so on. This could potentially break a few things here and there so make sure to check everything after rebooting. Modify the following lines or add them if needed:
/etc/sysctl.conf net.ipv4.conf.default.rp_filter = 1 net.ipv4.conf.all.rp_filter = 1 net.ipv4.icmp_echo_ignore_broadcasts = 1 net.ipv4.tcp_syncookies = 1 net.ipv4.conf.all.accept_source_route = 0 net.ipv6.conf.all.accept_source_route = 0 net.ipv4.conf.default.accept_source_route = 0 net.ipv6.conf.default.accept_source_route = 0 net.ipv4.conf.all.log_martians = 1
This one should be obvious. Add 2FA if it makes sense and if supported.
AppArmor can protect you from different kind of threats. Downside is that it can potentially take a lot of time to get it all working and will propably break things if you lock down too "hard"
For a quick setup do the following:
# Install AppArmor apt install apparmor-easyprof apparmor-notify apparmor-utils certspotter
Afterwards follow this guide from Ubuntu and make sure you understand the terminology. Basically it works like this:
- Lock down service
- Check "complains"
- Allow what is needed
- Continue with next service
Because a lot of our services are running in a container we should also invest some time to lock those down and tighten security.
- Different user
One simple change is to use a different user instead of root to run our service. Those users need to exist in the image obviously. To do so, add the following lines in the dockerfile:
RUN adduser -D someuser USER someuser
Replace someuser with your actual user. In case you are not using a dockerfile you can manually add new users with:
docker run -it your_container /bin/bash
This will allow you to run a normal shell inside of your docker-container where you can then simply add new users.
When done then re-run the container with the switch -u USER and replace USER with your newly created user.
With that done, check with the following script if you are following best practice regarding docker-security.
What about fail2ban? What about SELinux?
I intentionally left out some solutions simply because they would not make sense or the trade off was too small. Some ports like SSH are only reachable over VPN so Fail2ban was not needed IMHO. SELinux is similar to AppArmor and will not be used.
The approach for SELinux is more like "allow nothing and then loosen up" while AppArmor is the opposite. Because of the setup I went with AppArmor and will stay with it.
There are other tools out there that you could try to tighten security such as Tiger which will scan your system for best practice and misconfigurations.
Last thing we want to setup is IPS on OpnSense. This will actually block invalid and suspicious traffic. For a basic setup navigate to Services > Intrusion Detection and enable IPS. At least enable the WAN-Interface and if needed also other interfaces.
Then navigate to Rules and enable the rules you want to use. I selected the following:
When this is done make sure to thoroughly check all your services for any blocked or dropped traffic. It is very likely that some may stop working or start dropping packets here and there.
If that happens then you may use other rules or in case of multiple interfaces, may need to disable IPS for some interfaces.
Next up we have DNS. If you followed my tutorial up to here then your OpnSense-FW is already running Unbound which will act as DNS-Server for all the other VMs in your network.
What we want is to never use the ISP or the hoster for DNS but instead use DNSSEC and OpenDNS. To achieve that we need to go to Settings > General and type in the OpenDNS-Servers which are 22.214.171.124 and 126.96.36.199. Also make sure to disable "Allow DNS server list to be overridden by DHCP/PPP on WAN"
With that done set up all your servers to use the OpnSense-FW as DNS-Server and you are good to go. :)
I mentioned in my first post that you should create backups for your VMs and store them in a safe place. All effort invested up to this point is pointless if you safe your backups unencrypted for any attacker to grab from your HDD.
I did not create any backup of Proxmox itself because setup is very simple and takes less time than restoring. That leaves us with the VMs only.
Proxmox supports encrypted backups but you need to make sure that this actually works. To do so run the following from a root-shell on your actual proxmox-server
pvesm set <storageID> --encryption-key YOUR_ENC_KEY
This should in theory encrypt each new backup from this point on. In case it does not and you do not feel like troubleshooting you can implement something manually.
Encrypt with AES
Instead of trusting Proxmox with the encryption part we can simply encrypt the backups with our own solution. This has a few downsides though. After a backup has been created it will be saved in unencrypted state and will then be encrypted. This means there is a short timeframe where the backup (if stolen) could be used by an attacker.
We have two options here. First, Proxmox supports hook scripts which you can call at different points of the backup process. However, I never got it working as stable and reliable solution which left me with option 2.
Here we simply do the encryption with openssl and use asymetric encryption to keep the decryption key on our side. This means that the backup (after encryption) is just data garbage and does not pose a security-risk anymore.
First, we will need to create the keys. To do so, run the following on your client:
openssl req -x509 -nodes -days 10000 -newkey rsa:2048 -keyout privatekey.pem -out publickey.pem
Next you will need to create the following on your Proxmox-Server:
- Transfer publickey.pem to your proxmox-server
- Script to do the actual encryption
- Cronjob to periodically call the script
Simply copy and paste the content of publickey.pem into a file publickey.pem on your server. Then create a file like /home/user/encryptBackup.sh and paste in the following:
for f in /mnt/pve/Backup/dump/*.vma.zst; do echo "Encrypting $f" openssl smime -encrypt -aes256 -in $f -binary -outform DEM -out $f.vma publickey.pem rm $f done
Next up give your script execute-permission with chmod +x and then try to run it. (You will need "normal" backups at this point, so create them if not done yet)
This should then go through all your backups and do the following steps:
- Create an encrypted version of backupX.vma.zst with public key
- Save encrypted version in the same folder as backupX.vma.zst.vma
- Delete the original unencrypted backup
Why vma.zst.vma? You could give your backup any file-extension but with the extension .vma you can still see it in the backup overview of proxmox even though it is useless data-garbage at this point.
Last thing you will need to do is to create the cronjob to run the script periodically. To do so switch to root and type in crontab -e and paste in the following:
0 5 * * 1 /bin/sh /home/user/encryptBackup.sh
With this setup the script will be run every monday at 5am. Modify the timing according to your backup needs and give your backup enough time to finish.
If you ever need to use your backup you will need to decrypt it first. To do so run the following:
openssl smime -decrypt -in backupX.vma.zst.vma -binary -inform DEM -inkey privatekey.pem -out backupX.vma.zst
After doing that transfer it back to /etc/pve/Backup/dump (replace Backup with your storage-id) and then you should be able to use it from there.
If you followed everything up until this point then you should now have most of your services in your own secure cloud. Congrats! :)
We are not done yet however. If you are still using Windows or Apple, iPhones or stock Androids, searching with Google, texting via Whatsapp or Facebook, your data is still being used for tracking and custom ads.
Next up we want to switch to something more private and leave Windows and OSX behind and show you how to configure your notebook/desktop for privacy and security.