DIY cloud - Being paranoid

Annoyed of being a glass citizen? Follow this how-to to create your own private cloud 1/5

“Good old days”

Do you remember the time when the internet was still a “new thing” and totally not riddled with ads and the need to commercialise everything? When niche hobbys were not controlled by “influencers”, huge sponsors and ad-networks spying on you, trying to collect as much data as possible?

When Facebook was not a thing, Google did not exist yet and free speech was actually possible without “getting cancelled”. I remember and I am not even that old yet.

Nowadays it simply feels like you are being watched constantly. Not only to sell you more bullshit but also to create a profile. To get to know more about you and to figure about what you are up to, who you are connected to, what your hobbies are, what you bought, what you are going to buy, your ideas and thoughts and basically anything else.

With smartphones, alexas, echos, glass (does anyone remember google glass?) it has become much easier for trackers to actually follow you around and be with you 24/7. Do you know where you have been 5 years ago, what you did, where you traveled to, who you spoke to and what you said? No? Well, Google sure does

Edward Snowden showed us that not only is Google and Facebook interested in your data but also the Government and that the phrase nothing to hide is dangerous and simply wrong. Your data will be used against you and there is not much you can do about it after it has gotten in the wrong hands. In fact, government agencies will actively work against privacy to be able to get access to your data.

To combat this companies started more and more to implement features like E2E (end-to-end-encryption), ZeroTrust, FOSS and so on. At the same time governments tried to fight those privacy features by implementing several different laws and regulations, such as ACTA (EU), Vorratsdatenspeicherung (Germany), Metadata Access (Australia) and several others. With Chat Control 2.0 being the latest bs.

The main reason for those laws and regulations is either to combat terrorism, childporn or literally anything else that makes you look like a “bad guy” if you speak up. The goal is always the same, get access to any citizens private data. This includes any form of data as we already know from the Snowden-leaks.

Those regulations and laws are getting pushed constantly with different levels of success. If a law does not make it all the way through (denied/vetoed/whatever) then you can be certain that the party/parties behind it will try it again with a slightly modified approach.

“Throw shit against the wall until something sticks” seems to be the modus operandi when it comes to bringing those laws into reality.

Offline is the new Online

Because of this and because I needed something to distract me from constantly looking at WinDBG I decided to write this lengthy blog, where the main idea is to host everything yourself and not rely on Google/Facebook/Dropbox or any other big company that is potentially selling and/or leaking your data. The goal here is pretty simple:

Self-hosted
Anything that needs to be online needs to be hosted yourself. Mail, Fileshare, IM and anything you could think of needs to go on your own server.

FOSS
What is the point of self-hosting if you can not trust the code. Is FOSS free of exploits or backdoors? No, but they could potentially be found. Going FOSS 100% is technically not always possible.

Firmware-Blobs or specific parts like the TPM can not be open-sourced or open-source code is not available at all. There are other options in place to get around those, by disabling those features/parts or blocking them or similar.

ALL DEVICES
I do not see the point in going the privacy-route if you are still using the same services on your smartphone and have to trust Google or Apple. Therefore, I am also switching from Apple/Google to something privacy-focused. This will be a 2-step process which I will explain later on.

105% paranoid
My data is my data and only goes out of my hand if I say so. To stay true to this, we need to implement security and privacy-features on every level. I am talking about hardware and software. Cold-Boot-Attacks, Supply chain attacks or UEFI rootkits are all things we need to be aware of if we want to minimize data loss at any point.

In the next pages I am going to show you how I took my data back in my hands and restored what should have been the norm all along…privacy on the internet.

In the first part I will focus on the server-side and how to get a basic setup running. The next posts will touch on the actual services and how to deploy them and how to replace your iPhone/Android with a more privacy-focused option.

Server

When choosing the right server we need to differentiate between keeping the server at home or renting one. I believe that keeping a server at home is not potentially safer than keeping it in a secure and cool datacenter. Our goal is to keep the data on the hard drives secure from everyone except ourself.

This should also be possible if the server or hard drives get seized by law enforcement or a rogue datacenter-employee. If you are not a whistleblower then this may sound a bit over the top but it for sure helps me to sleep better at night, knowing that my data is very likely safe.

Choosing the hoster

If you want to get your own server then this obviously does not apply to you

When choosing a hoster we should keep the following in mind:

Location
You should choose a server in your country or at least the country where you reside usually. This is not a privacy-concern but related to latency and speed. If your server is half a globe away then things will be slow. (Given of course that there are good hosters in the chosen country)

Dedicated root-server
We do not want to go with containers or VMs (provided by the hoster). Simply because those run on a hypervisor which is not under our control. This opens up timing and caching-attacks or simply sniffing out the network traffic. While the latter is also possible on a dedicated root-server, we at least do not have to worry about a hypervisor on top.

Avoid Intel
Now why should we avoid Intel? IME (Intel Management Engine) is a small piece of hardware running on every Intel-MB 10 years or younger. It could be used to access any part of your computer/server no matter what OS you are running.

This is a nightmare when it comes to security. Unfortunately AMD is not much better with AMD PSP but at least there is an option to disable it in new BIOS-versions, which does NOT disable it completely but stops the different parts talking to each other.

Even if you can not disable PSP, I would still go with AMD simply because IME could be interacted with remotely, PSP (AFAIK) could only be used locally. Therefore, avoid Intel IME and try to go with AMD and if possible disable PSP in the BIOS.

If you are running your own Server, then I would even suggest to check out Coreboot and Libreboot to replace your manufacturers BIOS.

Monitoring
This is mostly optional but if possible get a server with Idrac (Dell), Ilo (HP) or any other form of Hardware-Monitoring. One downside of running a dedicated root-server is that you are responsible for any HW-faults. May it be failing hard drive, failing fan or whatever.

This is usually taken care of by the hosting provider but they will not monitor your server so it is your responsibility to inform the hoster when something like this happens.

Some hosters also build their own system around their servers. Be careful with how much access and info you want to send through those.

The usual
When going for a dedicated server take some time to think about the specs and what you need. Better plan ahead and leave some space. We all got used to the simplicity of simply adding some RAM and cores to VMs with just a few clicks, but this is a real server where simply adding more memory or space needs someone to walk up to the server, shut it down and plugin the hardware.

This can become expensive really fast and technicians on-site are not cheap. Other than that, it really comes down to what you are about to run on your server, which will then decide how beefy your machine has to be.

tl;dr:

  • Location: Get a server close to your location
  • Dedicated root-server: NO VM, NO CONTAINER, choose a real server, not shared
  • Avoid Intel and disable AMD PSP if possible
  • Monitoring: Make sure you can monitor your Server. If not provided then go with something like Zabbix
  • Plan ahead. Adding hardware is possible but expensive usually

In my case I went with Hetzner and one of their smaller root-servers. The specs are:

  • AMD Ryzen 5 3600
  • 64GB DDR4
  • 2×2TB Ent HDD (Software Raid1)
  • 1 static IPv4

Altogether this will cost me 40€/month which I think is a good payoff in terms of security and privacy once its all done.

If you are going with Hetzner and this is your first time ordering, there will be a small delay where Hetzner will ask for proof of documents and so on. Otherwise most servers are deployed within a few minutes.

RAID
First of all, decide if you need Software- or HW-Raid or any RAID at all. My server comes with Software-Raid (RAID is no backup) which I am going to use. The services I intend to run are either backed up offline (@home) or will not be backed up at all because provisioning them takes less time than restoring (Docker FTW).

After you know which configuration you are going with, deploy the Unix-Distro of your choice. Debating distro-choices is like debating religion. It is usually pointless and ends in a fight.

IMHO I would choose either Debian stable or Fedora or SUSE or something from the ground up, like Arch or Ubuntu minimal for simplicity. We should avoid unstable or “testing” editions because those usually end up breaking something in the long run.

Going completely open-source is not that easy after all and we are trying to find the least painful option here.

Proxmox
As I said, the distro is your choice but I would recommend Proxmox which is a hypervisor build on Debian. Another option would be Containers like LXC or Docker. However, IMHO breakouts from containers are “easier” and more common than breakouts from a fully virtualized VM in a hypervisor like Proxmox. In addition to being in a VM, we will also harden Proxmox itself to increase security.

Debian

The scripts used here are based on a Hetzner Script. They should run anywhere with little modification. First thing we want to do is install Debian 11 which we will later put Proxmox on top.

The goal is to install Dropbear and Busybox which will unlock/decrypt our Debian which is running Proxmox. You will need to modify those scripts if you are going with another hoster.

Basic setup
After provisioning log in to your server and follow the script I linked. The only thing you need to replace is the actual image used, hostname and the secret for the encryption in /tmp/setup.conf

Afterwards change the ssh-key for the actual system so you end up with 2 ssh-keys. One for dropbear and one for your OS. To do so create a new user first with adduser user1 then add this user to the wheels/sudo group. The actual name differs with each distro. usermod -aG sudo user1.

Then edit your ssh-config in /etc/ssh/sshd_config and change (uncomment if needed) the following:

Port 22 (Change it to anything you want like 22334)
LoginGraceTime 2m
MaxAuthTries 6
MaxSessions 5
PermitRootLogin yes (We will disable this after we confirmed our user is working)
PubkeyAuthentication yes
PasswordAuthentication yes (Will be disabled as well later on)

Now create a new ssh-key for your user with ssh-keygen -t rsa and then copy it over (after unlocking) to the main OS on your server with ssh-copy-id -p 22334 -i user1 user1@server.example.com
When that is done log back in and set both PasswordAuthentication and PermitRootLogin to no and restart the ssh-service.

Proxmox-Setup
When this is done we simply add Proxmox on top. When asked for Postfix-Setup you can easily set it to “No configuration” or anything else as it can be rerun later on.

echo "deb [arch=amd64] http://download.proxmox.com/debian/pve buster pve-no-subscription" > /etc/apt/sources.list.d/pve-install-repo.list
wget http://download.proxmox.com/debian/proxmox-ve-release-6.x.gpg -O /etc/apt/trusted.gpg.d/proxmox-ve-release-6.x.gpg
apt update && apt full-upgrade -y && apt install proxmox-ve postfix open-iscsi -y && apt remove os-prober -y

With this setup done we can continue to either buy a license for proxmox or use the free license and edit the repo-list.
To do so, edit the the pve-enterprise.list in /etc/apt/sources.list.d/ and remove the enterprise and replace it with download and the name of the list pve-enterprise with pve-no-subscription. Then rename the file and give it the same name pve-no-subscription.

To remove the nag-screen we open the file /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js and replace Ext.Msg.show with void and then reboot the server.

Simple hardening
When everything is done we want Proxmox only accessible to us. Proxmox will be put behind a webserver which will be only reachable via VPN. For now we want to disable some services. Skip those steps if you need any of these of course.

Disable NFS:
/etc/default/nfs-common set NEED_STATD=no

Disable RPC:
systemctl disable --now rpcbind.service rpcbind.socket

Disable postfix IPv6:
/etc/postfix/main.cf set inet_protocols = ipv4

Disable Postfix completely if not needed:
systemctl disable postfix

Disable SPICE-Proxy (Proxmox) if not needed or lock it down:
systemctl disable spice

Given that you are not running anything else, your server should have only ssh, pvedaemon and the pveproxy open. You can check with netstat -tulpen

Proxy it
Next step is to put Proxmox behind a proxy. One downside of keeping the Proxmox-GUI out in the open is that the proxmox-proxy itself is reachable for potential vulnerabilities and exploits. If there is an unknown vuln. in the proxy component, we would potentially loose the whole server with it.

To get around this we put Proxmox behind Nginx and lock that down further down the road. Now is the time to get certificates for your server and a domain-name.

Technically this is not really needed when we access the server via IP over VPN but I’ll keep it in in case someone wants to access their server over the internet. Regarding certificates I do recommend LetsEncrypt but use what you are comfortable with. When you got your certificates, go ahead and install Nginx and prepare a config-file for your server.

apt install nginx -y
rm /etc/nginx/sites-available/default
rm /etc/nginx/sites-enabled/default

Then create a file in /etc/nginx/sites-available/proxmox.conf and paste in the following:

upstream proxmox {
    server YOUR.FQDN;
}

server {
    listen 80 default_server;
    rewrite ^(.*) https://$host$1 permanent;

}


server {
    listen 443 ssl;
    server_name _;
    ssl on;
    ssl_certificate /etc/letsencrypt/live/your.fqdn/fullchain.pem; 		# managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/your.fqdn/privkey.pem;
    include /etc/letsencrypt/options-ssl-nginx.conf;
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
    proxy_redirect off;
    
    location / {
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection “upgrade”;
            proxy_pass https://127.0.0.1:8006;
            proxy_buffering off;
            client_max_body_size 0;
            proxy_connect_timeout  3600s;
            proxy_read_timeout  3600s;
            proxy_send_timeout  3600s;
            send_timeout  3600s;
        }

    }

When this is done link it to be available for nginx with ln -s /etc/nginx/sites-available/proxmox.conf /etc/nginx/sites-enabled/proxmox.conf

Turn on nginx with systemctl enable --now nginx and add the following to the nginx-service with systemctl edit nginx

[Unit] Requires=pve-cluster.service After=pve-cluster.service At last we need to disallow any connection except from localhost to pveproxy. Create the file /etc/default/pveproxy and add the following: ALLOW_FROM="127.0.0.1" DENY_FROM="all" POLICY="allow"

After all that check if you can access the proxmox-gui via either IP or FQDN. It should not be possible anymore to access the GUI directly (8006) but only through nginx.

Wireguard
While proxmox-gui is only accessible via nginx we really do not want it to be accessible at all over the internet. Some providers/hosters allow you to create private networks which you can use to connect trough. Otherwise I would recommend to install a VPN like wireguard and only allow access through the VPN.

Fortunately setting up a VPN got much easier with wireguard and there are a ton of guides out there for every distro and OS possible. I can recommend Wireguard.How

The steps are:

apt install wireguard -y

Then setup the keys with (umask 077 && wg genkey > wg-private.key) && wg pubkey < wg-private.key > wg-public.key && cat wg-private.key

Afterwads create a file /etc/wireguard/wg0.conf and add the following (replace with your private key)

[Interface] PrivateKey = AAAAABBBBBCCCCCC ListenPort = 51820

Then create the wg-interface at /etc/network/interfaces.d/wg0

auto wg0 iface wg0 inet static address 10.0.2.1/24 pre-up ip link add $IFACE type wireguard pre-up wg setconf $IFACE /etc/wireguard/$IFACE.conf post-down ip link del $IFACE

That is it for the server part. To create a client download wireguard for your OS and create a tunnel (empty tunnel) with the following data on your client. 10.0.2.2 is the IP of your client and 10.0.2.1 of your server. The endpoint is the public IP of your wg-server. To see the PublicKey run wg show wg0 on your server.

[Interface]
PrivateKey = ALREADY_FILLED_BY_EMPTY_TUNNEL
Address = 10.0.2.2/32
[Peer]
PublicKey = PUBLIC_KEY_SERVER
AllowedIPs = 10.0.2.1/32
Endpoint = 22.33.44.55:51820

Now go back to your server and add your client. To do so edit the file /etc/wireguard/wg0.conf and add the following data. Replace your keys accordingly.

[Peer]
PublicKey = PUBLIC_KEY_OF_YOUR_CLIENT_CHECK_WIREGUARD_ON_YOUR_CLIENT AllowedIPs = 10.0.2.2/32

When that is done run wg syncconf wg0 /etc/wireguard/wg0.conf and wg show wg0 to see if you were able to add your client. Afterwards try to connect. Now you should be able to visit your proxmox-server by visiting https://10.0.2.1. Next step is to lock down nginx.

To lock down nginx add the following file /etc/nginx/blockips.conf and add the following data:

allow 10.0.2.0/24; deny all;

With this setup we only allow connections to nginx from the VPN-Network and deny everything else. If you want you can further lock it down to only allow one IP instead of the whole subnet 10.0.2.0/24. Afterwards systemctl restart nginx and check if you are still able to visit proxmox via internet (not VPN). You should get a 403.

What is next?
If you followed the whole thing then you should now have a running encrypted, hardened proxmox-server, accessible only via wireguard (except ssh). There is more you can do to lock down the server, like install 2FA, add an additional user and enable the firewall.

In addition AppArmor/SELinux should be configured to lock down Debian itself. I recommend doing that after you have all your services running to not break anything while setting those up. In the next post I will go ahead and deploy the first VMs to  actually put proxmox to use.

To be continued…