DIY cloud - Proxmox, Proxies & Seafile

Annoyed of being a glass citizen? Follow this how-to to create your own private cloud 2/5

More of the same

After we set up the basic requirements in the first post, we are now going to setup and install the basics for our services we are going to host. If you followed the first part of this blog-series, then you should now have the following:

  • Running Proxmox-Server
  • Encrypted disks with LUKS
  • Locked down Access via VPN
  • Domainname to use for the next steps

Network setup
After we got our Proxmox-Server online we now need to make sure that the VMs itself have access to the Internet as well. This step really differs from hoster to hoster. In case you are going with Hetzner then you can copy and paste my code and modify it to fit your environment.

Before you do that though, make sure that the MAC of your WAN-Interface (usually vmbr0) fits the MAC your hoster expects to see.

If you skip this step, you may get Abuse-Mails regarding MAC-Spoofing.

auto vmbr0
iface vmbr0 inet static
        address IPv4_FROM_HOSTER/26
        gateway GW_FROM_HOSTER
        bridge-ports enp41s0
        bridge-stp off
        bridge-fd 1
        pointopoint GW_FROM_HOSTER
        bridge_hello 2
        bridge_maxage 12

auto vmbr1
iface vmbr1 inet manual
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

Backup
To have at least a few snapshots of your VMs we should attach another HDD to our server. You can also use the same disk or even send the backup to a NFS. If you do not want to pay for another HDD, you can also disable RAID and use the other HDD as backup drive.

Simply make sure to also encrypt the data on disk. Proxmox provides a very simple solution which automatically encrpyts each backup saved on to the specific storage.

pvesm set ID_OF_STORAGE --encryption-key /path/of/enc-key.txt

Network Setup – Firewall

Now that the basics are done you should be ready to setup the first VM which is the Firewall. You can choose what you want but I would recommend either OpnSense, PfSense or IpFire.

I went with OpnSense and created a simple VM with 2 NICs, where vmbr0 is WAN and vmbr1 is going to be LAN. When setting up the VM make sure to toggle mitigations for Spectre and Meltdown.

Depending on your CPU you need to figure out what works and what does not. This can be done by simply changing the CPU Flags to “On” (example ibpb) when creating the VM. I also recommend to enable the “AES”-Flag for your Firewall-VM to increase perfomance.

Again, when you create the WAN-Interface make sure that the MAC-Adress matches with what your hoster is expecting and modify the rest to your needs.

Jump-Host or another VPN
After setting up the Firewall you will need to access the Web-GUI from the internal LAN. To do so you could either create a new VPN, putting you right into the LAN-Network (I chose the VPN) or set up another VM which will act as a sort of jumphost.

The choice is yours but I would recommend a VPN as it makes it simpler in the long-term and something I would consider the “cleaner” version of the two.

VLAN
After that I set up VLANs for each service I am about to host. This is done in order to further separate each VM. Worst case, one of my services gets popped and taken over.

In this case I “only” have to worry about what I lost with this specific service while those VMs have no access to any of my other VMs. To do so you need to prepare a few extra NICs on the Firewall and give them a specific VLAN-Tag. Positive sidenote, you do not need to shutdown the VMs or reboot the host itself. In short:

  • Add another vmbr on Proxmox
  • Proxmox > Network > Linux Bridge (VLAN-aware)
  • OpnSense > Hardware > Add NIC (VLAN 10)
  • OpnSense > Create Firewall Rule for new NIC to allow Internet-Access
  • OpnSense > Enable DHCP if needed for new NIC
  • OpnSense > add additional NICs if needed

When you are done it should look like the following:

Caddy
Now we will set up Caddy as our reverse proxy which will then proxy all the services to the right server. A more traditional setup would be to use something like Nginx as reverse proxy. You can even run Nginx on OpnSense as plugin however I never got it to a stable point and had a lot of connection issues.

There is also Trafik which is also able to proxy bare-metal or non-docker-endpoints. In this case I went with Caddy which makes it very simple to proxy several different services.

Not only that but Caddy also takes care of requesting and installing certificates for each of your services. To follow what I did, do the following:

  • Install caddy
  • edit the file in /etc/caddy/Caddyfile and comment out the default listener
  • Add the service you intend to proxy to:
subdomain.example.com {
        reverse_proxy 192.168.10.100
}

And that is all you need to do on the caddy-side. In case not done yet, forward HTTP and HTTPS from WAN to your proxy via OpnSense (Firewall > NAT > Port forward) and then reload caddy with systemctl reload caddy

Are we there yet?
Ok cool but when are we actually going to roll out services you may ask? Now!! ;)

This was actually the last step and we should just take a quick check if everything is running as expected. A short recap of what we should have achieved so far:

  • Proxmox:
  • Locked behind VPN
  • Working backup
  • Firewall:
  • Locked down only allowing HTTP/HTTPS
  • Working VLANs
  • Proxy:
  • Receiving HTTP/HTTPS from the Internet via NAT
  • Setup one service with caddy

Looking back at the graph our setup should look like the following now (the graph is only supposed to show the internet-connection not the whole network-setup):

If you got here then we can finally begin with the fun part…rolling out services

Awesome-Selfhosted

One of the best ressources for selfhosted solutions is reddit/r/selfhosted and this Github list. There you should find a solution to nearly any kind of service.

For starters I want to find solutions for the following services:

  • File-Storage
  • Bookmark-Sync
  • …and more…

This is of course not all and will be expanded further down the road but as a general example for this blog, I want to get those 2 services online first.

File-Storage

The first version of this post described a Filerun-Installation. Because I was not happy with Filerun being closed source, I switched to seafile. In case you still want to go with Filerun, visit this LINK.

MEGA, OneDrive, DropBox, GoogleDrive, iCloud and any other kind of file-storage that allows to sync and share data with other devices and users falls under this category. After a lot of trial and error I went with SeaFile because

  • Stability
  • Simplicity
  • No fluff on top
  • Open source

Another option would be Nextcloud or Owncloud, however IMHO those try to offer a ton of additional features that tend to break here and there and cause headaches in the long run.

This may be just me but rolling out Nextcloud always needed a lot of optimization and fixing afterwards while Filerun “just works” and setting it up is a no-brainer.

Setup
Setting up Seafile is pretty simple and very straightforward. Seafile is split into a community and enterprise version with the community version being for free. Seafile works on a lot of OS but for simplicity I rolled out another Ubuntu-Server minimal 20.04.

Then I created another VLAN and modified Firewall-Rules (as mentioned before). In the end the new Ubuntu-Server should not be able to access anything but Internet and the Proxy but not any of the other VLANs.

I also added another 1TB-HDD to the VM and mounted /var on that HDD. This is where all the Docker-containers will run.

Next install Docker and Docker-compose with apt install docker docker-compose -y and then download the docker-compose.yml.

docker-compose.yml
Modify the file to your liking but make sure to at least modify the following:

  • MYSQL_ROOT_PASSWORD
  • DB_ROOT_PASSWORD
  • memcached (I recommend at least 512mb)
  • Timezone
  • Ports, disable HTTP and enable HTTPS

Caddy
Now if not already done go back to your Caddyfile on the Proxy-Server and edit the file /etc/caddy/Caddyfile and add the following for your Seafile-Server.

subdomain.example.com {
        reverse_proxy IP_OF_YOUR_SEAFILE-SERVER
}

When that is done docker-compose up -d and visit your new Seafile-Server.

By the way, did you notice that we did not set up any kind of SSL-certificate?
All taken care of by caddy :)

Bookmark-Sync

With File-Storage completed it is time to set-up bookmark-sync for all my devices. The idea was to replace Apples iCloud and Firefox Bookmark sync. The requirements were simple: Be able to read and write bookmarks to the same profile on all of my devices (Linux and Android).

The sync-process should be as simple as adding a new bookmark locally without having to visit an extra website. On Android that could be solved by an app that does the job for you. Linux-Firefox usually relies on addons.

If you check the Github-List then you’ll find out that there are not that many solutions that check all the requirements. After going through a lot of them, I went with xBrowserSync which offers addons for Firefox and Chrome and an app for Android.

Basics first
First things first, which is to setup the following:

  • Proxmox -> New NIC
  • OpnSense -> Add new NIC and give it a new VLAN-Tag
  • Proxmox -> Create a new VM
  • Add new NIC with just created VLAN-Tag
  • OpnSense -> Set up FW-Rules
  • Finish basic installation
  • Install docker & docker-compose

Xsync
After thats done go ahead and login to your VM and make sure that docker is working. You can check with docker run hello-world which should give you the typical hello world message. Then clone the xsync-repo with:

git clone https://github.com/xbrowsersync/api-docker/

Now edit the file .env and change the username and password, the fqdn where your xsync-api can be found at and optionally the db-name. Then edit the file docker-compose.yml and remove the caddy-part.

Normally running this image would also spin up a container with caddy, proxying the whole setup. We do not need this because we got caddy running already. Fortunately we only need to change a few things, such as commenting out the whole caddy-part and adding the listener-ports to our API.

With that done we only need to spin up the containers and edit the Caddyfile again on our proxy. Before you do that, make sure that your DNS for your domain is set up correctly and pointing to your proxy-ip.

When done edit the file /etc/caddy/Caddyfile on your proxy and add another entry like so (replace fqdn and ip accordingly):

subdomain.example.com {
        reverse_proxy 192.168.20.100:8080
}

After one more systemctl reload caddy run docker-compose up -d on your xsync-server and then you should be able to see Xsync running. :)

Whats next?

If everything worked out, you should now have at least 2 services running on your own cloud. :) Even though Bookmark-Sync and FileShare are important services, setting them up was very simple. Next I am going to replace a few more complicated services like Messaging and Mail.

…To be continued…