KVM Virtualisation on a Hetzner Server

I recently wrote about my experience of setting up a dedicated server at Hetzner. I leased those machines with the goal of running KVM on them to provide many VMs (Virtual Machines).

NOTE: This article has been updated (2020-11-10) for Ubuntu 20.04.1 LTS.

Actually getting the network configured correctly on both the KVM Host (server) and in the KVM Guests for Hetzner's network proved to be quite tricky, so I thought I would write it up here.

Previously, when I used servers at SoYouStart, the process was simply to buy a block of IP addresses, and create a Virtual Mac address for each one in their Control Panel, after which you would configure each guest VM's NIC with the appropriate IP and Virtual Mac address. Hetzner do not offer Virtual Mac addresses, and so instead a static routing approach is required. This is not better or worse in-particular than SoYouStart, it is just different to what I had become accustomed to ;-)

Network Details

My Hetzner Server's NIC: enp4s0
My Hetzner Server's MAC: 10:20:30:40:50:60
My Hetzner Server's IP: 123.123.123.87
Hetzner's Gateway for my Server: 123.123.123.65

I purchased an additional IPv4 address block from Hetzner to use for my VMs: 222.222.222.240/28. This means that the usable IP addresses for the VMs will be from 222.222.222.241 through to 222.222.222.254 inclusive.

We will create a bridged network setup, and each Guest VM will forward its traffic via the host, the following diagram gives a basic outline.

Configuring the KVM Host

If you did not read my previous post, then our server is running Ubuntu 19.04 and we have already installed the KVM Ubuntu packages by running: sudo apt-get -y install qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils uvtool.

First, we need to modify the servers network configuration to create the bridge interface. The configuration file is /etc/netplan/01-netcfg.yaml and the out-of-the-box config for my server looks like:

### Hetzner Online GmbH installimage
network:
  version: 2
  renderer: networkd
  ethernets:
    mainif:
      match:
        macaddress: 10:20:30:40:50:60
      addresses:
        - 123.123.123.87/32
        - 2a01:2a01:2a01:2a01::2/64
      routes:
        - on-link: true
          to: 0.0.0.0/0
          via: 123.123.123.65
      gateway6: ee60::1
      nameservers:
        addresses:
          - 213.133.100.100
          - 213.133.99.99
          - 213.133.98.98
          - 2a01:4f8:0:1::add:9999
          - 2a01:4f8:0:1::add:1010
          - 2a01:4f8:0:1::add:9898

In case a mistake is made when modifying the network config, it is a good idea to make a backup of the original config: cp /etc/netplan/01-netcfg.yaml /etc/netplan/01-netcfg.yaml.BAK.

We will edit the netplan config file to create the bridge like so:

network:
  version: 2
  renderer: networkd
  ethernets:
    enp4s0:
      match:
        macaddress: 10:20:30:40:50:60
      dhcp4: no
      dhcp6: no
  bridges:
    br0:
      macaddress: 10:20:30:40:50:60
      interfaces:
        - enp4s0
      dhcp4: no
      dhcp6: no
      addresses:
        - 123.123.123.87/32
        - 2a01:2a01:2a01:2a01::2/64
      routes:
        - to: 0.0.0.0/0
          via: 123.123.123.65
          on-link: true
        - to: 222.222.222.240/28
          scope: link
        - to: "::/0"
          via: "ee60::1"
          on-link: true
      nameservers:
        addresses:
          - 213.133.100.100
          - 213.133.98.98
          - 213.133.99.99
          - 2a01:4f8:0:1::add:1010
          - 2a01:4f8:0:1::add:9999
          - 2a01:4f8:0:1::add:9898

NOTE: There are two key things about the above configuration:

The bridge interface br0 must have the same MAC address as the physical enp4s0 interface, otherwise Hetzner will not route the traffic from the server.
We have added a static routing rule for the br0 interface for the subnet 222.222.222.240/28 that we purchased for our VMs. Without this we will not be able to send IP traffic to our VMs.

To apply the network changes simply run sudo netplan apply. If anything goes horribly wrong and you can no longer SSH to the server, you will need to arrange to use the Hetzner remote physical console (also called a KVM) to fix the network config. Once you have access you can start by copying the backup of the netplan config back into place and running netplan apply.

Next we must enable IP forwarding, so that the traffic from our new subnet is routed via the servers main IP address, to do this run: sysctl -w net.ipv4.conf.all.forwarding=1. However, for this change to be persisted across reboots, we also need to edit the file /etc/sysctl.conf and within there set net.ipv4.ip_forward=1.

Remember that UFW firewall that we enabled in the previous post? We need to instruct it to allow the IP forwarding, otherwise we will not be able to communicate with out VMs via TCP/IP. This is done by running the command: sudo ufw route allow in on br0 out on br0.

Creating a KVM Guest VM

When creating a guest VM, we will use the Ubuntu Cloud images and uvtool.

As part of the VM setup, I want to specify the network settings that will be used by the Operating System within that VM. To do that, I would like to supply a Cloudinit config for netplan, unfortunately the versions of uvtool shipped before Ubuntu 19.10 do not support that. We can however patch our uvtool to support this. If you are using Ubuntu 19.10 or later you can skip this step. This step only needs to be done once:

cd /tmp
wget http://static.adamretter.org.uk/uvtool-cloudinit.patch
sudo patch /usr/lib/python2.7/dist-packages/uvtool/libvirt/kvm.py /tmp/uvtool-cloudinit.patch

We will now setup a new Guest VM, we will call it guest1. First, we generate an SSH key to use when connecting to our new Guest VM:

mkdir ~/guest-ssh-keys
chmod 700 ~/guest-ssh-keys
ssh-keygen -b 4096 -C "ubuntu@guest1" -f ~/guest-ssh-keys/guest1

Secondly, we will create a netplan configuration for the Guest VMs network settings:

cat > /tmp/guest1-netplan << EOL
version: 2
ethernets:
  enp1s0:
    addresses:
      - 222.222.222.241/28
    gateway4: 123.123.123.87
    nameservers:
      addresses:
        - 213.133.100.100
        - 213.133.98.98
        - 213.133.99.99
    routes:
      - to: 123.123.123.87/32
        via: 0.0.0.0
        scope: link
EOL

NOTE: There are two key things about the above configuration:

The guest's network configuration file does not start with the key network:, as it is passed as the --network-config argument to uvtool.
The gateway and route in the above configuration are set to the IP address of our Host (dedicated server). Our host will forward the IP traffic so that its source has the correct MAC address for Hetzner to route it.
The interface name enp1s0 is correct for Ubuntu 20.04.1 LTS, in previous versions of Ubuntu, this should be defined as ens3. It seems the interfaces were renamed due to changes in newer versions of Systemd :-/.

Before we use uvtool to create our VM, we want to make sure that we are using the latest Ubuntu Cloud images, so we refresh our images:

sudo uvt-simplestreams-libvirt sync --source=http://cloud-images.ubuntu.com/minimal/releases arch=amd64 release=disco

Finally, we can now use uvtool to create our VM:

uvt-kvm create \
    --ssh-public-key-file ~/guest-ssh-keys/guest1.pub \
    --memory 4096 \
    --disk 40 \
    --cpu 2 \
    --bridge br0 \
    --network-config /tmp/guest1-netplan \
    --packages language-pack-en,openssh-server,mosh,git,vim,puppet,screen,ufw \
    guest1 arch="amd64" release=disco label="minimal release"

All being well, running virsh list --all should show our newly created and running VM:

$ virsh list --all
 Id   Name                      State
-----------------------------------------
 1    guest1                    running

Now that our Guest VM is up and running, we can connect to it using the SSH key we created earlier:

ssh -i ~/guest-ssh-keys/guest1 ubuntu@222.222.222.241

And that's it, we now have our first Guest VM running on KVM on our Hetzner dedicated server.