Introduction

This is the second post covering deploying a Proxmox node for a cyber security home lab. With our first node configured and our firewall/router operational, we’ll move onto the next two nodes. There’s an important caveat here. I initially only had a Raspberry Pi for the quorum device. Please revisit the first article as I added a cheap GMKtec node that could actually run some workloads as well, and changed the VLAN20 cluster network setup. These instructions are left here for anyone who may just be using a third device for quorum only but are now no longe required.

This was also done the hard way, with a jump host into the quorum device to prevent us from needing to access the cluster traffic VLAN.

  • Set up the second node to mirror the first.
  • Set up a Raspberry Pi quorum device.
  • Create the Proxmox cluster on node one.

Proxmox Node Two & Node Three

  • Follow the same steps from the first article to install Proxmox and execute the storage plan. Both my nodes are identical in respect of hardware between node one and two, so with the exception of a different hostname and IP address, the remaining tasks were identical to the first node.
  • Configure the networking as per the first article. Noting an increment to the IP for the enp4s0f1 interface for the cluster network:
    • Node 2: 192.168.20.3
    • Node 3: 192.168.20.4

The GMKTec

  • This host only has one storage drive which is shared with Proxmox OS. I did not set up the ZFS pools.
  • Networking was also different. This is a transitional workaround due to limited switch ports. The ideal configuration would have enp3s0 on its own switch port carrying 192.168.20.4/24 directly, matching the Lenovo node configuration.

[!NOTE] Proxmox’s network GUI does not support adding secondary IP addresses to an existing interface. We edit /etc/network/interfaces directly for this reason.

  1. Open a shell.

  2. vi /etc/network/interfaces (or nano instead of vi if you prefer).

  3. Note the post-up and pre-down commands that I have added to the configuration:

       iface vmbr0 inet static
         address 192.168.0.22/24
         gateway 192.168.0.1
         bridge-ports enp1s0
         bridge-stp off
         bridge-fd 0
         bridge-vlan-aware yes
         bridge-vids 2-4094
         post-up ip addr add 192.168.20.4/24 dev vmbr0
         pre-down ip addr del 192.168.20.4/24 dev vmbr0

  4. :wq to save and exit.

  5. systemctl restart networking to apply our changes.

  6. ip addr show vmbr0 to verify we see the IP:

       ip addr show vmbr0
   : vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
   link/ether e0:51:d8:19:30:14 brd ff:ff:ff:ff:ff:ff
   inet 192.168.0.22/24 scope global vmbr0
      valid_lft forever preferred_lft forever
   inet 192.168.20.4/24 scope global vmbr0
      valid_lft forever preferred_lft forever
   inet6 fe80::e251:d8ff:fe19:3014/64 scope link proto kernel_ll 
      valid_lft forever preferred_lft forever

VLAN 20 Cluster Network

[!UPDATE] As per some updates in the first article, I moved away from VLAN 20 for cluster and replication traffic. I now configure the second interface directly into a private subnet. The majority of this article is no longer relevant as I also don’t use the Pi.

We have some decisions to make here from a networking and access perspective.

VLAN 20 is designed for cluster traffic, so there is no need for us to be able to access hosts on the 192.168.20.0/24 network. This means:

  • We don’t need to put a route in to access the 192.168.20.0/24 network from our workstation.
  • We don’t need to configure firewall rules from the workstation to this VLAN via the WAN.
  • We possibly wouldn’t even need a gateway in the Pi, but we want it to be able to do OS updates.

This isolation will come at a cost of accessing and managing the quorum device, as we will need a jump host to manage the Pi. We can later set up a jump host for other parts of the network too, but let’s prep a minimal Alpine Linux container to jump from.

[!TIP]

For this to work, I had to configure the switch port as an access port for VLAN20 in my switch and set it to untagged. I also had to ensure that the trunk VLAN ports were both tagged for VLAN20.

This configuration will be highly dependent on your switch model.

Jump Host

  1. Datacenter > [node name] > iso-linux > CT Templates
  2. Click on Templates
  3. Select the latest alpine image.
  4. Click Download.

With the template downloaded…

  1. Click Create CT in the top right.
  2. Hostname: jump,
  3. Enter a complex password. Next,
  4. Select the Alpine template. Next,
  5. Select bulk-pool-vmdata (configured from Part 1). Next,
  6. Leave CPU and memory at defaults,
  7. VLAN Tag: 20,
  8. IPv4: 192.168.20.10/24,
  9. Gateway: 192.168.20.1. Next,
  10. DNS domain: Your internal lab FQDN,
  11. DNS servers: 192.168.20.1. Next.
  12. Start after created: Check. Finish.

Upon boot, login as root with the password you set.

  1. apk update
  2. apk upgrade
  3. apk add openssh-client
  4. ssh-keygen
  5. I save mine with the default name but add _pi to the end. Copy the contents of the public key id_ed25519_pi.pub to use with the Pi.

Prep the Raspberry Pi

I’ll be using a Raspberry Pi 4 for my qdevice. I prepared this on Windows:

  1. Download the Raspberry Pi imager.

  2. Run the Raspberry Pi imager:

    • Raspberry Pi Device: Raspberry Pi 4
    • Operating System: Raspberry Pi OS (64-bit)
    • Select your SD card storage.

  3. Customise the settings:

    • Enter a hostname, username, password and locale settings. Move to services.
    • Enable SSH,
    • Select Allow public-key authentication only,
    • Paste the public key in as an authorized_key from the keypair you just generated (ends with .pub) as the authorized host,
    • Move to Options. Do not select eject media when finished.
    • Save, and write your image with the custom settings.

  4. I connect my pi via Ethernet to my network switch. Boot the pi and find it on your network from your router’s DHCP list.

  5. SSH into the pi with the key and we will begin to configure:

    ssh -i <raspberrypi_key_file> [email protected] # Replace with your Pi's DHCP address

sudo apt update
sudo apt upgrade

sudo nmtui

  6. Edit a Connection > Wired Connection.

    • Addresses: 192.168.20.5/24
    • Gateway: 192.168.20.1
    • DNS Servers 192.168.20.1
    • Search Domains: Your lab FQDN.

  7. Select OK.

  8. Back and Exit.

  9. Restart the Pi.

We should then be able to login to the Pi from the jump host:

ssh [email protected] -i id_ed25519_pi

Configure the Pi

We need to allow root login and password authentication for the cluster:

  1. sudo passwd root. Set strong password.
  2. sudo vi /etc/ssh/sshd_config.
  3. Swap PermitRootLogin prohibit-password for PermitRootLogin yes.
  4. Add PasswordAuthentication yes.
  5. :wq to save and quit.
  6. sudo systemctl restart ssh to restart the SSH service.
  7. Exit back to the container and try and login as root with the password you just set:
  8. ssh [email protected].
  9. If we have verified the root account is working then we will go ahead and install the corosync-qnetd package:
  10. apt install corosync-qnetd

Configure the Cluster and QDevice

As a reference we are using the Proxmox Cluster Manager instructions from here…

  1. Let’s add the network for the cluster.
  2. Node > Network > Create > Linux VLAN.
  3. Name: bond0.20
  4. No IPV4.
  5. Vlan raw device: bond0 our bonded Linux bridge. OK.
  6. Create > Linux Bridge.
  7. Name: vmbr20
  8. IPV4: 192.168.20.2/24.
  9. VLAN aware: Check.
  10. Bridge Ports: bond0.20
  11. VLAN IDs: 20
  12. Comment: Cluster Network. OK.
  13. Apply Configuration.

Repeat this on the second node, but increment the IPv4 to 192.168.20.3/24.

  1. Add the corosync-qdevice package on all cluster nodes. In a shell on each node run apt install corosync-qdevice.
  2. On the primary node:
  3. Datacenter > Cluster > Create Cluster.
  4. Cluster Name: Tiny Cluster or your domain name or whatever you like.
  5. Cluster Network: Select the 192.168.20.2 vmbr20 interface.

Replacing the Pi with a Third Node

I managed to grab a good deal on a GMKtec G2 mini at just over $100. It was substantially less than the ThinkStation nodes (with substantially less storage, RAM and networking). It is a slightly better quorum device as I can actually run Proxmox and some light workloads even with its modest specs.

Because it only has one drive in it, disregard a lot of the storage configuration and run with defaults. There are also only two 2.5GbE NICs so much of the networking configuration is ignored also.

To configure the network to be able to use the cluster replication network:

  • Configure the Linux Bridge vmbr0 to be VLAN aware.
  • Create a Linux VLAN vmbr0.20 with a CIDR of 192.168.20.4/24 and no gateway.
  • You may need to amend your switch port configuration.

Once this node has successfully joined the cluster, we can remove the quorum device:

pvecm qdevice remove.

What’s Next?

In the next article we’ll create the cluster and join the second and third node to it.