Netbird On PiKVM
PiKVM is a feature-rich, industrial grade, Open-Source KVM over IP device.
It’s built on top of RaspberryPi hardware and Arch Linux as an alternative to expensive professional KVM-over-IP devices commonly used in datacenters for full remote control over servers.
This guide is heavily based on the official PiKVM Tailscale guide.
There’s a little gotcha when running Netbird on PiKVM, similar to running on SteamDeck.
To ensure data integrity in case of power loss, PiKVM’s root filesystem is by default mounted in read-only mode.
This means that Netbird, which typically requires write access to its files in /var/lib/netbird, cannot function properly in this environment by default.
To work around this limitation, we can use a tmpfs to provide Netbird with the necessary write access during runtime.
However, this means that changes to files in /var/lib/netbird will not persist across reboots.
This is acceptable in many use cases, but if you really need persistent storage for Netbird’s data, you may want to consider alternative solutions, such as using a dedicated USB or SD card storage, or some systemd-systext magic.
Prepare
First, we need to shut down any other VPN services that may be running on our PiKVM and have conflicting routes (especially Tailscale).
If we don’t have direct network access to PiKVM without our existing VPN, we need to ensure that we have another way to access it remotely because we’ll lose connectivity after disabling other VPN services. For example, install Netbird on some other machine in the same network and make sure that we can SSH into our PiKVM from there. Or configure that machine as a Netbird router for that subnet.
Once we’re logged into the PiKVM shell using direct network access, we can stop and disable other VPN services. For example, to stop Tailscale:
rw
sudo systemctl stop tailscale
sudo systemctl disable tailscale
ro
Notice that we used rw and ro commands to remount the root filesystem in read-write and read-only modes in order to make changes.
Next we’ll create a Netbird setup for our PiKVM from the Netbird admin console.
We are using a setup key because we want to automatically assign groups to the PiKVM and also because we don’t want the device key on PiKVM to expire. Note that setup key expiration has nothing to do with the device key expiration. Setup key can and should expire as soon as possible.
Install Netbird
First we need to use the PiKVM shell via SSH or via browser terminal to remount the root filesystem in read-write mode:
rw
Then we follow the standard Netbird installation instructions for Arch Linux:
curl -fsSL https://pkgs.netbird.io/install.sh | sh
For our first connect (netbird up), we’ll keep the root filesystem in read-write mode to allow Netbird to create necessary files:
netbird up -k SETUP-KEY --allow-server-ssh --enable-lazy-connection --no-browser --log-file syslog
And once connected we’ll cleanly shut down Netbird because we need it to be in a clean state before we set up the tmpfs:
netbird down
systemctl stop netbird
While we’re still in read-write mode, we need to clean up a bit:
rm /var/lib/netbird/state.json
rm /var/lib/netbird/*.log
ls -lah /var/lib/netbird/
In case you notice any files other than ‘active_profile.json’ and ‘default.json’, remove them as well.
Setup tmpfs for Netbird
Now we need to create a helper script which sets up overlay mount for /var/lib/netbird so that Netbird can have write access during runtime without modifying the read-only root filesystem.
Save this script as /usr/local/bin/setup-netbird-overlay.sh
#!/bin/bash
set -e
# Make tmpfs for netbird overlay
mkdir -p /tmp/netbird-tmpfs
mountpoint -q /tmp/netbird-tmpfs || mount -t tmpfs tmpfs /tmp/netbird-tmpfs
# Prepare overlay dirs
mkdir -p /tmp/netbird-tmpfs/upper
mkdir -p /tmp/netbird-tmpfs/work
mkdir -p /tmp/netbird-merged
# Mount overlay (lowerdir = persistent readonly state in /root)
mountpoint -q /tmp/netbird-merged || mount -t overlay overlay \
-o lowerdir=/root/netbird-state,upperdir=/tmp/netbird-tmpfs/upper,workdir=/tmp/netbird-tmpfs/work \
/tmp/netbird-merged
# Bind merged to /var/lib/netbird
mountpoint -q /var/lib/netbird && umount /var/lib/netbird || true
mount --bind /tmp/netbird-merged /var/lib/netbird
We also need to create a systemd service to trigger our helper script at boot.
Create a file /etc/systemd/system/netbird-overlay.service with the following content:
[Unit]
Description=Setup overlayfs for netbird
After=local-fs.target tmp.mount
Before=netbird.service
[Service]
Type=oneshot
ExecStart=/usr/local/bin/setup-netbird-overlay.sh
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
And we need to enable our new service so that it runs at boot:
systemctl enable netbird-overlay.service
I also had to modify the Netbird service file to remove dependency on hardcoded log paths that don’t exist on PiKVM.
To do that, we use systemctl edit to create an override file for the Netbird service:
systemctl edit --full netbird.service
This will open a text editor where we can add our custom configuration. This is what my service file looks like after the modification:
[Unit]
Description=NetBird mesh network client
ConditionFileIsExecutable=/usr/bin/netbird
After=network.target syslog.target
[Service]
StartLimitInterval=5
StartLimitBurst=10
#ExecStart=/usr/bin/netbird "service" "run" "--log-level" "info" "--daemon-addr" "unix:///var/run/netbird.sock" "--log-file" "/var/log/netbird/client.log"
ExecStart=
ExecStart=/usr/bin/netbird "service" "run" "--log-level" "info" "--daemon-addr" "unix:///var/run/netbird.sock" "--log-file" "syslog"
#StandardOutput=file:/var/log/netbird/netbird.out
StandardOutput=
StandardOutput=journal
#StandardError=file:/var/log/netbird/netbird.err
StandardError=
StandardError=journal
Restart=always
RestartSec=120
EnvironmentFile=-/etc/sysconfig/netbird
Environment=SYSTEMD_UNIT=netbird
[Install]
WantedBy=multi-user.target
All done, now we can remount the root filesystem back to read-only mode:
ro
To verify that everything is working correctly, we can manually start the overlay service and then start Netbird:
sudo systemctl start netbird-overlay.service
sudo systemctl start netbird
netbird status -d
If everything looks good, we can reboot the PiKVM to ensure that the overlay service starts automatically at boot and that Netbird comes up correctly:
reboot
Known Issues
Changes not preserved across reboots
Since we are using a tmpfs for Netbird’s writable data, any changes made to files in /var/lib/netbird or using netbird up command will not persist across reboots. Also, Netbird won’t be able to rotate device key so it’s really important that we disable key expiration (already done by using setup key).
If we need to preserve configuration changes across reboots, we’ll need to manually modify files in /root/netbird-state.
Netbird can’t edit resolv.conf
- Ensure that systemd-resolved is enabled using
systemctl enable --now systemd-resolved - Uninstall openresolv using
rw; pacman -R openresolv ;ro - Delete
/root/netbird-state/resolv.conf