Not having access to a GPU from the last decade, I’ve instead been running these models via Ollama’s CPU-only support. It has been interesting to see how recent library improvements have made gpt-oss much more useable when operating in this way.
Here’s the hardware I’m using:
$ neofetch kernel cpu memory
kernel: 6.16.4-arch1-1
cpu: Intel Xeon E5-2698 v4 (40) @ 3.600GHz
memory: 15181MiB / 64208MiB
This is a Broadwell-EP CPU from early 2016, paired with quad-channel DDR4-2400. Memory bandwidth is king for LLM inference, and this hardware has a theoretical maximum of a little under 80GB/s.
I used the following prompt against the lib/ source of a small Ruby gem, which resulted in 1.01k input tokens being fed to the model:
MODEL="gpt-oss:20b"
GEM_CODE=$(
git ls-files -- lib \
| xargs -I {} bash -c 'echo -e "{} contains:\n$(cat {})"'
)
echo -e "offer a refactoring suggestion for this ruby gem:\n\n$(GEM_CODE)" \
| ollama run $MODEL --verbose -
A combination of Ollama enabling flash attention for CPU-only prompt processing, as well as efficiency improvements in the handling of the way gpt-oss model weights are stored resulted in a big performance improvement between 0.11.4 and 0.11.8:
| Model | Ollama Version | Prompt Eval Rate (tokens/s) | Generation Rate (tokens/s) |
|---|---|---|---|
| gpt-oss:20b | 0.11.4 | 10.17 | 6.16 |
| gpt-oss:20b | 0.11.8 | 57.50 | 10.16 |
| qwen3-coder:latest | 0.11.8 | 78.19 | 13.55 |
| qwen3:0.6b | 0.11.8 | 404.92 | 42.81 |
I’ve included a couple of extra models in there for comparison, but the gpt-oss improvements were ~50% token generation, and 5x prompt evaluation.
btop was showing some odd stats for a ZFS dataset; it was showing as 100% free, but with an unexpectedly small total capacity:
├─main-pool──────────────65,6─GiB─┤
│ IO% ........................... │
│ Used: 0% ■ 128 KiB │
│ Free:100% ■■■■■■■■■■■■ 65,6 GiB │
├─────────────────────────────────┤
Checking with ZFS, this total capacity actually appeared to be the remaining space for the dataset:
$ zfs list main-pool
NAME USED AVAIL REFER MOUNTPOINT
main-pool 365G 65.7G 128K /main-pool
At first, I guessed there might be some sort of off-by-one error in btop’s reading of the ZFS statistics, so I had a poke around in the source. While IO stats are read from the ZFS-provided interface /proc/spl/kstat/zfs/main-pool/objset-*, capacity stats are retrieved from a statvfs system call.
So in fact, I was able to reproduce btop’s numbers, both via a python script:
>>> from os import statvfs
>>> stats = statvfs("/main-pool")
>>> "{:.1f}GB".format(stats.f_blocks * stats.f_bsize / (2**30))
'65.7GB'
…and via df:
$ df -h /main-pool
Filesystem Size Used Avail Use% Mounted on
main-pool 66G 128K 66G 1% /main-pool
I then realised that I had nested datasets that weren’t mounted, but were still consuming pool capacity. If I temporarily mounted them, things began to make sense:
├─main-pool─────────────────────────────65,6─GiB─┤
│ IO% .......................................... │
│ Used: 0% ■ 128 KiB │
│ Free:100% ■■■■■■■■■■■■■■■■■■■■■■■■■■■ 65,6 GiB │
│ │
├─time-machine───────────────────────────418─GiB─┤
│ IO% .......................................... │
│ Used: 84% ■■■■■■■■■■■■■■■■■■■■■■ 353 GiB │
│ Free: 16% ■■■■■■■■■■■■■■■■■■■■■■■■■■■ 65,6 GiB │
│ │
├────────────────────────────────────────────────┤
So it turns out ZFS exposes each mounted dataset individually, and reports the pool’s available space as free against each. Mystery solved!
]]>PKGBUILD recipe. To install a package, you first check out the repository and review it, then use makepkg to install dependencies, download/verify the source material, and build it. You can then install the built package directly with pacman -U.
This process starts to get unwieldy once you’re using lots of inter-related packages from the AUR. At this point, people often start using an AUR helper like yay, which abstracts away much of the manual work that’s normally needed when using the AUR.
However, if you’re trying to be as selective as possible when using the AUR, an AUR helper might be overkill; a slightly more primitive solution would be sufficient.
It’s possible to set up a local repository, and configure pacman to use it. I do this in /srv/packages (a location that’s readily accessible to pacman’s alpm download user).
The first step is to configure makepkg to set up the local repo as the destination for built packages:
grep -B 1 ^PKGDEST /etc/makepkg.conf
#-- Destination: specify a fixed directory where all packages will be placed
PKGDEST=/srv/packages
We then need to compile a repository database file that indexes all these packages:
cd /srv/packages
repo-add --new local-aur.db.tar.zst *.pkg.tar.zst
This needs to be re-run each time a package is [re-]built with makepkg.
We can add the following to /etc/pacman.conf so it will synchronize packages from our local repo (as well as core and extra).
[local-aur]
SigLevel = Optional TrustAll
Server = file:///srv/packages
We can now search for and install packages from our local repository just like we would with any other! 🎉
As AUR packages get updated, you have to manually update your git checkouts. I do this with a script to iterate over all my local versions, which I keep in ~/AUR:
fd -td -d1 -x bash -c "echo -en '{}\n ' && git -C {} pull" \; . ~/AUR
I can then review any changes, and run makepkg / repo-add / pacman -Syu to install.
send/recv to efficiently back up the dataset to a separate machine, transmitting just the deltas between snapshots. Syncoid is a tool to help with this, bundled as part of sanoid. If used with something like tailscale, it can make offsite backups a breeze!
We’ll want a few ancillary optional packages installed (pv, mbuffer, lzop) to improve the experience. The main thing we’ll want to get configured is non-root users on the source (user: syncoid-sender) and destination (user: syncoid-receiver) machines with sufficient delegated ZFS permissions to transfer the snapshots.
We’ll allow the sending user to send data from the desired dataset(s), as well as to be able to place and release ZFS holds; this allows us to prevent snapshots that are being used by the process from being removed prematurely.
sudo zfs allow syncoid-sender hold,send,release main-pool/test-dataset
Note that send does permit already-decrypted data to be sent. At the original time of writing, it was not possible to grant only the use of send --raw. However a proposal to add such a grant has subsequently been accepted and released as part of ZFS 2.4.0, meaning send:raw can be used instead.
On the receiving end, we’ll want to configure a dataset in which to receive the backups:
sudo zfs create -o readonly=on main-pool/backup
…then allow our local user the ability to zfs recv into new datasets within it:
sudo zfs allow syncoid-receiver create,mount,receive,hold,release main-pool/backup
To start, we’ll push the baseline of the dataset over SSH in order to establish a backup dataset on the receiving machine. Once SSH keys have been set up, we can run:
syncoid \
--sendoptions=raw \
--no-privilege-elevation \
--no-sync-snap \
--no-rollback \
--use-hold \
main-pool/test-dataset \
syncoid-receiver@receiving-machine:main-pool/backup/test-dataset
Breaking down these options:
--sendoptions=raw sends still-encrypted data. We don’t need to load the decryption keys on the destination machine.--no-privilege-elevation prevents the use of sudo, as we’ve granted the necessary delegated permissions at both ends.--no-sync-snap avoids the creation of extra syncoid-specific snapshots; the existing snapshot from the preexisting sanoid policy will suffice.--no-rollback stops the backup dataset from being rolled back if snapshots go missing from the original dataset.--use-hold ensures that a ZFS hold is placed on the latest snapshots needed for deltas to be producible, preventing their removal (which would then require a entire re-sync).If we want, we can temporarily load the decryption key and mount the backup dataset to explore it:
zfs mount main-pool/backup/test-dataset
# cannot mount 'main-pool/backup/test-dataset': encryption key not loaded
sudo zfs load-key main-pool/backup/test-dataset
sudo zfs mount main-pool/backup/test-dataset
# [...] verify things are as expected
sudo zfs unmount main-pool/backup/test-dataset
sudo zfs unload-key main-pool/backup/test-dataset
We’ll pull data from the sending machine to the receiving machine as the syncoid-receiver user over SSH. With the appropriate SSH keys installed on the sending machine, we can incrementally pull any updates to our dataset with:
syncoid \
--sendoptions=raw \
--no-privilege-elevation \
--no-sync-snap \
--no-rollback \
--use-hold \
syncoid-sender@sending-machine:main-pool/test-dataset \
main-pool/backup/test-dataset
The snapshot history from the source dataset will be preserved in the backup too, but we’ll want to do similar housekeeping with sanoid to ensure they don’t build up too much. We can even choose to keep an eye on the age of the latest snapshots present as a way of validating that our backups are continuing to take place.
Sample backup server sanoid configuration:
# /etc/sanoid/sanoid.conf
[main-pool/backup/test-dataset]
frequently = 0
hourly = 24
daily = 5
monthly = 2
yearly = 0
# Retain snapshots according to ^:
autoprune = yes
# Don't take any new snapshots:
autosnap = no
# If using nagios or similar, this has
# sanoid --monitor-snapshots get worried
# after a few days of missing dailies:
monitor = yes
daily_warn = 48h
daily_crit = 60h
The final thing to do is to ensure that sanoid is run periodically to perform this maintenance:
sudo systemctl enable --now sanoid.timer
It’s possible that a DKMS build may fail, or that the kernel being provided by Arch’s rolling release model gets ahead and is temporarily unsupported by ZFS. For this reason, we’ll install multiple kernels so we can (hopefully!) have one as a fallback should something go wrong.
We’ll use both the standard linux kernel package, as well as linux-lts. We’ll need to install the *-headers packages for both too, so we can build the ZFS modules against them.
# Some of these will already be installed
sudo pacman --needed -S linux linux-headers linux-lts linux-lts-headers dkms mkinitcpio git
Packages for ZFS are available in the Arch User Repository (AUR), but these are just git repositories containing PKGBUILD recipes; you clone them and build yourself using the pacman-provided makepkg. You can layer additional tools like yay to automate more of the use of the AUR, but we’ll stick with the basics for now.
For ZFS, we’ll need two packages - the zfs-dkms kernel modules and the zfs-utils utilities.
git clone https://aur.archlinux.org/zfs-dkms.git
git clone https://aur.archlinux.org/zfs-utils.git
Within each checkout, we run makepkg -si to [s]ync any missing dependencies from the official repositories, download and build, then temporarily elevate to root in order to [i]nstall.
At this point, a number of pacman hooks (in /usr/share/libalpm/hooks) will come in to play - the dkms-install hook will detect that the zfs-dkms package has placed kernel modules into /usr/lib/modules/, and will therefore build them for each of the installed kernels.
A following mkinitcpio-install pacman hook will then create a new initramfs for each kernel, ensuring the new modules are available during early stage boot. In fact, mkinitcpio will by default also create a second -fallback.img initramfs for each kernel; this skips the autodetect mkinitcpio hook that would normally prune modules deemed unnecessary for the hardware, leading to a larger but hopefully more-compatible environment.
In order for our LTS kernel to give us a chance to easily recover into a bootable system should we have any issues with the latest linux package, we’ll need to ensure it’s selectable in the bootloader. I’m using systemd-boot with the EFI System Partition mounted at /boot, so we need to create entries as /boot/loader/entries/*.conf. As an example, here’s the config for booting with the LTS kernel and its corresponding pared-down initramfs:
# /boot/loader/entries/linux-lts.conf
title Arch Linux (LTS kernel)
linux /vmlinuz-linux-lts
initrd /initramfs-linux-lts.img
options root=PARTUUID=xxxx-xxxx-xxxx rw
We can then use bootctl list to check they’re all showing up, and expect to see entries like:
type: Boot Loader Specification Type #1 (.conf)
title: Arch Linux (LTS kernel)
id: arch-lts.conf
source: /boot//loader/entries/arch-lts.conf (on the EFI System Partition)
linux: /boot//vmlinuz-linux-lts
initrd: /boot//initramfs-linux-lts.img
options: root=PARTUUID=xxxx-xxxx-xxxx rw
At this point, we should be able to reboot into our ZFS-enabled kernel!
]]>In trying to update to Ubuntu 24.10, I was getting just a black screen (sometimes with a cursor) whenever trying to start gdm, followed by hangs / long pauses before a different virtual terminal could be switched to. A pre-release build of 25.04 exhibited the same symptoms, as did a very pared down Arch + GNOME install. No end of kernel flags were tried without success, and with no smoking gun in any logs I could find I became resigned to the idea that my GT210 had finally become obsolete.
However, I stumbled across a suggestion that the issue might be with Mutter, and found this minimal addition to /etc/environment fixed everything:
# cat /etc/environment
# Hail Mary to try and get GT210 working with GNOME:
GSK_RENDERER=gl
MUTTER_DEBUG_DISABLE_TRIPLE_BUFFERING=1
MUTTER_DEBUG_USE_KMS_MODIFIERS=0
# pacman -Q linux mesa gdm mutter
linux 6.13.5.arch1-1
mesa 1:24.3.4-1
gdm 47.0-2
mutter 47.6-1
The GT210 and its 1080p maximum resolution lives on, hurray!
]]>keylocation=prompt an interactive solution will be required, but as I’m using a keyfile held outside of ZFS an automatic solution is possible.
Conceptually, there are three things that need to happen:
# The system needs to learn of the key's location:
zfs get keylocation main-pool/test-dataset
# Once available, this key needs to be loaded in to ZFS:
zfs load-key main-pool/test-dataset
# Once loaded, the requiring dataset can be mounted:
zfs mount main-pool/test-dataset
It turns out that this is a solved problem, and a solution is achievable using a combination of the ZFS event daemon and a systemd generator.
OpenZFS bundles a system generator that gets installed at /usr/lib/systemd/system-generators/zfs-mount-generator. This runs very early in the boot process (before units are loaded) and has an opportunity to generate further units dynamically.
It is too early in the boot process to interrogate ZFS directly for mounts, so instead the mount generator is driven by a cached version of zfs list output. This cache is maintained by the ZEDLET history_event-zfs-list-cacher, which is in turn invoked by the ZFS Event Daemon zed.service when datasets are modified. From the source of this ZEDLET, we can see it will keep a cached version of ZFS datasets’ info written to /etc/zfs/zfs-list.cache/<POOL_NAME>, as long as the location exists.
This location doesn’t exist by default, but we can ensure it does:
sudo mkdir -p /etc/zfs/zfs-list.cache
sudo touch /etc/zfs/zfs-list.cache/main-pool
Once we’ve triggered some cacheable activity (e.g. tweaking a dataset property), this cache should be written to by the ZEDLET.
With a populated cache in place, all we should have to do is reboot! Once the system comes back up, we should find new systemd units that have been generated as a result; a mount unit bound to a key loading service.
If we look at the new mount unit for our encrypted dataset (with systemctl cat srv-test\\x2ddataset.mount) we can see that it now binds to a service for loading the dataset’s key:
# /run/systemd/generator/srv-test\x2ddataset.mount
# Automatically generated by zfs-mount-generator
[Unit]
SourcePath=/etc/zfs/zfs-list.cache/main-pool
Documentation=man:zfs-mount-generator(8)
Before=zfs-mount.service local-fs.target
After= zfs-load-key-main\x2dpool-test\x2ddataset.service
Wants=
BindsTo=zfs-load-key-main\x2dpool-test\x2ddataset.service
[Mount]
Where=/srv/test-dataset
What=main-pool/test-dataset
Type=zfs
Options=defaults,atime,relatime,dev,exec,rw,suid,nomand,zfsutil
Taking a look at that service’s definition, we can see it waits for the keylocation to be mounted, then runs zfs load-key for our dataset if it still needs it.
# /run/systemd/generator/zfs-load-key-main\x2dpool-test\x2ddataset.service
# Automatically generated by zfs-mount-generator
[Unit]
Description=Load ZFS key for main-pool/test-dataset
SourcePath=/etc/zfs/zfs-list.cache/main-pool
Documentation=man:zfs-mount-generator(8)
DefaultDependencies=no
Wants=
After=
RequiresMountsFor='/root/main-pool-test-dataset.key'
[Service]
Type=oneshot
RemainAfterExit=yes
# This avoids a dependency loop involving systemd-journald.socket if this
# dataset is a parent of the root filesystem.
StandardOutput=null
StandardError=null
ExecStart=/bin/sh -c 'set -eu;keystatus="$$(/sbin/zfs get -H -o value keystatus "main-pool/test-dataset")";[ "$$keystatus" = "unavailable" ] || exit 0;/sbin/zfs load-key "main-pool/test-dataset"'
ExecStop=/bin/sh -c 'set -eu;keystatus="$$(/sbin/zfs get -H -o value keystatus "main-pool/test-dataset")";[ "$$keystatus" = "available" ] || exit 0;/sbin/zfs unload-key "main-pool/test-dataset"'
All being well, these units will have done their job and the encrypted dataset should be mounted already:
# findmnt --mountpoint /srv/test-dataset
TARGET SOURCE FSTYPE OPTIONS
/srv/test-dataset main-pool/test-dataset zfs rw,relatime,xattr,noacl,casesensitive
This in turn should have triggered any dependent downstream services (like Samba) to start too.
]]>Luckily, these scenarios can be prevented by associating the SMB systemd .service unit with the .mount unit that’s created as a result of the ZFS setup.
We can use sudo systemctl edit smbd.service and then specify via an override that:
time-machine dataset is mounted (using After=), andBindsTo=)[Unit]
# Serve only when the ZFS mount is available
BindsTo=srv-time\x2dmachine.mount
After=srv-time\x2dmachine.mount
If we wanted to, we could go a step further and use sudo systemctl edit srv-time\\x2dmachine.mount to add a Wants=smbd.service declaration back in the other direction; this would attempt to automatically start the SMB daemon should the time-machine dataset get (re)mounted.
# Here's our existing dataset in the "main-pool" pool, without encryption:
sudo zfs create main-pool/test-dataset
So we’re working on something consistent, we can either unmount the dataset, or take a snapshot. We’ll do both; the former preventing further changes, and the latter so we can use the full --replicate option later on:
sudo zfs unmount main-pool/test-dataset
sudo zfs snapshot main-pool/test-dataset@encryption-time
We then need a key, which we’ll choose to store on the host filesystem (rather than require it to be provided interactively when mounting):
sudo openssl rand -hex -out /root/main-pool-test-dataset.key 32
It may be prudent to keep a copy of this somewhere safe, too.
We’ll now use ZFS’s send/receive to create a new dataset with encryption enabled:
sudo zfs send --replicate --verbose \
main-pool/test-dataset@encryption-time \
| sudo zfs receive \
-s \
-o encryption=on \
-o keylocation=file:///root/main-pool-test-dataset.key \
-o keyformat=hex \
main-pool/encrypted-test-dataset
Depending on the size of the source dataset and capabilities of the hardware, it may may take a long time to create the encrypted version of the dataset. This process may get interrupted; if so, the -s flag given to zfs receive will have caused a “progress” token to be saved against the new dataset. We can extract that, and resume the zfs send from that point:
# extract the resume token from the target dataset:
token=$(zfs get -o value -H receive_resume_token main-pool/encrypted-test-dataset)
# restart sending the source dataset from that point:
sudo zfs send --verbose -t $token \
| sudo zfs receive -s main-pool/encrypted-test-dataset
Once happy, we can rename the new encrypted version in place of the original:
sudo zfs rename main-pool/test-dataset main-pool/unencrypted-test-dataset
sudo zfs rename main-pool/encrypted-test-dataset main-pool/test-dataset
If desired, we can then destroy the unencrypted version of the dataset to release the capacity back to the pool:
sudo zfs destroy -r main-pool/unencrypted-test-dataset
In order make this a secure erase, we can either:
sudo zpool initialize -w main-pool to zero over the freed space, orsudo zpool trim --secure -w main-pool (hardware support allowing).I’ve written separately on automatically mounting encrypted dataset on boot, which is possible here due to the use of a file as the dataset’s keylocation.
Let’s address those one at a time.
This is something I’ve covered before, but in summary we have to include and prioritise a BOOT_ORDER value of 2 in the config stored in the RPi’s EEPROM, which will instruct it to try and source a kernel over the network via TFTP (Trivial File Transfer Protocol). While updating the EEPROM, we’ll also set TFTP_PREFIX=2 - more on that later.
For other hardware, you may find configuration in your BIOS that allows similar.
While it is possible to hard-code in the EEPROM config where the bootloader should go to find the TFTP server, I prefer to advertise this as part of the DHCP offer - but watch out for potential issues if you start doing this for your whole network!
I use ISC’s DHCP server on the server that is also acting as gateway for the network, and will additionally be taking on the role of TFTP server (and iSCSI server). Therefore, to the subnet configuration in /etc/dhcp/dhcpd.conf it is a simple matter of adding:
range 192.168.20.10 192.168.20.200;
+ option tftp-server-name "192.168.20.1";
option routers 192.168.20.1;
It’s worth noting that DHCP also supports advertising a “next server” for clients to contact, but the RPi firmware doesn’t support that, so I stuck with the BOOTP extension option 66.
We can then test this using one of nmap’s built-in scripts:
# sudo nmap --script broadcast-dhcp-discover
| Response 1 of 1:
| IP Offered: 192.168.20.26
| DHCP Message Type: DHCPOFFER
| Server Identifier: 192.168.20.1
| [ ... ]
|_ TFTP Server Name: 192.168.20.1
Now the RPi will know where to look for its boot image, we now need to provision something for it to find! A TFTP server can be installed with:
sudo apt-get install tftpd-hpa
It is configured in /etc/default/tftpd-hpa; the defaults may be fine for you, but I configured the following starting settings:
TFTP_DIRECTORY="/srv/tftp"
TFTP_ADDRESS="192.168.20.1:69"
The TFTP_PREFIX=2 we set earlier in the RPi’s EEPROM will mean that it requests files it needs prefixed with its MAC address (easy to find) rather than its serial number (less easy to find).
I’ll be using ZFS to store both the block device to be served over iSCSI as well as the boot files to be served over TFTP. Setting up individual datasets will allow for snapshotting and rollback.
# Set up a dataset for the client with MAC address xx-xx-xx-xx-xx-xx:
CLIENT_DATASET="main-pool/netboot/xx-xx-xx-xx-xx-xx"
sudo zfs create -p $CLIENT_DATASET
sudo zfs set mountpoint=none main-pool/netboot
# Create a child zvol block device for iSCSI at '/vol':
sudo zfs create -V 64G $CLIENT_DATASET/vol
# Fill it with a disk image:
sudo dd \
if=ubuntu-24.04.1-preinstalled-server-arm64+raspi.img \
of=/dev/zvol/$CLIENT_DATASET/vol \
status=progress
# Create a child dataset for TFTP at '/tftp':
sudo zfs create -o mountpoint=/srv/tftp/xx-xx-xx-xx-xx-xx $CLIENT_DATASET/tftp
# Scan for partitions in the new block device:
sudo fdisk -l /dev/zvol/$CLIENT_DATASET/vol
# Temporarily mount the first (boot) partition, and copy the contents to the TFTP directory:
sudo mount /dev/zvol/$CLIENT_DATASET/vol-part1 /mnt
sudo cp -r /mnt/* /srv/tftp/xx-xx-xx-xx-xx-xx/
sudo umount /mnt
# Finally, recursively snapshot this pristine state:
sudo zfs snapshot -r $CLIENT_DATASET@pristine-24.04
To observe things working, we can monitor DHCP logs and sniff for traffic on :69 (in my case, shown here on the vlan20 interface) when a client RPi tries to netboot:
# a DHCPREQUEST should arrive as per normal:
journalctl -u isc-dhcp-server.service -b -f
# then, boot files should be requested via TFTP:
sudo tcpdump -vv -i vlan20 port 69
At this point, our client RPi should be able to get a kernel started, but then panic as it won’t be able to find a root filesystem. For Ubuntu 24.04, the kernel cmdline.txt that we will have delivered over TFTP will have instructed it to look for a filesystem labelled as writeable to use as the root:
... root=LABEL=writable rootfstype=ext4 rootwait ...
This can still work, but we’ll need the kernel to act as an iSCSI initiator and fetch the block device over the network first before the relevant partition can be found. We can do that by editing cmdline.txt to include the following before root=... (again, substituting in the client’s MAC address):
iscsi_initiator=iqn.2025-02.dev.pencheon:cluster:xx-xx-xx-xx-xx-xx iscsi_target_name=iqn.2025-02.dev.pencheon:root:xx-xx-xx-xx-xx-xx iscsi_target_ip=192.168.20.1
This will look for a iSCSI target called iqn.2025-02.dev.pencheon:root:xx-xx-xx-xx-xx-xx on the 192.168.20.1 host, self-identifying as iqn.2025-02.dev.pencheon:cluster:xx-xx-xx-xx-xx-xx.
In order to serve this, we’ll need to set up the iSCSI target, with the following configuration in /etc/tgt/conf.d/xx-xx-xx-xx-xx-xx.conf:
<target iqn.2025-02.dev.pencheon:root:xx-xx-xx-xx-xx-xx>
initiator-name iqn.2025-02.dev.pencheon:cluster:xx-xx-xx-xx-xx-xx
backing-store /dev/zvol/main-pool/netboot/xx-xx-xx-xx-xx-xx/vol
</target>
All being well, the client should now fetch the initial boot image over TFTP, start the kernel, attach the block device over iSCSI, and continue booting:
[ 13.451314] Loading iSCSI transport class v2.0-870.
[ 13.462700] iscsi: registered transport (tcp)
[ 14.474254] scsi host0: iSCSI Initiator over TCP/IP
[ 14.483862] scsi 0:0:0:0: RAID IET Controller 0001 PQ: 0 ANSI: 5
[ 14.495861] scsi 0:0:0:0: Attached scsi generic sg0 type 12
[ 14.502170] scsi 0:0:0:1: Direct-Access IET VIRTUAL-DISK 0001 PQ: 0 ANSI: 5
[ 14.514185] sd 0:0:0:1: Attached scsi generic sg1 type 0
[ 14.514307] sd 0:0:0:1: Power-on or device reset occurred
[ 14.525439] sd 0:0:0:1: [sda] 134217728 512-byte logical blocks: (68.7 GB/64.0 GiB)
[ 14.533353] sd 0:0:0:1: [sda] Write Protect is off
[ 14.538571] sd 0:0:0:1: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 14.550184] sda: sda1 sda2
[ 14.553106] sd 0:0:0:1: [sda] Attached SCSI disk
Once logged in, we should see the two partitions on this device, labelled and mounted:
lsblk -o name,size,mountpoints,label
NAME SIZE MOUNTPOINTS LABEL
loop0 33.7M /snap/snapd/21761
sda 64G
├─sda1 512M /boot/firmware system-boot
└─sda2 63.5G / writable
So far, so good! But there’s one glaring issue that becomes apparent when a client tries to update its boot information at /boot/firmware. The bootloader is now getting this via TFTP, and being served an frozen extracted copy, rather than what it might have updated on /dev/sda1. This makes running system updates tricky!
If we were using NFS to serve the entire root filesystem, this would be easy enough to fix; we could bind mount the boot/firmware directory on the server to also be served via TFTP. Whilst this is technically circumventing the multi-access that the NFS server would manage, tftp-hpa is read-only by default, and the two access modes shouldn’t be required at the same time.
From the perspective of the server, the block device being used for the iSCSI target is nothing more than that; it’s not mounted or readable by the server, nor should it be - unless the filesystem contained within is suitable for concurrent access (e.g. glusterfs), even a filesystem mounted a second time read-only won’t work. A filesystem being mounted read-only is not the same as the underlying block device being read-only (ext4 writes to its journal regardless of the mount flag), and regardless of that a non-clustered filesystem mount would cache reads and thus go stale.
NFS to the rescue again - but we’ll instead use it to serve just the TFTP version of the boot partition back to the client, so it can re-mount it. With nfs-kernel-server installed, we can configure it:
# cat /etc/exports
/srv/tftp/xx-xx-xx-xx-xx-xx 192.168.20.0/24(rw,sync,no_subtree_check,no_root_squash)
…and then apply the config with sudo exportfs -a.
Because boot files need to be owned by root and sometimes not even world-readable, we unfortunately need the no_root_squash option on the mount, and to punch some holes in the TFTP server’s configuration too:
# in /etc/default/tftpd-hpa
# allow serving of files accessible only to root by running as root:
TFTP_USERNAME="root"
# 'secure' tries to sandbox access to just the TFTP directory
# 'permissive' skips extra checks that require world-readable files
TFTP_OPTIONS="--secure --permissive"
On the client, we’ll need to install nfs-common, then update /etc/fstab:
LABEL=writable / ext4 defaults 0 1
- LABEL=system-boot /boot/firmware vfat defaults 0 1
+ 192.168.20.1:/srv/tftp/xx-xx-xx-xx-xx-xx /boot/firmware nfs defaults 0 1
…running sudo mount -a to apply.
It is a shame we have to manually install a package outside the base install on the client to get this working. Such provisioning is something that can be achieved with cloud-init, but that’s for another day.
At this point, you might like to shut down the client, and take another set of snapshots with ZFS:
sudo zfs snapshot -r $CLIENT_DATASET@after-initial-config
In order to return to this state, we can use ZFS to roll back. Note that when rolling back, -r is used to remove any more recent snapshots than the targeted one, rather than meaning “recursive”. Rollbacks have to be done to each child individually:
sudo zfs rollback -r $CLIENT_DATASET@after-initial-config
sudo zfs rollback -r $CLIENT_DATASET/vol@after-initial-config
sudo zfs rollback -r $CLIENT_DATASET/tftp@after-initial-config