Home    Articles    Notes       Projects    About    
PXE booting a Raspberry Pi 4

PXE booting a Raspberry Pi 4

October 26th, 2020

When I originally ordered the first few Pi’s for the rack, I realized I had forgotten to get SD cards along with them. Not to worry though, because there was another solution: PXE. PXE is a technology which allows us to boot a machine over the network. In principle, the process is simple: the machine on startup utlizes some sort of low-level firmware to attempt to boot from various media, usually an HDD or SSD. In our case, this would be the SD card. If one isn’t found, then we send a request to the network for a bootloader. If we have a machine on the network listening for such requests, we can send it the necessary files.

Although I got PXE boot working a few weeks ago, it was not easy. I spent a lot of time reading various tutorials and documentation trying to solve the issues I was having. If you want to PXE boot your Raspberry Pi’s too, and are having similar struggles, read on—hopefully this post will have some useful information for you.

The last point of note: the information here is written for a Raspberry Pi 4, specifically because that is what I have. If you have an earlier version, you’ll most likely have to find another resource. Sorry!

Requirements

If you want to do this, you’ll need the following:

* At least one Raspberry Pi 4

* A network switch, or available ports on your router—PXE uses ethernet

* A Linux machine to use as the server

* At least one microSD card, \(\geq16\) GB

* A display for the Pi

Preparing the Raspberry Pi

Download the latest version of Raspberry Pi OS, and write it to the SD card with your utility of choice (I use dd). Boot the Raspberry Pi with the card as you would normally. Go ahead and log in, we are going to make a few changes.

First, do a full upgrade:

# apt-get update
# apt-get upgrade

as is good practice. This will save us a little bit of annoyance later, especially if you have a slow internet connection like I do.

Next, we will modify the bootloader. To do so, we first copy a template binary, convert back to text, make our modifications, rewrite to a binary, and then stage the bootloader to be flashed on the next reboot. Copy the binary (you can choose a different one if you want):

$ cp /lib/firmware/raspberrypi/bootloader/stable/pieeprom-2020-09-03.bin pieeprom.bin
$ rpi-eeprom-config pieeprom.bin > bootconf.txt

You can now edit bootconf.txt to make any changes you need. For PXE booting, this means the BOOT_ORDER variable must contain the digit 2. Usually I set this variable to f241The digits specified in this variable tell the bootloader which boot media to attempt, in which order. It is read from right to left. The options you can specify are: 0, none; 1, SD card; 2, network; 3, USB device; 4, USB mass storage; and f, continuously retry. , which means the Pi will first attempt to boot from an SD card, then from USB, and then from the network. If all of these fail, it will continuously retry in that order. In my opinion, it would be unwise to remove the digit 1 from the variable, as this would prevent booting from SD.

Once you’ve made your changes, we will use them to create a new binary:

$ rpi-eeprom-config --out new.bin --config bootconf.txt pieeprom.bin

And then flash the binary to the EEPROM:

# rpi-eeprom-update -d -f ./new.bin

This won’t take effect until the next reboot. Before we do that, we may want to consider disabling future automatic updates of this bootloader, as this could override the changes we just made. To do so:

# systemctl mask rpi-eeprom-update

Finally, now is a good time to grab the Pi’s MAC address.You can use ip link in order to do so. Go and make a DHCP reservation for this address. If you run DNS, give it an FQDN too.

Okay, we’re done! Go ahead and reboot:

# reboot

When the login prompt comes back up, you can cut power and remove the SD card.

Preparing the server

First things first, I’m the realest; and that’s why I use Arch, btw, so if you use another distribution on your server you may need to change your package names accordingly.

We will need to install dnsmasq, NFS, and kpartx:

$ yay -S dnsmasq multipath-tools-git nfs-utils

Now we need to start building the filesystem. How exactly you choose to do this is up to you. Because of the way the filesystem on my server is structured, my setup involves a lot of bindmounts and linking, so I’ll just give an example structure here. Let’s make a place to store the Pi’s root filesystem and boot files:

# mkdir -p /srv/pxe/root/DEFAULT
# mkdir -p /srv/pxe/boot/DEFAULT

As you can guess, these are just to store the default Pi OS image. We will make a new subdirectory in each of these locations for each additional Pi we want to PXE boot. The boot subdirectory will require a special name, which we will address later. The root subdirectory name can be anything you want. I usually use the hostname, and recommend you do the same. For our example, we will assume our hostname is going to be YOUR_PI_HOSTNAME:

# mkdir /srv/pxe/root/YOUR_PI_HOSTNAME

Now, we will need to image the updated SD card you just used. Before you do so, you may want to create a temporary working directory to keep things tidy:

$ mkdir temp && cd temp

When you have your SD card connected, you can use lsblk to check its device name. It will probably be something like /dev/mmcblk0. Make sure the device isn’t mounted, and use dd to image the card:

# dd if=/dev/mmcblk0 of=image.img status=progress

Now, we make two temporary mountpoints—one for each partition in the disk image:

$ mkdir root boot

Now we use kpartx on our image to create loop devices, which we will then mount:

# kpartx -a -v image.img
# mount /dev/mapper/loop0p2 root/
# mount /dev/mapper/loop0p1 boot/

Then we will copy these files to our default locations:

# cp -a root/* /srv/pxe/root/DEFAULT/
# cp -a boot/* /srv/pxe/boot/DEFAULT/

While we are at it, you can copy the files from the default location to where we will be serving the new Pi from. You’ll have to do this each time you want to add a new Pi to PXE boot:

# cp -a /srv/pxe/root/DEFAULT/* /srv/pxe/root/YOUR_PI_HOSTNAME/

Go ahead and edit /srv/pxe/root/YOUR_PI_HOSTNAME/etc/hosts and /srv/pxe/root/YOUR_PI_HOSTNAME/etc/hostname to reflect your chosen hostname.

We will have to replace a few files from the boot partition. Change directory:

$ cd /srv/nfs/boot/DEFAULT

Delete the required files, and download their replacements. I’ve mirrored the replacements from Hexxeh/rpi-firmware, if you’d prefer to get them from there.

# rm start4.elf
# rm fixup4.dat
# wget https://git.riscj.dev/mirror/rpi-firmware/raw/branch/master/fixup4.dat
# wget https://git.riscj.dev/mirror/rpi-firmware/raw/branch/master/start4.elf

Another addition we need to make here will be to add a file titled ssh. The Pi will see this, and enable SSH:

# touch ssh

The last change we will need to make in the boot directory is to cmdline.txt. Your contents should look like the following:

console=serial0,115200 console=tty1 root=/dev/nfs nfsroot=SERVER_IP_ADDRESS:/srv/pxe/root/YOUR_PI_HOSTNAME/,vers=3,proto=tcp rw ip=dhcp rootwait elevator=deadline

It is really important that you have a DHCP reservation for your server, so that its IP does not change. I have tried to use FQDNs here, since I run DNS on my network, but it seems spotty. Feel free to try it and let me know if it works for you, though.

Now we are going to add NFS exports, so that our clients can access the server’s filesystem. To do so, place something like the following into your /etc/exports:

/srv/pxe/boot                                  *(rw,sync,no_subtree_check,no_root_squash)
/srv/pxe/root/YOUR_PI_HOSTNAME  YOUR_PI_HOSTNAME(rw,sync,no_subtree_check,no_root_squash)

If you don’t run DNS on your network, either replace the hostname with an IP address or add the IP to your /etc/hosts file.

Start the relevent services and export the filesystems:

# systemctl enable rpcbind
# systemctl start rpcbind
# systemctl enable nfs-server
# systemctl start nfs-server
# exportfs -arv

We are getting close, I promise.

The last thing to do is to setup dnsmasq. You’ll need to know your broadcast address for this. Mine is 10.0.0.255. Anywhere you see that, replace it with your own. If you do not know yours, you can find it with ip addr and looking at the brd entry. You will also need to know the name of the network interface you are using. The previous command shows you that too. Mine is enp0s25, so be sure to replace that with your own anywhere you see it. Place the following into /etc/dnsmasq.conf:

interface=enp0s25
port=0
dhcp-range=10.0.0.255,proxy
log-dhcp
enable-tftp
tftp-root=/srv/pxe/boot
pxe-service=0,"Raspberry Pi PXE"

Now, enable the service:

# systemctl enable dnsmasq
# systemctl start dnsmasq

Final steps

Almost done! Now, remember how I said earlier that the boot subdirectory will need a special name? We will handle that here. First, we are going to start tailing the dnsmasq logs:

# journalctl -u dnsmasq -f

Go over to your Pi. Make sure it has ethernet connected, and turn it on. Nothing should happen, so don’t bother waiting around. Go back to your server, and monitor the logs. Eventually, you should see a message of the form file /srv/pxe/boot/########/FILE.EXT not found. When you do, note the number I’ve marked in my example with octothorpes. Ctrl+C to stop tailing the logs, and go turn off the Pi. Then, create a new boot subdirectory with that number as its name:

# mkdir /srv/pxe/boot/########

And copy the default files there:

# cp -a /srv/pxe/boot/DEFAULT/* /srv/pxe/boot/########/

We will need to now modify /srv/pxe/root/YOUR_PI_HOSTNAME/etc/fstab, so the Pi knows to mount using NFS. It should contain something like the following:

proc                                     /proc proc defaults        0 0
SERVER_IP_ADDRESS:/srv/pxe/boot/######## /boot nfs  defaults,vers=3 0 0

Once you’re done, you can turn the Pi back on. If everything worked properly, your Pi should come up as expected. You’re done now, and can SSH into the Pi to do whatever you need to. Hurray!

I wrote all of this from memory, and it’s super possible I forgot something. If something isn’t working for you, let me know. It was a pain for me setting this up the first time, and I’d like to help try and alleviate some of that pain for others.