![Picture of Raspberry Pi 4 box and Raspberry Pi 4 board below box](https://www.stephenwagner.com/wp-content/uploads/2020/03/Blog-Pi4Icon-150x150.jpg)
If you’re worried about destroying your SD Cards, need some more space, or just want to learn something new, I’m going to show you how to use an NFS root for the Raspberry Pi 4.
When you use an NFS Root with your Raspberry Pi, it stores the entire root filesystem on a remote NFS export (think of it as a network filesystem share). This means you’ll have as much space as the NFS export, and you’ll probably see way faster performance since it’ll be running at 1Gb/sec instead of the speed of the SD Card.
This also protects your SD card, as the majority of the reading and writing is performed on the physical storage of the NFS export, instead of the SD card in the Pi which has limited reads and writes.
What you’ll need
To get started, you’ll need:
- Raspberry Pi 4
- Ubuntu or Raspbian for Raspberry Pi 4 Image
- A small SD card for the Boot Partition (1-2GB)
- SD card for the Raspberry Pi Linux image
- Access to another Linux system (workstation, or a Raspberry Pi)
There are multiple ways to do this, but I’m providing instructions on the easiest way it was for me to do this with the resources I had immediately available.
Instructions
To boot your Raspberry Pi 4 from an NFS root, multiple steps are involved. Below you’ll find the summary, and further down you’ll find the full instructions. You can click on an item below to go directly to the section.
The process:
- Write the Linux image to an SD Card
- Create boot SD Card for NFS Root
- Prep the Linux install for NFS Root
- Create the NFS Export
- Copy the Linux install to the NFS Export
- Copy and Modify the boot SD Card to use NFS Root
- Boot using SD Card and test NFS Root
See below for the individual instructions for each step.
Write the Linux image to an SD Card
First, we need to write the SD Card Linux image to your SD card. You’ll need to know which device your SD card will appear to your computer. In my case it was /dev/sdb, make sure you verify the right device or you could damage your current Linux install.
- Download Ubuntu or Raspbian for Raspberry Pi.
- unzip or unxz depending on distribution to uncompress the image file.
- Write the SD card image to SD card.
dd if=imagename.img of=/dev/sdb bs=4M
You now have an SD Card Linux install for your Raspberry Pi. We will later modify and then copy this to the NFS root and boot SD card.
Create boot SD Card for NFS Root
In this step, we’re going to create a bootable SD card that contains the Linux kernel and other needed files for the Raspberry Pi to boot.
This card will be installed in the Pi, load the kernel, and then kick off the boot process to load the NFS root.
I previously created a post to create a boot partition layout for a Raspberry Pi. Please follow those instructions to complete this step.
Later on in this guide, you’ll be copying the boot partition from the SD Card Linux image, on to this newly created boot SD Card for the NFS Root.
Prep the Linux install for NFS Root
There’s a few things we have to do to prep the Ubuntu or Raspbian Linux install to be usable as an NFS Root.
- Boot the Raspbian or Ubuntu SD Card you create in the first step on your Raspberry Pi.
- Complete the first boot procedures. Create your account, and complete the setup.
- Enable and confirm SSH is working so you can troubleshoot.
- Install the NFS client files using the following command:
apt install nfs-common
- Open the /etc/network/interfaces file, and add the following line so that the Pi only get’s an IP once during boot:
iface eth0 inet manual
- Modify your /etc/fstab entries to reflect the NFS root and the new boot SD card as per below.
For step 6, we need to modify the /etc/fstab entry for the root fs. It is different depending on whether you’re using Ubuntu or Raspbian.
For Raspbian, your /etc/fstab should look like this:
proc /proc proc defaults 0 0 LABEL=boot /boot vfat defaults 0 2 NFS-SERVER-IP:/nfs-export/PI-Raspbian / nfs defaults 0 0
For Ubuntu, your /etc/fstab should look like this:
LABEL=system-boot /boot/firmware vfat defaults 0 2 /dev/nfs / nfs defaults 0 0
After you do this, the Linux SD image may not boot again if directly installed in the Raspberry Pi, so make sure you’ve made the proper modifications before powering it down.
Create the NFS Export
In my case I used a Synology DS1813+ as an NFS server to host my Raspberry Pi NFS root images. But you can use any Linux server to host it.
If you’re using a synology disk station, create a shared folder, disable the recycling bin, leave everything else default. Head over to the “NFS Permissions” tab and create an ACL entry for your PI and workstations. You can also add a network segment for your entire network (ex. 192.168.0.0/24″) instead of specifying individual IPs.
Once you create an entry, it’ll look like this. Note the “Mount path” in the lower part of the window.
Now, if you’re using a standard Linux server the steps are different.
- Install the require NFS packages:
apt install nfs-kernel-server
- Create a directory, we’ll call it “nfs-export” on your root fs on the server:
mkdir /nfs-export/
- Then create a directory for the Raspberry Pi NFS Root:
mkdir /nfs-export/PI-ImageName
- Now edit your /etc/exports file and add this line to the file to export the path:
/nfs-export/PI-ImageName IPorNetworkRange(rw,no_root_squash,async,insecure)
- Reload the NFS exports to take affect:
exportfs -ra
Take note of the mount point and/or NFS export path, as this is the directory your Raspberry Pi will need to mount to access it’s NFS root. This is also the directory you will be copying your SD Card Linux install root FS to.
Copy the Linux install to the NFS Export
When you’re ready to copy your SD Card Linux install to your NFS Export, you’ll need to do the following. In my case I’ll be using an Ubuntu desktop computer to perform these steps.
When I insert the SD Card containing the Raspberry Pi Linux image, it appeared as /dev/sdb on my system. Please make sure you are using your proper device names to avoid using the wrong one to avoid writing or using the wrong disk.
Instructions to copy the root fs from the SD card to the NFS root export:
- Mount the root partition of the SD Card Linux install to a directory. In my case I used directory called “old”.
mount /dev/sdb2 old/
- Mount the NFS Export for the NFS Root to a directory. In my case I used a directory called “nfs”.
mount IPADDRESS:/nfs-export/PI-ImageName nfs/
- Use the rsync command to transfer the SD card Linux install to the NFS Root Export.
rsync -avxHAXS --numeric-ids --info=progress2 --progress old/ nfs/
- Unmount the directories.
umount old/
umount nfs/
Once this is complete, your OS root is now copied to the NFS root.
Copy and Modify the boot SD Card to use NFS Root
First we have to copy the boot partition from the SD Card Linux install to the boot SD card, then we need to modify the contents of the new boot SD card.
Top copy the boot files, follow these instructions.
- Mount the boot partition of the SD Card Linux install to a directory. In my case I used directory called “old”.
mount /dev/sdb1 old/
- Mount the new boot partition of the boot SD card to a new directory. In my case I used the directory called “new”.
mount /dev/sdc1 new/
- Use the rsync command to transfer the SD card Linux install boot partition to the new boot SD card.
rsync -avxHAXS --numeric-ids --info=progress2 --progress old/ new/
- Unmount the directories.
umount old/
umount new/
Now there are few steps we have to take to make to the boot SD card boot to an NFS Root.
We have to make a modification to the PI boot command. It is different depending on which Linux image (Ubuntu or Raspbian) you’re using.
First, insert the boot SD card, and mount it to a temporary directory.
mount /dev/sdc1 new/
If you’re running Ubuntu, your existing nobtcmd.txt should look like this:
dwc_otg.lpm_enable=0 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait
We’ll modify and replace some text to make it look like this. Don’t forget to change the command to reflect your IP and directory:
dwc_otg.lpm_enable=0 console=tty1 root=/dev/nfs nfsroot=IPADDRESS:/nfs-export/PI-Ubuntu,tcp,rw ip=dhcp rootfstype=nfs elevator=deadline rootwait
For Raspbian, your existing cmdline.txt should look like this:
console=serial0,115200 console=tty1 root=PARTUUID=97709164-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait
We’ll modify and replace some text to make it look like this. Don’t forget to change the command to reflect your IP and directory:
console=serial0,115200 console=tty1 root=/dev/nfs nfsroot=IPADDRESS:/nfs-export/PI-Raspbian,tcp rw vers=3 ip=dhcp rootfstype=nfs elevator=deadline rootwait
Once you make the modifications, save the file and unmount the SD card.
Your SD card is now ready to boot.
Boot using SD Card and test NFS Root
At this point, insert the boot SD Card in your Raspberry Pi and attempt to boot. All should be working now and it should boot and use the NFS root!
If you’re having issues, if the boot process stalls, or something doesn’t work right, look back and confirm you followed all the steps above properly.
You’re done!
You’re now complete and have a fully working NFS root for your Raspberry Pi. You’ll no longer worry about storage, have high speed access to it, and you’ll have some new skills!
And don’t forget to check out these Handy Tips, Tricks, and Commands for the Raspberry Pi 4!
[…] was creating a new Micro SD card with the purpose of using an NFS Root for the Raspberry Pi. For those of you that don’t know, you can boot a Raspberry Pi (or Linux computer) from local […]
Thanks for your information.
I can boot ‘nfs root filesystem’ w/ my Raspberry Pi 3B+.
I will try to do no-SD card boot using tftp next.
Best Regards,
Hi Hisaaki,
Glad to hear it works! Feel free to post your findings once you try the TFTP boot! 🙂
Cheers,
Stephen
Hi,
trying this on a RPi 2, I found that the cmdline for raspbian needs to be modified:
the vers=3 need to be appended as an nfsroot= option like tcp, like this:
nfsroot=:/path/to/nfs/root,tcp,vers=3
Otherwise, the kernel panics (unable to mount root).
The rest works fine, thanks for the writeup.
Hi, was able to set up NFS root this way, Pi boots, however apt update fails with following message:
pi@raspberrypi:~ $ sudo apt-get update
Get:1 http://archive.raspberrypi.org/debian buster InRelease [32.8 kB]
Get:2 http://raspbian.raspberrypi.org/raspbian buster InRelease [15.0 kB]
Reading package lists… Done
E: Unable to determine file size for fd 9 – fstat (5: Input/output error)
E: Problem closing the file – close (116: Stale file handle)
Hi Adam,
It sounds like there may be a permission issue.
This is either due to the NFS server, an incorrect rootfs copy, or incorrect mount.
I’d check your config and try again.
Cheers
Hi Stephen,
Indeed, I use QNAP and raspberry was mounting NFS version 1 by default. I solved it with adding vers=3 at the end of nfsroot parameter, hence
nfsroot=IP_ADDRESS:/path,tcp,vers=3
Thanks for the quide and help.
Cheers!
Thank You! I’ve learned few things today. Now I understand that it’s all about root folder being moved to NFS share and the boot remaing on the SD.
Your solution is more convinient for those like me who cannot install dnsmasq/tftp on the NFS server.
The only thing I need right now is to find cheap SDs
Hi Mr. Stephen, First of all, thanks for your time and effort. This is quite an amazing tutorial for newbies like me 🙂
Although the tutorial is detailed sufficiently, I’ve faced some issues while trying to apply it on my Raspberry Pi 4. Here are some points I want to add:
When I edit the /etc/exports file and apply the changes via
exportfs -ra
command, an error occurs. The output is as follows:The solution I found is to replace the appended line with the following:
This way, I was able to run the
exportfs -ra
command without any problem.I also used restarted the nfs-kernel-server using the command below.The cmdline.txt file which includes initial kernel arguments needs to be changed a bit. I couldn’t achieve to bring up my board until I change the nfsroot parameter as follows
I guess that the problem was about the
nfsvers
thing. You may want to enlighten us about it.I also added nfsrootdebug option to kernel parameter to further inspect the debug information about the process.
Thanks again for your time and effort. The tutorial was absolutely beneficial!
Greetings from Turkey, Caglayan
Hi Caglayan DOKME,
Thanks for posting and leaving the comment! I’m sure it will help someone else. It’s normal to have to tweak the version numbers around depending on what distribution is being used, the NFS version, etc.
Glad you got it working!
Cheers,
Stephen
How did you configure Linux ACLs via NFS? I am not referring to the shared folder permissions that you are referring to as an ACL, I am referring to POSIX-compliant ACLs that allow the kernel to write to /var/log/journal, or use tools like dmesg. I get errors like:
>dmesg: read kernel buffer failed: Operation not permitted
or:
>”Failed to set ACL on /var/log/journal/UUID/user-1000.journal, ignoring: Operation not supported.”
This guide seems not to acknowledge the following issues regarding Linux ACLs, and Synology’s ACL setup (As of July 2022, but seem to pre-date this article)
* rsync being broken in Ubuntu/Debian for this very issue: https://unix.stackexchange.com/questions/12203/rsync-failed-to-set-permissions-on-error-with-rsync-a-or-p-option
* Synology not supporting Linux ACLs over NFS (v3/v4/v4.1):
https://community.synology.com/enu/forum/1/post/150225
https://community.synology.com/enu/forum/1/post/149472
The rpi4 boots to the root fs via NFS, but without logging to the journal, or having random sys utils not being able to run, it’s too unsafe to run any type of workload over NFS. I’d love to understand what you did here, because I feel like there’s a gap in documentation from this article.
Update Wrong stack link to rsync issue, correct one is here: https://serverfault.com/questions/1059618/rsync-generator-failed-to-set-permissions-operation-not-supported-95
Hi Chris,
The NFS export has to be squashed as root. This allows the connecting system attempting an NFS root to give full access to the kernel, etc… From that point on, the users only have access to what they would normally.
Cheers,
Stephen
Hi Stephen
The synology screenshot in the tutorial shows no mapping for squash, is that an error? Nevertheless, the synology shared folders do not have “squash as root” as an option, rather to map root to admin or guest. In all cases, I still can’t get proper ACL support in the filesystem. I suspect others are hitting this issue, and are unaware of it, as the boot errors regarding ACLs are consistent during init.
I was able to loopback mount an img file I created with some success, but there is some very strange behavior in which (I believe) the NFS page cache refuses to acknowledge multiple discrete mounts, causing (in my case, 6 raspberry pis) to all route to the same NFS mount, even though each pi is reporting that it is using separate, unique NFS mounts, both in each Pi’s fstab, and cmdline.txt. This also results in incredibly poor performance and is generally not recommended (see: https://www.suse.com/support/kb/doc/?id=000017812)
At this point, I’m ready to give up on this endeavor, sadly. I spent a couple of months working/resolving corner cases with Unifi equipment to try PXE/TFTP booting; the boot partition + NFS root filesystem was my backup. Looks like my choices are to roll my own NFS server (no thanks), or USB mount an SSD or two, which is orthogonal to the whole concept + benefit of networked filesystems.
Appreciate the help,
Christian
I agree with Chris about NSFv4 ACLs, the nfs4-acl-tools may work on other Linux systems, but as far as I’ve experimented with them, not on Raspberry Pi OS built off Debian Bullseye. If you have a share with either a Raspberry Pi server, or something else (In my experiments a Solaris 11.4 machine), the tools can’t change the NSFv4 ACLs on the client. I don’t fully understand what you’re doing here by putting the OS on an NFS server, I mean it looks and sounds great, but are you forcing the OS to be of file system type NFS, rather than EXT4? I couldn’t get the nfs4-acl-tools to work when the server was an NFS file system (Solaris 11.4). Perhaps many people are not that interested in levying NSFv4 ACLs on their files and directories, I don’t know. But if you needed big iron servers, maybe that would be important to you. I’m only toying around on a couple three Pi’s.