FreeBSD update 14.0

In this article, we will explain how to migrate a FreeBSD 13.2 system to 14.0. The main features to highlight in version 14 include improvements in NVMe disk management, Wireguard reimplementation, and enhancements in BHyve graphics pass-through, among others. For more details, we can refer to the release notes at this link: https://www.freebsd.org/releases/14.0R/relnotes/ . It is worth noting that this time I have waited about 4 months since its release to avoid potential launch bugs.

The migration will be carried out differently depending on whether it is a physical system or a virtualized jail. Additionally, there are variations depending on the boot system used. On the other hand, we will also explain some aspects to consider if we use ZFS or Poudriere.

Operating system
Loader update: EFI
Loader update: BIOS
ZFS update
Bastille update
IOCage update
IOCage - Template
Poudriere
Poudriere - clean
MTA

Operating system

Update the kernel and core tools to the latest version within 13.2:

freebsd-update fetch
freebsd-update install

We do the same with the binary packages:

pkg upgrade
pkg autoremove

We check the current version.

freebsd-version

13.2-RELEASE-p10

We upgrade to 14.0:

freebsd-update upgrade -r 14.0-RELEASE

Some files may require manual intervention by the user, such as the /etc/group file, for example.

<<<<<<< current version
# $FreeBSD$
#
wheel:*:0:root,kr0m
=======
wheel:*:0:root
>>>>>>> 14.0-RELEASE

We install the kernel and kernel modules.

freebsd-update install

shutdown -r now

We install userspace/binaries/libraries:

freebsd-update install

We reinstall according to whether we use binary packages or ports.

pkg upgrade

git -C /usr/ports pull
cd /usr/ports
make fetchindex
for PORT in $(pkg info|awk '{print$1}'); do PORT_PATH=$(pkg info $PORT|grep Origin|awk '{print$3}') && echo PORT: $PORT - $PORT_PATH && cd /usr/ports/$PORT_PATH && export BATCH="yes" && make clean reinstall clean; done

It will detect the change in ABI and reinstall all installed packages.

We remove old libraries and files.

freebsd-update install

NOTE: If we have a custom kernel we must recompile using the releng/14.0 branch.

Reboot the OS:

shutdown -r now

We check the OS version, both the kernel in /boot/kernel/kernel, the one loaded in RAM, and the userlands.

freebsd-version -kru

14.0-RELEASE-p5 /boot/kernel/kernel
14.0-RELEASE-p5 RAM loaded kernel
14.0-RELEASE-p5 Userland version

Loader update: EFI

This step is very important if we have the root on a ZFS pool because if we update the pool and not the EFI image, the OS may fail to boot.

We locate the EFI partition (ESP):

efibootmgr -v

BootCurrent: 0002
+Boot0002* FreeBSD HD(1,GPT,9ca4da68-f165-11eb-8209-3497f636bf45,0x28,0x82000)/File(\EFI\FREEBSD\LOADER.EFI)
                      nvd0p1:/EFI/FREEBSD/LOADER.EFI (null)

Mount the partition:

mount_msdosfs /dev/nvd0p1 /boot/efi

We verify that it has been mounted and that the filesystem is correct.

df -Th /boot/efi

Filesystem   Type       Size    Used   Avail Capacity  Mounted on
/dev/nvd0p1  msdosfs    260M    1.8M    258M     1%    /boot/efi

We can view the configured EFI boot image.

ls -la /boot/efi/efi/freebsd/loader.efi

-rwxr-xr-x  1 root  wheel  896000 Jul 30  2021 /boot/efi/efi/freebsd/loader.efi

We can also view the default EFI boot image if no entry is configured.

ls -la /boot/efi/efi/boot/bootx64.efi

-rwxr-xr-x  1 root  wheel  896000 Jul 30  2021 /boot/efi/efi/boot/bootx64.efi

We need to update both versions.

cp /boot/loader.efi /boot/efi/efi/freebsd/loader.efi
cp /boot/loader.efi /boot/efi/efi/boot/bootx64.efi

We restart to ensure that the EFI image loads without issues.

shutdown -r now

Loader update: BIOS

This step is very important if we have the root on a ZFS pool because if we update the pool and not the EFI image, the OS may fail to boot.

We locate the boot partition:

gpart show

=>        40  1953525088  ada0  GPT  (932G)
          40        1024     1  freebsd-boot  (512K)
        1064         984        - free -  (492K)
        2048     4194304     2  freebsd-swap  (2.0G)
     4196352  1949327360     3  freebsd-zfs  (930G)
  1953523712        1416        - free -  (708K)

We dump the contents of the file /boot/pmbr to the MBR (first 512 bytes) of the disk and the contents of the file /boot/gptzfsboot onto the boot partition.

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0

ZFS update

When the ZFS code is updated, we need to update the system pools to enable the new functionalities.

zpool status

  pool: zroot
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
	The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
	the pool may no longer be accessible by software that does not support
	the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 00:04:24 with 0 errors on Sun Mar 17 09:04:24 2024
config:

	NAME        STATE     READ WRITE CKSUM
	zroot       ONLINE       0     0     0
	  nda0p2    ONLINE       0     0     0

errors: No known data errors

Update the pool:

zpool upgrade zroot

This system supports ZFS pool feature flags.

Enabled the following features on 'zroot':
  edonr
  zilsaxattr
  head_errlog
  blake3
  block_cloning
  vdev_zaps_v2

Pool 'zroot' has the bootfs property set, you might need to update
the boot code. See gptzfsboot(8) and loader.efi(8) for details.

NOTE: As we can see, the command itself warns us that we have the zroot with the bootfs property enabled, which can cause problems if we do not update the loader according to our EFI / BIOS

zpool status

  pool: zroot
 state: ONLINE
  scan: scrub repaired 0B in 00:04:24 with 0 errors on Sun Mar 17 09:04:24 2024
config:

	NAME        STATE     READ WRITE CKSUM
	zroot       ONLINE       0     0     0
	  nda0p2    ONLINE       0     0     0

errors: No known data errors

Bastille update

The base system can be updated within the same version (e.g., 13.2-p1 -> 13.2-p2), within the same major version (e.g., 13.2 -> 13.3), or to a different major version (e.g., 13.2 -> 14.0).

The first step is to obtain the new version of FreeBSD. To do this, we bootstrap version 14.0 and apply the latest security patches.

bastille bootstrap 14.0-RELEASE update

We check the available releases.

bastille list release

13.1-RELEASE
13.2-RELEASE
14.0-RELEASE

Once we finish, we can delete the old releases using:

bastille destroy 13.2-RELEASE

NOTE: If any jail depends on binaries compiled from Poudriere, we must prepare Poudriere beforehand .

Thin jail:
If it’s a Thin jail, they share the base, so updating the RELEASE from Bastille will update the base system of all jails that share that RELEASE.

To determine if a jail is Thick or Thin, you can check the file /usr/local/bastille/jails/JAILNAME/jail.conf. If the line “osrelease” appears, it’s a Thin jail.

for JAIL_PATH in /usr/local/bastille/jails/*; do JAIL=$(basename $JAIL_PATH) && echo -n "-- $JAIL: " && grep osrelease $JAIL_PATH/jail.conf >/dev/null 2>/dev/null ; if [ $? -eq 0 ]; then echo "Thin jail" ; else echo "Thick jail"; fi ; done

-- Atlas: Thick jail
-- DataDyne: Thick jail
-- HellStorm: Thick jail
-- MetaCortex: Thick jail
-- PostgreSQL00-test: Thick jail
-- PostgreSQL01-test: Thick jail
-- PostgreSQLBackups-test: Thick jail
-- PostgreSQLRestore-test: Thick jail
-- RECLog: Thick jail
-- thinJail: Thin jail
-- thinJail2: Thin jail

Stop the jail:

bastille stop JAILNAME

We upgrade to version 14 as indicated in the official documentation :

bastille edit JAILNAME fstab

Change path from 13.2 to 14.0:

/usr/local/bastille/releases/13.2-RELEASE /usr/local/bastille/jails/JAILNAME/root/.bastille nullfs ro 0 0
/usr/local/bastille/releases/14.0-RELEASE /usr/local/bastille/jails/JAILNAME/root/.bastille nullfs ro 0 0

Boot the jail:

bastille start JAILNAME

We reinstall all software depending on whether we use binary packages or ports.

bastille console JAILNAME

If we see the following error message, we must change the shell to the default one before accessing the jail:

>ld-elf.so.1: Shared object "libncursesw.so.8" not found, required by "bash"

bastille cmd JAILNAME chsh -s /bin/sh

pkg-static upgrade -f
exit

git -C /usr/ports pull
cd /usr/ports
make fetchindex
for PORT in $(pkg info|awk \'{print$1}\'); do PORT_PATH=$(pkg info $PORT|grep Origin|awk \'{print$3}\') && echo PORT: $PORT - $PORT_PATH && cd /usr/ports/$PORT_PATH && export BATCH=\"yes\" && make clean reinstall clean; done
exit

We revert the shell change.

bastille cmd JAILNAME chsh -s /usr/local/bin/bash

We update in case any patches have been released since the last release.

bastille update JAILNAME

We restart the jail to ensure that everything works correctly.

bastille stop JAILNAME
bastille start JAILNAME

We can see that the thin jails are already updated to 14.0:

bastille list -a

 JID                     State  IP Address      Published Ports  Hostname                Release           Path
 Atlas                   Up     192.168.69.19   -                Atlas                   13.2-RELEASE-p10  /usr/local/bastille/jails/Atlas/root
 DataDyne                Up     192.168.69.25   -                DataDyne                13.2-RELEASE-p10  /usr/local/bastille/jails/DataDyne/root
 HellStorm               Up     192.168.69.17   -                HellStorm               13.2-RELEASE-p10  /usr/local/bastille/jails/HellStorm/root
 MetaCortex              Up     192.168.69.20   -                MetaCortex              13.2-RELEASE-p10  /usr/local/bastille/jails/MetaCortex/root
 PostgreSQL00-test       Up     192.168.69.26   -                PostgreSQL00-test       13.2-RELEASE-p10  /usr/local/bastille/jails/PostgreSQL00-test/root
 PostgreSQL01-test       Up     192.168.69.27   -                PostgreSQL01-test       13.2-RELEASE-p10  /usr/local/bastille/jails/PostgreSQL01-test/root
 PostgreSQLBackups-test  Up     192.168.69.29   -                PostgreSQLBackups-test  13.2-RELEASE-p10  /usr/local/bastille/jails/PostgreSQLBackups-test/root
 PostgreSQLRestore-test  Up     192.168.69.28   -                PostgreSQLRestore-test  13.2-RELEASE-p10  /usr/local/bastille/jails/PostgreSQLRestore-test/root
 RECLog                  Up     192.168.69.21   -                RECLog                  13.2-RELEASE-p10  /usr/local/bastille/jails/RECLog/root
 thinJail                Up     192.168.40.190  -                thinJail                14.0-RELEASE-p5   /usr/local/bastille/jails/thinJail/root
 thinJail2               Up     192.168.40.191  -                thinJail2               14.0-RELEASE-p5   /usr/local/bastille/jails/thinJail2/root

Thick jail:
On the contrary, if it’s a Thick jail, we’ll have to update each jail independently, but always from Bastille since jails start in securelevel 2 and don’t allow updates from within.

sysctl kern.securelevel

kern.securelevel: 2

To determine if a jail is Thick or Thin, you can check the file /usr/local/bastille/jails/JAILNAME/jail.conf. If the line “osrelease” appears, it’s a Thin jail.

-- Atlas: Thick jail
-- DataDyne: Thick jail
-- HellStorm: Thick jail
-- MetaCortex: Thick jail
-- PostgreSQL00-test: Thick jail
-- PostgreSQL01-test: Thick jail
-- PostgreSQLBackups-test: Thick jail
-- PostgreSQLRestore-test: Thick jail
-- RECLog: Thick jail
-- thinJail: Thin jail
-- thinJail2: Thin jail

To update the Thick jails, first, we need to know the current release version.

bastille list -a

 JID                     State  IP Address      Published Ports  Hostname                Release           Path
 Atlas                   Up     192.168.69.19   -                Atlas                   13.2-RELEASE-p10  /usr/local/bastille/jails/Atlas/root
 DataDyne                Up     192.168.69.25   -                DataDyne                13.2-RELEASE-p10  /usr/local/bastille/jails/DataDyne/root
 HellStorm               Up     192.168.69.17   -                HellStorm               13.2-RELEASE-p10  /usr/local/bastille/jails/HellStorm/root
 MetaCortex              Up     192.168.69.20   -                MetaCortex              13.2-RELEASE-p10  /usr/local/bastille/jails/MetaCortex/root
 PostgreSQL00-test       Up     192.168.69.26   -                PostgreSQL00-test       13.2-RELEASE-p10  /usr/local/bastille/jails/PostgreSQL00-test/root
 PostgreSQL01-test       Up     192.168.69.27   -                PostgreSQL01-test       13.2-RELEASE-p10  /usr/local/bastille/jails/PostgreSQL01-test/root
 PostgreSQLBackups-test  Up     192.168.69.29   -                PostgreSQLBackups-test  13.2-RELEASE-p10  /usr/local/bastille/jails/PostgreSQLBackups-test/root
 PostgreSQLRestore-test  Up     192.168.69.28   -                PostgreSQLRestore-test  13.2-RELEASE-p10  /usr/local/bastille/jails/PostgreSQLRestore-test/root
 RECLog                  Up     192.168.69.21   -                RECLog                  13.2-RELEASE-p10  /usr/local/bastille/jails/RECLog/root
 thinJail                Up     192.168.40.190  -                thinJail                14.0-RELEASE-p5   /usr/local/bastille/jails/thinJail/root
 thinJail2               Up     192.168.40.191  -                thinJail2               14.0-RELEASE-p5   /usr/local/bastille/jails/thinJail2/root

As of today (18/03/2024), only updating Thin jails is documented in the documentation. The procedure for updating a Thick jail has been deduced from the Bastille man pages and the command’s help.

man bastille
bastille upgrade
Usage: bastille upgrade release newrelease | target newrelease | target install | [force]

Update the jail:

bastille upgrade JAILNAME 14.0-RELEASE

Apply the update:

bastille upgrade JAILNAME install

Reboot the jail:

bastille stop JAILNAME
bastille start JAILNAME

Continue with the update process:

bastille upgrade JAILNAME install

We reinstall all software depending on whether we use binary packages or ports.

bastille console JAILNAME

If we see the following error message, we must change the shell to the default one before accessing the jail:

>ld-elf.so.1: Shared object "libncursesw.so.8" not found, required by "bash"

bastille cmd JAILNAME chsh -s /bin/sh

pkg-static upgrade -f
exit

git -C /usr/ports pull
cd /usr/ports
make fetchindex
for PORT in $(pkg info|awk \'{print$1}\'); do PORT_PATH=$(pkg info $PORT|grep Origin|awk \'{print$3}\') && echo PORT: $PORT - $PORT_PATH && cd /usr/ports/$PORT_PATH && export BATCH=\"yes\" && make clean reinstall clean; done
exit

We revert the shell change.

bastille cmd JAILNAME chsh -s /usr/local/bin/bash

We finish the update.

bastille upgrade JAILNAME install

We update in case any patches have been released since the last release.

bastille update JAILNAME

We restart the jail to ensure that everything works correctly.

bastille stop JAILNAME
bastille start JAILNAME

We can see all the jails updated.

bastille list -a

 JID                     State  IP Address      Published Ports  Hostname                Release          Path
 Atlas                   Up     192.168.69.19   -                Atlas                   14.0-RELEASE-p5  /usr/local/bastille/jails/Atlas/root
 DataDyne                Up     192.168.69.25   -                DataDyne                14.0-RELEASE-p5  /usr/local/bastille/jails/DataDyne/root
 HellStorm               Up     192.168.69.17   -                HellStorm               14.0-RELEASE-p5  /usr/local/bastille/jails/HellStorm/root
 MetaCortex              Up     192.168.69.20   -                MetaCortex              14.0-RELEASE-p5  /usr/local/bastille/jails/MetaCortex/root
 PostgreSQL00-test       Up     192.168.69.26   -                PostgreSQL00-test       14.0-RELEASE-p5  /usr/local/bastille/jails/PostgreSQL00-test/r
 PostgreSQL01-test       Up     192.168.69.27   -                PostgreSQL01-test       14.0-RELEASE-p5  /usr/local/bastille/jails/PostgreSQL01-test/r
 PostgreSQLBackups-test  Up     192.168.69.29   -                PostgreSQLBackups-test  14.0-RELEASE-p5  /usr/local/bastille/jails/PostgreSQLBackups-t
 PostgreSQLRestore-test  Up     192.168.69.28   -                PostgreSQLRestore-test  14.0-RELEASE-p5  /usr/local/bastille/jails/PostgreSQLRestore-t
 RECLog                  Up     192.168.69.21   -                RECLog                  14.0-RELEASE-p5  /usr/local/bastille/jails/RECLog/root
 thinJail                Up     192.168.40.190  -                thinJail                14.0-RELEASE-p5  /usr/local/bastille/jails/thinJail/root
 thinJail2               Up     192.168.40.191  -                thinJail2               14.0-RELEASE-p5  /usr/local/bastille/jails/thinJail2/root

IOCAGE update

First, we update the parent to 14.0 following the steps outlined earlier in this article.

The next step is to update both the OS and the packages to their latest version in each of the jails.

iocage update JAILNAME
iocage console JAILNAME
pkg upgrade
pkg autoremove
exit

Now we can update them to 14.0; we download the RELEASE from Iocage.

iocage fetch

[0] 12.4-RELEASE
[1] 13.2-RELEASE
[2] 13.3-RELEASE
[3] 14.0-RELEASE

Type the number of the desired RELEASE
Press [Enter] to fetch the default selection: (14.0-RELEASE)
Type EXIT to quit: 3
Fetching: 14.0-RELEASE

Downloading: MANIFEST [####################] 100%
Downloading: base.txz [####################] 100%
Downloading: lib32.txz [####################] 100%
Downloading: src.txz [####################] 100%
Extracting: base.txz...
Extracting: lib32.txz...
Extracting: src.txz...

* Updating 14.0-RELEASE to the latest patch level...

We update each of the jails:

iocage upgrade JAILNAME -r 14.0-RELEASE

We reinstall all software depending on whether we use binary packages or ports.

iocage console JAILNAME

If we see the following error message, we must change the shell to the default one before accessing the jail:

>ld-elf.so.1: Shared object "libncursesw.so.8" not found, required by "bash"

iocage exec JAILNAME "chsh -s /bin/sh"

pkg-static upgrade -f
exit

git -C /usr/ports pull
cd /usr/ports
make fetchindex
for PORT in $(pkg info|awk '{print$1}'); do PORT_PATH=$(pkg info $PORT|grep Origin|awk '{print$3}') && echo PORT: $PORT - $PORT_PATH && cd /usr/ports/$PORT_PATH && export BATCH="yes" && make clean reinstall clean; done
exit

We revert the shell change.

iocage exec $JAILNAME "chsh -s /usr/local/bin/bash"

We update in case any patches have been released since the last release.

iocage update JAILNAME

We restart the jail to ensure that everything works correctly.

iocage stop JAILNAME
iocage start JAILNAME

We check the version of the updated jail.

iocage list

+-----+--------------------+-------+--------------+---------------+
| JID |        NAME        | STATE |   RELEASE    |      IP4      |
+=====+====================+=======+==============+===============+
+-----+--------------------+-------+--------------+---------------+
| 3   | haproxy            | up    | 14.0-RELEASE | 192.168.1.202 |
+-----+--------------------+-------+--------------+---------------+
| 4   | web00              | up    | 13.2-RELEASE | 192.168.1.203 |
+-----+--------------------+-------+--------------+---------------+

NOTE: Updates using Iocage automatically create a snapshot with each update, so if anything goes wrong, it can be reverted easily. However, we should keep in mind that snapshots consume disk space.

Iocage - Template

To update templates, we first need to convert them to jails, update them, and then convert them back to templates.

iocage list -t

+-----+----------------+-------+--------------+---------------+
| JID |      NAME      | STATE |   RELEASE    |      IP4      |
+=====+================+=======+==============+===============+
| -   | basic_template | down  | 13.2-RELEASE | 192.168.1.201 |
+-----+----------------+-------+--------------+---------------+

iocage set template=no JAILNAME
iocage start JAILNAME

Update exactly as if they were regular jails.

We convert it back to a template.

iocage set template=yes JAILNAME

iocage list -t

+-----+----------------+-------+--------------+---------------+
| JID |      NAME      | STATE |   RELEASE    |      IP4      |
+=====+================+=======+==============+===============+
| -   | basic_template | down  | 14.0-RELEASE | 192.168.1.201 |
+-----+----------------+-------+--------------+---------------+

Poudriere

We download the new RELEASE version:

poudriere jail -c -j freebsd_14-0x64 -v 14.0-RELEASE

poudriere jail -l

JAILNAME        VERSION          ARCH  METHOD TIMESTAMP           PATH
freebsd_13-2x64 13.2-RELEASE-p10 amd64 http   2024-03-17 05:25:03 /usr/local/poudriere/jails/freebsd_13-2x64
freebsd_14-0x64 14.0-RELEASE-p5  amd64 http   2024-03-18 16:46:00 /usr/local/poudriere/jails/freebsd_14-0x64

We migrate the generic compilation options:

cp /usr/local/etc/poudriere.d/freebsd_13-2x64-make.conf /usr/local/etc/poudriere.d/freebsd_14-0x64-make.conf

We migrate the specific package options:

cp -r /usr/local/etc/poudriere.d/freebsd_13-2x64-latest-options /usr/local/etc/poudriere.d/freebsd_14-0x64-latest-options

We update the jail and the ports tree:

poudriere jail -u -j freebsd_14-0x64
poudriere ports -u -p latest

Compile the ports:

poudriere bulk -j freebsd_14-0x64 -p latest -f /usr/local/etc/poudriere.d/port-list

The client part would look like this:

vi /usr/local/etc/pkg/repos/poudriere.conf

poudriere: {
    url: "http://poudriere.alfaexploit.com/packages/freebsd_14-0x64-latest/",
    mirror_type: "http",
    signature_type: "pubkey",
    pubkey: "/usr/local/etc/ssl/certs/poudriere.cert",
    enabled: yes,
}

We recompile all packages:

pkg-static upgrade -f

Poudriere - clean

Delete the jail:

poudriere jail -d -j freebsd_13-2x64

We remove the source code of the packages:

poudriere distclean -a -p latest

We delete the logs:

poudriere logclean -a -j freebsd_13-2x64 -p latest

We remove some residual directories:

rm -rf /usr/local/poudriere/data/packages/freebsd_13-2x64-latest/
rm -rf /usr/local/etc/poudriere.d/freebsd_13-2x64-latest-options/
rm /usr/local/etc/poudriere.d/freebsd_13-2x64-make.conf

We remove the old version of the data.json file:

vi /usr/local/poudriere/data/logs/bulk/.data.json

MTA:

The default MTA has changed now is DragonFly Mail Agent, but if we want to continue with our good friend Sendmail you only have to copy the appropiate configuration file /etc/mail/mailer.conf:

The default mail transport agent (MTA) is now the Dragonfly Mail Agent (dma(8)) rather than sendmail(8). Configuration of the MTA is done via mailer.conf(5). sendmail(8) and its configuration remain available. a67b925ff3e5

cd /etc/mail
cp /usr/share/examples/sendmail/mailer.conf /etc/mail/mailer.conf
make
cp HellStorm.cf sendmail.cf
service sendmail restart