By default, FreeBSD provides a generic kernel image that includes a large number of drivers to support as much hardware as possible, as well as the most widely used functionalities and disabled less common ones.

Compiling the kernel will bring several benefits:

• Faster system boot, since the kernel only has to test the hardware that the system actually has.
• Lower RAM consumption since we have removed unnecessary parts.
• Smaller attack surface by removing parts of the code that will no longer be present.
• Enable support for specific hardware that is disabled in GENERIC.
• Enable exotic options that are disabled by default.

In my case, the motivation for recompiling the kernel comes from the desire to experiment with a minimal image for my home server.

The manual is composed of different sections:

• Installation of the kernel source code
• Hardware identification
• Compiling the kernel
• Updates
• Troubleshooting

Installation of the kernel source code:

Install Git:

The OS sources are stored in /usr/src, which is a ZFS dataset.

NAME                                         USED  AVAIL     REFER  MOUNTPOINT
zroot/usr/src                                827M  27.2G      827M  /usr/src

Therefore, if we want to start from scratch, we must recreate it:

Clone the repo in the /usr/src directory:

The main branch of the Git repository corresponds to CURRENT , but we want to stay on RELEASE, specifically on the same version with which the core userland tools of our system were compiled. This will prevent many strange and difficult-to-debug problems (such as some network connections failing while others don’t).

We check the version of the userlands:

13.0-RELEASE-p11

We can see the branches on this website , the most current 13.0 RELEASE is:

releng/13.0

We switch to the desired branch:

Eventually, we will want to update our repository to keep the kernel source code up to date. We can choose between two methods.

Manually:

Using freebsd-update:
We just need to make sure it is configured to update the kernel source code as well, which is enabled by default.

Components src world kernel

src indicates that both the kernel sources and the core userland tools should be updated.

Hardware identification:

The first thing we need to do is to know what hardware we have. We can find out using the following files/commands:

/var/run/dmesg.boot
dmidecode
lspci -lv
kldstat

In my case, it is an old DELL laptop that still works:

Del XPS Studio XPS 1340 - A15
BIOS System

Next, I list the hardware identified via /var/run/dmesg.boot. I don’t list all the detected hardware, only what I’m going to use. For example, I don’t use the PCM-CIA cards, the SD card reader and the sound card, so I won’t add support for them in my kernel:

CPU: Intel(R) Core(TM)2 Duo CPU     P9500  @ 2.53GHz (2533.32-MHz K8-class CPU)
ahci0: <NVIDIA MCP79 AHCI SATA controller> port 0x30e8-0x30ef,0x30dc-0x30df,0x30e0-0x30e7,0x30d8-0x30db,0x30c0-0x30cf mem 0xf0884000-0xf0885fff irq 19 at device 11.0 on pci0
ohci0: <nVidia nForce MCP79 USB Controller> mem 0xf0886000-0xf0886fff irq 20 at device 4.0 on pci0
ehci0: <NVIDIA nForce MCP79 USB 2.0 controller> mem 0xf0889000-0xf08890ff irq 21 at device 4.1 on pci0
ohci1: <nVidia nForce MCP79 USB Controller> mem 0xf0887000-0xf0887fff irq 22 at device 6.0 on pci0
nfe0: <NVIDIA nForce MCP79 Networking Adapter> port 0x30d0-0x30d7 mem 0xf0888000-0xf0888fff,0xf0889c00-0xf0889cff,0xf0889800-0xf088980f irq 18 at device 10.0 on pci0
miibus0: <MII bus> on nfe0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
ada0: <Samsung SSD 870 EVO 1TB SVT02B6Q> ACS-4 ATA SATA 3.x device
cd0: <HL-DT-ST DVD+-RW GS20N A106> Removable CD-ROM SCSI device
da0: <WD 10EZEX External 1.75> Fixed Direct Access SPC-2 SCSI device
ath0: <Atheros 9280> mem 0xf0400000-0xf040ffff irq 22 at device 0.0 on pci5

Kernel compilation:

First, we check which kernel image we currently have loaded:

FreeBSD MightyMax.alfaexploit.com 13.1-RELEASE-p2 FreeBSD 13.1-RELEASE-p2 GENERIC amd64

If we need to consult any parameter of the GENERIC kernel, we can do it in the following file:

/usr/src/sys/$(uname -m)/conf/GENERIC

Disabling kernel options only implies that such an option will not be compiled within the kernel. By default, ALL modules are always compiled, even for options included within the kernel. To avoid compiling unnecessary modules, FreeBSD supports several interesting compilation parameters:

     MODULES_OVERRIDE
		   (str) Set to	a list of modules to build instead of all of
		   them.

     WITHOUT_MODULES
		   (str) Set to	a list of modules to exclude from the build.
		   This	provides a somewhat easier way to exclude modules you
		   are certain you will	never need than	specifying
		   MODULES_OVERRIDE.  This is applied after MODULES_OVERRIDE.

In this way, the compilation time will be drastically reduced.

As a starting point, we will use the MINIMAL configuration that leaves the minimum options inside the kernel and externalizes as many as possible as modules:

/usr/src/sys/$(uname -m)/conf/MINIMAL

To have a kernel as clean as possible, we will have to follow the following steps:

• Copy MINIMAL and remove everything we don’t need from the config.
• Compile a MINIMAL kernel.
• Look at the compiled modules to add the ones for the most used hardware inside our kernel.
• Compile our kernel with the MODULES_OVERRIDE option indicating the modules for the least used hardware.

With this, we obtain a kernel with support for the most used hardware inside the kernel and some modules for the least used hardware.

One of the most complicated tasks when compiling the kernel is knowing which functionality corresponds to each option. To find out, we have several options:

Grep the name in the configs:

/usr/src/sys/amd64/conf/GENERIC:device		aac			# Adaptec FSA RAID

Consult the notes of the specific architecture:

/usr/src/sys/conf/NOTES
/usr/src/sys/amd64/conf/NOTES

Consult the online information of the handbook, man pages or the GhostBSD website:
https://docs.freebsd.org/en/books/handbook/kernelconfig/
https://www.freebsd.org/cgi/man.cgi?manpath=FreeBSD+13.1-RELEASE+and+Ports
https://wiki.ghostbsd.org/index.php/Kernel

Google:

X freebsd module

A good way to detect dependencies is by grepping the kernel source code, for example:

MODULE_DEPEND(smbfs, netsmb, NSMB_VERSION, NSMB_VERSION, NSMB_VERSION);
MODULE_DEPEND(smbfs, libiconv, 1, 1, 2);
MODULE_DEPEND(smbfs, libmchain, 1, 1, 1);

Depending on whether a functionality is widely used or not, we will add it to the kernel image or add it to the MODULES_OVERRIDE compilation variable.

Some modules cannot be compiled specifically but are part of a package, for example, TCP congestion control algorithms. They are either all compiled or none, as we can see there is only one module called cc, not cc_cdg, cc_chd, cc_cubic… the cc module includes them all:

total 33
drwxr-xr-x    9 root  wheel   10 Aug 12 04:48 .
drwxr-xr-x  442 root  wheel  444 Aug 12 04:48 ..
-rw-r--r--    1 root  wheel  432 Aug 12 04:48 Makefile
drwxr-xr-x    2 root  wheel    3 Aug 12 04:48 cc_cdg
drwxr-xr-x    2 root  wheel    3 Aug 12 04:48 cc_chd
drwxr-xr-x    2 root  wheel    3 Aug 12 04:48 cc_cubic
drwxr-xr-x    2 root  wheel    3 Aug 12 04:48 cc_dctcp
drwxr-xr-x    2 root  wheel    3 Aug 12 04:48 cc_hd
drwxr-xr-x    2 root  wheel    3 Aug 12 04:48 cc_htcp
drwxr-xr-x    2 root  wheel    3 Aug 12 04:48 cc_vegas

NOTE: The same happens with some other modules such as geom.

As long as we have the GENERIC kernel running, we can check which options were included in it:

device	aac

Or the options compiled as modules:

-r-xr-xr-x  1 root  wheel   123224 May 23 20:16 /boot/kernel/aac.ko

We proceed with the compilation of the MINIMAL kernel to generate a reduced image and a good number of modules.

We install it:

Now we can see the minimal configuration in the MINIMAL file and the modules in /boot/kernel.minimal/*.ko, now we just have to disable everything unnecessary in the MINIMAL config and decide which modules we will include in the image, which modules we will leave out, and which modules we will remove.

Before touching anything, we make a copy of the generic configuration:

And we copy the MINIMAL to our own config which we will call KR0M-MINIMAL:

lrwxr-xr-x  1 root  wheel  33 Aug 29 01:13 KR0M-MINIMAL -> /root/kernel_configs/KR0M-MINIMAL

We must keep in mind that the DEFAULTS file is automatically included when the buildkernel command is executed, so we don’t have to edit the file, but rather disable the options we don’t want in our custom config. For example, I don’t need UART(serial port).

We edit our configuration leaving it as follows:

cpu		HAMMER
ident		KR0M-MINIMAL

nodevice	uart_ns8250                 # No UART needed

device		acpi
device		atkbdc                      # AT keyboard controller
device		atkbd                       # AT keyboard
device		vt                          # Virtual terminal console driver
device		vt_vga                      # Virtual terminal console driver VGA mode
device		vt_vbefb                    # VBE framebuffer
device		loop                        # Network loopback
device		ether                       # Ethernet support
device		bpf                         # Berkeley packet filter
device		if_bridge                   # Bridge interfaces

device		acpi_wmi                    # Interface for vendor specific WMI implementations
device		ahci                        # Serial ATA Advanced Host Controller Interface driver
device		pci                         # PCI/PCIe bus driver: ahci dep
device		ata                         # Generic ATA/SATA controller driver: MCP79
device		scbus                       # Common Access Method Storage subsystem: ata/ahci dep
device 		ada                         # ATA Direct Access device driver
device 		da                          # SCSI Direct Access device driver
device 		cd                          # SCSI CD-ROM driver
device 		usb                         # Universal Serial Bus
device		ehci                        # USB Enhanced Host Controller
device		ohci                        # OHCI USB Host Controller driver
device 		ctl                         # CAM Target Layer
device 		cfumass                     # USB Mass Storage Class Transport
device 		coretemp                    # Intel Core on-die digital thermal sensor
device 		cpuctl                      # To retrieve CPUID information and perform CPU firmware updates.
device 		cpufreq                     # CPU frequency control framework
device		crypto                      # User-mode access to hardware-accelerated cryptography
device		cryptodev                   # User-mode access to hardware-accelerated cryptography
device 		firmware                    # Interface for loading firmware images into the kernel
device 		miibus                      # Media Independent Interface network bus: nForce MCP Ethernet dep
device 		nfe                         # NVIDIA nForce MCP Ethernet driver
device 		iflib                       # Network Interface	Driver Framework
device		mem                         # Interface to the physical memory of the computer
device		umass                       # USB Mass Storage Devices driver
device		pass                        # CAM application passthrough driver
device    ath			                    # Wireless network interface: Atheros IEEE 802.11 wireless network driver
device    ath_pci
device    ath_hal
device    ath_rate_sample
device    wlan
device    wlan_xauth		              # External authenticator support for 802.11 devices
device    wlan_tkip                   # TKIP and Michael crypto support for 802.11 devices
device    wlan_wep		                # WEP crypto support for 802.11 devices
device    wlan_ccmp		                # AES-CCMP crypto support for 802.11 devices
device    if_bridge                   # network bridge device
device 		epair			                  # A pair of virtual back-to-back connected Ethernet interfaces


options   VIMAGE 			                # VNET/Vimage support
options 	RACCT                       # Resource accounting framework
options 	RCTL                        # Resource limits

options 	SCHED_ULE                   # ULE scheduler
options 	PREEMPTION                  # Enable kernel thread preemption
options 	SMP                         # Symmetric MultiProcessor Kernel
options 	INET                        # InterNETworking
options 	INET6                       # IPv6 communications protocols: Iocage/Bastille dep
options 	TCP_OFFLOAD                 # TCP offload
options 	ZFS                         # Zettabyte File System
options 	MD_ROOT                     # MD is a potential root device
options 	COMPAT_FREEBSD11            # Compatible with FreeBSD11
options 	COMPAT_FREEBSD12            # Compatible with FreeBSD12
options 	SCSI_DELAY=5000             # Delay (in ms) before probing SCSI
options 	_KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions
options 	PRINTF_BUFR_SIZE=128        # Prevent printf output being interspersed.
options 	AUDIT                       # Security event auditing
options 	CAPABILITY_MODE             # Capsicum capability mode
options 	CAPABILITIES                # Capsicum capabilities
options 	INCLUDE_CONFIG_FILE         # Include this file in kernel
options 	EARLY_AP_STARTUP            # Start Application Processors earlier during boot
options 	IOMMU                       # input–output memory management unit
options 	VESA                        # Add support for VESA BIOS Extensions (VBE)
options 	MAC                         # TrustedBSD MAC Framework


options		HZ=1000                     # Set the timer granularity, recomended value by dummynet
options		DEVICE_POLLING              # Device Polling
options		SYSVMSG                     # SYSV-style message queues
options 	SYSVSEM                     # SYSV-style semaphores
options 	SYSVSHM                     # SYSV-style shared memory
options		TMPFS                       # Tmpfs file system
options 	ZSTDIO                      # zstd-compressed kernel and user dumps: ZFS dep
options 	NFSCL                       # Network Filesystem Client: ZFS dep
options 	NFSD                        # Network Filesystem Server: ZFS dep
options 	NFSLOCKD                    # Network Lock Manager: ZFS dep
options 	NFS_ROOT                    # NFS usable as /, requires NFSCL: ZFS dep

options 	KDTRACE_FRAME		            # Ensure frames are compiled in
options 	KDTRACE_HOOKS		            # Kernel DTrace hooks

options     COMPAT_LINUX32              # Required for linux compat layers
options     COMPAT_FREEBSD32            # Required for poudriere compilation

As for the modules, I have decided to leave these:

• accf_data : buffer incoming connections until data arrives.

• accf_dns : buffer incoming DNS requests until the whole first request is present.

• accf_http : buffer incoming connections until a certain complete HTTP requests arrive.

• acl_nfs4: introduction to the POSIX.1e/NFSv4 ACL security API.

• acl_posix1e: introduction to the POSIX.1e/NFSv4 ACL security API.

• cc: TCP Congestion Control algorithms.

• cd9660: ISO-9660 file system.

• cd9660_iconv: ISO-9660 file system.

• fdescfs: Provides access to the per-process file descriptor namespace in the global file system namespace.

• fusefs: File system that is serviced by a userspace program.

• mac_ntpd: Policy allowing ntpd to run as non-root user.

• nullfs: The nullfs driver will permit the FreeBSD kernel to mount a loopback file system sub-tree.

• opensolaris: ZFS requirement.

• udf: Universal Disk Format.

• udf_iconv: Universal Disk Format.

• pf: packet filter

• pflog: packet filter logging interface

• pfsync: packet filter state table synchronization interface

• ipfw: IP packet filter and traffic accounting

• ipfw_nat: ipfw NAT

• ipfw_nat64: ipfw NAT IPv4 to IPv6

• ipfw_nptv6: IPv6 network prefix translation facility

• ipfw_pmod: packet modification facility

• dtrace: DTrace dynamic tracing compiler and tracing utility

• dtraceall: DTrace dynamic tracing compiler and tracing utility

• linprocfs: Required to compile using poudriere

• pseudofs: Required to compile using poudriere

• linux_common: Required to compile using poudriere

• procfs: Required to compile using poudriere

• linux: Required to compile using poudriere

• linux64: Required to compile using poudriere

NOTE: If we include ipfw in the kernel, it will always be loaded regardless of what is specified in the /etc/rc.conf file. Even with firewall_enable=“NO”, it will still be loaded. The only way to disable it is through firewall_type=“open”.

We compile our custom kernel:

We install it:

The INSTKERNNAME variable has been defined so that it does not overwrite the original kernel, as it is very difficult to maintain the operating system updated with our custom kernel running and an updated version of the GENERIC image as recommended here . This way, freebsd-update will update /boot/kernel regardless of /boot/kernel.kr0m-minimal.

If necessary, we could further refine the compilation process through the /etc/make.conf file.

We start with the new kernel:

We check which image we have loaded:

FreeBSD MightyMax.alfaexploit.com 13.1-RELEASE-p2 FreeBSD 13.1-RELEASE-p2 #19 releng/13.1-n250155-514a191356c-dirty: Thu Sep  1 13:01:03 CEST 2022     root@MightyMax.alfaexploit.com:/usr/obj/usr/src/amd64.amd64/sys/KR0M-MINIMAL amd64

If it works correctly, we edit the loader configuration to make it the default kernel:

kernel=kernel.kr0m-minimal

NOTE: Kernel modules are usually located in /boot/KERNEL_NAME/*.ko, except for third-party modules such as Nvidia or VirtualBox, which are located in /boot/modules/*.ko.

The hardware where the kernel was compiled is as follows:

hw.model: Intel(R) Core(TM)2 Duo CPU     P9500  @ 2.53GHz
hw.machine: amd64
hw.ncpu: 2

With 8GB of RAM:

             total       used       free     shared    buffers     cached
Mem:          7664       1307       6356          0          0          0
Swap:         2048          0       2048

Next, the difference between compiling a GENERIC kernel and compiling a minimal one:

CUSTOM:
real	4m56.748s
user	8m54.733s
sys	0m33.359s

GENERIC:
real	30m56.621s
user	54m24.386s
sys	4m48.999s

The difference is huge, we went from taking approximately 30 minutes to about 5 minutes.

As for the size, we also notice a difference:

CUSTOM:
du -h /boot/kernel.kr0m-minimal/kernel
6.8M	/boot/kernel.kr0m-minimal/kernel

du -sch /boot/kernel.kr0m-minimal/*.ko
209K	total

GENERIC:
du -h /boot/kernel/kernel 
17M	/boot/kernel/kernel

du -sch /boot/kernel/*.ko
80M	total

Troubleshooting:

Disabling IPv6 can cause problems since many software packages such as IOCage/Bastille assume that it will be available even without using IPv6 in the jails, so I do not recommend disabling it.
Bastille requires PF for NAT and port redirection, so we must compile it in the kernel or as a module. I have opted for the latter.

There are times when an update will not update the kernel but will update other parts of the system. This level of patching can be checked using the command:

13.0-RELEASE-p11

To ensure that our custom kernel correctly reflects the system’s patch level, we must recompile it again. This way, the image will be generated with the updated patch level information. The same applies to the GENERIC kernel; if we do not load the new image, it will not correctly reflect the patch level.

Summary

• In all updates, we must recompile the kernel to match the version of the core userlandtools and sources of both the core userlandtools and the kernel.
• If we also want to correctly reflect the patchlevel of the core userlandtools, we must also restart with the newly compiled kernel.

If our new kernel has problems booting, we can always load the GENERIC image or the previous image from the FreeBSD boot loader, just select the “Escape to loader prompt” option and type the name of the image.

Escape to loader prompt
boot kernel
boot kernel.kr0m.old

It may happen that the kernel boots but some commands such as ps, vmstat or connections randomly fail, this is because the system utilities were compiled for a version of the kernel very different from the current one.

To solve it, we must proceed differently depending on whether we compile “world” and the core userland tools or use the binary version of these:

• Source: Recompile and install the core userlandtools using a version of the sources on par with the version of the kernel.
• Binaries: It should be updated using freebsd-update to a version of FreeBSD that carries the binaries compiled with the same kernel version that we are using.

We can check the version of the kernel and the userlandtools as follows:

• freebsd-version -r -> Kernel currently running.
• freebsd-version -k -> Kernel installed but not yet loaded because the computer has not been restarted.
• freebsd-version -u -> Version of the core userlandtools.

If we do not have a GENERIC image for whatever reason, we can regenerate it as follows:

This process only requires not having touched the GENERIC configuration or compiled with retouched parameters in /etc/make.conf

We can also check the parameters with which our kernel was compiled if it was compiled with the INCLUDE_CONFIG_FILE option enabled (in GENERIC it is):

kern.conftxt: options	CONFIG_AUTOGENERATED
ident	KR0M-MINIMAL
machine	amd64
cpu	HAMMER
options	NFS_ROOT
options	NFSLOCKD
options	NFSD
...

Here are some reference links that may be helpful:
https://docs.freebsd.org/en/books/handbook/kernelconfig/
https://docs.freebsd.org/en/books/handbook/mirrors/index.html#git