We all know the advantages of using custom compiled binaries:

• Lower RAM consumption
• Smaller binaries
• More modern software versions
• Lower attack surface since unnecessary binary functionalities have been removed

The only drawback is the time required for compilation, but what if we had a more powerful computer to perform the compilation and use these binaries on our server?

Poudriere precisely takes care of this task, compiling custom binaries from ports to be used by other computers.

All the information in this article has been collected from the following sources:

• https://github.com/freebsd/poudriere/wiki
• https://www.freebsd.org/doc/handbook/ports-poudriere.html
• https://www.digitalocean.com/community/tutorials/how-to-set-up-a-poudriere-build-system-to-create-packages-for-your-freebsd-servers
• https://www.freebsd.org/cgi/man.cgi?query=poudriere&sektion=8&manpath=freebsd-release-ports
• https://hackmd.io/@dch/HkwIhv6x7

To be able to generate binaries using Poudriere, we will have to install Portmaster, Nginx, and ccache:

When we build packages with Poudriere, we will sign them with a private key, thus ensuring that no one can impersonate our package server. We create the necessary directories:

We generate the private key:

We generate the certificate that will be included in the package signature:

If our server does not support ZFS:

NO_ZFS=yes

Otherwise, we indicate the pool and the rootfs:

ZPOOL=zroot  
NO_ZFS=no  
ZROOTFS=/poudriere

Poudriere mounts a jail where it will compile the ports, with the FREEBSD_HOST parameter we indicate where it should download the jail image from:

FREEBSD_HOST=ftp://ftp.freebsd.org

We indicate where it should generate the necessary files for its operation:

POUDRIERE_DATA=${BASEFS}/data

We indicate if we want to use RAM as a file system, several levels of use can be defined, in my case I have 32Gb of RAM so I leave it configured as all:

USE_TMPFS=all  

CHECK_CHANGED_OPTIONS indicates whether packages should be recompiled if the compilation options have changed. CHECK_CHANGED_DEPS indicates whether packages should be recompiled if the dependencies have changed since the last compilation.

CHECK_CHANGED_OPTIONS=verbose  
CHECK_CHANGED_DEPS=yes

We tell Poudriere where the key is that it should use to sign the packages:

PKG_REPO_SIGNING_KEY=/usr/local/etc/ssl/keys/poudriere.key 

The directory where to save the cached object files from previous compilations:

CCACHE_DIR=/var/cache/ccache

We allow each job to use all available cores in the system:
ALLOW_MAKE_JOBS=yes

Finally, we indicate the URL with which clients will have access to the packages:

URL_BASE=http://poudriere.alfaexploit.com/

We create the object caching directory and the downloaded file caching directory:

My entire configuration looks like this:

ZPOOL=zroot
NO_ZFS=no
ZROOTFS=/poudriere
FREEBSD_HOST=ftp://ftp.freebsd.org
RESOLV_CONF=/etc/resolv.conf
BASEFS=/usr/local/poudriere
POUDRIERE_DATA=${BASEFS}/data
USE_PORTLINT=no
USE_TMPFS=all
DISTFILES_CACHE=/usr/ports/distfiles
CHECK_CHANGED_OPTIONS=verbose
CHECK_CHANGED_DEPS=yes
PKG_REPO_SIGNING_KEY=/usr/local/etc/ssl/keys/poudriere.key
CCACHE_DIR=/var/cache/ccache
ALLOW_MAKE_JOBS=yes
URL_BASE=http://poudriere.alfaexploit.com/

If we consider that Poudriere is consuming too many resources, we can limit the number of PARALLEL_JOBS, modify the ALLOW_MAKE_JOBS parameter, or reconfigure the USE_TMPFS parameter.

For example, we could choose to compile 8 ports, with each port using 1 core:

PARALLEL_JOBS=8
ALLOW_MAKE_JOBS=no

Or compile 1 port and use all the cores of the system:

PARALLEL_JOBS=1
ALLOW_MAKE_JOBS=yes

Now that we have Poudriere configured, we need to set up our compilation environment. As previously indicated, Poudriere will set up a jail for this purpose, and the only requirement is that the jail must be the same version or lower than the parent system.

We create the jail by specifying the name and version:

NOTE: It is not recommended to use “.” in the jail name as it can cause problems with certain tools.

JAILNAME        VERSION         ARCH  METHOD TIMESTAMP           PATH
freebsd_12-1x64 12.1-RELEASE-p5 amd64 ftp    2020-06-05 20:24:50 /usr/local/poudriere/jails/freebsd_12-1x64

We install the ports tree, and later we will tell the jail to use it:

NOTE: We can have several jails with different OS versions and versions of the ports.

PORTSTREE METHOD    TIMESTAMP            PATH  
latest    git+https 2020-06-05 21:05:18 /usr/local/poudriere/ports/latest

We generate the list of ports to compile. The best way to do this is to get the list of packages currently installed on the servers that will use our repository. Before generating the list, we do a little cleaning to make the list as small as possible.

We get the list of packages we installed explicitly without dependencies:

We copy the list to the Poudriere server.

This process must be done on all servers, and the list must be unified on the compilation server.

If necessary, we can add additional ports to the list:

NOTE: Don’t worry about dependencies since they will be compiled automatically.

If we want to parameterize the make.conf, we must indicate it. In my case, I’m interested in compiling everything related to PHP with version 7.4 and binaries without X or ipv6 support:

OPTIONS_UNSET+= X11 IPV6  
DEFAULT_VERSIONS+= php=7.4

We tell Poudriere which options to use to compile the ports. The configuration dialog will only appear if it has not been previously configured in the /usr/local/etc/poudriere.d/freebsd_12-1x64-latest-options/PORTNAME/options file. Once the configuration is done, it will not ask again in subsequent compilations. It is important to choose the minimum options to have fewer dependencies and faster compilation.

The Poudriere web interface shows us why each port is being compiled. We can know who each of them depends on and thus find out incorrect configuration parameters.

If later we want to be asked again for the compilation options, we will execute the same command but with the -c option:

If we only want to reconfigure the options for one of the ports, the easiest way is to delete the associated configuration:

NOTE: If we have doubts about the correct parameters, we can always consult them on another server where the binary is installed from the FreeBSD repositories. For example, for php7.4: pkg info php74

Finally, we can start building ports, but first, we make sure to have the jail and the port tree updated.

We start the compilation in a screen session:

We get information about the compilation process:

load: 6.80  cmd: sh 96156 [nanslp] 669.81r 0.21u 0.84s 0% 3852k  
[freebsd_12-1x64-latest] [2020-06-05_20h46m50s] [parallel_build:] Queued: 325 Built: 59  Failed: 0   Skipped: 0   Ignored: 0   Tobuild: 266  Time: 00:11:10  
 [01]: lang/tcl86                | tcl86-8.6.10              fetch           (00:00:17 / 00:00:19)  
 [02]: devel/nspr                | nspr-4.25                 package         (00:00:01 / 00:00:12)  
 [03]: lang/perl5.30             | perl5-5.30.3              build           (00:02:45 / 00:03:26)  
 [04]: databases/db5             | db5-5.3.28_7              build           (00:00:28 / 00:00:39)  
 [05]: dns/libidn2               | libidn2-2.3.0_1           fetch           (00:00:02 / 00:00:04)  
 [06]: devel/swig30              | swig30-3.0.12_1           configure       (00:00:02 / 00:00:26)  
 [07]: graphics/giflib           | giflib-5.2.1              fetch           (00:00:51 / 00:00:52)  
 [08]: devel/gettext-tools       | gettext-tools-0.20.2      build           (00:01:36 / 00:02:09)

Now that we have the binaries compiled, we just need to serve them through Nginx, which will also serve to consult the Poudriere compilation statistics.

We enable the service at startup:

Through the Nginx configuration, we will have access to the Poudriere interface (/), logs (/data), and packages (/packages).

#http context
. . .
    server {
        listen 80 default;
        server_name poudriere.alfaexploit.com;
        root /usr/local/share/poudriere/html;

        location /data {
            alias /usr/local/poudriere/data/logs/bulk;
            autoindex on;
        }

        location /packages {
            root /usr/local/poudriere/data;
            autoindex on;
        }
    }
}

To be able to view the logs directly from the web interface and not force us to download the file, we will have to modify the Nginx mime.types file.

text/plain                          txt log;

We check the configuration:

Performing sanity check on nginx configuration:  
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok  
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful

We start Nginx:

If we have any configured fw, we adapt it, and we can already access the Poudriere web interface.
http://poudriere.alfaexploit.com

If we want to see the packages, we must access /packages:
http://poudriere.alfaexploit.com/packages

Only the client configuration remains, just as it is not recommended to mix binary packages with ports compiled by the different compilation options they may have, neither is it recommended to mix packages from the official FreeBSD repositories with our own Poudriere repositories. To exclusively use our repository, we will only have to omit the priority configuration parameter and disable the official ones.

poudriere: {  
    url: "http://poudriere.alfaexploit.com/packages/freebsd_12-1x64-latest/",  
    mirror_type: "http",  
    signature_type: "pubkey",  
    pubkey: "/usr/local/etc/ssl/certs/poudriere.cert",  
    enabled: yes,  
}

FreeBSD: {  
    enabled: no  
}

We dump the content of the poudriere.cert pubkey into the client configuration.

On the client:
If the poudriere server is a server on our LAN, we modify the client’s /etc/hosts. If it is a public server, this step is unnecessary.

192.168.69.4		poudriere.alfaexploit.com

We manually check that the client can access the packages:

<html>  
<head><title>Index of /packages/</title></head>  
<body>  
<h1>Index of /packages/</h1><hr><pre><a href="../">../</a>  
<a href="freebsd_12-1x64-latest/">freebsd_12-1x64-latest/</a>                              05-Jun-2020 18:46                   -  
</pre><hr></body>  
</html>

We force the reinstallation of all packages, but this time from our repository. This way, we make sure that no package from the FreeBSD repositories is left:

Now we can install packages from our binary package server ☺️

Every time we want to update our servers, we must first compile the latest binaries in Poudriere. The steps to follow are as follows.

We update the build jail:

We update the port tree:

We compile:

NOTE: Some ports such as lang/rust or lang/gcc may have problems with multi-threaded compilations. For this reason, they are compiled with a separate command indicating that only one core is used.

The “poudriere pkgclean” command will remove old packages that are no longer listed in /usr/local/etc/poudriere.d/port-list. If we want to regenerate all packages, we can delete them with the following command:

We can see the build progress on the web interface:
http://poudriere.alfaexploit.com

We update the servers with:

For the laziest, here’s a quick script:

#!/usr/local/bin/bash
poudriere jail -u -j freebsd_13-0x64
poudriere ports -u -p latest
poudriere options -j freebsd_13-0x64 -p latest -f /usr/local/etc/poudriere.d/port-list

# Compile lang/rust devel/llvm and lang/gcc using only 1 core because these ports frecuently crash with parallel compilation:
# man poudriere-bulk
poudriere bulk -r -t -J 1:8 -j freebsd_13-0x64 -p latest lang/rust lang/gcc10 

poudriere bulk -j freebsd_13-0x64 -p latest -f /usr/local/etc/poudriere.d/port-list
poudriere pkgclean -y -j freebsd_13-0x64 -p latest -f /usr/local/etc/poudriere.d/port-list

# poudriere distclean -a -p latest
# poudriere logclean -a
# poudriere pkgclean -A -j freebsd_13-0x64 -p latest

UPDATE

If we switch from one version of FreeBSD to another, there will be old jails left. To remove them, we will proceed as follows:

TROUBLESHOOTING

If we have problems compiling, we can try:

Creating the jail again and a new ports tree.

Deleting the cache:

Manually adding packages to the list. To know the name of the port, we must first search the ports system:

Now that we know the name of the port, we can add it to the list:

CATEGORY/NAME

We can also delete previously compiled ports and recompile them, with the time it takes:

If we see that a port is causing problems, as is the case with rust.
dmesg:

pid 17034 (rustc), jid 18, uid 0, was killed: out of swap space

Poudriere logs:

RuntimeError: failed to run: /wrkdirs/usr/ports/lang/rust/work/rustc-1.53.0-src/build/bootstrap/debug/bootstrap build –jobs=8

We will try to compile the port before the rest of the ports but with only one core: