Performing reliable backups quickly and easily can be a daunting task, but fortunately, by using ZFS snapshots, we can simplify it so much that it will be a piece of cake.
The article consists of different sections:
Creating a snapshot
We create the snapshot using Iocage , although we could do it using ZFS commands, I prefer to use Iocage.
As an example, we will do it on a Jail called HAProxy (it also works without problems with databases):
Snapshot: zroot/iocage/jails/HAProxy@BACKUP created.
We check the snapshots of the Jail:
+-----------------------------------------------------+-----------------------+-------+-------+
| NAME | CREATED | RSIZE | USED |
+=====================================================+=======================+=======+=======+
| BACKUP | Thu Jul 22 22:30 2021 | 112K | 0B |
+-----------------------------------------------------+-----------------------+-------+-------+
| BACKUP/root | Thu Jul 22 22:30 2021 | 4.28G | 136K |
+-----------------------------------------------------+-----------------------+-------+-------+
As we can see, there are two snapshots. This is because when we take a snapshot from Iocage, it takes it from the directory where the Jail configuration files and the root file system are located.
We will back up the root file system, and we will back up the configuration files manually without using ZFS.
Dumping the snapshot
We check the available ZFS snapshots:
NAME USED AVAIL REFER MOUNTPOINT
zroot/iocage/jails/HAProxy@BACKUP 0B - 112K -
zroot/iocage/jails/HAProxy/root@BACKUP 136K - 4.28G -
As we mentioned earlier, there are two snapshots, but we are only interested in the root file system of the Jail.
We dump the snapshot to a file:
If we check the type of file we have generated, we will see that it is a ZFS snapshot:
haproxy.raw: ZFS shapshot (little-endian machine), version 17, type: ZFS, destination GUID: 8A 36 8B AC 2E C4 C0 85, name: 'zroot/iocage/jails/HAProxy/root@BACKUP'
In addition to the snapshot dump, it is recommended to copy the Jail configuration files:
cp /zroot/iocage/jails/HAProxy/fstab /root/
We remove the snapshot:
Snapshot: zroot/iocage/jails/HAProxy@BACKUP destroyed
Restore dataset ZFS
We create the dataset where we will restore the snapshot:
zfs recv -dvu storage/restored < haproxy.raw
receiving full stream of zroot/iocage/jails/HAProxy/root@BACKUP into storage/restored/iocage/jails/HAProxy/root@BACKUP
received 6.01G stream in 350 seconds (17.6M/sec)
We locate the dataset to mount:
NAME USED AVAIL REFER MOUNTPOINT
storage/restored 4.28G 168G 96K /storage/restored
storage/restored/iocage 4.28G 168G 96K /storage/restored/iocage
storage/restored/iocage/jails 4.28G 168G 96K /storage/restored/iocage/jails
storage/restored/iocage/jails/HAProxy 4.28G 168G 96K /storage/restored/iocage/jails/HAProxy
storage/restored/iocage/jails/HAProxy/root 4.28G 168G 4.28G /storage/restored/iocage/jails/HAProxy/root
We mount it:
As we can see, the files are accessible at the mount point:
total 128
drwxr-xr-x 19 root wheel 24 May 17 08:38 .
drwxr-xr-x 3 root wheel 3 Jul 22 09:54 ..
-rw-r--r-- 2 root wheel 1023 May 17 08:38 .cshrc
-rw-r--r-- 2 root wheel 507 May 17 08:38 .profile
-r--r--r-- 1 root wheel 6109 May 17 08:38 COPYRIGHT
drwxr-xr-x 2 root wheel 46 Jul 5 07:53 bin
drwxr-xr-x 15 root wheel 66 Jul 5 07:53 boot
dr-xr-xr-x 2 root wheel 2 Oct 23 2020 dev
drwxr-xr-x 27 root wheel 109 Jul 19 19:23 etc
lrwxr-xr-x 1 root wheel 8 Jan 1 2021 home -> usr/home
drwxr-xr-x 5 root wheel 67 Jul 5 07:53 lib
drwxr-xr-x 3 root wheel 5 May 17 08:37 libexec
drwxr-xr-x 2 root wheel 2 Oct 23 2020 media
drwxr-xr-x 2 root wheel 2 Oct 23 2020 mnt
drwxr-xr-x 2 root wheel 2 Oct 23 2020 net
dr-xr-xr-x 2 root wheel 2 Oct 23 2020 proc
drwxr-xr-x 2 root wheel 150 May 17 08:39 rescue
drwxr-xr-x 4 root wheel 15 Jun 28 07:45 root
drwxr-xr-x 2 root wheel 137 May 17 13:55 sbin
drwxr-xr-x 3 root wheel 3 Mar 18 09:11 storage
lrwxr-xr-x 1 root wheel 11 Nov 2 2020 sys -> usr/src/sys
drwxrwxrwt 6 root wheel 6 Jul 22 05:00 tmp
drwxr-xr-x 15 root wheel 15 Jan 1 2021 usr
drwxr-xr-x 25 root wheel 25 Jul 19 19:23 var
Finally, we unmount and destroy the dataset:
zfs destroy -r storage/restored
Restore Jail Iocage
To restore the snapshot as a Jail, we must first create the datasets that Iocage expects:
zfs create zroot/iocage/jails/restored/root
We dump the snapshot:
We edit the configuration files so that the data matches:
{
"boot": 1,
"cloned_release": "12.2-RELEASE",
"host_hostname": "restored",
"host_hostuuid": "restored",
"ip4_addr": "nfe0|192.168.69.69/24",
"jail_zfs_dataset": "iocage/jails/restored/data",
"last_started": "2021-07-13 18:22:06",
"release": "13.0-RELEASE-p3"
}
We copy the configuration to the correct directory:
If the Jail had any mount point, we must also edit the fstab:
/storage/backups/HAProxy /zroot/iocage/jails/restored/root/storage/backups/HAProxy nullfs rw 00 # Added by iocage on 2021-03-18 22:12:31
We copy the configuration to the correct directory:
We start the Jail:
* Starting restored
+ Started OK
+ Using devfs_ruleset: 1011 (iocage generated default)
+ Using IP options: ip4.addr=nfe0|192.168.69.69/24 ip4.saddrsel=1 ip4=new ip6.saddrsel=1 ip6=new
+ Starting services OK
+ Executing poststart OK
We check that it is up:
+-----+----------+-------+--------------+---------------+
| JID | NAME | STATE | RELEASE | IP4 |
+=====+==========+=======+==============+===============+
| 1 | DrWho | up | 13.0-RELEASE | 192.168.69.6 |
+-----+----------+-------+--------------+---------------+
| 3 | HAProxy | up | 13.0-RELEASE | 192.168.69.11 |
+-----+----------+-------+--------------+---------------+
| 2 | Infinity | up | 13.0-RELEASE | 192.168.69.12 |
+-----+----------+-------+--------------+---------------+
| 4 | Lomax | up | 13.0-RELEASE | 192.168.69.7 |
+-----+----------+-------+--------------+---------------+
| - | Mistery | down | 13.0-RELEASE | 192.168.69.3 |
+-----+----------+-------+--------------+---------------+
| 5 | Potras | up | 13.0-RELEASE | 192.168.69.5 |
+-----+----------+-------+--------------+---------------+
| 15 | restored | up | 13.0-RELEASE | 192.168.69.69 |
+-----+----------+-------+--------------+---------------+
| 6 | rxWod | up | 13.0-RELEASE | 192.168.69.10 |
+-----+----------+-------+--------------+---------------+
Backups script
With the following script, we will be able to make ZFS backups of all the Jails that are started but whose name does not begin with test_ or is Lomax, in addition, we will keep a history of X days.
#!/usr/local/bin/bash
source /root/.scripts/cecho.sh
BACKUP_DIR='/storage/backups'
DAYS_OF_BACKUP='7'
clear
cecho "Cyan" "<== Iocage Jail backup script by kr0m ==>"
echo ""
echo ""
for JAIL in $(iocage list|grep up|grep -v 'test_'|grep -v 'Lomax'|awk '{print$4}'); do
cecho "Cyan" "-----------------------------------"
cecho "Cyan" ">> Making ZFS snapshot: $JAIL"
iocage snapshot -n BACKUP $JAIL
BACKUP_SNAPSHOT=$(zfs list -t snapshot|grep $JAIL|grep 'root@BACKUP'|awk '{print$1}')
cecho "Cyan" ">> Saving ZFS snapshot -> $JAIL.raw"
zfs send $BACKUP_SNAPSHOT > $BACKUP_DIR/$JAIL/$JAIL.raw
cecho "Cyan" ">> Copying config files"
cp /zroot/iocage/jails/$JAIL/config.json $BACKUP_DIR/$JAIL/
cp /zroot/iocage/jails/$JAIL/fstab $BACKUP_DIR/$JAIL/
cecho "Cyan" ">> Deleting ZFS snapshot"
iocage snapremove -n BACKUP $JAIL
cecho "Cyan" ">> Compressing ZFS snapshot and config files"
cd $BACKUP_DIR/$JAIL/
DATE=$(date +%d-%m-%Y)
tar czvf zfs_$DATE.tar.gz $JAIL.raw config.json fstab
rm $JAIL.raw config.json fstab
# Make cleaning of old backups:
TOTAL=$(ls -lt zfs_*|wc -l|awk '{print$1}')
if [ $TOTAL -gt $DAYS_OF_BACKUP ]; then
cecho "Cyan" ">> Deleting old ZFS backups"
let TO_DELETE=$TOTAL-$DAYS_OF_BACKUP
for BACKUP in $(ls -lt zfs_*|awk '{print$9}'|tail -n $TO_DELETE); do
cecho "Cyan" "-- Deleting ZFS backup: $BACKUP"
rm $BACKUP
done
fi
done
NOTE: The cecho command is simply to display text in color by terminal, we can download the file from here or simply replace it with normal echos.
We assign the necessary permissions:
We schedule the backup at 5:00 in the morning:
00 05 * * * /root/.scripts/jailsBackup.sh >/dev/null 2>&1