Bastille is a container (jails) manager with 0 dependencies since it is written in Bourne Shell. It makes use of the best functionalities and technologies that FreeBSD offers us, thus facilitating the management, creation, destruction, and updating of jails throughout their life cycle.
Some advantages over other virtualization systems such as
IOCage
are:
- Better maintained project, receives updates more often.
- 0 dependencies for being written in Bourne Shell.
- More complete template system.
- Possibility of reusing template recipes through Rocinante .
- Jails natting with integrated port redirection.
We can find a very illustrative table on the Bastille website.
The article is composed of several parts:
- Installation
- Networking
- Networking-NAT
- Networking-Alias
- Networking-VNET
- Networking-VNET-Bridge-Own-Bridge
- Networking-VNET-DHCP
- Port Forwarding
- Final Firewall Script
- Types of Jails
- Viewing Jails/Releases/Templates/Logs
- Templates
- Export/Import Jails
- Mounting Directories
- Useful Commands
- Update
- Update BASE
- Update PACKAGES/PORTS
- Update Script
- Binary Package Security
- Limiting Resources
- ZFS Parameters: Quotas/Snapshots/df
- Jails Configuration
- Migration from IOCage to Bastille
- Bash AutoCompletion
- Jails Auto-Start
- Linux Jails
- Troubleshooting
Installation
We install the dependencies:
We install Bastille:
We configure if we want to use ZFS or not:
## ZFS options
bastille_zfs_enable="YES" ## default: ""
bastille_zfs_zpool="zroot" ## default: ""
We enable the service:
service bastille start
We download the base files of the FreeBSD version we want to use and apply the latest security patches:
Optionally, we can verify the image:
Networking
Bastille has some limitations regarding jail names, for example:
- Jail names cannot contain the “.” character.
- Names cannot be too long or network configuration commands will fail:
ifconfig: ioctl SIOCSIFNAME (set name): File name too long
jail: alcatrazVnet2: ifconfig epair0a up name e0a_alcatrazVnet2: failed
All networking documentation can be found at this link .
To see the network type of each jail, we must consult its configuration directly:
The PF rules we will see later in this same article only affect NAT and Alias network modes, since in VNET and VNET-BRIDGE-PROPIO we explicitly disable traffic filtering on bridges, so they will not be affected.
Networking-NAT
Through NAT, we will have one or more internal networks that will be NATed using
PF
. To do this, we must perform the following configuration:
sysrc ifconfig_lo1_name="bastille0"
We apply the configuration:
We will write a simple PF script where we will NAT the “jails” table, redirect the rdr ports, and allow free access to the 22-ssh port:
ext_if="bge1"
set block-policy return
scrub in on $ext_if all fragment reassemble
set skip on lo
table <jails> persist
nat on $ext_if from <jails> to any -> ($ext_if:0)
rdr-anchor "rdr/*"
block in all
pass out quick keep state
antispoof for $ext_if inet
pass in inet proto tcp from any to any port ssh flags S/SA keep state
Enable and start the service:
service pf start
We can check the rules with:
scrub in on bge1 all fragment reassemble
block return in all
pass out quick all flags S/SA keep state
block drop in on ! bge1 inet from 192.168.40.0/24 to any
block drop in inet from 192.168.40.113 to any
pass in inet proto tcp from any to any port = ssh flags S/SA keep state
The recommended internal ranges by Bastille are:
IP options include: 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16.
By default, Bastille creates the NATed jails:
JID IP Address Hostname Path
natJail 10.17.89.50 natJail /usr/local/bastille/jails/natJail/root
If we access the jail, we can see the IP:
bastille0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet 10.17.89.50 netmask 0xffffffff
inet6 fe80::1%bastille0 prefixlen 64 scopeid 0x4
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
Enable and start the SSH service in the jail:
bastille service natJail sshd start
We can see how the socket is listening:
[natJail]:
USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS
root sshd 6180 3 tcp4 10.17.89.50:22 *:*
[natJail]: 0
We enter the jail to add a user to check SSH connectivity:
adduser
Connect:
Networking-Alias
As the name suggests, this mode uses alias IPs on the parent host to assign them to the jails.
We create the jail indicating the interface on which to configure the alias:
JID IP Address Hostname Path
natJail 10.17.89.50 natJail /usr/local/bastille/jails/natJail/root
aliasJail 192.168.40.136 aliasJail /usr/local/bastille/jails/aliasJail/root
We can see how the parent interface now has several configured IPs:
bge1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
ether 00:1a:64:6d:e6:f8
inet 192.168.40.113 netmask 0xffffff00 broadcast 192.168.40.255
inet 192.168.40.136 netmask 0xffffff00 broadcast 192.168.40.255
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
In Alias mode, we must keep in mind that the IPs are configured in the parent, so if we want to bind a service to the jail, we must configure the parent to only use its specific IP and not use wildcards:
ListenAddress 192.168.40.113
We enable and start the SSH service in the jail:
bastille service aliasJail sshd start
We enter the jail to add a user to check SSH connectivity:
adduser
We connect:
Networking-VNET
Bastille creates a bridge and automatically assigns the jail interface and the WAN interface to this bridge.
For VNET jails to function correctly, devd rules are required that should only be configured once:
[bastille_vnet=13]
add path 'bpf*' unhide
We restart devd:
We disable traffic filtering on the bridges, so each VNET jail will perform its own filtering:
# Bastille:
net.link.bridge.pfil_bridge=0 # Packet filter on the bridge interface
net.link.bridge.pfil_onlyip=0 # Only pass IP packets when pfil is enabled
net.link.bridge.pfil_member=0 # Packet filter on the member interface
sysctl net.link.bridge.pfil_onlyip=0
sysctl net.link.bridge.pfil_member=0
We create the jail:
JID IP Address Hostname Path
aliasJail 192.168.40.136 aliasJail /usr/local/bastille/jails/aliasJail/root
jailVnet jailVnet /usr/local/bastille/jails/jailVnet/root
natJail 10.17.89.50 natJail /usr/local/bastille/jails/natJail/root
We can see in the parent the bridge with the jail interface (e0a_bastille0) and the WAN interface (bge1):
bge1bridge: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 58:9c:fc:10:ff:a4
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: e0a_bastille0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 6 priority 128 path cost 2000
member: bge1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 2 priority 128 path cost 55
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>
e0a_bastille0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: vnet host interface for Bastille jail jailVnet
options=8<VLAN_MTU>
ether 02:20:98:6d:e6:f8
hwaddr 02:b5:d3:fe:09:0a
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
We enable and start the SSH service in the jail:
bastille service jailVnet sshd start
If we access the jail we can see the IP:
vnet0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 0e:20:98:6d:e6:f8
hwaddr 02:b5:d3:fe:09:0b
inet 192.168.40.137 netmask 0xffffff00 broadcast 192.168.40.255
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
We add a user to check SSH connectivity:
Sometimes the default route is not detected correctly, if this is the case we can adjust it for each jail:
bastille service TARGET routing restart
Or from Bastille for new jails:
bastille_network_gateway=aa.bb.cc.dd
Networking-VNET-Bridge-Own-Bridge
The same configuration as VNET must be followed.
[bastille_vnet=13]
add path 'bpf*' unhide
We restart devd:
We disable traffic filtering on bridges, so each VNET jail will perform its own filtering:
# Bastille:
net.link.bridge.pfil_bridge=0 # Packet filter on the bridge interface
net.link.bridge.pfil_onlyip=0 # Only pass IP packets when pfil is enabled
net.link.bridge.pfil_member=0 # Packet filter on the member interface
sysctl net.link.bridge.pfil_onlyip=0
sysctl net.link.bridge.pfil_member=0
In automatic VNET mode we have put the WAN interface in a bridge and interfaces cannot be in two bridges simultaneously, so first we will remove the VNET jail we created in the previous step:
bastille destroy jailVnet
ifconfig bge1bridge destroy
We create the bridge in the parent:
sysrc ifconfig_bridge0=“addm bge1 up”
We apply the configuration:
We create the jail:
JID IP Address Hostname Path
aliasJail 192.168.40.136 aliasJail /usr/local/bastille/jails/aliasJail/root
natJail 10.17.89.50 natJail /usr/local/bastille/jails/natJail/root
jailVnet2 jailVnet2 /usr/local/bastille/jails/jailVnet2/root
We can see in the parent the bridge with the jail interface (e0a_jailVnet2) and the WAN interface (bge1):
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 58:9c:fc:10:ff:a4
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: e0a_jailVnet2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 6 priority 128 path cost 2000
member: bge1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 2 priority 128 path cost 200000
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>
e0a_jailVnet2: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:f2:2e:f9:94:0a
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
We enable and start the SSH service in the jail:
bastille sysrc jailVnet2 sshd_enable=YES
bastille service jailVnet2 sshd start
If we access the jail, we can see the IP:
vnet0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:f2:2e:f9:94:0b
inet 192.168.40.138 netmask 0xffffff00 broadcast 192.168.40.255
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
We add a user to check SSH connectivity:
Networking-VNET-DHCP:
If we want to use DHCP, the only option is the VNET interfaces, the same configuration as VNET must be followed.
[bastille_vnet=13]
add path 'bpf*' unhide
We restart devd:
We disable traffic filtering on bridges, so each VNET jail will perform its own filtering:
# Bastille:
net.link.bridge.pfil_bridge=0 # Packet filter on the bridge interface
net.link.bridge.pfil_onlyip=0 # Only pass IP packets when pfil is enabled
net.link.bridge.pfil_member=0 # Packet filter on the member interface
sysctl net.link.bridge.pfil_onlyip=0
sysctl net.link.bridge.pfil_member=0
We just need to create the jail with the IP 0.0.0.0:
We can see an interface e0b_bastille0:
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
groups: pflog
e0b_bastille0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 0e:20:98:eb:c7:e9
hwaddr 02:f8:eb:0f:50:0b
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
In which we can request an IP manually:
DHCPREQUEST on e0b_bastille0 to 255.255.255.255 port 67
DHCPACK from 192.168.69.200
bound to 192.168.69.205 -- renewal in 43200 seconds.
Or through this crontab, as I have been unable to configure it through the operating system (RC) configuration:
@reboot /sbin/dhclient e0b_bastille0
In all cases, the interface will be in the following state:
e0b_bastille0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 0e:20:98:eb:c7:e9
hwaddr 02:f8:eb:0f:50:0b
inet 192.168.69.205 netmask 0xffffff00 broadcast 192.168.69.255
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
And the routes:
Routing tables
Internet:
Destination Gateway Flags Netif Expire
default 192.168.69.200 UGS e0b_bast
Port forwarding:
Using the rdr command, we can redirect ports to the Nateadas jails:
Usage: bastille rdr TARGET [clear] | [list] | [tcp <host_port> <jail_port>] | [udp <host_port> <jail_port>]
Bastille uses a PF anchor, so the rules can be dynamic in real-time:
We check the jail’s redirects:
rdr pass on bge1 inet proto tcp from any to any port = 2001 -> 10.17.89.50 port 22
JID State IP Address Published Ports Hostname Release Path
HAProxy Up 192.168.40.198 - HAProxy.alfaexploit.com 13.1-RELEASE-p5 /usr/local/bastille/jails/HAProxy/root
NATJail Up 10.17.89.50 tcp/2001:22 NATJail 13.1-RELEASE-p5 /usr/local/bastille/jails/NATJail/root
aliasJail Up 192.168.40.136 - aliasJail 13.1-RELEASE-p5 /usr/local/bastille/jails/aliasJail/root
aliasJail2 Up 192.168.40.139 - aliasJail2 13.1-RELEASE-p5 /usr/local/bastille/jails/aliasJail2/root
jailVnet2 Up 192.168.40.138 - jailVnet2 13.1-RELEASE-p5 /usr/local/bastille/jails/jailVnet2/root
nginx Up 192.168.40.140 - nginx 13.1-RELEASE-p5 /usr/local/bastille/jails/nginx/root
thickJail Up 192.168.40.190 - thickJail 13.1-RELEASE-p5 /usr/local/bastille/jails/thickJail/root
If we want to remove the redirects of a jail:
Final firewall script
In my case, the final firewall script would be as follows:
ext_if = "nfe0"
set block-policy return
scrub in on $ext_if all fragment reassemble
set skip on lo
table <badguys> persist
table <jails> persist
nat on $ext_if from <jails> to any -> ($ext_if:0)
rdr-anchor "rdr/*"
antispoof for $ext_if inet
block log all
pass out quick
pass in proto tcp to 192.168.69.2 port 7777
# SMTP -> HellStorm
pass in proto tcp to 192.168.69.17 port 25
# STATS/HTTP/HTTPS -> Atlas
pass in proto tcp from 192.168.69.4 to 192.168.69.19 port 8404
pass in proto tcp to 192.168.69.19 port 80
pass in proto tcp to 192.168.69.19 port 443
# Continue attending badguys in HA-Proxy, we have a surprise for them:
pass in quick proto tcp from <badguys> to 192.168.69.19 port 80
pass in quick proto tcp from <badguys> to 192.168.69.19 port 443
# Xbox -> Paradox
pass in proto tcp from 192.168.69.196 to 192.168.69.18 port 80
# TARS -> Paradox
pass in proto tcp from 192.168.69.198 to 192.168.69.18 port 80
# Garrus -> Paradox, testing purpose
pass in proto tcp from 192.168.69.4 to 192.168.69.18 port 80
# HACKERS script
pass in proto tcp to 192.168.69.2 port 6666
pass in proto tcp to any port 22
# Garrus -> Flatland testing
pass in proto tcp from 192.168.69.4 to 192.168.69.24 port 80
# Block all traffic from badguys, except 80,443 that was allowed previously:
block in from <badguys>
Types of jails
Bastille allows creating jails of several different types:
-E | --empty -- Creates an empty container, intended for custom jail builds (thin/thick/linux or unsupported).
-L | --linux -- This option is intended for testing with Linux jails, this is considered experimental.
-T | --thick -- Creates a thick container, they consume more space as they are self contained and independent.
But the most commonly used are Thin (default mode) and Thick:
-
Thin:
bastille create thinJail 13.1-RELEASE 192.168.40.190 bge1- All jails share a base in RO mode.
- When the RELEASE is updated, all jails that share the base are updated.
- Recommended for jails that are not planned to be updated from one major RELEASE to another, but only within the same RELEASE.
-
Thick:
bastille create -T thickJail 13.1-RELEASE 192.168.40.190 bge1- Each jail has a copy of the RELEASE, they are updated independently and the entire file system is RW.
- Recommended for jails that will be updated between major RELEASEs over time.
It is possible to convert a Thin jail to Thick, but we must bear in mind that the process is irreversible, that is, it is NOT possible to switch from Thick to Thin:
bastille convert aliasJail2
The only way to know if it is a Thin or Thick jail is to check if the base is mounted:
/usr/local/bastille/releases/13.1-RELEASE nullfs 130G 488M 129G 0% /usr/local/bastille/jails/aliasJail/root/.bastille
/usr/local/bastille/releases/13.1-RELEASE nullfs 130G 488M 129G 0% /usr/local/bastille/jails/jailVnet2/root/.bastille
/usr/local/bastille/releases/13.1-RELEASE nullfs 130G 488M 129G 0% /usr/local/bastille/jails/natJail/root/.bastille
/usr/local/bastille/releases/13.1-RELEASE nullfs 130G 488M 129G 0% /usr/local/bastille/jails/nginx/root/.bastille
Viewing jails/releases/templates/logs
List of started jails:
aliasJail
aliasJail2
jailVnet2
natJail
List of started jails with more information:
JID IP Address Hostname Path
aliasJail 192.168.40.136 aliasJail /usr/local/bastille/jails/aliasJail/root
natJail 10.17.89.50 natJail /usr/local/bastille/jails/natJail/root
jailVnet2 jailVnet2 /usr/local/bastille/jails/jailVnet2/root
To see all jails and their details including the VNET jail IPs:
JID State IP Address Published Ports Hostname Release Path
aliasJail Up 192.168.40.136 - aliasJail 13.1-RELEASE-p5 /usr/local/bastille/jails/aliasJail/root
jailVnet2 Up 192.168.40.138 - jailVnet2 13.1-RELEASE-p5 /usr/local/bastille/jails/jailVnet2/root
natJail Up 10.17.89.50 - natJail 13.1-RELEASE-p5 /usr/local/bastille/jails/natJail/root
List of releases:
13.1-RELEASE
List of templates:
/usr/local/bastille/templates
List of jail log files:
/var/log/bastille/aliasJail_console.log
/var/log/bastille/aliasJail2_console.log-2022-12-18
/var/log/bastille/jailVnet2_console.log
/var/log/bastille/natJail_console.log
/var/log/bastille/jailVnet_console.log-2022-12-17
Templates
Templates
are a way of creating “recipes”, similar to Ansible but for Bastille jails. In these templates, we must indicate the steps to be performed on the jail, for example, install X package, start a certain service, etc.
https://gitlab.com/bastillebsd-templates
https://docs.bastillebsd.org/en/latest/chapters/template.html
As an example, we download the Nginx template:
Cloning into '/usr/local/bastille/templates/bastillebsd-templates/nginx'...
warning: redirecting to https://gitlab.com/bastillebsd-templates/nginx.git/
remote: Enumerating objects: 54, done.
remote: Counting objects: 100% (12/12), done.
remote: Compressing objects: 100% (12/12), done.
remote: Total 54 (delta 2), reused 1 (delta 0), pack-reused 42
Receiving objects: 100% (54/54), 7.53 KiB | 3.76 MiB/s, done.
Resolving deltas: 100% (9/9), done.
Detected Bastillefile hook.
[Bastillefile]:
PKG nginx
CP usr /
SYSRC nginx_enable=YES
CMD nginx -t
SERVICE nginx restart
Template ready to use.
Now we see that we have one more template:
/usr/local/bastille/templates
/usr/local/bastille/templates/bastillebsd-templates
/usr/local/bastille/templates/bastillebsd-templates/nginx
We create a jail to which we will apply the template:
We apply the template:
We can see how Nginx is listening:
[nginx]:
USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS
nobody nginx 19341 7 tcp4 192.168.40.140:80 *:*
nobody nginx 18809 7 tcp4 192.168.40.140:80 *:*
root nginx 17697 7 tcp4 192.168.40.140:80 *:*
[nginx]: 0
And we can connect without problems:
HTTP/1.1 200 OK
Server: nginx/1.22.0
Date: Sun, 18 Dec 2022 14:40:55 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Mon, 23 May 2022 23:59:19 GMT
Connection: keep-alive
ETag: "628c1fd7-267"
Accept-Ranges: bytes
Of course, we can create our own templates, it does not use any markup language or any exotic language, just real commands.
The best way is to copy an existing one and modify it:
PKG nginx
CP usr /
SYSRC nginx_enable=YES
CMD nginx -t
SERVICE nginx restart
Currently supported template hooks are: CMD, CP, INCLUDE, LIMITS, MOUNT, PKG, RDR, SERVICE, SYSRC:
| HOOK | Format | Example |
|---|---|---|
| CMD | /bin/sh | command /usr/bin/chsh -s /usr/local/bin/zsh |
| CP | path(s) | etc root usr (one per line) |
| INCLUDE | template path/URL | http://TEMPLATE_URL or project/path |
| LIMITS | resource value | memoryuse 1G |
| MOUNT | fstab syntax | /host/path container/path nullfs ro 0 0 |
| PKG | port/pkg name(s) | vim-console zsh git-lite tree htop |
| RDR | tcp port port | tcp 2200 22 (hostport jailport) |
| SERVICE | service command | ’nginx start’ OR ‘postfix reload’ |
| SYSRC | sysrc command(s) | nginx_enable=YES |
A very interesting aspect of templates is that they are not limited to jails created by Bastille, but can also be applied to any system, whether physical, virtual machine, or jail, using a software called
Rocinante
. This way, the recipes we have in template format will be reusable:
We install Nginx using Rocinante:
Export/Import Jails
When we export a jail, we are generating a compressed image that can then be imported into another system. This way, we can move jails between hosts and perform backups quickly and easily.
The export command works on both UFS and ZFS:
- UFS: The jail must be stopped, and only supports txz.
- ZFS: It will take a snapshot and extract the compressed file from it without the need to stop the jail.
Usage: bastille export | option(s) | TARGET | PATH
--gz -- Export a ZFS jail using GZIP(.gz) compressed image.
-r | --raw -- Export a ZFS jail to an uncompressed RAW image.
-s | --safe -- Safely stop and start a ZFS jail before the exporting process.
--tgz -- Export a jail using simple .tgz compressed archive instead.
--txz -- Export a jail using simple .txz compressed archive instead.
-v | --verbose -- Be more verbose during the ZFS send operation.
--xz -- Export a ZFS jail using XZ(.xz) compressed image.
We export a jail as an example:
Exporting 'aliasJail' to a compressed .tgz archive...
84.5%
Exported '/usr/local/bastille/backups/aliasJail_2022-12-27-121707.tgz' successfully.
We can see the size of the backup:
1.1M /usr/local/bastille/backups/aliasJail_2022-12-27-121707.tgz
We can also get a list of backups. The subcommands export/import/backup are equivalent:
bastille list import
bastille list backup
aliasJail_2022-12-27-121707.tgz
The restoration is done with the same name, so to restore, we first delete the original jail:
bastille destroy aliasJail
bastille import /usr/local/bastille/backups/aliasJail_2022-12-27-121707.tgz
Mounting Directories
Using the mount command, we can mount directories from the parent host into the jail itself:
Usage: bastille mount TARGET host_path container_path [filesystem_type options dump pass_number]
We set up a test directory:
bastille mount aliasJail2 /mnt/storage mnt/storage nullfs rw 0 0
[aliasJail2]:
Added: /mnt/storage /usr/local/bastille/jails/aliasJail2/root/mnt/storage nullfs rw 0 0
We can see how the jail configuration now has an additional entry:
/usr/local/bastille/releases/13.1-RELEASE /usr/local/bastille/jails/aliasJail2/root/.bastille nullfs ro 0 0
/mnt/storage /usr/local/bastille/jails/aliasJail2/root/mnt/storage nullfs rw 0 0
We create a file from the parent:
We access the jail and verify that the file is indeed visible:
ls -al /mnt/storage/
total 2
drwxr-xr-x 2 root wheel 3 Dec 18 16:16 .
drwxr-xr-x 3 root wheel 3 Dec 18 16:15 ..
-rw-r--r-- 1 root wheel 0 Dec 18 16:16 AA
To unmount it, it is as simple as executing the umount command:
[aliasJail2]:
Unmounted: /mnt/storage
NOTE: The umount command will NOT remove the entry from the file /usr/local/bastille/jails/JAIL_NAME/fstab, but it will no longer be available inside the jail.
Useful commands
Login with another user:
We can directly install software:
bastille pkg ALL install -y htop
Running top/htop is such a common task that Bastille incorporates a command for it:
note: won’t work if you don’t have htop installed in the container.
bastille htop aliasJail
To delete a jail, first stop it, then destroy it:
bastille destroy alcatraz
Run sysrc commands in the jail:
bastille sysrc ALL sshd_enable=YES
Manage services:
bastille service nginx nginx configtest
bastille service ALL nginx status
We can copy files directly to the jail:
bastille cp ALL /etc/resolv.conf etc/resolv.conf
Clone a jail:
bastille start aliasJail2
Rename a jail:
bastille rename natJail NATJail
Restart a jail:
bastille restart ALL
Update
As in any FreeBSD system, the operating system update is divided into two parts, the base system and the installed packages/ports.
Update BASE
The base system can be upgraded within the same version 13.2-p1 -> 13.2-p2, the same major version 13.2 -> 13.3, or another major version 13.2 -> 14.0.
The first step will be to obtain the new version of FreeBSD. To do this, we must bootstrap it and apply the latest security patches.
We check the available releases.
13.1-RELEASE
13.2-RELEASE
14.0-RELEASE
When we finish, we can remove the old releases with:
NOTE: If any jail depends on binaries compiled from Poudriere, we must prepare Poudriere beforehand .
Thin jail:
If it’s a Thin jail, they share the base, so updating the RELEASE from Bastille will update the base system of all jails that share that RELEASE.
To determine if a jail is Thick/Thin, you can check the file /usr/local/bastille/jails/JAILNAME/jail.conf. If the osrelease line appears, it’s a thin jail.
-- Atlas: Thick jail
-- DataDyne: Thick jail
-- HellStorm: Thick jail
-- MetaCortex: Thick jail
-- PostgreSQL00-test: Thick jail
-- PostgreSQL01-test: Thick jail
-- PostgreSQLBackups-test: Thick jail
-- PostgreSQLRestore-test: Thick jail
-- RECLog: Thick jail
-- thinJail: Thin jail
-- thinJail2: Thin jail
If it’s an update within the same version, simply execute:
If we are going to update to another version outside of the one we are currently on or to another major version, the process will be longer.
Stop the jail:
We update to the new version as indicated in the
official documentation
:
In this case, we change the path from 13.2 to 14.0:
/usr/local/bastille/releases/13.2-RELEASE /usr/local/bastille/jails/JAILNAME/root/.bastille nullfs ro 0 0
/usr/local/bastille/releases/14.0-RELEASE /usr/local/bastille/jails/JAILNAME/root/.bastille nullfs ro 0 0
Boot the jail:
We reinstall all software depending on whether we use binary packages or ports.
If we see the following error message, we should change the shell to the default before accessing the jail:
>ld-elf.so.1: Shared object "libncursesw.so.8" not found, required by "bash"
pkg-static upgrade -f
exit
git -C /usr/ports pull
cd /usr/ports
make fetchindex
for PORT in $(pkg info|awk \'{print$1}\'); do PORT_PATH=$(pkg info $PORT|grep Origin|awk \'{print$3}\') && echo PORT: $PORT - $PORT_PATH && cd /usr/ports/$PORT_PATH && export BATCH=\"yes\" && make clean reinstall clean; done
exit
We revert the shell change.
We update in case any patches have been released since the last release.
We restart the jail to ensure everything works correctly.
bastille start JAILNAME
Thick jail:
On the contrary, if it’s a Thick jail, we’ll need to update each jail independently, but always from Bastille since the jails start at securelevel 2 and don’t allow it from within.
kern.securelevel: 2
To determine if a jail is Thick/Thin, you can check the file /usr/local/bastille/jails/JAILNAME/jail.conf. If the ‘osrelease’ line appears, it’s a thin jail.
-- Atlas: Thick jail
-- DataDyne: Thick jail
-- HellStorm: Thick jail
-- MetaCortex: Thick jail
-- PostgreSQL00-test: Thick jail
-- PostgreSQL01-test: Thick jail
-- PostgreSQLBackups-test: Thick jail
-- PostgreSQLRestore-test: Thick jail
-- RECLog: Thick jail
-- thinJail: Thin jail
-- thinJail2: Thin jail
To update the thick jails, first, we need to know which release version is the current one.
JID State IP Address Published Ports Hostname Release Path
Atlas Up 192.168.69.19 - Atlas 13.2-RELEASE-p10 /usr/local/bastille/jails/Atlas/root
DataDyne Up 192.168.69.25 - DataDyne 13.2-RELEASE-p10 /usr/local/bastille/jails/DataDyne/root
HellStorm Up 192.168.69.17 - HellStorm 13.2-RELEASE-p10 /usr/local/bastille/jails/HellStorm/root
MetaCortex Up 192.168.69.20 - MetaCortex 13.2-RELEASE-p10 /usr/local/bastille/jails/MetaCortex/root
PostgreSQL00-test Up 192.168.69.26 - PostgreSQL00-test 13.2-RELEASE-p10 /usr/local/bastille/jails/PostgreSQL00-test/root
PostgreSQL01-test Up 192.168.69.27 - PostgreSQL01-test 13.2-RELEASE-p10 /usr/local/bastille/jails/PostgreSQL01-test/root
PostgreSQLBackups-test Up 192.168.69.29 - PostgreSQLBackups-test 13.2-RELEASE-p10 /usr/local/bastille/jails/PostgreSQLBackups-test/root
PostgreSQLRestore-test Up 192.168.69.28 - PostgreSQLRestore-test 13.2-RELEASE-p10 /usr/local/bastille/jails/PostgreSQLRestore-test/root
RECLog Up 192.168.69.21 - RECLog 13.2-RELEASE-p10 /usr/local/bastille/jails/RECLog/root
thinJail Up 192.168.40.190 - thinJail 14.0-RELEASE-p5 /usr/local/bastille/jails/thinJail/root
thinJail2 Up 192.168.40.191 - thinJail2 14.0-RELEASE-p5 /usr/local/bastille/jails/thinJail2/root
If it’s an update within the same version, simply execute:
If we are going to update to another version outside of the one we are currently on or to another major version, the process will be longer. The documentation currently only covers how to update thin jails as of today (03/18/2024), the procedure for updating a thick jail has been deduced from the Bastille man pages and the help command.
man bastille
bastille upgrade
Usage: bastille upgrade release newrelease | target newrelease | target install | [force]
Update it:
Apply the update:
Reboot the jail:
bastille start JAILNAME
Continue with the update:
We reinstall all software depending on whether we use binary packages or ports.
If we see the following error message, we should change the shell to the default before accessing the jail:
>ld-elf.so.1: Shared object "libncursesw.so.8" not found, required by "bash"
pkg-static upgrade -f
exit
git -C /usr/ports pull
cd /usr/ports
make fetchindex
for PORT in $(pkg info|awk \'{print$1}\'); do PORT_PATH=$(pkg info $PORT|grep Origin|awk \'{print$3}\') && echo PORT: $PORT - $PORT_PATH && cd /usr/ports/$PORT_PATH && export BATCH=\"yes\" && make clean reinstall clean; done
exit
We revert the shell change.
We conclude the update.
We update in case any patches have been released since the last release.
We restart the jail to ensure everything works correctly.
bastille start JAILNAME
Update PACKAGES/PORTS
Update binary packages of a jail:
bastille pkg ALL upgrade
bastille pkg ALL autoremove
Update ports of a jail:
git -C /usr/ports pull
for PORT in $(pkg info|awk '{print$1}'); do PORT_PATH=$(pkg info $PORT|grep Origin|awk '{print$3}') && echo PORT: $PORT - $PORT_PATH && cd /usr/ports/$PORT_PATH && export BATCH="yes" && make clean reinstall clean; done
Update script
Here is my update script in case it is useful to someone, keep in mind that all my jails are Thick type:
cecho.sh script
#!/usr/local/bin/bash
source /root/.scripts/cecho.sh
function sendTelegram {
message=${@:1}
curl -s -X POST https://api.telegram.org/bot5XXXXXXXXXXXXXXXX/sendMessage -d chat_id=XXXXXXXXXX -d text="$message"
}
clear
cecho "Cyan" "<== Bastille Jail updater by kr0m ==>"
echo ""
echo ""
FREEBSD_VERSION=$(freebsd-version -u|awk -F "-RELEASE" '{print$1}')
cecho "Cyan" ">> Updating Bastille RELEASE: $FREEBSD_VERSION"
bastille update $FREEBSD_VERSION-RELEASE
echo ""
cecho "Green" "----------------------------------------------------"
DATE=$(date +%d_%m_%Y-%H:%M:%S)
for JAIL in $(bastille list jail); do
cecho "Cyan" ">> Snapshooting: $JAIL@$DATE"
bastille zfs $JAIL snap UPDATE-$DATE
done
echo ""
cecho "Green" "----------------------------------------------------"
cecho "Cyan" ">> Snapshots check:"
for JAIL in $(bastille list jail); do
N=$(bastille zfs $JAIL df|grep UPDATE|awk -F '@' '{print$2}'|awk '{print$1}'|sort -u|wc -l|awk '{print$1}')
if [ $N -le 1 ]; then
continue
fi
let N=$N-1
for SNAPSHOT_TOREMOVE in $(bastille zfs $JAIL df|grep UPDATE|grep -v "root@UPDATE"|awk -F '@' '{print$2}'|awk '{print$1}'|head -n $N); do
cecho "Cyan" ">> Clearing $JAIL@$SNAPSHOT_TOREMOVE"
zfs destroy zroot/bastille/jails/$JAIL@$SNAPSHOT_TOREMOVE
zfs destroy zroot/bastille/jails/$JAIL/root@$SNAPSHOT_TOREMOVE
done
done
echo ""
cecho "Green" "----------------------------------------------------"
for JAIL in $(bastille list jail); do
cecho "Cyan" ">> Updating base: $JAIL"
bastille update $JAIL
done
echo ""
cecho "Green" "----------------------------------------------------"
cecho "Cyan" ">> Updating ALL jails packages"
bastille cmd ALL env ASSUME_ALWAYS_YES=YES pkg upgrade
echo ""
cecho "Green" "----------------------------------------------------"
cecho "Cyan" ">> AutoRemoving ALL jails packages"
bastille cmd ALL env ASSUME_ALWAYS_YES=YES pkg autoremove
echo ""
cecho "Green" "----------------------------------------------------"
cecho "Cyan" ">> Updating ALL jails PIP"
bastille cmd ALL pip install --upgrade pip
echo ""
cecho "Green" "----------------------------------------------------"
cecho "Cyan" ">> Updating SpamAssasin: HellStorm"
bastille cmd HellStorm sa-update -v
bastille service HellStorm sa-spamd restart
cecho "Cyan" ">> Restarting Mail services: HellStorm"
bastille service HellStorm sendmail restart
bastille service HellStorm dovecot restart
echo ""
cecho "Green" "----------------------------------------------------"
cecho "Cyan" ">> Visit: https://mail.alfaexploit.com/?admin#/about"
echo ""
cecho "Cyan" ">> Checking if Mail system still works: HellStorm"
DATE=$(date "+%d/%m/%Y %H:%M:%S")
echo -e "Subject: HellStorm updated: $DATE" | sendmail -f root@alfaexploit.com kr0m@alfaexploit.com
sleep 10
grep -r "HellStorm updated: $DATE" /usr/local/bastille/jails/HellStorm/root/var/mail/kr0m 1>/dev/null
if [ $? -ne 0 ]; then
cecho "Red" "++ ERROR: Mail system is not working"
MESSAGE="ERROR: Mail system is not working"
sendTelegram $MESSAGE
else
cecho "Cyan" ">> Mail system is working"
fi
echo ""
cecho "Green" "----------------------------------------------------"
echo ""
echo ""
cecho "Cyan" ">> Updating owasp-modsecurity: MetaCortex"
bastille cmd MetaCortex bash -c 'cd /usr/local/owasp-modsecurity-crs/ && git pull'
echo ""
cecho "Cyan" ">> Updating Hugo theme: MetaCortex"
bastille cmd MetaCortex bash -c "su -l kr0m -c 'cd /home/kr0m/AlfaExploit/alfaexploit_zzo/themes/zzo && git submodule update --remote --merge'"
echo ""
cecho "Cyan" ">> Redeploying Alfaexploit: MetaCortex"
bastille cmd MetaCortex bash -c "su -l kr0m -c 'cd /home/kr0m/AlfaExploit/alfaexploit_zzo/ && hugo && rm -rf /usr/local/www/alfaexploit/* && cp -r public/* /usr/local/www/alfaexploit/'"
echo ""
cecho "Green" "----------------------------------------------------"
cecho "Cyan" ">> Security check:"
bastille cmd ALL pkg audit -F
Binary packages security
Check for security issues in a jail’s packages:
bastille cmd ALL pkg audit -F
Limiting resources
Using the limits command, we can limit the resources available to a jail. Bastille relies on
rctl
for this.
We enable rctl:
shutdown -r now
We can check the resources to limit at this link :
RESOURCES
cputime CPU time, in seconds
datasize data size, in bytes
stacksize stack size, in bytes
coredumpsize core dump size, in bytes
memoryuse resident set size, in bytes
memorylocked locked memory, in bytes
maxproc number of processes
openfiles file descriptor table size
vmemoryuse address space limit, in bytes
pseudoterminals number of PTYs
swapuse swap space that may be reserved or used, in bytes
nthr number of threads
msgqqueued number of queued SysV messages
msgqsize SysV message queue size, in bytes
nmsgq number of SysV message queues
nsem number of SysV semaphores
nsemop number of SysV semaphores modified in a single
semop(2) call
nshm number of SysV shared memory segments
shmsize SysV shared memory size, in bytes
wallclock wallclock time, in seconds
pcpu %CPU, in percents of a single CPU core
readbps filesystem reads, in bytes per second
writebps filesystem writes, in bytes per second
readiops filesystem reads, in operations per second
writeiops filesystem writes, in operations per second
If we want to check the limits:
jail:aliasJail:memoryuse:deny=1g/jail
jail:aliasJail:memoryuse:log=1g/jail
Or for all jails using the command:
jail:aliasJail:memoryuse:log=1024M
jail:aliasJail:memoryuse:deny=1024M
If we want to remove them:
rctl -r jail:aliasJail:memoryuse:log=1024M
rctl -r jail:aliasJail:memoryuse:deny=1024M
We can also remove all restrictions:
rctl -r :
NOTE: Keep in mind that through ZFS parameters we can also limit certain aspects such as disk quota.
ZFS Parameters: Quotas/Snapshots/df
The zfs command allows us to assign any zfs parameter to the dataset of a specific jail, for example, we can limit the available disk space:
If we check the available space, we will see that it is close to 1G:
[aliasJail]:
NAME USED AVAIL REFER MOUNTPOINT COMPRESS RATIO
zroot/bastille/jails/aliasJail 88.2M 936M 108K /usr/local/bastille/jails/aliasJail lz4 1.75x
zroot/bastille/jails/aliasJail/root 88.1M 936M 88.1M /usr/local/bastille/jails/aliasJail/root lz4 1.75x
We can also assign values to ZFS parameters massively:
bastille zfs ALL df
Bastille allows us to create snapshots:
To check them:
[aliasJail]:
NAME USED AVAIL REFER MOUNTPOINT COMPRESS RATIO
zroot/bastille/jails/aliasJail 88.2M 936M 108K /usr/local/bastille/jails/aliasJail lz4 1.75x
zroot/bastille/jails/aliasJail@SNAP0 0B - 108K - - 1.02x
zroot/bastille/jails/aliasJail/root 88.1M 936M 88.1M /usr/local/bastille/jails/aliasJail/root lz4 1.75x
zroot/bastille/jails/aliasJail/root@SNAP0 0B - 88.1M - - 1.75x
This command can also be applied to all jails:
bastille zfs ALL df
If we want to check the date on which the snapshots were created, we must do it directly through ZFS commands:
NAME CREATION
zroot/bastille/jails/Paradox@UPDATE-18_01_2023-22:50:51 Wed Jan 18 22:50 2023
zroot/bastille/jails/Paradox/root@UPDATE-18_01_2023-22:50:51 Wed Jan 18 22:50 2023
Deleting snapshots is a manual process, first, we must locate the name of the snapshot using the df command and then execute the ZFS commands:
zfs destroy zroot/bastille/jails/aliasJail/root@SNAP0
As there are two datasets per jail, a quick way to delete both would be to use the following loop:
Restoring snapshots is also a manual task, first, we must locate the name of the snapshot using the df command and then execute the ZFS commands:
zfs rollback -r zroot/bastille/jails/aliasJail/root@SNAP0
NOTE: Remember that in ZFS, if we have, for example, 5 snapshots (0,1,2,3,4) and we want to revert to the first one (0), snapshots 1,2,3,4 will be lost.
Jails Configuration
We can edit the configuration of a jail using the edit command:
aliasJail {
devfs_ruleset = 4;
enforce_statfs = 2;
exec.clean;
exec.consolelog = /var/log/bastille/aliasJail_console.log;
exec.start = '/bin/sh /etc/rc';
exec.stop = '/bin/sh /etc/rc.shutdown';
host.hostname = aliasJail;
mount.devfs;
mount.fstab = /usr/local/bastille/jails/aliasJail/fstab;
path = /usr/local/bastille/jails/aliasJail/root;
securelevel = 2;
interface = bge1;
ip4.addr = 192.168.40.136;
ip6 = disable;
}
Or consult/modify specific parameters using the config get/set command:
192.168.40.136
bastille restart aliasJail
Migration from IOCage to Bastille
It should be noted that jail migration has caused problems, at least when migrating an HAProxy. Refer to the troubleshooting section in this same article for more details.
Stop the jail in IOCage:
Export it:
Copy the image to the Bastille server:
Import the image:
We can see that there is a new jail called HAProxy:
JID State IP Address Published Ports Hostname Release Path
HAProxy Down nfe0|192.168.69.11 - HAProxy 13.1-RELEASE-p5 /usr/local/bastille/jails/HAProxy/root
aliasJail Up 192.168.40.136 - aliasJail 13.1-RELEASE-p5 /usr/local/bastille/jails/aliasJail/root
aliasJail2 Up 192.168.40.139 - aliasJail2 13.1-RELEASE-p5 /usr/local/bastille/jails/aliasJail2/root
jailVnet2 Up 192.168.40.138 - jailVnet2 13.1-RELEASE-p5 /usr/local/bastille/jails/jailVnet2/root
natJail Up 10.17.89.50 - natJail 13.1-RELEASE-p5 /usr/local/bastille/jails/natJail/root
nginx Up 192.168.40.140 - nginx 13.1-RELEASE-p5 /usr/local/bastille/jails/nginx/root
Depending on the network configuration we want to use in the jail, we will make the relevant adjustments:
NAT:
interface = bastille0;
ip4.addr = 10.17.89.50;
ALIAS:
interface = bge1;
ip4.addr = 192.168.40.136;
BRIDGE:
vnet;
vnet.interface = "e0b_jailVnet2";
exec.prestart += "ifconfig epair0 create";
exec.prestart += "ifconfig bridge0 addm epair0a";
exec.prestart += "ifconfig epair0a up name e0a_jailVnet2";
exec.prestart += "ifconfig epair0b up name e0b_jailVnet2";
exec.poststop += "ifconfig bridge0 deletem e0a_jailVnet2";
exec.poststop += "ifconfig e0a_jailVnet2 destroy";
In bridge mode, we will also have to perform the configuration inside the jail.
In my case, it is a jail with an IP address configured as an alias:
interface = bge1;
ip4.addr = 192.168.40.198;
Start the jail:
Now it has the correct IP address:
JID State IP Address Published Ports Hostname Release Path
HAProxy Up 192.168.40.198 - HAProxy.alfaexploit.com 13.1-RELEASE-p5 /usr/local/bastille/jails/HAProxy/root
aliasJail Up 192.168.40.136 - aliasJail 13.1-RELEASE-p5 /usr/local/bastille/jails/aliasJail/root
aliasJail2 Up 192.168.40.139 - aliasJail2 13.1-RELEASE-p5 /usr/local/bastille/jails/aliasJail2/root
jailVnet2 Up 192.168.40.138 - jailVnet2 13.1-RELEASE-p5 /usr/local/bastille/jails/jailVnet2/root
natJail Up 10.17.89.50 - natJail 13.1-RELEASE-p5 /usr/local/bastille/jails/natJail/root
nginx Up 192.168.40.140 - nginx 13.1-RELEASE-p5 /usr/local/bastille/jails/nginx/root
Bash autoCompletion
We install the software:
We tell Bash to load the autocompletion:
[[ $PS1 && -f /usr/local/share/bash-completion/bash_completion.sh ]] && source /usr/local/share/bash-completion/bash_completion.sh
The autocompletion script would be the following:
_bastille () {
local cur
# cur: Current word where cursor is located
cur=${COMP_WORDS[COMP_CWORD]}
# Bastille first level commands:
if [ ${#COMP_WORDS[@]} -eq 2 ]; then
# Show only the options that start with $cur
COMPREPLY=($(compgen -W "$(bastille --help|grep '^ '|grep -v 'bastille command TARGET \[args\]'|sed 's/^[[:space:]]*//g'|sed 's/[[:space:]].*//g')" -- $cur))
return 0
# Bastille second level commands:
elif [ ${#COMP_WORDS[@]} -eq 3 ]; then
case ${COMP_WORDS[1]} in
cmd)
JAILS=$(bastille list|grep -v 'JID IP Address Hostname Path'|sed 's/^[[:space:]]*//g'|sed 's/[[:space:]].*//g')
COMPREPLY=($(compgen -W "$JAILS" -- $cur))
;;
clone)
JAILS=$(bastille list -a|grep -v 'JID State IP Address Published Ports Hostname Release Path'|sed 's/^[[:space:]]*//g'|sed 's/[[:space:]].*//g')
COMPREPLY=($(compgen -W "$JAILS" -- $cur))
;;
config)
JAILS=$(bastille list -a|grep -v 'JID State IP Address Published Ports Hostname Release Path'|sed 's/^[[:space:]]*//g'|sed 's/[[:space:]].*//g')
COMPREPLY=($(compgen -W "$JAILS" -- $cur))
;;
console)
JAILS=$(bastille list|grep -v 'JID IP Address Hostname Path'|sed 's/^[[:space:]]*//g'|sed 's/[[:space:]].*//g')
COMPREPLY=($(compgen -W "$JAILS" -- $cur))
;;
convert)
JAILS=$(bastille list -a|grep -v 'JID State IP Address Published Ports Hostname Release Path'|sed 's/^[[:space:]]*//g'|sed 's/[[:space:]].*//g')
COMPREPLY=($(compgen -W "$JAILS" -- $cur))
;;
cp)
JAILS=$(bastille list -a|grep -v 'JID State IP Address Published Ports Hostname Release Path'|sed 's/^[[:space:]]*//g'|sed 's/[[:space:]].*//g')
COMPREPLY=($(compgen -W "$JAILS" -- $cur))
;;
create)
CREATE_OPTIONS=$(bastille create 2>/dev/null|grep -v 'Options:'|awk '{print$1"\n"$3}'|sed '/^[[:space:]]*$/d')
COMPREPLY=($(compgen -W "$CREATE_OPTIONS" -- $cur))
;;
destroy)
JAILS=$(bastille list -a|grep -v 'JID State IP Address Published Ports Hostname Release Path'|sed 's/^[[:space:]]*//g'|sed 's/[[:space:]].*//g')
COMPREPLY=($(compgen -W "$JAILS" -- $cur))
;;
edit)
JAILS=$(bastille list -a|grep -v 'JID State IP Address Published Ports Hostname Release Path'|sed 's/^[[:space:]]*//g'|sed 's/[[:space:]].*//g')
COMPREPLY=($(compgen -W "$JAILS" -- $cur))
;;
export)
EXPORT_OPTIONS=$(bastille export 2>/dev/null|grep -v 'Options:'|grep -v 'Note: If no export option specified, the container should be redirected to standard output.'|grep -v '|'|sed '/^[[:space:]]*$/d'|awk '{print$1}')
EXPORT_OPTIONS=$EXPORT_OPTIONS" "$(bastille export 2>/dev/null|grep -v 'Options:'|grep -v 'Note: If no export option specified, the container should be redirected to standard output.'|grep '|'|sed '/^[[:space:]]*$/d'|awk '{print$1"\n"$3}')
COMPREPLY=($(compgen -W "$EXPORT_OPTIONS" -- $cur))
;;
htop)
JAILS=$(bastille list|grep -v 'JID IP Address Hostname Path'|sed 's/^[[:space:]]*//g'|sed 's/[[:space:]].*//g')
COMPREPLY=($(compgen -W "$JAILS" -- $cur))
;;
import)
IMPORT_OPTIONS=$(bastille import 2>/dev/null|grep -v 'Options:'|grep -v 'Tip: If no option specified, container should be imported from standard input.'|awk '{print$1"\n"$3}'|sed '/^[[:space:]]*$/d')
COMPREPLY=($(compgen -W "$IMPORT_OPTIONS" -- $cur))
;;
limits)
JAILS=$(bastille list -a|grep -v 'JID State IP Address Published Ports Hostname Release Path'|sed 's/^[[:space:]]*//g'|sed 's/[[:space:]].*//g')
COMPREPLY=($(compgen -W "$JAILS" -- $cur))
;;
list)
OPTIONS_LIST='-j -a release template log limit import export backup'
COMPREPLY=($(compgen -W "$OPTIONS_LIST" -- $cur))
;;
mount)
JAILS=$(bastille list -a|grep -v 'JID State IP Address Published Ports Hostname Release Path'|sed 's/^[[:space:]]*//g'|sed 's/[[:space:]].*//g')
COMPREPLY=($(compgen -W "$JAILS" -- $cur))
;;
pkg)
JAILS=$(bastille list|grep -v 'JID IP Address Hostname Path'|sed 's/^[[:space:]]*//g'|sed 's/[[:space:]].*//g')
COMPREPLY=($(compgen -W "$JAILS" -- $cur))
;;
rdr)
JAILS=$(bastille list|grep -v 'JID IP Address Hostname Path'|sed 's/^[[:space:]]*//g'|sed 's/[[:space:]].*//g')
COMPREPLY=($(compgen -W "$JAILS" -- $cur))
;;
rename)
JAILS=$(bastille list -a|grep -v 'JID State IP Address Published Ports Hostname Release Path'|sed 's/^[[:space:]]*//g'|sed 's/[[:space:]].*//g')
COMPREPLY=($(compgen -W "$JAILS" -- $cur))
;;
restart)
JAILS=$(bastille list|grep -v 'JID IP Address Hostname Path'|sed 's/^[[:space:]]*//g'|sed 's/[[:space:]].*//g')
COMPREPLY=($(compgen -W "$JAILS" -- $cur))
;;
service)
JAILS=$(bastille list|grep -v 'JID IP Address Hostname Path'|sed 's/^[[:space:]]*//g'|sed 's/[[:space:]].*//g')
COMPREPLY=($(compgen -W "$JAILS" -- $cur))
;;
start)
JAILS=$(bastille list -a|grep 'Down'|sed 's/^[[:space:]]*//g'|sed 's/[[:space:]].*//g')
COMPREPLY=($(compgen -W "$JAILS" -- $cur))
;;
stop)
JAILS=$(bastille list|grep -v 'JID IP Address Hostname Path'|sed 's/^[[:space:]]*//g'|sed 's/[[:space:]].*//g')
COMPREPLY=($(compgen -W "$JAILS" -- $cur))
;;
sysrc)
JAILS=$(bastille list|grep -v 'JID IP Address Hostname Path'|sed 's/^[[:space:]]*//g'|sed 's/[[:space:]].*//g')
COMPREPLY=($(compgen -W "$JAILS" -- $cur))
;;
template)
JAILS=$(bastille list|grep -v 'JID IP Address Hostname Path'|sed 's/^[[:space:]]*//g'|sed 's/[[:space:]].*//g')
COMPREPLY=($(compgen -W "$JAILS" -- $cur))
;;
top)
JAILS=$(bastille list|grep -v 'JID IP Address Hostname Path'|sed 's/^[[:space:]]*//g'|sed 's/[[:space:]].*//g')
COMPREPLY=($(compgen -W "$JAILS" -- $cur))
;;
umount)
JAILS=$(bastille list -a|grep -v 'JID State IP Address Published Ports Hostname Release Path'|sed 's/^[[:space:]]*//g'|sed 's/[[:space:]].*//g')
COMPREPLY=($(compgen -W "$JAILS" -- $cur))
;;
update)
UPDATE_LIST=$(bastille list release)
UPDATE_LIST_TMP=$(bastille list container)
UPDATE_LIST=$UPDATE_LIST" "$UPDATE_LIST_TMP
UPDATE_LIST_TMP=$(for DIR in $(ls -l /usr/local/bastille/templates|grep -v default|awk '{print$9}'); do TEMPLATENAME=$(ls -l /usr/local/bastille/templates/$DIR|awk '{print$9}') && echo "$DIR/$TEMPLATENAME"; done)
UPDATE_LIST=$UPDATE_LIST" "$UPDATE_LIST_TMP
COMPREPLY=($(compgen -W "$UPDATE_LIST" -- $cur))
;;
upgrade)
UPGRADE_LIST=$(bastille list release)
COMPREPLY=($(compgen -W "$UPGRADE_LIST" -- $cur))
;;
verify)
VERIFY_LIST=$(bastille list release)
VERIFY_LIST_TMP=$(for DIR in $(ls -l /usr/local/bastille/templates|grep -v default|awk '{print$9}'); do TEMPLATENAME=$(ls -l /usr/local/bastille/templates/$DIR|awk '{print$9}') && echo "$DIR/$TEMPLATENAME"; done)
VERIFY_LIST=$VERIFY_LIST" "$VERIFY_LIST_TMP
COMPREPLY=($(compgen -W "$VERIFY_LIST" -- $cur))
;;
zfs)
JAILS=$(bastille list -a|grep -v 'JID State IP Address Published Ports Hostname Release Path'|sed 's/^[[:space:]]*//g'|sed 's/[[:space:]].*//g')
COMPREPLY=($(compgen -W "$JAILS" -- $cur))
;;
esac
# Bastille third level commands:
elif [ ${#COMP_WORDS[@]} -eq 4 ]; then
case ${COMP_WORDS[1]} in
config)
CONFIG_COMMANDS="set get"
COMPREPLY=($(compgen -W "$CONFIG_COMMANDS" -- $cur))
;;
export)
JAILS=$(bastille list -a|grep -v 'JID State IP Address Published Ports Hostname Release Path'|sed 's/^[[:space:]]*//g'|sed 's/[[:space:]].*//g')
COMPREPLY=($(compgen -W "$JAILS" -- $cur))
;;
pkg)
JAILS=$(bastille list -a|grep -v 'JID State IP Address Published Ports Hostname Release Path'|sed 's/^[[:space:]]*//g'|sed 's/[[:space:]].*//g')
COMPREPLY=($(compgen -W "$JAILS" -- $cur))
;;
template)
TEMPLATE_LIST=$(for DIR in $(ls -l /usr/local/bastille/templates|grep -v default|awk '{print$9}'); do TEMPLATENAME=$(ls -l /usr/local/bastille/templates/$DIR|awk '{print$9}') && echo "$DIR/$TEMPLATENAME"; done)
COMPREPLY=($(compgen -W "$TEMPLATE_LIST" -- $cur))
;;
zfs)
ZFS_LIST='set get snap'
COMPREPLY=($(compgen -W "$ZFS_LIST" -- $cur))
;;
esac
else
return 0
fi
}
complete -F _bastille bastille
Jails auto start
By default, it starts all of them:
: ${bastille_list:="ALL"}
We can define a list of jails to start:
To disable the auto start of all jails, it is best to disable the entire service:
If we want to start them all again, we just have to remove the bastille_list parameter:
Linux Jails
This functionality is in experimental phase, some things fail and the system works partially so I do not recommend under any circumstances to put a Linux in production through this system, if we need a Linux by force it is preferable to mount it through vm-bhyve since being a complete virtual machine it should not give any problem, in case of using Bastille/VM-Bhyve I recommend having two physical network interfaces to be able to have a bridge for the jails and another for the virtual machines and thus avoid problems between softwares.
Some drawbacks found in a basic test are:
-
Only allows configuring the network in NAT/Alias mode
-
dmesg does not work:
root@aliasLinux:~# dmesg
dmesg: read kernel buffer failed: Operation not permitted
- Some network tools fail:
root@aliasLinux:~# netstat -nputa
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
netstat: no support for `AF INET (tcp)' on this system.
- Reboot fails:
root@aliasLinux:~# reboot
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down
Failed to talk to init daemon.
- SSH access does not work due to something related to /dev/pts:
ssh kr0m@192.168.40.141 -p22
kr0m@192.168.40.141's password:
PTY allocation request failed on channel 0
Welcome to Ubuntu 20.04 LTS (GNU/Linux 3.17.0 x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
- It only supports a very limited number of distributions:
#adding Ubuntu Bionic as valid "RELEASE" for POC @hackacad
ubuntu_bionic|bionic|ubuntu-bionic)
PLATFORM_OS="Ubuntu/Linux"
LINUX_FLAVOR="bionic"
DIR_BOOTSTRAP="Ubuntu_1804"
ARCH_BOOTSTRAP=${HW_MACHINE_ARCH_LINUX}
debootstrap_release
;;
ubuntu_focal|focal|ubuntu-focal)
PLATFORM_OS="Ubuntu/Linux"
LINUX_FLAVOR="focal"
DIR_BOOTSTRAP="Ubuntu_2004"
ARCH_BOOTSTRAP=${HW_MACHINE_ARCH_LINUX}
debootstrap_release
;;
debian_stretch|stretch|debian-stretch)
PLATFORM_OS="Debian/Linux"
LINUX_FLAVOR="stretch"
DIR_BOOTSTRAP="Debian9"
ARCH_BOOTSTRAP=${HW_MACHINE_ARCH_LINUX}
debootstrap_release
;;
debian_buster|buster|debian-buster)
PLATFORM_OS="Debian/Linux"
LINUX_FLAVOR="buster"
DIR_BOOTSTRAP="Debian10"
ARCH_BOOTSTRAP=${HW_MACHINE_ARCH_LINUX}
debootstrap_release
;;
debian_bullseye|bullseye|debian-bullseye)
PLATFORM_OS="Debian/Linux"
LINUX_FLAVOR="bullseye"
DIR_BOOTSTRAP="Debian11"
ARCH_BOOTSTRAP=${HW_MACHINE_ARCH_LINUX}
debootstrap_release
;;
If, despite all the problems, we decide to try Linux jails, the first step is to download the image:
It will ask several questions about the necessary configuration in the system for Linux to run, we answer yes to everything:
fdescfs not enabled in /boot/loader.conf, Should I do that for you? (N|y)
y
Loading kernel module: fdescfs
Loaded fdescfs, id=17
Persisting module: fdescfs
fdescfs_load: -> YES
linprocfs not enabled in /boot/loader.conf, Should I do that for you? (N|y)
y
Loading kernel module: linprocfs
Loaded linprocfs, id=18
Persisting module: linprocfs
linprocfs_load: -> YES
linsysfs not enabled in /boot/loader.conf, Should I do that for you? (N|y)
y
Loading kernel module: linsysfs
Loaded linsysfs, id=20
Persisting module: linsysfs
linsysfs_load: -> YES
tmpfs not enabled in /boot/loader.conf, Should I do that for you? (N|y)
y
Persisting module: tmpfs
tmpfs_load: -> YES
Loading kernel module: linux
Loaded linux, id=21
Loading kernel module: linux64
Loaded linux64, id=22
linux_enable: NO -> YES
Debootstrap not found. Should it be installed? (N|y)
y
...
...
debootstrap bionic /compat/ubuntu
Bootstrapping Ubuntu/Linux distfiles...
...
...
Bootstrap successful.
Now we can see a new “release”:
13.1-RELEASE
Ubuntu_2004
We create the jail indicating that it will be of type linux:
We can see that it has started correctly:
JID State IP Address Published Ports Hostname Release Path
HAProxy Up 192.168.40.198 - HAProxy.alfaexploit.com 13.1-RELEASE-p5 /usr/local/bastille/jails/HAProxy/root
NATJail Up 10.17.89.50 tcp/2001:22 NATJail 13.1-RELEASE-p5 /usr/local/bastille/jails/NATJail/root
aliasJail Up 192.168.40.136 - aliasJail 13.1-RELEASE-p5 /usr/local/bastille/jails/aliasJail/root
aliasJail2 Up 192.168.40.139 - aliasJail2 13.1-RELEASE-p5 /usr/local/bastille/jails/aliasJail2/root
aliasLinux Up 192.168.40.141 - aliasLinux focal (Ubuntu 20.04) /usr/local/bastille/jails/aliasLinux/root
jailVnet2 Up 192.168.40.138 - jailVnet2 13.1-RELEASE-p5 /usr/local/bastille/jails/jailVnet2/root
nginx Up 192.168.40.140 - nginx 13.1-RELEASE-p5 /usr/local/bastille/jails/nginx/root
thickJail Up 192.168.40.190 - thickJail 13.1-RELEASE-p5 /usr/local/bastille/jails/thickJail/root
We access the jail:
[aliasLinux]:
Welcome to Ubuntu 20.04 LTS (GNU/Linux 3.17.0 x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.
root@aliasLinux:~# uname -a
Linux aliasLinux 3.17.0 FreeBSD 13.1-RELEASE-p3 GENERIC x86_64 x86_64 x86_64 GNU/Linux
root@aliasLinux:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04 LTS
Release: 20.04
Codename: focal
Troubleshooting:
Bastille works quite well, I have only detected some anomalous behavior very punctually when doing more complicated things.
Logs
An easy way to see if a jail is causing problems is to consult its log file:
Import problems
- We import a jail that already exists, logically it gives an error.
- We delete the jail.
- We import the jail again, this time successfully.
- We delete the jail, it gives an RO error of the base, a bastille list DOES NOT show the jail.
The problem is that it left remains and this causes problems.
zroot/bastille/jails/aliasJail2 on /usr/local/bastille/jails/aliasJail2 (zfs, local, noatime, nfsv4acls)
zroot/bastille/jails/aliasJail2/root on /usr/local/bastille/jails/aliasJail2/root (zfs, local, noatime, nfsv4acls)
/usr/local/bastille/releases/13.1-RELEASE on /usr/local/bastille/jails/aliasJail2/root/.bastille (nullfs, local, noatime, read-only, nfsv4acls)
We unmount all mount points:
umount /usr/local/bastille/jails/aliasJail2/root
umount /usr/local/bastille/jails/aliasJail2
If an umount gives problems, we force it from ZFS:
We delete the directory:
We destroy the dataset associated with the jail:
Import issues from IOCage
While importing IOCage jails to Bastille, we detected a problem. If we restart the parent server, it starts the jails before the network is functioning, causing problems when shutting down the jails:
[HAProxy]:
jail: HAProxy: /bin/sh /etc/rc.shutdown: exited on signal 9
If we access the jail, we can see the HAProxy process:
nobody 22059 0.0 0.5 31152 10956 - SsJ 08:38 0:00.19 /usr/local/sbin/haproxy -q -f /usr/local/etc/haproxy.conf -p /var/run/haproxy.pid
But it doesn’t have any socket listening:
USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS PATH STATE CONN STATE
root sendmail 18066 5 tcp4 192.168.40.198:25 *:* LISTEN
root sendmail 18066 6 tcp4 192.168.40.198:587 *:* LISTEN
root sshd 17328 3 tcp4 192.168.40.198:22 *:* LISTEN
root syslogd 398 5 udp4 192.168.40.198:514 *:*
If we kill the service and start it again, everything works normally, but to avoid doing this on every startup, we can configure netwait. This way, services wait for the network interface to be up and the IP we indicate to be reachable by ping before starting the services.
If set to “YES”, delays the start of network-reliant services until netwait_if is up and ICMP packets to a
destination defined in netwait_ip are flowing. Link state is examined first, followed by “pinging” an IP address to verify
network usability. If no destination can be reached or timeouts are exceeded, network services are started anyway
with no guarantee that the network is usable. Use of this variable requires both netwait_ip and netwait_if to be set.
The operating system will wait for a while, but if it exceeds the time, the services will start anyway, with no guarantee that the network is usable. In our case, it is enough to avoid the problem of starting Bastille jails.
If no destination can be reached or timeouts are exceeded, network services are started anyway with no guarantee that the network is usable.
We proceed with the configuration:
sysrc netwait_if=‘bge1’
sysrc netwait_ip=‘192.168.40.1’
Orphaned IP alias issues
Sometimes, when stopping and destroying a jail, the IP remains configured as an alias. When reusing the IP in a new jail, Bastille complains:
Warning: IP address already in use (192.168.40.136).
Valid: (bge1).
Creating a thinjail...
Error: IP address (192.168.40.136) already in use.
[aliasJail]: Not started. See 'bastille start aliasJail'.
[aliasJail]: Not started. See 'bastille start aliasJail'.
Error: IP address (192.168.40.136) already in use.
We check that there is no other jail started with that IP:
JID State IP Address Published Ports Hostname Release Path
HAProxy Up 192.168.40.198 - HAProxy.alfaexploit.com 13.1-RELEASE-p5 /usr/local/bastille/jails/HAProxy/root
NATJail Up 10.17.89.50 - NATJail 13.1-RELEASE-p5 /usr/local/bastille/jails/NATJail/root
aliasJail Down 192.168.40.136 - aliasJail 13.1-RELEASE-p5 /usr/local/bastille/jails/aliasJail/root
aliasJail2 Up 192.168.40.139 - aliasJail2 13.1-RELEASE-p5 /usr/local/bastille/jails/aliasJail2/root
jailVnet2 Up 192.168.40.138 - jailVnet2 13.1-RELEASE-p5 /usr/local/bastille/jails/jailVnet2/root
nginx Up 192.168.40.140 - nginx 13.1-RELEASE-p5 /usr/local/bastille/jails/nginx/root
thickJail Down 192.168.40.190 - thickJail 13.1-RELEASE-p5 /usr/local/bastille/jails/thickJail/root
As there is no jail configured with the IP, we can safely remove the alias manually:
We start the jail:
Issues with partially created jails
If for some reason a VNET jail cannot be created, the network interfaces may be partially configured. In such a case, we must unconfigure them and run the command again:
Inmortal jail
If any jail cant be stopped, we should execute a KILL command from inside jail:
JID IP Address Hostname Path
135 192.168.69.22 PostgreSQL00-test /usr/local/bastille/jails/PostgreSQL00-test/root
136 192.168.69.23 PostgreSQL01-test /usr/local/bastille/jails/PostgreSQL01-test/root
138 192.168.69.26 test /usr/local/bastille/jails/test/root
140 192.168.69.19 Atlas /usr/local/bastille/jails/Atlas/root
141 192.168.69.25 DataDyne /usr/local/bastille/jails/DataDyne/root
142 192.168.69.17 HellStorm /usr/local/bastille/jails/HellStorm/root
143 192.168.69.20 MetaCortex /usr/local/bastille/jails/MetaCortex/root
144 192.168.69.18 Paradox /usr/local/bastille/jails/Paradox/root
145 192.168.69.21 RECLog /usr/local/bastille/jails/RECLog/root
We execute the KILL command:
kill -KILL -1
exit
The jail should be stopped:
JID State IP Address Published Ports Hostname Release Path
Atlas Up 192.168.69.19 - Atlas 13.2-RELEASE-p1 /usr/local/bastille/jails/Atlas/root
DataDyne Up 192.168.69.25 - DataDyne 13.2-RELEASE-p1 /usr/local/bastille/jails/DataDyne/root
HellStorm Up 192.168.69.17 - HellStorm 13.2-RELEASE-p1 /usr/local/bastille/jails/HellStorm/root
MetaCortex Up 192.168.69.20 - MetaCortex 13.2-RELEASE-p1 /usr/local/bastille/jails/MetaCortex/root
Paradox Up 192.168.69.18 - Paradox 13.2-RELEASE-p1 /usr/local/bastille/jails/Paradox/root
PostgreSQL00-test Down 192.168.69.22 - PostgreSQL00-test 13.2-RELEASE-p1 /usr/local/bastille/jails/PostgreSQL00-test/root
PostgreSQL01-test Up 192.168.69.23 - PostgreSQL01-test 13.2-RELEASE-p1 /usr/local/bastille/jails/PostgreSQL01-test/root
RECLog Up 192.168.69.21 - RECLog 13.2-RELEASE-p1 /usr/local/bastille/jails/RECLog/root
test Up 192.168.69.26 - test 13.2-RELEASE-p1 /usr/local/bastille/jails/test/root
If we are going to deletes a jail it is recomended to stop it and before deleting it, execute the sync command:
sync
bastille destroy JAILNAME