This page looks best with JavaScript enabled

Managing Bhyve VMs on FreeBSD with Iohyve

 ·  🎃 kr0m

Iohyve is a Bhyve virtual machine manager that uses ZFS functionalities to offer a simpler and more intuitive user experience. Its features include the ability to take snapshots, manage ISOs, integrated VNC server, cloning, and VM backup. It could be said that it is the equivalent of Iocage but for Bhyve VMs. On the other hand, if we prefer a unified interface, we can always use CBSD .

This manual is composed of different parts:


Installation

We install everything we need:

pkg install iohyve grub2-bhyve bhyve-firmware

If we have more than one zpool, we must tell Iohyve which one to use:

zpool list

NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT  
zroot   109G  78.5G  30.5G        -         -    44%    71%  1.00x  ONLINE  -
iohyve setup pool=zroot

The Iohyve network works through bridging, we must indicate the network interface that we want to put inside the bridge, so when we create a VM, the tap interface of that VM will be attached to the existing bridge:

iohyve setup net=em0

We allow Iohyve to load the kernel modules it needs:

iohyve setup kmod=1

To make it permanent at server startup, we can use the following RC parameters:

sysrc iohyve_enable="yes"
sysrc iohyve_flags=“kmod=1 net=em0”

We restart to make sure everything loads automatically:

reboot


Downloading ISO images

To perform the OS installation, we need the installation ISO image, which we download using the fetchiso command:

We rename it to make it easier to write:

iohyve renameiso kali-linux-2020.3-installer-amd64.iso kali2020.3.iso

We list the available ISOs:

iohyve isolist

Listing ISO's...  
kali2020.3.iso

Creating VMs

We create the VM with a 15G disk:

iohyve create kali 15G

NOTE: The virtual disk space reservation is NOT dynamic, which means that we must have the indicated space available at the time of creating the VM.


Installation via serial port:

We assign the VM parameters:

iohyve set kali ram=2048M loader=grub-bhyve os=debian

The os parameter does not indicate the OS to install, but only some adjustments that grub-bhyve will take into account to boot the VM, for example, debian is any Linux without LVM.

The possible options are: openbsd57, openbsd58, openbsd59, openbsd60, openbsd61, netbsd, debian (non-LVM), d8lvm (LVM installs), centos6, centos7 (non-XFS), ubuntu (LVM installs, non-LVM use debian), arch, and custom.

The custom option means that the files /iohyve/VM_NAME/device.map and /iohyve/VM_NAME/grub.cfg will be read at boot.

NOTE: If we are going to install a FreeBSD system, it is not necessary to specify the loader/os parameters.

Before installing the OS, we leave a console ready and waiting:

iohyve console kali

In a second terminal, we start the installation:

iohyve install kali kali2020.3.iso

We will see that the OS installation wizard appears in the first terminal, we select the second option:

GNU GRUB version 2.00

+--------------------------------------------------------------------------+
|Graphical install                                                         |
|Install                                                                   |
|Advanced options ...                                                      |
|Accessible dark contrast installer menu ...                               |
|Install with speech synthesis                                             |
|                                                                          |
|                                                                          |
|                                                                          |
|                                                                          |
|                                                                          |
|                                                                          |
|                                                                          |
+--------------------------------------------------------------------------+

 Use the ^ and v keys to select which entry is highlighted.
 Press enter to boot the selected OS, `e' to edit the commands
 before booting or `c' for a command-line.

After finishing the installation, we will have the VM turned off, we start it and access the console:

iohyve console kali
iohyve start kali

kali login: kr0m  
Password:  
  
Linux kali 5.7.0-kali1-amd64 #1 SMP Debian 5.7.6-1kali2 (2020-07-01) x86_64  
The programs included with the Kali GNU/Linux system are free software;  
the exact distribution terms for each program are described in the  
individual files in /usr/share/doc/*/copyright.  
  
Kali GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent  
permitted by applicable law.  
Last login: Sun Nov 8 10:59:10 CET 2020 on ttyS0  
  
kr0m@kali:~$

To exit the console, we write Alt+4.: ~.


Installation via VNC

By default, Iohyve uses the serial console to access the OS, in most cases it will be more than enough since if we need to start any graphical application through ssh-Xforward we can do it without having to start the X.

Another way to get access to the X is by installing a VNC server in the VM itself, this way we would no longer depend on Iohyve’s VNC.

But if we need graphical access in the OS installation as with Windows, we will have to enable access by VNC through the uefi loader, so that it is available we must install the bhyve-firmware package as done at the beginning of the manual and import the firmware file with the cpfw command.

iohyve cpfw /usr/local/share/uefi-firmware/BHYVE_UEFI.fd
iohyve fwlist

Listing Firmware...  
BHYVE_UEFI.fd

Now we must decide whether we want to sacrifice an IP address from the network for VNC access or if we bind the service to the loopback and access via Ssh tunnel, through loopback access will be safer since it will only be exposed when we mount the tunnel and we will not waste IP addresses but we will have to configure a different VNC port for each VM.


VNC-Loopback

We create the VM:

iohyve create kaliVNC 15G

We set the necessary parameters:

iohyve set kaliVNC ram=2048M loader=uefi vnc=YES fw=BHYVE_UEFI.fd

We start the installation:

iohyve install kaliVNC kali2020.3.iso

We see that there is a service on the loopback on port 5900:

sockstat -sv|grep LISTEN|grep bhyve

root bhyve 9029 7 tcp4 127.0.0.1:5900 *:* LISTEN

We set up the Ssh tunnel to access the VNC from outside:

ssh -L 5900:127.0.0.1:5900 -A -g kr0m@PHYSICAL_HOST

We start our VNC client attacking our PC which will take us via the tunnel to the VM’s VNC:

vinagre 127.0.0.1:5900

If we create a second VM we will have to choose a different VNC port:

iohyve create kaliVNC2 15G
iohyve set kaliVNC2 ram=2048M loader=uefi vnc=YES vnc_port=5901 fw=BHYVE_UEFI.fd
iohyve install kaliVNC2 kali2020.3.iso

We see that there are two services bound to the loopback but on different ports:

sockstat -sv|grep LISTEN|grep bhyve

root bhyve 4166 7 tcp4 127.0.0.1:5901 *:* LISTEN  
root bhyve 9029 7 tcp4 127.0.0.1:5900 *:* LISTEN

We set up the Ssh tunnel to give access to the VM’s VNC:

ssh -L 5901:127.0.0.1:5901 -A -g kr0m@PHYSICAL_HOST

We access the installer via VNC:

vinagre 127.0.0.1:5901



VNC-Ip

If we choose to assign an IP address, the VM will have an IP address for VNC access and that of the VM itself.
We will have to configure the VNC-IP as an alias in the parent and indicate it in the VM configuration.

We are going to leave it configured in the rc.conf and manually, in this way in case of a restart no configuration will be lost:

sysrc ifconfig_nfe0_alias0=“inet IP_VM netmask NETMASK”
ifconfig INTERFACE IP_VM netmask NETMASK alias

We create the VM:

iohyve create kaliVNC3 15G

We set the necessary parameters:

iohyve set kaliVNC3 ram=2048M loader=uefi vnc=YES vnc_ip=192.168.69.254 fw=BHYVE_UEFI.fd

We start the installation:

iohyve install kaliVNC3 kali2020.3.iso

We can see that there is a VNC bound to the indicated IP address:

sockstat -sv|grep LISTEN|grep bhyve

root bhyve 14438 7 tcp4 192.168.69.254:5900 *:* LISTEN  
root bhyve 4166 7 tcp4 127.0.0.1:5901 *:* LISTEN  
root bhyve 9029 7 tcp4 127.0.0.1:5900 *:* LISTEN

We connect directly to the VM’s VNC without the need for any tunnel:

vinagre 192.168.69.254:5900

Actually, for a Unix system administrator, VNC access doesn’t make sense since VMs are usually accessed via SSH. If we needed any graphical application, it would be sufficient to start it using ssh-Xforward, and if we needed urgent access, we would access it via the serial port with the Iohyve console command.


Xorg Bug

If X has been installed, it is buggy ( 1 , 2 ). The bug in question presents the following output when trying to start X:

To fix it, we have to enter via SSH and uninstall the xserver-xorg-video-vesa package. The problem is that when we start the graphical environment, it hangs due to the bug, and when we press Ctrl+Alt+F1, we are sending the keystrokes to our FreeBSD. To be able to do this within the VNC client, we must first disable the keys in our OS.

setxkbmap -option srvrkeys:none

Now we can log in from the non-graphical tty and start the SSH service to uninstall the conflicting package:

service ssh start

We access the VM via SSH:

We uninstall the package and shut down the VM:

apt remove xserver-xorg-video-vesa
halt

We enable Ctrl+Alt+FX in our OS:

setxkbmap -option

We start the VM:

iohyve start kaliVNC

As we can see, Kali now starts X normally. If we log in, we will see that the graphical environment works without problems:



Running graphical applications without X via SSH

One way to run graphical applications without the need to start X is through ssh-xforward. To do this, we configure ssh access and find out the VM’s IP address:

iohyve console kali

Starting console on kali...  
~~. to escape console [uses cu(1) for console]  
Connected  
  
kali login: kr0m  
Password:  
  
Linux kali 5.7.0-kali1-amd64 #1 SMP Debian 5.7.6-1kali2 (2020-07-01) x86_64  
The programs included with the Kali GNU/Linux system are free software;  
the exact distribution terms for each program are described in the  
individual files in /usr/share/doc/*/copyright.  
  
Kali GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent  
permitted by applicable law.  
Last login: Sun Nov 8 11:50:53 CET 2020 on ttyS0
Habilitamos el servico Ssh:
kr0m@kali:~$ sudo su
root@kali:/home/kr0m# systemctl enable ssh.service
root@kali:/home/kr0m# systemctl start ssh.service

We check the VM’s IP address:

root@kali:/home/kr0m# ifconfig

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500  
 inet 192.168.69.205 netmask 255.255.255.0 broadcast 192.168.69.255

We allow displaying graphical applications that come from the indicated IP on our PC using xhost:

xhost +inet:192.168.69.205

We run burpsuite as a test:

ssh -vYC kr0m@192.168.69.205 burpsuite



Listing VMs

iohyve list
Guest     VMM?  Running  rcboot?  Description  
kali      NO    NO       NO       Sun Nov  8 20:28:35 CET 2020  
kaliVNC3  NO    NO       NO       Mon Nov  9 12:19:21 CET 2020

VMs Information

We can check information about the VMs in the following way:

iohyve info -sv
Guest           Size  CPU  RAM    Pool   OS       Loader      Tap   Con    VMM?  Running  rcboot?  
kali            -     1    2048M  zroot  debian   grub-bhyve  tap0  nmdm0  NO    NO       NO  
kali/disk0      15G   -    -      zroot  -        -           -     -      -     -        -  
kaliVNC3        -     1    2048M  zroot  default  uefi        tap1  nmdm1  NO    NO       NO  
kaliVNC3/disk0  15G   -    -      zroot  -        -           -     -      -     -        -
iohyve getall kali
Getting kali iohyve properties...  
bargs            -A_-H_-P  
bhyve_path       /usr/sbin/bhyve  
boot             0  
con              nmdm0  
cpu              1  
description      Sun Nov  8 20:28:35 CET 2020  
install          no  
loader           grub-bhyve  
name             kali  
os               debian  
persist          1  
ram              2048M  
size             15G  
tap              tap0  
template         NO  
utc              YES  
vnc              NO  
vnc_h            600  
vnc_ip           127.0.0.1  
vnc_tablet       NO  
vnc_w            800  
vnc_wait         NO  

Parameters

We can assign values to the VM’s parameters. Some examples can be:

iohyve set kali ram=512M
iohyve set kali cpu=1
iohyve set kali description=“This is my Kali box”

We can even do device passthrough:

iohyve set kali pcidev:1=passthru,2/0/0

To get parameter values:

iohyve get kali ram
iohyve getall kali


Snapshots

Create a snapshot:

iohyve snap kali@beforeupdate

Taking snapshot kali@beforeupdate

List available snapshots:

iohyve snaplist

kali@beforeupdate

Revert to a snapshot:

iohyve stop kali
iohyve roll kali@beforeupdate

Rolling back to kali@beforeupdate

Delete snapshots:

iohyve rmsnap kali@beforeupdate


Cloning VMs

By cloning, we are making a copy of the VM, not a clone at the ZFS level. This can result in considerable space consumption since in addition to the space that the two VMs will occupy, Iohyve needs additional space while cloning.

iohyve clone kali dolly
iohyve list

Guest  VMM?  Running  rcboot?  Description  
dolly  NO    NO       NO       Tue_Nov__8_21:04:05_CET_2020  
kali   NO    NO       NO       Sun Nov  8 20:28:35 CET 2020

Templates

If we don’t want anyone to accidentally start a VM or make any modifications to this VM by starting it, reinstalling it, renaming it, or deleting it, we will have to set the template parameter to YES.

Let’s take the following example where kali is stopped:

iohyve list

Guest  VMM?  Running  rcboot?  Description  
kali   NO    NO       NO       Sun Nov  8 20:28:35 CET 2020

We assign the template=YES parameter:

iohyve set kali template=YES

We try to start the VM:

iohyve start kali

kali is a template. Cannot start

We try to delete it:

iohyve delete kali

kali is a template. Cannot delete

Force stop

In some extreme cases, the stop command will not be able to stop the VM. We can force stop it, but this command should be used as a last resort:

iohyve forcekill kali


Backup through exports

Through the export command we can generate a tarball with all the data of the VM, including its Iohyve configuration, this tarball will be stored in the path: /iohyve/VM_NAME/VM_NAME.tar.gz the only drawback is that it requires us to shut down the VM. But we can always clone the VM, make the export of the clone, copy the backup and delete the clone.

iohyve clone kali backup
iohyve list

Guest  VMM?  Running  rcboot?  Description  
backup NO    NO       NO       Tue_Nov__8_21:14:05_CET_2020  
kali   NO    NO       NO       Sun Nov  8 20:28:35 CET 2020
iohyve export backup
Exporting backup. Note this may take some time depending on the size.  
Creating temp directory...  
Exporting properties...  
Exporting disks...  
15360+0 records in  
15360+0 records out  
16106127360 bytes transferred in 170.564576 secs (94428326 bytes/sec)  
Compressing to archive...  
Removing temp directory...

We check the size of the export:

du -h /iohyve/backup/backup.tar.gz

3.2G /iohyve/backup/backup.tar.gz

We move the tarball to a safe place, it would be advisable to move it to a remote server in another physical location, as a proof of concept I will move it to /root:

mv /iohyve/backup/backup.tar.gz /root/

We delete the cloned VM:

iohyve delete backup

[WARNING] Are you sure you want to permanently delete backup and all child datasets?  
Location: zroot/iohyve/backup [Y/N]? Y  
Deleting backup at zroot/iohyve/backup...

We delete the original VM:

iohyve stop kali
iohyve delete kali

[WARNING] Are you sure you want to permanently delete kali and all child datasets?  
Location: zroot/iohyve/kali [Y/N]? Y  
Deleting kali at zroot/iohyve/kali...

To restore it we just have to decompress it, create a VM with the parameters of the configuration file and dump the contents of the disk image:

cd /root/
tar xvzf backup.tar.gz

x ./  
x ./disk0.img  
x ./properties.ucl
cat ./properties.ucl  
bargs=-A_-H_-P  
bhyve_path=/usr/sbin/bhyve  
boot=0  
con=nmdm0  
cpu=1  
description=Sun_Nov__8_20:08:03_CET_2020  
install=no  
loader=grub-bhyve  
name=backup  
os=debian  
persist=1  
ram=2048M  
size=15G  
tap=tap0  
template=NO  
utc=YES  
vnc=NO  
vnc_h=600  
vnc_ip=127.0.0.1  
vnc_tablet=NO  
vnc_w=800  
vnc_wait=NO

We create the VM with the disk size indicated in the ucl file:

iohyve create kali 15G

We assign the VM parameters as they appear in the ucl file:

iohyve set kali ram=2048M loader=grub-bhyve os=debian

We check that it has been created correctly:

iohyve list

Guest  VMM?  Running  rcboot?  Description  
kali   NO    NO       NO       Sun Nov  8 20:28:35 CET 2020

We dump the contents of the disk image:

dd if=/root/disk0.img of=/dev/zvol/zroot/iohyve/kali/disk0

31457280+0 records in  
31457280+0 records out  
16106127360 bytes transferred in 397.031970 secs (40566324 bytes/sec)

We start the VM and access it:

iohyve start kali
iohyve console kali


Difference between VMM and Running

When we start a VM, it goes through the bootloader where the kernel has not yet been loaded. This phase is VMM. If the kernel starts to load, it appears as Running.

If we stop a VM but VMM does not stop, resources may remain occupied by this VM. In this case, we must execute the destroy command:

iohyve destroy kali


Troubleshooting

Sometimes the console displays incorrect characters. Resetting it solves the problem:

iohyve conreset kali
iohyve console kali

If we are testing Iohyve on a desktop computer, it is very likely that we have the gvfs process running. This can cause problems when we delete VMs since the gvfs Trash process may be accessing the filesystem. In this case, the error shown when trying to delete the VM is as follows:

iohyve delete kaliVNC

[WARNING] Are you sure you want to permanently delete kaliVNC and all child datasets?  
Location: zroot/iohyve/kaliVNC [Y/N]? Y  
Deleting kaliVNC at zroot/iohyve/kaliVNC...  
cannot unmount '/iohyve/kaliVNC': Device busy

We locate the process that keeps the device occupied:

lsof +f -- /iohyve/kaliVNC

COMMAND   PID USER   FD   TYPE                DEVICE SIZE/OFF NODE NAME  
gvfsd-tra 68671 kr0m   29r  VDIR 2635944363,1737022982      512   34 /iohyve/kaliVNC

We kill the process:

kill -9 68671

Now we can delete the VM without problems:

iohyve delete kaliVNC

If the error message we get is the following:

iohyve delete kali

[WARNING] Are you sure you want to permanently delete kali and all child datasets?  
Location: zroot/iohyve/kali [Y/N]? Y  
Deleting kali at zroot/iohyve/kali...  
cannot destroy 'zroot/iohyve/kali/disk0': dataset is busy

We will have to restart the service. The drawback of this is that it will stop all VMs:

service iohyve stop
iohyve delete kaliVNC
service iohyve start

If we have any network problems, we can quickly identify the tap interfaces of the VMs with the following command:

iohyve taplist

Listing all network taps:  
zroot/iohyve/kali......tap0

We can manually check how the bridge has both the parent host and the VM’s tap network interfaces:

ifconfig
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500  
 options=812099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>  
 ether 34:97:f6:36:bf:45  
 inet 192.168.69.4 netmask 0xffffff00 broadcast 192.168.69.255  
 media: Ethernet autoselect (1000baseT <full-duplex>)  
 status: active  
 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>  
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384  
 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>  
 inet6 ::1 prefixlen 128  
 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2  
 inet 127.0.0.1 netmask 0xff000000  
 groups: lo  
 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>  
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500  
 description: iohyve-bridge-em0  
 ether 02:89:af:f3:c6:00  
 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15  
 maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200  
 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0  
 member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>  
        ifmaxaddr 0 port 5 priority 128 path cost 2000000  
 member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>  
         ifmaxaddr 0 port 1 priority 128 path cost 20000  
 groups: bridge  
 nd6 options=9<PERFORMNUD,IFDISABLED>  
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500  
 description: iohyve-kali--  
 options=80000<LINKSTATE>  
 ether 58:9c:fc:10:ff:fd  
 groups: tap  
 media: Ethernet autoselect  
 status: active  
 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>  
 Opened by PID 30627

I must warn you that disk management seems to work well in most of the tested cases, but deleting disk0 from a VM is problematic. There seems to be some bug, and it becomes impossible to boot the VM with disk1 afterwards.

If you liked the article, you can treat me to a RedBull here