Iohyve is a Bhyve virtual machine manager that uses ZFS functionalities to offer a simpler and more intuitive user experience. Its features include the ability to take snapshots, manage ISOs, integrated VNC server, cloning, and VM backup. It could be said that it is the equivalent of Iocage but for Bhyve VMs. On the other hand, if we prefer a unified interface, we can always use CBSD .

This manual is composed of different parts:

• Installation
• Downloading ISO images
• Creating VMs
• Installation via serial port
• Installation by VNC
• VNC-Loopback
• VNC-Ip
• Xorg Bug
• Running graphical applications without X via Ssh
• Listing VMs
• VMs Information
• Parameters
• Snapshots
• Cloning VMs
• Templates
• Force stop
• Backup through exports
• Difference between VMM and Running
• Troubleshooting

Installation

We install everything we need:

If we have more than one zpool, we must tell Iohyve which one to use:

NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT  
zroot   109G  78.5G  30.5G        -         -    44%    71%  1.00x  ONLINE  -

The Iohyve network works through bridging, we must indicate the network interface that we want to put inside the bridge, so when we create a VM, the tap interface of that VM will be attached to the existing bridge:

We allow Iohyve to load the kernel modules it needs:

To make it permanent at server startup, we can use the following RC parameters:

We restart to make sure everything loads automatically:

Downloading ISO images

To perform the OS installation, we need the installation ISO image, which we download using the fetchiso command:

We rename it to make it easier to write:

We list the available ISOs:

Listing ISO's...  
kali2020.3.iso

Creating VMs

We create the VM with a 15G disk:

NOTE: The virtual disk space reservation is NOT dynamic, which means that we must have the indicated space available at the time of creating the VM.

Installation via serial port:

We assign the VM parameters:

The os parameter does not indicate the OS to install, but only some adjustments that grub-bhyve will take into account to boot the VM, for example, debian is any Linux without LVM.

The possible options are: openbsd57, openbsd58, openbsd59, openbsd60, openbsd61, netbsd, debian (non-LVM), d8lvm (LVM installs), centos6, centos7 (non-XFS), ubuntu (LVM installs, non-LVM use debian), arch, and custom.

The custom option means that the files /iohyve/VM_NAME/device.map and /iohyve/VM_NAME/grub.cfg will be read at boot.

NOTE: If we are going to install a FreeBSD system, it is not necessary to specify the loader/os parameters.

Before installing the OS, we leave a console ready and waiting:

In a second terminal, we start the installation:

We will see that the OS installation wizard appears in the first terminal, we select the second option:

GNU GRUB version 2.00

+--------------------------------------------------------------------------+
|Graphical install                                                         |
|Install                                                                   |
|Advanced options ...                                                      |
|Accessible dark contrast installer menu ...                               |
|Install with speech synthesis                                             |
|                                                                          |
|                                                                          |
|                                                                          |
|                                                                          |
|                                                                          |
|                                                                          |
|                                                                          |
+--------------------------------------------------------------------------+

 Use the ^ and v keys to select which entry is highlighted.
 Press enter to boot the selected OS, `e' to edit the commands
 before booting or `c' for a command-line.

After finishing the installation, we will have the VM turned off, we start it and access the console:

kali login: kr0m  
Password:  
  
Linux kali 5.7.0-kali1-amd64 #1 SMP Debian 5.7.6-1kali2 (2020-07-01) x86_64  
The programs included with the Kali GNU/Linux system are free software;  
the exact distribution terms for each program are described in the  
individual files in /usr/share/doc/*/copyright.  
  
Kali GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent  
permitted by applicable law.  
Last login: Sun Nov 8 10:59:10 CET 2020 on ttyS0  
  
kr0m@kali:~$

To exit the console, we write Alt+4.: ~.

Installation via VNC

By default, Iohyve uses the serial console to access the OS, in most cases it will be more than enough since if we need to start any graphical application through ssh-Xforward we can do it without having to start the X.

Another way to get access to the X is by installing a VNC server in the VM itself, this way we would no longer depend on Iohyve’s VNC.

But if we need graphical access in the OS installation as with Windows, we will have to enable access by VNC through the uefi loader, so that it is available we must install the bhyve-firmware package as done at the beginning of the manual and import the firmware file with the cpfw command.

Listing Firmware...  
BHYVE_UEFI.fd

Now we must decide whether we want to sacrifice an IP address from the network for VNC access or if we bind the service to the loopback and access via Ssh tunnel, through loopback access will be safer since it will only be exposed when we mount the tunnel and we will not waste IP addresses but we will have to configure a different VNC port for each VM.

VNC-Loopback

We create the VM:

We set the necessary parameters:

We start the installation:

We see that there is a service on the loopback on port 5900:

root bhyve 9029 7 tcp4 127.0.0.1:5900 *:* LISTEN

We set up the Ssh tunnel to access the VNC from outside:

We start our VNC client attacking our PC which will take us via the tunnel to the VM’s VNC:

If we create a second VM we will have to choose a different VNC port:

We see that there are two services bound to the loopback but on different ports:

root bhyve 4166 7 tcp4 127.0.0.1:5901 *:* LISTEN  
root bhyve 9029 7 tcp4 127.0.0.1:5900 *:* LISTEN

We set up the Ssh tunnel to give access to the VM’s VNC:

We access the installer via VNC:

VNC-Ip

If we choose to assign an IP address, the VM will have an IP address for VNC access and that of the VM itself.
We will have to configure the VNC-IP as an alias in the parent and indicate it in the VM configuration.

We are going to leave it configured in the rc.conf and manually, in this way in case of a restart no configuration will be lost:

We create the VM:

We set the necessary parameters:

We start the installation:

We can see that there is a VNC bound to the indicated IP address:

root bhyve 14438 7 tcp4 192.168.69.254:5900 *:* LISTEN  
root bhyve 4166 7 tcp4 127.0.0.1:5901 *:* LISTEN  
root bhyve 9029 7 tcp4 127.0.0.1:5900 *:* LISTEN

We connect directly to the VM’s VNC without the need for any tunnel:

Actually, for a Unix system administrator, VNC access doesn’t make sense since VMs are usually accessed via SSH. If we needed any graphical application, it would be sufficient to start it using ssh-Xforward, and if we needed urgent access, we would access it via the serial port with the Iohyve console command.

Xorg Bug

If X has been installed, it is buggy ( 1 , 2 ). The bug in question presents the following output when trying to start X:

To fix it, we have to enter via SSH and uninstall the xserver-xorg-video-vesa package. The problem is that when we start the graphical environment, it hangs due to the bug, and when we press Ctrl+Alt+F1, we are sending the keystrokes to our FreeBSD. To be able to do this within the VNC client, we must first disable the keys in our OS.

Now we can log in from the non-graphical tty and start the SSH service to uninstall the conflicting package:

We access the VM via SSH:

We uninstall the package and shut down the VM:

We enable Ctrl+Alt+FX in our OS:

We start the VM:

As we can see, Kali now starts X normally. If we log in, we will see that the graphical environment works without problems:

Running graphical applications without X via SSH

One way to run graphical applications without the need to start X is through ssh-xforward. To do this, we configure ssh access and find out the VM’s IP address:

Starting console on kali...  
~~. to escape console [uses cu(1) for console]  
Connected  
  
kali login: kr0m  
Password:  
  
Linux kali 5.7.0-kali1-amd64 #1 SMP Debian 5.7.6-1kali2 (2020-07-01) x86_64  
The programs included with the Kali GNU/Linux system are free software;  
the exact distribution terms for each program are described in the  
individual files in /usr/share/doc/*/copyright.  
  
Kali GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent  
permitted by applicable law.  
Last login: Sun Nov 8 11:50:53 CET 2020 on ttyS0
Habilitamos el servico Ssh:

We check the VM’s IP address:

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500  
 inet 192.168.69.205 netmask 255.255.255.0 broadcast 192.168.69.255

We allow displaying graphical applications that come from the indicated IP on our PC using xhost:

We run burpsuite as a test:

Listing VMs

Guest     VMM?  Running  rcboot?  Description  
kali      NO    NO       NO       Sun Nov  8 20:28:35 CET 2020  
kaliVNC3  NO    NO       NO       Mon Nov  9 12:19:21 CET 2020

VMs Information

We can check information about the VMs in the following way:

Guest           Size  CPU  RAM    Pool   OS       Loader      Tap   Con    VMM?  Running  rcboot?  
kali            -     1    2048M  zroot  debian   grub-bhyve  tap0  nmdm0  NO    NO       NO  
kali/disk0      15G   -    -      zroot  -        -           -     -      -     -        -  
kaliVNC3        -     1    2048M  zroot  default  uefi        tap1  nmdm1  NO    NO       NO  
kaliVNC3/disk0  15G   -    -      zroot  -        -           -     -      -     -        -

Getting kali iohyve properties...  
bargs            -A_-H_-P  
bhyve_path       /usr/sbin/bhyve  
boot             0  
con              nmdm0  
cpu              1  
description      Sun Nov  8 20:28:35 CET 2020  
install          no  
loader           grub-bhyve  
name             kali  
os               debian  
persist          1  
ram              2048M  
size             15G  
tap              tap0  
template         NO  
utc              YES  
vnc              NO  
vnc_h            600  
vnc_ip           127.0.0.1  
vnc_tablet       NO  
vnc_w            800  
vnc_wait         NO  

Parameters

We can assign values to the VM’s parameters. Some examples can be:

We can even do device passthrough:

To get parameter values:

Snapshots

Create a snapshot:

Taking snapshot kali@beforeupdate

List available snapshots:

kali@beforeupdate

Revert to a snapshot:

Rolling back to kali@beforeupdate

Delete snapshots:

Cloning VMs

By cloning, we are making a copy of the VM, not a clone at the ZFS level. This can result in considerable space consumption since in addition to the space that the two VMs will occupy, Iohyve needs additional space while cloning.

Guest  VMM?  Running  rcboot?  Description  
dolly  NO    NO       NO       Tue_Nov__8_21:04:05_CET_2020  
kali   NO    NO       NO       Sun Nov  8 20:28:35 CET 2020

Templates

If we don’t want anyone to accidentally start a VM or make any modifications to this VM by starting it, reinstalling it, renaming it, or deleting it, we will have to set the template parameter to YES.

Let’s take the following example where kali is stopped:

Guest  VMM?  Running  rcboot?  Description  
kali   NO    NO       NO       Sun Nov  8 20:28:35 CET 2020

We assign the template=YES parameter:

We try to start the VM:

kali is a template. Cannot start

We try to delete it:

kali is a template. Cannot delete

Force stop

In some extreme cases, the stop command will not be able to stop the VM. We can force stop it, but this command should be used as a last resort:

Backup through exports

Through the export command we can generate a tarball with all the data of the VM, including its Iohyve configuration, this tarball will be stored in the path: /iohyve/VM_NAME/VM_NAME.tar.gz the only drawback is that it requires us to shut down the VM. But we can always clone the VM, make the export of the clone, copy the backup and delete the clone.

Guest  VMM?  Running  rcboot?  Description  
backup NO    NO       NO       Tue_Nov__8_21:14:05_CET_2020  
kali   NO    NO       NO       Sun Nov  8 20:28:35 CET 2020

Exporting backup. Note this may take some time depending on the size.  
Creating temp directory...  
Exporting properties...  
Exporting disks...  
15360+0 records in  
15360+0 records out  
16106127360 bytes transferred in 170.564576 secs (94428326 bytes/sec)  
Compressing to archive...  
Removing temp directory...

We check the size of the export:

3.2G /iohyve/backup/backup.tar.gz

We move the tarball to a safe place, it would be advisable to move it to a remote server in another physical location, as a proof of concept I will move it to /root:

We delete the cloned VM:

[WARNING] Are you sure you want to permanently delete backup and all child datasets?  
Location: zroot/iohyve/backup [Y/N]? Y  
Deleting backup at zroot/iohyve/backup...

We delete the original VM:

[WARNING] Are you sure you want to permanently delete kali and all child datasets?  
Location: zroot/iohyve/kali [Y/N]? Y  
Deleting kali at zroot/iohyve/kali...

To restore it we just have to decompress it, create a VM with the parameters of the configuration file and dump the contents of the disk image:

x ./  
x ./disk0.img  
x ./properties.ucl
cat ./properties.ucl  
bargs=-A_-H_-P  
bhyve_path=/usr/sbin/bhyve  
boot=0  
con=nmdm0  
cpu=1  
description=Sun_Nov__8_20:08:03_CET_2020  
install=no  
loader=grub-bhyve  
name=backup  
os=debian  
persist=1  
ram=2048M  
size=15G  
tap=tap0  
template=NO  
utc=YES  
vnc=NO  
vnc_h=600  
vnc_ip=127.0.0.1  
vnc_tablet=NO  
vnc_w=800  
vnc_wait=NO

We create the VM with the disk size indicated in the ucl file:

We assign the VM parameters as they appear in the ucl file:

We check that it has been created correctly:

Guest  VMM?  Running  rcboot?  Description  
kali   NO    NO       NO       Sun Nov  8 20:28:35 CET 2020

We dump the contents of the disk image:

31457280+0 records in  
31457280+0 records out  
16106127360 bytes transferred in 397.031970 secs (40566324 bytes/sec)

We start the VM and access it:

Difference between VMM and Running

When we start a VM, it goes through the bootloader where the kernel has not yet been loaded. This phase is VMM. If the kernel starts to load, it appears as Running.

If we stop a VM but VMM does not stop, resources may remain occupied by this VM. In this case, we must execute the destroy command:

Troubleshooting

Sometimes the console displays incorrect characters. Resetting it solves the problem:

If we are testing Iohyve on a desktop computer, it is very likely that we have the gvfs process running. This can cause problems when we delete VMs since the gvfs Trash process may be accessing the filesystem. In this case, the error shown when trying to delete the VM is as follows:

[WARNING] Are you sure you want to permanently delete kaliVNC and all child datasets?  
Location: zroot/iohyve/kaliVNC [Y/N]? Y  
Deleting kaliVNC at zroot/iohyve/kaliVNC...  
cannot unmount '/iohyve/kaliVNC': Device busy

We locate the process that keeps the device occupied:

COMMAND   PID USER   FD   TYPE                DEVICE SIZE/OFF NODE NAME  
gvfsd-tra 68671 kr0m   29r  VDIR 2635944363,1737022982      512   34 /iohyve/kaliVNC

We kill the process:

Now we can delete the VM without problems:

If the error message we get is the following:

[WARNING] Are you sure you want to permanently delete kali and all child datasets?  
Location: zroot/iohyve/kali [Y/N]? Y  
Deleting kali at zroot/iohyve/kali...  
cannot destroy 'zroot/iohyve/kali/disk0': dataset is busy

We will have to restart the service. The drawback of this is that it will stop all VMs:

If we have any network problems, we can quickly identify the tap interfaces of the VMs with the following command:

Listing all network taps:  
zroot/iohyve/kali......tap0

We can manually check how the bridge has both the parent host and the VM’s tap network interfaces:

em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500  
 options=812099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>  
 ether 34:97:f6:36:bf:45  
 inet 192.168.69.4 netmask 0xffffff00 broadcast 192.168.69.255  
 media: Ethernet autoselect (1000baseT <full-duplex>)  
 status: active  
 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>  
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384  
 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>  
 inet6 ::1 prefixlen 128  
 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2  
 inet 127.0.0.1 netmask 0xff000000  
 groups: lo  
 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>  
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500  
 description: iohyve-bridge-em0  
 ether 02:89:af:f3:c6:00  
 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15  
 maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200  
 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0  
 member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>  
        ifmaxaddr 0 port 5 priority 128 path cost 2000000  
 member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>  
         ifmaxaddr 0 port 1 priority 128 path cost 20000  
 groups: bridge  
 nd6 options=9<PERFORMNUD,IFDISABLED>  
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500  
 description: iohyve-kali--  
 options=80000<LINKSTATE>  
 ether 58:9c:fc:10:ff:fd  
 groups: tap  
 media: Ethernet autoselect  
 status: active  
 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>  
 Opened by PID 30627

I must warn you that disk management seems to work well in most of the tested cases, but deleting disk0 from a VM is problematic. There seems to be some bug, and it becomes impossible to boot the VM with disk1 afterwards.