In this article, I will explain a real case of a physical server where the system is installed on a ZFS mirror in FreeBSD. I will show the steps to follow in case of failure of either disk and we will see some peculiarities of the system.
The specific hardware is:
HP ProLiant ML310e Gen8 v2
HP Smart Array B120i RAID -> Disabled to pass the disks directly to the operating system.
- Initial state
- Simulating failure of the first disk
- Simulating failure of the second disk
- Troubleshooting
Initial state
The system was installed with GPT partitions and BIOS system since it is very old and does not support EFI. Its initial state is as follows:
<WDC WDS500G1B0A-00H9H0 X41100WD> at scbus0 target 0 lun 0 (ada0,pass0)
<Samsung SSD 850 EVO 500GB EMT02B6Q> at scbus1 target 0 lun 0 (ada1,pass1)
pool: zroot
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ada0p3 ONLINE 0 0 0
ada1p3 ONLINE 0 0 0
errors: No known data errors
Simulating failure of the first disk
We remove the ada0 disk:
WDC WDS500G1B0A-00H9H0 X41100WD
We check the disks and we will see that it has disappeared:
<Samsung SSD 850 EVO 500GB EMT02B6Q> at scbus1 target 0 lun 0 (ada0,pass0)
The pool status will be as follows:
pool: zroot
state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
scan: resilvered 20.5G in 00:00:52 with 0 errors on Thu May 12 09:40:08 2022
config:
NAME STATE READ WRITE CKSUM
zroot DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
ada0p3 FAULTED 0 0 0 corrupted data
ada0p3 ONLINE 0 0 0
errors: No known data errors
When ada0 is removed, ada1 becomes ada0, so ada0p3 is FAULTED because that was the name of the disk before it disappeared and it is ONLINE because it is the old ada1p3 remapped to ada0p3 in this last boot.
That’s why zpool status shows ambiguous data like this:
ada0p3 FAULTED 0 0 0 corrupted data
ada0p3 ONLINE 0 0 0
What should be clear is that the disk that is shown ONLINE is always the one that is working correctly regardless of the rest of the information displayed.
In this case ada0p3:
ada0p3 ONLINE 0 0 0
Which corresponds to the Samsung disk:
<Samsung SSD 850 EVO 500GB EMT02B6Q> at scbus1 target 0 lun 0 (ada0,pass0)
The BIOS of this server has a peculiarity and that is that it always starts from the first disk it finds connected, regardless of whether it works or not.
If the disk in slot 1 is removed, it will start from the disk in slot 2.
If a disk fails, the survivor must be put in slot 1 and the new disk in slot 2, so that it will start with the good one and we can start the resilvering process.
Taking this into account, the steps to follow to change the disk are:
-
Make sure that the survivor disk is plugged into the first slot (important)
-
Plug the replacement into the second slot:
<Samsung SSD 850 EVO 500GB EMT02B6Q> at scbus0 target 0 lun 0 (ada0,pass0)
<TOSHIBA MQ01ABF050 AM0P4C> at scbus1 target 0 lun 0 (ada1,pass1)
pool: zroot
state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
scan: resilvered 20.5G in 00:00:52 with 0 errors on Thu May 12 09:40:08 2022
config:
NAME STATE READ WRITE CKSUM
zroot DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
ada0p3 FAULTED 0 0 0 corrupted data
ada0p3 ONLINE 0 0 0
errors: No known data errors
- Clone the partition table on the second disk:
- Locate the freebsd-boot partition (for BIOS, UEFI is another command ):
40 1024 1 freebsd-boot (512K)
- Install the bootloader in the MBR and the boot files in the freebsd-boot partition (in this case the first partition: -i 1) for UEFI is another command :
- Replace the disk, the parameters of the replace command indicate to replace this disk (1st param) with this other (2nd param).
When doing the replace, ZFS will automatically search for the correct disk from which to do resilvering even if the FAULTED and ONLINE have the same name.
zpool replace -f zroot FAULTED_DISK NEW_DISK
pool: zroot
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Thu May 12 12:47:17 2022
19.3G scanned at 123M/s, 8.09G issued at 51.5M/s, 19.3G total
8.60G resilvered, 41.89% done, 00:03:43 to go
config:
NAME STATE READ WRITE CKSUM
zroot DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
replacing-0 DEGRADED 0 0 0
ada0p3 FAULTED 0 0 0 corrupted data
ada1p3 ONLINE 0 0 0 (resilvering)
ada0p3 ONLINE 0 0 0
errors: No known data errors
When finished:
pool: zroot
state: ONLINE
scan: resilvered 20.5G in 00:06:35 with 0 errors on Thu May 12 12:53:52 2022
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ada1p3 ONLINE 0 0 0
ada0p3 ONLINE 0 0 0
errors: No known data errors
Simulating failure of the second disk
We remove the ada1 disk:
Samsung SSD 850 EVO 500GB EMT02B6Q
We check the disks and we will see that it has disappeared:
<WDC WDS500G1B0A-00H9H0 X41100WD> at scbus0 target 0 lun 0 (ada0,pass0)
The state of the pool will be as follows:
pool: zroot
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-2Q
config:
NAME STATE READ WRITE CKSUM
zroot DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
ada0p3 ONLINE 0 0 0
ada1p3 UNAVAIL 0 0 0 cannot open
errors: No known data errors
Since there has been no disk remapping, it would be necessary to copy the partition scheme from the surviving disk, install the bootloader and replace the disk.
- Plug in the replacement in the second slot.
<WDC WDS500G1B0A-00H9H0 X41100WD> at scbus0 target 0 lun 0 (ada0,pass0)
<TOSHIBA MQ01ABF050 AM0P4C> at scbus1 target 0 lun 0 (ada1,pass1)
- Clone the partition table on the second disk:
- Locate the freebsd-boot partition (for BIOS, UEFI is another command ):
40 1024 1 freebsd-boot (512K)
- Install the bootloader in the MBR and the boot files in the freebsd-boot partition (in this case the first partition: -i 1) for UEFI it is another command :
- Replace the disk, the parameters of the replace command indicate replace this disk (1st param) with this other (2nd param).
zpool replace -f zroot UNAVAIL NEW_DISK
pool: zroot
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Thu May 12 15:14:54 2022
19.3G scanned at 1.07G/s, 526M issued at 29.2M/s, 19.3G total
567M resilvered, 2.66% done, 00:10:59 to go
config:
NAME STATE READ WRITE CKSUM
zroot DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
ada0p3 ONLINE 0 0 0
replacing-1 DEGRADED 0 0 0
ada1p3/old UNAVAIL 0 0 0 cannot open
ada1p3 ONLINE 0 0 0 (resilvering)
errors: No known data errors
When finished:
pool: zroot
state: ONLINE
scan: resilvered 20.5G in 00:06:35 with 0 errors on Thu May 12 15:21:29 2022
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ada0p3 ONLINE 0 0 0
ada1p3 ONLINE 0 0 0
errors: No known data errors
Troubleshooting
Manipulating the partition table on some disks may give us problems:
gpart: Device busy
It may be for two reasons:
- The new disk had a swap partition that the system has recognized in the boot and is using:
Device 1K-blocks Used Avail Capacity
/dev/ada0p2 2097152 0 2097152 0%
/dev/ada1p2 2097152 0 2097152 0%
Total 4194304 0 4194304 0%
- Or because we have confused the disk and we are backing up the wrong one, since it is being used by the OS it does not let us.
What should be clear is that the disk that is shown ONLINE is always the one that is working correctly regardless of the rest of the information shown.
ada0p3 ONLINE 0 0 0
<Samsung SSD 850 EVO 500GB EMT02B6Q> at scbus1 target 0 lun 0 (ada0,pass0)
If the good disk is ada0, we will have to back up from this to ada1.