BTRFS has many interesting features but when things go wrong, it can be complicated to recover the file system or even the data. In this article, I will explain the basic steps to follow in case of disaster.
To perform the tests, we generate an 8G file that we will mount using loop and format in BTRFS:
losetup /dev/loop1 mountedImage.img
mkfs.btrfs /dev/loop1
mount /dev/loop1 /mnt/aux/
DETECT ERRORS
Using btrfs scrub, the checksum of all data in the file system will be verified. If an error is detected and is correctable by reading the correct data from another part of the file system (RAID1, for example), it will be automatically corrected.
The scrub will consume approximately 80% of the I/O of the device in question. For most scenarios, this is something that can be assumed without any problem.
We launch it in foreground to see the results, otherwise we would consult: /var/lib/btrfs/scrub.status.ID
scrub done for 101ffe75-d95b-41bf-a383-893e9a5f5e21
scrub started at Wed May 17 10:13:33 2017 and finished after 00:00:00
total bytes scrubbed: 320.00KiB with 0 errors
scrub status:1
101ffe75-d95b-41bf-a383-893e9a5f5e21:1|data_extents_scrubbed:1|tree_extents_scrubbed:16|data_bytes_scrubbed:65536|tree_bytes_scrubbed:262144|read_errors:0|csum_errors:0|verify_errors:0|no_csum:16|csum_discards:0|super_errors:0|malloc_errors:0|uncorrectable_errors:0|corrected_errors:0|last_physical:896663552|t_start:1495008813|t_resumed:0|duration:0|canceled:0|finished:1
We can check error counters with:
[/dev/loop1].write_io_errs 0
[/dev/loop1].read_io_errs 0
[/dev/loop1].flush_io_errs 0
[/dev/loop1].corruption_errs 0
[/dev/loop1].generation_errs 0
One way to automate BTRFS monitoring is through SEC. To do this, we must launch scrubs in the background so that they leave a record of the errors found in the system logs. Through SEC, we will review these logs and alert based on the parsing results:
vi /etc/conf.d/sec
# configuration file for /etc/init.d/sec
# flags to pass to sec (see 'sec --help')
SEC_FLAGS=""
# Define a debug level (1..6)
DEBUG_LEVEL="4"
# define where sec reads messages from for translating them
#INPUT_FILES="${INPUT_FILES} -input=/var/log/some.log"
#INPUT_FILES="${INPUT_FILES} -input=/tmp/other.file"
INPUT_FILES="-input=/var/log/messages"
\# http://simple-evcorr.sourceforge.net/man.html
\# http://sixshooter.v6.thrupoint.net/SEC-examples/article.html
\# http://sixshooter.v6.thrupoint.net/SEC-examples/article-part2.html
type=SingleWithSuppress
ptype=RegExp
pattern=(?i)kernel.*btrfs: (?!disk space caching is enabled|use ssd allocation|use .* compression|unlinked .* orphans|turning on discard|device label .* devid .* transid|detected SSD devices, enabling SSD mode|has skinny extents|device label|creating UUID tree|checking UUID tree|setting .* feature flag|bdev.* flush 0, corrupt 0, gen 0)
window=60
desc=Btrfs unexpected log
action=pipe '%t: $0' /usr/bin/mail -s "sec: %s" root
DUMP DATA
With a little luck, mounting in RO will be enough:
A restore is harmless, it will not attempt to repair anything, just read:
The available options for restore are:
-s: Also restore snapshots. Without this option snapshots will not be restored.
-x: Get extended attributes. Without this option, extented attributes will not be retrieved.
-m: Restore metadata: owner, mode and times.
-S: Restore symbolic links.
-v: Increase verbosity. May be given multiple times.
-i: Ignore errors. Normally the restore tool exits immediately for any error. This option forces it to keep going if it can, usually this results in some missing data.
-o: Overwrite existing files. If files exist at the output location with the same name, normally the restore utility will skip restoring that file. This option will overwrite the existing files instead.
-t: Tree location. The location of the tree of tree roots.
-f: Filesystem location. The byte number given will be used as the root of the filesystem, instead of the location specified by the superblock. This can be useful if your superblocks are inconsistent.
-u: Superblock mirror. Valid values are 0,1,2. Specifies an alternate superblock copy to use. This may be useful if your 0th superblock is damaged.
-d: ???
-r: Root objectid. The objectid given will be used as the root of the filesystem, instead of the location specified by the superblock. This may assist in recovery of subvolumes where the real root is damaged.
-c: Case insensitive regex matching.
-l: List tree roots.
-D: Dry run (only list files that would be recovered).
--path-regex: Regex for files to restore. In order to restore only a single folder somewhere in the btrfs tree, it is unfortunately necessary to construct a slightly nontrivial regex, e.g.: ‘^/(|home(|/username(|/Desktop(|/.*))))$’
Another option is to dump from a certain tree, but it is a more complicated method. To show how it works, we create 3 files:
touch 1 2 3
We create some snapshots to make the tree grow:
btrfs subvolume delete /mnt/aux/snap1/
btrfs subvolume create /mnt/aux/subvol1
btrfs subvolume delete /mnt/aux/subvol1
Despite not having mistreated the file system at all, somewhat worrying messages appear: “seems good, but generation/level doesn’t match.”
Superblock thinks the generation is 17
Superblock thinks the level is 0
Found tree root at 29982720 gen 17 level 0
Well block 29917184(gen: 16 level: 0) seems good, but generation/level doesn't match, want gen: 17 level: 0
Well block 29868032(gen: 15 level: 0) seems good, but generation/level doesn't match, want gen: 17 level: 0
Well block 29704192(gen: 14 level: 0) seems good, but generation/level doesn't match, want gen: 17 level: 0
Well block 29605888(gen: 13 level: 0) seems good, but generation/level doesn't match, want gen: 17 level: 0
Well block 29392896(gen: 12 level: 0) seems good, but generation/level doesn't match, want gen: 17 level: 0
Well block 29523968(gen: 11 level: 0) seems good, but generation/level doesn't match, want gen: 17 level: 0
A root has been found: 29982720 with the tree: 16,15,14,13,12,11 with their respective block numbers: 29917184, 29868032, 29704192, 29605888, 29392896, 29523968.
btrfs restore -t 29917184 /dev/loop1 /mnt/restore
parent transid verify failed on 29917184 wanted 17 found 16
parent transid verify failed on 29917184 wanted 17 found 16
parent transid verify failed on 29917184 wanted 17 found 16
parent transid verify failed on 29917184 wanted 17 found 16
Ignoring transid failure
1 2 3
REPAIR FILE SYSTEM
There is a very DANGEROUS tool called btrfs-zero-log that we should ONLY use if we see exactly this error:
BTRFS: failed to read log tree
NOTE: If we run the tool unnecessarily, it will be more complicated to recover the data!!
Another option is to run a check, but it is not recommended since the code needs to be polished and could cause damage (It is possible (but highly unlikely in the most recent versions) that it may cause additional damage in the process of repair).
btrfs check --repair /dev/loop1 2>&1 | tee /tmp/repair
checking extents
Fixed 0 roots.
checking free space cache
checking fs roots
checking csums
checking root refs
enabling repair mode
Checking filesystem on /dev/loop1
UUID: 101ffe75-d95b-41bf-a383-893e9a5f5e21
cache and super generation don't match, space cache will be invalidated
found 196608 bytes used err is 0
total csum bytes: 0
total tree bytes: 131072
total fs tree bytes: 32768
total extent tree bytes: 16384
btree space waste bytes: 123761
file data blocks allocated: 65536
referenced 65536
DUMP METADATA
btrfs-debug-tree /dev/loop1
btrfs-progs v4.6.1
root tree
leaf 29360128 items 13 free space 12844 generation 18 owner 1
fs uuid 101ffe75-d95b-41bf-a383-893e9a5f5e21
chunk uuid eec5eaf1-daa5-4329-b185-a8117524e57c
item 0 key (EXTENT_TREE ROOT_ITEM 0) itemoff 15844 itemsize 439
root data bytenr 29376512 level 0 dirid 0 refs 1 gen 18 lastsnap 0
uuid 00000000-0000-0000-0000-000000000000
item 1 key (DEV_TREE ROOT_ITEM 0) itemoff 15405 itemsize 439
root data bytenr 29573120 level 0 dirid 0 refs 1 gen 12 lastsnap 0
uuid 00000000-0000-0000-0000-000000000000
item 2 key (FS_TREE INODE_REF 6) itemoff 15388 itemsize 17
inode ref index 0 namelen 7 name: default
item 3 key (FS_TREE ROOT_ITEM 0) itemoff 14949 itemsize 439
root data bytenr 29900800 level 0 dirid 256 refs 1 gen 16 lastsnap 14
uuid 00000000-0000-0000-0000-000000000000
ctransid 16 otransid 0 stransid 0 rtransid 0
item 4 key (ROOT_TREE_DIR INODE_ITEM 0) itemoff 14789 itemsize 160
inode generation 3 transid 0 size 0 nbytes 16384
block group 0 mode 40755 links 1 uid 0 gid 0
rdev 0 flags 0x0
item 5 key (ROOT_TREE_DIR INODE_REF 6) itemoff 14777 itemsize 12
inode ref index 0 namelen 2 name: ..
item 6 key (ROOT_TREE_DIR DIR_ITEM 2378154706) itemoff 14740 itemsize 37
location key (FS_TREE ROOT_ITEM -1) type DIR
namelen 7 datalen 0 name: default
item 7 key (CSUM_TREE ROOT_ITEM 0) itemoff 14301 itemsize 439
root data bytenr 30015488 level 0 dirid 0 refs 1 gen 17 lastsnap 0
uuid 00000000-0000-0000-0000-000000000000
item 8 key (UUID_TREE ROOT_ITEM 0) itemoff 13862 itemsize 439
root data bytenr 29933568 level 0 dirid 0 refs 1 gen 16 lastsnap 0
uuid ce1e052c-8632-a34b-ad7c-cc2dd9ea4944
item 9 key (256 INODE_ITEM 0) itemoff 13702 itemsize 160
inode generation 17 transid 17 size 65536 nbytes 786432
block group 0 mode 100600 links 1 uid 0 gid 0
rdev 0 flags 0x1b
item 10 key (256 EXTENT_DATA 0) itemoff 13649 itemsize 53
extent data disk byte 12648448 nr 65536
extent data offset 0 nr 65536 ram 65536
extent compression(none)
item 11 key (FREE_SPACE UNTYPED 29360128) itemoff 13608 itemsize 41
location key (256 INODE_ITEM 0)
cache generation 17 entries 7 bitmaps 0
item 12 key (DATA_RELOC_TREE ROOT_ITEM 0) itemoff 13169 itemsize 439
root data bytenr 29442048 level 0 dirid 256 refs 1 gen 4 lastsnap 0
uuid 00000000-0000-0000-0000-000000000000
chunk tree
leaf 458817536 items 4 free space 15781 generation 11 owner 3
fs uuid 101ffe75-d95b-41bf-a383-893e9a5f5e21
chunk uuid eec5eaf1-daa5-4329-b185-a8117524e57c
item 0 key (DEV_ITEMS DEV_ITEM 1) itemoff 16185 itemsize 98
dev item devid 1 total_bytes 8589934592 bytes used 934412288
dev uuid 7aefe6b3-949e-4389-9baa-a392970cec30
item 1 key (FIRST_CHUNK_TREE CHUNK_ITEM 12582912) itemoff 16105 itemsize 80
chunk length 8388608 owner 2 stripe_len 65536
type DATA num_stripes 1
stripe 0 devid 1 offset 12582912
dev uuid: 7aefe6b3-949e-4389-9baa-a392970cec30
item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 29360128) itemoff 15993 itemsize 112
chunk length 429457408 owner 2 stripe_len 65536
type METADATA|DUP num_stripes 2
stripe 0 devid 1 offset 37748736
dev uuid: 7aefe6b3-949e-4389-9baa-a392970cec30
stripe 1 devid 1 offset 467206144
dev uuid: 7aefe6b3-949e-4389-9baa-a392970cec30
item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 458817536) itemoff 15881 itemsize 112
chunk length 33554432 owner 2 stripe_len 65536
type SYSTEM|DUP num_stripes 2
stripe 0 devid 1 offset 896663552
dev uuid: 7aefe6b3-949e-4389-9baa-a392970cec30
stripe 1 devid 1 offset 930217984
dev uuid: 7aefe6b3-949e-4389-9baa-a392970cec30
extent tree key (EXTENT_TREE ROOT_ITEM 0)
leaf 29376512 items 12 free space 15594 generation 18 owner 2
fs uuid 101ffe75-d95b-41bf-a383-893e9a5f5e21
chunk uuid eec5eaf1-daa5-4329-b185-a8117524e57c
item 0 key (12582912 BLOCK_GROUP_ITEM 8388608) itemoff 16259 itemsize 24
block group used 65536 chunk_objectid 256 flags DATA
item 1 key (12648448 EXTENT_ITEM 65536) itemoff 16206 itemsize 53
extent refs 1 gen 17 flags DATA
extent data backref root 1 objectid 256 offset 0 count 1
item 2 key (29360128 METADATA_ITEM 0) itemoff 16173 itemsize 33
extent refs 1 gen 18 flags TREE_BLOCK
tree block skinny level 0
tree block backref root 1
item 3 key (29360128 BLOCK_GROUP_ITEM 429457408) itemoff 16149 itemsize 24
block group used 114688 chunk_objectid 256 flags METADATA|DUP
item 4 key (29376512 METADATA_ITEM 0) itemoff 16116 itemsize 33
extent refs 1 gen 18 flags TREE_BLOCK
tree block skinny level 0
tree block backref root 2
item 5 key (29442048 METADATA_ITEM 0) itemoff 16083 itemsize 33
extent refs 1 gen 4 flags TREE_BLOCK
tree block skinny level 0
tree block backref root 18446744073709551607
item 6 key (29573120 METADATA_ITEM 0) itemoff 16050 itemsize 33
extent refs 1 gen 12 flags TREE_BLOCK
tree block skinny level 0
tree block backref root 4
item 7 key (29900800 METADATA_ITEM 0) itemoff 16017 itemsize 33
extent refs 1 gen 16 flags TREE_BLOCK
tree block skinny level 0
tree block backref root 5
item 8 key (29933568 METADATA_ITEM 0) itemoff 15984 itemsize 33
extent refs 1 gen 16 flags TREE_BLOCK
tree block skinny level 0
tree block backref root 9
item 9 key (30015488 METADATA_ITEM 0) itemoff 15951 itemsize 33
extent refs 1 gen 17 flags TREE_BLOCK
tree block skinny level 0
tree block backref root 7
item 10 key (458817536 METADATA_ITEM 0) itemoff 15918 itemsize 33
extent refs 1 gen 11 flags TREE_BLOCK
tree block skinny level 0
tree block backref root 3
item 11 key (458817536 BLOCK_GROUP_ITEM 33554432) itemoff 15894 itemsize 24
block group used 16384 chunk_objectid 256 flags SYSTEM|DUP
device tree key (DEV_TREE ROOT_ITEM 0)
leaf 29573120 items 6 free space 15853 generation 12 owner 4
fs uuid 101ffe75-d95b-41bf-a383-893e9a5f5e21
chunk uuid eec5eaf1-daa5-4329-b185-a8117524e57c
item 0 key (0 DEV_STATS 1) itemoff 16243 itemsize 40
device stats
item 1 key (1 DEV_EXTENT 12582912) itemoff 16195 itemsize 48
dev extent chunk_tree 3
chunk objectid 256 chunk offset 12582912 length 8388608
item 2 key (1 DEV_EXTENT 37748736) itemoff 16147 itemsize 48
dev extent chunk_tree 3
chunk objectid 256 chunk offset 29360128 length 429457408
item 3 key (1 DEV_EXTENT 467206144) itemoff 16099 itemsize 48
dev extent chunk_tree 3
chunk objectid 256 chunk offset 29360128 length 429457408
item 4 key (1 DEV_EXTENT 896663552) itemoff 16051 itemsize 48
dev extent chunk_tree 3
chunk objectid 256 chunk offset 458817536 length 33554432
item 5 key (1 DEV_EXTENT 930217984) itemoff 16003 itemsize 48
dev extent chunk_tree 3
chunk objectid 256 chunk offset 458817536 length 33554432
fs tree key (FS_TREE ROOT_ITEM 0)
leaf 29900800 items 14 free space 15062 generation 16 owner 5
fs uuid 101ffe75-d95b-41bf-a383-893e9a5f5e21
chunk uuid eec5eaf1-daa5-4329-b185-a8117524e57c
item 0 key (256 INODE_ITEM 0) itemoff 16123 itemsize 160
inode generation 3 transid 16 size 6 nbytes 16384
block group 0 mode 40755 links 1 uid 0 gid 0
rdev 0 flags 0x0
item 1 key (256 INODE_REF 256) itemoff 16111 itemsize 12
inode ref index 0 namelen 2 name: ..
item 2 key (256 DIR_ITEM 2086311400) itemoff 16080 itemsize 31
location key (259 INODE_ITEM 0) type FILE
namelen 1 datalen 0 name: 3
item 3 key (256 DIR_ITEM 2385581803) itemoff 16049 itemsize 31
location key (258 INODE_ITEM 0) type FILE
namelen 1 datalen 0 name: 2
item 4 key (256 DIR_ITEM 2640438559) itemoff 16018 itemsize 31
location key (257 INODE_ITEM 0) type FILE
namelen 1 datalen 0 name: 1
item 5 key (256 DIR_INDEX 2) itemoff 15987 itemsize 31
location key (257 INODE_ITEM 0) type FILE
namelen 1 datalen 0 name: 1
item 6 key (256 DIR_INDEX 3) itemoff 15956 itemsize 31
location key (258 INODE_ITEM 0) type FILE
namelen 1 datalen 0 name: 2
item 7 key (256 DIR_INDEX 4) itemoff 15925 itemsize 31
location key (259 INODE_ITEM 0) type FILE
namelen 1 datalen 0 name: 3
item 8 key (257 INODE_ITEM 0) itemoff 15765 itemsize 160
inode generation 10 transid 10 size 0 nbytes 0
block group 0 mode 100644 links 1 uid 0 gid 0
rdev 0 flags 0x0
item 9 key (257 INODE_REF 256) itemoff 15754 itemsize 11
inode ref index 2 namelen 1 name: 1
item 10 key (258 INODE_ITEM 0) itemoff 15594 itemsize 160
inode generation 10 transid 10 size 0 nbytes 0
block group 0 mode 100644 links 1 uid 0 gid 0
rdev 0 flags 0x0
item 11 key (258 INODE_REF 256) itemoff 15583 itemsize 11
inode ref index 3 namelen 1 name: 2
item 12 key (259 INODE_ITEM 0) itemoff 15423 itemsize 160
inode generation 10 transid 10 size 0 nbytes 0
block group 0 mode 100644 links 1 uid 0 gid 0
rdev 0 flags 0x0
item 13 key (259 INODE_REF 256) itemoff 15412 itemsize 11
inode ref index 4 namelen 1 name: 3
checksum tree key (CSUM_TREE ROOT_ITEM 0)
leaf 30015488 items 0 free space 16283 generation 17 owner 7
fs uuid 101ffe75-d95b-41bf-a383-893e9a5f5e21
chunk uuid eec5eaf1-daa5-4329-b185-a8117524e57c
uuid tree key (UUID_TREE ROOT_ITEM 0)
leaf 29933568 items 0 free space 16283 generation 16 owner 9
fs uuid 101ffe75-d95b-41bf-a383-893e9a5f5e21
chunk uuid eec5eaf1-daa5-4329-b185-a8117524e57c
data reloc tree key (DATA_RELOC_TREE ROOT_ITEM 0)
leaf 29442048 items 2 free space 16061 generation 4 owner 18446744073709551607
fs uuid 101ffe75-d95b-41bf-a383-893e9a5f5e21
chunk uuid eec5eaf1-daa5-4329-b185-a8117524e57c
item 0 key (256 INODE_ITEM 0) itemoff 16123 itemsize 160
inode generation 3 transid 0 size 0 nbytes 16384
block group 0 mode 40755 links 1 uid 0 gid 0
rdev 0 flags 0x0
item 1 key (256 INODE_REF 256) itemoff 16111 itemsize 12
inode ref index 0 namelen 2 name: ..
total bytes 8589934592
bytes used 196608
uuid 101ffe75-d95b-41bf-a383-893e9a5f5e21