Esta pagina se ve mejor con JavaScript habilitado

BTRFS recovery

 ·  🎃 kr0m

BTRFS tiene muchas características interesantes pero cuando la cosa se pone fea también puede ser complicado recuperar el sistema de ficheros o incluso los datos, en este artículo voy a explicar los pasos básicos a seguir en caso de desastre.

Para realizar las pruebas generamos un fichero de 8G que montaremos mediante loop y que formatearemos en BTRFS:

dd if=/dev/zero of=mountedImage.img bs=1g count=8
losetup /dev/loop1 mountedImage.img
mkfs.btrfs /dev/loop1
mount /dev/loop1 /mnt/aux/

DETECTAR ERRORES

Mediante btrfs scrub se verificará el checksum de todos los datos del sistema de ficheros, si se detecta un error y es corregible leyendo el dato correcto de otra parte del sistema de ficheros(RAID1 por ejemplo) se corregirá de forma automática.

El scrub consumirá aproximadamente el 80% del I/O del dispositivo en cuestión, para la mayoría de escenarios es algo asumible sin problema.
Lo lanzamos en foreground para ver los resultado, en caso contrario consultaríamos: /var/lib/btrfs/scrub.status.ID

btrfs scrub start /mnt/aux/ -B
scrub done for 101ffe75-d95b-41bf-a383-893e9a5f5e21
scrub started at Wed May 17 10:13:33 2017 and finished after 00:00:00
total bytes scrubbed: 320.00KiB with 0 errors

cat /var/lib/btrfs/scrub.status.101ffe75-d95b-41bf-a383-893e9a5f5e21
scrub status:1
101ffe75-d95b-41bf-a383-893e9a5f5e21:1|data_extents_scrubbed:1|tree_extents_scrubbed:16|data_bytes_scrubbed:65536|tree_bytes_scrubbed:262144|read_errors:0|csum_errors:0|verify_errors:0|no_csum:16|csum_discards:0|super_errors:0|malloc_errors:0|uncorrectable_errors:0|corrected_errors:0|last_physical:896663552|t_start:1495008813|t_resumed:0|duration:0|canceled:0|finished:1

Podemos consultar los contadores de errores con:

btrfs device stats /mnt/aux
[/dev/loop1].write_io_errs 0
[/dev/loop1].read_io_errs 0
[/dev/loop1].flush_io_errs 0
[/dev/loop1].corruption_errs 0
[/dev/loop1].generation_errs 0

Una forma de automatizar la monitorización BTRFS es mediante SEC, para ello debemos lanzar scrubs en background para que dejen constancia de los errores encontrados en los logs del sistema, mediante SEC revisaremos dichos logs y alertaremos en función de los resultados del parseo:

emerge -av net-analyzer/sec
vi /etc/conf.d/sec

# configuration file for /etc/init.d/sec
# flags to pass to sec (see 'sec --help')
SEC_FLAGS=""
# Define a debug level (1..6)
DEBUG_LEVEL="4"
# define where sec reads messages from for translating them
#INPUT_FILES="${INPUT_FILES} -input=/var/log/some.log"
#INPUT_FILES="${INPUT_FILES} -input=/tmp/other.file"
INPUT_FILES="-input=/var/log/messages"
vi /etc/sec.conf
\# http://simple-evcorr.sourceforge.net/man.html 
\# http://sixshooter.v6.thrupoint.net/SEC-examples/article.html 
\# http://sixshooter.v6.thrupoint.net/SEC-examples/article-part2.html 
type=SingleWithSuppress 
ptype=RegExp 
pattern=(?i)kernel.*btrfs: (?!disk space caching is enabled|use ssd allocation|use .* compression|unlinked .* orphans|turning on discard|device label .* devid .* transid|detected SSD devices, enabling SSD mode|has skinny extents|device label|creating UUID tree|checking UUID tree|setting .* feature flag|bdev.* flush 0, corrupt 0, gen 0) 
window=60 
desc=Btrfs unexpected log 
action=pipe '%t: $0' /usr/bin/mail -s "sec: %s" root

DUMPEAR DATOS

Con un poco de suerte montando en RO será suficiente:

mount -o ro,recovery

Un restore es inocuo, no intentará reparar nada, tan solo leer:

btrfs restore /dev/loop1 /mnt/aux

Las opciones disponibles para restore son:

-s: Also restore snapshots. Without this option snapshots will not be restored.
-x: Get extended attributes. Without this option, extented attributes will not be retrieved.
-m: Restore metadata: owner, mode and times.
-S: Restore symbolic links.
-v: Increase verbosity. May be given multiple times.
-i: Ignore errors. Normally the restore tool exits immediately for any error. This option forces it to keep going if it can, usually this results in some missing data.
-o: Overwrite existing files. If files exist at the output location with the same name, normally the restore utility will skip restoring that file. This option will overwrite the existing files instead.
-t: Tree location. The location of the tree of tree roots.
-f: Filesystem location. The byte number given will be used as the root of the filesystem, instead of the location specified by the superblock. This can be useful if your superblocks are inconsistent.
-u: Superblock mirror. Valid values are 0,1,2. Specifies an alternate superblock copy to use. This may be useful if your 0th superblock is damaged.
-d: ???
-r: Root objectid. The objectid given will be used as the root of the filesystem, instead of the location specified by the superblock. This may assist in recovery of subvolumes where the real root is damaged.
-c: Case insensitive regex matching.
-l: List tree roots.
-D: Dry run (only list files that would be recovered).
--path-regex: Regex for files to restore. In order to restore only a single folder somewhere in the btrfs tree, it is unfortunately necessary to construct a slightly nontrivial regex, e.g.: ‘^/(|home(|/username(|/Desktop(|/.*))))$’

Otra opción es dumpear a partir de cierto arbol, pero es un método un poco mas complicado, para mostrar el funcionamiento creamos 3 ficheros:

cd /mnt/aux/
touch 1 2 3

Creamos algunos snapshots para que el arbol crezca:

btrfs subvolume snapshot /mnt/aux/ /mnt/aux/snap1
btrfs subvolume delete /mnt/aux/snap1/
btrfs subvolume create /mnt/aux/subvol1
btrfs subvolume delete /mnt/aux/subvol1

A pesar de no haber maltratado el sistema de ficheros en absoluto aparecen mensajes un tanto preocupantes: seems good, but generation/level doesn’t match

btrfs-find-root /dev/loop1

Superblock thinks the generation is 17
Superblock thinks the level is 0
Found tree root at 29982720 gen 17 level 0
Well block 29917184(gen: 16 level: 0) seems good, but generation/level doesn't match, want gen: 17 level: 0
Well block 29868032(gen: 15 level: 0) seems good, but generation/level doesn't match, want gen: 17 level: 0
Well block 29704192(gen: 14 level: 0) seems good, but generation/level doesn't match, want gen: 17 level: 0
Well block 29605888(gen: 13 level: 0) seems good, but generation/level doesn't match, want gen: 17 level: 0
Well block 29392896(gen: 12 level: 0) seems good, but generation/level doesn't match, want gen: 17 level: 0
Well block 29523968(gen: 11 level: 0) seems good, but generation/level doesn't match, want gen: 17 level: 0

Se ha encontrado un root: 29982720 con el arbol: 16,15,14,13,12,11 con sus respectivos números de bloque: 29917184, 29868032, 29704192, 29605888, 29392896, 29523968.

mkdir /mnt/restore
btrfs restore -t 29917184 /dev/loop1 /mnt/restore

parent transid verify failed on 29917184 wanted 17 found 16
parent transid verify failed on 29917184 wanted 17 found 16
parent transid verify failed on 29917184 wanted 17 found 16
parent transid verify failed on 29917184 wanted 17 found 16
Ignoring transid failure
ls /mnt/restore/
1 2 3

REPARAR SISTEMA DE FICHEROS

Existe una herramienta muy PELIGROSA llamada btrfs-zero-log que SOLO debemos utilizar si vemos exactamente este error:

BTRFS: failed to read log tree

NOTA: Si ejecutamos la herramienta sin necesidad será mas complicado recuperar los datos!!

Otra opción es pasar un check pero es desaconsejable ya que el código está por pulir y podría llegar a causar daños(It is possible (but highly unlikely in the most recent versions) that it may cause additional damage in the process of repair)

umount /mnt/aux
btrfs check --repair /dev/loop1 2>&1 | tee /tmp/repair

checking extents
Fixed 0 roots.
checking free space cache
checking fs roots
checking csums
checking root refs
enabling repair mode
Checking filesystem on /dev/loop1
UUID: 101ffe75-d95b-41bf-a383-893e9a5f5e21
cache and super generation don't match, space cache will be invalidated
found 196608 bytes used err is 0
total csum bytes: 0
total tree bytes: 131072
total fs tree bytes: 32768
total extent tree bytes: 16384
btree space waste bytes: 123761
file data blocks allocated: 65536
 referenced 65536

DUMPEAR METADATOS

btrfs-debug-tree /dev/loop1
btrfs-debug-tree /dev/loop1
btrfs-progs v4.6.1
root tree
leaf 29360128 items 13 free space 12844 generation 18 owner 1
fs uuid 101ffe75-d95b-41bf-a383-893e9a5f5e21
chunk uuid eec5eaf1-daa5-4329-b185-a8117524e57c
    item 0 key (EXTENT_TREE ROOT_ITEM 0) itemoff 15844 itemsize 439
        root data bytenr 29376512 level 0 dirid 0 refs 1 gen 18 lastsnap 0
        uuid 00000000-0000-0000-0000-000000000000
    item 1 key (DEV_TREE ROOT_ITEM 0) itemoff 15405 itemsize 439
        root data bytenr 29573120 level 0 dirid 0 refs 1 gen 12 lastsnap 0
        uuid 00000000-0000-0000-0000-000000000000
    item 2 key (FS_TREE INODE_REF 6) itemoff 15388 itemsize 17
        inode ref index 0 namelen 7 name: default
    item 3 key (FS_TREE ROOT_ITEM 0) itemoff 14949 itemsize 439
        root data bytenr 29900800 level 0 dirid 256 refs 1 gen 16 lastsnap 14
        uuid 00000000-0000-0000-0000-000000000000
        ctransid 16 otransid 0 stransid 0 rtransid 0
    item 4 key (ROOT_TREE_DIR INODE_ITEM 0) itemoff 14789 itemsize 160
        inode generation 3 transid 0 size 0 nbytes 16384
        block group 0 mode 40755 links 1 uid 0 gid 0
        rdev 0 flags 0x0
    item 5 key (ROOT_TREE_DIR INODE_REF 6) itemoff 14777 itemsize 12
        inode ref index 0 namelen 2 name: ..
    item 6 key (ROOT_TREE_DIR DIR_ITEM 2378154706) itemoff 14740 itemsize 37
        location key (FS_TREE ROOT_ITEM -1) type DIR
        namelen 7 datalen 0 name: default
    item 7 key (CSUM_TREE ROOT_ITEM 0) itemoff 14301 itemsize 439
        root data bytenr 30015488 level 0 dirid 0 refs 1 gen 17 lastsnap 0
        uuid 00000000-0000-0000-0000-000000000000
    item 8 key (UUID_TREE ROOT_ITEM 0) itemoff 13862 itemsize 439
        root data bytenr 29933568 level 0 dirid 0 refs 1 gen 16 lastsnap 0
        uuid ce1e052c-8632-a34b-ad7c-cc2dd9ea4944
    item 9 key (256 INODE_ITEM 0) itemoff 13702 itemsize 160
        inode generation 17 transid 17 size 65536 nbytes 786432
        block group 0 mode 100600 links 1 uid 0 gid 0
        rdev 0 flags 0x1b
    item 10 key (256 EXTENT_DATA 0) itemoff 13649 itemsize 53
        extent data disk byte 12648448 nr 65536
        extent data offset 0 nr 65536 ram 65536
        extent compression(none)
    item 11 key (FREE_SPACE UNTYPED 29360128) itemoff 13608 itemsize 41
        location key (256 INODE_ITEM 0)
        cache generation 17 entries 7 bitmaps 0
    item 12 key (DATA_RELOC_TREE ROOT_ITEM 0) itemoff 13169 itemsize 439
        root data bytenr 29442048 level 0 dirid 256 refs 1 gen 4 lastsnap 0
        uuid 00000000-0000-0000-0000-000000000000
chunk tree
leaf 458817536 items 4 free space 15781 generation 11 owner 3
fs uuid 101ffe75-d95b-41bf-a383-893e9a5f5e21
chunk uuid eec5eaf1-daa5-4329-b185-a8117524e57c
    item 0 key (DEV_ITEMS DEV_ITEM 1) itemoff 16185 itemsize 98
        dev item devid 1 total_bytes 8589934592 bytes used 934412288
        dev uuid 7aefe6b3-949e-4389-9baa-a392970cec30
    item 1 key (FIRST_CHUNK_TREE CHUNK_ITEM 12582912) itemoff 16105 itemsize 80
        chunk length 8388608 owner 2 stripe_len 65536
        type DATA num_stripes 1
            stripe 0 devid 1 offset 12582912
            dev uuid: 7aefe6b3-949e-4389-9baa-a392970cec30
    item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 29360128) itemoff 15993 itemsize 112
        chunk length 429457408 owner 2 stripe_len 65536
        type METADATA|DUP num_stripes 2
            stripe 0 devid 1 offset 37748736
            dev uuid: 7aefe6b3-949e-4389-9baa-a392970cec30
            stripe 1 devid 1 offset 467206144
            dev uuid: 7aefe6b3-949e-4389-9baa-a392970cec30
    item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 458817536) itemoff 15881 itemsize 112
        chunk length 33554432 owner 2 stripe_len 65536
        type SYSTEM|DUP num_stripes 2
            stripe 0 devid 1 offset 896663552
            dev uuid: 7aefe6b3-949e-4389-9baa-a392970cec30
            stripe 1 devid 1 offset 930217984
            dev uuid: 7aefe6b3-949e-4389-9baa-a392970cec30
extent tree key (EXTENT_TREE ROOT_ITEM 0)
leaf 29376512 items 12 free space 15594 generation 18 owner 2
fs uuid 101ffe75-d95b-41bf-a383-893e9a5f5e21
chunk uuid eec5eaf1-daa5-4329-b185-a8117524e57c
    item 0 key (12582912 BLOCK_GROUP_ITEM 8388608) itemoff 16259 itemsize 24
        block group used 65536 chunk_objectid 256 flags DATA
    item 1 key (12648448 EXTENT_ITEM 65536) itemoff 16206 itemsize 53
        extent refs 1 gen 17 flags DATA
        extent data backref root 1 objectid 256 offset 0 count 1
    item 2 key (29360128 METADATA_ITEM 0) itemoff 16173 itemsize 33
        extent refs 1 gen 18 flags TREE_BLOCK
        tree block skinny level 0
        tree block backref root 1
    item 3 key (29360128 BLOCK_GROUP_ITEM 429457408) itemoff 16149 itemsize 24
        block group used 114688 chunk_objectid 256 flags METADATA|DUP
    item 4 key (29376512 METADATA_ITEM 0) itemoff 16116 itemsize 33
        extent refs 1 gen 18 flags TREE_BLOCK
        tree block skinny level 0
        tree block backref root 2
    item 5 key (29442048 METADATA_ITEM 0) itemoff 16083 itemsize 33
        extent refs 1 gen 4 flags TREE_BLOCK
        tree block skinny level 0
        tree block backref root 18446744073709551607
    item 6 key (29573120 METADATA_ITEM 0) itemoff 16050 itemsize 33
        extent refs 1 gen 12 flags TREE_BLOCK
        tree block skinny level 0
        tree block backref root 4
    item 7 key (29900800 METADATA_ITEM 0) itemoff 16017 itemsize 33
        extent refs 1 gen 16 flags TREE_BLOCK
        tree block skinny level 0
        tree block backref root 5
    item 8 key (29933568 METADATA_ITEM 0) itemoff 15984 itemsize 33
        extent refs 1 gen 16 flags TREE_BLOCK
        tree block skinny level 0
        tree block backref root 9
    item 9 key (30015488 METADATA_ITEM 0) itemoff 15951 itemsize 33
        extent refs 1 gen 17 flags TREE_BLOCK
        tree block skinny level 0
        tree block backref root 7
    item 10 key (458817536 METADATA_ITEM 0) itemoff 15918 itemsize 33
        extent refs 1 gen 11 flags TREE_BLOCK
        tree block skinny level 0
        tree block backref root 3
    item 11 key (458817536 BLOCK_GROUP_ITEM 33554432) itemoff 15894 itemsize 24
        block group used 16384 chunk_objectid 256 flags SYSTEM|DUP
device tree key (DEV_TREE ROOT_ITEM 0)
leaf 29573120 items 6 free space 15853 generation 12 owner 4
fs uuid 101ffe75-d95b-41bf-a383-893e9a5f5e21
chunk uuid eec5eaf1-daa5-4329-b185-a8117524e57c
    item 0 key (0 DEV_STATS 1) itemoff 16243 itemsize 40
        device stats
    item 1 key (1 DEV_EXTENT 12582912) itemoff 16195 itemsize 48
        dev extent chunk_tree 3
        chunk objectid 256 chunk offset 12582912 length 8388608
    item 2 key (1 DEV_EXTENT 37748736) itemoff 16147 itemsize 48
        dev extent chunk_tree 3
        chunk objectid 256 chunk offset 29360128 length 429457408
    item 3 key (1 DEV_EXTENT 467206144) itemoff 16099 itemsize 48
        dev extent chunk_tree 3
        chunk objectid 256 chunk offset 29360128 length 429457408
    item 4 key (1 DEV_EXTENT 896663552) itemoff 16051 itemsize 48
        dev extent chunk_tree 3
        chunk objectid 256 chunk offset 458817536 length 33554432
    item 5 key (1 DEV_EXTENT 930217984) itemoff 16003 itemsize 48
        dev extent chunk_tree 3
        chunk objectid 256 chunk offset 458817536 length 33554432
fs tree key (FS_TREE ROOT_ITEM 0)
leaf 29900800 items 14 free space 15062 generation 16 owner 5
fs uuid 101ffe75-d95b-41bf-a383-893e9a5f5e21
chunk uuid eec5eaf1-daa5-4329-b185-a8117524e57c
    item 0 key (256 INODE_ITEM 0) itemoff 16123 itemsize 160
        inode generation 3 transid 16 size 6 nbytes 16384
        block group 0 mode 40755 links 1 uid 0 gid 0
        rdev 0 flags 0x0
    item 1 key (256 INODE_REF 256) itemoff 16111 itemsize 12
        inode ref index 0 namelen 2 name: ..
    item 2 key (256 DIR_ITEM 2086311400) itemoff 16080 itemsize 31
        location key (259 INODE_ITEM 0) type FILE
        namelen 1 datalen 0 name: 3
    item 3 key (256 DIR_ITEM 2385581803) itemoff 16049 itemsize 31
        location key (258 INODE_ITEM 0) type FILE
        namelen 1 datalen 0 name: 2
    item 4 key (256 DIR_ITEM 2640438559) itemoff 16018 itemsize 31
        location key (257 INODE_ITEM 0) type FILE
        namelen 1 datalen 0 name: 1
    item 5 key (256 DIR_INDEX 2) itemoff 15987 itemsize 31
        location key (257 INODE_ITEM 0) type FILE
        namelen 1 datalen 0 name: 1
    item 6 key (256 DIR_INDEX 3) itemoff 15956 itemsize 31
        location key (258 INODE_ITEM 0) type FILE
        namelen 1 datalen 0 name: 2
    item 7 key (256 DIR_INDEX 4) itemoff 15925 itemsize 31
        location key (259 INODE_ITEM 0) type FILE
        namelen 1 datalen 0 name: 3
    item 8 key (257 INODE_ITEM 0) itemoff 15765 itemsize 160
        inode generation 10 transid 10 size 0 nbytes 0
        block group 0 mode 100644 links 1 uid 0 gid 0
        rdev 0 flags 0x0
    item 9 key (257 INODE_REF 256) itemoff 15754 itemsize 11
        inode ref index 2 namelen 1 name: 1
    item 10 key (258 INODE_ITEM 0) itemoff 15594 itemsize 160
        inode generation 10 transid 10 size 0 nbytes 0
        block group 0 mode 100644 links 1 uid 0 gid 0
        rdev 0 flags 0x0
    item 11 key (258 INODE_REF 256) itemoff 15583 itemsize 11
        inode ref index 3 namelen 1 name: 2
    item 12 key (259 INODE_ITEM 0) itemoff 15423 itemsize 160
        inode generation 10 transid 10 size 0 nbytes 0
        block group 0 mode 100644 links 1 uid 0 gid 0
        rdev 0 flags 0x0
    item 13 key (259 INODE_REF 256) itemoff 15412 itemsize 11
        inode ref index 4 namelen 1 name: 3
checksum tree key (CSUM_TREE ROOT_ITEM 0)
leaf 30015488 items 0 free space 16283 generation 17 owner 7
fs uuid 101ffe75-d95b-41bf-a383-893e9a5f5e21
chunk uuid eec5eaf1-daa5-4329-b185-a8117524e57c
uuid tree key (UUID_TREE ROOT_ITEM 0)
leaf 29933568 items 0 free space 16283 generation 16 owner 9
fs uuid 101ffe75-d95b-41bf-a383-893e9a5f5e21
chunk uuid eec5eaf1-daa5-4329-b185-a8117524e57c
data reloc tree key (DATA_RELOC_TREE ROOT_ITEM 0)
leaf 29442048 items 2 free space 16061 generation 4 owner 18446744073709551607
fs uuid 101ffe75-d95b-41bf-a383-893e9a5f5e21
chunk uuid eec5eaf1-daa5-4329-b185-a8117524e57c
    item 0 key (256 INODE_ITEM 0) itemoff 16123 itemsize 160
        inode generation 3 transid 0 size 0 nbytes 16384
        block group 0 mode 40755 links 1 uid 0 gid 0
        rdev 0 flags 0x0
    item 1 key (256 INODE_REF 256) itemoff 16111 itemsize 12
        inode ref index 0 namelen 2 name: ..
total bytes 8589934592
bytes used 196608
uuid 101ffe75-d95b-41bf-a383-893e9a5f5e21
Si te ha gustado el artículo puedes invitarme a un RedBull aquí