Running btrfs check
shows several errors like this (13 in total):
checksum verify failed on 369098752 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
All wanted checksums are zeroes, which is obviously suspicious.
After running btrfs scrub
I had exactly 13 corrected errors in metadata and multiple uncorrectable errors in data blocks (fixed by restoring affected files from backup).
However, running btrfs check
again shows the same 13 checksum errors.
I have two questions: what are the numbers in the error referring to? All search results mentioning this question result in no answers. My guess is it's a logical address, but btrfs inspect-internal logical-resolve number /path
gives ERROR: logical ino ioctl: No such file or directory
, and it's obviously not an inode as well (dmesg actually explicitly refers to these numbers as logical addresses, pointing to the metadata leaves, but it doesn't specify of what kind).
And second question, what can I do about these checksums? All the btrfs issues are promptly answered with running btrfs check --init-csum-tree
but my understanding is this will reset csum tree completely and I want to understand where exactly is the issue first.
Filesystem info:
[root@ls-fedora ArchiMedia]# btrfs filesystem usage .
Overall:
Device size: 27.28TiB
Device allocated: 12.02TiB
Device unallocated: 15.27TiB
Device missing: 0.00B
Used: 11.84TiB
Free (estimated): 15.44TiB (min: 7.80TiB)
Free (statfs, df): 15.44TiB
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 0.00B)
Multiple profiles: no
Data,single: Size:11.82TiB, Used:11.65TiB (98.57%)
/dev/mapper/ArchiMedia 11.82TiB
Metadata,RAID1: Size:102.00GiB, Used:100.80GiB (98.82%)
/dev/mapper/ArchiMedia 102.00GiB
/dev/mapper/AM-2 102.00GiB
System,RAID1: Size:32.00MiB, Used:1.30MiB (4.05%)
/dev/mapper/ArchiMedia 32.00MiB
/dev/mapper/AM-2 32.00MiB
Unallocated:
/dev/mapper/ArchiMedia 4.45TiB
/dev/mapper/AM-2 10.81TiB
Full btrfs check
output:
[root@ls-fedora ~]# btrfs check -p /dev/mapper/ArchiMedia
Opening filesystem to check...
Checking filesystem on /dev/mapper/ArchiMedia
UUID: 6e222b94-0fa2-237d-284c-176e55472e30
[1/7] checking root items (0:06:50 elapsed, 17461366 items checked)
checksum verify failed on 369098752 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 369115136 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 369131520 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 369147904 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5962932944896 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5962932961280 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5962932977664 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5962932994048 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5962933010432 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x929dbce6274a6515cb28dcc6e1a5ee75bfdd7247c15ca701b2d272c358d1c544
checksum verify failed on 5964006686720 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5964006703104 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5964006719488 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5964006735872 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
[2/7] checking extents (0:17:42 elapsed, 6613880 items checked)
[3/7] checking free space cache (0:06:57 elapsed, 12204 items checked)
checksum verify failed on 5962933010432 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x929dbce6274a6515cb28dcc6e1a5ee75bfdd7247c15ca701b2d272c358d1c544
checksum verify failed on 5962932994048 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5962932944896 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5962932961280 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5962932977664 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5964006686720 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5964006703104 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5964006719488 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5964006735872 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 369098752 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 369115136 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 369147904 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 369131520 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
[4/7] checking fs roots (1:18:02 elapsed, 129168 items checked)
checksum verify failed on 5962933010432 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x929dbce6274a6515cb28dcc6e1a5ee75bfdd7247c15ca701b2d272c358d1c544
checksum verify failed on 5962932994048 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5962932944896 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5962932961280 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5962932977664 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5964006686720 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5964006703104 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5964006719488 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5964006735872 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 369098752 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 369115136 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 369147904 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 369131520 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
[5/7] checking csums (without verifying data) (1:10:29 elapsed, 25512445 items checked)
[6/7] checking root refs (0:00:00 elapsed, 4 items checked)
[7/7] checking quota groups skipped (not enabled on this FS)
found 12933797396480 bytes used, no error found
total csum bytes: 100175282368
total tree bytes: 108361662464
total fs tree bytes: 2123726848
total extent tree bytes: 1481572352
btree space waste bytes: 3398233945
file data blocks allocated: 15714799702016
referenced 13476542222336
dmesg lines corresponding to the addresses:
[ 6663.817055] BTRFS info (device dm-1): scrub: started on devid 1
[ 6663.818097] BTRFS info (device dm-1): scrub: started on devid 2
[ 6683.992056] BTRFS warning (device dm-1): checksum error at logical 5962932944896 on dev /dev/mapper/AM-2, physical 1498415104: metadata leaf (level 0) in tree 7
[ 6683.992079] BTRFS warning (device dm-1): checksum error at logical 5962932944896 on dev /dev/mapper/AM-2, physical 1498415104: metadata leaf (level 0) in tree 7
[ 6683.992092] BTRFS error (device dm-1): bdev /dev/mapper/AM-2 errs: wr 29, rd 0, flush 0, corrupt 22, gen 0
[ 6683.992254] BTRFS warning (device dm-1): checksum error at logical 5962932961280 on dev /dev/mapper/AM-2, physical 1498431488: metadata leaf (level 0) in tree 7
[ 6683.992263] BTRFS warning (device dm-1): checksum error at logical 5962932961280 on dev /dev/mapper/AM-2, physical 1498431488: metadata leaf (level 0) in tree 7
[ 6683.992270] BTRFS error (device dm-1): bdev /dev/mapper/AM-2 errs: wr 29, rd 0, flush 0, corrupt 23, gen 0
[ 6683.992477] BTRFS warning (device dm-1): checksum error at logical 5962932977664 on dev /dev/mapper/AM-2, physical 1498447872: metadata leaf (level 0) in tree 7
[ 6683.992485] BTRFS warning (device dm-1): checksum error at logical 5962932977664 on dev /dev/mapper/AM-2, physical 1498447872: metadata leaf (level 0) in tree 7
[ 6683.992492] BTRFS error (device dm-1): bdev /dev/mapper/AM-2 errs: wr 29, rd 0, flush 0, corrupt 24, gen 0
[ 6683.992647] BTRFS warning (device dm-1): checksum error at logical 5962932994048 on dev /dev/mapper/AM-2, physical 1498464256: metadata leaf (level 0) in tree 7
[ 6683.992653] BTRFS warning (device dm-1): checksum error at logical 5962932994048 on dev /dev/mapper/AM-2, physical 1498464256: metadata leaf (level 0) in tree 7
[ 6683.992663] BTRFS error (device dm-1): bdev /dev/mapper/AM-2 errs: wr 29, rd 0, flush 0, corrupt 25, gen 0
[ 6683.997006] BTRFS warning (device dm-1): checksum error at logical 5962933010432 on dev /dev/mapper/AM-2, physical 1498480640: metadata leaf (level 0) in tree 7
[ 6683.997028] BTRFS warning (device dm-1): checksum error at logical 5962933010432 on dev /dev/mapper/AM-2, physical 1498480640: metadata leaf (level 0) in tree 7
[ 6683.997041] BTRFS error (device dm-1): bdev /dev/mapper/AM-2 errs: wr 29, rd 0, flush 0, corrupt 26, gen 0
[ 6702.992865] BTRFS warning (device dm-1): checksum error at logical 5964006686720 on dev /dev/mapper/AM-2, physical 2572156928: metadata leaf (level 0) in tree 7
[ 6702.992880] BTRFS warning (device dm-1): checksum error at logical 5964006686720 on dev /dev/mapper/AM-2, physical 2572156928: metadata leaf (level 0) in tree 7
[ 6702.992888] BTRFS error (device dm-1): bdev /dev/mapper/AM-2 errs: wr 29, rd 0, flush 0, corrupt 27, gen 0
[ 6702.992985] BTRFS warning (device dm-1): checksum error at logical 5964006703104 on dev /dev/mapper/AM-2, physical 2572173312: metadata leaf (level 0) in tree 7
[ 6702.992990] BTRFS warning (device dm-1): checksum error at logical 5964006703104 on dev /dev/mapper/AM-2, physical 2572173312: metadata leaf (level 0) in tree 7
[ 6702.992994] BTRFS error (device dm-1): bdev /dev/mapper/AM-2 errs: wr 29, rd 0, flush 0, corrupt 28, gen 0
[ 6702.993083] BTRFS warning (device dm-1): checksum error at logical 5964006719488 on dev /dev/mapper/AM-2, physical 2572189696: metadata leaf (level 0) in tree 7
[ 6702.993086] BTRFS warning (device dm-1): checksum error at logical 5964006719488 on dev /dev/mapper/AM-2, physical 2572189696: metadata leaf (level 0) in tree 7
[ 6702.993090] BTRFS error (device dm-1): bdev /dev/mapper/AM-2 errs: wr 29, rd 0, flush 0, corrupt 29, gen 0
[ 6702.993177] BTRFS warning (device dm-1): checksum error at logical 5964006735872 on dev /dev/mapper/AM-2, physical 2572206080: metadata leaf (level 0) in tree 7
[ 6702.993181] BTRFS warning (device dm-1): checksum error at logical 5964006735872 on dev /dev/mapper/AM-2, physical 2572206080: metadata leaf (level 0) in tree 7
[ 6702.993185] BTRFS error (device dm-1): bdev /dev/mapper/AM-2 errs: wr 29, rd 0, flush 0, corrupt 30, gen 0
[ 6721.512615] BTRFS warning (device dm-1): checksum error at logical 369098752 on dev /dev/mapper/AM-2, physical 3645898752: metadata leaf (level 0) in tree 7
[ 6721.512627] BTRFS warning (device dm-1): checksum error at logical 369098752 on dev /dev/mapper/AM-2, physical 3645898752: metadata leaf (level 0) in tree 7
[ 6721.512633] BTRFS error (device dm-1): bdev /dev/mapper/AM-2 errs: wr 29, rd 0, flush 0, corrupt 31, gen 0
[ 6721.512703] BTRFS warning (device dm-1): checksum error at logical 369115136 on dev /dev/mapper/AM-2, physical 3645915136: metadata leaf (level 0) in tree 7
[ 6721.512707] BTRFS warning (device dm-1): checksum error at logical 369115136 on dev /dev/mapper/AM-2, physical 3645915136: metadata leaf (level 0) in tree 7
[ 6721.512709] BTRFS error (device dm-1): bdev /dev/mapper/AM-2 errs: wr 29, rd 0, flush 0, corrupt 32, gen 0
[ 6721.512775] BTRFS warning (device dm-1): checksum error at logical 369131520 on dev /dev/mapper/AM-2, physical 3645931520: metadata leaf (level 0) in tree 7
[ 6721.512777] BTRFS warning (device dm-1): checksum error at logical 369131520 on dev /dev/mapper/AM-2, physical 3645931520: metadata leaf (level 0) in tree 7
[ 6721.512780] BTRFS error (device dm-1): bdev /dev/mapper/AM-2 errs: wr 29, rd 0, flush 0, corrupt 33, gen 0
[ 6721.516630] BTRFS warning (device dm-1): checksum error at logical 369147904 on dev /dev/mapper/AM-2, physical 3645947904: metadata leaf (level 0) in tree 7
[ 6721.516655] BTRFS warning (device dm-1): checksum error at logical 369147904 on dev /dev/mapper/AM-2, physical 3645947904: metadata leaf (level 0) in tree 7
[ 6721.516668] BTRFS error (device dm-1): bdev /dev/mapper/AM-2 errs: wr 29, rd 0, flush 0, corrupt 34, gen 0
[ 6771.128355] BTRFS error (device dm-1): bad tree block start, want 5962933010432 have 0
[ 6771.157972] BTRFS info (device dm-1): read error corrected: ino 0 off 5962933010432 (dev /dev/mapper/AM-2 sector 2926720)
[ 6771.157979] BTRFS info (device dm-1): read error corrected: ino 0 off 5962933014528 (dev /dev/mapper/AM-2 sector 2926728)
[ 6771.159093] BTRFS info (device dm-1): read error corrected: ino 0 off 5962933018624 (dev /dev/mapper/AM-2 sector 2926736)
[ 6771.159422] BTRFS info (device dm-1): read error corrected: ino 0 off 5962933022720 (dev /dev/mapper/AM-2 sector 2926744)
[ 6913.335267] BTRFS warning (device dm-1): checksum error at logical 43113840640 on dev /dev/mapper/ArchiMedia, physical 44184436736, root 5, inode 4981, offset 131072, length 4096, links 1 (path: Archive/Develop/SDKs/Mobile/Android/android-sdk-windows/docs/reference/android/widget/MediaController.html)
[ 6913.335277] BTRFS error (device dm-1): bdev /dev/mapper/ArchiMedia errs: wr 0, rd 0, flush 0, corrupt 135, gen 0
[ 7229.380941] BTRFS error (device dm-1): bad tree block start, want 5962932994048 have 0
[ 7229.425140] BTRFS info (device dm-1): read error corrected: ino 0 off 5962932994048 (dev /dev/mapper/AM-2 sector 2926688)
[ 7229.425160] BTRFS info (device dm-1): read error corrected: ino 0 off 5962932998144 (dev /dev/mapper/AM-2 sector 2926696)
[ 7229.425170] BTRFS info (device dm-1): read error corrected: ino 0 off 5962933002240 (dev /dev/mapper/AM-2 sector 2926704)
[ 7229.425176] BTRFS info (device dm-1): read error corrected: ino 0 off 5962933006336 (dev /dev/mapper/AM-2 sector 2926712)
[ 7241.946885] BTRFS error (device dm-1): bad tree block start, want 5962932944896 have 0
[ 7241.982749] BTRFS info (device dm-1): read error corrected: ino 0 off 5962932944896 (dev /dev/mapper/AM-2 sector 2926592)
[ 7241.982775] BTRFS info (device dm-1): read error corrected: ino 0 off 5962932948992 (dev /dev/mapper/AM-2 sector 2926600)
[ 7241.982788] BTRFS info (device dm-1): read error corrected: ino 0 off 5962932953088 (dev /dev/mapper/AM-2 sector 2926608)
[ 7241.982797] BTRFS info (device dm-1): read error corrected: ino 0 off 5962932957184 (dev /dev/mapper/AM-2 sector 2926616)
[ 7314.107076] BTRFS error (device dm-1): bad tree block start, want 5962932961280 have 0
[ 7314.131362] BTRFS info (device dm-1): read error corrected: ino 0 off 5962932961280 (dev /dev/mapper/AM-2 sector 2926624)
[ 7314.131386] BTRFS info (device dm-1): read error corrected: ino 0 off 5962932965376 (dev /dev/mapper/AM-2 sector 2926632)
[ 7314.131396] BTRFS info (device dm-1): read error corrected: ino 0 off 5962932969472 (dev /dev/mapper/AM-2 sector 2926640)
[ 7314.131403] BTRFS info (device dm-1): read error corrected: ino 0 off 5962932973568 (dev /dev/mapper/AM-2 sector 2926648)
[ 7353.926345] BTRFS error (device dm-1): bad tree block start, want 5962932977664 have 0
[ 7353.964151] BTRFS info (device dm-1): read error corrected: ino 0 off 5962932977664 (dev /dev/mapper/AM-2 sector 2926656)
[ 7353.964169] BTRFS info (device dm-1): read error corrected: ino 0 off 5962932981760 (dev /dev/mapper/AM-2 sector 2926664)
[ 7353.964178] BTRFS info (device dm-1): read error corrected: ino 0 off 5962932985856 (dev /dev/mapper/AM-2 sector 2926672)
[ 7353.964185] BTRFS info (device dm-1): read error corrected: ino 0 off 5962932989952 (dev /dev/mapper/AM-2 sector 2926680)