Bug #5315
closedjenkins node rpi4 throws i/o errors, mounted root partition read-only
Added by osmith over 2 years ago. Updated over 2 years ago.
100%
Description
After write errors, the SD card has been mounted read-only.
I guess somebody in the office should power off, dd the contents of this SD card to a new one, and boot again?
I'll mark the node in jenkins as temporary offline for now.
[14762353.310716] blk_update_request: I/O error, dev mmcblk0, sector 34172976 op 0x1:(WRITE) flags 0x103000 phys_seg 2 prio class 0 [14762353.310765] Buffer I/O error on dev mmcblk0p2, logical block 4205062, lost async page write [14762353.311434] Buffer I/O error on dev mmcblk0p2, logical block 4205063, lost async page write [14762353.312127] EXT4-fs error (device mmcblk0p2): ext4_check_bdev_write_error:216: comm rs:main Q:Reg: Error while async write back metadata [14762353.312203] blk_update_request: I/O error, dev mmcblk0, sector 34173024 op 0x1:(WRITE) flags 0x103000 phys_seg 4 prio class 0 [14762353.312238] Buffer I/O error on dev mmcblk0p2, logical block 4205068, lost async page write [14762353.312279] Buffer I/O error on dev mmcblk0p2, logical block 4205069, lost async page write [14762353.312316] Buffer I/O error on dev mmcblk0p2, logical block 4205070, lost async page write [14762353.312352] Buffer I/O error on dev mmcblk0p2, logical block 4205071, lost async page write [14762353.313686] blk_update_request: I/O error, dev mmcblk0, sector 34173080 op 0x1:(WRITE) flags 0x103000 phys_seg 1 prio class 0 [14762353.313719] Buffer I/O error on dev mmcblk0p2, logical block 4205075, lost async page write [14762353.314917] blk_update_request: I/O error, dev mmcblk0, sector 34173120 op 0x1:(WRITE) flags 0x103000 phys_seg 2 prio class 0 [14762353.314949] Buffer I/O error on dev mmcblk0p2, logical block 4205080, lost async page write [14762353.314988] Buffer I/O error on dev mmcblk0p2, logical block 4205081, lost async page write [14762353.316075] blk_update_request: I/O error, dev mmcblk0, sector 34173728 op 0x1:(WRITE) flags 0x103000 phys_seg 1 prio class 0 [14762353.316106] Buffer I/O error on dev mmcblk0p2, logical block 4205156, lost async page write [14762353.317330] blk_update_request: I/O error, dev mmcblk0, sector 34173744 op 0x1:(WRITE) flags 0x103000 phys_seg 2 prio class 0 [14762353.318572] blk_update_request: I/O error, dev mmcblk0, sector 34173904 op 0x1:(WRITE) flags 0x103000 phys_seg 5 prio class 0 [14762353.319822] blk_update_request: I/O error, dev mmcblk0, sector 34174896 op 0x1:(WRITE) flags 0x103000 phys_seg 5 prio class 0 [14762353.322015] blk_update_request: I/O error, dev mmcblk0, sector 34174944 op 0x1:(WRITE) flags 0x103000 phys_seg 12 prio class 0 [14762353.323172] blk_update_request: I/O error, dev mmcblk0, sector 34175248 op 0x1:(WRITE) flags 0x103000 phys_seg 1 prio class 0 [14762353.644971] EXT4-fs warning (device mmcblk0p2): ext4_end_bio:349: I/O error 10 writing to inode 1061363 starting block 4324406) [14762353.645025] EXT4-fs warning (device mmcblk0p2): ext4_end_bio:349: I/O error 10 writing to inode 1061365 starting block 4324457) [14762353.645788] EXT4-fs warning (device mmcblk0p2): ext4_end_bio:349: I/O error 10 writing to inode 1061367 starting block 4324458) [14762353.645818] EXT4-fs warning (device mmcblk0p2): ext4_end_bio:349: I/O error 10 writing to inode 1061368 starting block 4324459) [14762353.646294] EXT4-fs warning (device mmcblk0p2): ext4_end_bio:349: I/O error 10 writing to inode 1061370 starting block 4324460) [14762353.646325] EXT4-fs warning (device mmcblk0p2): ext4_end_bio:349: I/O error 10 writing to inode 1061371 starting block 4324461) [14762353.646354] EXT4-fs warning (device mmcblk0p2): ext4_end_bio:349: I/O error 10 writing to inode 1061372 starting block 4324462) [14762353.646386] EXT4-fs warning (device mmcblk0p2): ext4_end_bio:349: I/O error 10 writing to inode 1061377 starting block 4324463) [14762353.646414] EXT4-fs warning (device mmcblk0p2): ext4_end_bio:349: I/O error 10 writing to inode 1061380 starting block 4324464) [14762353.646445] EXT4-fs warning (device mmcblk0p2): ext4_end_bio:349: I/O error 10 writing to inode 1061373 starting block 4324468) [14762353.646604] EXT4-fs error (device mmcblk0p2): ext4_check_bdev_write_error:216: comm kworker/u8:3: Error while async write back metadata [14762353.717321] Aborting journal on device mmcblk0p2-8. [14762353.717393] Buffer I/O error on device mmcblk0p2, logical block 606797 [14762353.718780] Buffer I/O error on device mmcblk0p2, logical block 606799 [14762353.720152] Buffer I/O error on device mmcblk0p2, logical block 2562736 [14762353.723258] Buffer I/O error on device mmcblk0p2, logical block 3340204 [14762353.780374] EXT4-fs (mmcblk0p2): previous I/O error to superblock detected [14762353.786283] JBD2: Error -5 detected when updating journal superblock for mmcblk0p2-8. [14762353.786927] EXT4-fs (mmcblk0p2): I/O error while writing superblock [14762353.786934] EXT4-fs (mmcblk0p2): previous I/O error to superblock detected [14762353.786951] EXT4-fs error (device mmcblk0p2) in ext4_reserve_inode_write:5749: Journal has aborted [14762353.789117] EXT4-fs (mmcblk0p2): I/O error while writing superblock [14762353.789133] EXT4-fs error (device mmcblk0p2): ext4_journal_check_start:83: Detected aborted journal [14762353.789168] EXT4-fs (mmcblk0p2): Remounting filesystem read-only [14762353.792028] EXT4-fs (mmcblk0p2): I/O error while writing superblock [14762353.792038] EXT4-fs error (device mmcblk0p2): __ext4_ext_dirty:169: inode #1061363: comm kworker/u8:3: mark_inode_dirty error [14762353.792082] EXT4-fs error (device mmcblk0p2) in ext4_reserve_inode_write:5749: Journal has aborted [14762353.799004] EXT4-fs (mmcblk0p2): I/O error while writing superblock [14762353.799043] EXT4-fs error (device mmcblk0p2): ext4_check_bdev_write_error:216: comm kworker/u8:3: Error while async write back metadata [14762353.801662] EXT4-fs (mmcblk0p2): I/O error while writing superblock [14762353.801668] EXT4-fs (mmcblk0p2): previous I/O error to superblock detected [14762353.801687] EXT4-fs error (device mmcblk0p2) in ext4_reserve_inode_write:5749: Journal has aborted [14762353.804059] EXT4-fs (mmcblk0p2): I/O error while writing superblock [14762353.804087] EXT4-fs error (device mmcblk0p2): ext4_dirty_inode:5951: inode #1286: comm rs:main Q:Reg: mark_inode_dirty error [14762353.808689] EXT4-fs error (device mmcblk0p2): ext4_convert_unwritten_extents:4768: inode #1061363: comm kworker/u8:3: mark_inode_dirty error [14762353.812005] Buffer I/O error on device mmcblk0p2, logical block 4257835 [14762353.812034] Buffer I/O error on device mmcblk0p2, logical block 4257836 [14762353.812055] Buffer I/O error on device mmcblk0p2, logical block 4257837 [14762353.812074] Buffer I/O error on device mmcblk0p2, logical block 4257838 [14762353.812094] Buffer I/O error on device mmcblk0p2, logical block 4257839 [14762353.812113] Buffer I/O error on device mmcblk0p2, logical block 4257840
Updated by osmith over 2 years ago
powered it off via ssh, there's no use keeping it running if the SD card is damaged
Updated by laforge over 2 years ago
- Assignee changed from 4368 to roh
please let me know which size of uSD card we need to purchase. I'll try to get a high quality one.
Updated by roh over 2 years ago
- Status changed from New to In Progress
refitted a new sdcard, installed recent raspios buster (2021-05-07-raspios-buster-armhf-lite.img)
i tried giving it the correct ip via /etc/network/interfaces but that fails due to bridge-utils or sth missing... i guess the ansible pulls that in? not sure...
its accessible as 10.9.25.132 for now with 'sysmocom23' as password for user pi.
the old sdcard was a sandisk extreme pro 64gb
the new one is a samsung evo 64gb - should be ok.
osmith can you take over from here?
Updated by roh over 2 years ago
added bridge-utils, networking should be fixed: accessible on 10.9.25.50 again
Updated by osmith over 2 years ago
- Status changed from In Progress to Resolved
- % Done changed from 0 to 100