[...] [4401904.509179] hpsa 0000:03:00.0: HP SSD Smart Path: Retrying on standard path. [4401907.211824] sd 0:0:0:0: [sda] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [4401907.211843] sd 0:0:0:0: [sda] Sense Key : Medium Error [current] [4401907.211849] sd 0:0:0:0: [sda] Add. Sense: Unrecovered read error [4401907.211852] sd 0:0:0:0: [sda] CDB: [4401907.211856] Write(10): 2a 00 00 00 08 08 00 00 01 00 [4401907.211869] blk_update_request: critical medium error, dev sda, sector 2056 [4401907.244220] md: super_written gets error=-61, uptodate=0 [4401907.244238] md/raid1:md0: Disk failure on sda1, disabling device. md/raid1:md0: Operation continuing on 3 devices. [4401907.308528] RAID1 conf printout: [4401907.308531] --- wd:3 rd:4 [4401907.308535] disk 0, wo:1, o:0, dev:sda1 [4401907.308537] disk 1, wo:0, o:1, dev:sdb1 [4401907.308539] disk 2, wo:0, o:1, dev:sdc1 [4401907.308541] disk 3, wo:0, o:1, dev:sdd1 [4401907.325791] RAID1 conf printout: [4401907.325795] --- wd:3 rd:4 [4401907.325798] disk 1, wo:0, o:1, dev:sdb1 [4401907.325799] disk 2, wo:0, o:1, dev:sdc1 [4401907.325801] disk 3, wo:0, o:1, dev:sdd1 [4401932.348955] hpsa 0000:03:00.0: Direct-Access device c0b0t0l0 updated. [4401966.301902] hpsa 0000:03:00.0: Abort request on C0:B0:T0:L0 [4401966.302019] hpsa 0000:03:00.0: invalid command: LUN:0000004000000000 CDB:0000000000000000f000000000000000 [4401966.302023] hpsa 0000:03:00.0: probably means device no longer present [4401966.302087] hpsa 0000:03:00.0: invalid command: LUN:0000004000000000 CDB:0000000000000000000000f000000000 [4401966.302089] hpsa 0000:03:00.0: probably means device no longer present [4401966.302092] hpsa 0000:03:00.0: FAILED abort on device C0:B0:T0:L0 [4401966.302097] hpsa 0000:03:00.0: Abort request on C0:B0:T0:L0 [...] [4402114.450593] INFO: task md0_raid1:340 blocked for more than 120 seconds. [4402114.481110] Not tainted 3.19.0-2-amd64 #1 [4402114.502340] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [4402114.538966] md0_raid1 D ffff881034465230 0 340 2 0x00000000 [4402114.538969] ffff881034465230 0000000000000001 ffff88103924f3f0 0000000000014180 [4402114.538971] ffff88103446bfd8 0000000000014180 ffff881034465230 0000000000000246 [4402114.538973] ffff8810349e0800 ffff8810349e0a98 ffff8810349e0a90 ffff8810349e0818 [4402114.538975] Call Trace: [4402114.539014] [] ? md_super_wait+0x45/0x80 [md_mod] [4402114.539035] [] ? wait_woken+0x90/0x90 [4402114.539042] [] ? md_update_sb+0x23e/0x680 [md_mod] [4402114.539046] [] ? md_check_recovery+0x18a/0x470 [md_mod] [4402114.539050] [] ? raid1d+0x49/0xd50 [raid1] [4402114.539055] [] ? __schedule+0x7/0x810 [4402114.539057] [] ? schedule_timeout+0x21a/0x290 [4402114.539062] [] ? md_thread+0x126/0x130 [md_mod] [4402114.539064] [] ? wait_woken+0x90/0x90 [4402114.539067] [] ? md_stop+0x40/0x40 [md_mod] [4402114.539071] [] ? kthread+0xd3/0xf0 [4402114.539073] [] ? kthread_create_on_node+0x180/0x180 [4402114.539075] [] ? ret_from_fork+0x58/0x90 [4402114.539077] [] ? kthread_create_on_node+0x180/0x180 [4402114.539078] INFO: task jbd2/md0-8:369 blocked for more than 120 seconds. [4402114.570074] Not tainted 3.19.0-2-amd64 #1 [4402114.591358] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [4402114.627887] jbd2/md0-8 D ffff881034464920 0 369 2 0x00000000 [4402114.627889] ffff881034464920 ffffea0081fedf00 ffff881d7847ebe0 0000000000014180 [4402114.627890] ffff8810349d3fd8 0000000000014180 ffff881034464920 0000000000000246 [4402114.627893] ffff88108b1ee100 ffff88108b1ee160 ffff8810349950b8 ffff8810349d3db0 [4402114.627894] Call Trace: [4402114.627915] [] ? jbd2_journal_commit_transaction+0x216/0x1740 [jbd2] [4402114.627918] [] ? set_next_entity+0x56/0x70 [4402114.627930] [] ? pick_next_task_fair+0x1b0/0x850 [4402114.627935] [] ? wait_woken+0x90/0x90 [4402114.627938] [] ? try_to_del_timer_sync+0x4c/0x60 [4402114.627942] [] ? kjournald2+0xcd/0x270 [jbd2] [4402114.627944] [] ? wait_woken+0x90/0x90 [4402114.627963] [] ? commit_timeout+0x20/0x20 [jbd2] [4402114.627964] [] ? kthread+0xd3/0xf0 [4402114.627966] [] ? kthread_create_on_node+0x180/0x180 [4402114.627968] [] ? ret_from_fork+0x58/0x90 [4402114.627970] [] ? kthread_create_on_node+0x180/0x180 [4402114.627974] INFO: task jbd2/dm-0-8:611 blocked for more than 120 seconds. [4402114.659471] Not tainted 3.19.0-2-amd64 #1 [4402114.682356] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [4402114.718771] jbd2/dm-0-8 D ffff88203778c250 0 611 2 0x00000000 [4402114.718927] ffff88203778c250 ffff8810381e6000 ffff88103924f3f0 0000000000014180 [4402114.718928] ffff882036013fd8 0000000000014180 ffff88203778c250 ffffffff8101cc05 [4402114.719081] ffff88207fc74a30 ffff88107ffd6a60 0000000000000002 ffffffff8154fde0 [4402114.719084] Call Trace: [4402114.719089] [] ? read_tsc+0x5/0x10 [4402114.719090] [] ? bit_wait_timeout+0x80/0x80 [4402114.719093] [] ? io_schedule+0xaa/0x140 [4402114.719094] [] ? bit_wait_io+0x34/0x60 [4402114.719096] [] ? __wait_on_bit+0x5f/0x90 [4402114.719100] [] ? queue_unplugged+0x25/0x90 [4402114.719101] [] ? bit_wait_timeout+0x80/0x80 [4402114.719102] [] ? out_of_line_wait_on_bit+0x6e/0x80 [4402114.719105] [] ? autoremove_wake_function+0x30/0x30 [4402114.719110] [] ? jbd2_journal_commit_transaction+0xeaf/0x1740 [jbd2] [4402114.719115] [] ? kjournald2+0xcd/0x270 [jbd2] [4402114.719117] [] ? wait_woken+0x90/0x90 [4402114.719121] [] ? commit_timeout+0x20/0x20 [jbd2] [4402114.719124] [] ? kthread+0xd3/0xf0 [4402114.719126] [] ? kthread_create_on_node+0x180/0x180 [4402114.719127] [] ? ret_from_fork+0x58/0x90 [4402114.719129] [] ? kthread_create_on_node+0x180/0x180 [...] 4403145.320775] blk_update_request: critical medium error, dev sda, sector 1029788264 [4403145.357401] Aborting journal on device dm-0-8. [4403145.357491] EXT4-fs (dm-0): ext4_writepages: jbd2_start: 1022 pages, ino 143135284; err -30 [4403145.357516] EXT4-fs (dm-0): ext4_writepages: jbd2_start: 1024 pages, ino 34865179; err -30 [4403145.357518] EXT4-fs (dm-0): ext4_writepages: jbd2_start: 1024 pages, ino 34865379; err -30 [4403145.357520] EXT4-fs (dm-0): ext4_writepages: jbd2_start: 1024 pages, ino 34865383; err -30 [4403145.357727] EXT4-fs (dm-0): ext4_writepages: jbd2_start: 1024 pages, ino 143135886; err -30 [4403145.357729] EXT4-fs (dm-0): ext4_writepages: jbd2_start: 1024 pages, ino 143136052; err -30 [4403145.357731] EXT4-fs (dm-0): ext4_writepages: jbd2_start: 1024 pages, ino 143136167; err -30 [4403145.357733] EXT4-fs (dm-0): ext4_writepages: jbd2_start: 1024 pages, ino 34865265; err -30 [4403175.705868] hpsa 0000:03:00.0: Abort request on C0:B0:T0:L0 [4403176.711420] hpsa 0000:03:00.0: device is ready. [4403176.711429] hpsa 0000:03:00.0: Reset as abort: Device recovered from reset: scsi3addr 0x000000c000000001 [4403176.711433] hpsa 0000:03:00.0: ABORT REQUEST on C0:B0:T0:L0 Tag:0x00000000:00000010 Command:0x2a SN:0x0 REQUEST SUCCEEDED. [4403176.711456] hpsa 0000:03:00.0: Abort request on C0:B0:T0:L0 [4403176.749374] EXT4-fs error (device dm-0): ext4_journal_check_start:56: Detected aborted journal [4403176.790437] EXT4-fs (dm-0): Remounting filesystem read-only [4403192.102023] hpsa 0000:03:00.0: Direct-Access device c0b0t0l0 updated. [4403199.302030] sd 0:0:0:0: [sda] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [4403199.302050] sd 0:0:0:0: [sda] Sense Key : Medium Error [current] [4403199.302053] sd 0:0:0:0: [sda] Add. Sense: Unrecovered read error [4403199.302054] sd 0:0:0:0: [sda] CDB: [4403199.302056] Write(10): 2a 00 03 9f e4 00 00 00 08 00 [4403199.302061] blk_update_request: critical medium error, dev sda, sector 60810240 [4403199.335848] Buffer I/O error on dev dm-0, logical block 0, lost sync page write [4403199.369561] EXT4-fs error (device dm-0): ext4_journal_check_start:56: Detected aborted journal [4403208.354499] hpsa 0000:03:00.0: HP SSD Smart Path: task complete with check condition. [4403208.354503] hpsa 0000:03:00.0: HP SSD Smart Path: Retrying on standard path. [4403222.114900] hpsa 0000:03:00.0: Direct-Access device c0b0t0l0 updated. [4403229.573738] hpsa 0000:03:00.0: Abort request on C0:B0:T0:L0 [4403230.579311] hpsa 0000:03:00.0: device is ready. [4403230.579321] hpsa 0000:03:00.0: Reset as abort: Device recovered from reset: scsi3addr 0x000000c000000001 [4403230.579325] hpsa 0000:03:00.0: ABORT REQUEST on C0:B0:T0:L0 Tag:0x00000000:00000010 Command:0x2a SN:0x0 REQUEST SUCCEEDED. [4403252.127419] hpsa 0000:03:00.0: Direct-Access device c0b0t0l0 updated. [4403253.283431] EXT4-fs error (device dm-0): ext4_journal_check_start:56: [4403253.283436] EXT4-fs error (device dm-0): ext4_journal_check_start:56: Detected aborted journal [4403253.354938] Detected aborted journal [4403253.354957] EXT4-fs (dm-0): ext4_writepages: jbd2_start: 1024 pages, ino 34865378; err -30 [4406814.597501] hpsa 0000:03:00.0: HP SSD Smart Path: task complete with check condition. [4406814.597509] hpsa 0000:03:00.0: HP SSD Smart Path: Retrying on standard path. [4406820.818190] hpsa 0000:03:00.0: Direct-Access device c0b0t0l0 updated. [4410445.560631] hpsa 0000:03:00.0: HP SSD Smart Path: task complete with check condition. [4410445.560639] hpsa 0000:03:00.0: HP SSD Smart Path: Retrying on standard path. [4410449.487597] hpsa 0000:03:00.0: Direct-Access device c0b0t0l0 updated. [4414069.666451] hpsa 0000:03:00.0: HP SSD Smart Path: task complete with check condition. [4414069.666459] hpsa 0000:03:00.0: HP SSD Smart Path: Retrying on standard path. [4414078.158926] hpsa 0000:03:00.0: Direct-Access device c0b0t0l0 updated. [4417690.664284] hpsa 0000:03:00.0: HP SSD Smart Path: task complete with check condition. [4417690.664292] hpsa 0000:03:00.0: HP SSD Smart Path: Retrying on standard path. [4417706.829618] hpsa 0000:03:00.0: Direct-Access device c0b0t0l0 updated.