Nodes (not) crashed: node stopped + mount UUID not available; suspension score impacted

Hi there,

today I had 2 strange behaviours:

  • I have 3 HDDs, one of them for logging; connected via USB. RPi 4B.
  • my 2 nodes are stopping without errors (!?)
  • then I check, if the HDDs are available, but it says, the UUID cannot be found, all 3
  • rebooting RPi recovers everything, but suspension score is impacted → therefore I need to figure out, what’s happening

It does not seem to be the connection of the HDDs. Would a node do a full stop?

It does not seem to be a docker issue, because the mounts would stay available.

The RPi does not seem to reboot on its own, because the system would work and recover itself.

Any ideas, where to search for the issue?

Raspberry pi is super flaky on usb hard drive.
I had constant crashes using raspberry and eventually switched to a regular desktop for it

There should be errors, if the disk for logs were available on time of the crash, otherwise it cannot save it.

It’s likely either USB controller or not enough power. The node will crash, if the disk with data become unavailable for reads or writes.

1 Like

Hmm, it must have been some power „hiccups“… It did not occur since yesterday. Thank you for your support!

1 Like

happened again last night.

first step would be to move the logs to e.g. the internal SSD, in order to see, what has been logged, right?

any other hint, what i can do to figure out, where the problem is?

run a

dmesg

this usually tells you if the harddrive got disconnected

2 Likes

Yes, or switch back to the docker logs (but you will lose these logs when the container is removed).

indeed:

[13257.620217] usb 2-1.2: cmd cmplt err -71
...
[13263.373226] usb 2-1.2: cmd cmplt err -71
[13264.570232] sd 1:0:0:0: [sdb] tag#22 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD IN 
[13264.570262] sd 1:0:0:0: [sdb] tag#22 CDB: opcode=0xa1 a1 08 2e d0 01 00 4f c2 00 b0 00 00
[13288.638459] sd 1:0:0:0: [sdb] tag#23 uas_eh_abort_handler 0 uas-tag 6 inflight: CMD OUT 
[13288.638485] sd 1:0:0:0: [sdb] tag#23 CDB: opcode=0x8a 8a 00 00 00 00 03 9a fd ba 28 00 00 00 28 00 00
[13288.638904] sd 1:0:0:0: [sdb] tag#21 uas_eh_abort_handler 0 uas-tag 8 inflight: CMD OUT 
...
[13293.426498] sd 1:0:0:0: [sdb] tag#5 uas_eh_abort_handler 0 uas-tag 22 inflight: CMD IN 
[13293.426517] sd 1:0:0:0: [sdb] tag#5 CDB: opcode=0x88 88 00 00 00 00 03 3f 40 a8 b8 00 00 00 08 00 00
[13293.446522] scsi host1: uas_eh_device_reset_handler start
[13293.452246] xhci_hcd 0000:01:00.0: WARNING: Host System Error
[13298.578536] xhci_hcd 0000:01:00.0: xHCI host not responding to stop endpoint command
[13298.610557] xhci_hcd 0000:01:00.0: Host halt failed, -110
[13298.610564] xhci_hcd 0000:01:00.0: xHCI host controller not responding, assume dead
[13298.610662] xhci_hcd 0000:01:00.0: HC died; cleaning up
[13298.610844] scsi host1: uas_eh_device_reset_handler FAILED err -22
[13298.610867] sd 1:0:0:0: Device offlined - not ready after error recovery
...
[13298.610977] sd 1:0:0:0: Device offlined - not ready after error recovery
[13298.611009] sd 1:0:0:0: [sdb] tag#5 UNKNOWN(0x2003) Result: hostbyte=0x03 driverbyte=DRIVER_OK cmd_age=35s
[13298.611018] sd 1:0:0:0: [sdb] tag#5 CDB: opcode=0x88 88 00 00 00 00 03 3f 40 a8 b8 00 00 00 08 00 00
[13298.611025] I/O error, dev sdb, sector 13946103992 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 2
[13298.611065] sd 1:0:0:0: [sdb] tag#4 UNKNOWN(0x2003) Result: hostbyte=0x03 driverbyte=DRIVER_OK cmd_age=35s
[13298.611073] sd 1:0:0:0: [sdb] tag#4 CDB: opcode=0x88 88 00 00 00 00 03 ed c0 b9 80 00 00 00 08 00 00
[13298.611079] I/O error, dev sdb, sector 16873732480 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
[13298.611106] sd 1:0:0:0: [sdb] tag#3 UNKNOWN(0x2003) Result: hostbyte=0x03 driverbyte=DRIVER_OK cmd_age=37s
[13298.611113] sd 1:0:0:0: [sdb] tag#3 CDB: opcode=0x8a 8a 00 00 00 00 03 9b 03 20 f0 00 00 00 08 00 00
[13298.611119] I/O error, dev sdb, sector 15485575408 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2
[13298.611129] EXT4-fs warning (device sdb1): ext4_end_bio:347: I/O error 10 writing to inode 219358404 starting block 1935696926)
[13298.611156] sd 1:0:0:0: [sdb] tag#2 UNKNOWN(0x2003) Result: hostbyte=0x03 driverbyte=DRIVER_OK cmd_age=37s
[13298.611163] sd 1:0:0:0: [sdb] tag#2 CDB: opcode=0x8a 8a 00 00 00 00 03 9b 03 fa d8 00 00 00 30 00 00
[13298.611168] I/O error, dev sdb, sector 15485631192 op 0x1:(WRITE) flags 0x800 phys_seg 6 prio class 2
[13298.611177] EXT4-fs warning (device sdb1): ext4_end_bio:347: I/O error 10 writing to inode 219358403 starting block 1935703899)
[13298.611202] sd 1:0:0:0: [sdb] tag#1 UNKNOWN(0x2003) Result: hostbyte=0x03 driverbyte=DRIVER_OK cmd_age=38s
[13298.611208] sd 1:0:0:0: [sdb] tag#1 CDB: opcode=0x88 88 00 00 00 00 03 aa a6 a5 20 00 00 00 08 00 00
[13298.611214] I/O error, dev sdb, sector 15747949856 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
[13298.611214] Buffer I/O error on device sdb1, logical block 1935696670
[13298.611221] EXT4-fs warning (device sdb1): dx_probe:818: inode #263651505: lblock 0: comm storagenode: error -5 reading directory block
[13298.611244] EXT4-fs warning (device sdb1): dx_probe:818: inode #245366858: lblock 0: comm storagenode: error -5 reading directory block
[13298.611252] sd 1:0:0:0: [sdb] tag#9 UNKNOWN(0x2003) Result: hostbyte=0x03 driverbyte=DRIVER_OK cmd_age=38s
[13298.611259] sd 1:0:0:0: [sdb] tag#9 CDB: opcode=0x8a 8a 00 00 00 00 03 9b 27 e7 18 00 00 00 50 00 00
[13298.611262] Buffer I/O error on device sdb1, logical block 1935703643
[13298.611265] I/O error, dev sdb, sector 15487985432 op 0x1:(WRITE) flags 0x800 phys_seg 10 prio class 2
[13298.611274] EXT4-fs warning (device sdb1): ext4_end_bio:347: I/O error 10 writing to inode 219358402 starting block 1935998179)
[13298.611276] Buffer I/O error on device sdb1, logical block 1935703644
[13298.611284] Buffer I/O error on device sdb1, logical block 1935703645
[13298.611292] Buffer I/O error on device sdb1, logical block 1935703646
[13298.611301] Buffer I/O error on device sdb1, logical block 1935703647
[13298.611302] sd 1:0:0:0: [sdb] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x03 driverbyte=DRIVER_OK cmd_age=38s
[13298.611309] sd 1:0:0:0: [sdb] tag#0 CDB: opcode=0x88 88 00 00 00 00 03 a7 c1 93 20 00 00 00 08 00 00
[13298.611309] Buffer I/O error on device sdb1, logical block 1935703648
[13298.611315] I/O error, dev sdb, sector 15699383072 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
[13298.611336] Buffer I/O error on device sdb1, logical block 1935997923
[13298.611339] sd 1:0:0:0: [sdb] tag#28 UNKNOWN(0x2003) Result: hostbyte=0x03 driverbyte=DRIVER_OK cmd_age=38s
[13298.611346] sd 1:0:0:0: [sdb] tag#28 CDB: opcode=0x8a 8a 00 00 00 00 03 9b 03 fa b8 00 00 00 20 00 00
[13298.611348] Buffer I/O error on device sdb1, logical block 1935997924
[13298.611353] I/O error, dev sdb, sector 15485631160 op 0x1:(WRITE) flags 0x800 phys_seg 4 prio class 2
[13298.611356] Buffer I/O error on device sdb1, logical block 1935997925
[13298.611361] EXT4-fs warning (device sdb1): ext4_end_bio:347: I/O error 10 writing to inode 219358401 starting block 1935703895)
[13298.611382] sd 1:0:0:0: [sdb] tag#27 UNKNOWN(0x2003) Result: hostbyte=0x03 driverbyte=DRIVER_OK cmd_age=38s
[13298.611389] sd 1:0:0:0: [sdb] tag#27 CDB: opcode=0x88 88 00 00 00 00 03 ac 3c 92 b8 00 00 00 08 00 00
[13298.611394] I/O error, dev sdb, sector 15774552760 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
[13298.611417] sd 1:0:0:0: [sdb] tag#26 UNKNOWN(0x2003) Result: hostbyte=0x03 driverbyte=DRIVER_OK cmd_age=38s
[13298.611424] sd 1:0:0:0: [sdb] tag#26 CDB: opcode=0x88 88 00 00 00 00 03 ee 00 91 08 00 00 00 08 00 00
[13298.611430] I/O error, dev sdb, sector 16877916424 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
[13298.611528] EXT4-fs warning (device sdb1): dx_probe:818: inode #245301365: lblock 0: comm storagenode: error -5 reading directory block
[13298.611574] EXT4-fs warning (device sdb1): ext4_end_bio:347: I/O error 10 writing to inode 108397143 starting block 1935640447)
[13298.611611] EXT4-fs warning (device sdb1): ext4_end_bio:347: I/O error 10 writing to inode 219358400 starting block 49338240)
[13298.611631] EXT4-fs warning (device sdb1): ext4_end_bio:347: I/O error 10 writing to inode 108397143 starting block 1935652677)
[13298.611674] sd 1:0:0:0: rejecting I/O to offline device
[13298.611926] EXT4-fs (sdb1): Delayed block allocation failed for inode 108397142 at logical offset 26 with max blocks 2 with error 5
[13298.611947] EXT4-fs (sdb1): This should not happen!! Data will be lost

[13298.612239] JBD2: Detected IO errors while flushing file data on sdb1-8
[13298.612303] EXT4-fs error (device sdb1): __ext4_get_inode_loc_noinmem:4613: inode #266021484: block 2128085702: comm storagenode: unable to read itable block
[13298.612324] Aborting journal on device sdb1-8.
[13298.612337] EXT4-fs error (device sdb1) in ext4_do_update_inode:5211: Journal has aborted
[13298.612351] EXT4-fs error (device sdb1): ext4_convert_unwritten_extents:4820: inode #108397143: comm kworker/u8:4: mark_inode_dirty error
[13298.612364] Buffer I/O error on dev sdb1, logical block 1220575232, lost sync page write
[13298.612370] EXT4-fs error (device sdb1) in ext4_convert_unwritten_io_end_vec:4859: IO failure
[13298.612381] Buffer I/O error on dev sdb1, logical block 0, lost sync page write
[13298.612381] EXT4-fs (sdb1): failed to convert unwritten extents to written extents -- potential data loss!  (inode 108397143, error -30)
[13298.612399] JBD2: I/O error when updating journal superblock for sdb1-8.
[13298.612400] EXT4-fs (sdb1): I/O error while writing superblock
[13298.612438] Buffer I/O error on dev sdb1, logical block 0, lost sync page write
[13298.612454] EXT4-fs (sdb1): I/O error while writing superblock
[13298.612609] Buffer I/O error on dev sdb1, logical block 1936719911, lost async page write
...
[13298.612672] Buffer I/O error on dev sdb1, logical block 2014840277, lost async page write
[13298.613483] EXT4-fs error (device sdb1) in __ext4_new_inode:1084: Journal has aborted
[13298.613590] EXT4-fs error (device sdb1) in __ext4_new_inode:1084: Journal has aborted
[13298.613698] EXT4-fs error (device sdb1): ext4_journal_check_start:83: comm storagenode: Detected aborted journal
[13298.613735] EXT4-fs error (device sdb1): ext4_journal_check_start:83: comm storagenode: Detected aborted journal
[13298.613814] EXT4-fs (sdb1): previous I/O error to superblock detected
[13298.613833] EXT4-fs error (device sdb1) in ext4_setattr:5582: Journal has aborted
[13298.613859] EXT4-fs (sdb1): I/O error while writing superblock
[13298.613860] EXT4-fs (sdb1): Remounting filesystem read-only
[13298.613944] usb 1-1: USB disconnect, device number 2
[13298.613957] usb 1-1.1: USB disconnect, device number 3
[13298.613967] usb 1-1.1.5: USB disconnect, device number 4
[13298.617281] usb 2-1: USB disconnect, device number 2
[13298.617312] usb 2-1.1: USB disconnect, device number 3
[13298.621498] EXT4-fs (sdb1): I/O error while writing superblock
[13298.634828] EXT4-fs warning (device sda1): ext4_end_bio:347: I/O error 17 writing to inode 51642371 starting block 31588771)
[13298.634883] EXT4-fs warning (device sda1): ext4_end_bio:347: I/O error 17 writing to inode 51642371 starting block 31588772)
[13298.635001] EXT4-fs warning (device sda1): ext4_end_bio:347: I/O error 17 writing to inode 51642370 starting block 31587219)
[13298.635020] Aborting journal on device sda1-8.
[13298.635033] EXT4-fs warning (device sda1): ext4_convert_unwritten_extents:4819: inode #51642371: block 65700: len 2: ext4_ext_map_blocks returned -30
[13298.635053] EXT4-fs error (device sda1) in ext4_reserve_inode_write:5839: Journal has aborted
[13298.635053] EXT4-fs (sda1): Delayed block allocation failed for inode 51642370 at logical offset 78484 with max blocks 1 with error 30
[13298.635062] EXT4-fs (sda1): This should not happen!! Data will be lost

[13298.635073] EXT4-fs error (device sda1) in ext4_writepages:2854: Journal has aborted
[13298.635078] EXT4-fs error (device sda1): ext4_convert_unwritten_extents:4820: inode #51642371: comm kworker/u8:4: mark_inode_dirty error
[13298.635083] JBD2: I/O error when updating journal superblock for sda1-8.
[13298.635092] EXT4-fs error (device sda1) in ext4_convert_unwritten_io_end_vec:4859: Journal has aborted
[13298.635102] EXT4-fs (sda1): failed to convert unwritten extents to written extents -- potential data loss!  (inode 51642371, error -30)
[13298.635117] EXT4-fs error (device sda1) in ext4_reserve_inode_write:5839: Journal has aborted
[13298.635131] EXT4-fs error (device sda1): ext4_dirty_inode:6043: inode #51642370: comm storagenode: mark_inode_dirty error
[13298.635145] EXT4-fs error (device sda1) in ext4_dirty_inode:6044: Journal has aborted
[13298.635206] EXT4-fs (sda1): I/O error while writing superblock
[13298.635212] EXT4-fs error (device sda1): ext4_journal_check_start:83: comm storagenode: Detected aborted journal
[13298.635264] EXT4-fs (sda1): previous I/O error to superblock detected
[13298.635303] EXT4-fs (sda1): I/O error while writing superblock
[13298.635310] EXT4-fs (sda1): Remounting filesystem read-only
[13298.635310] EXT4-fs (sda1): I/O error while writing superblock
[13298.635342] EXT4-fs error (device sda1) in ext4_reserve_inode_write:5839: Journal has aborted
[13298.635400] EXT4-fs (sda1): I/O error while writing superblock
[13298.635788] EXT4-fs error (device sda1): ext4_dirty_inode:6043: inode #51642371: comm storagenode: mark_inode_dirty error
[13298.635921] EXT4-fs (sda1): I/O error while writing superblock
[13298.679055] sd 0:0:0:0: [sda] Synchronizing SCSI cache
[13298.707546] systemd[1]: Stopped target Local File Systems.
[13298.759226] systemd[18992]: mnt-WD1003.mount: Failed to connect stdout to the journal socket, ignoring: No such file or directory
[13298.759697] systemd[1]: Unmounting /mnt/WD1003...
[13298.799019] systemd[1]: mnt-WD1003.mount: Succeeded.
[13298.800601] systemd[1]: Unmounted /mnt/WD1003.
[13298.801722] systemd[1]: systemd-fsck@dev-disk-by\x2duuid-60f9452a\x2dc2da\x2d48fd\x2da156\x2d53d15e67102c.service: Succeeded.
[13298.802479] systemd[1]: Stopped File System Check on /dev/disk/by-uuid/60f9452a-c2da-48fd-a156-53d15e67102c.
[13298.803169] systemd[1]: systemd-fsck@dev-disk-by\x2duuid-60f9452a\x2dc2da\x2d48fd\x2da156\x2d53d15e67102c.service: Consumed 10.557s CPU time.
[13298.942591] sd 0:0:0:0: [sda] Synchronize Cache(10) failed: Result: hostbyte=0x07 driverbyte=DRIVER_OK
[13298.943169] xhci_hcd 0000:01:00.0: WARN Can't disable streams for endpoint 0x85, streams are being disabled already
[13298.946412] usb 2-1.2: USB disconnect, device number 4
[13300.110654] EXT4-fs error (device sdb1): __ext4_get_inode_loc_noinmem:4613: inode #13: block 2081: comm storagenode: unable to read itable block
[13300.110772] EXT4-fs (sdb1): I/O error while writing superblock
[13300.362155] systemd[1]: docker-7480f25e1fbd7256604a55f167aca4e00afcd58a108f9ab1832e112c8c3a7110.scope: Succeeded.
[13300.363658] systemd[1]: docker-7480f25e1fbd7256604a55f167aca4e00afcd58a108f9ab1832e112c8c3a7110.scope: Consumed 40min 17.103s CPU time.
[13301.357082] docker0: port 3(veth418e3dd) entered disabled state
[13301.357734] vethca79324: renamed from eth0
[13301.411564] IPv6: ADDRCONF(NETDEV_CHANGE): vethca79324: link becomes ready
[13301.411770] IPv6: ADDRCONF(NETDEV_CHANGE): veth418e3dd: link becomes ready
[13301.411834] docker0: port 3(veth418e3dd) entered blocking state
[13301.411852] docker0: port 3(veth418e3dd) entered forwarding state
[13301.436324] docker0: port 3(veth418e3dd) entered disabled state
[13301.439139] device veth418e3dd left promiscuous mode
[13301.439159] docker0: port 3(veth418e3dd) entered disabled state
[13301.532753] systemd[1]: run-docker-netns-c162f7aa0e14.mount: Succeeded.
[13301.582291] systemd[1]: var-lib-docker-overlay2-16c35c4c8de2148662106d80d818b4be53bbe89bf96e8e5c47ba269a829f3ad2-merged.mount: Succeeded.
[13301.786799] docker0: port 3(vethad171e3) entered blocking state
[13301.786823] docker0: port 3(vethad171e3) entered disabled state
[13301.787059] device vethad171e3 entered promiscuous mode
[13301.787336] docker0: port 3(vethad171e3) entered blocking state
[13301.787348] docker0: port 3(vethad171e3) entered forwarding state
[13301.791171] IPv6: ADDRCONF(NETDEV_CHANGE): vethad171e3: link becomes ready
[13302.499194] systemd[1]: Started libcontainer container 7480f25e1fbd7256604a55f167aca4e00afcd58a108f9ab1832e112c8c3a7110.
[13302.536239] systemd[1]: docker-7480f25e1fbd7256604a55f167aca4e00afcd58a108f9ab1832e112c8c3a7110.scope: Succeeded.
[13303.432121] docker0: port 3(vethad171e3) entered disabled state
[13303.435833] device vethad171e3 left promiscuous mode
[13303.435854] docker0: port 3(vethad171e3) entered disabled state
[13303.632921] systemd[1]: var-lib-docker-overlay2-16c35c4c8de2148662106d80d818b4be53bbe89bf96e8e5c47ba269a829f3ad2-merged.mount: Succeeded.
[13303.870671] sd 1:0:0:0: [sdb] Synchronizing SCSI cache
[13303.903072] systemd[19262]: mnt-WD1001.mount: Failed to connect stdout to the journal socket, ignoring: No such file or directory
[13303.903566] systemd[1]: Unmounting /mnt/WD1001...
[13303.915348] EXT4-fs (sdb1): unmounting filesystem.
[13303.962400] systemd[1]: mnt-WD1001.mount: Succeeded.
[13303.963995] systemd[1]: Unmounted /mnt/WD1001.
[13303.965008] systemd[1]: systemd-fsck@dev-disk-by\x2duuid-2c691f8a\x2d637d\x2d49a5\x2d8198\x2d82b34661e658.service: Succeeded.
[13303.965611] systemd[1]: Stopped File System Check on /dev/disk/by-uuid/2c691f8a-637d-49a5-8198-82b34661e658.
[13303.966168] systemd[1]: systemd-fsck@dev-disk-by\x2duuid-2c691f8a\x2d637d\x2d49a5\x2d8198\x2d82b34661e658.service: Consumed 29.029s CPU time.
[13304.114615] sd 1:0:0:0: [sdb] Synchronize Cache(10) failed: Result: hostbyte=0x07 driverbyte=DRIVER_OK
[13304.130898] xhci_hcd 0000:01:00.0: WARN Can't disable streams for endpoint 0x85, streams are being disabled already
[13304.132862] usb 2-1.3: USB disconnect, device number 5
[13304.133180] sd 2:0:0:0: [sdc] tag#9 uas_zap_pending 0 uas-tag 1 inflight: CMD 
[13304.133190] sd 2:0:0:0: [sdc] tag#9 CDB: opcode=0x88 88 00 00 00 00 01 32 00 17 08 00 00 00 b8 00 00
[13304.133199] sd 2:0:0:0: [sdc] tag#10 uas_zap_pending 0 uas-tag 2 inflight: CMD 
[13304.133204] sd 2:0:0:0: [sdc] tag#10 CDB: opcode=0x88 88 00 00 00 00 01 32 00 17 c0 00 00 00 08 00 00
[13304.133210] sd 2:0:0:0: [sdc] tag#8 uas_zap_pending 0 uas-tag 3 inflight: CMD 
[13304.133215] sd 2:0:0:0: [sdc] tag#8 CDB: opcode=0x88 88 00 00 00 00 01 32 00 17 c8 00 00 00 40 00 00
[13304.133220] scsi_io_completion_action: 11 callbacks suppressed
[13304.133230] sd 2:0:0:0: [sdc] tag#9 UNKNOWN(0x2003) Result: hostbyte=0x01 driverbyte=DRIVER_OK cmd_age=10s
[13304.133240] sd 2:0:0:0: [sdc] tag#9 CDB: opcode=0x88 88 00 00 00 00 01 32 00 17 08 00 00 00 b8 00 00
[13304.133245] blk_print_req_error: 132 callbacks suppressed
[13304.133250] I/O error, dev sdc, sector 5133833992 op 0x0:(READ) flags 0x80700 phys_seg 23 prio class 2
[13304.133320] device offline error, dev sdc, sector 9753636608 op 0x1:(WRITE) flags 0x800 phys_seg 23 prio class 2
[13304.133332] EXT4-fs warning (device sdc1): ext4_end_bio:347: I/O error 17 writing to inode 38483079 starting block 1219204576)
[13304.133364] device offline error, dev sdc, sector 1916839752 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
[13304.133381] device offline error, dev sdc, sector 1916840800 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
[13304.133396] device offline error, dev sdc, sector 1916843216 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
[13304.133407] EXT4-fs warning (device sdc1): dx_probe:818: inode #29950063: lblock 0: comm storagenode: error -5 reading directory block
[13304.133411] device offline error, dev sdc, sector 1937805008 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 2
[13304.133411] EXT4-fs warning (device sdc1): dx_probe:818: inode #29950036: lblock 0: comm storagenode: error -5 reading directory block
[13304.133429] device offline error, dev sdc, sector 2449507328 op 0x0:(READ) flags 0x80700 phys_seg 14 prio class 2
[13304.133459] device offline error, dev sdc, sector 2449507440 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 2
[13304.133474] device offline error, dev sdc, sector 2449507448 op 0x0:(READ) flags 0x80700 phys_seg 11 prio class 2
[13304.133509] device offline error, dev sdc, sector 2449507544 op 0x0:(READ) flags 0x80700 phys_seg 6 prio class 2
[13304.133552] EXT4-fs warning (device sdc1): ext4_end_bio:347: I/O error 17 writing to inode 38483080 starting block 1201178540)
[13304.133552] EXT4-fs error (device sdc1): __ext4_get_inode_loc_noinmem:4613: inode #38336226: block 306188174: comm storagenode: unable to read itable block
[13304.133568] EXT4-fs warning (device sdc1): ext4_end_bio:347: I/O error 17 writing to inode 38483081 starting block 1203829975)
[13304.133581] EXT4-fs warning (device sdc1): ext4_end_bio:347: I/O error 17 writing to inode 187695595 starting block 1217824700)
[13304.133591] buffer_io_error: 121 callbacks suppressed
[13304.133594] Buffer I/O error on device sdc1, logical block 1217824444
[13304.133601] EXT4-fs warning (device sdc1): ext4_dx_find_entry:1775: inode #29556846: lblock 90: comm storagenode: error -5 reading directory block
[13304.133674] Buffer I/O error on device sdc1, logical block 1219204320
...
[13304.133719] Buffer I/O error on device sdc1, logical block 1219204324
[13304.133720] sd 2:0:0:0: [sdc] tag#10 UNKNOWN(0x2003) Result: hostbyte=0x01 driverbyte=DRIVER_OK cmd_age=10s
[13304.133726] Buffer I/O error on device sdc1, logical block 1219204325
[13304.133730] sd 2:0:0:0: [sdc] tag#10 CDB: opcode=0x88 88 00 00 00 00 01 32 00 17 c0 00 00 00 08 00 00
[13304.133734] Buffer I/O error on device sdc1, logical block 1219204326
[13304.133741] Buffer I/O error on device sdc1, logical block 1219204327
[13304.133749] Buffer I/O error on device sdc1, logical block 1219204328
[13304.133753] sd 2:0:0:0: [sdc] tag#8 UNKNOWN(0x2003) Result: hostbyte=0x01 driverbyte=DRIVER_OK cmd_age=10s
[13304.133760] sd 2:0:0:0: [sdc] tag#8 CDB: opcode=0x88 88 00 00 00 00 01 32 00 17 c8 00 00 00 40 00 00
[13304.133777] EXT4-fs warning (device sdc1): dx_probe:818: inode #29950065: lblock 0: comm storagenode: error -5 reading directory block
[13304.133787] EXT4-fs warning (device sdc1): dx_probe:818: inode #30277819: lblock 0: comm storagenode: error -5 reading directory block
[13304.133965] JBD2: Detected IO errors while flushing file data on sdc1-8
[13304.134038] Aborting journal on device sdc1-8.
[13304.134093] EXT4-fs warning (device sdc1): ext4_end_bio:347: I/O error 17 writing to inode 187695595 starting block 1217824948)
[13304.134104] buffer_io_error: 38 callbacks suppressed
[13304.134110] Buffer I/O error on dev sdc1, logical block 1220575232, lost sync page write
[13304.134130] JBD2: I/O error when updating journal superblock for sdc1-8.
[13304.134204] EXT4-fs error (device sdc1): ext4_journal_check_start:83: comm kworker/u8:0: Detected aborted journal
[13304.134264] Buffer I/O error on dev sdc1, logical block 0, lost sync page write
[13304.134290] EXT4-fs (sdc1): previous I/O error to superblock detected
[13304.134318] Buffer I/O error on dev sdc1, logical block 0, lost sync page write
[13304.134337] EXT4-fs (sdc1): I/O error while writing superblock
[13304.134367] EXT4-fs error (device sdc1): ext4_journal_check_start:83: comm kworker/u8:1: Detected aborted journal
[13304.134401] Buffer I/O error on dev sdc1, logical block 0, lost sync page write
[13304.134416] EXT4-fs (sdc1): I/O error while writing superblock
[13304.134422] EXT4-fs (sdc1): Remounting filesystem read-only
[13304.134431] EXT4-fs (sdc1): ext4_writepages: jbd2_start: 9223372036854775806 pages, ino 187695597; err -30
[13304.134437] EXT4-fs error (device sdc1) in ext4_reserve_inode_write:5839: Journal has aborted
[13304.134501] Buffer I/O error on dev sdc1, logical block 0, lost sync page write
[13304.134521] EXT4-fs (sdc1): I/O error while writing superblock
[13304.134545] EXT4-fs error (device sdc1) in __ext4_new_inode:1092: Journal has aborted
[13304.134634] Buffer I/O error on dev sdc1, logical block 0, lost sync page write
[13304.134651] EXT4-fs (sdc1): I/O error while writing superblock
[13304.135079] EXT4-fs (sdc1): failed to convert unwritten extents to written extents -- potential data loss!  (inode 187695595, error -30)
[13304.135147] Buffer I/O error on dev sdc1, logical block 235405364, lost async page write
...
[13304.135213] Buffer I/O error on dev sdc1, logical block 242745386, lost async page write
[13304.136013] EXT4-fs error (device sdc1): __ext4_get_inode_loc_noinmem:4613: inode #80223629: block 641729016: comm storagenode: unable to read itable block
[13304.136120] EXT4-fs (sdc1): I/O error while writing superblock
[13304.136192] EXT4-fs error (device sdc1): ext4_dirty_inode:6043: inode #38483082: comm storagenode: mark_inode_dirty error
[13304.136260] EXT4-fs (sdc1): I/O error while writing superblock
[13304.141082] EXT4-fs error (device sdc1): __ext4_get_inode_loc_noinmem:4613: inode #29884612: block 239075372: comm storagenode: unable to read itable block
[13304.141412] EXT4-fs error (device sdc1): __ext4_get_inode_loc_noinmem:4613: inode #30540017: block 244318255: comm storagenode: unable to read itable block
[13305.394715] EXT4-fs error (device sdc1): __ext4_get_inode_loc_noinmem:4613: inode #16: block 2081: comm storagenode: unable to read itable block
[13306.543141] EXT4-fs error (device sda1): ext4_wait_block_bitmap:534: comm supervisord: Cannot read block bitmap - block_group = 964, block_bitmap = 31457284
[13306.543220] EXT4-fs error (device sda1): ext4_discard_preallocations:5025: comm supervisord: Error -5 loading buddy information for 964
[13306.543636] EXT4-fs error (device sda1): ext4_wait_block_bitmap:534: comm supervisord: Cannot read block bitmap - block_group = 963, block_bitmap = 31457283
[13306.543674] EXT4-fs error (device sda1): ext4_discard_preallocations:5025: comm supervisord: Error -5 loading buddy information for 963
[13306.543855] EXT4-fs (sda1): unmounting filesystem.
[13306.569569] systemd[1]: docker-abac240296c90e8c64e6080055f3cfae1c3299bb7e86d65cc47d94184fee1a7f.scope: Succeeded.
[13306.571232] systemd[1]: docker-abac240296c90e8c64e6080055f3cfae1c3299bb7e86d65cc47d94184fee1a7f.scope: Consumed 41min 4.565s CPU time.
[13306.959999] docker0: port 1(vethe7a652a) entered disabled state
[13306.960530] veth2a5c703: renamed from eth0
[13307.026769] IPv6: ADDRCONF(NETDEV_CHANGE): veth2a5c703: link becomes ready
[13307.026977] IPv6: ADDRCONF(NETDEV_CHANGE): vethe7a652a: link becomes ready
[13307.027076] docker0: port 1(vethe7a652a) entered blocking state
[13307.027097] docker0: port 1(vethe7a652a) entered forwarding state
[13307.043760] docker0: port 1(vethe7a652a) entered disabled state
[13307.046896] device vethe7a652a left promiscuous mode
[13307.046942] docker0: port 1(vethe7a652a) entered disabled state
[13307.099436] systemd[1]: run-docker-netns-2566ec3ddec5.mount: Succeeded.
[13307.123792] systemd[1]: var-lib-docker-overlay2-34668aca5102229aa6b2dfdb7fad869c992295d4f0f41c0e9583869bfcc1c743-merged.mount: Succeeded.
[13308.398727] sd 2:0:0:0: [sdc] Synchronizing SCSI cache
[13308.408791] docker0: port 1(veth9df8735) entered blocking state
[13308.408809] docker0: port 1(veth9df8735) entered disabled state
[13308.409028] device veth9df8735 entered promiscuous mode
[13308.409325] docker0: port 1(veth9df8735) entered blocking state
[13308.409340] docker0: port 1(veth9df8735) entered forwarding state
[13308.412203] systemd[1]: Unmounting /mnt/WD1002...
[13308.418888] IPv6: ADDRCONF(NETDEV_CHANGE): veth9df8735: link becomes ready
[13308.427834] systemd[19362]: mnt-WD1002.mount: Failed to connect stdout to the journal socket, ignoring: No such file or directory
[13308.509798] systemd[1]: mnt-WD1002.mount: Succeeded.
[13308.511623] systemd[1]: Unmounted /mnt/WD1002.
[13308.512592] systemd[1]: systemd-fsck@dev-disk-by\x2duuid-e7d6dabc\x2d57da\x2d481d\x2daee0\x2d29cf106966ec.service: Succeeded.
[13308.513107] systemd[1]: Stopped File System Check on /dev/disk/by-uuid/e7d6dabc-57da-481d-aee0-29cf106966ec.
[13308.513598] systemd[1]: systemd-fsck@dev-disk-by\x2duuid-e7d6dabc\x2d57da\x2d481d\x2daee0\x2d29cf106966ec.service: Consumed 31.851s CPU time.
[13308.638713] sd 2:0:0:0: [sdc] Synchronize Cache(10) failed: Result: hostbyte=0x07 driverbyte=DRIVER_OK
[13308.671254] xhci_hcd 0000:01:00.0: WARN Can't disable streams for endpoint 0x85, streams are being disabled already
[13308.672384] usb 2-1.4: USB disconnect, device number 6
[13308.844325] systemd[1]: run-docker-runtime\x2drunc-moby-abac240296c90e8c64e6080055f3cfae1c3299bb7e86d65cc47d94184fee1a7f-runc.b6mzff.mount: Succeeded.
[13308.907130] systemd[1]: Started libcontainer container abac240296c90e8c64e6080055f3cfae1c3299bb7e86d65cc47d94184fee1a7f.
[13308.939788] systemd[1]: docker-abac240296c90e8c64e6080055f3cfae1c3299bb7e86d65cc47d94184fee1a7f.scope: Succeeded.
[13309.279354] docker0: port 1(veth9df8735) entered disabled state
[13309.281199] device veth9df8735 left promiscuous mode
[13309.281221] docker0: port 1(veth9df8735) entered disabled state
[13309.344860] systemd[1]: var-lib-docker-overlay2-34668aca5102229aa6b2dfdb7fad869c992295d4f0f41c0e9583869bfcc1c743-merged.mount: Succeeded.
[13329.793825] systemd[1]: run-docker-runtime\x2drunc-moby-96675d970e311b99294cb8532dfac800ac72f9e100efacc2dc7ac0939c1e96b7-runc.TH1wLu.mount: Succeeded.

It seems to start with WD1003, which is the logging HDD.

But I’m not sure, if this is the root cause (meaning HDD defect) or if the USB connector has issues.

ouch!

:face_with_open_eyes_and_hand_over_mouth:

I would recommend to unmount this drive and perform

sudo fsck /dev/sdb1

Perhaps you would need to perform it more than once until all errors are fixed.
Then you may try to mount it back and try to run a node.
But it’s possible, that this drive developed bad sectors (you need to check S.M.A.R.T.), and it could be dying.

oh no, the larger node of both.

result is:

/dev/sdb1: clean, 14035127/305201152 files, 1821313782/2441608955 blocks

same for the other to HDDs:

/dev/sdc1: clean, 7215945/305201152 files, 853247118/2441608955 blocks
/dev/sda1: clean, 262/61054976 files, 4503362/244190385 blocks

So, now?

Short SMART test of the HDD sdb also shows no errors:

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     17143         -

I am currently running the long test until tomorrow.

I’ve read a couple of times, that zeroing the partition and reformatting it could help. Any advice how to do that? (of course, after copying the data.

Nobody’s able to help? Can’t belief. :innocent:

this is good!

I do not know, how it could help with the hardware issue like this:

Perhaps updating OS could help (kernel to be clear), but I do not know. Linux is very sensitive to problems with hardware.

1 Like

Ok lets give it a try.

Can you post the part of the dmesg log that shows the exact moment when you connect your drive to the USB port?. If it is not longer visible, do this.

1.- Disconect the cable from the USB
2.- Run sudo dmesg --clear
3.- Connect the USB cable.
4.- Post here here the contents of dmesg

1 Like

Thank you.

Had the idea that it might be a heating issue, due to the high load in the last few weeks.
I’ve added an additional fan to the HDDs enclosure.

Output from dmesg:

[182598.745183] usb 1-1.1: new high-speed USB device number 5 using xhci_hcd
[182598.848947] usb 1-1.1: New USB device found, idVendor=2109, idProduct=2822, bcdDevice= 5.a3
[182598.848979] usb 1-1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[182598.848998] usb 1-1.1: Product: USB2.0 Hub
[182598.849017] usb 1-1.1: Manufacturer: VIA Labs, Inc.
[182598.849038] usb 1-1.1: SerialNumber: 000000001
[182598.850980] hub 1-1.1:1.0: USB hub found
[182598.851392] hub 1-1.1:1.0: 5 ports detected
[182598.976366] usb 2-1: new SuperSpeed USB device number 7 using xhci_hcd
[182599.027496] usb 2-1: New USB device found, idVendor=2109, idProduct=8822, bcdDevice= 5.a3
[182599.027533] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[182599.027550] usb 2-1: Product: USB3.1 Hub
[182599.027563] usb 2-1: Manufacturer: VIA Labs, Inc.
[182599.027576] usb 2-1: SerialNumber: 000000001
[182599.031137] hub 2-1:1.0: USB hub found
[182599.031508] hub 2-1:1.0: 4 ports detected
[182599.517200] usb 1-1.1.5: new high-speed USB device number 6 using xhci_hcd
[182599.618703] usb 1-1.1.5: New USB device found, idVendor=2109, idProduct=8818, bcdDevice= 0.01
[182599.618722] usb 1-1.1.5: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[182599.618729] usb 1-1.1.5: Product: USB Billboard Device   
[182599.618736] usb 1-1.1.5: Manufacturer: VIA Labs, Inc.
[182599.618741] usb 1-1.1.5: SerialNumber: 0000000000000001
[182602.601495] usb 2-1.1: new SuperSpeed USB device number 8 using xhci_hcd
[182602.628529] usb 2-1.1: New USB device found, idVendor=2109, idProduct=0715, bcdDevice= 6.14
[182602.628539] usb 2-1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[182602.628546] usb 2-1.1: Product: USB3.1 SATA Bridge
[182602.628551] usb 2-1.1: Manufacturer: VIA Labs,Inc.
[182602.628557] usb 2-1.1: SerialNumber: 0000000000000001
[182602.668369] scsi host0: uas
[182602.669760] scsi 0:0:0:0: Direct-Access     WDC WD10 JFCX-68N6GN0     01.0 PQ: 0 ANSI: 6
[182602.673415] sd 0:0:0:0: Attached scsi generic sg0 type 0
[182602.673684] sd 0:0:0:0: [sda] 1953525168 512-byte logical blocks: (1.00 TB/932 GiB)
[182602.673693] sd 0:0:0:0: [sda] 4096-byte physical blocks
[182602.673878] sd 0:0:0:0: [sda] Write Protect is off
[182602.673886] sd 0:0:0:0: [sda] Mode Sense: 2f 00 00 00
[182602.674251] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[182602.674690] sd 0:0:0:0: [sda] Preferred minimum I/O size 4096 bytes
[182602.674698] sd 0:0:0:0: [sda] Optimal transfer size 33553920 bytes not a multiple of preferred minimum block size (4096 bytes)
[182602.737420]  sda: sda1
[182602.738257] sd 0:0:0:0: [sda] Attached SCSI disk
[182603.769493] usb 2-1.4: new SuperSpeed USB device number 9 using xhci_hcd
[182603.796711] usb 2-1.4: New USB device found, idVendor=2109, idProduct=0715, bcdDevice= 6.14
[182603.796723] usb 2-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[182603.796730] usb 2-1.4: Product: USB3.1 SATA Bridge
[182603.796736] usb 2-1.4: Manufacturer: VIA Labs,Inc.
[182603.796741] usb 2-1.4: SerialNumber: 0000000000000004
[182603.799662] usb-storage 2-1.4:1.0: USB Mass Storage device detected
[182603.800085] scsi host1: usb-storage 2-1.4:1.0
[182604.807129] scsi 1:0:0:0: Direct-Access              VirtualDisk           PQ: 0 ANSI: 6
[182604.808057] sd 1:0:0:0: Attached scsi generic sg1 type 0
[182604.809003] sd 1:0:0:0: [sdb] Media removed, stopped polling
[182604.811011] sd 1:0:0:0: [sdb] Attached SCSI removable disk
[182619.997854] usb 2-1.3: new SuperSpeed USB device number 10 using xhci_hcd
[182620.025088] usb 2-1.3: New USB device found, idVendor=2109, idProduct=0715, bcdDevice= 6.14
[182620.025117] usb 2-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[182620.025130] usb 2-1.3: Product: USB3.1 SATA Bridge
[182620.025140] usb 2-1.3: Manufacturer: VIA Labs,Inc.
[182620.025151] usb 2-1.3: SerialNumber: 0000000000000003
[182620.038544] scsi host2: uas
[182620.040536] scsi 2:0:0:0: Direct-Access     WDC WD10 1EFBX-68B0AN     85.0 PQ: 0 ANSI: 6
[182620.045272] sd 2:0:0:0: Attached scsi generic sg2 type 0
[182620.045529] sd 2:0:0:0: [sdc] 19532873728 512-byte logical blocks: (10.0 TB/9.10 TiB)
[182620.045545] sd 2:0:0:0: [sdc] 4096-byte physical blocks
[182620.045874] sd 2:0:0:0: [sdc] Write Protect is off
[182620.045891] sd 2:0:0:0: [sdc] Mode Sense: 2f 00 00 00
[182620.046364] sd 2:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[182620.046885] sd 2:0:0:0: [sdc] Preferred minimum I/O size 4096 bytes
[182620.046900] sd 2:0:0:0: [sdc] Optimal transfer size 33553920 bytes not a multiple of preferred minimum block size (4096 bytes)
[182620.154251]  sdc: sdc1
[182620.154966] sd 2:0:0:0: [sdc] Attached SCSI disk
[182620.297822] usb 2-1.2: new SuperSpeed USB device number 11 using xhci_hcd
[182620.325093] usb 2-1.2: New USB device found, idVendor=2109, idProduct=0715, bcdDevice= 6.14
[182620.325124] usb 2-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[182620.325131] usb 2-1.2: Product: USB3.1 SATA Bridge
[182620.325136] usb 2-1.2: Manufacturer: VIA Labs,Inc.
[182620.325142] usb 2-1.2: SerialNumber: 0000000000000002
[182620.337216] scsi host3: uas
[182620.338779] scsi 3:0:0:0: Direct-Access     WDC WD10 1EFBX-68B0AN     85.0 PQ: 0 ANSI: 6
[182620.343299] sd 3:0:0:0: Attached scsi generic sg3 type 0
[182620.343451] sd 3:0:0:0: [sdd] 19532873728 512-byte logical blocks: (10.0 TB/9.10 TiB)
[182620.343460] sd 3:0:0:0: [sdd] 4096-byte physical blocks
[182620.343694] sd 3:0:0:0: [sdd] Write Protect is off
[182620.343706] sd 3:0:0:0: [sdd] Mode Sense: 2f 00 00 00
[182620.344093] sd 3:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[182620.344575] sd 3:0:0:0: [sdd] Preferred minimum I/O size 4096 bytes
[182620.344587] sd 3:0:0:0: [sdd] Optimal transfer size 33553920 bytes not a multiple of preferred minimum block size (4096 bytes)
[182620.465180]  sdd: sdd1
[182620.465711] sd 3:0:0:0: [sdd] Attached SCSI disk

I’ve seen sdb is not mounted anymore, there is sdd now.

Which one of all those disks ( sda, sdb, sdc, sdc) is the one that is giving you problems?

To be honest: I don’t know, as all 3 mounts are lost in those cases. A reboot of the RPi helps.

WD1001 is the largest one (80% filled) / 10 TB capacity
WD1002 is the second node (35% filled or so) / 10 TB capacity
WD1003 is used for logging only / 1 TB capacity

Could it be the SD (long endurance from SanDisk)? Just to bring all possibilities on the table.

OK, lets disable the uas mode for all of them, and see how it goes. (its common that some USB chipsets present problems working in uas mode under linux)

please do the following:

1.- Disconnect all your disks from the USB. (obviously first stop your nodes and umount the partitions)
2.- run: echo "2109:2822:u,2109:8822:u,2109:8818:u,2109:0715:u" | sudo tee /sys/module/usb_storage/parameters/quirks
3.- reconnect your discs and test them thoroughly if you can, if not, simply mount partitions, start your nodes and keep an eye.

Note. The setting above doesn’t survive a reboot.

2 Likes

thank you, just waited for the next “disconnect”, which was today.

done.

dmesg, if it is helpful:

[ 3586.116491] usb 1-1.1: new high-speed USB device number 5 using xhci_hcd
[ 3586.220164] usb 1-1.1: New USB device found, idVendor=2109, idProduct=2822, bcdDevice= 5.a3
[ 3586.220179] usb 1-1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 3586.220186] usb 1-1.1: Product: USB2.0 Hub
[ 3586.220192] usb 1-1.1: Manufacturer: VIA Labs, Inc.
[ 3586.220197] usb 1-1.1: SerialNumber: 000000001
[ 3586.221791] hub 1-1.1:1.0: USB hub found
[ 3586.222003] hub 1-1.1:1.0: 5 ports detected
[ 3586.346357] usb 2-1: new SuperSpeed USB device number 7 using xhci_hcd
[ 3586.394747] usb 2-1: New USB device found, idVendor=2109, idProduct=8822, bcdDevice= 5.a3
[ 3586.394766] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 3586.394772] usb 2-1: Product: USB3.1 Hub
[ 3586.394778] usb 2-1: Manufacturer: VIA Labs, Inc.
[ 3586.394783] usb 2-1: SerialNumber: 000000001
[ 3586.397091] hub 2-1:1.0: USB hub found
[ 3586.397269] hub 2-1:1.0: 4 ports detected
[ 3586.888564] usb 1-1.1.5: new high-speed USB device number 6 using xhci_hcd
[ 3586.990570] usb 1-1.1.5: New USB device found, idVendor=2109, idProduct=8818, bcdDevice= 0.01
[ 3586.990602] usb 1-1.1.5: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 3586.990618] usb 1-1.1.5: Product: USB Billboard Device   
[ 3586.990632] usb 1-1.1.5: Manufacturer: VIA Labs, Inc.
[ 3586.990645] usb 1-1.1.5: SerialNumber: 0000000000000001
[ 3590.049047] usb 2-1.1: new SuperSpeed USB device number 8 using xhci_hcd
[ 3590.076473] usb 2-1.1: New USB device found, idVendor=2109, idProduct=0715, bcdDevice= 6.14
[ 3590.076495] usb 2-1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 3590.076512] usb 2-1.1: Product: USB3.1 SATA Bridge
[ 3590.076526] usb 2-1.1: Manufacturer: VIA Labs,Inc.
[ 3590.076540] usb 2-1.1: SerialNumber: 0000000000000001
[ 3590.091760] usb 2-1.1: UAS is ignored for this device, using usb-storage instead
[ 3590.091891] usb 2-1.1: UAS is ignored for this device, using usb-storage instead
[ 3590.091908] usb-storage 2-1.1:1.0: USB Mass Storage device detected
[ 3590.095967] usb-storage 2-1.1:1.0: Quirks match for vid 2109 pid 0715: 800000
[ 3590.096256] scsi host0: usb-storage 2-1.1:1.0
[ 3591.122186] scsi 0:0:0:0: Direct-Access     WDC WD10 JFCX-68N6GN0     01.0 PQ: 0 ANSI: 6
[ 3591.122634] sd 0:0:0:0: Attached scsi generic sg0 type 0
[ 3591.123003] sd 0:0:0:0: [sda] 1953525168 512-byte logical blocks: (1.00 TB/932 GiB)
[ 3591.123788] sd 0:0:0:0: [sda] Write Protect is off
[ 3591.123799] sd 0:0:0:0: [sda] Mode Sense: 2f 00 00 00
[ 3591.124613] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 3591.198813]  sda: sda1
[ 3591.199264] sd 0:0:0:0: [sda] Attached SCSI disk
[ 3591.217075] usb 2-1.4: new SuperSpeed USB device number 9 using xhci_hcd
[ 3591.248206] usb 2-1.4: New USB device found, idVendor=2109, idProduct=0715, bcdDevice= 6.14
[ 3591.248226] usb 2-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 3591.248233] usb 2-1.4: Product: USB3.1 SATA Bridge
[ 3591.248239] usb 2-1.4: Manufacturer: VIA Labs,Inc.
[ 3591.248244] usb 2-1.4: SerialNumber: 0000000000000004
[ 3591.251143] usb-storage 2-1.4:1.0: USB Mass Storage device detected
[ 3591.251462] usb-storage 2-1.4:1.0: Quirks match for vid 2109 pid 0715: 800000
[ 3591.251560] scsi host1: usb-storage 2-1.4:1.0
[ 3592.274325] scsi 1:0:0:0: Direct-Access              VirtualDisk           PQ: 0 ANSI: 6
[ 3592.274753] sd 1:0:0:0: Attached scsi generic sg1 type 0
[ 3592.275501] sd 1:0:0:0: [sdb] Media removed, stopped polling
[ 3592.276232] sd 1:0:0:0: [sdb] Attached SCSI removable disk
[ 3602.980288] EXT4-fs (sda1): mounted filesystem with ordered data mode. Quota mode: none.
[ 3607.377827] usb 2-1.3: new SuperSpeed USB device number 10 using xhci_hcd
[ 3607.404931] usb 2-1.3: New USB device found, idVendor=2109, idProduct=0715, bcdDevice= 6.14
[ 3607.404950] usb 2-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 3607.404957] usb 2-1.3: Product: USB3.1 SATA Bridge
[ 3607.404963] usb 2-1.3: Manufacturer: VIA Labs,Inc.
[ 3607.404968] usb 2-1.3: SerialNumber: 0000000000000003
[ 3607.408109] usb 2-1.3: UAS is ignored for this device, using usb-storage instead
[ 3607.408215] usb 2-1.3: UAS is ignored for this device, using usb-storage instead
[ 3607.408225] usb-storage 2-1.3:1.0: USB Mass Storage device detected
[ 3607.408596] usb-storage 2-1.3:1.0: Quirks match for vid 2109 pid 0715: 800000
[ 3607.408706] scsi host2: usb-storage 2-1.3:1.0
[ 3607.713768] usb 2-1.2: new SuperSpeed USB device number 11 using xhci_hcd
[ 3607.741021] usb 2-1.2: New USB device found, idVendor=2109, idProduct=0715, bcdDevice= 6.14
[ 3607.741035] usb 2-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 3607.741041] usb 2-1.2: Product: USB3.1 SATA Bridge
[ 3607.741047] usb 2-1.2: Manufacturer: VIA Labs,Inc.
[ 3607.741053] usb 2-1.2: SerialNumber: 0000000000000002
[ 3607.744023] usb 2-1.2: UAS is ignored for this device, using usb-storage instead
[ 3607.744090] usb 2-1.2: UAS is ignored for this device, using usb-storage instead
[ 3607.744098] usb-storage 2-1.2:1.0: USB Mass Storage device detected
[ 3607.744453] usb-storage 2-1.2:1.0: Quirks match for vid 2109 pid 0715: 800000
[ 3607.744553] scsi host3: usb-storage 2-1.2:1.0
[ 3608.435446] scsi 2:0:0:0: Direct-Access     WDC WD10 1EFBX-68B0AN     85.0 PQ: 0 ANSI: 6
[ 3608.436304] sd 2:0:0:0: Attached scsi generic sg2 type 0
[ 3608.437487] sd 2:0:0:0: [sdc] Very big device. Trying to use READ CAPACITY(16).
[ 3608.437926] sd 2:0:0:0: [sdc] 19532873728 512-byte logical blocks: (10.0 TB/9.10 TiB)
[ 3608.437943] sd 2:0:0:0: [sdc] 4096-byte physical blocks
[ 3608.438927] sd 2:0:0:0: [sdc] Write Protect is off
[ 3608.438945] sd 2:0:0:0: [sdc] Mode Sense: 2f 00 00 00
[ 3608.439859] sd 2:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 3608.500556]  sdc: sdc1
[ 3608.501161] sd 2:0:0:0: [sdc] Attached SCSI disk
[ 3608.754990] scsi 3:0:0:0: Direct-Access     WDC WD10 1EFBX-68B0AN     85.0 PQ: 0 ANSI: 6
[ 3608.755438] sd 3:0:0:0: Attached scsi generic sg3 type 0
[ 3608.755715] sd 3:0:0:0: [sdd] Very big device. Trying to use READ CAPACITY(16).
[ 3608.755935] sd 3:0:0:0: [sdd] 19532873728 512-byte logical blocks: (10.0 TB/9.10 TiB)
[ 3608.755943] sd 3:0:0:0: [sdd] 4096-byte physical blocks
[ 3608.756801] sd 3:0:0:0: [sdd] Write Protect is off
[ 3608.756813] sd 3:0:0:0: [sdd] Mode Sense: 2f 00 00 00
[ 3608.757622] sd 3:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 3608.814222]  sdd: sdd1
[ 3608.814598] sd 3:0:0:0: [sdd] Attached SCSI disk
[ 3622.333736] EXT4-fs (sdd1): mounted filesystem with ordered data mode. Quota mode: none.
[ 3622.615833] EXT4-fs (sdc1): mounted filesystem with ordered data mode. Quota mode: none.

nodes seem to run properly for the moment.

Happened again today.

please post the dmesg log