Input/output error while canceling to write a piece

nerdatwork · May 22, 2024, 7:49am

2024-05-22T05:44:26Z    ERROR   piecestore      error during canceling a piece write    {"Process": "storagenode", "Piece ID": "7KPI7Y34GHEEHTLHBYNQW5JKYE6MJOLJK5XHZ5K5WNLAGJ3L6JNQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote
Address": "172.17.0.1:58188", "error": "pieces error: filestore error: close config/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/7k/pi7y34gheehtlhbynqw5jkye6mjoljk5xhz5k5wnlagj3l6jnq.sj1: input/output error", "errorVerbose": "pieces error:
filestore error: close config/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/7k/pi7y34gheehtlhbynqw5jkye6mjoljk5xhz5k5wnlagj3l6jnq.sj1: input/output error\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobWriter).Cancel:121\n\tstorj.io/storj/
storagenode/pieces.(*MonitoredBlobWriter).Cancel:61\n\tstorj.io/storj/storagenode/pieces.(*Writer).Cancel:210\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func4:399\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:538\n\tstorj.io/common/pb.DR
PCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:167\n\tstorj.io/drpc/d
rpcserver.(*Server).ServeOne:109\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:157\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}

I have 35 of these in current log.

I think its a slow disk error so more in line with context canceled than input/output error. Please share your thoughts.

Mitsos · May 22, 2024, 8:32am

Does this disk pass full smart scan? I/O errors could be a defective disk or bad connection to the drive, or controller giving up.

nerdatwork · May 22, 2024, 10:25am

Everything says OK

Mitsos · May 22, 2024, 10:30am

No, that disk is dying, and judging by the numbers it’s dying fast.

Read error rate: worst is 64, threshold for failed is 44. Raw error rate that constantly changes (last column) is never a good sign. Every manufacturer reports this differently, so YMMV.
Seek error rate: worst 60, threshold 45. Again, last column should be changing constantly, see above.
Emergency retract: is that drive connected to a bicycle chained to a generator? why did it pull the heads off the platter 2296 times?
ECC on the fly: the disk is reporting failed checksums. The sectors it’s reading don’t match their checksum, worst just inches off threshold.
ULTRA DMA CRC = bad cable. just 1 raw reported, so let’s ignore that.

nerdatwork · May 22, 2024, 10:33am

This is using Seagates’s tool. Are there better tools to get SMART results ?

Mitsos · May 22, 2024, 10:37am

No, just pay attention how big the worst > threshold difference is. In order for something to trigger as failed, the threshold must be reached. Some attributes are more useful when looking at their raw data (ie pending sectors, reallocated sectors etc). It’s all a correlation of what the drive is reporting and what it is doing. It’s reporting I/O errors to the host. I see that the head is having a hard time getting positioned (could be the trigger for all those emergency retracts). On top of that, the head isn’t reading back as cleanly as it should.

My guess? failed head. Can’t read current sector => seek errors. seek errors leading to emergency retracts so it resets its position. Read error + ecc means the sectors read aren’t matching the data on the platter.

donald.m.motsinger · May 22, 2024, 11:40am

Don’t take the raw values at face value. Seagate’s SMART data uses 48 bit hexidecimal readings converted to decimal.

With Seagate drives I need to run smartctl -a -v 1,raw48:54 -v 7,raw48:54 ...

Here’s the output of one of my drives, first without the extra options and then with.

smartctl -a  /dev/sde
...
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   073   064   044    Pre-fail  Always       -       21205362
  3 Spin_Up_Time            0x0003   093   093   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       719
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   081   060   045    Pre-fail  Always       -       124260026
  9 Power_On_Hours          0x0032   064   064   000    Old_age   Always       -       32355 (212 231 0)
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       172
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   093   000    Old_age   Always       -       17180131340
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   064   060   040    Old_age   Always       -       36 (Min/Max 32/38)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       390
193 Load_Cycle_Count        0x0032   083   083   000    Old_age   Always       -       34316
194 Temperature_Celsius     0x0022   036   040   000    Old_age   Always       -       36 (0 20 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   001   000    Old_age   Always       -       799
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       32081 (25 17 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       211242908404
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       279680067603

smartctl -a -v 1,raw48:54 -v 7,raw48:54 /dev/sde
...
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   073   064   044    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0003   093   093   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       719
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   081   060   045    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   064   064   000    Old_age   Always       -       32355 (170 228 0)
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       172
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   093   000    Old_age   Always       -       17180131340
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   064   060   040    Old_age   Always       -       36 (Min/Max 32/38)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       390
193 Load_Cycle_Count        0x0032   083   083   000    Old_age   Always       -       34319
194 Temperature_Celsius     0x0022   036   040   000    Old_age   Always       -       36 (0 20 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   001   000    Old_age   Always       -       799
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       32082 (156 192 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       211242920900
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       279680206803

Mitsos · May 22, 2024, 12:11pm

The only thing I personally trust to give a go/no-go at a drive is a

smartctl -x /dev/sda

I agree that the raw value is most of the times meaningless. But the person you linked to is viewing every attribute in isolation. Here is a smart report for a now failed drive. This drive was flagged by me more than 1 year ago as being on its last breath and the owner should consider replacing it. It failed (sigh…) two months ago and the owner has yet to replace it.
Serial removed in order to protect the guilty.

Ignore the failed item. Would you say this drive is good or not? If that item hadn’t triggered, it would be a pass by any SMART suite.

smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.0-20-amd64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     SAMSUNG SpinPoint F3
Device Model:     SAMSUNG HD103SJ
Serial Number:    S246J1B
LU WWN Device Id: 5 0024e9 004ef273b
Firmware Version: 1AJ10001
User Capacity:    1,000,204,886,016 bytes [1.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database 7.3/5577
ATA Version is:   ATA8-ACS T13/1699-D revision 6
SATA Version is:  SATA 2.6, 3.0 Gb/s
Local Time is:    Wed May 22 15:09:14 2024 EEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Disabled
APM level is:     254 (maximum performance)
Rd look-ahead is: Enabled
Write cache is:   Enabled
DSN feature is:   Unavailable
ATA Security is:  Disabled, frozen [SEC2]
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: FAILED!
Drive failure expected in less than 24 hours. SAVE ALL DATA.
See vendor-specific Attribute list for failed Attributes.

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      ( 121) The previous self-test completed having
                                        the read element of the test failed.
Total time to complete Offline
data collection:                ( 9480) seconds.
Offline data collection
capabilities:                    (0x5b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 158) minutes.
SCT capabilities:              (0x003f) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR-K   001   001   051    NOW  1038967
  2 Throughput_Performance  -OS--K   045   045   000    -    10722
  3 Spin_Up_Time            PO---K   070   069   025    -    9119
  4 Start_Stop_Count        -O--CK   100   100   000    -    100
  5 Reallocated_Sector_Ct   PO--CK   252   252   010    -    0
  7 Seek_Error_Rate         -OSR-K   252   252   051    -    0
  8 Seek_Time_Performance   --S--K   252   252   015    -    0
  9 Power_On_Hours          -O--CK   100   100   000    -    54915
 10 Spin_Retry_Count        -O--CK   252   252   051    -    0
 11 Calibration_Retry_Count -O--CK   252   252   000    -    0
 12 Power_Cycle_Count       -O--CK   100   100   000    -    78
191 G-Sense_Error_Rate      -O---K   100   100   000    -    40
192 Power-Off_Retract_Count -O---K   252   252   000    -    0
194 Temperature_Celsius     -O----   064   053   000    -    36 (Min/Max 15/47)
195 Hardware_ECC_Recovered  -O-RCK   100   100   000    -    0
196 Reallocated_Event_Count -O--CK   252   252   000    -    0
197 Current_Pending_Sector  -O--CK   100   100   000    -    8
198 Offline_Uncorrectable   ----CK   252   252   000    -    0
199 UDMA_CRC_Error_Count    -OS-CK   100   100   000    -    120
200 Multi_Zone_Error_Rate   -O-R-K   100   100   000    -    351
223 Load_Retry_Count        -O--CK   252   252   000    -    0
225 Load_Cycle_Count        -O--CK   100   100   000    -    103
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      2  Comprehensive SMART error log
0x03       GPL     R/O      2  Ext. Comprehensive SMART error log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      2  Extended self-test log
0x08       GPL     R/O      2  Power Conditions log
0x09           SL  R/W      1  Selective self-test log
0x10       GPL     R/O      1  NCQ Command Error log
0x11       GPL     R/O      1  SATA Phy Event Counters log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (2 sectors)
Device Error Count: 41304 (device log contains only the most recent 8 errors)
        CR     = Command Register
        FEATR  = Features Register
        COUNT  = Count (was: Sector Count) Register
        LBA_48 = Upper bytes of LBA High/Mid/Low Registers ]  ATA-8
        LH     = LBA High (was: Cylinder High) Register    ]   LBA
        LM     = LBA Mid (was: Cylinder Low) Register      ] Register
        LL     = LBA Low (was: Sector Number) Register     ]
        DV     = Device (was: Device/Head) Register
        DC     = Device Control Register
        ER     = Error register
        ST     = Status register
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 41304 [7] occurred at disk power-on lifetime: 54915 hours (2288 days + 3 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 06 00 00 27 81 98 6a e0 00  Error: UNC 6 sectors at LBA = 0x2781986a = 662804586

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  25 00 00 00 08 00 00 27 81 98 68 e0 00     00:27:26.609  READ DMA EXT
  25 00 00 00 08 00 00 13 d1 31 40 e0 00     00:27:26.609  READ DMA EXT
  25 00 00 00 08 00 00 51 90 18 18 e0 00     00:27:26.609  READ DMA EXT
  25 00 00 00 08 00 00 28 90 18 00 e0 00     00:27:26.609  READ DMA EXT
  25 00 00 00 08 00 00 15 cd b2 c0 e0 00     00:27:26.609  READ DMA EXT

Error 41303 [6] occurred at disk power-on lifetime: 54914 hours (2288 days + 2 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 06 00 00 27 81 98 6a e0 00  Error: UNC 6 sectors at LBA = 0x2781986a = 662804586

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  25 00 00 00 08 00 00 27 81 98 68 e0 00     00:27:23.009  READ DMA EXT
  25 00 00 00 08 00 00 15 d0 18 30 e0 00     00:27:23.009  READ DMA EXT
  25 00 00 00 08 00 00 42 10 18 70 e0 00     00:27:23.009  READ DMA EXT
  25 00 00 00 08 00 00 42 10 18 78 e0 00     00:27:23.009  READ DMA EXT
  25 00 00 00 08 00 00 11 90 18 60 e0 00     00:27:23.009  READ DMA EXT

Error 41302 [5] occurred at disk power-on lifetime: 54913 hours (2288 days + 1 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 06 00 00 27 81 98 6a e0 00  Error: UNC 6 sectors at LBA = 0x2781986a = 662804586

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  25 00 00 00 08 00 00 27 81 98 68 e0 00     00:27:19.409  READ DMA EXT
  25 00 00 00 08 00 00 15 d0 18 38 e0 00     00:27:19.409  READ DMA EXT
  25 00 00 00 08 00 00 3b 50 18 60 e0 00     00:27:19.409  READ DMA EXT
  25 00 00 00 08 00 00 15 ee be 38 e0 00     00:27:19.409  READ DMA EXT
  25 00 00 00 d8 00 00 15 d0 f1 30 e0 00     00:27:19.409  READ DMA EXT

Error 41301 [4] occurred at disk power-on lifetime: 54912 hours (2288 days + 0 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 06 00 00 27 81 98 6a e0 00  Error: UNC 6 sectors at LBA = 0x2781986a = 662804586

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  25 00 00 00 08 00 00 27 81 98 68 e0 00     00:27:15.809  READ DMA EXT
  25 00 00 00 08 00 00 15 d0 18 48 e0 00     00:27:15.809  READ DMA EXT
  25 00 00 00 08 00 00 40 d0 18 50 e0 00     00:27:15.809  READ DMA EXT
  25 00 00 00 08 00 00 40 d0 18 58 e0 00     00:27:15.809  READ DMA EXT
  25 00 00 00 08 00 00 40 d0 18 60 e0 00     00:27:15.809  READ DMA EXT

Error 41300 [3] occurred at disk power-on lifetime: 54911 hours (2287 days + 23 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 06 00 00 27 81 98 6a e0 00  Error: UNC 6 sectors at LBA = 0x2781986a = 662804586

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  25 00 00 00 08 00 00 27 81 98 68 e0 00     00:27:12.209  READ DMA EXT
  25 00 00 00 08 00 00 15 d0 18 58 e0 00     00:27:12.209  READ DMA EXT
  25 00 00 00 08 00 00 15 fe 86 78 e0 00     00:27:12.209  READ DMA EXT
  25 00 00 00 08 00 00 15 b4 9e 70 e0 00     00:27:12.209  READ DMA EXT
  25 00 00 00 08 00 00 4e 10 18 30 e0 00     00:27:12.209  READ DMA EXT

Error 41299 [2] occurred at disk power-on lifetime: 54910 hours (2287 days + 22 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 06 00 00 27 81 98 6a e0 00  Error: UNC 6 sectors at LBA = 0x2781986a = 662804586

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  25 00 00 00 08 00 00 27 81 98 68 e0 00     00:27:08.610  READ DMA EXT
  ef 00 10 00 02 00 00 00 00 00 00 a0 00     00:27:08.610  SET FEATURES [Enable SATA feature]
  27 00 00 00 00 00 00 00 00 00 00 e0 00     00:27:08.610  READ NATIVE MAX ADDRESS EXT [OBS-ACS-3]
  ec 00 00 00 00 00 00 00 00 00 00 a0 00     00:27:08.610  IDENTIFY DEVICE
  ef 00 03 00 46 00 00 00 00 00 00 a0 00     00:27:08.610  SET FEATURES [Set transfer mode]

Error 41298 [1] occurred at disk power-on lifetime: 54910 hours (2287 days + 22 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 06 00 00 27 81 98 6a e0 00  Error: UNC 6 sectors at LBA = 0x2781986a = 662804586

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  25 00 00 00 08 00 00 27 81 98 68 e0 00     00:27:08.609  READ DMA EXT
  25 00 00 00 08 00 00 2a 90 18 00 e0 00     00:27:08.609  READ DMA EXT
  25 00 00 00 08 00 00 15 91 74 c0 e0 00     00:27:08.609  READ DMA EXT
  25 00 00 00 08 00 00 13 d4 4d 98 e0 00     00:27:08.609  READ DMA EXT
  25 00 00 00 08 00 00 13 d1 19 a0 e0 00     00:27:08.609  READ DMA EXT

Error 41297 [0] occurred at disk power-on lifetime: 54909 hours (2287 days + 21 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 06 00 00 27 81 98 6a e0 00  Error: UNC 6 sectors at LBA = 0x2781986a = 662804586

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  25 00 00 00 08 00 00 27 81 98 68 e0 00     00:27:05.009  READ DMA EXT
  c8 00 00 00 08 00 00 00 d0 18 68 e0 00     00:27:05.009  READ DMA
  c8 00 00 00 08 00 00 00 d0 18 70 e0 00     00:27:05.009  READ DMA
  c8 00 00 00 08 00 00 00 d0 18 78 e0 00     00:27:05.009  READ DMA
  c8 00 00 00 08 00 00 00 10 18 00 e1 00     00:27:05.009  READ DMA

SMART Extended Self-test Log Version: 1 (2 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       90%     52923         194307160
# 2  Extended offline    Completed: read failure       90%     52922         194307159
# 3  Short offline       Completed without error       00%     52922         -
# 4  Short offline       Completed: read failure       90%     52922         1360010152
# 5  Short offline       Completed: read failure       90%     52922         1024505857
# 6  Extended offline    Completed: read failure       90%     52917         194307155
# 7  Extended offline    Completed: read failure       90%     52916         194307153
# 8  Extended offline    Completed: read failure       90%     52916         194307157
# 9  Short offline       Completed without error       00%     52916         -
#10  Short offline       Completed: read failure       90%     52915         1024489976
#11  Short offline       Completed: read failure       90%     52915         944778267
#12  Short offline       Completed without error       00%     52913         -
#13  Short offline       Completed: read failure       90%     52913         1196432800
#14  Short offline       Completed: read failure       90%     52913         1024506224
#15  Short offline       Completed: read failure       90%     52913         1024574512
#16  Short offline       Completed: read failure       90%     52913         1024475985
#17  Short offline       Completed: read failure       90%     52913         1024475909
#18  Short offline       Completed: read failure       90%     52913         856695069
#19  Short offline       Completed: read failure       90%     52913         1024475904
#20  Short offline       Completed: read failure       90%     52913         1024471609
#21  Short offline       Completed: read failure       90%     52913         1024469722

SMART Selective self-test log data structure revision number 0
Note: revision number not 1 implies that no selective self-test has ever been run
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Completed_read_failure [90% left] (0-65535)
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:                  2
SCT Version (vendor specific):       256 (0x0100)
Device State:                        Active (0)
Current Temperature:                    36 Celsius
Power Cycle Min/Max Temperature:     27/37 Celsius
Lifetime    Min/Max Temperature:     18/64 Celsius
Specified Max Operating Temperature:    80 Celsius
Under/Over Temperature Limit Count:   0/0

SCT Temperature History Version:     2
Temperature Sampling Period:         5 minutes
Temperature Logging Interval:        5 minutes
Min/Max recommended Temperature:     -5/80 Celsius
Min/Max Temperature Limit:           -10/85 Celsius
Temperature History Size (Index):    128 (39)

Index    Estimated Time   Temperature Celsius
  40    2024-05-22 04:30    36  *****************
 ...    ..( 48 skipped).    ..  *****************
  89    2024-05-22 08:35    36  *****************
  90    2024-05-22 08:40    35  ****************
 ...    ..( 12 skipped).    ..  ****************
 103    2024-05-22 09:45    35  ****************
 104    2024-05-22 09:50    36  *****************
 105    2024-05-22 09:55    35  ****************
 ...    ..(  8 skipped).    ..  ****************
 114    2024-05-22 10:40    35  ****************
 115    2024-05-22 10:45    36  *****************
 116    2024-05-22 10:50    35  ****************
 117    2024-05-22 10:55    35  ****************
 118    2024-05-22 11:00    36  *****************
 119    2024-05-22 11:05    35  ****************
 120    2024-05-22 11:10    35  ****************
 121    2024-05-22 11:15    35  ****************
 122    2024-05-22 11:20    36  *****************
 123    2024-05-22 11:25    35  ****************
 124    2024-05-22 11:30    36  *****************
 125    2024-05-22 11:35    36  *****************
 126    2024-05-22 11:40    35  ****************
 127    2024-05-22 11:45    36  *****************
   0    2024-05-22 11:50    35  ****************
   1    2024-05-22 11:55    36  *****************
   2    2024-05-22 12:00    35  ****************
   3    2024-05-22 12:05    36  *****************
   4    2024-05-22 12:10    35  ****************
   5    2024-05-22 12:15    36  *****************
 ...    ..(  3 skipped).    ..  *****************
   9    2024-05-22 12:35    36  *****************
  10    2024-05-22 12:40    35  ****************
  11    2024-05-22 12:45    36  *****************
 ...    ..(  6 skipped).    ..  *****************
  18    2024-05-22 13:20    36  *****************
  19    2024-05-22 13:25    35  ****************
  20    2024-05-22 13:30    35  ****************
  21    2024-05-22 13:35    36  *****************
 ...    ..( 17 skipped).    ..  *****************
  39    2024-05-22 15:05    36  *****************

SCT Error Recovery Control:
           Read: Disabled
          Write: Disabled

Device Statistics (GP/SMART Log 0x04) not supported

Pending Defects log (GP Log 0x0c) not supported

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x0001  4            6  Command failed due to ICRC error
0x0002  4            6  R_ERR response for data FIS
0x0003  4            6  R_ERR response for device-to-host data FIS
0x0004  4            0  R_ERR response for host-to-device data FIS
0x0005  4            0  R_ERR response for non-data FIS
0x0006  4            0  R_ERR response for device-to-host non-data FIS
0x0007  4            0  R_ERR response for host-to-device non-data FIS
0x0008  4            0  Device-to-host non-data FIS retries
0x0009  4            8  Transition from drive PhyRdy to drive PhyNRdy
0x000a  4            8  Device-to-host register FISes sent due to a COMRESET
0x000b  4            0  CRC errors within host-to-device FIS
0x000d  4            0  Non-CRC errors within host-to-device FIS
0x000f  4            0  R_ERR response for host-to-device data FIS, CRC
0x0010  4            0  R_ERR response for host-to-device data FIS, non-CRC
0x0012  4            0  R_ERR response for host-to-device non-data FIS, CRC
0x0013  4            0  R_ERR response for host-to-device non-data FIS, non-CRC
0x8e00  4            0  Vendor specific
0x8e01  4            6  Vendor specific
0x8e02  4            0  Vendor specific
0x8e03  4            0  Vendor specific
0x8e04  4            0  Vendor specific
0x8e05  4            0  Vendor specific
0x8e06  4        42567  Vendor specific
0x8e07  4        70933  Vendor specific
0x8e08  4         2321  Vendor specific
0x8e09  4         7423  Vendor specific
0x8e0a  4       148414  Vendor specific
0x8e0b  4    104757888  Vendor specific
0x8e0c  4       519261  Vendor specific
0x8e0d  4        15394  Vendor specific
0x8e0e  4        55138  Vendor specific
0x8e0f  4     24252672  Vendor specific
0x8e10  4       220552  Vendor specific
0x8e11  4            6  Vendor specific