Hello, recently I have updated my storage node to version v1.71.2 and now it keep restarting every so often. in the logs I have found that just before the restart there is a FATAL error but not entirely sure what is causing it. I have check the SMART data of the HDD and everything seems okay.
2023-02-01T13:18:15.841-0600 INFO piecestore uploaded {“Process”: “storagenode”, “Piece ID”: “EGVSTJKRBPDSIIKUIDHPK44HCYPMILNZQWFKM7WGSIC7CWFGLOCA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 19456}
2023-02-01T13:18:16.239-0600 INFO piecestore uploaded {“Process”: “storagenode”, “Piece ID”: “3AA4LLGVSOYS4QNCRVN7UCZN5J6QCVVGADER4NRRTQGEOEXMVCTQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT_REPAIR”, “Size”: 86272}
2023-02-01T13:18:16.308-0600 INFO piecestore downloaded {“Process”: “storagenode”, “Piece ID”: “6ZMO6OEPMLJRJF3QZQ7UFBYVJFX2OP7GSJ3JKX5TAXL3TJ2S756A”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”}
2023-02-01T13:18:17.203-0600 INFO piecestore upload started {“Process”: “storagenode”, “Piece ID”: “66ADRPYS7DZMQ3AF5ZG3W2QCO6ETQNG73LUHMUJOUIFCQWIXN7LA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 488018982784}
2023-02-01T13:18:17.340-0600 INFO piecestore uploaded {“Process”: “storagenode”, “Piece ID”: “INE4KADL3KSB2ZFIJKEKXEKISHG6L5VBY6VR4TBCHF4KCLE5UJNQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 73728}
2023-02-01T13:18:17.662-0600 INFO piecestore download started {“Process”: “storagenode”, “Piece ID”: “O6SJV2PJ4IGM5PLQ7ZKFOIVNBLI4E72RRDFFYK3QXLRRHFDXCGDQ”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “GET_REPAIR”}
2023-02-01T13:18:17.753-0600 INFO piecestore download started {“Process”: “storagenode”, “Piece ID”: “QYJADHU2D2NXZYV3U5SE6WGQT7FOJLWYTKLNDNKGEGGV7VDPNLIQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”}
2023-02-01T13:18:17.980-0600 ERROR piecestore:cache error getting current used space: {“Process”: “storagenode”, “error”: “readdirent /mnt/node3/storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/32: bad message; readdirent /mnt/node3/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/op: bad message; readdirent /mnt/node3/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/jh: bad message”, “errorVerbose”: “group:\n— readdirent /mnt/node3/storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/32: bad message\n— readdirent /mnt/node3/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/op: bad message\n— readdirent /mnt/node3/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/jh: bad message”}
2023-02-01T13:18:17.981-0600 ERROR services unexpected shutdown of a runner {“Process”: “storagenode”, “name”: “piecestore:cache”, “error”: “readdirent /mnt/node3/storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/32: bad message; readdirent /mnt/node3/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/op: bad message; readdirent /mnt/node3/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/jh: bad message”, “errorVerbose”: “group:\n— readdirent /mnt/node3/storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/32: bad message\n— readdirent /mnt/node3/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/op: bad message\n— readdirent /mnt/node3/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/jh: bad message”}
2023-02-01T13:18:17.990-0600 INFO piecestore downloaded {“Process”: “storagenode”, “Piece ID”: “QYJADHU2D2NXZYV3U5SE6WGQT7FOJLWYTKLNDNKGEGGV7VDPNLIQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”}
2023-02-01T13:18:17.990-0600 INFO piecestore downloaded {“Process”: “storagenode”, “Piece ID”: “O6SJV2PJ4IGM5PLQ7ZKFOIVNBLI4E72RRDFFYK3QXLRRHFDXCGDQ”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “GET_REPAIR”}
2023-02-01T13:18:17.991-0600 INFO piecestore downloaded {“Process”: “storagenode”, “Piece ID”: “C5LXRRACGPGFY4IHS3EYQ6AIQBAN6ZKEIOV3SATPFCIE75MLDM7A”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”}
2023-02-01T13:18:18.009-0600 INFO piecestore upload canceled {“Process”: “storagenode”, “Piece ID”: “AGCHU4K42CBLMOCONF4GP6EQWPJYCRSZ7WGKBMIRDEPXGMYSBSOA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “PUT”, “Size”: 532480}
2023-02-01T13:18:18.010-0600 INFO piecestore upload canceled {“Process”: “storagenode”, “Piece ID”: “K2LH5EMK2PHDDOOFUN5MRAN5ES24BOIETMPEFNC6DB3UN2AUJOGA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 0}
2023-02-01T13:18:18.011-0600 INFO piecestore upload canceled {“Process”: “storagenode”, “Piece ID”: “66ADRPYS7DZMQ3AF5ZG3W2QCO6ETQNG73LUHMUJOUIFCQWIXN7LA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 0}
2023-02-01T13:18:18.011-0600 INFO piecestore upload canceled {“Process”: “storagenode”, “Piece ID”: “G2EKSK7V74OAKSPVAJXPBJ5MJOHJL7G4TCPM27GSYU6MSDMKG3LA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 0}
2023-02-01T13:18:18.011-0600 INFO piecestore upload canceled {“Process”: “storagenode”, “Piece ID”: “N7V2EBC7ASRHBNBGJGCVD5KKAKQFFW5O3PQDTNY7NQVUXHMB737Q”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 1581056}
2023-02-01T13:18:18.048-0600 FATAL Unrecoverable error {“Process”: “storagenode”, “error”: “readdirent /mnt/node3/storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/32: bad message; readdirent /mnt/node3/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/op: bad message; readdirent /mnt/node3/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/jh: bad message”, “errorVerbose”: “group:\n— readdirent /mnt/node3/storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/32: bad message\n— readdirent /mnt/node3/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/op: bad message\n— readdirent /mnt/node3/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/jh: bad message”}
2023-02-01T13:19:18.779-0600 INFO Configuration loaded {“Process”: “storagenode”, “Location”: “/mnt/node3/config.yaml”}
2023-02-01T13:19:19.095-0600 INFO Anonymized tracing enabled {“Process”: “storagenode”}
2023-02-01T13:19:19.104-0600 INFO Operator email {“Process”: “storagenode”, “Address”: “”}
2023-02-01T13:19:19.104-0600 INFO Operator wallet {“Process”: “storagenode”, “Address”: “”}
2023-02-01T13:19:21.823-0600 INFO Telemetry enabled {“Process”: “storagenode”, “instance ID”: “1g8hm7zCcxx1csw9qadSfv7XY2mmDSi8DJSWRPj4D6DGR2Y4wo”}
2023-02-01T13:19:21.823-0600 INFO Event collection enabled {“Process”: “storagenode”, “instance ID”: “1g8hm7zCcxx1csw9qadSfv7XY2mmDSi8DJSWRPj4D6DGR2Y4wo”}
2023-02-01T13:19:21.854-0600 INFO db.migration Database Version {“Process”: “storagenode”, “version”: 54}
2023-02-01T13:19:22.252-0600 INFO preflight:localtime start checking local system clock with trusted satellites’ system clock. {“Process”: “storagenode”}
2023-02-01T13:19:23.258-0600 INFO preflight:localtime local system clock is in sync with trusted satellites’ system clock. {“Process”: “storagenode”}
2023-02-01T13:19:23.259-0600 INFO bandwidth Performing bandwidth usage rollups {“Process”: “storagenode”}
2023-02-01T13:19:23.260-0600 INFO Node 1g8hm7zCcxx1csw9qadSfv7XY2mmDSi8DJSWRPj4D6DGR2Y4wo started {“Process”: “storagenode”}
2023-02-01T13:19:23.260-0600 INFO Public server started on [::]:28969 {“Process”: “storagenode”}
2023-02-01T13:19:23.260-0600 INFO Private server started on 127.0.0.1:7780 {“Process”: “storagenode”}
2023-02-01T13:19:23.261-0600 INFO trust Scheduling next refresh {“Process”: “storagenode”, “after”: “3h30m19.494718856s”}
2023-02-01T13:19:23.261-0600 INFO pieces:trash emptying trash started {“Process”: “storagenode”, “Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”}
Restart Cycle
