I’m looking into something that started showing after the updates to v1.104 line. So far it’s on a couple of nodes only.
Environment:
- Linux baremetal
- Node v1.104.5
- running error logs (so no info)
The current disk usage for the trash folder of this node:
du -sh /mnt/node3/storagenode/storage/trash/
83G /mnt/node3/storagenode/storage/trash/
Current structure of trash folder:
ls /mnt/node3/storagenode/storage/trash/*
/mnt/node3/storagenode/storage/trash/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa:
2024-05-09
/mnt/node3/storagenode/storage/trash/qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa:
2024-05-10
/mnt/node3/storagenode/storage/trash/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa:
/mnt/node3/storagenode/storage/trash/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa:
2024-05-11
Relevant error log:
2024-05-12T19:09:10+03:00 ERROR db failed to stat blob in trash {"Process": "storagenode", "namespace": "ootPBOELroXWf0xsuCv41MDw9HqOpyYnUk3rbsAAAAA=", "key": "MdRT+UJr/XejttfvSMvkfUhRSQtdkfLq9/ERRLSoQds=", "error": "lstat /mnt/node3/storagenode/storage/trash/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/2024-05-03/gh/kfh6kcnp6xpi5w27xurs7epvefcsillwi7f2xx6eiujnfiihnq.sj1: no such file or directory"}
2024-05-12T19:09:10+03:00 ERROR db failed to stat blob in trash {"Process": "storagenode", "namespace": "ootPBOELroXWf0xsuCv41MDw9HqOpyYnUk3rbsAAAAA=", "key": "MfgqbrOR9m3UiCsQ0e97B+vPNOJHwPf5ear45jRMew4=", "error": "lstat /mnt/node3/storagenode/storage/trash/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/2024-05-03/gh/4cu3vtsh3g3veifmind333a7v46nhci7app6lzvl4omncmpmha.sj1: no such file or directory"}
2024-05-12T19:09:10+03:00 ERROR db failed to stat blob in trash {"Process": "storagenode", "namespace": "ootPBOELroXWf0xsuCv41MDw9HqOpyYnUk3rbsAAAAA=", "key": "McQqmNjar0bIsLCGSktZgJSeLNeTIL6YGPy5y9meZy4=", "error": "lstat /mnt/node3/storagenode/storage/trash/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/2024-05-03/gh/ccvggy3kxunsfqwcdeus2zqckj4lgxsmql5gay7s44xwm6m4xa.sj1: no such file or directory"}
2024-05-12T19:09:10+03:00 ERROR db failed to stat blob in trash {"Process": "storagenode", "namespace": "ootPBOELroXWf0xsuCv41MDw9HqOpyYnUk3rbsAAAAA=", "key": "MdF113Td8pimJ30BkhMMI73YTcfx9fhMVHqOAUlZNQ4=", "error": "lstat /mnt/node3/storagenode/storage/trash/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/2024-05-03/gh/ixlv3u3xzjrjrhpuazeeymeo65qtoh6h27qtcupkhacskzguha.sj1: no such file or directory"}
2024-05-12T19:09:10+03:00 ERROR db failed to stat blob in trash {"Process": "storagenode", "namespace": "ootPBOELroXWf0xsuCv41MDw9HqOpyYnUk3rbsAAAAA=", "key": "MdSESgeduGkBaitLFVXqDso9zYGzFjiIWBYXiiKtENQ=", "error": "lstat /mnt/node3/storagenode/storage/trash/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/2024-05-03/gh/kiisqhtw4gsalkfnfrkvpkb3fd3tmbwmldrccycylyuivncdka.sj1: no such file or directory"}
2024-05-14T17:30:43+03:00 ERROR blobscache trashTotal < 0 {"Process": "storagenode", "trashTotal": -27588549910}
2024-05-14T17:31:25+03:00 ERROR blobscache trashTotal < 0 {"Process": "storagenode", "trashTotal": -1079187456}
2024-05-14T18:50:59+03:00 ERROR blobscache trashTotal < 0 {"Process": "storagenode", "trashTotal": -22135780828}
Disk fs ok.
My comments: I think the missing file errors may have been due to used-space running together with trash cleanup (technically trash cleanup got triggered while used-space was still running).
Update: On another node that had this issue, the trash got updated, so the error seems to suggest that the trash cleanup is trying to remove a value from the database that is too large?
