Let me add these:
2024-04-28T19:22:28Z INFO pieces:trash emptying trash started {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-04-28T19:22:28Z ERROR pieces:trash emptying trash failed {"Process": "storagenode", "error": "pieces error: filestore error: remove storage/trash/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/2024-04-13: directory not empty", "errorVerbose": "pieces error: filestore error: remove storage/trash/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/2024-04-13: directory not empty\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).EmptyTrash:189\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).EmptyTrash:319\n\tstorj.io/storj/storagenode/pieces.(*Store).EmptyTrash:431\n\tstorj.io/storj/storagenode/pieces.(*TrashChore).Run.func1.1:84\n\tstorj.io/common/sync2.(*Workplace).Start.func1:89"}
2024-04-28T19:22:28Z INFO pieces:trash emptying trash started {"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-04-28T19:22:28Z ERROR pieces:trash emptying trash failed {"Process": "storagenode", "error": "pieces error: filestore error: remove storage/trash/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/2024-04-13: directory not empty", "errorVerbose": "pieces error: filestore error: remove storage/trash/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/2024-04-13: directory not empty\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).EmptyTrash:189\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).EmptyTrash:319\n\tstorj.io/storj/storagenode/pieces.(*Store).EmptyTrash:431\n\tstorj.io/storj/storagenode/pieces.(*TrashChore).Run.func1.1:84\n\tstorj.io/common/sync2.(*Workplace).Start.func1:89"}
2024-04-28T19:22:28Z INFO pieces:trash emptying trash started {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-04-28T19:22:28Z ERROR pieces:trash emptying trash failed {"Process": "storagenode", "error": "pieces error: filestore error: remove storage/trash/qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa/2024-04-13: directory not empty", "errorVerbose": "pieces error: filestore error: remove storage/trash/qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa/2024-04-13: directory not empty\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).EmptyTrash:189\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).EmptyTrash:319\n\tstorj.io/storj/storagenode/pieces.(*Store).EmptyTrash:431\n\tstorj.io/storj/storagenode/pieces.(*TrashChore).Run.func1.1:84\n\tstorj.io/common/sync2.(*Workplace).Start.func1:89"}
2024-04-28T19:22:28Z INFO pieces:trash emptying trash started {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-04-28T19:22:28Z ERROR pieces:trash emptying trash failed {"Process": "storagenode", "error": "pieces error: filestore error: remove storage/trash/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/2024-04-13: directory not empty", "errorVerbose": "pieces error: filestore error: remove storage/trash/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/2024-04-13: directory not empty\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).EmptyTrash:189\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).EmptyTrash:319\n\tstorj.io/storj/storagenode/pieces.(*Store).EmptyTrash:431\n\tstorj.io/storj/storagenode/pieces.(*TrashChore).Run.func1.1:84\n\tstorj.io/common/sync2.(*Workplace).Start.func1:89"}
That is one node that has the nested directories in trash like: /storage/trash/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/2024-04-13/2024-04-13/2024-04-13/2a/
Obviously they do not get deleted because they are not empty as there are the nested directories in it. If it is 100% safe to delete those folders after 7 days there should be no need to check if they are empty or not.
Version was 1.99.3 and it seems it is now 1.102.3 since today or yesterday. If that version already contains a fix for that, maybe the nested folders will get removed on the next run of the trash chore. If it has not been fixed yet, then that’s the actual status.