Piecedeleter could not send delete piece to trash operation not permitted

Node Operators troubleshooting
1

I moved to a new storage and getting a ot of those errors now.
User is storj-storagenode storj-storagenode (linux) changed also to chmod 777
any tipp for exactly to look for?

Nov 18 05:43:24 xxxxxxxxxxxxx storagenode[50165]: 2022-11-18T05:43:24.163Z        ERROR        piecedeleter        could not send delete piece to trash        
{"Process": "storagenode", "Satellite ID": "xxxxxxxxxx", "Piece ID": "xxxxxxxxxxxxxxxx", 
"error": "pieces error: filestore error: chtimes /media/storagenode/config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/oi/eduhxogqezdquzgcjaed27xva3lqqutqtapn5rmh7fgx5kbs3q.sj1: 
operation not permitted", "errorVerbose": "pieces error: filestore error: chtimes /media/storagenode/config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/oi/eduhxogqezdquzgcjaed27xva3lqqutqtapn5rmh7fgx5kbs3q.sj1: 
operation not permitted\n\tstorj.io/storj/storage/filestore.(*blobStore).Trash:140\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Trash:295\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:365
\n\tstorj.io/storj/storagenode/pieces.(*Deleter).deleteOrTrash:185\n\tstorj.io/storj/storagenode/pieces.
(*Deleter).work:135\n\tstorj.io/storj/storagenode/pieces.(*Deleter).Run.func1:72\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}

if I look into the trash there are new directories

root@xxx:/media/storagenode/config/storage/trash# ls -lart
total 8
drwxrwxrwx   5 storj-storagenode storj-storagenode  202 Nov 18 05:46 .
drwxrwxrwx   6 storj-storagenode storj-storagenode 4096 Nov 18 06:09 ..
drwx------  15 storj-storagenode storj-storagenode  192 Nov 18 06:10 v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa
drwx------  12 storj-storagenode storj-storagenode  150 Nov 18 06:12 qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa
drwx------ 120 storj-storagenode storj-storagenode 4096 Nov 18 06:12 ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa

maybe just hickups after node restart? At least I see also successful send to trash

2

It doesn’t have permissions to /media/storagenode/config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/oi/eduhxogqezdquzgcjaed27xva3lqqutqtapn5rmh7fgx5kbs3q.sj1, not to the trash folder.
If you uses docker and parameter --user $(id -u):$(id -g) in your docker run command, then you need to configure your docker to run without a sudo and change the owner to your user (I suppose it’s storj-storagenode) and change permissions to give a full rights to it recursively for all storage location.

3

storj is installed in a VM and running as a service as user storj-storagenode (not using docker). I mainly spotted the errors when starting the node. Will observe it for a longer period as after a while the # of errors reduced.

4

Please check permissions in the blobs folder. What’s your filesystem there? Is it a network filesystem?

5

xfs local disk. I checked some files that were reported and it had 777.

Could it be a multithreading load issue while startup phase of the node? I had the impression the numbers reported reducing when CPU load was going down (4 cores assigned and ~20-25% at startup later dropping to ~5% - very old Intel i7)

6

It should not produce the permission error.
But I suspect the xfs filesystem now…
Or the middleware to connect your disk to the VM.

7

Checked the logs - the error stopped to be created ~1h after node start.
So currently all fine - I will keep it under monitoring.