The v0 mention appears only on warn level? So I can’t verify it at error level… anyway, I’ll keep an eye on logs. If the errors muliply, I’ll switch to warn level.
It should be right on the error level with the error together.
Din’t find v0 mentioned anywhere. I think I’m good. These are all the errors, one on each machine, but not on all machines:
As you can see above in my comment, v0’s are also at warn level. That’s why I asked what he logs, because there could be more to it.
they should be stop after a while. If not, you have a workaround:
But not in the error, so likely should not be the case. However, this warn is about a corrupted piece.
But as his Errors are v0 free, like mine, i thought it’s maybe the same and he just don’t see them.
In any case, our errors are identical and different than the ones the topic started with.
1 Like
I also got a log like this today:
{L:WARN,T:2023-11-01T16:23:25Z,N:pieces,M:failed to migrate v0 piece. Piece may not be recoverable,process:storagenode}
{L:WARN,T:2023-11-01T16:23:25Z,N:retain,M:failed to delete piece,process:storagenode,Satellite ID:12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S,Piece ID:QS52GRAG5U6V2A4TTILSBTJSEUDBIJQF3F7D6Y2YNK37IMA3QI4Q,error:pieces error: filestore error: file does not exist,errorVerbose:pieces error: filestore error: file does not exist
storj.io/storj/storagenode/blobstore/filestore.(blobStore).Stat:110
storj.io/storj/storagenode/pieces.(BlobsUsageCache).pieceSizes:245
storj.io/storj/storagenode/pieces.(BlobsUsageCache).Trash:290
storj.io/storj/storagenode/pieces.(Store).Trash:404
storj.io/storj/storagenode/retain.(Service).trash:364
storj.io/storj/storagenode/retain.(Service).retainPieces:341
storj.io/storj/storagenode/retain.(Service).Run.func2:221
golang.org/x/sync/errgroup.(Group).Go.func1:75}
checked the uploaded logs:
{"L":"INFO","T":"2023-10-24T15:22:31Z","N":"piecestore","M":"upload started","process":"storagenode","Piece ID":"QS52GRAG5U6V2A4TTILSBTJSEUDBIJQF3F7D6Y2YNK37IMA3QI4Q","Satellite ID":"12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S","Action":"PUT","Available Space":1106730333321,"Remote Address":"172.18.0.7:40196"}
{"L":"INFO","T":"2023-10-24T15:22:32Z","N":"piecestore","M":"uploaded","process":"storagenode","Piece ID":"QS52GRAG5U6V2A4TTILSBTJSEUDBIJQF3F7D6Y2YNK37IMA3QI4Q","Satellite ID":"12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S","Action":"PUT","Size":46080,"Remote Address":"172.18.0.7:40196"}
also checked the filesystem with fsck. no errors.
any advices?
your node lost that piece. I would suggest to check S.M.A.R.T. of this disk.
From the other side it should be removed from your node anyway, since the garbage collector wanted to move it to the trash.
Please try to search in the trash as well.
To convert PieceId to the filename you may use this method:
Do you maybe have more logs for that piece since upload?
I would suggest to check S.M.A.R.T. of this disk.
SMART is fine.
Do you maybe have more logs for that piece since upload?
no more interactions with this piece id.
Now I have a similar log on a completely different node… looks strange to me.
not so strange… If the piece is uploaded with an expiration date, it could be removed by storagenode, without a command from the satellite.
However, it looks weird, if it’s removed the piece and then the Garbage Collector coming in with the request to remove it.
Do you have an evidence to confirm?
I did a cat/grep of the logs with piece id. All logs have already been posted. Log level is set to info and logs are kept “forever”.
what should I look for?
Your log should show something like
INFO collector deleted expired piece …Piece ID: your piece id
shows the piece was deleted after it got expired.
1 Like
Did you check the file with this piece? Does it exist in the blobs or trash folder?
i did check that.
It was not in trash nor in blobs.
1 Like
Then it can only confirm that we have a several bugs there.
See
Then it can only confirm that we have a several bugs there.
as long as the audit score does not weaken - it’s not a problem for me.
It should not as far as I know.