Huge amount of errors!

naxbc · January 27, 2020, 4:40pm

Guys, what´s this I´m checking?

Preformatted text|2020-01-27T16:38:14.825Z|WARN|retain|failed to delete piece|{“Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Piece ID”: “T6U3KA3GWWHLIYALW6HUVZAV4CAELPQ7BOLM6IRFZWRR42F5DIYQ”, “error”: “pieces error: file does not exist”, “errorVerbose”: “pieces error: file does not exist\n\tstorj.io/storj/storagenode/pieces.(*Store).MigrateV0ToV1:392\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:304\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces.func1:388\n\tstorj.io/storj/storagenode/storagenodedb.(*v0PieceInfoDB).WalkSatelliteV0Pieces:108\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:468\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces:364\n\tstorj.io/storj/storagenode/retain.(*Service).Run.func2:220\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}|
|—|---|—|---|—|
|2020-01-27T16:38:14.832Z|WARN|retain|failed to delete piece|{“Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Piece ID”: “T6YSIYAETLIJ6HCJ3E4GF2AK4774SQQNNL4BND4IXBKQT3XH676A”, “error”: “pieces error: file does not exist”, “errorVerbose”: “pieces error: file does not exist\n\tstorj.io/storj/storagenode/pieces.(*Store).MigrateV0ToV1:392\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:304\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces.func1:388\n\tstorj.io/storj/storagenode/storagenodedb.(*v0PieceInfoDB).WalkSatelliteV0Pieces:108\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:468\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces:364\n\tstorj.io/storj/storagenode/retain.(*Service).Run.func2:220\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}|
|2020-01-27T16:38:15.779Z|INFO|piecestore|download started|{“Piece ID”: “MZRQJ4OJV2PODEGAMNU6QGEIMFEMPFLZEMHBMKJY4RYMAU2ZO4JQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”}|
|2020-01-27T16:38:15.952Z|WARN|retain|failed to delete piece|{“Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Piece ID”: “VZXBHORAR5CW4MB2QQHCTCANA4YRNTLNPYNLBY2V7VZQ4SYCLQIQ”, “error”: “pieces error: file does not exist”, “errorVerbose”: “pieces error: file does not exist\n\tstorj.io/storj/storagenode/pieces.(*Store).MigrateV0ToV1:392\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:304\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces.func1:388\n\tstorj.io/storj/storagenode/storagenodedb.(*v0PieceInfoDB).WalkSatelliteV0Pieces:108\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:468\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces:364\n\tstorj.io/storj/storagenode/retain.(*Service).Run.func2:220\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}|
|2020-01-27T16:38:16.098Z|WARN|retain|failed to delete piece|{“Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Piece ID”: “WVQCZ3HCNGLRA632AQYSBDTZZJMSVPK6EPA4OYVMMJU5BXXLIPIA”, “error”: “pieces error: file does not exist”, “errorVerbose”: “pieces error: file does not exist\n\tstorj.io/storj/storagenode/pieces.(*Store).MigrateV0ToV1:392\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:304\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces.func1:388\n\tstorj.io/storj/storagenode/storagenodedb.(*v0PieceInfoDB).WalkSatelliteV0Pieces:108\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:468\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces:364\n\tstorj.io/storj/storagenode/retain.(*Service).Run.func2:220\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}|
|2020-01-27T16:38:16.131Z|WARN|retain|failed to delete piece|{“Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Piece ID”: “W6W5IZWATC3GTC6UBX7HG6IBRKKUTKR5YSWB6XMAR6EZEOKW6MRA”, “error”: “pieces error: file does not exist”, “errorVerbose”: “pieces error: file does not exist\n\tstorj.io/storj/storagenode/pieces.(*Store).MigrateV0ToV1:392\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:304\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces.func1:388\n\tstorj.io/storj/storagenode/storagenodedb.(*v0PieceInfoDB).WalkSatelliteV0Pieces:108\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:468\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces:364\n\tstorj.io/storj/storagenode/retain.(*Service).Run.func2:220\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}|
|2020-01-27T16:38:16.711Z|INFO|piecestore|download failed|{“Piece ID”: “MZRQJ4OJV2PODEGAMNU6QGEIMFEMPFLZEMHBMKJY4RYMAU2ZO4JQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”, “error”: “piecestore: piecestore protocol: context canceled”, “errorVerbose”: “piecestore: piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func3:653\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}|
|2020-01-27T16:38:17.024Z|WARN|retain|failed to delete piece|{“Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Piece ID”: “XHB34JNNDGR5CO5SY77BOFSEUTCBNGU6YOTIABZVUY6FRJO4FK7A”, “error”: “pieces error: file does not exist”, “errorVerbose”: “pieces error: file does not exist\n\tstorj.io/storj/storagenode/pieces.(*Store).MigrateV0ToV1:392\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:304\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces.func1:388\n\tstorj.io/storj/storagenode/storagenodedb.(*v0PieceInfoDB).WalkSatelliteV0Pieces:108\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:468\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces:364\n\tstorj.io/storj/storagenode/retain.(*Service).Run.func2:220\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}|
|2020-01-27T16:38:17.085Z|WARN|retain|failed to delete piece|{“Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Piece ID”: “X4RZXJD2LX3GGJN3HSAZ6DMPBHVHQG5OT4MFS5XFJLLZE7DA6PGQ”, “error”: “pieces error: file does not exist”, “errorVerbose”: “pieces error: file does not exist\n\tstorj.io/storj/storagenode/pieces.(*Store).MigrateV0ToV1:392\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:304\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces.func1:388\n\tstorj.io/storj/storagenode/storagenodedb.(*v0PieceInfoDB).WalkSatelliteV0Pieces:108\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:468\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces:364\n\tstorj.io/storj/storagenode/retain.(*Service).Run.func2:220\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}|
|2020-01-27T16:38:17.110Z|WARN|retain|failed to delete piece|{“Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Piece ID”: “X6YGTZXRQG2LG3Y5ZJLSSNFSOMHEES47SW57X4II5ON535H5G6PA”, “error”: “pieces error: file does not exist”, “errorVerbose”: “pieces error: file does not exist\n\tstorj.io/storj/storagenode/pieces.(*Store).MigrateV0ToV1:392\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:304\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces.func1:388\n\tstorj.io/storj/storagenode/storagenodedb.(*v0PieceInfoDB).WalkSatelliteV0Pieces:108\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:468\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces:364\n\tstorj.io/storj/storagenode/retain.(*Service).Run.func2:220\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}|Preformatted text

nerdatwork · January 27, 2020, 4:43pm

Did you manually delete files from the trash folder ?

naxbc · January 27, 2020, 4:44pm

Absolutely not. This is my oldest node, online since May 2019.

And it´s only on delete operation and on @stefanbenten satellite :

2020-01-27T16:38:16.131Z WARN retain failed to delete piece {“Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Piece ID”: “W6W5IZWATC3GTC6UBX7HG6IBRKKUTKR5YSWB6XMAR6EZEOKW6MRA”, “error”: “pieces error: file does not exist”, “errorVerbose”: “pieces error: file does not exist\n\tstorj.io/storj/storagenode/pieces.(*Store).MigrateV0ToV1:392\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:304\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces.func1:388\n\tstorj.io/storj/storagenode/storagenodedb.(*v0PieceInfoDB).WalkSatelliteV0Pieces:108\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:468\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces:364\n\tstorj.io/storj/storagenode/retain.(*Service).Run.func2:220\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

nerdatwork · January 27, 2020, 5:06pm

Here’s what I understand so far. Its a warning not an error that would negatively affect you. Retain service is run for garbage collection & seems its trying to delete something which is already deleted.

I have these errors too & when I search my log for the piece id I see as many as 5 attempts.

naxbc · January 27, 2020, 5:09pm

Ufff . That’s a relief. Thank you.

flo · January 28, 2020, 6:31am

Getting exactly the same - these errors appear regularly and don’t go away.

A bug that needs to be fixed IMHO.

naxbc · January 28, 2020, 8:11am

As @nerdatwork mentioned above, it’s not errors, it’s warning, and long as everything is ok on my side, good.
We’re still in Beta and this kinda things are “expected”.
As long as in Production things are smoother, cool.

flo · January 31, 2020, 1:27pm

Still getting this in v0.31.12. Since it keeps reappearing for the same pieces, my logfile grows to several gigabytes full of those errors.

Would be nice if @littleskunk (or somebody else from the dev team) could fix this.

naxbc · January 31, 2020, 1:32pm

There was a procedure to limit log file size. Search for Docker post installation steps.

littleskunk · January 31, 2020, 1:34pm

There is no need to fix it. Just let it run.

flo · January 31, 2020, 2:04pm

So why exactly am I seeing the same errors over and over again?

BrightSilence · January 31, 2020, 2:14pm

I’ll repeat it again, they are NOT errors. They are warnings.
I think this is garbage collection or cleanup of zombie segments trying to clean up pieces that were already removed from your node. It’s not a problem, they shouldn’t be on your node and they aren’t on your node. All good.

flo · January 31, 2020, 3:16pm

Can we agree that it makes zero sense to try to delete the same pieces over and over again?

littleskunk · January 31, 2020, 3:18pm

No we can’t. We could remove the message but it wouldn’t change the behavior.

BrightSilence · January 31, 2020, 3:20pm

I don’t see any evidence that it’s the same piece over and over again. All logs show different piece ID’s. I’m not going to agree to irrelevant hypotheticals.

nerdatwork · January 31, 2020, 4:58pm

Not a hypothetical