ERROR collector unable to delete piece

barroso1994 · April 9, 2021, 3:02am

I’m getting this error in the log only on that “Piece ID”: “NCUUF5ALPEVEZED7CUM2ZL7QKUPGT3BLS2R4ZFIKDM5P26TRH2BA”

2021-04-07T01:09:28.106-0300 ERROR collector unable to delete piece {“Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Piece ID”: “NCUUF5ALPEVEZED7CUM2ZL7QKUPGT3BLS2R4ZFIKDM5P26TRH2BA”, “error”: “pieces error: filestore error: file does not exist”, “errorVerbose”: “pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:99\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:298\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:81\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:80\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

some solution to that?

Pac · April 10, 2021, 9:47pm

Does this suggest one file was lost on your disk?
Are your scores looking good?
Does your node appear to have normal activity?

barroso1994 · July 21, 2021, 7:02pm

it was after a migration…

how can i fix this?

barroso1994 · July 21, 2021, 7:08pm

2021-07-20T21:21:35.345-0300	ERROR	collector	unable to delete piece	{Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Piece ID: VH3WB36T5EX5IK3VYJQN34RR76DLXV77EETG7BGOEE5RQL4XT4DQ, error: pieces error: filestore error: file does not exist, errorVerbose: pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(Service).Run.func1:57\n\tstorj.io/common/sync2.(Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(Service).Run:53\n\tstorj.io/storj/private/lifecycle.(Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57}
2021-07-20T21:21:35.560-0300	ERROR	collector	unable to delete piece	{Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Piece ID: Z2DRMNZSLDBQEX5UPU2SNDTY2BFG6RGDKJ4UVGISJ56HIQGKPEMA, error: pieces error: filestore error: file does not exist, errorVerbose: pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(Service).Run.func1:57\n\tstorj.io/common/sync2.(Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(Service).Run:53\n\tstorj.io/storj/private/lifecycle.(Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57}
2021-07-20T21:21:35.605-0300	ERROR	collector	unable to delete piece	{Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Piece ID: UTPCI62T4RBWOGZ7F3BPHIRF5X5F6X6M3I5QERBNUB26L45ZVFVQ, error: pieces error: filestore error: file does not exist, errorVerbose: pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(Service).Run.func1:57\n\tstorj.io/common/sync2.(Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(Service).Run:53\n\tstorj.io/storj/private/lifecycle.(Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57}
2021-07-20T21:21:35.713-0300	ERROR	collector	unable to delete piece	{Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Piece ID: I3OF2UASQM35AJCEWKZRHMUXGVGCXNZVLHGUF3TVL5FWQXIAYPZQ, error: pieces error: filestore error: file does not exist, errorVerbose: pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(Service).Run.func1:57\n\tstorj.io/common/sync2.(Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(Service).Run:53\n\tstorj.io/storj/private/lifecycle.(Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57}
2021-07-20T21:21:35.894-0300	ERROR	collector	unable to delete piece	{Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Piece ID: YWE4MT27BNCGW2EIET2Q3I7CKRNDC3ZUYFOSDGT6JJ2XNACALQFA, error: pieces error: filestore error: file does not exist, errorVerbose: pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(Service).Run.func1:57\n\tstorj.io/common/sync2.(Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(Service).Run:53\n\tstorj.io/storj/private/lifecycle.(Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57}
2021-07-20T21:21:36.149-0300	ERROR	collector	unable to delete piece	{Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Piece ID: 4YMBI2JTOND3TLV7PF6LYOWT6DAKWGWBOAG6QOK4NN5WK3AIH54A, error: pieces error: filestore error: file does not exist, errorVerbose: pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(Service).Run.func1:57\n\tstorj.io/common/sync2.(Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(Service).Run:53\n\tstorj.io/storj/private/lifecycle.(Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57}
2021-07-20T21:21:36.240-0300	ERROR	collector	unable to delete piece	{Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Piece ID: YW3ENT2MIOXGA3KCRCMFV4WKVVSNKTHUFACFSTCS5SBISTBKR3AA, error: pieces error: filestore error: file does not exist, errorVerbose: pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(Service).Run.func1:57\n\tstorj.io/common/sync2.(Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(Service).Run:53\n\tstorj.io/storj/private/lifecycle.(Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57}
2021-07-20T21:21:36.306-0300	ERROR	collector	unable to delete piece	{Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Piece ID: YWL4PTK5DRJWWMGSBTEV6QEZSQEJYS2EQQ3J4PN377K5WNPXHWPA, error: pieces error: filestore error: file does not exist, errorVerbose: pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(Service).Run.func1:57\n\tstorj.io/common/sync2.(Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(Service).Run:53\n\tstorj.io/storj/private/lifecycle.(Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57}
2021-07-20T21:21:36.485-0300	ERROR	collector	unable to delete piece	{Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Piece ID: NDLSEVODN2R6CLGDUVPXZXTZNS2BZDBQKJJ642GRKI36DCUD575Q, error: pieces error: filestore error: file does not exist, errorVerbose: pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(Service).Run.func1:57\n\tstorj.io/common/sync2.(Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(Service).Run:53\n\tstorj.io/storj/private/lifecycle.(Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57}
2021-07-20T21:21:36.652-0300	ERROR	collector	unable to delete piece	{Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Piece ID: R7SJKHWYY274NBSIMHMCFGNU7KYD65VTQYNKCJ6J6ZGPFVBI6FSA, error: pieces error: filestore error: file does not exist, errorVerbose: pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(Service).Run.func1:57\n\tstorj.io/common/sync2.(Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(Service).Run:53\n\tstorj.io/storj/private/lifecycle.(Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57}
2021-07-20T21:21:36.703-0300	ERROR	collector	unable to delete piece	{Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Piece ID: I363Y6QZW54FKQ2TWYYQT2SPLTZYJQ5H5X2GTCTQQA636JL2KHVQ, error: pieces error: filestore error: file does not exist, errorVerbose: pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(Service).Run.func1:57\n\tstorj.io/common/sync2.(Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(Service).Run:53\n\tstorj.io/storj/private/lifecycle.(Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57}
2021-07-20T21:21:36.926-0300	ERROR	collector	unable to delete piece	{Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Piece ID: FRJGXEF52MAXHCIHTTDSBJSEZ6VPMDQQW6L7ZXP2JYKASD37PUSQ, error: pieces error: filestore error: file does not exist, errorVerbose: pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(Service).Run.func1:57\n\tstorj.io/common/sync2.(Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(Service).Run:53\n\tstorj.io/storj/private/lifecycle.(Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57}

Code_Breaker · July 21, 2021, 9:39pm

Hello, I hope you are doing well. This looks like garbage collection is attempting to collect a file that may have already been collected or purged from the disk somehow. I believe I saw a similar post recently where the user had come up with an easy work around. I have posted a link to the post below so you could get some more context, but essentially they verified that the file did not exist in the location specified and then they simply created a file by using the touch command. If you find that your file exists in the location specified in log file then I would check the permissions on the file and verify it matches the permissions of the other files in the directory.

I hope this helps.

Warm Regards,

Antonio

barroso1994 · July 21, 2021, 9:49pm

I saw this post, but I don’t know how to do this on windows. do you have any windows tutoring?

Code_Breaker · July 21, 2021, 10:16pm

If you are working from the command line you can execute the command below replacing file.txt with the filename of the file you wish to create.

echo.> file.txt

You can also easily do this using windows explorer.
Open the folder → Right click in folder → Hover over new → select Text Document. This should create a new blank file waiting to be named. Just name it to match the missing file.

Let me know if this doesn’t work for you and I can explore other potential issues.

barroso1994 · July 21, 2021, 10:29pm

but in which folder should I put the created file?

exempl

barroso1994 · July 21, 2021, 10:56pm

in which folder do i see this piece?

Piece ID: FRJGXEF52MAXHCIHTTDSBJSEZ6VPMDQQW6L7ZXP2JYKASD37PUSQ

Code_Breaker · July 21, 2021, 11:07pm

I am looking into what the path to the blob file is currently. I’ll get back to you as soon as possible.

BrightSilence · July 22, 2021, 10:16am

Each satellite has their own folder in the blobs folder in your storage location. For the satellite this piece is on that’s v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa. The subfolders you see inside there represent the first 2 characters of the piece id. So in your case the folder should be fr. The filename inside that folder is the rest of the piece id with a .sj1 extension. So jgxef52maxhcihttdsbjsez6vpmdqqw6l7zxp2jykasd37pusq.sj1. Path and filenames are in lower case, which is a little annoying since they are logged in upper case. On windows systems this may not matter, but linux/unix systems are case sensitive.

Entire path should be:
[StorjDataLocation]\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\fr\jgxef52maxhcihttdsbjsez6vpmdqqw6l7zxp2jykasd37pusq.sj1

Please replace the [StorjDataLocation] part with where your storj data is stored.

Last note: If you use explorer to create the file as @Code_Breaker mentioned. Make sure you have displaying of known file extensions switched on. Otherwise you may unknowingly be naming your file jgxef52maxhcihttdsbjsez6vpmdqqw6l7zxp2jykasd37pusq.sj1.txt.

It may be easier to just run this in CMD:

echo.> [StorjDataLocation]\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\fr\jgxef52maxhcihttdsbjsez6vpmdqqw6l7zxp2jykasd37pusq.sj1

Again, please replace the [StorjDataLocation] part with where your storj data is stored.

From the logs I see that you’ve got multiple pieces that have this issue though. So you will need to do this for all of them. You can use https://convertcase.net/ to convert the upper case piece id’s to lower case.

BrightSilence · July 22, 2021, 10:19am

Edit: This list may now be outdated, latest list is maintained here:

Original message for reference

In the logs you posted it seems to have all been for the same satellites. If you have the same issue on other satellites, here’s a reference for satellites and corresponding ID’s and blobs folders.

Satellite Name: satellite.stefan-benten.de
Satellite ID: 118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW
Satellite ID hex: 004AE89E970E703DF42BA4AB1416A3B30B7E1D8E14AA0E558F7EE26800000000
Blobs Folder: abforhuxbzyd35blusvrifvdwmfx4hmocsva4vmpp3rgqaaaaaaa

Satellite Name: ap1.storj.io (formerly: asia-east-1.tardigrade.io)
Satellite ID: 121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6
Satellite ID hex: 84A74C2CD43C5BA76535E1F42F5DF7C287ED68D33522782F4AFABFDB40000000
Blobs Folder: qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa

Satellite Name: us1.storj.io (formerly: us-central-1.tardigrade.io)
Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S
Satellite ID hex: A28B4F04E10BAE85D67F4C6CB82BF8D4C0F0F47A8EA72627524DEB6EC0000000
Blobs Folder: ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa

Satellite Name: eu1.storj.io (formerly: europe-west-1.tardigrade.io)
Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs
Satellite ID hex: AF2C42003EFC826AB4361F73F9D890942146FE0EBE806786F8E7190800000000
Blobs Folder: v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa

Satellite Name: saltlake.tardigrade.io
Satellite ID: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE
Satellite ID hex: 7B2DE9D72C2E935F1918C058CAAF8ED00F0581639008707317FF1BD000000000
Blobs Folder: pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa

Satellite Name: europe-north-1.tardigrade.io
Satellite ID: 12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB
Satellite ID hex: F474535A19DB00DB4F8071A1BE6C2551F4DED6A6E38F0818C68C68D000000000
blobs folder: 6r2fgwqz3manwt4aogq343bfkh2n5vvg4ohqqgggrrunaaaaaaaa

Satellite Name: us2.storj.io (formerly: us2.tardigrade.io)
Satellite ID: 12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo
Satellite ID hex: 04489F5245DED48D2A8AC8FB5F5CD1C6A638F7C6E75EFD800EF2D72000000000
blobs folder: arej6usf33ki2kukzd5v6xgry2tdr56g45pp3aao6llsaaaaaaaa

barroso1994 · July 22, 2021, 11:53am

thank you very much…resolved. Code_Breaker, BrightSilence.

barroso1994 · July 22, 2021, 12:32pm

it was just this new error = could not send delete piece to trash

What to do in this case?

2021-07-22T09:18:38.418-0300 ERROR piecedeleter could not send delete piece to trash {“Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Piece ID”: “IXZSEWGXS7QNUD6YJBQ672VQZYWE7462NYEVZBYRJZUNH7HHX57Q”, “error”: “pieces error: v0pieceinfodb: sql: no rows in result set”, “errorVerbose”: “pieces error: v0pieceinfodb: sql: no rows in result set\n\tstorj.io/storj/storagenode/storagenodedb.(*v0PieceInfoDB).Get:131\n\tstorj.io/storj/storagenode/pieces.(*Store).MigrateV0ToV1:404\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:348\n\tstorj.io/storj/storagenode/pieces.(*Deleter).deleteOrTrash:185\n\tstorj.io/storj/storagenode/pieces.(*Deleter).work:135\n\tstorj.io/storj/storagenode/pieces.(*Deleter).Run.func1:72\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

BrightSilence · July 22, 2021, 1:08pm

These should be a one off. It only tries to do regular deletes once. So there isn’t really a need to fix this.

But these errors do suggest you may have had file loss. Do you have other errors in the logs as well? Most importantly in lines with GET_AUDIT or GET_REPAIR in them. But it’s worth posting any type of error you encounter.

jammerdan · August 24, 2021, 6:50am

@Alexey
Maybe the solution can be made a topic in a FAQ category? Something like “Which data folder belongs to which satellite?” or something like that?

jammerdan · September 15, 2021, 3:00pm

I see another error exclusively on the EU1 satellite:

ERROR   collector       unable to delete piece{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "FIJ57I2RNE4SKNKA7BTKBUSGWCGVOXXN53OMF4WDR57S5AXAAPBQ", "error": "pieces error: filestore error: file does not exist"

Alexey · September 16, 2021, 6:19am

Please, try to search for this piece in your logs to see a history - when it was removed (or maybe lost)

jammerdan · September 16, 2021, 7:09am

I only see log entries with the request to delete. Earlier logs have been truncated already.
What puzzles me is that this error is on the EU satellite only.

Alexey · September 16, 2021, 7:26am

Unlikely. We need to have a history for this piece to make such a claim.
Until than - the node has lost this piece for some reason. We need to figure out this reason. The good news - your node is lucky because the customer has deleted a piece before it was audited.