Disk usage discrepancy?

Hello, Quick question,
I had to restart my storj services to update the dashboard address and to update the storage size
This Drive just has Storj installed nothing else is installed and used space is 5.78


i upgraded Total disk space to 6.75 TB from 6.5 so i will still have around 500 Gb of Free space
but now on the dashboard i m getting this

Used Space 4.76TB instead of 5.78
image

Can someone please advise on this
Thanks

After a graceful reboot, the node dashboard reported a lower used space value (5.75TB) then a refreshed reported amount (5.94TB) a few minutes prior to reboot. Is this correct? Seems strange to drop 190GB is such a short time period.

Storj v1.104.5
Windows 10

Do yourself a favour and disable Indexing on that drive. Uncheck that box, click OK and Ignore all if you get an error message.

1 Like

This is mean that your databases are not updated due to a database errors and/or failed a filewalker.

Hello,

I am trying to figure out why there is a discrepancy here:

I tried searching the logs but found no errors or failed messages. What I did notice is that there are no “used space walker” or similar messages. I can only find the “lazy garbage walker” messages.

How can I check if the other walkers have finished or even started running?

I have to add that these issues are occurring across seven nodes. I also turned on the walker at node startup and restarted the nodes, but there was no difference either.

My system reports the Blobs folder at the right amount. (I checked the worsted node with a 2Tb HDD. It has a discrepancy of about 500GB.)

I just saw this on one node:

All other nodes show all 0% on reputation. its just this one. What could cause this?

Because its a average between 13 and 21Tb.
Some of the 21tb are not for the whole month.
Used space may always be bigger than last peak while growing the node.

Thats why i marked yesterday. It reported for yesterday 17.5Tb but i am over 20 for longer than that. Or is that normal too? That the average for the month is marked at 13 is ok for me, because there was quite a big grow this month. But a day differenc of almost 4Tb looks quite big to me.

17.5TB is reported from satellite, this is what you get paid for. The difference is uncollected trash.

I found this Problem. The HDD on this node is Failing. RIP this HDD after about 24k Hours run time. Will let it run until its completly dead.

The other thing is still unresolved I think i will wait till the End of next week and come back if its still there or got worse. Thank you.

1 Like

so what should i do restart the computer ?
Also as some said untick the allow files on this drive to have contents indexed in addition to file properties ?


Depending on filewalks etc sat reports and stuff, mine is looking quite same.
also we see the batman curve everywhere.

The Batman is everywhere… :shushing_face:

1 Like

Haha, thats a good one :joy:
But i know, that this is a reporting error. I was just wondering, why the reporting of the day was that far off. But as long as its working, its fine for me

You need to search for errors related to a database or a filewalker. Depending on what error do you have we can suggest a solution.
Restarting the computer or reinstalling the node likely will not help, because likely you need to optimize the disk subsystem instead.
Since you use Windows and NTFS, you need to do:

  1. Check and fix errors on the disk
  2. NTFS Disable 8dot3name
  3. [Solved] Win10 20GB Ram Usage - #17 by arrogantrabbit
  4. defragment the disk
  5. If you have a managed UPS, then you may also enable a write cache for the disk (both checkboxes), however, if you do not have a managed UPS, you should not enable it (especially the second checkbox).

The next step would be to search errors in your logs with PowerShell:

sls error "$env:ProgramFiles\Storj\Storage Node\storagenode.log" | sls "database|used-space" | select -last 10

Hi, I noticed recently that the drive I have my node on was getting close to being full. Upon investigation I realized that my Node is reporting 23.76tb used with Trash at 240.31gb. If this were true, my drive which has around double that amount shouldn’t be full. When I look at the properties of the Storj folder it says size = 30.3tb and size on disk 37.4tb. It has been this way for over a week now. Why isn’t the garbage getting cleaned up properly and why is way more being used than Storj is reporting? I have attached a screen shot as well.

Thread merge in 3… 2… 1…

2 Likes

We should open a topic that deals with frequently asked questions, but instead titled “Daily Asked Questions”.

1 Like

Hello @bdg0296,
Welcome back!
I do not know, why the search didn’t work for you, but:

So I found the following errors in my logs for file walker how do I resolve this?

Search "lazyfile" (97 hits in 1 file of 1 searched) [Normal]
  C:\Program Files\Storj\Storage Node\storagenode.log (97 hits)
	Line       16: 2024-06-28T11:56:26-04:00	INFO	lazyfilewalker.used-space-filewalker	starting subprocess	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
	Line       17: 2024-06-28T11:56:26-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	starting subprocess	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
	Line       21: 2024-06-28T11:56:26-04:00	INFO	lazyfilewalker.used-space-filewalker	subprocess started	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
	Line       22: 2024-06-28T11:56:26-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess started	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
	Line       23: 2024-06-28T11:56:26-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker started	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode", "dateBefore": "2024-06-21T11:56:26-04:00"}
	Line       24: 2024-06-28T11:56:26-04:00	INFO	lazyfilewalker.used-space-filewalker.subprocess	Database started	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}
	Line       25: 2024-06-28T11:56:26-04:00	INFO	lazyfilewalker.used-space-filewalker.subprocess	used-space-filewalker started	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}
	Line       26: 2024-06-28T11:56:26-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	Database started	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode"}
	Line  7506193: 2024-06-29T02:33:06-04:00	INFO	lazyfilewalker.gc-filewalker	starting subprocess	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
	Line  7506194: 2024-06-29T02:33:06-04:00	INFO	lazyfilewalker.gc-filewalker	subprocess started	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
	Line  7506336: 2024-06-29T02:33:27-04:00	INFO	lazyfilewalker.gc-filewalker.subprocess	Database started	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode"}
	Line  7506337: 2024-06-29T02:33:27-04:00	INFO	lazyfilewalker.gc-filewalker.subprocess	gc-filewalker started	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode", "createdBefore": "2024-06-25T17:59:59Z", "bloomFilterSize": 719420}
	Line  7557679: 2024-06-29T04:46:57-04:00	INFO	lazyfilewalker.gc-filewalker	starting subprocess	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
	Line  7557680: 2024-06-29T04:46:58-04:00	INFO	lazyfilewalker.gc-filewalker	subprocess started	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
	Line  7557851: 2024-06-29T04:47:17-04:00	INFO	lazyfilewalker.gc-filewalker.subprocess	Database started	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode"}
	Line  7557852: 2024-06-29T04:47:17-04:00	INFO	lazyfilewalker.gc-filewalker.subprocess	gc-filewalker started	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode", "createdBefore": "2024-06-25T17:59:59Z", "bloomFilterSize": 3377571}
	Line  7614470: 2024-06-29T07:05:06-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker completed	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "bytesDeleted": 43593223936, "numKeysDeleted": 120140, "Process": "storagenode"}
	Line  7614472: 2024-06-29T07:05:06-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess finished successfully	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
	Line  7614475: 2024-06-29T07:05:06-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	starting subprocess	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
	Line  7614476: 2024-06-29T07:05:06-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess started	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
	Line  7614477: 2024-06-29T07:05:06-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker started	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode", "dateBefore": "2024-06-22T07:05:06-04:00"}
	Line  7614525: 2024-06-29T07:05:14-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	Database started	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode"}
	Line  7614526: 2024-06-29T07:05:14-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker completed	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "numKeysDeleted": 0, "Process": "storagenode", "bytesDeleted": 0}
	Line  7614527: 2024-06-29T07:05:14-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess finished successfully	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
	Line  7614530: 2024-06-29T07:05:14-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	starting subprocess	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
	Line  7614531: 2024-06-29T07:05:14-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess started	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
	Line  7614532: 2024-06-29T07:05:14-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker started	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "dateBefore": "2024-06-22T07:05:14-04:00", "Process": "storagenode"}
	Line  7614533: 2024-06-29T07:05:14-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	Database started	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}
	Line  7614536: 2024-06-29T07:05:15-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker completed	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode", "bytesDeleted": 0, "numKeysDeleted": 0}
	Line  7614537: 2024-06-29T07:05:15-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess finished successfully	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
	Line  7614540: 2024-06-29T07:05:15-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	starting subprocess	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
	Line  7614541: 2024-06-29T07:05:15-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess started	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
	Line  7614542: 2024-06-29T07:05:15-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker started	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode", "dateBefore": "2024-06-22T07:05:15-04:00"}
	Line  7614543: 2024-06-29T07:05:15-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	Database started	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode"}
	Line  8354071: 2024-06-29T11:01:06-04:00	ERROR	lazyfilewalker.gc-filewalker.subprocess	failed to save progress in the database	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode", "error": "gc_filewalker_progress_db: database is locked", "errorVerbose": "gc_filewalker_progress_db: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*gcFilewalkerProgressDB).Store:33\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePiecesToTrash.func2:188\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces.func1:66\n\tstorj.io/storj/storagenode/blobstore/filestore.walkNamespaceWithPrefix:999\n\tstorj.io/storj/storagenode/blobstore/filestore.(*Dir).walkNamespaceUnderPath:897\n\tstorj.io/storj/storagenode/blobstore/filestore.(*Dir).walkNamespaceInPath:842\n\tstorj.io/storj/storagenode/blobstore/filestore.(*Dir).WalkNamespace:835\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).WalkNamespace:315\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:54\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePiecesToTrash:178\n\tstorj.io/storj/cmd/storagenode/internalcmd.gcCmdRun:104\n\tstorj.io/storj/cmd/storagenode/internalcmd.NewGCFilewalkerCmd.func1:35\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:267"}
	Line  8402549: 2024-06-29T12:27:49-04:00	INFO	lazyfilewalker.gc-filewalker.subprocess	gc-filewalker completed	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "trashPiecesCount": 65239, "piecesTrashed": 65239, "piecesSkippedCount": 0, "Process": "storagenode", "piecesCount": 1279965}
	Line  8402567: 2024-06-29T12:27:52-04:00	INFO	lazyfilewalker.gc-filewalker	subprocess finished successfully	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
	Line  8503418: 2024-06-29T15:33:21-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess exited with status	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "status": 1, "error": "exit status 1"}
	Line  8503419: 2024-06-29T15:33:21-04:00	INFO	lazyfilewalker.used-space-filewalker	subprocess exited with status	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "status": 1, "error": "exit status 1"}
	Line  8503420: 2024-06-29T15:33:21-04:00	ERROR	pieces:trash	emptying trash failed	{"error": "pieces error: lazyfilewalker: exit status 1", "errorVerbose": "pieces error: lazyfilewalker: exit status 1\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:85\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkCleanupTrash:187\n\tstorj.io/storj/storagenode/pieces.(*Store).EmptyTrash:419\n\tstorj.io/storj/storagenode/pieces.(*TrashChore).Run.func1.1:84\n\tstorj.io/common/sync2.(*Workplace).Start.func1:89"}
	Line  8503421: 2024-06-29T15:33:21-04:00	ERROR	pieces	failed to lazywalk space used by satellite	{"error": "lazyfilewalker: exit status 1", "errorVerbose": "lazyfilewalker: exit status 1\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:85\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:130\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:704\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
	Line  8503422: 2024-06-29T15:33:21-04:00	INFO	lazyfilewalker.used-space-filewalker	starting subprocess	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
	Line  8503423: 2024-06-29T15:33:21-04:00	ERROR	lazyfilewalker.used-space-filewalker	failed to start subprocess	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "error": "context canceled"}
	Line  8503424: 2024-06-29T15:33:21-04:00	ERROR	pieces	failed to lazywalk space used by satellite	{"error": "lazyfilewalker: context canceled", "errorVerbose": "lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:73\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:130\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:704\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
	Line  8503425: 2024-06-29T15:33:21-04:00	INFO	lazyfilewalker.used-space-filewalker	starting subprocess	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
	Line  8503426: 2024-06-29T15:33:21-04:00	ERROR	lazyfilewalker.used-space-filewalker	failed to start subprocess	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "error": "context canceled"}
	Line  8503427: 2024-06-29T15:33:21-04:00	ERROR	pieces	failed to lazywalk space used by satellite	{"error": "lazyfilewalker: context canceled", "errorVerbose": "lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:73\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:130\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:704\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
	Line  8503428: 2024-06-29T15:33:21-04:00	INFO	lazyfilewalker.used-space-filewalker	starting subprocess	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
	Line  8503429: 2024-06-29T15:33:21-04:00	ERROR	lazyfilewalker.used-space-filewalker	failed to start subprocess	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "error": "context canceled"}
	Line  8503430: 2024-06-29T15:33:21-04:00	ERROR	pieces	failed to lazywalk space used by satellite	{"error": "lazyfilewalker: context canceled", "errorVerbose": "lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:73\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:130\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:704\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
	Line  8503432: 2024-06-29T15:33:21-04:00	INFO	lazyfilewalker.gc-filewalker	subprocess exited with status	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "status": 1, "error": "exit status 1"}
	Line  8503433: 2024-06-29T15:33:21-04:00	ERROR	pieces	lazyfilewalker failed	{"error": "lazyfilewalker: exit status 1", "errorVerbose": "lazyfilewalker: exit status 1\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:85\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkSatellitePiecesToTrash:160\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePiecesToTrash:558\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces:369\n\tstorj.io/storj/storagenode/retain.(*Service).Run.func2:258\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
	Line  8503452: 2024-06-29T15:33:22-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	starting subprocess	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
	Line  8503453: 2024-06-29T15:33:22-04:00	INFO	lazyfilewalker.used-space-filewalker	starting subprocess	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
	Line  8503454: 2024-06-29T15:33:22-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess started	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
	Line  8503455: 2024-06-29T15:33:22-04:00	INFO	lazyfilewalker.used-space-filewalker	subprocess started	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
	Line  8503458: 2024-06-29T15:33:22-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker started	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode", "dateBefore": "2024-06-22T15:33:22-04:00"}
	Line  8503459: 2024-06-29T15:33:22-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	Database started	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode"}
	Line  8503460: 2024-06-29T15:33:22-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker completed	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "bytesDeleted": 0, "numKeysDeleted": 0, "Process": "storagenode"}
	Line  8503461: 2024-06-29T15:33:22-04:00	INFO	lazyfilewalker.used-space-filewalker.subprocess	Database started	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}
	Line  8503462: 2024-06-29T15:33:22-04:00	INFO	lazyfilewalker.used-space-filewalker.subprocess	used-space-filewalker started	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}
	Line  8503463: 2024-06-29T15:33:22-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess finished successfully	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
	Line  8503466: 2024-06-29T15:33:22-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	starting subprocess	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
	Line  8503468: 2024-06-29T15:33:22-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess started	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
	Line  8503469: 2024-06-29T15:33:22-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker started	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode", "dateBefore": "2024-06-22T15:33:22-04:00"}
	Line  8503470: 2024-06-29T15:33:22-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	Database started	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}
	Line  8503472: 2024-06-29T15:33:22-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker completed	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode", "bytesDeleted": 0, "numKeysDeleted": 0}
	Line  8503473: 2024-06-29T15:33:22-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess finished successfully	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
	Line  8503476: 2024-06-29T15:33:22-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	starting subprocess	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
	Line  8503477: 2024-06-29T15:33:22-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess started	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
	Line  8503479: 2024-06-29T15:33:23-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker started	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "dateBefore": "2024-06-22T15:33:22-04:00", "Process": "storagenode"}
	Line  8503480: 2024-06-29T15:33:23-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	Database started	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode"}
	Line  8503482: 2024-06-29T15:33:23-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker completed	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode", "bytesDeleted": 0, "numKeysDeleted": 0}
	Line  8503483: 2024-06-29T15:33:23-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess finished successfully	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
	Line  8503486: 2024-06-29T15:33:23-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	starting subprocess	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
	Line  8503487: 2024-06-29T15:33:23-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess started	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
	Line  8503488: 2024-06-29T15:33:23-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker started	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode", "dateBefore": "2024-06-22T15:33:23-04:00"}
	Line  8503489: 2024-06-29T15:33:23-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	Database started	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode"}