[Solved] Trash Reported at 0Bytes but it's clearly there

[thelastspark]

Hi,

I have a node that is reporting 0 trash (which I found suspicious considering how much activity there has been):

When I check the trash folder I can see that it was successfully migrated to the new structure:

I checked and all folders are within 7 days, however, if I run properties on the trash folder:

You can see that there is still quite a fair number leftover. Any advice?

Also, I don’t know if it’s connected or needs a separate issue, but it seems like all my storagenode hard drives (1 HDD per node) are marked as Read-only (you can see this above) - is this an issue? Everything seems to be working fine otherwise. I’d rather not have to enable this if I don’t have to (this operation will take ages )

Thanks!

[ACarneiro]

Are you running the file walker at startup or have you disabled it?
If you’ve not disabled it, have you enabled the lazy file walker?

(I am not an expert by any means but I wonder if the file walker also does the garbage accounting thing)

[thelastspark]

I don’t have it disabled, but lazy walker is on

[ACarneiro]

Has it finished?
(Couldn’t see the uptime on your screenshot)

A reading of 0 is suspicious for database shenanigans, isn’t it?

[thelastspark]

How can I check that? I’m guessing the log file, but does it need a specific debug level (my current one is on ERROR)

[ACarneiro]

You can try

sudo docker logs <your docker container here> |grep filewalker.*subprocess.*finished

That should bring completion messages for each satellite

[thelastspark]

will that work for windows?

[ACarneiro]

Whoops! Sorry, I had a brain fart…
No, not sure how to do something similar on Windows… I guess you could open the log with some text editor and search for “filewalker subprocess finished”?

[thelastspark]

I don’t think INFO log level shows this so I just restarted the node with INFO. I’ll give it a few hours to run and report back here

[ACarneiro]

Sure thing. Sorry about that…

In the mean time, perhaps someone more knowledgeable like @Alexey can give you some more tips

[thelastspark]

So it just finished:

2024-05-21T16:33:22-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess started	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-05-21T16:33:22-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker started	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode", "dateBefore": "2024-05-14T16:33:22-04:00"}
2024-05-21T16:33:22-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	Database started	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode"}
2024-05-21T16:33:22-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker completed	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "numKeysDeleted": 0, "Process": "storagenode", "bytesDeleted": 0}
2024-05-21T16:33:22-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess finished successfully	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-05-21T16:33:22-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess started	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-21T16:33:22-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker started	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode", "dateBefore": "2024-05-14T16:33:22-04:00"}
2024-05-21T16:33:22-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	Database started	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}
2024-05-21T16:33:22-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker completed	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "bytesDeleted": 0, "numKeysDeleted": 0, "Process": "storagenode"}
2024-05-21T16:33:22-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess finished successfully	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-21T16:33:22-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess started	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-05-21T16:33:22-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker started	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode", "dateBefore": "2024-05-14T16:33:22-04:00"}
2024-05-21T16:33:22-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	Database started	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode"}
2024-05-21T16:33:22-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker completed	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode", "bytesDeleted": 0, "numKeysDeleted": 0}
2024-05-21T16:33:22-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess finished successfully	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-05-21T16:33:22-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess started	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-05-21T16:33:22-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker started	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode", "dateBefore": "2024-05-14T16:33:22-04:00"}
2024-05-21T16:33:22-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	Database started	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode"}
2024-05-21T16:33:22-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker completed	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "bytesDeleted": 0, "numKeysDeleted": 0, "Process": "storagenode"}
2024-05-21T16:33:22-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess finished successfully	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-05-22T16:33:22-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess started	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-05-22T16:33:22-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker started	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode", "dateBefore": "2024-05-15T16:33:22-04:00"}
2024-05-22T16:33:27-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	Database started	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode"}
2024-05-22T16:33:27-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker completed	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode", "bytesDeleted": 0, "numKeysDeleted": 0}
2024-05-22T16:33:27-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess finished successfully	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-05-22T16:33:27-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess started	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-05-22T16:33:27-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker started	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode", "dateBefore": "2024-05-15T16:33:27-04:00"}
2024-05-22T16:33:27-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	Database started	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode"}
2024-05-22T16:33:27-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker completed	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode", "bytesDeleted": 0, "numKeysDeleted": 0}
2024-05-22T16:33:27-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess finished successfully	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-05-22T16:33:27-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess started	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-22T16:33:27-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker started	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode", "dateBefore": "2024-05-15T16:33:27-04:00"}
2024-05-22T16:33:27-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	Database started	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}
2024-05-22T16:33:27-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker completed	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode", "bytesDeleted": 0, "numKeysDeleted": 0}
2024-05-22T16:33:27-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess finished successfully	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-22T16:33:27-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess started	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-05-22T16:33:27-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker started	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "dateBefore": "2024-05-15T16:33:27-04:00", "Process": "storagenode"}
2024-05-22T16:33:27-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	Database started	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode"}
2024-05-22T16:33:27-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker completed	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode", "bytesDeleted": 0, "numKeysDeleted": 0}
2024-05-22T16:33:27-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess finished successfully	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-05-23T12:03:04-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess started	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-05-23T12:03:04-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker started	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode", "dateBefore": "2024-05-16T12:03:04-04:00"}
2024-05-23T12:03:04-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	Database started	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode"}
2024-05-23T12:03:04-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker completed	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode", "bytesDeleted": 0, "numKeysDeleted": 0}
2024-05-23T12:03:04-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess finished successfully	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-05-23T12:03:04-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess started	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-23T12:03:04-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker started	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode", "dateBefore": "2024-05-16T12:03:04-04:00"}
2024-05-23T12:03:04-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	Database started	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}
2024-05-23T12:03:04-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker completed	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode", "bytesDeleted": 0, "numKeysDeleted": 0}
2024-05-23T12:03:04-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess finished successfully	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-23T12:03:04-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess started	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-05-23T12:03:04-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker started	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode", "dateBefore": "2024-05-16T12:03:04-04:00"}
2024-05-23T12:03:04-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	Database started	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode"}
2024-05-23T12:03:05-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker completed	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode", "bytesDeleted": 0, "numKeysDeleted": 0}
2024-05-23T12:03:05-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess finished successfully	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-05-23T12:03:05-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess started	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-05-23T12:03:05-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker started	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode", "dateBefore": "2024-05-16T12:03:05-04:00"}
2024-05-23T12:03:05-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	Database started	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode"}
2024-05-23T12:03:05-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker completed	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode", "bytesDeleted": 0, "numKeysDeleted": 0}
2024-05-23T12:03:06-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess finished successfully	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-05-23T22:18:10-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess started	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-05-23T22:18:10-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker started	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode", "dateBefore": "2024-05-16T22:18:10-04:00"}
2024-05-23T22:18:10-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	Database started	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode"}
2024-05-23T23:15:53-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker completed	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode", "bytesDeleted": 64350752768, "numKeysDeleted": 91349}
2024-05-23T23:15:53-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess finished successfully	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-05-23T23:15:53-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess started	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-23T23:15:53-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker started	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode", "dateBefore": "2024-05-16T23:15:53-04:00"}
2024-05-23T23:15:54-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	Database started	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}
2024-05-24T01:04:02-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker completed	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode", "bytesDeleted": 523897923840, "numKeysDeleted": 248384}
2024-05-24T01:04:02-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess finished successfully	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-24T01:04:02-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess started	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-05-24T01:04:02-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker started	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode", "dateBefore": "2024-05-17T01:04:02-04:00"}
2024-05-24T01:04:03-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	Database started	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode"}
2024-05-24T01:29:44-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker completed	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode", "bytesDeleted": 24157450496, "numKeysDeleted": 51417}
2024-05-24T01:29:44-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess finished successfully	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-05-24T01:29:44-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess started	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-05-24T01:29:44-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker started	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode", "dateBefore": "2024-05-17T01:29:44-04:00"}
2024-05-24T01:29:48-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	Database started	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode"}
2024-05-24T01:29:48-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker completed	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode", "bytesDeleted": 0, "numKeysDeleted": 0}
2024-05-24T01:29:49-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess finished successfully	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-05-24T22:18:10-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess started	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-05-24T22:18:10-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker started	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode", "dateBefore": "2024-05-17T22:18:10-04:00"}
2024-05-24T22:18:15-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	Database started	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode"}
2024-05-24T22:18:15-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker completed	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode", "bytesDeleted": 0, "numKeysDeleted": 0}
2024-05-24T22:18:15-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess finished successfully	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-05-24T22:18:15-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess started	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-24T22:18:15-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker started	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode", "dateBefore": "2024-05-17T22:18:15-04:00"}
2024-05-24T22:18:15-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	Database started	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}
2024-05-24T22:18:15-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker completed	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode", "bytesDeleted": 0, "numKeysDeleted": 0}
2024-05-24T22:18:15-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess finished successfully	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-24T22:18:15-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess started	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-05-24T22:18:15-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker started	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode", "dateBefore": "2024-05-17T22:18:15-04:00"}
2024-05-24T22:18:15-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	Database started	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode"}
2024-05-24T22:18:15-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker completed	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "bytesDeleted": 0, "numKeysDeleted": 0, "Process": "storagenode"}
2024-05-24T22:18:15-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess finished successfully	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-05-24T22:18:15-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess started	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-05-24T22:18:15-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker started	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode", "dateBefore": "2024-05-17T22:18:15-04:00"}
2024-05-24T22:18:15-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	Database started	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode"}
2024-05-24T22:18:15-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker completed	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode", "bytesDeleted": 0, "numKeysDeleted": 0}
2024-05-24T22:18:15-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess finished successfully	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-05-26T17:58:38-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess started	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-05-26T17:58:39-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker started	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode", "dateBefore": "2024-05-19T17:58:38-04:00"}
2024-05-26T17:58:39-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	Database started	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode"}
2024-05-26T17:58:39-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker completed	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode", "bytesDeleted": 0, "numKeysDeleted": 0}
2024-05-26T17:58:39-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess finished successfully	{"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-05-26T17:58:39-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess started	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-05-26T17:58:39-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker started	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode", "dateBefore": "2024-05-19T17:58:39-04:00"}
2024-05-26T17:58:39-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	Database started	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode"}
2024-05-26T17:58:39-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker completed	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode", "bytesDeleted": 0, "numKeysDeleted": 0}
2024-05-26T17:58:39-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess finished successfully	{"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-05-26T17:58:39-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess started	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-26T17:58:39-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker started	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "dateBefore": "2024-05-19T17:58:39-04:00", "Process": "storagenode"}
2024-05-26T17:58:39-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	Database started	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}
2024-05-26T17:58:40-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker completed	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode", "bytesDeleted": 0, "numKeysDeleted": 0}
2024-05-26T17:58:40-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess finished successfully	{"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-26T17:58:40-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess started	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-05-26T17:58:40-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker started	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode", "dateBefore": "2024-05-19T17:58:40-04:00"}
2024-05-26T17:58:40-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	Database started	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode"}
2024-05-26T17:58:41-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker.subprocess	trash-filewalker completed	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode", "bytesDeleted": 0, "numKeysDeleted": 0}
2024-05-26T17:58:41-04:00	INFO	lazyfilewalker.trash-cleanup-filewalker	subprocess finished successfully	{"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}

But dashboard still says 0B when the trash folder has stuff in it (a good amount too relative to total capacity)

[Alexey]

You need to find the string “used-space-filewalker” with “finished” word:

Storagenode.exe stopping for too long, for 10h+

Windows/PowerShell Logs should be at least info level Used space filewalker Regulated by the parameter in the config.yaml file or as a command line argument (add -- before the parameter, remove :, place it after the image name in your docker run command): # if set to true, all pieces disk usage is recalculated on startup (default true) storage2.piece-scan-on-startup: true to watch for a progress: cat "C:\Program Files\Storj\Storage Node\*.log" -Wait | sls "used-space-filewalker" | sls "sta…

[thelastspark]

Thanks for that, I’m guessing you mean specifically the

cat "C:\Program Files\Storj\Storage Node\*.log" -Wait | sls "used-space-filewalker" | sls "started|finished"
Command? If so, here is the result:

2024-05-21T16:33:22-04:00       INFO    lazyfilewalker.used-space-filewalker    subprocess started      {"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-21T16:33:22-04:00       INFO    lazyfilewalker.used-space-filewalker.subprocess Database started        {"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}
2024-05-21T16:33:22-04:00       INFO    lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started  {"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}
2024-05-21T16:35:41-04:00       INFO    lazyfilewalker.used-space-filewalker    subprocess finished successfully       {"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-21T16:35:41-04:00       INFO    lazyfilewalker.used-space-filewalker    subprocess started      {"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-05-21T16:35:41-04:00       INFO    lazyfilewalker.used-space-filewalker.subprocess Database started        {"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode"}
2024-05-21T16:35:41-04:00       INFO    lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started  {"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode"}
2024-05-21T17:26:59-04:00       INFO    lazyfilewalker.used-space-filewalker    subprocess finished successfully       {"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-05-21T17:26:59-04:00       INFO    lazyfilewalker.used-space-filewalker    subprocess started      {"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-05-21T17:26:59-04:00       INFO    lazyfilewalker.used-space-filewalker.subprocess Database started        {"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode"}
2024-05-21T17:26:59-04:00       INFO    lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started  {"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode"}
2024-05-22T16:44:10-04:00       INFO    lazyfilewalker.used-space-filewalker    subprocess finished successfully       {"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-05-22T16:44:10-04:00       INFO    lazyfilewalker.used-space-filewalker    subprocess started      {"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-05-22T16:44:10-04:00       INFO    lazyfilewalker.used-space-filewalker.subprocess Database started        {"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode"}
2024-05-22T16:44:10-04:00       INFO    lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started  {"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode"}
2024-05-22T19:36:17-04:00       INFO    lazyfilewalker.used-space-filewalker    subprocess finished successfully       {"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-05-23T12:03:04-04:00       INFO    lazyfilewalker.used-space-filewalker    subprocess started      {"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-23T12:03:04-04:00       INFO    lazyfilewalker.used-space-filewalker.subprocess Database started        {"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}
2024-05-23T12:03:04-04:00       INFO    lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started  {"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}
2024-05-23T12:20:06-04:00       INFO    lazyfilewalker.used-space-filewalker    subprocess finished successfully       {"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-23T12:20:06-04:00       INFO    lazyfilewalker.used-space-filewalker    subprocess started      {"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-05-23T12:20:07-04:00       INFO    lazyfilewalker.used-space-filewalker.subprocess Database started        {"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode"}
2024-05-23T12:20:07-04:00       INFO    lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started  {"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode"}
2024-05-23T13:22:00-04:00       INFO    lazyfilewalker.used-space-filewalker    subprocess finished successfully       {"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-05-23T13:22:00-04:00       INFO    lazyfilewalker.used-space-filewalker    subprocess started      {"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-05-23T13:22:00-04:00       INFO    lazyfilewalker.used-space-filewalker.subprocess Database started        {"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode"}
2024-05-23T13:22:00-04:00       INFO    lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started  {"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode"}
2024-05-23T22:18:10-04:00       INFO    lazyfilewalker.used-space-filewalker    subprocess started      {"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-23T22:18:10-04:00       INFO    lazyfilewalker.used-space-filewalker.subprocess Database started        {"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}
2024-05-23T22:18:10-04:00       INFO    lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started  {"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}
2024-05-23T23:31:43-04:00       INFO    lazyfilewalker.used-space-filewalker    subprocess finished successfully       {"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-23T23:31:43-04:00       INFO    lazyfilewalker.used-space-filewalker    subprocess started      {"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-05-23T23:31:43-04:00       INFO    lazyfilewalker.used-space-filewalker.subprocess Database started        {"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode"}
2024-05-23T23:31:43-04:00       INFO    lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started  {"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode"}
2024-05-24T01:41:04-04:00       INFO    lazyfilewalker.used-space-filewalker    subprocess finished successfully       {"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-05-24T01:41:04-04:00       INFO    lazyfilewalker.used-space-filewalker    subprocess started      {"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-05-24T01:41:04-04:00       INFO    lazyfilewalker.used-space-filewalker.subprocess Database started        {"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode"}
2024-05-24T01:41:04-04:00       INFO    lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started  {"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode"}
2024-05-26T17:58:38-04:00       INFO    lazyfilewalker.used-space-filewalker    subprocess started      {"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-26T17:58:39-04:00       INFO    lazyfilewalker.used-space-filewalker.subprocess Database started        {"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}
2024-05-26T17:58:39-04:00       INFO    lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started  {"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}
2024-05-26T19:04:43-04:00       INFO    lazyfilewalker.used-space-filewalker    subprocess finished successfully       {"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-26T19:04:43-04:00       INFO    lazyfilewalker.used-space-filewalker    subprocess started      {"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-05-26T19:04:44-04:00       INFO    lazyfilewalker.used-space-filewalker.subprocess Database started        {"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode"}
2024-05-26T19:04:44-04:00       INFO    lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started  {"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode"}
2024-05-26T20:05:16-04:00       INFO    lazyfilewalker.used-space-filewalker    subprocess finished successfully       {"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-05-26T20:05:16-04:00       INFO    lazyfilewalker.used-space-filewalker    subprocess started      {"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-05-26T20:05:16-04:00       INFO    lazyfilewalker.used-space-filewalker.subprocess Database started        {"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode"}
2024-05-26T20:05:16-04:00       INFO    lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started  {"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode"}

It seems like things finish successfully?

[donald.m.motsinger]

@thelastspark

It seems like things finish successfully?

No, it runs once per satellite and the last line says started.

[thelastspark]

Is subprocess finished successfully not it finishing?

[donald.m.motsinger]

Yes, but it finished for

"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"

After that the next process started for the next satellite.

2024-05-26T20:05:16-04:00       INFO    lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started  {"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode"}

[thelastspark]

How many satellites are there?
I see 4 different ones being finished in this screenshot alone:

[donald.m.motsinger]

@thelastspark

I see 4 different ones being finished in this screenshot alone:

Yes, there are 4. Look at the date and time of the logs. You must have restarted that node between 2024-05-24T01:41:04-04:00 and 2024-05-26T17:58:38-04:00.

The startup scan runs on every node start, hence the name

Look at the log entries from 2024-05-26 and you will see only 2 finished and the 3rd scan started.

Your logs also show that the scan which started 2024-05-24T01:41:04-04:00 didn’t finish when you restarted the node.

[thelastspark]

Nevermind my initial message - I see what you mean, the last line is saying started, not finished so it’s currently still trying to process.

In that case, how often does it try to start a new one? And does it try to start a new one if the old one hasn’t finished (assuming no node restart)

[donald.m.motsinger]

@thelastspark

In that case, how often does it try to start a new one? And does it try to start a new one if the old one hasn’t finished (assuming no node restart)

It starts 1 scan for each satellite one after the other on every start of the node. If you don’t restart the node there will be no more startup scans.

Let the scans finish. It looks like it’s on the 3rd satellite right now. When you interrupt it by restarting the node it will start from scratch again.