How do I empty the trash?

[naxbc]

They all seems to finish sucessfully but I still have the issue:

PS C:\Users\Admin> sls “used-space-filewalker” “C:\Program Files\Storj\Storage Node\storagenode.log” | sls “finish”

C:\Program Files\Storj\Storage Node\storagenode.log:83:2024-04-28T15:00:29+01:00 INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {“satelliteID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”}
C:\Program Files\Storj\Storage Node\storagenode.log:27519:2024-04-28T17:52:28+01:00 INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {“satelliteID”:
“1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”}
C:\Program Files\Storj\Storage Node\storagenode.log:78869:2024-04-28T23:02:01+01:00 INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {“satelliteID”:
“121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”}
C:\Program Files\Storj\Storage Node\storagenode.log:378641:2024-04-30T11:15:00+01:00 INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {“satelliteID”:
“118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”}
C:\Program Files\Storj\Storage Node\storagenode.log:379405:2024-04-30T11:20:04+01:00 INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {“satelliteID”:
“118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”}
C:\Program Files\Storj\Storage Node\storagenode.log:386935:2024-04-30T11:58:42+01:00 INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {“satelliteID”:
“118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”}
C:\Program Files\Storj\Storage Node\storagenode.log:426732:2024-04-30T15:17:49+01:00 INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {“satelliteID”:
“1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”}
C:\Program Files\Storj\Storage Node\storagenode.log:485764:2024-04-30T21:26:31+01:00 INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {“satelliteID”:
“121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”}
C:\Program Files\Storj\Storage Node\storagenode.log:866909:2024-05-02T17:59:46+01:00 INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {“satelliteID”:
“118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”}
C:\Program Files\Storj\Storage Node\storagenode.log:886071:2024-05-02T20:31:13+01:00 INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {“satelliteID”:
“1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”}
C:\Program Files\Storj\Storage Node\storagenode.log:951473:2024-05-03T04:37:03+01:00 INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {“satelliteID”:
“121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”}
C:\Program Files\Storj\Storage Node\storagenode.log:967305:2024-05-03T06:36:36+01:00 INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {“satelliteID”:
“118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”}
C:\Program Files\Storj\Storage Node\storagenode.log:989000:2024-05-03T11:13:08+01:00 INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {“satelliteID”:
“1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”}
C:\Program Files\Storj\Storage Node\storagenode.log:990440:2024-05-03T11:30:11+01:00 INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {“satelliteID”:
“118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”}
C:\Program Files\Storj\Storage Node\storagenode.log:998765:2024-05-03T12:38:56+01:00 INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {“satelliteID”:
“1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”}
C:\Program Files\Storj\Storage Node\storagenode.log:1018131:2024-05-03T15:42:08+01:00 INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {“satelliteID”:
“121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”}
C:\Program Files\Storj\Storage Node\storagenode.log:1353470:2024-05-05T10:08:57+01:00 INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {“satelliteID”:
“118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”}
C:\Program Files\Storj\Storage Node\storagenode.log:1387691:2024-05-05T16:33:24+01:00 INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {“satelliteID”:
“118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”}

[Alexey]

I see only two satellites, 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE

@BrightSilence

Satellite Address: saltlake.tardigrade.io:7777
Satellite ID: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE

and 121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6

@BrightSilence

Satellite Address: ap1.storj.io:7777 (formerly: asia-east-1.tardigrade.io:7777 )
Satellite ID: 121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6

the 118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW satellite

@BrightSilence

Satellite Address: satellite.stefan-benten.de:7777 (decommissioned)
Satellite ID: 118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW

is decommissioned many years ago.
Where the others?

[naxbc]

No idea. This is the output for the PowerShell command.
What should I do @Alexey? I have the database reporting 1.72TB Trash and it’s not accurate.
This impact’s node performance. How can I fix this?

[Alexey]

Only by finishing used-space-filewalker, which is running only after a restart.

[donald.m.motsinger]

On some of my nodes there are mixed folders (old an new style) in the trash for us1. The old style folders are all from 27/04 and don’t seem to get cleaned up.

I just finished a “scan on startup” scan on one of my smaller nodes with 1.2TB of data. I scanned the trash folder with du and the old style folders don’t seem to count as trash according to the now corrected dashboard. Will they be cleaned up with the next update(s)?

Also, the startup scan took >12h for 1.2TB. I’m reluctant to enable it on my other nodes with >3TB. Will the disk usage discrepancy fix itself without enabling the startup scan once you fixed the trash problem?

Is there an ELI5 for all the filewalkers etc. and their switches and the effect when enabling/disabling them? It’s all spread around in several threads and hard to understand for the average joe.

[striker43]

@donald.m.motsinger

On some of my nodes there are mixed folders (old an new style) in the trash for us1. The old style folders are all from 27/04 and don’t seem to get cleaned up.

Same for all of my nodes. I just deleted all trash folders manually now to solve this issue…

[naxbc]

Either there’s something I’m missing or something is very wrong.
It’s been almost 24 hours; I have “enabled pieces on startup” but when I run powershell, scan only starts for 2 sattelites @Alexey
Anything I should check in particular?

Started Logs
PS C:\Users\Admin> sls “used-space-filewalker” “C:\Program Files\Storj\Storage Node\storagenode.log” | sls “started”

C:\Program Files\Storj\Storage Node\storagenode.log:20:2024-05-09T13:12:37+01:00 INFO lazyfilewalker.used-space-filewalker subprocess started {“satelliteID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”}
C:\Program Files\Storj\Storage Node\storagenode.log:34:2024-05-09T13:12:38+01:00 INFO lazyfilewalker.used-space-filewalker.subprocess Database started {“satelliteID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”,
“Process”: “storagenode”}
C:\Program Files\Storj\Storage Node\storagenode.log:35:2024-05-09T13:12:38+01:00 INFO lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started {“satelliteID”:
“1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Process”: “storagenode”}
C:\Program Files\Storj\Storage Node\storagenode.log:349:2024-05-09T13:13:37+01:00 INFO lazyfilewalker.used-space-filewalker subprocess started {“satelliteID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”}
C:\Program Files\Storj\Storage Node\storagenode.log:362:2024-05-09T13:13:38+01:00 INFO lazyfilewalker.used-space-filewalker.subprocess Database started {“satelliteID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”,
“Process”: “storagenode”}
C:\Program Files\Storj\Storage Node\storagenode.log:363:2024-05-09T13:13:38+01:00 INFO lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started {“satelliteID”:
“1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Process”: “storagenode”}
C:\Program Files\Storj\Storage Node\storagenode.log:1051:2024-05-09T13:16:19+01:00 INFO lazyfilewalker.used-space-filewalker subprocess started {“satelliteID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”}
C:\Program Files\Storj\Storage Node\storagenode.log:1084:2024-05-09T13:16:20+01:00 INFO lazyfilewalker.used-space-filewalker.subprocess Database started {“satelliteID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”,
“Process”: “storagenode”}
C:\Program Files\Storj\Storage Node\storagenode.log:1085:2024-05-09T13:16:20+01:00 INFO lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started {“satelliteID”:
“1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Process”: “storagenode”}
C:\Program Files\Storj\Storage Node\storagenode.log:56204:2024-05-09T15:11:03+01:00 INFO lazyfilewalker.used-space-filewalker subprocess started {“satelliteID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”}
C:\Program Files\Storj\Storage Node\storagenode.log:56211:2024-05-09T15:11:04+01:00 INFO lazyfilewalker.used-space-filewalker.subprocess Database started {“satelliteID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”,
“Process”: “storagenode”}
C:\Program Files\Storj\Storage Node\storagenode.log:56212:2024-05-09T15:11:04+01:00 INFO lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started {“satelliteID”:
“121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Process”: “storagenode”}
C:\Program Files\Storj\Storage Node\storagenode.log:160980:2024-05-09T18:23:10+01:00 INFO lazyfilewalker.used-space-filewalker subprocess started {“satelliteID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”}
C:\Program Files\Storj\Storage Node\storagenode.log:160986:2024-05-09T18:23:10+01:00 INFO lazyfilewalker.used-space-filewalker.subprocess Database started {“satelliteID”:
“12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Process”: “storagenode”}
C:\Program Files\Storj\Storage Node\storagenode.log:160987:2024-05-09T18:23:10+01:00 INFO lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started {“satelliteID”:
“12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Process”: “storagenode”}
C:\Program Files\Storj\Storage Node\storagenode.log:284012:2024-05-09T22:08:31+01:00 INFO lazyfilewalker.used-space-filewalker subprocess started {“satelliteID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”}
C:\Program Files\Storj\Storage Node\storagenode.log:284046:2024-05-09T22:08:33+01:00 INFO lazyfilewalker.used-space-filewalker.subprocess Database started {“satelliteID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”,
“Process”: “storagenode”}
C:\Program Files\Storj\Storage Node\storagenode.log:284047:2024-05-09T22:08:33+01:00 INFO lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started {“satelliteID”:
“1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Process”: “storagenode”}
C:\Program Files\Storj\Storage Node\storagenode.log:364602:2024-05-10T00:42:36+01:00 INFO lazyfilewalker.used-space-filewalker subprocess started {“satelliteID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”}
C:\Program Files\Storj\Storage Node\storagenode.log:364609:2024-05-10T00:42:37+01:00 INFO lazyfilewalker.used-space-filewalker.subprocess Database started {“satelliteID”:
“121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Process”: “storagenode”}
C:\Program Files\Storj\Storage Node\storagenode.log:364610:2024-05-10T00:42:37+01:00 INFO lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started {“satelliteID”:
“121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Process”: “storagenode”}
C:\Program Files\Storj\Storage Node\storagenode.log:499457:2024-05-10T05:31:14+01:00 INFO lazyfilewalker.used-space-filewalker subprocess started {“satelliteID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”}
C:\Program Files\Storj\Storage Node\storagenode.log:499463:2024-05-10T05:31:15+01:00 INFO lazyfilewalker.used-space-filewalker.subprocess Database started {“satelliteID”:
“12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Process”: “storagenode”}
C:\Program Files\Storj\Storage Node\storagenode.log:499464:2024-05-10T05:31:15+01:00 INFO lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started {“satelliteID”:
“12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Process”: “storagenode”}

Finished Log:
PS C:\Users\Admin> sls “used-space-filewalker” “C:\Program Files\Storj\Storage Node\storagenode.log” | sls “finished”

C:\Program Files\Storj\Storage Node\storagenode.log:56202:2024-05-09T15:11:03+01:00 INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {“satelliteID”:
“1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”}
C:\Program Files\Storj\Storage Node\storagenode.log:160977:2024-05-09T18:23:10+01:00 INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {“satelliteID”:
“121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”}
C:\Program Files\Storj\Storage Node\storagenode.log:364600:2024-05-10T00:42:36+01:00 INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {“satelliteID”:
“1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”}
C:\Program Files\Storj\Storage Node\storagenode.log:499455:2024-05-10T05:31:14+01:00 INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {“satelliteID”:
“121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”}

[donald.m.motsinger]

I think the startup scan is a different process which doesn’t write to the logs. When I ran it on 2 nodes recently it scanned about 100GB per hour.

Just watch the iops of the hard disk. While it’s 100% the scan is still running.

[Alexey]

@naxbc

but when I run powershell, scan only starts for 2 sattelites @Alexey
Anything I should check in particular?

the scan may start a few hours later (to do not overload the node):

It seems that the new feature “save-state-resume GC filewalker” isn’t functioning as expected

If you check your logs, you will see that it will start not immediately but with some delay. $ cat /mnt/x/storagenode2/storagenode.log | grep "Node " | tail 2024-05-01T10:01:01Z INFO Node 1HhG6tE58gGMUXch83b71p63JEyewy6haL5npqEgGQKwpdEhxo started {"Process": "storagenode"} $ cat /mnt/x/storagenode2/storagenode.log | grep "used-space" | grep "started" | tail 2024-05-01T10:01:08Z INFO lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started {"Process": "storag…

@donald.m.motsinger

I think the startup scan is a different process which doesn’t write to the logs. When I ran it on 2 nodes recently it scanned about 100GB per hour.

it’s reported as used-space-filewalker, see logs from my node above.

@donald.m.motsinger

While it’s 100% the scan is still running.

this could be any of 5 filewalkers, not only scan on startup, especially if it’s starting with a delay (see my logs above).

@naxbc

Started Logs
PS C:\Users\Admin> sls “used-space-filewalker” “C:\Program Files\Storj\Storage Node\storagenode.log” | sls “started”

please try this one:

(sls "used-space-filewalker" "C:\Program Files\Storj\Storage Node\storagenode.log" | sls "started").line | select -last 20

and post results between two lines with three backticks, like this:

```
2024-05-09T13:12:37+01:00 INFO lazyfilewalker.used-space-filewalker subprocess started {“satelliteID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”}
```

it will be rendered like this

2024-05-09T13:12:37+01:00 INFO lazyfilewalker.used-space-filewalker subprocess started {“satelliteID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”}

[naxbc]

Looks like it started for all 4 sattelites now right?
Below logs are with the settings you said @Alexey:
“# storage2.piece-scan-on-startup: true”
“pieces.enable-lazy-filewalker: false”

PS C:\Users\Admin> (sls “used-space-filewalker” “C:\Program Files\Storj\Storage Node\storagenode.log” | sls “started”).line | select -last 20

2024-05-09T15:11:04+01:00       INFO    lazyfilewalker.used-space-filewalker.subprocess Database started        {"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode"}
2024-05-09T15:11:04+01:00       INFO    lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started   {"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode"}
2024-05-09T18:23:10+01:00       INFO    lazyfilewalker.used-space-filewalker    subprocess started      {"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-05-09T18:23:10+01:00       INFO    lazyfilewalker.used-space-filewalker.subprocess Database started        {"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode"}
2024-05-09T18:23:10+01:00       INFO    lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started   {"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode"}
2024-05-09T22:08:31+01:00       INFO    lazyfilewalker.used-space-filewalker    subprocess started      {"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-09T22:08:33+01:00       INFO    lazyfilewalker.used-space-filewalker.subprocess Database started        {"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}
2024-05-09T22:08:33+01:00       INFO    lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started   {"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}
2024-05-10T00:42:36+01:00       INFO    lazyfilewalker.used-space-filewalker    subprocess started      {"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-05-10T00:42:37+01:00       INFO    lazyfilewalker.used-space-filewalker.subprocess Database started        {"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode"}
2024-05-10T00:42:37+01:00       INFO    lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started   {"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode"}
2024-05-10T05:31:14+01:00       INFO    lazyfilewalker.used-space-filewalker    subprocess started      {"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-05-10T05:31:15+01:00       INFO    lazyfilewalker.used-space-filewalker.subprocess Database started        {"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode"}
2024-05-10T05:31:15+01:00       INFO    lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started   {"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode"}
2024-05-10T11:10:44+01:00       INFO    lazyfilewalker.used-space-filewalker    subprocess started      {"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-10T11:10:45+01:00       INFO    lazyfilewalker.used-space-filewalker.subprocess Database started        {"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}
2024-05-10T11:10:45+01:00       INFO    lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started   {"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}
2024-05-10T14:27:09+01:00       INFO    lazyfilewalker.used-space-filewalker    subprocess started      {"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-05-10T14:27:09+01:00       INFO    lazyfilewalker.used-space-filewalker.subprocess Database started        {"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode"}
2024-05-10T14:27:09+01:00       INFO    lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started   {"satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode"}

[donald.m.motsinger]

@Alexey

it’s reported as used-space-filewalker, see logs from my node above.

Are you talking about this setting?
storage2.piece-scan-on-startup: true

None of my nodes has “used-space-filewalker” in the logs. I enabled this setting a few days ago on 2 of my nodes and the trash “use” went from ~800GB down to <50GB. I also set

pieces.enable-lazy-filewalker: false

as it was recommended everywhere.

Ot is there another filewalker which corrected the trash use? The amount of filewalkers is still so confusing.

[Alexey]

Filewalker status

Logs should be at least info level Used space filewalker docker logs storagenode -f 2>&1 | grep "used-space-filewalker" | grep -E "started|finished" GC filewalker docker logs storagenode -f 2>&1 | grep "gc-filewalker" | grep -E "started|finished" Trash-cleanup filewalker docker logs storagenode -f 2>&1 | grep "trash-cleanup-filewalker" | grep -E "started|finished" retain docker logs storagenode -f 2>&1 | grep "retain" collector docker logs storagenode -f 2>&1 | grep "collector" trash docke…

[snorkel]

If you use custom.log level and disable piecestore, it won’t log the walker also.

[donald.m.motsinger]

I DO NOT have “used-space-filewalker” in any log in any of my nodes and I keep all logs since the beginning!

# zgrep "used-space-filewalker" /mnt/storagenode*/node/node.log*|wc -l
0

I don’t have changed the log level anywhere, so it is INFO. Something is wrong here.

[Alexey]

Then it’s disabled on start.
Please search for

storage2.piece-scan-on-startup: false

option in your config.yaml or command line argument --storage2.piece-scan-on-startup=false in your docker run command or in the docker-compose.yaml.

[donald.m.motsinger]

@Alexey

Then it’s disabled on start.

I have it usually disabled, but to fix the wrong trash value I enabled it for 2 nodes. Nowhere in the log I can see used-space-filewalker. Also, I don’t think that I set storage2.piece-scan-on-startup: false right away when this setting got introduced. So I would expect used-space-filewalker in the logs of some nodes in the past.

To get to the bottom of it I stopped and removed all nodes and enabled storage2.piece-scan-on-startup and disabled pieces.enable-lazy-filewalker for one node and started all nodes again.

After 3h the %util column of iostat is still constantly ~100% and I expect it to be for another day. Still no used-space-filewalker in the logs.

config.yaml

contact.external-address: ""
identity.cert-path: identity/identity.cert
identity.key-path: identity/identity.key
log.level: info
operator.email: ""
operator.wallet: ""
server.address: :28967
server.debug-log-traffic: false
server.private-address: 127.0.0.1:7778
storage.allocated-bandwidth: 2.0 TB
storage.allocated-disk-space: 1.0 TB
log.output: "/app/config/node.log"
console.address: :14002
debug.addr: ":7777"
preflight.database-check: true
storage2.piece-scan-on-startup: true
pieces.enable-lazy-filewalker: false

docker run -d --restart unless-stopped --stop-timeout 300 \
        -p 28969:28967/tcp \
        -p 28969:28967/udp \
        -e WALLET="$WALLET" \
        -e EMAIL="$EMAIL" \
        -e ADDRESS="$ADDRESS:28969" \
        -e STORAGE="$STORAGE" \
        -p 14004:14002 \
        -p 127.0.0.1:7779:7777 \
        --sysctl net.ipv4.tcp_fastopen=3 \
        --mount type=bind,source="$identityDir",destination=/app/identity \
        --mount type=bind,source="$storageDir",destination=/app/config \
        --name $NAME storjlabs/storagenode:latest \
        --operator.wallet-features="zksync-era"

I can’t imagine that I’m the only one with this problem. It happens on ALL nodes where I enabled storage2.piece-scan-on-startup so far.

[jammerdan]

Have you tried to put it in the docker run command?

Turning off used space filewalker

Try to use a different notation: --storage2.piece-scan-on-startup=false \ --pieces.enable-lazy-filewalker=false \ Make sure that they are provided after the image name.

[donald.m.motsinger]

@jammerdan

Have you tried to put it in the docker run command?

I don’t want to interrupt the current scan, so I’ll try with the next node.

The thing is the scan seems to run. Disk utilisation is 100% for over a day and at the end the value for trash gets corrected. What other process would be responsible for that?

Now I added

        --storage2.piece-scan-on-startup=true \
        --pieces.enable-lazy-filewalker=false \

to the docker run command of the next node and after >30 minutes at 100% disk utilization still nothing in the logs.

grep -v piecestore /mnt/storagenode4/node/node.log
2024-05-11T16:29:08Z    INFO    Configuration loaded    {"Process": "storagenode", "Location": "/app/config/config.yaml"}
2024-05-11T16:29:08Z    INFO    Anonymized tracing enabled      {"Process": "storagenode"}
2024-05-11T16:29:08Z    INFO    Operator email  {"Process": "storagenode", "Address": "xxx"}
2024-05-11T16:29:08Z    INFO    Operator wallet {"Process": "storagenode", "Address": "xxx"}
2024-05-11T16:29:09Z    INFO    server  existing kernel support for server-side tcp fast open detected  {"Process": "storagenode"}
2024-05-11T16:29:09Z    INFO    Telemetry enabled       {"Process": "storagenode", "instance ID": "xxx"}
2024-05-11T16:29:09Z    INFO    Event collection enabled        {"Process": "storagenode", "instance ID": "xxx"}
2024-05-11T16:29:09Z    INFO    db.migration    Database Version        {"Process": "storagenode", "version": 56}
2024-05-11T16:29:09Z    INFO    preflight:localtime     start checking local system clock with trusted satellites' system clock.        {"Process": "storagenode"}
2024-05-11T16:29:10Z    INFO    preflight:localtime     local system clock is in sync with trusted satellites' system clock.    {"Process": "storagenode"}
2024-05-11T16:29:10Z    INFO    Node xxx started {"Process": "storagenode"}
2024-05-11T16:29:10Z    INFO    Public server started on [::]:28967     {"Process": "storagenode"}
2024-05-11T16:29:10Z    INFO    Private server started on 127.0.0.1:7778        {"Process": "storagenode"}
2024-05-11T16:29:10Z    INFO    trust   Scheduling next refresh {"Process": "storagenode", "after": "5h18m26.174964708s"}
2024-05-11T16:29:10Z    INFO    pieces:trash    emptying trash started  {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-11T16:29:10Z    INFO    bandwidth       Performing bandwidth usage rollups      {"Process": "storagenode"}
2024-05-11T16:29:10Z    INFO    pieces:trash    emptying trash finished {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "elapsed": "155.474602ms"}
2024-05-11T16:29:10Z    INFO    pieces:trash    emptying trash started  {"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-05-11T16:29:10Z    INFO    pieces:trash    emptying trash finished {"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "elapsed": "10.117865ms"}
2024-05-11T16:29:10Z    INFO    pieces:trash    emptying trash started  {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-05-11T16:29:10Z    INFO    pieces:trash    emptying trash finished {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "elapsed": "35.35088ms"}
2024-05-11T16:29:10Z    INFO    pieces:trash    emptying trash started  {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-05-11T16:29:10Z    INFO    pieces:trash    emptying trash finished {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "elapsed": "93.757232ms"}
2024-05-11T16:29:11Z    INFO    collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "ADT6V63XNRWHI4RUHDG4DEIF42E2HL2CKSDI6BFMVMSB73SQ7VYA"}
2024-05-11T16:29:11Z    INFO    collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "UKIVXYVJHAXLNUGFFUZTBRPXMGJMHWYLOB3CUHIL3CY7K75WDBIQ"}
2024-05-11T16:29:11Z    INFO    collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "XMJ2UN4AJKBTP2VGIMO5D7RU6WCXKJ3Q27XH6DJ47WNYOCM7X45A"}
2024-05-11T16:29:11Z    INFO    collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "DAOWZPQEHC2AIZ7XXPXJR5F6QV2BX2P6V6MLJLIYX5NYJDXTCTSQ"}
2024-05-11T16:29:11Z    INFO    collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "ZB27RL6XS4FYWSYEITMUS2TLOU3UELGQ63MK6WJOM75O5XVHH6EQ"}
2024-05-11T16:29:11Z    INFO    collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "A5BND2I3ABI4K6E2CUHR4T3RD35GW2UYLSBEACEJJQHHXMKHFGCA"}
2024-05-11T16:29:11Z    INFO    collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "VWKIZO2ZKDUQK2EIXADNFKWKCNGUWTUUAQQ5Y4WZYEC6GEJDNHKA"}
2024-05-11T16:29:11Z    INFO    collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "YPV76P7U4VSNPTPWWLF75K4DBNOPCMSYXN4SA5CTUVN7H63QBLRQ"}
2024-05-11T16:29:11Z    INFO    collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "MWB6KQNQHCE3EO2MRMLZOXRYSC7CHOLVFOGXM2NYCLMRD4SCYXKA"}
2024-05-11T16:29:12Z    INFO    collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "73CVUTFU3LW4WLWU6P26ZW75L6YRUF7CO2IXZNVQYDY5IVPBCZZA"}
2024-05-11T16:29:12Z    INFO    collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "ZOPPVF6HZVVENT5DOXRQ4QXYRSVIAXSRRLPZNTKMACLXPVWQAKQQ"}
2024-05-11T16:29:12Z    INFO    collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "Q56Z6MBQQHETPXIMAGVQYPLQHWVSVFVGYSPACXYJ3E22ZTNRLSSA"}
2024-05-11T16:29:12Z    INFO    collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "5SAIUNMMLV66OU4GGGDNFLFAWXZOZ7I3VDMPVNJVPIAVSEPBY2GA"}
2024-05-11T16:29:12Z    INFO    collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "YQKRXLCPYBQINCNHRFDBEGNHXHN3544QXFELGXFBVKCSECC6PHUQ"}
2024-05-11T16:29:12Z    INFO    collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "2WJPVKOKMD75LSF6EU35OSC7D2TCDZDBVPADONJSXF62XS5FXX3Q"}
2024-05-11T16:29:12Z    INFO    collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "KAZSLGSF3XDJF4LND5ZDO6ZA2EWNIRMGQ4VPCZDXTWJJAYGS3LCA"}
2024-05-11T16:29:12Z    INFO    collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "HSFZBOL64PU2KFIFDD35TK77YTKZDNWW3RLWD6QNFS6AB3AYAVQA"}
2024-05-11T16:29:12Z    INFO    collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "QBPIZQA2VKUWUT7SPVACF63IYKER5HK6VY22D3HFGE6GNO5N3ABQ"}
2024-05-11T16:29:12Z    INFO    collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "RMCMC7DVIEI5CPFYDY5VQSEOUWCLVE663SL3TEP3OUDEXC6QLQEQ"}
2024-05-11T16:29:13Z    INFO    collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Piece ID": "4YNR5Z5HOER2DEYLSXT4AEODPG4HHXGFWRGTKIJPIMUOXXL2QBNQ"}
2024-05-11T16:29:13Z    INFO    collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "DTEIP6MJ52IXLOFDWM4QTKQXTEFSBWR4XVAW6TZFKRDCWTCFMMMA"}
2024-05-11T16:29:13Z    INFO    collector       collect {"Process": "storagenode", "count": 21}
2024-05-11T16:29:17Z    INFO    Configuration loaded    {"Process": "storagenode", "Location": "/app/config/config.yaml"}
2024-05-11T16:29:17Z    INFO    Invalid configuration file key  {"Process": "storagenode", "Key": "operator.email"}
2024-05-11T16:29:17Z    INFO    Invalid configuration file key  {"Process": "storagenode", "Key": "operator.wallet"}
2024-05-11T16:29:17Z    INFO    Invalid configuration file key  {"Process": "storagenode", "Key": "server.address"}
2024-05-11T16:29:17Z    INFO    Invalid configuration file key  {"Process": "storagenode", "Key": "storage.allocated-disk-space"}
2024-05-11T16:29:17Z    INFO    Invalid configuration file key  {"Process": "storagenode", "Key": "console.address"}
2024-05-11T16:29:17Z    INFO    Invalid configuration file key  {"Process": "storagenode", "Key": "contact.external-address"}
2024-05-11T16:29:17Z    INFO    Invalid configuration file key  {"Process": "storagenode", "Key": "server.private-address"}
2024-05-11T16:29:17Z    INFO    Invalid configuration file key  {"Process": "storagenode", "Key": "storage2.piece-scan-on-startup"}
2024-05-11T16:29:17Z    INFO    Invalid configuration file key  {"Process": "storagenode", "Key": "server.debug-log-traffic"}
2024-05-11T16:29:17Z    INFO    Invalid configuration file key  {"Process": "storagenode", "Key": "storage.allocated-bandwidth"}
2024-05-11T16:29:17Z    INFO    Invalid configuration file key  {"Process": "storagenode", "Key": "preflight.database-check"}
2024-05-11T16:29:17Z    INFO    Anonymized tracing enabled      {"Process": "storagenode"}
2024-05-11T16:29:17Z    INFO    Identity loaded.        {"Process": "storagenode", "Node ID": "1WU7Snm18BWvbYWwfWHEag9LzQRqibV8zrn2NwJJjhcYnBR5Jh"}
2024-05-11T16:29:33Z    INFO    orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6      sending {"Process": "storagenode", "count": 57}
2024-05-11T16:29:33Z    INFO    orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S      sending {"Process": "storagenode", "count": 2204}
2024-05-11T16:29:33Z    INFO    orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs      sending {"Process": "storagenode", "count": 225}
2024-05-11T16:29:33Z    INFO    orders.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE       sending {"Process": "storagenode", "count": 23}
2024-05-11T16:29:33Z    INFO    orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs      finished        {"Process": "storagenode"}
2024-05-11T16:29:33Z    INFO    orders.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE       finished        {"Process": "storagenode"}
2024-05-11T16:29:34Z    INFO    orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6      finished        {"Process": "storagenode"}
2024-05-11T16:29:34Z    INFO    orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S      finished        {"Process": "storagenode"}
2024-05-11T16:32:45Z    INFO    reputation:service      node scores updated     {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Total Audits": 290436, "Successful Audits": 279641, "Audit Score": 1, "Online Score": 0.9956102024187534, "Suspension Score": 1, "Audit Score Delta": 0, "Online Score Delta": 0, "Suspension Score Delta": 0}
2024-05-11T16:32:46Z    INFO    reputation:service      node scores updated     {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Total Audits": 523476, "Successful Audits": 504336, "Audit Score": 1, "Online Score": 0.9960179494768052, "Suspension Score": 1, "Audit Score Delta": 0, "Online Score Delta": 0, "Suspension Score Delta": 0}
2024-05-11T16:32:46Z    INFO    reputation:service      node scores updated     {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Total Audits": 665064, "Successful Audits": 649249, "Audit Score": 1, "Online Score": 0.9952570622286699, "Suspension Score": 1, "Audit Score Delta": 0, "Online Score Delta": 0, "Suspension Score Delta": 0}
2024-05-11T16:32:46Z    INFO    reputation:service      node scores updated     {"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Total Audits": 734931, "Successful Audits": 702644, "Audit Score": 1, "Online Score": 0.9960900025141867, "Suspension Score": 1, "Audit Score Delta": 0, "Online Score Delta": 0, "Suspension Score Delta": 0}

[snorkel]

They said they did some log cleaning. Maybe they removed more than was needed.
Maybe only lazy scan gets a log entry, and the normal one, with lazy off, gets none. I didn’t saw the piece scan log entry either, but I thought the custom level is to blame.
Also I didn’t saw a trash cleanup entry or GC entry. Maybe logs on docker/linux have different logs than Windows ones.
I only get the collector, pieces:trash and retain entries.

[jammerdan]

@donald.m.motsinger

Now I added

        --storage2.piece-scan-on-startup=true \
        --pieces.enable-lazy-filewalker=false \

Oh sorry. I believe there is a bug and you must run it with lazy filewalker on. Otherwise there is no log entry I think I have read that somewhere.
It would be

        --storage2.piece-scan-on-startup=true \
        --pieces.enable-lazy-filewalker=true \

then.