How do you solve slow file deletion on ext4?

[JWvdV]

@Roxor

Do you have a feel for how long your used-space-filewalker takes? I don’t have it set up yet to test: but I’m hoping it’s like 5min/TB/satellite or something.

root@STORJ7:~# docker logs storagenode 2> /dev/null | grep -i walk
2024-07-25T17:42:25Z    INFO    lazyfilewalker.trash-cleanup-filewalker starting subprocess                            {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-07-25T17:42:25Z    INFO    lazyfilewalker.trash-cleanup-filewalker subprocess started                             {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-07-25T17:42:25Z    INFO    lazyfilewalker.trash-cleanup-filewalker.subprocess                                     trash-filewalker started {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode", "dateBefore": "2024-07-18T17:42:25Z"}
2024-07-25T17:42:25Z    INFO    lazyfilewalker.trash-cleanup-filewalker.subprocess                                     Database started {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode"}
2024-07-25T17:42:25Z    INFO    lazyfilewalker.trash-cleanup-filewalker.subprocess                                     trash-filewalker completed       {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode", "bytesDeleted": 0, "numKeysDeleted": 0}
2024-07-25T17:42:25Z    INFO    pieces  used-space-filewalker started   {"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-07-25T17:42:25Z    INFO    lazyfilewalker.used-space-filewalker    starting subprocess                            {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-07-25T17:42:25Z    INFO    lazyfilewalker.used-space-filewalker    subprocess started                             {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-07-25T17:42:25Z    INFO    lazyfilewalker.trash-cleanup-filewalker subprocess finished successfully               {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-07-25T17:42:25Z    INFO    lazyfilewalker.trash-cleanup-filewalker starting subprocess                            {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-07-25T17:42:25Z    INFO    lazyfilewalker.trash-cleanup-filewalker subprocess started                             {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-07-25T17:42:25Z    INFO    lazyfilewalker.trash-cleanup-filewalker.subprocess                                     trash-filewalker started {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode", "dateBefore": "2024-07-18T17:42:25Z"}
2024-07-25T17:42:25Z    INFO    lazyfilewalker.used-space-filewalker.subprocess                                        Database started {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode"}
2024-07-25T17:42:25Z    INFO    lazyfilewalker.trash-cleanup-filewalker.subprocess                                     Database started {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}
2024-07-25T17:42:25Z    INFO    lazyfilewalker.trash-cleanup-filewalker.subprocess                                     trash-filewalker completed       {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode", "bytesDeleted": 0, "numKeysDeleted": 0}
2024-07-25T17:42:25Z    INFO    lazyfilewalker.trash-cleanup-filewalker subprocess finished successfully               {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-07-25T17:42:25Z    INFO    lazyfilewalker.trash-cleanup-filewalker starting subprocess                            {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-07-25T17:42:25Z    INFO    lazyfilewalker.trash-cleanup-filewalker subprocess started                             {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-07-25T17:42:25Z    INFO    lazyfilewalker.trash-cleanup-filewalker.subprocess                                     trash-filewalker started {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode", "dateBefore": "2024-07-18T17:42:25Z"}
2024-07-25T17:42:25Z    INFO    lazyfilewalker.trash-cleanup-filewalker.subprocess                                     Database started {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode"}
2024-07-25T17:42:25Z    INFO    lazyfilewalker.trash-cleanup-filewalker.subprocess                                     trash-filewalker completed       {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode", "bytesDeleted": 0, "numKeysDeleted": 0}
2024-07-25T17:42:25Z    INFO    lazyfilewalker.trash-cleanup-filewalker subprocess finished successfully               {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-07-25T17:42:25Z    INFO    lazyfilewalker.trash-cleanup-filewalker starting subprocess                            {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-07-25T17:42:25Z    INFO    lazyfilewalker.trash-cleanup-filewalker subprocess started                             {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-07-25T17:42:25Z    INFO    lazyfilewalker.trash-cleanup-filewalker.subprocess                                     trash-filewalker started {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode", "dateBefore": "2024-07-18T17:42:25Z"}
2024-07-25T17:42:25Z    INFO    lazyfilewalker.trash-cleanup-filewalker.subprocess                                     Database started {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode"}
2024-07-25T17:42:25Z    INFO    lazyfilewalker.trash-cleanup-filewalker.subprocess                                     trash-filewalker completed       {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode", "bytesDeleted": 0, "numKeysDeleted": 0}
2024-07-25T17:42:25Z    INFO    lazyfilewalker.trash-cleanup-filewalker subprocess finished successfully               {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-07-25T17:45:02Z    INFO    lazyfilewalker.used-space-filewalker    subprocess finished successfully               {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-07-25T17:45:02Z    INFO    pieces  used-space-filewalker completed {"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Lazy File Walker": true, "Total Pieces Size": 67104322048, "Total Pieces Content Size": 66981463552}
2024-07-25T17:45:02Z    INFO    pieces  used-space-filewalker started   {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-07-25T17:45:02Z    INFO    lazyfilewalker.used-space-filewalker    starting subprocess                            {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-07-25T17:45:02Z    INFO    lazyfilewalker.used-space-filewalker    subprocess started                             {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-07-25T17:45:02Z    INFO    lazyfilewalker.used-space-filewalker.subprocess                                        Database started {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode"}
2024-07-25T17:48:33Z    INFO    lazyfilewalker.used-space-filewalker    subprocess finished successfully               {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-07-25T17:48:33Z    INFO    pieces  used-space-filewalker completed {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Lazy File Walker": true, "Total Pieces Size": 324812015368, "Total Pieces Content Size": 324030394632}
2024-07-25T17:48:33Z    INFO    pieces  used-space-filewalker started   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-07-25T17:48:33Z    INFO    lazyfilewalker.used-space-filewalker    starting subprocess                            {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-07-25T17:48:33Z    INFO    lazyfilewalker.used-space-filewalker    subprocess started                             {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-07-25T17:48:33Z    INFO    lazyfilewalker.used-space-filewalker.subprocess                                        Database started {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}
2024-07-25T18:03:16Z    INFO    lazyfilewalker.used-space-filewalker    subprocess finished successfully               {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-07-25T18:03:16Z    INFO    pieces  used-space-filewalker completed {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Lazy File Walker": true, "Total Pieces Size": 394558090752, "Total Pieces Content Size": 393343318016}
2024-07-25T18:03:16Z    INFO    pieces  used-space-filewalker started   {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-07-25T18:03:16Z    INFO    lazyfilewalker.used-space-filewalker    starting subprocess                            {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-07-25T18:03:16Z    INFO    lazyfilewalker.used-space-filewalker    subprocess started                             {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-07-25T18:03:16Z    INFO    lazyfilewalker.used-space-filewalker.subprocess                                        Database started {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode"}
2024-07-25T18:03:50Z    INFO    lazyfilewalker.used-space-filewalker    subprocess finished successfully               {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-07-25T18:03:50Z    INFO    pieces  used-space-filewalker completed {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Lazy File Walker": true, "Total Pieces Size": 8268178176, "Total Pieces Content Size": 8256300288}
2024-07-26T01:28:32Z    INFO    lazyfilewalker.gc-filewalker    starting subprocess                                    {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-07-26T01:28:32Z    INFO    lazyfilewalker.gc-filewalker    subprocess started                                     {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-07-26T01:28:32Z    INFO    lazyfilewalker.gc-filewalker.subprocess Database started                               {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode"}
2024-07-26T01:28:32Z    INFO    lazyfilewalker.gc-filewalker.subprocess gc-filewalker started                          {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "createdBefore": "2024-07-21T17:59:57Z", "bloomFilterSize": 12864, "Process": "storagenode"}
2024-07-26T01:28:36Z    INFO    lazyfilewalker.gc-filewalker.subprocess gc-filewalker completed                        {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "piecesSkippedCount": 0, "Process": "storagenode", "piecesCount": 23199, "trashPiecesCount": 1442, "piecesTrashed": 1442}
2024-07-26T01:28:36Z    INFO    lazyfilewalker.gc-filewalker    subprocess finished successfully                       {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-07-26T17:42:25Z    INFO    lazyfilewalker.trash-cleanup-filewalker starting subprocess                            {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-07-26T17:42:25Z    INFO    lazyfilewalker.trash-cleanup-filewalker subprocess started                             {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-07-26T17:42:25Z    INFO    lazyfilewalker.trash-cleanup-filewalker.subprocess                                     trash-filewalker started {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode", "dateBefore": "2024-07-19T17:42:25Z"}
2024-07-26T17:42:26Z    INFO    lazyfilewalker.trash-cleanup-filewalker.subprocess                                     Database started {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode"}
2024-07-26T17:46:55Z    INFO    lazyfilewalker.trash-cleanup-filewalker.subprocess                                     trash-filewalker completed       {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "numKeysDeleted": 28124, "Process": "storagenode", "bytesDeleted": 26861542400}
2024-07-26T17:46:55Z    INFO    lazyfilewalker.trash-cleanup-filewalker subprocess finished successfully               {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-07-26T17:46:55Z    INFO    lazyfilewalker.trash-cleanup-filewalker starting subprocess                            {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-07-26T17:46:55Z    INFO    lazyfilewalker.trash-cleanup-filewalker subprocess started                             {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-07-26T17:46:55Z    INFO    lazyfilewalker.trash-cleanup-filewalker.subprocess                                     trash-filewalker started {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode", "dateBefore": "2024-07-19T17:46:55Z"}
2024-07-26T17:46:56Z    INFO    lazyfilewalker.trash-cleanup-filewalker.subprocess                                     Database started {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}
2024-07-26T17:46:56Z    INFO    lazyfilewalker.trash-cleanup-filewalker.subprocess                                     trash-filewalker completed       {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode", "bytesDeleted": 0, "numKeysDeleted": 0}
2024-07-26T17:46:56Z    INFO    lazyfilewalker.trash-cleanup-filewalker subprocess finished successfully               {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-07-26T17:46:56Z    INFO    lazyfilewalker.trash-cleanup-filewalker starting subprocess                            {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-07-26T17:46:56Z    INFO    lazyfilewalker.trash-cleanup-filewalker subprocess started                             {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-07-26T17:46:56Z    INFO    lazyfilewalker.trash-cleanup-filewalker.subprocess                                     trash-filewalker started {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode", "dateBefore": "2024-07-19T17:46:56Z"}
2024-07-26T17:46:56Z    INFO    lazyfilewalker.trash-cleanup-filewalker.subprocess                                     Database started {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode"}
2024-07-26T17:46:56Z    INFO    lazyfilewalker.trash-cleanup-filewalker.subprocess                                     trash-filewalker completed       {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "bytesDeleted": 0, "numKeysDeleted": 0, "Process": "storagenode"}
2024-07-26T17:46:56Z    INFO    lazyfilewalker.trash-cleanup-filewalker subprocess finished successfully               {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-07-26T17:46:56Z    INFO    lazyfilewalker.trash-cleanup-filewalker starting subprocess                            {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-07-26T17:46:56Z    INFO    lazyfilewalker.trash-cleanup-filewalker subprocess started                             {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-07-26T17:46:56Z    INFO    lazyfilewalker.trash-cleanup-filewalker.subprocess                                     trash-filewalker started {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode", "dateBefore": "2024-07-19T17:46:56Z"}
2024-07-26T17:46:56Z    INFO    lazyfilewalker.trash-cleanup-filewalker.subprocess                                     Database started {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode"}
2024-07-26T17:46:56Z    INFO    lazyfilewalker.trash-cleanup-filewalker.subprocess                                     trash-filewalker completed       {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode", "bytesDeleted": 0, "numKeysDeleted": 0}
2024-07-26T17:46:56Z    INFO    lazyfilewalker.trash-cleanup-filewalker subprocess finished successfully               {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}

For a 1TB node. All walkers within some minutes…

[pangolin]

ZFS is great for filewalkers but ingress dropped 50% after moving from ext4 to zfs.

[JWvdV]

No problem at my side, even iostat shows the hard drives are all in relaxed spheres since I converted them to ZFS.

But I’m getting the feeling we’re going off track: everything won’t speed up file deletion on ext4. Except for running regularly e2fsck -fD, which takes ages. Or reverting to SSD totally or to another filesystem on which you can split the data (on HDD) and metadata (on SSD) of the filesystem.

[andrew2.hart]

Or my favourite- add another storagenode on the same IP

Edit: When you add another storagenode on the same IP you double your delete speed but only after a few months when there are files on the new node

[JWvdV]

@andrew2.hart

When you add another storagenode on the same IP you double your delete speed but only after a few months when there are files on the new node

How do you mean? Your delete speed might increase a bit, because you’ve less intercurrent random IO because of lower ingress? Than it’s not the delete speed that’s increasing, that’s called less competition I would say.

[littleskunk]

2 drives = 2 times the iops while the amount of TTL deletes keeps the same. So each drive needs to handle a fraction of the total TTL pressure

[Alexey]

We also got a very weird result with test of the filewalker and GC on BTRFS and ext4.
We have a benchmark tool for the filewalkers and the garbage collector: storj/cmd/tools/filewalker-benchmark at main · storj/storj · GitHub

git clone https://github.com/storj/storj.git
cd /mnt/x
go run ~/storj/cmd/tools/filewalker-benchmark

[JWvdV]

@Alexey

We also got a very weird result with test of the filewalker and GC on BTRFS and ext4.

Can you elaborate on this?

[JWvdV]

@littleskunk

So each drive needs to handle a fraction of the total TTL pressure

That fraction is at least 0.5, but probably (much) higher. When you have one node, it probably only handless 5MB/s ingress probably less. If you add another drive, the first thing happening is an increase of ingress, under current circumstances. So probably, you need at least 7 drives or something before it will alleviate your problem even a bit. At this moment I have an ingress of 30-40MB/s.

[Alexey]

Accordingly our tests BTRFS was faster than ext4 on that.
But I would prefer to see results from the Community.

Because other operations usually much slower on BTRFS up to the crash due to failed write/read checks with a 1m0s timeout.

[EasyRhino]

So, DisaSoft made a suggestion of just increasing the number of directory used by storj to reduce the number of files per folder…

Do we know, with the storj nodes right now, if you increase the size of a node (say from 1TB → 10TB) does the number of directories increase, or just the number of files per directory?

Because if it’s files per directory then ext4 (and maybe other file systems) could indeed hit a tipping point where performance collapses when doing file operations.

[Roxor]

@EasyRhino

So, DisaSoft made a suggestion of just increasing the number of directory used by storj to reduce the number of files per folder…

That will probably make things slower… but it’s easy to check. They suggested having more directories (with fewer files-per-directory) but they didn’t actually test spreading the same number of files over more directories.

@EasyRhino

Do we know, with the storj nodes right now, if you increase the size of a node (say from 1TB → 10TB) does the number of directories increase, or just the number of files per directory?

A node gets larger by storing more .sj1 files in the same config/storage/blobs/[satellite]/[prefix] directories. No more layers of subdirectories.

@EasyRhino

Because if it’s files per directory then ext4 (and maybe other file systems) could indeed hit a tipping point where performance collapses when doing file operations.

Storj tailored the algo that splits all the files over those directories to not hold more than modern filesystems support. I think Pentium100 recently posted a 50TB node with no issues with numbers of files. But you’re right that as you simply have more files… all housekeeping tasks take longer. At some point (if you don’t have metadata in RAM/SSD) you’ll spend all your time running filewalkers . Another reason to run around 20TB/node these days?

[DisaSoft]

Finally I made a few more tests which are more close to reality - deleting 1000 random files from millions files spreaded in 1024 folders (“/ab/filename” stricture) and 1000 files from millions spreaded in 1024*32 folders (“/ab/c/filename”) and didn’t find a reason to change folder strucutre. Somewhere “/ab/filename” is faster, somewhere “/ab/c/filename” is faster (depending on number of millions files), but it’s still tens of seconds to delete 1000 old files. And more than 100 seconds when total number of files >= 5 mil.

After that I made similar experiment (only for “/ab/filename” stricture) on ZFS filesystem with metadata cache on SSD. Deletion of 1000 random files from 10mil took… 0.13 seconds.

So I don’t see any reason to stay with ext4 and try to tune is with any small optimizations. It’s just not viable for large node use case. Cache of filesystem metadata looks inevitable.

[donald.m.motsinger]

time ll /mnt/storagenode4/node/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/gd/|wc -l
76368

real    5m48.369s
user    0m0.544s
sys     0m11.422s

time ll /mnt/storagenode4/node/storage/trash/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/2024-07-24/tf/|wc -l
766

real    0m6.558s
user    0m0.001s
sys     0m0.209s

[JWvdV]

Very often referenced: Benchmark: Deep directory structure vs. flat directory structure to store millions of files on ext4 | by hartator | Medium

[agente]

@DisaSoft

Cache of filesystem metadata looks inevitable.

many of us have multiple nodes in ext4 without metadata caching system. Still usefull to understand how make it work on ext4 faster

PS: migrating to zfs will be a VERY long process in some cases

[JWvdV]

@agente

Still usefull to understand how make it work on ext4 faster

We’re saying everytime the same thing: won’t happen, in a way that it really can keep up with the demands. So ext4 just can’t stand the heat at the moment. I’m holding my fingers crossed for the recently implemented file-stat cache.

@agente

PS: migrating to zfs will be a VERY long process in some cases

I know, but in the end it costs me less time now than all errors everytime on ext4.

So, make your choice: wait some weeks hoping file-stat is a big success. Or just convert to full-SSD nodes. Or switch filesystem to to zfs or bcachefs with metadata on SSD.

[DisaSoft]

Fortunately, these nodes are just 2 month old with ingress stopped 3 weeks ago. Almost all data on them is already expired test data, so it’s more easy to lose these 2 month and just to start them from the beginning with new identities than to spend another 2 month for moving expired data.

[JWvdV]

Yeah, meaning two months additional heldback payment. So, why not just convert?

[DisaSoft]

1. Too much efforts in general to save a few bucks.
2. Need another one 10TB+ drive or to kill at least one node anyway to use it’s drive.
3. Even saving of these few bucks is questionable, because until the end of migration every node will stay with expired data and no new ingress, while new node can take ingress immediately.