Badger cache filewalker test results

Here comparison of same thing, but than using ZFS with meta on SSD:

STORJ12
2024-08-12T06:39:24Z    INFO    lazyfilewalker.gc-filewalker    starting subprocess                                    {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-08-12T06:39:25Z    INFO    lazyfilewalker.gc-filewalker    subprocess started                                     {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-08-12T06:39:25Z    INFO    lazyfilewalker.gc-filewalker.subprocess Database started                               {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}2024-08-12T06:39:25Z    INFO    lazyfilewalker.gc-filewalker.subprocess gc-filewalker started                          {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "bloomFilterSize": 1847427, "Process": "storagenode", "createdBefore": "2024-08-06T17:59:59Z"}
2024-08-12T06:40:08Z    INFO    lazyfilewalker.gc-filewalker.subprocess gc-filewalker completed                        {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode", "piecesCount": 2604091, "trashPiecesCount": 5456, "piecesTrashed": 5456, "piecesSkippedCount": 0}
2024-08-12T06:40:08Z    INFO    lazyfilewalker.gc-filewalker    subprocess finished successfully                       {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
# 2.6M in 44s, trashed 5.5k

STORJ20
2024-08-12T09:38:05Z    INFO    lazyfilewalker.gc-filewalker    starting subprocess                                    {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-08-12T09:38:05Z    INFO    lazyfilewalker.gc-filewalker    subprocess started                                     {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-08-12T09:38:05Z    INFO    lazyfilewalker.gc-filewalker.subprocess Database started                               {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}2024-08-12T09:38:05Z    INFO    lazyfilewalker.gc-filewalker.subprocess gc-filewalker started                          {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "bloomFilterSize": 1792288, "Process": "storagenode", "createdBefore": "2024-08-06T17:59:59Z"}
2024-08-12T09:38:53Z    INFO    lazyfilewalker.gc-filewalker.subprocess gc-filewalker completed                        {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "trashPiecesCount": 13554, "piecesTrashed": 13554, "piecesSkippedCount": 0, "Process": "storagenode", "piecesCount": 2706347}
2024-08-12T09:38:53Z    INFO    lazyfilewalker.gc-filewalker    subprocess finished successfully                       {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
# 2.7M in 48s, trashed 13.5k

STORJ3 # only one still using ext4, lazy + 
2024-08-11T15:52:52Z    INFO    lazyfilewalker.gc-filewalker    starting subprocess                                    {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-08-11T15:52:52Z    INFO    lazyfilewalker.gc-filewalker    subprocess started                                     {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-08-11T15:52:54Z    INFO    lazyfilewalker.gc-filewalker.subprocess Database started                               {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}2024-08-11T15:52:54Z    INFO    lazyfilewalker.gc-filewalker.subprocess gc-filewalker started                          {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode", "createdBefore": "2024-08-04T17:59:59Z", "bloomFilterSize": 156143}
2024-08-11T16:11:58Z    INFO    lazyfilewalker.gc-filewalker.subprocess gc-filewalker completed                        {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode", "piecesCount": 234338, "trashPiecesCount": 19040, "piecesTrashed": 19040, "piecesSkippedCount": 0}
2024-08-11T16:11:59Z    INFO    lazyfilewalker.gc-filewalker    subprocess finished successfully                       {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
#0.23M in 19m6s, trashing 19k

STORJ7
2024-08-12T06:29:00Z    INFO    lazyfilewalker.gc-filewalker    starting subprocess                                    {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-08-12T06:29:00Z    INFO    lazyfilewalker.gc-filewalker    subprocess started                                     {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-08-12T06:29:01Z    INFO    lazyfilewalker.gc-filewalker.subprocess Database started                               {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}2024-08-12T06:29:01Z    INFO    lazyfilewalker.gc-filewalker.subprocess gc-filewalker started                          {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode", "createdBefore": "2024-08-06T17:59:59Z", "bloomFilterSize": 2048585}
2024-08-12T06:31:48Z    INFO    lazyfilewalker.gc-filewalker.subprocess gc-filewalker completed                        {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "piecesTrashed": 5080, "piecesSkippedCount": 0, "Process": "storagenode", "piecesCount": 2296377, "trashPiecesCount": 5080}
2024-08-12T06:31:48Z    INFO    lazyfilewalker.gc-filewalker    subprocess finished successfully                       {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
# 2.3M in 2m48s, trashing 5k

STORJ11
2024-08-12T02:10:21Z    INFO    lazyfilewalker.gc-filewalker    starting subprocess                                    {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-08-12T02:10:21Z    INFO    lazyfilewalker.gc-filewalker    subprocess started                                     {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-08-12T02:10:21Z    INFO    lazyfilewalker.gc-filewalker.subprocess Database started                               {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}2024-08-12T02:10:21Z    INFO    lazyfilewalker.gc-filewalker.subprocess gc-filewalker started                          {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode", "createdBefore": "2024-08-06T17:59:59Z", "bloomFilterSize": 654238}
2024-08-12T02:10:27Z    INFO    lazyfilewalker.gc-filewalker.subprocess gc-filewalker completed                        {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "piecesTrashed": 469, "piecesSkippedCount": 0, "Process": "storagenode", "piecesCount": 374739, "trashPiecesCount": 469}
2024-08-12T02:10:27Z    INFO    lazyfilewalker.gc-filewalker    subprocess finished successfully                       {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
# 0.37M in 6s, trashing 469

STORJ16
2024-08-12T02:26:36Z    INFO    lazyfilewalker.gc-filewalker    subprocess started                                     {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-08-12T02:26:36Z    INFO    lazyfilewalker.gc-filewalker.subprocess Database started                               {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}2024-08-12T02:26:36Z    INFO    lazyfilewalker.gc-filewalker.subprocess gc-filewalker started                          {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "createdBefore": "2024-08-06T17:59:59Z", "bloomFilterSize": 251871, "Process": "storagenode"}
2024-08-12T02:26:40Z    INFO    lazyfilewalker.gc-filewalker.subprocess gc-filewalker completed                        {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode", "piecesCount": 327556, "trashPiecesCount": 226, "piecesTrashed": 226, "piecesSkippedCount": 0}
2024-08-12T02:26:40Z    INFO    lazyfilewalker.gc-filewalker    subprocess finished successfully                       {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
# 0.33M in 4s, trashing 226

STORJ17
2024-08-12T10:42:19Z    INFO    lazyfilewalker.gc-filewalker    starting subprocess                                    {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-08-12T10:42:19Z    INFO    lazyfilewalker.gc-filewalker    subprocess started                                     {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-08-12T10:42:19Z    INFO    lazyfilewalker.gc-filewalker.subprocess Database started                               {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}2024-08-12T10:42:19Z    INFO    lazyfilewalker.gc-filewalker.subprocess gc-filewalker started                          {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "createdBefore": "2024-08-06T17:59:59Z", "bloomFilterSize": 240654, "Process": "storagenode"}
2024-08-12T10:42:23Z    INFO    lazyfilewalker.gc-filewalker.subprocess gc-filewalker completed                        {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "trashPiecesCount": 364, "piecesTrashed": 364, "piecesSkippedCount": 0, "Process": "storagenode", "piecesCount": 161390}
2024-08-12T10:42:23Z    INFO    lazyfilewalker.gc-filewalker    subprocess finished successfully                       {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
# 0.16M in 4s, trashing 364.

STORJ22
2024-08-12T12:06:08Z    INFO    lazyfilewalker.gc-filewalker    starting subprocess                                    {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-08-12T12:06:08Z    INFO    lazyfilewalker.gc-filewalker    subprocess started                                     {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-08-12T12:06:08Z    INFO    lazyfilewalker.gc-filewalker.subprocess Database started                               {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}2024-08-12T12:06:08Z    INFO    lazyfilewalker.gc-filewalker.subprocess gc-filewalker started                          {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode", "createdBefore": "2024-08-06T17:59:59Z", "bloomFilterSize": 2114666}
2024-08-12T12:06:19Z    INFO    lazyfilewalker.gc-filewalker.subprocess gc-filewalker completed                        {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode", "piecesCount": 752278, "trashPiecesCount": 1833, "piecesTrashed": 1833, "piecesSkippedCount": 0}
2024-08-12T12:06:19Z    INFO    lazyfilewalker.gc-filewalker    subprocess finished successfully                       {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
# 0.75M in 11s, trashing 1.8k

STORJ9
2024-08-12T11:44:49Z    INFO    lazyfilewalker.gc-filewalker    starting subprocess                                    {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-08-12T11:44:49Z    INFO    lazyfilewalker.gc-filewalker    subprocess started                                     {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-08-12T11:44:49Z    INFO    lazyfilewalker.gc-filewalker.subprocess Database started                               {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}2024-08-12T11:44:49Z    INFO    lazyfilewalker.gc-filewalker.subprocess gc-filewalker started                          {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode", "createdBefore": "2024-08-06T17:59:59Z", "bloomFilterSize": 1583738}
2024-08-12T11:45:16Z    INFO    lazyfilewalker.gc-filewalker.subprocess gc-filewalker completed                        {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode", "piecesCount": 2189290, "trashPiecesCount": 6066, "piecesTrashed": 6066, "piecesSkippedCount": 0}
2024-08-12T11:45:16Z    INFO    lazyfilewalker.gc-filewalker    subprocess finished successfully                       {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
# 2.2M in 27s, trashing 6k

So, yeah badger (on XFS) is a real improvement. But still taking about a factor 8x ZFS+meta on SSD. Although apparently already about 7x quicker than just ext4 on SMR.

5 Likes