Badger cache filewalker test results

Node Operators
1

System Windows 10 OS.
RAM 24GB RAM
total 10 nodes

HDD 2.5 ST1000LM 1TB HDD
900GB Alocated, it overused to 931 GB so Node is Full
First Run

2024-07-28T13:52:17+03:00 INFO pieces used-space-filewalker started {Satellite ID: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE}
2024-07-28T16:47:32+03:00 INFO pieces used-space-filewalker completed {Satellite ID: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE, Lazy File Walker: false, Total Pieces Size: 591950544640, Total Pieces Content Size: 591609134848}
2024-07-28T16:47:32+03:00 INFO pieces used-space-filewalker started {Satellite ID: 121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6}
2024-07-28T16:49:31+03:00 INFO pieces used-space-filewalker completed {Satellite ID: 121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6, Lazy File Walker: false, Total Pieces Size: 3213761536, Total Pieces Content Size: 3210807808}
2024-07-28T16:49:31+03:00 INFO pieces used-space-filewalker started {Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S}
2024-07-29T00:12:41+03:00 INFO pieces used-space-filewalker completed {Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Lazy File Walker: false, Total Pieces Size: 222399251946, Total Pieces Content Size: 221670944234}
2024-07-29T00:12:41+03:00 INFO pieces used-space-filewalker started {Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs}
2024-07-29T00:28:19+03:00 INFO pieces used-space-filewalker completed {Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Lazy File Walker: false, Total Pieces Size: 18147162624, Total Pieces Content Size: 18121439232}

Second run

2024-07-29T12:35:58+03:00 INFO pieces used-space-filewalker started {Satellite ID: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE}
2024-07-29T12:42:12+03:00 INFO pieces used-space-filewalker completed {Satellite ID: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE, Lazy File Walker: false, Total Pieces Size: 592347381760, Total Pieces Content Size: 592005934080}
2024-07-29T12:42:12+03:00 INFO pieces used-space-filewalker started {Satellite ID: 121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6}
2024-07-29T12:42:40+03:00 INFO pieces used-space-filewalker completed {Satellite ID: 121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6, Lazy File Walker: false, Total Pieces Size: 3214163968, Total Pieces Content Size: 3211208192}
2024-07-29T12:42:40+03:00 INFO pieces used-space-filewalker started {Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S}
2024-07-29T12:51:40+03:00 INFO pieces used-space-filewalker completed {Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Lazy File Walker: false, Total Pieces Size: 222028834834, Total Pieces Content Size: 221302886930}
2024-07-29T12:51:40+03:00 INFO pieces used-space-filewalker started {Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs}
2024-07-29T12:52:59+03:00 INFO pieces used-space-filewalker completed {Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Lazy File Walker: false, Total Pieces Size: 18147162624, Total Pieces Content Size: 18121439232}
5 Likes
2

@agente you can put your test results here

3

@elek can you make path for budger cache configurable? It will more speedup reading, as then we can put it to SSD/NVME with databases, then it will become lightning fast.
As i see it can be problem when there is big upload/download amount, it can be not written or something, like problem with DBs.

1 Like
4

If you use Docker you can mount folder you want with parameter

–mount type=bind,source=“‘[path to folder on SSD/NVME]’”,destination=/app/config/storage/filestatcache

4 Likes
5

What does this mean? First run without and 2nd run with cache? Or both with cache?

6

this mean, that first time it run, it create cache, second time it is with persistent cache, so it faster.

2 Likes
7

so the used space filewalker only took 10 minutes the second pass?

8

17 min for 0.9 TB to be exact

9

That is really encouraging! especially with a slow laptop drive!

Was the badger located on the drive, or on a SSD?

10

I just tried to test there, I used what i have, drive working OK.
It is not possible jet to setup budget to SSD on windows GUI, also it is 7200 hdd.

11

What happens if Badger becomes corrupt, unavailable or deleted for some reason? What is the fallback strategy?

And what size is required per TB?

12

In my setup 0.9 TB cache is 2.25GB

13

I guess disable it and restart. And, perhaps turn the lazy mode on back.

14

No I mean will the node crash or stop working or something?

15

I think there is only one way to know the answer :slight_smile:, please enable it and monitor.
I guess that in a worse case it would crash, or show the incorrect numbers, but I think that just removing and restart would be enough - it will be re-created again.

16

Been doing some testing with Badger filestatcache on a couple of newer nodes with ~5TB data on them.

They’re running on (for comparison context):

  • Plenty of CPU/RAM
  • Ubuntu 22 LTS
  • Docker
  • LSI 2008 Firmware IT (9211-8i)
  • Seagate Exos X16 ST16000
  • 1 disk / 1 node / ext4 with noatime + nodiratime

On avg the performance gain was +70% between initial warmup run and second. Any subsequent runs varies a little, but only a few %.

Theres certainly a large performance gain from the cache on these nodes.
I would assume the speed gain is greater on larger nodes?

Node A
Dataset size: 5.059 TB ~14.8 million pieces
Badger cache size counted with du -ms is 876MB

Initial run / cache warmup (2h55m18s)

2024-07-30T07:24:26Z    INFO    pieces  used-space-filewalker started   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-07-30T09:03:09Z    INFO    pieces  used-space-filewalker completed {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Lazy File Walker": false, "Total Pieces Size": 4126731072256, "Total Pieces Content Size": 4121882582784}
2024-07-30T09:03:09Z    INFO    pieces  used-space-filewalker started   {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-07-30T10:14:41Z    INFO    pieces  used-space-filewalker completed {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Lazy File Walker": false, "Total Pieces Size": 746611870056, "Total Pieces Content Size": 743588001128}
2024-07-30T10:14:41Z    INFO    pieces  used-space-filewalker started   {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-07-30T10:15:53Z    INFO    pieces  used-space-filewalker completed {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Lazy File Walker": false, "Total Pieces Size": 17927263744, "Total Pieces Content Size": 17911560704}
2024-07-30T10:15:53Z    INFO    pieces  used-space-filewalker started   {"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-07-30T10:19:44Z    INFO    pieces  used-space-filewalker completed {"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Lazy File Walker": false, "Total Pieces Size": 167443472384, "Total Pieces Content Size": 167177128960}

Second run (0h41m39s)

2024-07-30T11:05:39Z    INFO    pieces  used-space-filewalker started   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-07-30T11:25:23Z    INFO    pieces  used-space-filewalker completed {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Lazy File Walker": false, "Total Pieces Size": 4177078237184, "Total Pieces Content Size": 4172223609856}
2024-07-30T11:25:23Z    INFO    pieces  used-space-filewalker started   {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-07-30T11:45:09Z    INFO    pieces  used-space-filewalker completed {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Lazy File Walker": false, "Total Pieces Size": 753145557372, "Total Pieces Content Size": 750099064188}
2024-07-30T11:45:09Z    INFO    pieces  used-space-filewalker started   {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-07-30T11:45:21Z    INFO    pieces  used-space-filewalker completed {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Lazy File Walker": false, "Total Pieces Size": 18107989248, "Total Pieces Content Size": 18092201216}
2024-07-30T11:45:21Z    INFO    pieces  used-space-filewalker started   {"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-07-30T11:47:18Z    INFO    pieces  used-space-filewalker completed {"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Lazy File Walker": false, "Total Pieces Size": 168166924800, "Total Pieces Content Size": 167898848256}

Improvement from 1st to 2nd run: (2h55m18s - 0h41m39s) = 2h13m39s ~ 76%

Node B
Dataset size: 5.287 TB ~15.9 million pieces
Badger cache size counted with du -ms is 939MB

Initial run / cache warmup (2h11m22s)

2024-07-30T07:21:50Z    INFO    pieces  used-space-filewalker started   {"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-07-30T07:26:19Z    INFO    pieces  used-space-filewalker completed {"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Lazy File Walker": false, "Total Pieces Size": 159129016064, "Total Pieces Content Size": 158870001408}
2024-07-30T07:26:19Z    INFO    pieces  used-space-filewalker started   {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-07-30T07:26:52Z    INFO    pieces  used-space-filewalker completed {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Lazy File Walker": false, "Total Pieces Size": 17016483584, "Total Pieces Content Size": 17001458432}
2024-07-30T07:26:52Z    INFO    pieces  used-space-filewalker started   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-07-30T08:42:03Z    INFO    pieces  used-space-filewalker completed {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Lazy File Walker": false, "Total Pieces Size": 4351233119488, "Total Pieces Content Size": 4345841282816}
2024-07-30T08:42:03Z    INFO    pieces  used-space-filewalker started   {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-07-30T09:33:12Z    INFO    pieces  used-space-filewalker completed {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Lazy File Walker": false, "Total Pieces Size": 759631409828, "Total Pieces Content Size": 756550708388}

Second run (0h39m10s)

2024-07-30T11:05:38Z    INFO    pieces  used-space-filewalker started   {"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-07-30T11:07:56Z    INFO    pieces  used-space-filewalker completed {"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Lazy File Walker": false, "Total Pieces Size": 161322160384, "Total Pieces Content Size": 161058892032}
2024-07-30T11:07:56Z    INFO    pieces  used-space-filewalker started   {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-07-30T11:08:08Z    INFO    pieces  used-space-filewalker completed {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Lazy File Walker": false, "Total Pieces Size": 17554413568, "Total Pieces Content Size": 17539145728}
2024-07-30T11:08:08Z    INFO    pieces  used-space-filewalker started   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-07-30T11:29:58Z    INFO    pieces  used-space-filewalker completed {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Lazy File Walker": false, "Total Pieces Size": 4406232061696, "Total Pieces Content Size": 4400833520896}
2024-07-30T11:29:58Z    INFO    pieces  used-space-filewalker started   {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-07-30T11:44:48Z    INFO    pieces  used-space-filewalker completed {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Lazy File Walker": false, "Total Pieces Size": 768186856632, "Total Pieces Content Size": 765076884152}

Improvement from 1st to 2nd run: (2h11m22s - 0h39m10s) = 1h32m12s ~ 70%

3 Likes
17

Seems promissing. How can I activate that feature on Windows nodes?
I’m running versions 108.

18

pieces.file-stat-cache: badger and pieces.enable-lazy-filewalker: false in the config, then restart

5 Likes
19

FYI:

  • noatime: do not update access time for anything (files and/or directories)
  • nodiratime: do not update access time for directories (but still update it for files)

and because there is that kind of people between us, here’s the direct man quote:

       noatime
           Do not update inode access times on this filesystem (e.g. for
           faster access on the news spool to speed up news servers). This
           works for all inode types (directories too), so it implies
           nodiratime.

TL;DR: noatime implies nodiratime, you don’t need to use both.

9 Likes
20

Thanks for the heads up. I thought I need to add a new parameter in mount command.

1 Like