Badger cache: are we ready?

Node Operators
61

Budger cache is failing time to time when you just normally restart PC, or i have even updater stoped node for update and it gone broken. this cache is to havy IO for HDD, it need to be moved to SSD, but today there is no possibility for windows users to do that.
I yesterday stoped my pc to swap to new case, 5 nodes cache gone broke.

1 Like
62

I thought it will reduce the load on HDDs?

63

It hold lot of data in 1 place, this reduce lot of long scanning, but when it read or write havy IO happens in small period of time. I tried it on my node that is on NVME, 1,5 TB of data, all was dun within a minut or 2, but 25-45Mbit/s all the time. HDD is not capable of this with all fragmentation. So if we move it to SSD it will help a lot.

64

You can probably use symlinks to move the cache to another place.

65

it used to break when it most needed. Also it overcomplicating setup, this thing usually not work good. Better if it will be like Databases, just can set where it will be.

3 Likes
66

I worry slightly that see a lot of people on a lot of threads enabling Badger cache.
Seems quite risky to me and the official position from Storj (as far as I understand) is that it’s still experimental and not ready for prime-time although preliminary results seem encouraging.

I’m staying well away until it’s officially supported and you can have multiple node databases on the same SSD (seems like that is causing issues at the moment).

4 Likes
67

The risk is minimal, you may always to turn it off, and the node would walk as usual - two weeks or so.

1 Like
68

I’m using Docker.

--pieces.file-stat-cache=badger \
--pieces.enable-lazy-filewalker=false

Can I add this command?

69

I did exactly that in my docker-compose.yaml, it’s working. But you need to provide them after the image name (to pass to the node).

1 Like
70

Thanks, I’ll try it today :slight_smile:

71

quick question do I need to add badger line to every nodes config.yaml or only my main node?

72

Every node is a node. There is no “main node” as such and each one has its own configuration.

Unless I’m missing something obvious?

6 Likes
73

I seems that if in docker we stop and restart a node that it will simply continue using the already built badger cache. If we stop and remove the node then run a fresh instance of it, providing we maintain the same settings will it still make use of whatever existing badger cache or will it expect an entirely new cache to be built?

I suspect it would resume using what cache has already been built, but I’d like to know for sure. Thanks!

74

The cache is held in the storage folder. It should be retained regardless of what you do to your container.

2 Likes
75

agree that’s what I’ve noticed, if I bring docker down the badger cache stays there and seems to be used on restart (still working through my initial tests)

1 Like
76

So… are there any new versions of this badger coming up? Should we wait for a more stable version? Is there an ETA?
On Docker/Linux can we move the badger cache to other drive? How?

77

You just duplicate however youre mounting things now, and change target to be the filestatcache folder within the storage folder on the node, and set source to wherever you want it in your filesystem. For instance in docker compose:

volumes:
   - type: 'bind'
       source: '/storj/storage/filestatcache'
       target: '/app/config/storage/filestatcache'
2 Likes
78

This is my docker startup with datbases and badger cache on nvme:

docker run -d --restart unless-stopped --stop-timeout 300 --network host \
    -e WALLET="xxxx" \
    -e EMAIL="xxxx" \
    -e ADDRESS="xxxx" \
    -e STORAGE="11TB" \
    --mount type=bind,source=/volume1/backup/storj,destination=/app/identity \
    --mount type=bind,source=/volume1/storj,destination=/app/config \
    --mount type=bind,source=/volume2/storjdb,destination=/app/dbs \
    --mount type=bind,source=/volume2/filestatcache,destination=/app/config/storage/filestatcache \
    --name storagenode storjlabs/storagenode:latest --storage2.database-dir="dbs" --storage2.piece-scan-on-startup=true --pieces.enable-lazy-filewalker=false --pieces.file-stat-cache=badger
4 Likes
79

If the badger cache gets corrupted in case of a sudden power down and has to be manualy deleted and then rebuit, why not having a little cache for each folder in the storagenode?
Each 2 char folder for each sat with it’s own badger cache file…
This way, after a sudden power loss or whatever, when a corrupted db is detected, storagenode software should autoremove it and rebuild it or, at least, log the name of the corrupted file to help the SNO remove it manualy.
If the autoremoval is implemented, it can prevent the node crashing, and we don’t have to babysit the nodes every day. This is the reason why I don’t enable it; even though I have UPSes, and SSD, you can’t prevent a sudden power off forever, and I don’t watch nodes everyday.

3 Likes
80

My badger cache survives two updates. So, perhaps only the abrupt restart can broke it?

I do not like anything which starts with “auto” and ended with “removal”. I would like to have a control.
Auto removal of you funds from your account, if there was unauthorized transaction of $10. For me it always sounds like this. Fix is worse than a cause.