[Tech Preview] Hashstore backend for storage nodes

Alexey · January 18, 2025, 6:48am

Is it fully migrated?

CONTAINER ID   NAME                               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
e9e91890cdca   build-storagenode5-1               0.06%     34.2MiB / 1.857GiB    1.80%     1.48MB / 1.32MB   6.48MB / 3.99MB   10
5e23493c3585   build-storagenode11-1              1.88%     54.73MiB / 1.857GiB   2.88%     1.47MB / 3.16MB   7.05MB / 4.53MB   10
d54f58540535   build-storagenode2-1               0.04%     33.96MiB / 1.857GiB   1.79%     1.48MB / 1.32MB   5.31MB / 4.42MB   11
e1ed6a68c961   build-storagenode1-1               0.06%     37.16MiB / 1.857GiB   1.95%     1.51MB / 3.23MB   11.3MB / 3.87MB   10
a9257ba7f4a2   build-storagenode3-1               0.93%     59.65MiB / 1.857GiB   3.14%     1.45MB / 1.32MB   12.1MB / 4.28MB   10

storagenode1 - the whole disk, storagenode2 - piecestore, storagenode3 - badger, storagenode5 - hashstore, storagenode11 - badger and the whole disk

snorkel · January 18, 2025, 7:51am

Interesting. So the badger uses the most CPU and RAM.

Alexey · January 18, 2025, 7:59am

I think I need to configure another one to use hashstore and the whole disk to have a full stack.

CONTAINER ID   NAME                               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
e58b828daaf5   build-storagenode10-1              0.00%     36.14MiB / 1.857GiB   1.90%     16MB / 93.2kB     16.4kB / 34MB     11
e9e91890cdca   build-storagenode5-1               0.00%     46.23MiB / 1.857GiB   2.43%     37.5MB / 3.39MB   6.49MB / 40.7MB   10
5e23493c3585   build-storagenode11-1              0.63%     88.7MiB / 1.857GiB    4.66%     37.4MB / 5.19MB   7.05MB / 41.1MB   10
d54f58540535   build-storagenode2-1               0.00%     46.27MiB / 1.857GiB   2.43%     54.4MB / 3.44MB   5.31MB / 57.9MB   11
e1ed6a68c961   build-storagenode1-1               0.02%     54.8MiB / 1.857GiB    2.88%     71.3MB / 5.32MB   11.3MB / 74.1MB   11
a9257ba7f4a2   build-storagenode3-1               0.99%     93.44MiB / 1.857GiB   4.91%     54.3MB / 3.42MB   12.2MB / 57.9MB   10

done, storagenode10 - hashstore and the whole disk

Mark · January 18, 2025, 6:37pm

TLDR: Add the .migrate_chore file mentioned in the original post even for passive migration.

I noticed unexpected behavior with passive migration on my node running v1.120.4. It seems that even if you are not running active migration, the .migrate_chore file needs to be present (even if the file is empty) for passive migration to work correctly. If the file is not present, passive migration will add data to the hash store but it will not remove the old migrated files from the piece store. As a result I believe I now have some duplicate customer data in both the old piece store and inside the new hash store. Hopefully garbage collection will deal with it.

I used the following linux command from inside my blobs folder to monitor the number of files in my piece store as customers were uploading and downloading files:

find . -type f | wc -l

PS: I also have WriteToNew set to true in addition to PassiveMigrate. The number of files in my blobs folder would remain constant until I added the .migrate_chore file(set to false or set to nothing at all), and now the blobs file count decreases slowly as passive migration works correctly as customers download their files.

Aleksman4o · January 18, 2025, 7:21pm

Old files will not be migrated without .migrate_chore files with “true” values, so they are not deleted

Mark · January 18, 2025, 7:50pm

If I set .migrate_chore to true, it enables active migration. I only want passive migration right now. I have it set to false and it’s still deleting files but in passive mode.

Aleksman4o · January 18, 2025, 7:52pm

It’s not deleting files. Just saving new in hashstore.
Passive mode because of old files will be deleted by garbage collectors and retains in a couple of years and there will be only new files in hashstore.

Mark · January 18, 2025, 8:06pm

The screen shot I posted shows file count decreasing in real time.

Years? haha. ok.

Aleksman4o · January 22, 2025, 6:15am

After migrating to hashstore some nodes has stopped getting ingress. They really have free space and their dashboards shows ~2TB free space on each, but in multinode dashboard i see this

p.s.: increasing storage.allocated-disk-space in config file has no effect

Alexey · January 22, 2025, 6:49am

Of course it will not. Sometimes hashstore enabled nodes reports a negative free space.
The only workaround is to enable also a dedicated disk feature,

unfortunately there is no rollback to piecestore.

Aleksman4o · January 22, 2025, 7:30am

Hmm, now i see a bunch of filewalkers eating my processor time. Node dashboards shows something after several page reload

Alexey · January 23, 2025, 2:02am

The dedicated disk feature is not calculating used space properly for the the dashboard, but at least it reports a correct free space to the satellites.

This PR should fix all used space issues when the hashstore backend is enabled: https://review.dev.storj.io/c/storj/storj/+/16040

Aleksman4o · January 25, 2025, 10:46pm

Yes, it’s working. Hope storj not going to ban me for this version ))

Ambifacient · January 25, 2025, 11:47pm

Believe it or not, straight to jail!

Alexey · January 26, 2025, 3:47am

Depends on with what flags you built it. If you used a developers defaults, then it wouldn’t be selected by production satellites for uploads.

snorkel · January 26, 2025, 9:29am

I like the true date versioning. It tells you for how long bugs stayed unsquashed.

Aleksman4o · January 26, 2025, 10:34am

cd "${_basename}"/web/storagenode && npm install && npm run build && cd ../.. \
    && go get github.com/go-bindata/go-bindata/go-bindata \
    && go-bindata -prefix web/storagenode/ -o storagenode/console/consoleassets/bindata.resource.go -pkg consoleassets web/storagenode/dist/... web/storagenode/static/... \
    && /usr/bin/env echo -e '\nfunc init() { FileSystem = AssetFile() }' >> storagenode/console/consoleassets/bindata.resource.go \
    && gofmt -w -s storagenode/console/consoleassets/bindata.resource.go \
    && git checkout .
    # Build using the release script
    scripts/release.sh build -o build ./cmd/storagenode

There was -fs flag, but i’ve got flag provided but not defined: -fs so i’ve removed it

Alexey · January 27, 2025, 6:58am

You need to add flag after building, --defaults=release

Aleksman4o · January 27, 2025, 8:47am

Not sure where i should add it.

Running go build -o build -defaults=release ./cmd/storagenode
flag provided but not defined: -defaults

But with my variant i see incoming traffic. It seems there release by default )

flo82 · January 27, 2025, 8:53pm

is hashstore enabled by default?

i see a folder …/storage/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/meta/ and some files in it?!