[Tech Preview] Hashstore backend for storage nodes

3 posts were split to a new topic: Piecemigrate:chore couldn’t migrate {“error”: "duplicate pieces differ: headers don’t match

If you wonder about migration speed, here it is:
Synology NAS 216+ 8GB RAM, 2 nodes on 2 IronWolf 8TB.
Node 2 no migration and badger active.
Node 1 migration active and badger on: 10000 pieces/ 12min processed.
I don’t know if the hardware (CPU/RAM) has a noticeable influence. I will see with the next migration on a more powerful machine.
To put into perspective, my biggest node has 11.15TB data, 53798100 pieces; so the migration should take 45 days.

2 Likes

where do you see the migration progress ? i started the migration yesterday evening but the only progress i am seeing is the increasing number of files in the hashstore directory, but nothing in the logs. in 12 hours it went up by ~ 200 GB. So my 8 TB node would take like 20 days (on a 4-core E3 v6 Xeon CPU)

In logs, info level, with some custom filtering. Search my post in “log custom level” thread.
The startup piece scan gives you the no. of pieces on each sat and the migration chore gives you a log entry after each 10.000 pieces processed. I can’t post the specific entries right now, but they are obvious.

1 Like

thanks i see. mine looks like 10.000 pieces/7 minutes. i dont see the no. of pieces in the logs, but my storj folder contains 35 Million files. If 1 piece = 1 file it will be around 17 days for the migration.

1 Like

Sounds reasonable if other I/O is happening. It might become somewhat faster as it progresses, as the node will reference files from the old storage for serving regular traffic less and less.

I would expect offline migration to be 5-10 times faster.

2 Likes

Today I see 10000 pieces/ 7 min, too. Maybe yesterday some cleanup happened.
This looks more promising.

1 Like

I have a fairly new node that I can spare to the test. Is the activation still the same as the OP explained? (asking since has been few months).

Also, hashstore adds any benefit on top of non-hashstore nodes? For example, faster saves, could mean more data?

The activation is the one mentioned in the 1st post. There are fewer files, so this alone should speed up all the walkers. Also, maybe you loose more races if you are on piecestore and the majority are migrated to hashstore. This remains to be tested, but this is in theory.

I have a test setup where a piecestore node and a hashstore node sharing one IP. The piecestore node having slightly better ingress.

I’ve seen this discrepancy between identical nodes sharing the same IP. I’m not sure if the hashstore made any difference on yours.
On a machine with 2 identical nodes as settings and HDD, under the same IP, 1 month age difference between them, 1st one got 186GB bandwidth ingress this month, and the 2nd one got 272GB. No hashstore, just badger active.

Honestly, I don’t think the backend matters for success rate.

1 Like

actually this bug is quite useful to see how much data remains to be migrated :smiley: I kinda hope the fix won’t reach my node until the migration is over

2 Likes

workflow

10 Likes

Migrating my nodes in to hashstore, I noticed a drop in used space in my GRAFANA and in the Storj GUI interface, it’s a knowed bug?

1 Like

Yeah hashstore disk space is not calculated, fixed in 1.122.

4 Likes

Hi,

I have followed the first post, setting .migrate_chore to true and the options in .migrate all to true for each satellite. This was all set in mid-January, and it looked like it was migrating well.

But it looks to have stopped with the following error in the Storj Logs.

2025-02-10T20:14:14Z INFO piecemigrate:chore couldn’t migrate {“Process”: “storagenode”, “error”: “opening the old reader: pieces error: invalid piece file for storage format version 1: too small for header (0 < 512)”, “errorVerbose”: “opening the old reader: pieces error: invalid piece file for storage format version 1: too small for header (0 < 512)\n\tstorj.io/storj/storagenode/piecemigrate.(*Chore).migrateOne:318\n\tstorj.io/storj/storagenode/piecemigrate.(*Chore).processQueue:260\n\tstorj.io/storj/storagenode/piecemigrate.(*Chore).Run.func2:167\n\tstorj.io/common/errs2.(*Group).Go.func1:23”, “sat”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “id”: “TQR6KSBAOJJZ3I77D4M7VKBMS6KWPO5NY44ZFTBH43VBMGJP5GJA”}

Can anyone help with this issue?

Storj Version 1.2.1.2, running on RPI 4 (6.6.62+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.6.62-1+rpt1 (2024-11-25) aarch64)

1 Like

Might be a corrupt piece. I’d get that file out of there. Based on the satellite ID, I think it will be in the EU1 blobs folder called: v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa
Look in the two letter “TQ” prefix folder for a file called: R6KSBAOJJZ3I77D4M7VKBMS6KWPO5NY44ZFTBH43VBMGJP5GJA

Try removing that file (after you back it up to a safe place) and see if the migration continues without error.

2 Likes

Hi Mark,

Thanks for the info. I found the file in the QX folder, and it was zero in size.
After restarting the node, I will monitor the logs for any other migration errors.

1 Like

my migration is almost over, but i stopped getting new ingress 2 days ago.

there should be still a 1TB free space available from the allocated 10 TB. The disk shows 7.81TB of used space.

maybe its related somehow to the overhead the hashstore brings?