3 posts were split to a new topic: Piecemigrate:chore couldn’t migrate {“error”: "duplicate pieces differ: headers don’t match
If you wonder about migration speed, here it is:
Synology NAS 216+ 8GB RAM, 2 nodes on 2 IronWolf 8TB.
Node 2 no migration and badger active.
Node 1 migration active and badger on: 10000 pieces/ 12min processed.
I don’t know if the hardware (CPU/RAM) has a noticeable influence. I will see with the next migration on a more powerful machine.
To put into perspective, my biggest node has 11.15TB data, 53798100 pieces; so the migration should take 45 days.
where do you see the migration progress ? i started the migration yesterday evening but the only progress i am seeing is the increasing number of files in the hashstore directory, but nothing in the logs. in 12 hours it went up by ~ 200 GB. So my 8 TB node would take like 20 days (on a 4-core E3 v6 Xeon CPU)
In logs, info level, with some custom filtering. Search my post in “log custom level” thread.
The startup piece scan gives you the no. of pieces on each sat and the migration chore gives you a log entry after each 10.000 pieces processed. I can’t post the specific entries right now, but they are obvious.
thanks i see. mine looks like 10.000 pieces/7 minutes. i dont see the no. of pieces in the logs, but my storj folder contains 35 Million files. If 1 piece = 1 file it will be around 17 days for the migration.
Sounds reasonable if other I/O is happening. It might become somewhat faster as it progresses, as the node will reference files from the old storage for serving regular traffic less and less.
I would expect offline migration to be 5-10 times faster.
Today I see 10000 pieces/ 7 min, too. Maybe yesterday some cleanup happened.
This looks more promising.
I have a fairly new node that I can spare to the test. Is the activation still the same as the OP explained? (asking since has been few months).
Also, hashstore adds any benefit on top of non-hashstore nodes? For example, faster saves, could mean more data?
The activation is the one mentioned in the 1st post. There are fewer files, so this alone should speed up all the walkers. Also, maybe you loose more races if you are on piecestore and the majority are migrated to hashstore. This remains to be tested, but this is in theory.
I have a test setup where a piecestore node and a hashstore node sharing one IP. The piecestore node having slightly better ingress.
I’ve seen this discrepancy between identical nodes sharing the same IP. I’m not sure if the hashstore made any difference on yours.
On a machine with 2 identical nodes as settings and HDD, under the same IP, 1 month age difference between them, 1st one got 186GB bandwidth ingress this month, and the 2nd one got 272GB. No hashstore, just badger active.
Honestly, I don’t think the backend matters for success rate.
actually this bug is quite useful to see how much data remains to be migrated I kinda hope the fix won’t reach my node until the migration is over
Migrating my nodes in to hashstore, I noticed a drop in used space in my GRAFANA and in the Storj GUI interface, it’s a knowed bug?
Yeah hashstore disk space is not calculated, fixed in 1.122.
Hi,
I have followed the first post, setting .migrate_chore to true and the options in .migrate all to true for each satellite. This was all set in mid-January, and it looked like it was migrating well.
But it looks to have stopped with the following error in the Storj Logs.
2025-02-10T20:14:14Z INFO piecemigrate:chore couldn’t migrate {“Process”: “storagenode”, “error”: “opening the old reader: pieces error: invalid piece file for storage format version 1: too small for header (0 < 512)”, “errorVerbose”: “opening the old reader: pieces error: invalid piece file for storage format version 1: too small for header (0 < 512)\n\tstorj.io/storj/storagenode/piecemigrate.(*Chore).migrateOne:318\n\tstorj.io/storj/storagenode/piecemigrate.(*Chore).processQueue:260\n\tstorj.io/storj/storagenode/piecemigrate.(*Chore).Run.func2:167\n\tstorj.io/common/errs2.(*Group).Go.func1:23”, “sat”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “id”: “TQR6KSBAOJJZ3I77D4M7VKBMS6KWPO5NY44ZFTBH43VBMGJP5GJA”}
Can anyone help with this issue?
Storj Version 1.2.1.2, running on RPI 4 (6.6.62+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.6.62-1+rpt1 (2024-11-25) aarch64)
Might be a corrupt piece. I’d get that file out of there. Based on the satellite ID, I think it will be in the EU1 blobs folder called: v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa
Look in the two letter “TQ” prefix folder for a file called: R6KSBAOJJZ3I77D4M7VKBMS6KWPO5NY44ZFTBH43VBMGJP5GJA
Try removing that file (after you back it up to a safe place) and see if the migration continues without error.
Hi Mark,
Thanks for the info. I found the file in the QX folder, and it was zero in size.
After restarting the node, I will monitor the logs for any other migration errors.