Hashstore rollout commencing!

jammerdan · May 26, 2026, 3:31am

There is one thing that I have just noticed today: On several nodes that have already completed migration, the switch only in 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE.migrate_chore has reverted to true. This comes after migration had already been fully completed and all migration switched had been set to false to prevent further IO spent on this.
All other migration chore files have their switches set to false. From the date of the file it seems the changes has been made on the 15th of May.

Has the switch been set by Storj?

Alexey · May 26, 2026, 5:42am

Or by your docker. Did you remove the container before changing these files?

But perhaps you are correct. I saw a message on May 15 for active migration on SLC 100% (it was gradual)

jammerdan · May 26, 2026, 5:47am

Yes. Each container was stopped and removed before setting the values to false.

I see modification date 14th of May on one node and 15th on multiple others. That would fit.

Alexey · May 26, 2026, 5:51am

It’s possible that the storage node synchronized the state from the satellite, at least the date matches.

jammerdan · May 26, 2026, 5:53am

Ok, but the nodes are full migrated. Can we prevent it somehow that it keeps turning itself back on?

Alexey · May 26, 2026, 5:57am

What is issue with having it enabled, if blobs are empty? Also you may try to change it and set as read-only.

also

$ storagenode setup --help | grep migr
...
      --storage2migration.suppress-central-migration             if true, whether to suppress central control of
migration initiation
...

jammerdan · May 26, 2026, 6:03am

This:

It shold not turn itself back on. I mean the problem is not that I can’t disable it again. The problem is that I would have to monitor the status for each satellite per node if Storj turns it on and off whenever they want to. If I turn it off it shold remain off.

Alexey · May 26, 2026, 6:03am

You don’t. You can disable it entirely.

Right. But now there is no directories after successful migration, isn’t it?

jammerdan · May 26, 2026, 6:15am

Does this work in the Docker run command?

There are still those 2-letter subfolders. Mostly empty but some may contain broken pieces in it.

Alexey · May 26, 2026, 6:36am

Yes, you need to put it after the image name.

I thought you cleaned up them? It also should remove empty directories after a successful migration:

github.com/storj/storj

storagenode/piecemigrate/chore.go

a3b8d3a88


      
          }
          
          // cleanupEmptyDirectories attempts to remove empty prefix directories for a satellite.
          // It tries os.Remove on each prefix directory directly — empty directories are removed,
          // non-empty ones fail with ENOTEMPTY instantly. This avoids expensive ReadDir calls on
          // previously-large directories whose on-disk block structure is still huge.
          //
          // For directories that can't be removed, a second pass checks for zero-sized file
          // debris (artifacts from incomplete writes). If a directory contains only zero-sized
          // files older than minZeroFileAge, those files are deleted and removal is retried.
          func (chore *Chore) cleanupEmptyDirectories(ctx context.Context, satellite storj.NodeID) {
          	defer mon.Task()(&ctx)(nil)
          
          	if chore.oldBlobsPath == "" {
          		return
          	}
          
          	if chore.writeChecker != nil && !chore.writeChecker.IsWritingToNew(satellite) {
          		chore.log.Debug("skipping empty directory cleanup: satellite is still writing to old backend",
          			zap.Stringer("sat", satellite))
          		return

Looks like this was never enabled? I found only mentions in tests.

jammerdan · May 26, 2026, 6:40am

as --storage2migration.suppress-central-migration=true or --storage2migration.suppress-central-migration="true" or --storage2migration.suppress-central-migration true?

No. I couldn’t be bothered to do that manually. And some are still there so it does not seem to happen automatically, I don’t know.

Alexey · May 26, 2026, 6:45am

All three are valid.
I checked with CoPilot, the cleanup is happening during migration. So if all pieces are migrated, all their folders should be removed as well, including 2-letters directories. It even removes zero-sized objects and their directories. But didn’t remove non zero files and non empty directories.

jammerdan · May 26, 2026, 6:48am

Thanks. Will add that option then.

Well all I can say is that I can still see the subfolders… At some point I will have to remove them manuall if automatic removal is not working.

Alexey · May 26, 2026, 6:50am

It is performed only during migration and never after. So if they are left, then they are likely not empty and you need to cleanup them manually, if the remained pieces cannot be migrated (you should see errors in your logs).

jammerdan · May 26, 2026, 6:57am

They look very empty to me:

ls storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/2j | wc -l
0

Maybe automatic removal was not implemented when I migrated or it did not work. I have no idea. All I can say is that the folders are there and on most nodes completely empty.

elek · May 26, 2026, 8:31am

The mentioned problem (slow ext4 with empty dirs) shouldn’t be a problem any more, as newest version cleans up the empty (but slow) directories.

Actually it makes more sense to turn in migration (at least a short period of time).

It’s merged in 1.149:

What is your version?

jammerdan · May 26, 2026, 8:57am

That’s interesting. Does it cleanup even though migration has been completed already? In that case the best way might be indeed to set the migration chores back to true.
What will happen if broken pieces are left in subfolders?

Alexey · May 27, 2026, 6:38am

Subfolders with broken pieces will remain until you delete them. This chore can delete only zero-sized pieces and their subfolders.

Roxor · May 28, 2026, 3:07pm

Has Storj only enabled active-migration of SLC so far, or have a portion of nodes begun for other satellites too?

elek · May 28, 2026, 3:16pm

SLC is full done, AP is in progres (I think it’s at 50% now). US1 and EU1 are not yet touched.