Hashstore rollout commencing!

[Ottetal]

Very nice! Questions: Is there a place where a user can go and look, so see how many nodes on what sattelites have begun active migration, pushed from your side?

Also; once all migrations are finished, will StorJ push code changes to get rid of ´blobs´ directory and its presumably many damaged files from some users perhaps less than stellar setups?

Kind regards Elek

[elek]

@Ottetal

Is there a place where a user can go and look, so see how many nodes on what sattelites have begun active migration, pushed from your side?

Unfortunately there is no such page

@Ottetal

Also; once all migrations are finished, will StorJ push code changes to get rid of ´blobs´ directory and its presumably many damaged files from some users perhaps less than stellar setups?

The exact plan is still forming, but yes, we are planning to fully migrate to hashstore, and remove old code + old garbage. The exact approach is still under discussion and once we have a plan, it will be published on this forum

[Ottetal]

Thank you so much, for the always swift and accurate responses. Good day to you

[jammerdan]

@Alexey

This chore can delete only zero-sized pieces and their subfolders.

I am still seeing mixed results and the removal of subfolders does not seem to work as I expect it:

Initially after migration I had set the cores back to false to avoid the empty subfolders to be re-queried over an over again. I never manually deleted them.
Then obviously the central migration started. But I see different results on the nodes.

For example I have:

=== storagenode 1 ===
  ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa: 1024 subfolders

but

=== storagenode 2 ===
  ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa: 1024 subfolders
  v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa: 1024 subfolders

On 1 I see 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S.migrate_chore set to false with file modification date as of today and 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs.migrate_chore set to true with file modification date 1st of July. This is consistent with the result that the EU satellite folder has been completely removed.

While on storagenode 2 I see 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S.migrate_chore set to false and 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs.migrate_chore also set to false, both with file modification date as of today. This is also consistent with both folders still there.

But it is not consistent over all nodes. Some have all satellite folders removes. Some still have all subfolders although empty.

And I am wondering about the file modifaction dates and if their values are correctly set. As we had it that in the past that the values were resetted to previous values:

[Tech Preview] Hashstore backend for storage nodes

It is still weird in the light of what I have just observed: I had set everything to true in the files with migrate_chore and migrate suffix. So for example: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs.migrate_chore > true and 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs.migrate > {"PassiveMigrate":true,"WriteToNew":true,"ReadNewFirst":true,"TTLToNew":true} Then I have restarted the node. After 16 hours when I checked, the files with suffix migrate_chore contained no longer tr…

[Alexey]

Yes, enabling is performed gradually, so it might not be enabled on one and might be enabled on another.

[jammerdan]

But why am I seeing the changes of the timestamps then?
File last modified:

 5 Jul 15 06:50 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs.migrate_chore

which is 20 minutes ago. Now set to false. It would be a strange coincidence if the central migration just set the value to false. Which means it had to be set to true before. However we are still at

=== storagenode 2 ===
  ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa: 1024 subfolders
  v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa: 1024 subfolders

Also weird:
Based on metric function{scope="storj_io_storj_storagenode_piecemigrate",name="__Chore__cleanupEmptyDirectories",field="successes"} 222 as I understand it it was supposed to cleanup 222 directories already. However we are still seeing the number of subfolders at 1024. Isn’t this function supposed to delete the subfolder if they are found to be empty?

[Alexey]

@jammerdan

But why am I seeing the changes of the timestamps then?
File last modified:

I guess this is how this central migration works.

@jammerdan

function{scope="storj_io_storj_storagenode_piecemigrate",name="__Chore__cleanupEmptyDirectories",field="successes"} 222 as I understand it it was supposed to cleanup 222 directories already. However we are still seeing the number of subfolders at 1024. Isn’t this function supposed to delete the subfolder if they are found to be empty?

It should. Did you check these directories, are they empty?

[jammerdan]

@Alexey

I guess this is how this central migration works.

How? Timestamp is now: 5 Jul 15 07:50 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs.migrate_chore

Does the central migration modify this file every hour?

@Alexey

It should. Did you check these directories, are they empty?

# find "storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/" -type d -empty | wc -l     
1024

[jammerdan]

@jammerdan

function{scope=“storj_io_storj_storagenode_piecemigrate”,name=“__Chore__cleanupEmptyDirectories”,field=“successes”} 222

Now at:
function{name="__Chore__cleanupEmptyDirectories",scope="storj_io_storj_storagenode_piecemigrate",field="successes"} 240 although

2026-07-15T06:40:23Z    INFO    piecemigrate:chore      all enqueued for migration; will sleep before next pooling      {"process": "storagenode", "active": {"121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6": true, "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S": false, "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs": false, "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE": true}, "interval": "10m0s"}

and

=== storagenode 2 ===
  ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa: 1024 subfolders
  v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa: 1024 subfolders

[Alexey]

Possible, I don’t know. Likely after each check-in, which is happening hourly by default. The file is modified not by the satellite, but by the node.
I would like to suggest to let it run and finish. When the migration will be finished, folders should be deleted. However, you may do it yourself since it’s concerning you, just make sure that they are truly empty.

[jammerdan]

@Alexey

since it’s concerning you

What concerns me are the ambigious or contradicting signals that do not give a clear indication what is going on.
E.G: If migration chore is set to false for both satellites US and EU, why is the the cleanupEmptyDirectories success chore counting up succesful directory cleanups. And if there succesful directory cleanups, why don’t they get deleted?
What I am seeing is not what I am expecting to see to be able to say it is working.
Also why is the process so slow? Checking empty subfolders and deleting them should not take weeks or months. Deleting 1024 empty subfolders should take seconds. But now we are at function{name="__Chore__cleanupEmptyDirectories",scope="storj_io_storj_storagenode_piecemigrate",field="successes"} 246. So +2 in 5 minutes? What is this thing even doing as migration chore is set to false?
This all just looks weird to me.

[Alexey]

@jammerdan

why is the the cleanupEmptyDirectories success chore counting up succesful directory cleanups.

Perhaps you need to restart the node to apply the change, until then it’s likely considering it as still enabled. The second thing - the passive migration is likely enabled, so it migrates remained pieces when they requested.

@jammerdan

Also why is the process so slow?

It runs after the migration is completed and will fail, if there are some files remained. Perhaps you might have migration errors too.

[dlmorgan]

So, there are a couple issues here it seems.

1. I verified that yes, the node is modifying the file. While a satellite’s flag is still false, every check-in rewrites the .migrate_chore file regardless of whether the value actually changed. So the mtime bumps hourly with zero real signal until the day it flips true.

2. On the counter incrementing, the counter increases are likely coming from your other two satellites (the ones showing true), not EU/US. Neither of those two counters are tagged per-satellite. They’re global across all the satellites, so you can’t tell from the metrics alone which satellite is being cleaned.

Those EU/US subfolders are likely still empty because cleanup for them hasn’t run at all yet since their flag is still false. Once it flips true centrally (or you flip it true yourself), it should clear out within one 10-minute cycle. If you want to flip it yourself, i’d recomend flipping it by bringing the node down first, changing the value, then starting it up again.

To track the actual progress, I recommend watching each satellite’s own active bool in the “all enqueued for migration…” log line rather than the aggregate counters.

On another note, I think there is a bug here that should be fixed. There are some false positives in the metric, so i’ll look into getting that fixed.

[jammerdan]

@dlmorgan

While a satellite’s flag is still false, every check-in rewrites the .migrate_chore file regardless of whether the value actually changed. So the mtime bumps hourly with zero real signal until the day it flips true.

That explains it. Thank you for confirming that.

@dlmorgan

On the counter incrementing, the counter increases are likely coming from your other two satellites (the ones showing true), not EU/US. Neither of those two counters are tagged per-satellite.

I am aware that the counter is not per satellite. But how can they increase if there are no folders left for the 2 other satellites?

storagenode 2
ls /storage/blobs
ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa  v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa

=== storagenode 2 ===
  ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa: 1024 subfolders
  v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa: 1024 subfolders

But still:

function{name="__Chore__cleanupEmptyDirectories",scope="storj_io_storj_storagenode_piecemigrate",field="successes"} 262

Now at 262? while still
12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S.migrate_chore false and 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs.migrate_chore false.

Whatever cleanup is happening, it is not visible.
Maybe the increase due to this?

@Alexey

The second thing - the passive migration is likely enabled

Passive migration is set to true of course. Is this supposed to trigger the cleanupEmptyDirectories chore? In case it does, why does it not delete the empty subfolders? I am just seeing the counter increase but nothing happens.

It is simply not clear what to expect and what it is actually doing.

@dlmorgan

If you want to flip it yourself, i’d recomend flipping it by bringing the node down first, changing the value, then starting it up again.

This is what I am doing right now on all of my nodes: Manually switching the migration back on. Let’s see what will happen to the empty folders. My expectation is that they should get deleted.

@dlmorgan

To track the actual progress, I recommend watching each satellite’s own active bool in the “all enqueued for migration…” log line rather than the aggregate counters.

I will check on that.

[dlmorgan]

@jammerdan

I am aware that the counter is not per satellite. But how can they increase if there are no folders left for the 2 other satellites?

This may be related to the bug I mentioned. There are some false positives in that metric that need to be fixed. I will work on this, and push out a new release with the fix.

[jammerdan]

@dlmorgan

i’d recomend flipping it by bringing the node down first, changing the value, then starting it up again.

For some reason I can see on some nodes (not all) that the value reverts back to false although it has been done exactly like you have proposed. Any ideas why this is happening? Of course this means that the subfolders on these nodes have not been deleted yet.

[Alexey]

@Alexey

enabling is performed gradually, so it might not be enabled on one and might be enabled on another.

and as turned out, it also can write false, if the node is not eligible

@Alexey

Likely after each check-in, which is happening hourly by default. The file is modified not by the satellite, but by the node.

[jammerdan]

So the only way would be to additionally turn off/supress central migration with setting

# if true, whether to suppress central control of migration initiation
# storage2migration.suppress-central-migration: false

?

[Alexey]

Like it’s stated in the description of this option.

[Roxor]

Are we… maybe not going to have the US Satellite signal hashstore migration?