So in this case… you did something wrong 
Did you used it? You can take a look at what I posted above. That command is literaly copy-pasted from my run command. The node keeps restarting after 10 sec.
Yes, but I’m not on docker. I just added this:
--storage2migration.suppress-central-migration=true
You can use these parameters:
The version v1.133.8 has only these options (the option to opt-out is absent):
docker exec -it storagenode2 /app/bin/storagenode setup --help | sls migrat
--storage2migration.buffer-size int how many pieces to buffer (default 1)
--storage2migration.delay duration constant delay between migration of two pieces. 0 means no delay
--storage2migration.delete-expired whether to also delete expired pieces; has no effect if expired are migrated (default true)
--storage2migration.interval duration how long to wait between pooling satellites for active migration (default 10m0s)
--storage2migration.jitter whether to add jitter to the delay; has no effect if delay is 0 (default true)
--storage2migration.migrate-expired whether to also migrate expired pieces (default true)
--storage2migration.migrate-regardless whether to also migrate pieces for satellites outside currently set
The version v1.135.5 has these options:
PS > & 'C:\Program Files\Storj\Storage Node\storagenode.exe' setup --help | sls migrat
--storage2migration.buffer-size int how many pieces to buffer (default 1)
--storage2migration.delay duration constant delay between migration of two pieces. 0 means no delay
--storage2migration.delete-expired whether to also delete expired pieces; has no effect if expired are migrated (default true)
--storage2migration.interval duration how long to wait between pooling satellites for active migration (default 10m0s)
--storage2migration.jitter whether to add jitter to the delay; has no effect if delay is 0 (default true)
--storage2migration.migrate-expired whether to also migrate expired pieces (default true)
--storage2migration.migrate-regardless whether to also migrate pieces for satellites outside currently set
--storage2migration.suppress-central-migration if true, whether to suppress central control of migration initiation
Seems the option name has been changed. Then the variable name should be STORJ_STORAGE2MIGRATION_SUPPRESS_CENTRAL_MIGRATION=true
@jtolio would you like to update a topic post?
1 Like
As I understand it that sets a path. But that is not what I meant.
From piecestore if you have a file id you could tell -obviously you had the satellite and the storage dir - in which subdir the piece was located. Like from 2CABCDE… → it was the satellite/2c/ dir with piece ABCDE…
Now with hashstore we still have the pieceid but what log file is the piece stored in?
I have no idea. And there is unlikely an easy method to find it out.
But first - why do you need it at all?
2 Likes
I can’t find the Total Pieces Count as reported in the logs by filewalker on the metrics endpoint. Is this not there or am I missing it?
Is there any reason to keep “true” in *.migrate_chore files after it’s done for a specific satellite, or should we flip it back to false if everything has been migrated over?
The initial instruction doesn’t mention it. I guess false will stop migration and data may land in piecestore again.
So, I would like to suggest to do not change it, unless you want to switch to piecestore back.
1 Like
You can keep it on true. It will trigger the migration every 10 minutes and find out that there is nothing to migrate. It does create some log lines.
You can also switch it to false as long as you keep the other values in the .migrate file on true.
2 Likes
There are some additional debugging tools available. GitHub - elek/stbb
stbb hashstore --help
Usage: stbb hashstore <command>
commands related to the new hashtable based store
Flags:
-h, --help Show context-sensitive help.
Commands:
hashstore convert
import data to the hashtable
hashstore list <path>
list content of a hashtable
hashstore stat <path>
list content of a hashtable stat
hashstore compact
compact a hashtable store
hashstore report <path>
show additional reports on a hashtable store
hashstore logs <path> <logs>
show current log file load
hashstore ttl-report <path>
print out ttl expiration per file
hashstore recover
recover hashtable (metadata) from a hashstore log files
hashstore restore-time
get/set restore time for a satellite
hashstore get <path> <id>
get a record from a hashtable
hashstore diff <left> <right>
diff two hashstore
hashstore audit
audit a hashstore: check if pieces are included
hashstore log-read <piece>
find record in hashstore log files without using metadata
Edit: I haven’t used them all. Here are some commands from my bash history that might be usefull:
find ./ -name “log*” -exec /mnt/sn1/binaries/stbb hashstore logs /mnt/sn1/storagenode1/storagenode/storage/hashstore/121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6/s0/meta/ {} >> debug.txt ;
The resultfile will contain informations about the used space, dead space and ttl timestamps of each log file.
Here is another one:
/mnt/sn1/binaries/stbb hashstore list /mnt/sn1/storagenode1/storagenode/storage/hashstore/121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6/s0/meta
This will log all the pieceIDs. It takes a long time to run. Last time I canceled that command. I am planning to run it on a smaller test node. It looks very usefull to track down piece not found errors. I believe it does scan even the dead pieces in the log files. If it finds one we would know it was there but got deleted for some reason. That would be a bug. One item on my todo list. You are welcome to help debugging it 
3 Likes
This looks interesting. Thanks.
Oops, thank you @Alexey, the topic post has been fixed. Sorry everyone!
1 Like
How it can use only 16GB ram on memtbl it is only 0.25GB per TB why recommended 1.3GB per TB? it is very big difference
One value is our estimation and the other one is from my machine. I don’t expect it to stay that low. If compact gets triggered it might need a bit more. I do have 32 GB available so there is room for more. It also depends on the piece size. If the customer decides to upload a lot of tiny pieces my RAM usage should go up a bit.
And if I ever get close to 32 GB RAM usage I would simply switch a few nodes that are already full back to hashtable. So no reason to panic. My recommendation would be to give it a try and see how much RAM it used in your case.
Thank you for explanation, I just want to know how it corelated, because difference is 5x. How can i check ram usage on windows by memtbl? is it some process?
for example i activated memtbl on 1 server, server consume 90GB of RAM(all consumption)
and it has only 38TB of Data
Regardig the rollout, please take note that during migration all allocated space will be reported as full and the node will stop receiving ingress. So, if the rollout is done in big batches, those big batches of nodes will be excluded from taking new pieces for a few weeks. It is solved after migration, with a restart and a finished startup file walker.
1 Like
I can confirm that part of my nodes, show that it is full, looks like some times it same ammount looks like 2x
Why is this a desired behavior?