[Tech Preview] Hashstore backend for storage nodes

[Pentium100]

And if you use NTFS with 64KB clusters (to reduce fragmentatio), one bad sector inside the MFT makes 64 files disappear.

[snorkel]

Sooo… I think I will stay with the current format and badger cash. I just activated it on all nodes.
It’s safer, has the same speeds as hashstore and dosen’t waste 25% of my space.
Curious thing: when updated to 119 ver. a new directory appeared in the storage dir: hashstore.
It has some dirs inside, one is meta, and some files. I didn’t switched to hashstore, they just poped up after update.

[Alexey]

Actually, I have an answer:

the hashtable datastructure is designed to be resilient to localized errors. like, it doesn’t have some small set of bits where if they get corrupted the entire file is corrupted. additionally, the hashtable can be reconstructed from the log files in principle and there’s a test for that but there’s no code yet written to actually reconstruct it

[Alexey]

Yes, it’s expected, see the initial post. After the supported version is released, these folders and files are created automatically.
And if you want to enable, you know how to do it.

[snorkel]

So 2 things have to be worrie about than:

1. The code to reconstruct the hashtable to be written and implemented.
2. Loosing more than 4% of log files.
   Well, Storj can take care of point 1.
   The point 2 is SNOs problem. Got it.

[alpharabbit]

There could be a variable for hashfile directory like we have for databases. So SNOs can use a folder with redundancy for those files. In a low RAM setup it also might be a benefit to have hashfiles on SSD.

[MarviBiene]

@snorkel

Loosing more than 4% of log files

If the chance is high enough that this can happen, why doesn’t the Node have the ability to request the needed information from the satellite? (Or does it?) Wouldn’t it be a good thing, that the node can say “Hey, this file got corrupted and this piece(s) is (are) destroyed”. So the satellite can keep track of healthy pieces too?

And my second question is, how does a satellite know a piece is corrupt? Is there a checksum that will get checked? If someone is requesting a file and a piece is corrupt, how does someone know that this exact piece was wrong?

[snorkel]

The logic is: if you lost a piece, that means you can’t be trusted with that piece anymore. Why they should give you that piece again for keeping?
In the end, this is your main job, to keep pieces healty and available. If you don’t do your job for which you are paid to do, why should they give you the same job again?
And the repair is costly for the network. Why they should pay for repairing the lost pieces in the same spot they were lost the first time, and for which they paid already to be kept safe?

A midle ground would be to have like an option (opt in/out) to pay yourself for repairing the lost/corrupted pieces on your node. But this can be disputed, and many SNO would start arguing that Storj is taking their money with no reason.
You, as a SNO, can’t realy verify the health of pieces; you have to trust the software and the network.

I believe there is a checksum/hash in the header, and the sat compares it with it’s own, or something like that; I didn’t dive into this, but I recall reading something about how it works.
And lost piece means 1 of 3 things: corrupted, unavailable for more than 4 hours (node offline), deleted by operator.

[Alexey]

It works with symlinks, we checked. However, I wouldn’t recommend to do so, especially split hashtables and logs. Especially, if SSD is reused by several nodes - you may lose ALL nodes at once on case of SSD failure.

[Alexey]

@MarviBiene

why doesn’t the Node have the ability to request the needed information from the satellite? (Or does it?)

Because it’s not worth it. See many discussions like

Storage node to Satellite : I've lost this block, sorry

Just a simple scrub. If a block is lost tell the satellite. This is better for both ends. The satellite wants to know and the storagenode would prefer not to get disqualified. The storagenode pays for this leniency by disk bandwidth and power. #enable scrub constant, weekly, monthly #scrub=weekly The satellite manages this using reputation calculations (and held amount) So you can’t use it to cheat or graceful exit

[Aleksman4o]

@Alexey

I wouldn’t recommend to do so, especially split hashtables and logs

Why not? I think it’s good idea to move hashtables to ssd for less fragmentation, especially if it can be reconstructed from logs in future.

[andrew2.hart]

It still sounds like a good idea to me…I used to be so clever!

[Alexey]

It’s a bad idea if the SSD would die, your node will die too.
Please note - not your storage!

[snorkel]

If I start a new node and I want to go with the hashstore since start, is there a different way/command/parameter to use?
Or start as usual and than do the steps described in the first post?

[Alexey]

These folders and files are created when the node check-in on the satellite. So, you need to run it online at least once.
The only thing which I can think of - is to try to pre-create these folders and files after the SETUP step.
I think you can run the script from the first topic, just specify your location and replace false to true.

[vovannovig]

@littleskunk

# Change this path to match your storage node location
cd /mnt/sn1/storagenode/storage/hashstore/meta/

# Enable passive migration. (requires version v1.119)
# WriteToNew will send all incoming uploads to the new hashstore backend
# TTLToNew will send only uploads with a TTL to the new hashstore backend
# ReadNewFirst will migrate any piece that gets hit by a download request
# Not sure what PassiveMigrate does. By the time you think about this one you most likely have them all set to true anyway.
echo '{"PassiveMigrate":false,"WriteToNew":false,"ReadNewFirst":false,"TTLToNew":false}' > 121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6.migrate
echo '{"PassiveMigrate":false,"WriteToNew":false,"ReadNewFirst":false,"TTLToNew":false}' > 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S.migrate
echo '{"PassiveMigrate":false,"WriteToNew":false,"ReadNewFirst":false,"TTLToNew":false}' > 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs.migrate
echo '{"PassiveMigrate":false,"WriteToNew":false,"ReadNewFirst":false,"TTLToNew":false}' > 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE.migrate

# Enable active migration (requires version v1.120)
# Will take multiple days with high CPU time.
# You can enable it for all satellites at the same time.
# The migration will run through them one by one.
echo -n 'false' > 121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6.migrate_chore
echo -n 'false' > 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S.migrate_chore
echo -n 'false' > 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs.migrate_chore
echo -n 'false' > 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE.migrate_chore

please tell me what will happen if “false” is specified and what will happen if everything is replaced with “true”
What’s the logic?

I started it like this(Windows):

# Enable passive migration (requires version v1.119)
Set-Content -Path "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6.migrate" -Value '{"PassiveMigrate":true,"WriteToNew":true,"ReadNewFirst":true,"TTLToNew":true}'
Set-Content -Path "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S.migrate" -Value '{"PassiveMigrate":true,"WriteToNew":true,"ReadNewFirst":true,"TTLToNew":true}'
Set-Content -Path "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs.migrate" -Value '{"PassiveMigrate":true,"WriteToNew":true,"ReadNewFirst":true,"TTLToNew":true}'
Set-Content -Path "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE.migrate" -Value '{"PassiveMigrate":true,"WriteToNew":true,"ReadNewFirst":true,"TTLToNew":true}'

# Enable active migration (requires version v1.120)
Set-Content -Path "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6.migrate_chore" -Value 'true'
Set-Content -Path "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S.migrate_chore" -Value 'true'
Set-Content -Path "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs.migrate_chore" -Value 'true'
Set-Content -Path "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE.migrate_chore" -Value 'true'

I want to figure out how to do it correctly, please help me.

[Aleksman4o]

After node restart on v1.119 all new pieces will go to hashstore, on v1.120 old pieces also will be migrated to hashstore

[Vadim]

Does someone know why hashstore folder have some files, I an not turned anything on yet.

[snorkel]

@Alexey

Yes, it’s expected, see the initial post. After the supported version is released, these folders and files are created automatically.
And if you want to enable, you know how to do it.

Expected behavior. See my post above this one.

[nitrobass24]

Where have you seen that as of 120 we will be migrated to hashstore? I didnt see that in the changelog.