[Tech Preview] Hashstore backend for storage nodes

[littleskunk]

@nerdatwork

Did I interpret it correctly ?

No. I guess I have to explain that part a bit more in detail.
Hashtable needs 0 bytes of memory but it writes 128 bytes of metadata per piece.
Memtable needs 18 bytes of memory but it cuts the metadata in halve so 64 bytes of metadata. You can store that metadata on HDD or SSD. This translate into 1 GiB of memory per 10 TiB of space for memtable. Always regardless of where you store the 64 bytes of metadata.

[littleskunk]

@Vadim

After migrating to hashstore, my node shows 2x amount of data

A 100% overhead? Or do you mean your used space has doubled?

[Vadim]

real space on disk is OK, but node show used is more.

I have 8 hashstore nodes most of them 6tk OK they are smaller. 2 bigger they node HDD are 12 and 16 TB and they both show doble from real amount.

[littleskunk]

@Vadim

real space on disk is OK, but node show used is more.

Used space scan on startup is disabled I guess? After the migration is finished you should be able to enable it again and it should take only seconds to run.

[Vadim]

Only if it disabled by default.

[DisaSoft]

Previously you didn’t recommend to use SSD for hashtables. I also found very intensive writes to hashtables during compaction (I guess, full rewire of hashtable while compacting each log) when tried to use SSD. Is anything changing in this behavior when memtables are used?

[snorkel]

I deleted all db-es after migration and restarted the node. It solved it.
You could just restart the node first, wait 8 days and restart again, to be sure the startup file walker ran. See if it solves it. If not, delete the db-es.
Why 8 days? Because it seems that startup FW runs only once in 7 days.

[Aleksman4o]

just delete used_space_per_prefix.db

[Toyoo]

@littleskunk

9 bytes per piece

So, I assume you’re not storing the piece ID in memory at all, and instead verify it on disk?

[zip]

Is there a way to configure trash threshold for the file rewrite, like rewrite only if there is more than 50% of trash?
Some of the migrated nodes are spending more than half a day almost every day rewriting the files, causing the success rates to sink, to only gain insignificant amounts of free space.

[Vadim]

Does hashstore automaticly mean that used space will be calculated by real HDD size and usage? I mean not like old way

[Alexey]

I got an answer from the team:

not directly, but we have two knobs to help:

STORJ_HASHSTORE_COMPACTION_ALIVE_FRAC
STORJ_HASHSTORE_COMPACTION_PROBABILITY_POWER

the ALIVE_FRAC is described in storagenode/hashstore: stochastic log rewriting · storj/storj@2436132 · GitHub and if you make it a number smaller than 0.25, log files will be less likely to be compacted. the PROBABILITY_POWER is described in storagenode/hashstore: raise rewrite probability to a power · storj/storj@82674e6 · GitHub and the higher the number, log files will be less likely to be compacted.

you can play around with the graph here: plot min(((a/(1-a))*((1-x)/x))^p, 1) from 0 to 1 where a = 0.25 and p = 1 - Wolfram|Alpha and change out the a and p variables to see the probability curve for a log file, where the x axis is what percent alive the log file is, and the y axis is the probability.

so, the old behavior used to be like ALIVE_FRAC=0.75 and PROBABILITY_POWER=100 which isn’t exactly what they’re asking for because they want to rewrite if the aggregate amount of trash is over some percent, whereas compaction only considers individual log files.

anyway, i would suggest maybe leaving ALIVE_FRAC the default and setting PROBABILITY_POWER=3. i think that should only clean up log files with a significant amount of dead data. (try looking at the curve by setting a=0.25 and p=3 on the wolframalpha link above)