Hashstore rollout commencing!

[Aleksman4o]

It’s normal, you’ll see them at the end of each satellite’s migration

[Alexey]

Did you enabled like described there?

[Tech Preview] Hashstore backend for storage nodes

Do you remember that one big customer we simulated on SLC? If not here is a short recap. That customer uploads a high amount of small and medium size files with a short TTL. It turns out storage nodes can handle the initial uploads but things gets expensive when the TTL kicks in. The storage nodes simply needs too much time deleting such an high amount of pieces. The solution for that problem is the new hashstore backend. Instead of storing all pieces as a file on disk the hashstore backend com…

Because this one

@cryptobiose

hashstore:
enabled: true
write-to-new: true # Send all new writes to the hashstore
ttl-to-new: true # Only redirect chunks with a TTL to the hashstore
read-new-first: true # When downloading, try the hashstore first
path: /home/storagenode/SJNs005n2504HDD4_MID0011/hashstore # The directory where

Looks totally incorrect.

[Alexey]

It should be after the image name, since it’s a node’s argument.

You may place it before the image name too but as a variable name, e.g.

@jtolio

STORJ_STORAGE2_MIGRATION_SUPPRESS_CENTRAL_MIGRATION=true

with a -e option prefix.

[Alexey]

Hashstore rollout commencing!

We predominantly use hashtbl, but have started using memtbl more in cases where the RAM is available. Both are very fast. memtbl is only faster if the RAM is available for it, but if there isn’t enough RAM, then hashtbl is faster. Ah yes, thank you for the context. It looks like since none of our code used it, it was removed in a refactor without causing anything to trip. It should be straightforward to add back. In fact, the telemetry data your node maintains does actually report satellite u…

[Julio]

@jtolio

There is a similar config option for the logs path (hashstore.logs-path)

So this is where I can specify a completely different drive/folder path to migrate the main data to?

[Alexey]

./storagenode setup --help | sls hash

      --hashstore.logs-path string                               path to store log files in (by default, it's relative
to the storage directory)' (default "hashstore")
      --hashstore.table-path string                              path to store tables in. Can be same as LogsPath, as
subdirectories are used (by default, it's relative to the storage directory) (default "hashstore")

So, yes.

[Julio]

Thanks A.I.exey, you are a wealth of knowledge, as always!

5 cents,

Julio

[Alexey]

I’m a human, not A.I. So, N.I.

[snorkel]

So to opt-out from migration to hashstore, you should use one of these 2 variants in Docker run command:

Var.1:

--mount type=bind,source="/volume1/Storj/Identity/storagenode/",destination=/app/identity \
--mount type=bind,source="/volume1/Storj/",destination=/app/config \
--name storagenode storjlabs/storagenode:latest \
--storage2migration.suppress-central-migration=true \

Var.2:

-e STORJ_STORAGE2MIGRATION_SUPPRESS_CENTRAL_MIGRATION=true \
--mount type=bind,source="/volume1/Storj/Identity/storagenode/",destination=/app/identity \
--mount type=bind,source="/volume1/Storj/",destination=/app/config \
--name storagenode storjlabs/storagenode:latest \

It works from v1.135.5.
P.S. I corrected the flags. Now it works.

[jammerdan]

Question on hashstore reads and data location. As I understand it, with hashstore pieces are no longer file pieces but data within “log” files?

I read that this is supposed to help writes.

@jtolio

Hashstore also has the goal of improving alignment overhead and minimizing random writes.

But what about reads? As the pieces are no longer file system objects do frequently accessed pieces get cached? Because there are frequently accessed so called lucky pieces, causing massive amount of downloads:

@jammerdan

I recently had a lucky piece that was being downloaded at a rate of like 100 times per minute and more.

@Krawi

This is a single piece, logs from today, time: 8 hours 45 minutes.
That’s about one download request every 2 seconds, it started on June 12. and continuing since then.

@snorkel

… and the bit goes on…
(if we were back 4 years ago when egress was 20$… )

@EasyRhino

oh I got my first magic data fragment yesterday on one node. normal egress is around 8GB. yesterday it was 146GB

@snorkel

The lucky pieces still on… same 2 nodes last month:

So 2 questions regarding pieces in log files:

1. How (good) are reads cached with hashstore with the big logfiles in place now (because in my opinion such massive downloads should come entirely from RAM)?
2. Is there still a way to tell where a specific piece is stored which was easy with piecestore of course?

[snorkel]

From which version can we use the parameter to opt out?

[Alexey]

I suppose that the whole file would be cached, if it’s requested too frequently.

[Alexey]

I think the next one. It’s a preparation to get early feedback, before we would implement anything on the public network.
We will use a partial updates exactly how we rollout a new version - with a possibility to stop and start over.

[alpharabbit]

With ZFS blocks are cached not files. So there is no difference for caching the lucky pieces.

[Alexey]

If this is true, then there should be no difference between backends. Perhaps some tunings will be obsolete, but I guess that’s not a problem.

[jammerdan]

Is there a way to tell in which log file/path a piece is stored like it was possible with piecestore?

[flo]

@snorkel

From which version can we use the parameter to opt out?

v1.135.5 (currently being rolled out)

[flo82]

i see it every hour and it causes CPU spikes and log floods.

[snorkel]

Nope. Dosen’t work on that either.

[cryptobiose]

Yeah it’s the i.a fault I trusted the tools and they got worse and worse. But at the same time, I should have thought about it a little more…

Yes, I did this [Tech Preview] Hashstore backend for storage nodes - #567 by jammerdan

and this then this [Tech Preview] Hashstore backend for storage nodes - #569 by Alexey .

I think it is working because I have log file in hashstore in s0 and s1, more the 20 GO for now.

But I don’t move them to the ssd for now.