[Tech Preview] Hashstore backend for storage nodes

Product Discussions STORJLINGS
21

All other options works as before as far as I know. So, you can use either an allocation or a dedicated disk features as well.

22

I rephrase, will it be possible to config store file size? for example for small disk 32k for bigger disk is 64 or 1m store size.

23

Oh, a wonderful story about how we create problems for ourselves and then heroically solve them! :smile:

Let’s imagine this epic saga of data storage:

Act One: “The Great Multiplication” “Hey, let’s create a storage system where we’ll have… drum roll… 10,000 files per terabyte!” “What? Only 10,000? That’s amateur hour! Let’s make it 100,000!” “No, wait! How about… A MILLION?” “A million is for weaklings! FIVE MILLION FILES!”

Act Two: “The Consequences” Disks start crying Storage system falls into philosophical depression Administrators grab their heads in despair

Act Three: “The Genius Solution” “Oh! I have an idea! Let’s now… merge all these files back together!”

Epilogue: Somewhere in a parallel universe, there exists a simple solution: “What if we just used 100-1000 times fewer files?” But no, that’s too simple and not heroic enough!

The moral of this tale: sometimes we get so caught up in creating complex solutions that we forget about the simple ones. And then we invent “anti-rakes” to fight the rakes we stepped on ourselves! :man_facepalming:

7 Likes
24

This adventure was a more dramatic.
We stored files in a database. What could be better?! Compacted, easily indexed and so on… But. There are SNOs who may disrupt their nodes in a random time, and maybe also their power providers can do so… As a result - the database become corrupted. Oh, well…
We decided to store pieces in the filesystem in a much efficient manner - in trees. Oh. Some FSes are not efficient to work with a tree, but they works… Until the high upload traffic is arrived…
So, we need a different solution. For most of FS the append only files are the most efficient regarding IOPS. But we need a search and quick access to the part of these files and also preferable do not move pieces (small files) to a different subfolder and also do not delete these millions of files using usual unlink functions of any OS (which is SLOW, incredible SLOW), what’s here? Hashmap!
Ok, now we implemented a hashstore, you may try it out (but please, on a new node, which would be not so painful to lose).

2 Likes
25

I still do not understand. You can change the allocation, as before, nothing has changed.

26

you make small files to bigger files, the size of this file should be configurable.

28

“Could you clarify something - if I create a node on Docker now, will it already have hashing enabled by default?”

29

Already answered in the start post:

1 Like
30

Is there a ready made v1.119 image for use with docker or do I need to compile one myself?

31

“Version”: “v1.116.7”} my version

updates

alpha
674068a5f-go1.18.8
674068a5f-go1.18.8-arm64v8
674068a5f-go1.18.8-arm32v5
674068a5f-go1.18.8-amd64
bb2ac4279-go1.18.8
bb2ac4279-go1.18.8-arm64v8
bb2ac4279-go1.18.8-arm32v5
bb2ac4279-go1.18.8-amd64
7c152f7ea-go1.18.8
7c152f7ea-go1.18.8-arm64v8
7c152f7ea-go1.18.8-arm32v5
7c152f7ea-go1.18.8-amd64
15efa1e31-go1.18.8
15efa1e31-go1.18.8-arm64v8
15efa1e31-go1.18.8-arm32v5
15efa1e31-go1.18.8-amd64
e7b35381f-go1.18.8
e7b35381f-go1.18.8-arm64v8
e7b35381f-go1.18.8-arm32v5
e7b35381f-go1.18.8-amd64
d0686648d-go1.18.8
d0686648d-go1.18.8-arm64v8
d0686648d-go1.18.8-arm32v5
d0686648d-go1.18.8-amd64
e9bc06608-go1.18.8
e9bc06608-go1.18.8-arm64v8
e9bc06608-go1.18.8-arm32v5
e9bc06608-go1.18.8-amd64
e40191afd-go1.18.8
e40191afd-go1.18.8-arm64v8
e40191afd-go1.18.8-arm32v5
e40191afd-go1.18.8-amd64
c98ef8931-go1.18.8
c98ef8931-go1.18.8-arm64v8
c98ef8931-go1.18.8-arm32v5
c98ef8931-go1.18.8-amd64
eab595397-go1.18.8
eab595397-go1.18.8-arm64v8
eab595397-go1.18.8-arm32v5
eab595397-go1.18.8-amd64
3639c5ee1-go1.18.8
3639c5ee1-go1.18.8-arm64v8
3639c5ee1-go1.18.8-arm32v5
3639c5ee1-go1.18.8-amd64
95960572b-go1.18.8
95960572b-go1.18.8-arm64v8
95960572b-go1.18.8-arm32v5
95960572b-go1.18.8-amd64
6f87ea801-v1.71.2-go1.18.8
6f87ea801-v1.71.2-go1.18.8-arm64v8
6f87ea801-v1.71.2-go1.18.8-arm32v5
6f87ea801-v1.71.2-go1.18.8-amd64
cb01aca13-go1.18.8
cb01aca13-go1.18.8-arm64v8
cb01aca13-go1.18.8-arm32v5
cb01aca13-go1.18.8-amd64
740cb0d9c-go1.18.8
740cb0d9c-go1.18.8-arm64v8
740cb0d9c-go1.18.8-arm32v5
740cb0d9c-go1.18.8-amd64
8850fde9f-go1.18.8
8850fde9f-go1.18.8-arm64v8
8850fde9f-go1.18.8-arm32v5
8850fde9f-go1.18.8-amd64
bf5b37883-go1.18.8
bf5b37883-go1.18.8-arm64v8
bf5b37883-go1.18.8-arm32v5
bf5b37883-go1.18.8-amd64
b86ce0d52-go1.18.8
b86ce0d52-go1.18.8-arm64v8
b86ce0d52-go1.18.8-arm32v5
b86ce0d52-go1.18.8-amd64

As I understand it, it will take another 3 months to press until the node can receive 1.19?

32

The version server url is configurable. So you can just setup your own one to make the official image downloading any storagenode version you want.

33

@alpharabbit thank you for that information. It seems I can switch in https://version.qa.storj.io/ which returns minimum version 1.118.7 and suggested version 1.119.1-rc. Then wait till it updates to 1.119

34

Ok, I am up and running 1.119 and I am seeing a bunch of log files that are slowly growing.

I am using
{“PassiveMigrate”:true,“WriteToNew”:true,“ReadNewFirst”:true,“TTLToNew”:false}

35

Wouldn’t it make sense to increase the maximum file size on the SNO side?
Splitting up huge backup files to 2MB chunks sounds pretty inefficient, or am I missing someting?

36

I think since data is diced up with erasure encoding… clients would need to download at least 28 other pieces from other nodes… so really they’d have to grab at least (29x2MB=) 58MB to have anything usable… and that’s a decent size?

37

I noticed that when running with hashstore migrations on, the used space amount shown in the nodes stats does not change. When switching all hashstore migrations off, the used space amount changes again.

Is this the right place to give feedback?

38

Used space in dashboard obviously doesn’t include the hashstore atm. I guess this is still on todo list.

1 Like
39

Do you mean the maximum log file size in hashstore?
then no, it’s hardcoded storj/storagenode/hashstore/store.go at 78f1637010dc51126a22886d7c1271ffacb4ff0b · storj/storj · GitHub

40

Give me a link on how to create your own update server.
Or write here.

41

I wouldn’t suggest to do so, just wait for the update.