# \[Tech Preview\] Hashstore backend for storage nodes **URL:** \ **Category:** STORJLINGS\ **Tags:** hashstore\ **Created:** [December 5, 2024, 3:24pm UTC](https://forum.storj.io/t/tech-preview-hashstore-backend-for-storage-nodes/28724 "2024-12-05T15:24:20Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [December 8, 2024, 3:29am UTC](https://forum.storj.io/t/tech-preview-hashstore-backend-for-storage-nodes/28724/21 "2024-12-08T03:29:19Z")
All other options works as before as far as I know. So, you can use either an allocation or a dedicated disk features as well. ---
**Author:** ![Vadim](https://storj.bcdn.literatehosting.com/letter_avatar/vadim/32/5_5575768a8748004e209b776fc1b2916d.png) [@Vadim](https://forum.storj.io/u/Vadim)\ **Post date:** [December 8, 2024, 7:49am UTC](https://forum.storj.io/t/tech-preview-hashstore-backend-for-storage-nodes/28724/22 "2024-12-08T07:49:29Z")
I rephrase, will it be possible to config store file size? for example for small disk 32k for bigger disk is 64 or 1m store size. ---
**Author:** ![peps](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/peps/32/1137_2.png) [@peps](https://forum.storj.io/u/peps)\ **Post date:** [December 8, 2024, 9:32am UTC](https://forum.storj.io/t/tech-preview-hashstore-backend-for-storage-nodes/28724/23 "2024-12-08T09:32:07Z")
Oh, a wonderful story about how we create problems for ourselves and then heroically solve them! 😄 Let’s imagine this epic saga of data storage: Act One: “The Great Multiplication” “Hey, let’s create a storage system where we’ll have… _drum roll_… 10,000 files per terabyte!” “What? Only 10,000? That’s amateur hour! Let’s make it 100,000!” “No, wait! How about… A MILLION?” “A million is for weaklings! FIVE MILLION FILES!” Act Two: “The Consequences” _Disks start crying_ _Storage system falls into philosophical depression_ _Administrators grab their heads in despair_ Act Three: “The Genius Solution” “Oh! I have an idea! Let’s now… merge all these files back together!” Epilogue: Somewhere in a parallel universe, there exists a simple solution: “What if we just used 100-1000 times fewer files?” But no, that’s too simple and not heroic enough! The moral of this tale: sometimes we get so caught up in creating complex solutions that we forget about the simple ones. And then we invent “anti-rakes” to fight the rakes we stepped on ourselves! 🤦‍♂️ ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [December 8, 2024, 9:58am UTC](https://forum.storj.io/t/tech-preview-hashstore-backend-for-storage-nodes/28724/24 "2024-12-08T09:58:40Z")
This adventure was a more dramatic. We stored files in a database. What could be better?! Compacted, easily indexed and so on… But. There are SNOs who may disrupt their nodes in a random time, and maybe also their power providers can do so… As a result - the database become corrupted. Oh, well… We decided to store pieces in the filesystem in a much efficient manner - in trees. Oh. Some FSes are not efficient to work with a tree, but they works… Until the high upload traffic is arrived… So, we need a different solution. For most of FS the append only files are the most efficient regarding IOPS. But we need a search and quick access to the part of these files and also preferable do not move pieces (small files) to a different subfolder and also do not delete these millions of files using usual unlink functions of any OS (which is SLOW, incredible SLOW), what’s here? Hashmap! Ok, now we implemented a hashstore, you may try it out (but please, on a new node, which would be not so painful to lose). ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [December 8, 2024, 10:05am UTC](https://forum.storj.io/t/tech-preview-hashstore-backend-for-storage-nodes/28724/25 "2024-12-08T10:05:01Z")
I still do not understand. You can change the allocation, as before, nothing has changed. ---
**Author:** ![Vadim](https://storj.bcdn.literatehosting.com/letter_avatar/vadim/32/5_5575768a8748004e209b776fc1b2916d.png) [@Vadim](https://forum.storj.io/u/Vadim)\ **Post date:** [December 8, 2024, 10:59am UTC](https://forum.storj.io/t/tech-preview-hashstore-backend-for-storage-nodes/28724/26 "2024-12-08T10:59:21Z")
you make small files to bigger files, the size of this file should be configurable. ---
**Author:** ![peps](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/peps/32/1137_2.png) [@peps](https://forum.storj.io/u/peps)\ **Post date:** [December 8, 2024, 11:06am UTC](https://forum.storj.io/t/tech-preview-hashstore-backend-for-storage-nodes/28724/28 "2024-12-08T11:06:58Z")
“Could you clarify something - if I create a node on Docker now, will it already have hashing enabled by default?” ---
**Author:** ![littleskunk](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/littleskunk/32/63_2.png) [@littleskunk](https://forum.storj.io/u/littleskunk)\ **Post date:** [December 8, 2024, 11:30am UTC](https://forum.storj.io/t/tech-preview-hashstore-backend-for-storage-nodes/28724/29 "2024-12-08T11:30:16Z")
Already answered in the start post: > [@littleskunk](#): > > By default all migrations will be disabled and the new hashstore folders will stay empty while the storage nodes keeps writing new pieces to the old piecestore backend. ---
**Author:** ![PieceKeeper](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/piecekeeper/32/1125_2.png) [@PieceKeeper](https://forum.storj.io/u/PieceKeeper)\ **Post date:** [December 8, 2024, 11:57am UTC](https://forum.storj.io/t/tech-preview-hashstore-backend-for-storage-nodes/28724/30 "2024-12-08T11:57:15Z")
Is there a ready made v1.119 image for use with docker or do I need to compile one myself? ---
**Author:** ![peps](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/peps/32/1137_2.png) [@peps](https://forum.storj.io/u/peps)\ **Post date:** [December 8, 2024, 12:27pm UTC](https://forum.storj.io/t/tech-preview-hashstore-backend-for-storage-nodes/28724/31 "2024-12-08T12:27:33Z")
“Version”: “v1.116.7”} my version updates alpha 674068a5f-go1.18.8 674068a5f-go1.18.8-arm64v8 674068a5f-go1.18.8-arm32v5 674068a5f-go1.18.8-amd64 bb2ac4279-go1.18.8 bb2ac4279-go1.18.8-arm64v8 bb2ac4279-go1.18.8-arm32v5 bb2ac4279-go1.18.8-amd64 7c152f7ea-go1.18.8 7c152f7ea-go1.18.8-arm64v8 7c152f7ea-go1.18.8-arm32v5 7c152f7ea-go1.18.8-amd64 15efa1e31-go1.18.8 15efa1e31-go1.18.8-arm64v8 15efa1e31-go1.18.8-arm32v5 15efa1e31-go1.18.8-amd64 e7b35381f-go1.18.8 e7b35381f-go1.18.8-arm64v8 e7b35381f-go1.18.8-arm32v5 e7b35381f-go1.18.8-amd64 d0686648d-go1.18.8 d0686648d-go1.18.8-arm64v8 d0686648d-go1.18.8-arm32v5 d0686648d-go1.18.8-amd64 e9bc06608-go1.18.8 e9bc06608-go1.18.8-arm64v8 e9bc06608-go1.18.8-arm32v5 e9bc06608-go1.18.8-amd64 e40191afd-go1.18.8 e40191afd-go1.18.8-arm64v8 e40191afd-go1.18.8-arm32v5 e40191afd-go1.18.8-amd64 c98ef8931-go1.18.8 c98ef8931-go1.18.8-arm64v8 c98ef8931-go1.18.8-arm32v5 c98ef8931-go1.18.8-amd64 eab595397-go1.18.8 eab595397-go1.18.8-arm64v8 eab595397-go1.18.8-arm32v5 eab595397-go1.18.8-amd64 3639c5ee1-go1.18.8 3639c5ee1-go1.18.8-arm64v8 3639c5ee1-go1.18.8-arm32v5 3639c5ee1-go1.18.8-amd64 95960572b-go1.18.8 95960572b-go1.18.8-arm64v8 95960572b-go1.18.8-arm32v5 95960572b-go1.18.8-amd64 6f87ea801-v1.71.2-go1.18.8 6f87ea801-v1.71.2-go1.18.8-arm64v8 6f87ea801-v1.71.2-go1.18.8-arm32v5 6f87ea801-v1.71.2-go1.18.8-amd64 cb01aca13-go1.18.8 cb01aca13-go1.18.8-arm64v8 cb01aca13-go1.18.8-arm32v5 cb01aca13-go1.18.8-amd64 740cb0d9c-go1.18.8 740cb0d9c-go1.18.8-arm64v8 740cb0d9c-go1.18.8-arm32v5 740cb0d9c-go1.18.8-amd64 8850fde9f-go1.18.8 8850fde9f-go1.18.8-arm64v8 8850fde9f-go1.18.8-arm32v5 8850fde9f-go1.18.8-amd64 bf5b37883-go1.18.8 bf5b37883-go1.18.8-arm64v8 bf5b37883-go1.18.8-arm32v5 bf5b37883-go1.18.8-amd64 b86ce0d52-go1.18.8 b86ce0d52-go1.18.8-arm64v8 b86ce0d52-go1.18.8-arm32v5 b86ce0d52-go1.18.8-amd64 As I understand it, it will take another 3 months to press until the node can receive 1.19? ---
**Author:** ![alpharabbit](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alpharabbit/32/1129_2.png) [@alpharabbit](https://forum.storj.io/u/alpharabbit)\ **Post date:** [December 8, 2024, 2:10pm UTC](https://forum.storj.io/t/tech-preview-hashstore-backend-for-storage-nodes/28724/32 "2024-12-08T14:10:00Z")
> [@PieceKeeper](#): > > Is there a ready made v1.119 image for use with docker or do I need to compile one myself? The version server url is configurable. So you can just setup your own one to make the official image downloading any storagenode version you want. ---
**Author:** ![PieceKeeper](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/piecekeeper/32/1125_2.png) [@PieceKeeper](https://forum.storj.io/u/PieceKeeper)\ **Post date:** [December 8, 2024, 3:05pm UTC](https://forum.storj.io/t/tech-preview-hashstore-backend-for-storage-nodes/28724/33 "2024-12-08T15:05:51Z")
@alpharabbit thank you for that information. It seems I can switch in [https://version.qa.storj.io/](https://version.qa.storj.io/) which returns minimum version 1.118.7 and suggested version 1.119.1-rc. Then wait till it updates to 1.119 ---
**Author:** ![PieceKeeper](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/piecekeeper/32/1125_2.png) [@PieceKeeper](https://forum.storj.io/u/PieceKeeper)\ **Post date:** [December 8, 2024, 4:26pm UTC](https://forum.storj.io/t/tech-preview-hashstore-backend-for-storage-nodes/28724/34 "2024-12-08T16:26:13Z")
Ok, I am up and running 1.119 and I am seeing a bunch of log files that are slowly growing. I am using {“PassiveMigrate”:true,“WriteToNew”:true,“ReadNewFirst”:true,“TTLToNew”:false} ---
**Author:** ![flo](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/flo/32/64_2.png) [@flo](https://forum.storj.io/u/flo)\ **Post date:** [December 8, 2024, 7:03pm UTC](https://forum.storj.io/t/tech-preview-hashstore-backend-for-storage-nodes/28724/35 "2024-12-08T19:03:01Z")
Wouldn’t it make sense to increase the maximum file size on the SNO side? Splitting up huge backup files to 2MB chunks sounds pretty inefficient, or am I missing someting? ---
**Author:** ![Roxor](https://storj.bcdn.literatehosting.com/letter_avatar/roxor/32/5_5575768a8748004e209b776fc1b2916d.png) [@Roxor](https://forum.storj.io/u/Roxor)\ **Post date:** [December 8, 2024, 7:41pm UTC](https://forum.storj.io/t/tech-preview-hashstore-backend-for-storage-nodes/28724/36 "2024-12-08T19:41:07Z")
I think since data is diced up with erasure encoding… clients would need to download at least 28 other pieces from other nodes… so really they’d have to grab at least (29x2MB=) 58MB to have anything usable… and that’s a decent size? ---
**Author:** ![PieceKeeper](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/piecekeeper/32/1125_2.png) [@PieceKeeper](https://forum.storj.io/u/PieceKeeper)\ **Post date:** [December 9, 2024, 7:47am UTC](https://forum.storj.io/t/tech-preview-hashstore-backend-for-storage-nodes/28724/37 "2024-12-09T07:47:14Z")
I noticed that when running with hashstore migrations on, the used space amount shown in the nodes stats does not change. When switching all hashstore migrations off, the used space amount changes again. Is this the right place to give feedback? ---
**Author:** ![alpharabbit](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alpharabbit/32/1129_2.png) [@alpharabbit](https://forum.storj.io/u/alpharabbit)\ **Post date:** [December 9, 2024, 7:56am UTC](https://forum.storj.io/t/tech-preview-hashstore-backend-for-storage-nodes/28724/38 "2024-12-09T07:56:11Z")
Used space in dashboard obviously doesn’t include the hashstore atm. I guess this is still on todo list. ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [December 9, 2024, 8:14am UTC](https://forum.storj.io/t/tech-preview-hashstore-backend-for-storage-nodes/28724/39 "2024-12-09T08:14:34Z")
Do you mean the maximum log file size in hashstore? then no, it’s hardcoded [storj/storagenode/hashstore/store.go at 78f1637010dc51126a22886d7c1271ffacb4ff0b · storj/storj · GitHub](https://github.com/storj/storj/blob/78f1637010dc51126a22886d7c1271ffacb4ff0b/storagenode/hashstore/store.go#L28) ---
**Author:** ![peps](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/peps/32/1137_2.png) [@peps](https://forum.storj.io/u/peps)\ **Post date:** [December 12, 2024, 5:30pm UTC](https://forum.storj.io/t/tech-preview-hashstore-backend-for-storage-nodes/28724/40 "2024-12-12T17:30:48Z")
Give me a link on how to create your own update server. Or write here. ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [December 13, 2024, 3:37am UTC](https://forum.storj.io/t/tech-preview-hashstore-backend-for-storage-nodes/28724/41 "2024-12-13T03:37:48Z")
I wouldn’t suggest to do so, just wait for the update. [Previous page](https://forum.storj.io/t/tech-preview-hashstore-backend-for-storage-nodes/28724.md?page=1) [Next page](https://forum.storj.io/t/tech-preview-hashstore-backend-for-storage-nodes/28724.md?page=3)