My "memtables" kludge

JuhaML · March 30, 2025, 2:20pm

Backgrounder:

node located in northern Europe
only 10Mbps upload, 6Mbps minimum dedicated to Storj (using QoS), in order to meet the minimum node requirements
running a 1.8TB node on router/firewall Linux distro (IPFire), bare metal, launching node from CLI (not set up as a service)
US satellite only, gracefully exited from SL, EU and AP
Fully migrated to hashstore
databases and orders relocated to an SSD (which is also boot/root/home device)

The Kludge:

storj@jmlfire:~$ cat storj.sh
#!/bin/bash
RAMPATH=/mnt/ramdisk
DATPATH=/mnt/storj/StorjDat/hashstore
US_SATELLITE=12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S

# Move US satellite hashtables to ramdisk and create symlinks
cd $DATPATH/$US_SATELLITE/s0 > /dev/null
mv meta $RAMPATH/s0meta
ln -s $RAMPATH/s0meta meta
cd ../s1
mv meta $RAMPATH/s1meta
ln -s $RAMPATH/s1meta meta

# Try to yank payload file/dir metadata to cache
ls -lsaR $DATPATH > /dev/null

# Start storagenode at home dir, direct log from stdout to ramdisk
$HOME/storagenode run > $RAMPATH/tmp.log

# Move hashtables back after kill -SIGINT <storagenode process>
rm meta
mv $RAMPATH/s1meta meta
cd ../s0
rm meta
mv $RAMPATH/s0meta meta

# append RAM disk log to persistent log at home dir and empty ramdisk
cat $RAMPATH/tmp.log >> $HOME/node.log
rm -rf $RAMPATH/*
storj@jmlfire:~$ ./storj.sh &

Really ugly, I know, but the metadata will be in RAM - virtual memory, to be more specific - when the node is running, right? Definitely will lose the node if there is a system crash or power outage, so I won’t be recommending this kind of tweak for “real” nodes.

Having said that, this seems to work for me, although I’ve been running it like this for only a day or so. My question is, how would this compare, performance wise, to having hashtables on SSD. Not to mention actual hashtbl implementation in v1.125.2?

Some random stats while node is running:

storj@jmlfire:~$ lsblk ; echo ; df -h ; echo ; free -h
NAME           MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda              8:0    0   1.8T  0 disk /mnt/storj
sdb              8:16   0   1.8T  0 disk
└─span-spanned 253:0    0   2.3T  0 lvm  /mnt/nas
sdc              8:32   0 465.8G  0 disk
└─span-spanned 253:0    0   2.3T  0 lvm  /mnt/nas
sdd              8:48   0  55.9G  0 disk
├─sdd1           8:49   0   256M  0 part /boot
├─sdd2           8:50   0    32M  0 part /boot/efi
├─sdd3           8:51   0 864.9M  0 part [SWAP]
└─sdd4           8:52   0  54.8G  0 part /

Filesystem                Size  Used Avail Use% Mounted on
devtmpfs                  3.9G  4.0K  3.9G   1% /dev
tmpfs                     3.9G     0  3.9G   0% /dev/shm
tmpfs                     3.9G  748K  3.9G   1% /run
/dev/sdd4                  53G  6.1G   45G  12% /
/dev/sda                  1.9T  1.8T   29G  99% /mnt/storj
/dev/sdd1                 236M   64M  155M  30% /boot
/dev/sdd2                  32M  288K   32M   1% /boot/efi
/dev/mapper/span-spanned  2.3T  925G  1.4T  40% /mnt/nas
/var/lock                 8.0M   12K  8.0M   1% /var/lock
tmpfs                     7.0G  2.6G  4.5G  36% /mnt/ramdisk

               total        used        free      shared  buff/cache   available
Mem:           7.7Gi       2.9Gi       2.0Gi       2.5Gi       5.3Gi       4.7Gi
Swap:          864Mi          0B       864Mi
storj@jmlfire:~$

Success stats so far (15MB log file):

storj@jmlfire:~$ nice ./succrate.sh node.log
========== AUDIT ==============
Critically failed:     0
Critical Fail Rate:    0.000%
Recoverable failed:    0
Recoverable Fail Rate: 0.000%
Successful:            317
Success Rate:          100.000%
========== DOWNLOAD ===========
Failed:                3
Fail Rate:             0.006%
Canceled:              885
Cancel Rate:           1.829%
Successful:            47489
Success Rate:          98.164%
========== UPLOAD =============
Rejected:              0
Acceptance Rate:       0.000%
---------- accepted -----------
Failed:                0
Fail Rate:             0.000%
Canceled:              0
Cancel Rate:           0.000%
Successful:            0
Success Rate:          0.000%
========== REPAIR DOWNLOAD ====
Failed:                0
Fail Rate:             0.000%
Canceled:              0
Cancel Rate:           0.000%
Successful:            442
Success Rate:          100.000%
========== REPAIR UPLOAD ======
Failed:                0
Fail Rate:             0.000%
Canceled:              0
Cancel Rate:           0.000%
Successful:            0
Success Rate:          0.000%
========== DELETE =============
Failed:                0
Fail Rate:             0.000%
Successful:            0
Success Rate:          0.000%
storj@jmlfire:~$

Roxor · March 30, 2025, 4:19pm

Cached file metadata certainly speeds up maintenance operations: like anything involving filewalker. But it’s hard to tell if it benefits anything related to making money today.

The main bottleneck to making-more-money is the uploads from paying customers. And then if there are lots of uploads… the Internet connection is the bottleneck. And even with fast Internet… HDDs are still usually way faster and can easily keep up with customer random-writes. But if demand is high enough… and the network connection is fast enough… and maybe the disk is a bit busy… THEN badger/hashstore/memtable can help. And maybe they help you win a couple more races. To maybe store more data. To maybe make more money.

A lot of maybes

Hashstore/memtable is cool new tech! But it’s also performance tuning for a load the network has only seen from test data last July-Sept. I hope one day we’ll actually need the speed!

alpharabbit · March 30, 2025, 4:38pm

This hashstore/memtable development is driven by select network. For my public network nodes I see not much benefit, it’s just another playground.

JuhaML · March 30, 2025, 5:17pm

For me, this is a hobby. I just happen to have a gizmo which was already on 24/7, well before storj became a thing for me. Essentially a homespun equivalent of this, cobbled together from repurposed parts:
IPFire Mini Appliance (EU) - Firewall Appliances - Lightning Wire Labs Store

I have no plans for a dedicated storj setup, just glad to add storj functionality to an existing device doing firewall/router/home network drive, etc tasks to begin with - and nowadays storj, too.

Therefore, I only count the additional cost of spinning an extra drive as an expense. With that mindset, my break-even point is around 1TB harddisk, so I am happy to let storj wring out every last bit of use from my otherwise retired spinning rust. I suppose that is well aligned with the original storj idea, eh?

However, I would like it to be the best storj node it can be, given the shared resources. So I look for tweaks like the one above to make sure that storj gets the “lion’s share” whenever it needs; the leftovers are perfectly sufficient for my relatively modest uses.

JuhaML · March 31, 2025, 4:15pm

Some further observations:

If you actually try this for yourself, do make the ramdisk at least 2x metadata size. Compaction creates temporary doubles (triples, quadruples, quintets, ?) of hashtables, too. Fortunately I was prepared for that, I suspect I might have ruined my node otherwise.
It does look like compaction actually writes hashtables a lot. Had a couple of compaction events today:

Compacting1d2828×1648 366 KB

Apparently, the first one did s0, file timestamp supports that:

storj@jmlfire:~$ ll /mnt/ramdisk/s0meta ; ll /mnt/ramdisk/s1meta
total 513M
   0 drwxr-x--- 2 storj storj   80 Mar 31 11:10 .
   0 drwxrwxrwt 4 root  root   100 Mar 30 20:37 ..
513M -rw------- 1 storj storj 513M Mar 31 11:10 hashtbl-00000000000002d3
   0 -rw-rw-r-- 1 storj storj    0 Jan 13 05:26 lock
total 4.1G
   0 drwxr-x--- 2 storj storj  100 Mar 31 15:35 .
   0 drwxrwxrwt 4 root  root   100 Mar 30 20:37 ..
2.1G -rw------- 1 storj storj 2.1G Mar 31 15:35 hashtbl-00000000000002cb
2.1G -rw------- 1 storj storj 2.1G Mar 31 15:37 hashtbl-00000000000002cc.tmp
   0 -rw-rw-r-- 1 storj storj    0 Jan 13 05:26 lock
storj@jmlfire:~$

As for the second round, apparently for s1, I polled the hashdata folder (on ramdisk) at 60s intervals and logged this:

total 4.1G
   0 drwxr-x--- 2 storj storj  100 Mar 31 16:43 .
   0 drwxrwxrwt 4 root  root   100 Mar 30 20:37 ..
2.1G -rw------- 1 storj storj 2.1G Mar 31 16:43 hashtbl-00000000000002ee
2.1G -rw------- 1 storj storj 2.1G Mar 31 16:43 hashtbl-00000000000002ef.tmp
   0 -rw-rw-r-- 1 storj storj    0 Jan 13 05:26 lock
total 4.1G
   0 drwxr-x--- 2 storj storj  100 Mar 31 16:43 .
   0 drwxrwxrwt 4 root  root   100 Mar 30 20:37 ..
2.1G -rw------- 1 storj storj 2.1G Mar 31 16:43 hashtbl-00000000000002ee
2.1G -rw------- 1 storj storj 2.1G Mar 31 16:44 hashtbl-00000000000002ef.tmp
   0 -rw-rw-r-- 1 storj storj    0 Jan 13 05:26 lock
total 4.1G
   0 drwxr-x--- 2 storj storj  100 Mar 31 16:44 .
   0 drwxrwxrwt 4 root  root   100 Mar 30 20:37 ..
2.1G -rw------- 1 storj storj 2.1G Mar 31 16:44 hashtbl-00000000000002ef
2.1G -rw------- 1 storj storj 2.1G Mar 31 16:45 hashtbl-00000000000002f0.tmp
   0 -rw-rw-r-- 1 storj storj    0 Jan 13 05:26 lock
total 4.1G
   0 drwxr-x--- 2 storj storj  100 Mar 31 16:45 .
   0 drwxrwxrwt 4 root  root   100 Mar 30 20:37 ..
2.1G -rw------- 1 storj storj 2.1G Mar 31 16:45 hashtbl-00000000000002f0
2.1G -rw------- 1 storj storj 2.1G Mar 31 16:46 hashtbl-00000000000002f1.tmp
   0 -rw-rw-r-- 1 storj storj    0 Jan 13 05:26 lock
total 4.1G
   0 drwxr-x--- 2 storj storj  100 Mar 31 16:47 .
   0 drwxrwxrwt 4 root  root   100 Mar 30 20:37 ..
2.1G -rw------- 1 storj storj 2.1G Mar 31 16:47 hashtbl-00000000000002f1
2.1G -rw------- 1 storj storj 2.1G Mar 31 16:47 hashtbl-00000000000002f2.tmp
   0 -rw-rw-r-- 1 storj storj    0 Jan 13 05:26 lock
total 4.1G
   0 drwxr-x--- 2 storj storj  100 Mar 31 16:47 .
   0 drwxrwxrwt 4 root  root   100 Mar 30 20:37 ..
2.1G -rw------- 1 storj storj 2.1G Mar 31 16:47 hashtbl-00000000000002f1
2.1G -rw------- 1 storj storj 2.1G Mar 31 16:48 hashtbl-00000000000002f2.tmp
   0 -rw-rw-r-- 1 storj storj    0 Jan 13 05:26 lock
total 4.1G
   0 drwxr-x--- 2 storj storj  100 Mar 31 16:48 .
   0 drwxrwxrwt 4 root  root   100 Mar 30 20:37 ..
2.1G -rw------- 1 storj storj 2.1G Mar 31 16:48 hashtbl-00000000000002f2
2.1G -rw------- 1 storj storj 2.1G Mar 31 16:49 hashtbl-00000000000002f3.tmp
   0 -rw-rw-r-- 1 storj storj    0 Jan 13 05:26 lock
total 4.1G
   0 drwxr-x--- 2 storj storj  100 Mar 31 16:50 .
   0 drwxrwxrwt 4 root  root   100 Mar 30 20:37 ..
2.1G -rw------- 1 storj storj 2.1G Mar 31 16:49 hashtbl-00000000000002f3
2.1G -rw------- 1 storj storj 2.1G Mar 31 16:50 hashtbl-00000000000002f4.tmp
   0 -rw-rw-r-- 1 storj storj    0 Jan 13 05:26 lock
total 4.1G
   0 drwxr-x--- 2 storj storj  100 Mar 31 16:50 .
   0 drwxrwxrwt 4 root  root   100 Mar 30 20:37 ..
2.1G -rw------- 1 storj storj 2.1G Mar 31 16:49 hashtbl-00000000000002f3
2.1G -rw------- 1 storj storj 2.1G Mar 31 16:51 hashtbl-00000000000002f4.tmp
   0 -rw-rw-r-- 1 storj storj    0 Jan 13 05:26 lock
total 4.1G
   0 drwxr-x--- 2 storj storj  100 Mar 31 16:50 .
   0 drwxrwxrwt 4 root  root   100 Mar 30 20:37 ..
2.1G -rw------- 1 storj storj 2.1G Mar 31 16:49 hashtbl-00000000000002f3
2.1G -rw------- 1 storj storj 2.1G Mar 31 16:52 hashtbl-00000000000002f4.tmp
   0 -rw-rw-r-- 1 storj storj    0 Jan 13 05:26 lock
total 4.1G
   0 drwxr-x--- 2 storj storj  100 Mar 31 16:52 .
   0 drwxrwxrwt 4 root  root   100 Mar 30 20:37 ..
2.1G -rw------- 1 storj storj 2.1G Mar 31 16:52 hashtbl-00000000000002f4
2.1G -rw------- 1 storj storj 2.1G Mar 31 16:53 hashtbl-00000000000002f5.tmp
   0 -rw-rw-r-- 1 storj storj    0 Jan 13 05:26 lock
total 4.1G
   0 drwxr-x--- 2 storj storj  100 Mar 31 16:54 .
   0 drwxrwxrwt 4 root  root   100 Mar 30 20:37 ..
2.1G -rw------- 1 storj storj 2.1G Mar 31 16:54 hashtbl-00000000000002f5
2.1G -rw------- 1 storj storj 2.1G Mar 31 16:54 hashtbl-00000000000002f6.tmp
   0 -rw-rw-r-- 1 storj storj    0 Jan 13 05:26 lock
total 4.1G
   0 drwxr-x--- 2 storj storj  100 Mar 31 16:54 .
   0 drwxrwxrwt 4 root  root   100 Mar 30 20:37 ..
2.1G -rw------- 1 storj storj 2.1G Mar 31 16:54 hashtbl-00000000000002f5
2.1G -rw------- 1 storj storj 2.1G Mar 31 16:55 hashtbl-00000000000002f6.tmp
   0 -rw-rw-r-- 1 storj storj    0 Jan 13 05:26 lock
total 4.1G
   0 drwxr-x--- 2 storj storj  100 Mar 31 16:56 .
   0 drwxrwxrwt 4 root  root   100 Mar 30 20:37 ..
2.1G -rw------- 1 storj storj 2.1G Mar 31 16:55 hashtbl-00000000000002f6
2.1G -rw------- 1 storj storj 2.1G Mar 31 16:56 hashtbl-00000000000002f7.tmp
   0 -rw-rw-r-- 1 storj storj    0 Jan 13 05:26 lock
total 4.1G
   0 drwxr-x--- 2 storj storj  100 Mar 31 16:57 .
   0 drwxrwxrwt 4 root  root   100 Mar 30 20:37 ..
2.1G -rw------- 1 storj storj 2.1G Mar 31 16:57 hashtbl-00000000000002f7
2.1G -rw------- 1 storj storj 2.1G Mar 31 16:57 hashtbl-00000000000002f8.tmp
   0 -rw-rw-r-- 1 storj storj    0 Jan 13 05:26 lock
total 4.1G
   0 drwxr-x--- 2 storj storj  100 Mar 31 16:57 .
   0 drwxrwxrwt 4 root  root   100 Mar 30 20:37 ..
2.1G -rw------- 1 storj storj 2.1G Mar 31 16:57 hashtbl-00000000000002f7
2.1G -rw------- 1 storj storj 2.1G Mar 31 16:58 hashtbl-00000000000002f8.tmp
   0 -rw-rw-r-- 1 storj storj    0 Jan 13 05:26 lock
total 4.1G
   0 drwxr-x--- 2 storj storj  100 Mar 31 16:59 .
   0 drwxrwxrwt 4 root  root   100 Mar 30 20:37 ..
2.1G -rw------- 1 storj storj 2.1G Mar 31 16:59 hashtbl-00000000000002f8
2.1G -rw------- 1 storj storj 2.1G Mar 31 16:59 hashtbl-00000000000002f9.tmp
   0 -rw-rw-r-- 1 storj storj    0 Jan 13 05:26 lock
total 4.1G
   0 drwxr-x--- 2 storj storj  100 Mar 31 16:59 .
   0 drwxrwxrwt 4 root  root   100 Mar 30 20:37 ..
2.1G -rw------- 1 storj storj 2.1G Mar 31 16:59 hashtbl-00000000000002f8
2.1G -rw------- 1 storj storj 2.1G Mar 31 17:00 hashtbl-00000000000002f9.tmp
   0 -rw-rw-r-- 1 storj storj    0 Jan 13 05:26 lock
total 4.1G
   0 drwxr-x--- 2 storj storj  100 Mar 31 17:01 .
   0 drwxrwxrwt 4 root  root   100 Mar 30 20:37 ..
2.1G -rw------- 1 storj storj 2.1G Mar 31 17:01 hashtbl-00000000000002f9
2.1G -rw------- 1 storj storj 2.1G Mar 31 17:01 hashtbl-00000000000002fa.tmp
   0 -rw-rw-r-- 1 storj storj    0 Jan 13 05:26 lock
total 4.1G
   0 drwxr-x--- 2 storj storj  100 Mar 31 17:02 .
   0 drwxrwxrwt 4 root  root   100 Mar 30 20:37 ..
2.1G -rw------- 1 storj storj 2.1G Mar 31 17:02 hashtbl-00000000000002fa
2.1G -rw------- 1 storj storj 2.1G Mar 31 17:02 hashtbl-00000000000002fb.tmp
   0 -rw-rw-r-- 1 storj storj    0 Jan 13 05:26 lock
total 4.1G
   0 drwxr-x--- 2 storj storj  100 Mar 31 17:02 .
   0 drwxrwxrwt 4 root  root   100 Mar 30 20:37 ..
2.1G -rw------- 1 storj storj 2.1G Mar 31 17:02 hashtbl-00000000000002fa
2.1G -rw------- 1 storj storj 2.1G Mar 31 17:03 hashtbl-00000000000002fb.tmp
   0 -rw-rw-r-- 1 storj storj    0 Jan 13 05:26 lock
total 4.1G
   0 drwxr-x--- 2 storj storj  100 Mar 31 17:04 .
   0 drwxrwxrwt 4 root  root   100 Mar 30 20:37 ..
2.1G -rw------- 1 storj storj 2.1G Mar 31 17:04 hashtbl-00000000000002fb
2.1G -rw------- 1 storj storj 2.1G Mar 31 17:04 hashtbl-00000000000002fc.tmp
   0 -rw-rw-r-- 1 storj storj    0 Jan 13 05:26 lock
total 4.1G
   0 drwxr-x--- 2 storj storj  100 Mar 31 17:04 .
   0 drwxrwxrwt 4 root  root   100 Mar 30 20:37 ..
2.1G -rw------- 1 storj storj 2.1G Mar 31 17:04 hashtbl-00000000000002fb
2.1G -rw------- 1 storj storj 2.1G Mar 31 17:05 hashtbl-00000000000002fc.tmp
   0 -rw-rw-r-- 1 storj storj    0 Jan 13 05:26 lock
total 4.1G
   0 drwxr-x--- 2 storj storj  100 Mar 31 17:05 .
   0 drwxrwxrwt 4 root  root   100 Mar 30 20:37 ..
2.1G -rw------- 1 storj storj 2.1G Mar 31 17:05 hashtbl-00000000000002fc
2.1G -rw------- 1 storj storj 2.1G Mar 31 17:06 hashtbl-00000000000002fd.tmp
   0 -rw-rw-r-- 1 storj storj    0 Jan 13 05:26 lock
total 4.1G
   0 drwxr-x--- 2 storj storj  100 Mar 31 17:05 .
   0 drwxrwxrwt 4 root  root   100 Mar 30 20:37 ..
2.1G -rw------- 1 storj storj 2.1G Mar 31 17:05 hashtbl-00000000000002fc
2.1G -rw------- 1 storj storj 2.1G Mar 31 17:07 hashtbl-00000000000002fd.tmp
   0 -rw-rw-r-- 1 storj storj    0 Jan 13 05:26 lock
total 4.1G
   0 drwxr-x--- 2 storj storj  100 Mar 31 17:07 .
   0 drwxrwxrwt 4 root  root   100 Mar 30 20:37 ..
2.1G -rw------- 1 storj storj 2.1G Mar 31 17:07 hashtbl-00000000000002fd
2.1G -rw------- 1 storj storj 2.1G Mar 31 17:08 hashtbl-00000000000002fe.tmp
   0 -rw-rw-r-- 1 storj storj    0 Jan 13 05:26 lock
total 4.1G
   0 drwxr-x--- 2 storj storj  100 Mar 31 17:09 .
   0 drwxrwxrwt 4 root  root   100 Mar 30 20:37 ..
2.1G -rw------- 1 storj storj 2.1G Mar 31 17:09 hashtbl-00000000000002fe
2.1G -rw------- 1 storj storj 2.1G Mar 31 17:09 hashtbl-00000000000002ff.tmp
   0 -rw-rw-r-- 1 storj storj    0 Jan 13 05:26 lock
total 2.1G
   0 drwxr-x--- 2 storj storj   80 Mar 31 17:10 .
   0 drwxrwxrwt 4 root  root   100 Mar 30 20:37 ..
2.1G -rw------- 1 storj storj 2.1G Mar 31 17:10 hashtbl-00000000000002ff
   0 -rw-rw-r-- 1 storj storj    0 Jan 13 05:26 lock
total 2.1G
   0 drwxr-x--- 2 storj storj   80 Mar 31 17:10 .
   0 drwxrwxrwt 4 root  root   100 Mar 30 20:37 ..
2.1G -rw------- 1 storj storj 2.1G Mar 31 17:10 hashtbl-00000000000002ff
   0 -rw-rw-r-- 1 storj storj    0 Jan 13 05:26 lock

Uh-oh, a fresh ~2GB hashtable file every ~2 minutes or so → updating metadata at ~1GB/min rate?
Sounds quite a lot to me, especially considering that my spinning rust wasn’t even moving actual payload data around particularly quickly at that time: