Block size recommendations for different subdirectories

Node Operators
,
1

Hello! I’m planning a node on ZFS and want to understand how can I properly split files to maximize IO.

  • most blobs are probably ok with higher recordsize;
  • sqlite databases: I guess would be ok with blocks up to 64k, if updated accordingly (in case there’s no much data written to them;
  • hashstore logs - unclear, huge files, can’t get the pattern;
  • meta/hashtbl - looks like they are queried by 64 byte offset, so probably should be treated like a database, like 16k?

Anyone experimented splitting those files per type and giving them separate ZFS filesystems?

2

128k blobs. 64k databases. Or leave them at 128k as well.

Don’t overthink it. Don’t change defaults unless you have measured and found a bottleneck.

3

You have a point, but I’m not that kind of guy. I’m always overthinking.

For example I know SMR drives are really not good for any consistent load. On the other hand I already have them, and I have a tiered storage that mostly nullifies negative impact. To do that I’m trying to limit write multiplication as much as I can, so I’m using 4M recordsize globally for all non-database workloads, that helps prevent defragmentation and speeds up contiguous RW considerably. I think 4M is also an old default for piece storage used in config previously. The bad side of the coin is that you need to directly highlight where databases are and split them from anything else. And here comes storj, that packs differently used files in adjacent directories, requiring a bushy tree to separate databases and bulk load files:

storage:

- root:
  - sqlite database files live here
  - blobs:
   - data files stored here
  - hashstore:
   - meta:
    - more database files
   - ID:
    - s0:
     - 01 - FF data files are here
     - meta hashtbl (?database) is here

This means it’s not trivial to clearly separate database/data, as some folders are even automatically created. This is why I’m trying to collect as much data as I can, to understand how can I improve IO for storj.

And I’m already past “don’t touch it” phase. Everything works properly without issues, until I lost power and hashtbl’s fail to load, so I need to recreate them from scratch, and when I do that — I need a day or two of high IO load to get node back online.

4

With hashstore, I wouldn’t bother tuning different ZFS record sizes anymore. ZFS had its place with piecestore, but it’s no longer the sweet spot for hashstore.

Just use XFS with the defaults and move on. Trust the guys who have been running Storj nodes at scale for years :slightly_smiling_face:

5

With hashstore file system doesn’t matter tbh. I stick with zfs and default settings. Even 4..8 nodes per single hdd pool works without problems.

6

I don’t think XFS is an option for me, ZFS is first class citizen for me, and I’m just trying to minimize storj impact.