New operator need sugestions/tips/advice

Node Operators getting started
1

Hello everyone I have decided to start a node and contribute to Storj.
I have made research but would still like some sugestions/advice from the veteras before I start.

Curently I will be a small node as I only have space in the server case for 13 drives that will be 13 x 18TB disks.
The memory and cpu will not be really powerfull 8GB of DDR4 2133mhz and CPU Intel pentium G4560.
My entire setup will be conected to a 5G router with 5G conectivity its not the fastest in my area but should be OK (~135Mbps down and around 25Mbps up).
The main reason im using 5G as its the fastest I got as im not able to get fiber
The power for the server will be free as I have setup my own 6kw solar array with storage also cell towers do not usualy go down with power outage as they have their own generators and that will keep my server up all times.

As far as im aware cpu and memory are not a big factor so that should be OK? (I had this hardware laying around so I did not want to be buying new stuff if not needed).

Than my OS of choise will be TrueNas as it supports Storj via Apps page and I already use TrueNas for a ton of stuff so im familiar with it (p.s. this will be a separate machine it will not be used with my other stuff as I want to isolate it).

So my first question that I have is do you recomend runing each drive as a separate node or should I just make one big VDEV and have it as one disk?

The conclusion I came on is that its better to rin each disk as its own node due to disk failure and than Storj will blacklist/penalize my entire array instead of just 1 node.
I could run a SAS expander and hotswap mechanism but this will require a couple of disks to be used as parity so that will give me less space.
Do you think this is the correct way of going at it?

One more thing im not sure is in TrueNas its giving me the option to specifiy size i want to use in GB.. Now im aware that the 18TB disk is not 18TB should I leave some space like 1GB room so its never full? I belive its like what 16.4TB per disk so maybe I should set 15.5 GB?

Well this are some concers andnquestion I got so any advice/suggestions are welcome!

2

Welcome @Sale666 :slight_smile:

Sounds like it’s not a small node if you are planning to run 234TB RAW space?

It’s fine - if you are planning to run multiple nodes in the future, a bit more memory would be nice. TOS states 1 core = 1 node

Remember than you will need inbound traffic for UDP/TCP on the node - if you do not have a public IP assigned to the 5G connection, you will need to run it via an external VPN with portforwarding capabilities.

Also, this will use quite a lot of traffic, if you have ANY limitations on this 5G connection, it could become a problem.

For example, a 4-6TB node (about 1 year old) easily uses 2-3TB traffic a month on its own.

This is a qustion with many answers. TOS / StorJ recommendations are clear - 1 drive = 1 node.
HOWEVER - it all depends on many factors. Here’s a few things to consider:

Backend store - Piecestore vs Hashstore

Piecestore:
.. uses single files per upload and will accumulate 10’s of millions of files over time. This can be a strain on the filesystem when housekeeping tasks are running (scan files for usage/garbage collectors etc)

Hashstore:
.. uses index/log files of 1GB, and will not cause issues with metadata on any filesystem. However, it is not the default and can corrupt if a power failure is introduced.

Filesystem
I guess ZFS will be you champion of choise, as it’s default in Truenas. In that case, you should consider if you want ZIL / L2ARC / Special device. Especially if you go with piecestore, some kind of caching / special dev would be very helpful.

Raid / redundancy in ZFS will certainly take away its chunk of parity space, however it’s a matter of personal preferences. The general recommendation from StorJ is 1 disk = 1 node, but it’s also that you should use what you have running already.. so these two can often become counterparts. In my case I have a ZFS system with all the bells and wistles running, so for me it makes sense to use it for nodes as well.

Some does something inbetween with a single disk vdev and adds special dev or cache partition with tweaks to favor metadata caching. (there are MANY good threads in the forum for these discussions)

Don’t worry too much about it, it will take a looooong time before you fill a disk, and in the meantime you will have gained more than enough wisdom to set this right. Also, the node will stop accepting upload if it gets near the available space on the device - regardless of what you have specified. Also, this can be set in the startup parameters, so you can change it with a simple restart of the node.

For example, my nodes are generally run on a ZFS pool used for other purposes as well - I manage allocations with quotas for the dataset used by the node. With this it can be changed as needed, and gives more flexibility.

This is really cool - I wish I had the option to run more solar! :sun_with_face:

–

I hope these quick points helps you on the way to becoming a happy SNO :slight_smile:

1 Like
3

It doesn’t matter because it will never fill anyway. My oldest node is at 7.5 TB after more than 4 years. :wink:

7 Likes
4

Thank you very much for all this valuable info! My 5G connection is open as I asked my provider to move me from CGNAT and I am able to portforward without issues.
I do not have a static ip but DDNS works great and I do run some things such as Jellyfin, Jellyseer and a game server over DuckDNS and it has been working well for me so im not worried there also I do not have traffic restricitons its “flat rate” so I can “mooch” as much as I want :sweat_smile:.

You are correct I will use ZFS maybe add a cache drive/ssd would be a good idea here!

5

Oh ok maybe should I downsize to like 5 drives so I dont have all the drives spining for no reason? I could always hotswap more drives to the HBA at anytime!

1 Like
6

This:

Unless you want multiple disks for redundancy reasons, suggestion is to start with 1 node and 1 drive and add more space when the node grows.

3 Likes
7

All nodes in a /24 subnet are handled by storj like one node. So multiple nodes need multiple IPs to make sense.

8

The last thread like this I think I was hearing a fill rate of 2-4TB/year (per /24 IP) was a reasonable expectation? But if you have the gear anyways: may as well try!

9

On my setup I see closer to 4 than 2 / year. This is different fiber connections in EU.

10

So internaly its seen as one node.. its not an issue to make multiple vlans on my switch if thats the case would you recomend than adding multiple vlans so I can separate the nodes? Or it just does not matter at all according to you?

11

I can’t think of a reason to isolate nodes on individual vlans: but you-do-you ;). All Storj is looking at externally is the /24 of the IP. Like if you had three public IPs of:

123.156.189.1
123.156.189.2
123.156.189.3

… the satellites would treat them like one, and spread traffic across them. So they wouldn’t fill any faster. But if they were:

123.156.189.1
123.156.190.2
123.156.191.3

…they would each get their own individual traffic (so more uploads, so fill faster, so more $$$)

Edit: Yes, that means if say all other ISP customers in your area/building have IPs with the same three initial numbers… you share traffic with them. They’re your “neighbors”: you can check to see if you have any here.

2 Likes
12

There have been quite a few instances reported here of DuckDNS not updating correctly when IPs have changed. Do a search of the forum to see the issues reported.

Cheers.

2 Likes
13

You can’t know that. You only know that it resolves the same address from the same approximately network location that updated the address, and therefore likely when talking to the same instance of the service. If the replication is borked (because DuckDNS is a shit on a stick an unimportant hobby project of two dudes who lost interest in it immediately after starting it, it has been borked on many occasions) nobody else will be able to resolve your host.

Don’t use crappy DDNS providers. Use Cloudflare. Because if cloudflare breaks – we’ll have much bigger issues to deal with, and therefore it will be quickly fixed.

2 Likes
14

to chime in with what everyone else has said…

if you have any kind of bandwidth costs, you may not want to host. you will start using terabytes per month with storj.

The fill rate will be crazy slow. I have two year old nodes hosting 5.5TB. You don’t need to allocate all your disks. You would either peel apart one disk and use it separately from the rest of the array, for storj. Or just use your regular array with whatever redundancy level you have set up.

if you use the current standard pieces store, you’ll have millions of files on your node. So a SSD for l2arc or mirrored SSD’s as a ZFS special vdev can help performance. The in-the-works hashstore may make this less important.

2 Likes
15

Truenas will be sluggish with 8Gb, really want a min of 64Gb.
I would not recommend one large vdev. Be slow, and high risk of failure.
I would use either 3x 4 wide raidz1 vdevs or 4x 3 wide vdevs. Good balance of speed, storage space, redundancy.
Special Dev (metadata cache) is desirable, but not required.

1 Like
16

Please consider to replace it with No-IP or Cloudflare: Search results for 'duckdns offline order:latest' - Storj Community Forum (official) and Public Service Announcement: DuckDNS.

17

Thank you all for your suggestions I see a lot of you have issues with DuckDNS while I run a cronjob every 5 minutes it has worked fine for now for my projects I see it can cause many issues and use CloudFlare instead!
@Roxor ha yes I was confused I must say I thought you where talking about internal network so I figured well its not a problem to separate them internally :sweat_smile:

Thank you all for suggestions I know this are based on your experience and will do as suggested so I dont have to redo the entire setup down the line for rookie mistakes!

18

Seems you missed the point, not we have issues with DuckDNS, their servers misbehaving in other regions. So basically they returns an error to the satellites instead of your IP, thus online score is dropping or your node even considered as offline, it also affects QUIC somehow, at least when other SNOs changed the DDNS provider from DuckDNS these problems are completely gone.
But yes, it can work stable in your region. You may consider to setup a monitoring like UptimeRobot or similar to see how your domain is actually visible from other places of the world.

You do not need to separate networks, you just need to use unique ports. See How to add an additional drive? - Storj Docs

19

Absolutely useless and a waste of time. You will understand in time.

20

What do you mean by this?

Running a StorJ node - if done as suggested - on hardware that’s underutilized and already online is completely free, and any non-zero amount of money paid is free money.

2 Likes