TLDR: Please post the things you wished you’d known when you first got started, and I’ll create a summarised list.
Background
So, three years ago, I found a fully functioning gaming PC in the street! Someone was throwing it away! I thought this is a great chance to get started with Storj and so I installed a 18TB ultrastar WD HDD and got it live on Windows 10.
It’s been running basically flawlessly for the last 3 years and the ROI, compared to other asset classes, is actually pretty impressive.
So, now I’ve decided to scale it!
Request
I thought it might be useful for the community if people could add some things they really wished they’d known right at the beginning (for instance @snorkel has very helpfully pointed me in the direction of switching to Linux).
I’ll periodically come back to this initial message and update it with a summarised list of the key things discovered, thus created a valuable, growing depository of best practice!
I disagree on this one.
Depends whether you have a SSD you can use as special dev for meta data (preferably mirrorred) or L2ARC with only metadata caching.
In which case you can even use SMR.
@JWvdV Let’s not go into the whole zfs discussion again. There are topics dedicated to this, and mostly other topics that have been taken over by such discussions. There are some people here that think that it’s their whole life’s purpose to defend storj on zfs, otherwise they can’t fill the void in their soul.
I started my reply with “my list”. It may or may not apply to everyone.
Well, I agree. I hated them too, till two months back. Till some months back, this and this topic came in. Which changed my stance against ZFS, with a couple of other remarks in some posts.
So my list, I wished I knew:
MergerFS or any RAID0-like solution, is a bad idea increasing your likelihood to lose the whole node. Just keep with 1 HDD per node.
That it really takes some time to set things up in a good way. Still learning. It really doesn’t make me money, but it does teach me still every month some skills.
ZFS isn’t memory intensive, if your use a mirrored special dev (needs two partitions of equal size on two different SSDs, to be safe) or an L2ARC device with metadata caching (needs only one partition on an SSD).
I started on Windows, repurposing a mining rig that already had Win 10 pro (bad OS for mining, but I didn’t knew better). After that, I moved the nodes in Synology NAS 1GB RAM, because that was what I had.
And than I bought newer NAS, upgraded to max recommended mem, dicovered the Synology lies, and upgraded to “unsupported” max mem.
And after all this jurney, I discovered the limitations of OEM NASes and the fact that the best system for Storj is a PC that you put yourself together, with a flavour of linux server, low power CPU and max memory you can put on it. And no USB connections to drives.
So, the main things I discovered and hadn’t a clue when I started was the importance of RAM and the fact that Synology lies about hardware limits. I didn’t knew about buffers and cache and how well Linux knows to use all the resourses available, and not waste them unutilised.
And second was that Linux beats Windows for Storj usecase and server stuff in general.
And third is that ext4 is/was the best file system for Storj.
DBs on SSD (even if just for more reliable metrics)
expand when full
sell every coin as soon as you can: never hodl: never buy
patience
Then basically ignore things unless you get an email saying your node is offline.
(Edit: one more)
People in the forums will have different opinions. It’s OK if they’re wrong: there’s no need to correct them. But speak up if they’re misleading others in an egregious way: or if it would be funny )
My kind of SNO.
“Start and forget” it’s my way to go too. No point in watching graphs and logs. There are a lot more funny and useful ways to spend your time.
What if, I have a Dell server with 10x 4tb drives in a raid 5. Clears 30tb for storj after 10% overhead host has 32gb of ram, and 2 other drives to host the OS and DBs
Would it be better to unraid, and run 10 nodes on one device?
My 2 cents for myself or anyone looking to start a node:
Get comfortable with Linux and the command line. There is a lot of tuning for both the OS and Docker that are not part of the standard Storj install that could improve the setup.
RAM is king. You can never have enough RAM. Max out all the RAM you can cram in to your system.
Back up your node identities
For instant payouts, just stick to mainnet and use a CEX address (double check their minimum deposit amount however).
Do not run more than one node on an UNRAID array setup.
I don’t think it has to be unraid: but the standard answer is to not waste any HDDs on parity/mirroring and to always use them to make more money (run another node) instead. But maybe you need RAID5 for the other apps you run on that server?
Of course it will be better. You can upgrade the nodes one by one as you go. You can’t do that with a raid5. Not to mention that 10x raid5 is a recipe for disaster.
Incubate lots of nodes. Keep your future-use nodes small so that the data is easier to migrate as you eventually acquire storage for them.
Do everything via docker-compose. Makes it very easy to change node configurations and view your configuration at a glance.
Whether you’re using ZFS or not, get your filesystem settings right before your node gets big. Wrong settings can cause wasted space and write amplification.
Be patient and wait for great deals on hardware. Much easier to make a profit off drives if you’re able to get them for pennies on the dollar.
This is great!! So helpful, thank you all! I’ll put together the summarised list really soon!
One thing a lot of you have mentioned (@Mitsos, @snorkel, @Ambifacient) is RAM, but it seems no one has yet to put a number as to how much is needed per TB
Anyone have any thoughts on this? Let’s assume Linux setup, since that seems to be most favoured.
Yea. we got some estimations for 0.5gb/TB for linux and 1(-2)GB/TB for windows.
Personaly me did not hesitate to go from 32gb 3200mhz to 64gb 2400mhz ddr4 (due to maiboard restrictions)
I think the minimum would be 16GB windows and 8GB linux. BUT: simply the most possible/affordable amount should be right.
(so it ranges from 8 (raspi5 with 16TB drive) to 128GB or more (more will be overkill in most cases, if not involved in primocache) per system.
Also having the cheapest UPS was not a bad advice.
But as long the “use what you have”/“buy used” is king: its dreaming.
Nice to know: SAS-controlers are compatible to SATA with the right cable!
Plan some Slots for SSDs/M2 (2+) and some for 3.5" (3-10?) according to your (max) internet bandwidh 25Mbitdown and 5up per node if possible.
DO NOT put the system in your bedroom/living room.
The node software has gotten much better in terms of performance. The satellites are also better at not overloading an underperforming node.
You don’t need as beefy of a system anymore. You also don’t need as much disk write performance, as syncing is disabled by default. In other words, your system really only has to buffer the write and it’s good to go.
As for reads, it matters a little, but the payout for egress isn’t as high now, so it’s not quite as important to be able to win upload races.
However, things have changed in the past, and they’ll probably change again. Maybe node performance will become more important, or there will be some minimum allowed level of performance. Who knows.
