Just based on that screenshot alone I feel like I might need some help from everyone here to see what could be the bottleneck for my node?
System Specs:
OS: Win10
HDD: (the 20GB RAM one): ST16000NM001G-2KK103 (16TB)
Ram: 32GB DDR3
CPU: A10-7860k
Edit:
I just watched the RAM usage drop to 12gb rapidly (and the network usage dropped as well - seems like I started failing races and satellite stopped selecting it for a bit) but now it’s picking back up.
I’d venture to guess – NTFS filesystem is a bottleneck, that cannot batch writes, cannot cache metadata sufficiently, and does not take advantage of available ram.
Did it max out disk IOPS? You can glean much more information from performance monitor, task manager is pretty rudimentary.
Not sure what I should be looking at exactly here…
Disk F: is the one in trouble
IOPS. Input/output operations per second. I’m prrety sure that’s the problem. The Task Manager also shows it on one of the plots.
The bandaid solution would be using something like Primo Cache to buffer writes to the disk.
I can look into that - but is there a non-bandaid solution (that isn’t migrating from Windows since I have too much data and no drives to hold it to move it over onto)
go to Overview tab and click on a disk on the left
Nope. Bandaid can work pretty well for years though.
Hmm… (This is where me not using windows for the past 7 years shows…) There used to be a rectangle with a disk … googled the picture:
if the active time is maxed out – disk can’t keep up with the io load, and node start buffering in ram. The Primo Cache will help offload some of the IO from the disk by caching (reads and writes) and thus make it bearable.
By “performance monitor” I meant this one: How to use Performance Monitor on Windows 10 | Windows Central . You can add very specific counters, including disk IO load.
I’m restarting the device now to test out primocache - any recommendations how to set it up? 
And I will also look into the performance monitor (I confused it with resource monitor)
For Primo Cache you would need to add an SSD and then you can configure two tiers of cache – one in RAM, and one on SSD. For node I’d say I would completely forego the in-ram cache, and allocate the largest SSD you can find for the second tier cache (I think they call it L2). Since this workload is mostly write intensive, cached writes is what you want.
If you don’t have an SSD but have tons of ram - in-ram cache might actually help somewhat too. You would need to expereiment and watch the cache hit rates; then adjust, wait for a while, measure again, etc…
I’ll assume I shouldn’t use my OS ssd here as a cache because it would kill the SSD quick
Right, but not because it will kill it. I think you have to give it full physical SSD. I would not worry about exhausting SSSD resource , modern SSDs last forever. I’m buying used SSDs on eBay (enterprise grade, from old servers, with 10-20% useful life remaining, and they work for ages)
BTW, another option is Enmotus FuzeDrive (but I believe it’s discontinued) – unlike cache, its a tiered storage solution – so under right conditions your metadata will end up on SSD and data on HDD, and it will be even better. But I believe it was discontinued (at some point AMD rebranded it as StoreMI I believe – so if you have AMD motherboard maybe you already have access to it)
My OS drive is Samsung 870EVO 1TB. I can make a partition for ~256GB easily would the be fine? Do you say it needs the whole SSD because the OS does a lot of random writes/reads?
I think Primo cache consumes the whole SSD, but I may be wrong. Try it.
The problem with using 870EVO is that it itself is a tiered solution: has a small SLC cache SSD in front of slow large portion of TLC/QLC storage. It works well as boot drive exactly because frequently used data ends on that fast SLC cache and the rest on TLC/QLC.
It is not a good idea to use such tiered SSDs as cache SSDs – they will indeed wear out much faster.
If you have PCIE ports on your PC you can buy ultra cheap (sub $10) 16 or 32GB of Optane SSDs on eBay. That’s an ultimate cache device. But this is later, after your proof of concept actually demonstrates that primo cache or fuse drive actually help.
Yea I had a feeling - I remember choosing it for one reason or another but forgot why I decided not to use it for caching it’s coming back now 
Oh, separately, make sure you have last access time disabled on the volume that holds storage node data – this will remove quite a lot of IO:
And if you haven’t done so yet – move databases to your system drive.
We should have started with this (but I assumed you have already done that, but I"m not sure, so double check 
That doesn’t seem to line up?
You want DisableAccess to be 1.
I think it’s poor wording. The “DisableAccesss” feature is “Enabled”, so updates of access time are disabled.
I see it’s inverted - do I need to do this per drive?
