after i upgraded server with 8 nodes, from 8gb ram to 24 Gb ram it getting around 20% more ingress, so RAM is helping significantly, also after i upgraded lot of nodes from 32GB ram to 48-64 nodes working much stable and not turn off under big load. Before they turned off time to time by writability check.
gr8 but that will not speed up filewalker, more RAM helps for stability yes. Iām maxed out on mobo for RAM too at 64GB 
anyone here running 128gb of ram? does storj even use half of it?
storj is not useing it, but Windows will use to store MFT and reason used files
that will speed up storj.
I use 64 GB of ram work better than before
Individual nodes wonāt benefit from that much: but when you have multiple nodes, and the OS can use a healthy chunk of RAM for filesystem/metadata caching, it will help keep things running smoothly.
If I had 128GB RAMā¦Iād probably set up an Ununtu 24.04 install and let ZFS ARC have half. Then steal 16GB as a compressed-RAM-swap-device. Which would leave about 48GB for OS+nodes. Probably would run 50+ nodes easily?
How much ram usage you get? I use 32gb and like 25-27gb used constantly
So probably my 3 nodes wont benefit from 128gb 
1 Like
I was thinging about tiered storage but somepeople said it makes your hard drives work harder.
Now I was thinging about increacing ram to 128gb and maybe that would work better with less stress on hdds?
It sounds like help may be on the way without changing anything. (But more RAM is always better 
)
I have 14 nodes here.
Les nodes you have and more memory, this mean your pc can have more MFT per node in ram.
Hey, it looks you have deep knowledge about how MS Windows storage tiers work, especially metadata caching. Or maybe not knowledge but emotions.
I searched this forum and not found anyone using it. And since I use what I already have running anyway, I will give it a go. And if such setup will prove as not working great, Iām ok with it, nothing to loseā¦
you can try budger cache, it work from v1.108 I tried it on some nodes, i like it.
First time it takes time, then it faster. no need anything additional.
1 Like
What does the badger thingy do exactly?
1 Like
builds a little database about pieces in use, stuff that is currently only retrieved by querying the file system. That makes subsequent filewalkers and garbage collection much faster.
still considered experimental by storj.
2 Likes
Sounds good. Storage Spaces has itās uses, certainly.
If you plan to use parity at all, the gui defaults to a raid 5, with three columns, and 256kb interleave. If youād like to make a proper/better setup, then itās Powershell command line stuff.
For example:
The ultimate preference for a parity set-up, would be 5 disk, 16k interleave, amounting to 64k Cluster size. The reason being with stripes of 4 data disks at 16k interleave, you would match your formatting cluster 4x16 = 64k; meaning there would be no compute overhead at all for files larger than 64k, it would just bypass the writeback cache, and could sequentially write at max speed.
If you were to use a three column, you could set an interleave of 16, and with two data disks, you could match that by using 32k cluster size.
The difference is meaningful, 40MB/s vs. 160-200 MB/s (max of one disk)
Of course the parity read speed and the above efficient write settings, would be beneficial for nodes.
Anyways thatās a newbie trap to avoidā¦ 256k interleave, default gui setup.
Quick synopsis of other considerations.
-1GB writeback cache is the default. You can change that, but itās suggested not to go above 50% of the tier size.
-The automated heatmap scan of your disk every 4 hours, can be disabled; itās merely an entry in scheduler, that you can disable.
-When making a pool, the order of the disks added makes a difference.
-There is the opportunity when adding disks, to have the system self align data (evenly balance between disks) - think carefully ā¦ very, if thatās advantageous to your use case.
-Iāve had problems when doing fancy mirroring outside of server versions (in Win 10/11), where the mirror tier can break to a point of identical disks. Only one (1/2 the mirror) can be in the pool at a time - and thus, both cannot be loaded driver level at OS boot; your bootup will brick. As an unsolvable kernel mode panic ensues. Your drive & data is toast.
-Simple tier may be your friend.
-Storage spaces works nicely with USB disks, generally. Even if you controller is flakey, has bad cabling, or over taxed bus., frequent disconnects, etc.
-Tiering used to work the opposite way it does now. It used to push all new data into the fast tier, then draw from that fast tier into the slow tier. Now it will push data to the slow tier (aside from the writeback cache), and by schedule, pull the āhotā data back into the fast tier. As mentioned in some previous post of mine, this causes major - 100% fragmentation of your main body of slow tier data over a short amount of time.
-Using a simple pool, is a good way to just stack disks, and make a jbod.
-Once you choose your configuration for a tiered system, you cannot change it.
-REFS format will do some special things with metadata, and make a black box of itās caching techniques. NTFS will not, you would have to do your own special caching of file records. REFS is only flavoured in 4k, and 64k clusters.
-Also, Sectorsize, Physical sector size, and file record size are all specifically adjustable. For instance if you have 4k native disks, very useful - no write amplification, etc. You can even make file record sizes match, thus saving about 15-25% of your nodes files; fitting them nicely into file records; however of course files >4k will waste this space. Nevertheless, you do stuff like that correctly and even a hdd filesystem can be faster than badger cache, you donāt even need a fast tier.
-you can pin files specifically to the fast tier, so if say you want your .db files on there.
-Do not ever put an actively used dynamic .vhdx to a tiered system, the fragmentation will destroy itās usefulness; unless itās small enough to pin to the fast tier.
Etc., etc.
Any specific questions, Iām happy to answer.
6 cents
1 Like
Ah, and the fine thing is, you can change/pause cache configuration as you want, without restart.
I created the cache nvme partition while in use, and put 12GB ram on the databases drive, while reducing the OS ssd cache.
Nice upgrade especialy when hardware free.
1 Like
oh men itās getting bit expensive, not sure I can afford it 
This is not the case. When setting up Tiered space there is nothing like Interleave size and Number of Columns. Therefore, all the calculations do not apply here.
I donāt posses the server anymore, but will be getting similar setup in a couple of days, so I will get deep into this laterā¦
I tested Primocache on NTFS shortly, it did not cache metadata and did not help at all with filewalkers (what is the heaviest stuff on storagenode). Maybe I set it up wrongly, really didnāt have spare time. Maybe someone will be more lucky. When I search this forum for Primocache related posts, I donāt se anyone really using it long term and sharing their experiences.
A simple space, obviously not. And yes, any tier of a tiered space can have whatever parameters you want, the fast tier itself, could be mirrored, 3 way mirror, parity, whatever - in addition to any combination desired of the slow tier, or even a mid-tier; think multiple classes even. Apparently I wasted a detailed explanation for you. Above, I specifically referenced the backing data of any tier, for purposes of efficient parity acceleration of rust spindles - the slow tier. Any fast tier, will beget utter and complete fragmentation of the slow tier, if itās just a single hdd; without the concept of parity read acceleration/column spanning/etc. to counter balance such effects. For the data pattern of Storj, a simple tiered system would be pointless. Even so, my advice above merely scratches the surface of whatās necessary to utilize SS efficiently.
Good luck with those calculations!
Primocache is decent software.
Itās not a matter of luck to make it work. First you have to understand itās nature. It acts primarily on idle, so unless you park your node and purposely & properly seed itās cache, it will fall on itās face. Using the correct parameters itās fine, will fully accelerate your file-walker activities, etc. So long as you have the ram (patience and intelligence/knowledge), but then again you could just let the file system do that - and go by luck. Caching file records on windows isnāt rocket scienceā¦ itās read the file metadata from ram, at ram speed. vs. a max hdd speed of ~50k files per second. And itās simple math too, although nobody seems to have a clue on these forums. You have 10 million files to traverse/ātouchā? You need a cache available of 9.54 Gigs. And if the OS does it itās known as metafile cache allocation; be it files themselves (in their whole) or just metafile info.
Excellent for gamers who donāt need/have a clue. Itās best efficiency comes from exploring last access FILE times while target disk/volume is idle. ie: overnight. All the lovely games that ran and loaded their data files previous to itās last scanā¦it aggregates a heatmap of all your most popular run time hits. By pulling the underlying blocks of those files into itās own cache; one can consider itās file caching onto, and from - at a block level - thatās by design.
2 1/2 cents ā¦ 3 cents now (had to edit this)
2 Likes