Question about some config.yaml options

Node Operators FAQ
,
1

I’ve just finished copying my movies and series collection from the same disk I run Storj on. Later, I’ll remove them completely. I also cleared some space and went to config.yaml file to add another terabyte of space to Storj. I got interested in some other parameters, that might make the node work better:

  • in-memory buffer for uploads
    filestore.write-buffer-size: 512.0 KiB (I’ve increased it from 128 KiB)
  • file preallocated for uploading
    pieces.write-prealloc-size: 16.0 MiB (I’ve increased it from 4 MiB)
  • how many concurrent retain requests can be processed at the same time.
    retain.concurrency: 20 (I’ve increased it from 5)

Could someone explain those 3 parameters to me and how increasing them would affect the node?

2

Why would you change settings if you didn’t know what there for and how it can affect your node?

3 Likes
3

I have some idea what they do, but I would like a more detailed explanation.

4

Oh ok well
What they do is in the name,

  • filestore.write-buffer-size
    Which loads your ram with data before going to the drive. Good for SMR drives.

  • pieces.write-prealloc-size
    It preloads data before the data is there. So it will prevent anything else using the space.

  • how many concurrent retain requests can be processed at the same time
    This is how many files can be uploaded at once before it will give an error. Normally you wouldn’t touch this because it will handle it on its own. Unless your having issues with losing the race or low powered nodes.

If your question is does it help the performance of your node in anyway. This really depends on the system that the node is running on the hard drives that are being used. If its a pretty high end system I would say no this really doesn’t do much at all keeping settings at default is usually the best bet.

2 Likes
5

Thanks, the first option is definitely the most important for me as my disk is starting to work slower.

6

I would be careful with it though if your drive is too slow your ram will fill up error out and you may cause more problems.

7

You’re preallocating 16MiB for files that can never be larger than 4 MiB.

It’s not… this is a different setting for the retain process (garbage collection). A higher number will only put more load on HDD.

In general, don’t mess with settings you don’t understand. It seems like at least two of the changes would have adverse effects.

Setting this one to 4 MiB might help if you have enough RAM. That would allow it to keep the entire file in RAM and write it to the disk at once.

1 Like
8

I’ve also stopped the node right now and started defragmentation. The orders.db file has over 13k fragments scatted on the whole disk - it’s madness. And 2 MB files have over 500 fragments. No wonder it’s working slow.

9

I have 32GB of RAM, so it’s not a problem.

10

I leave the write buffer at 4 MiB and the rest on default.

1 Like
11

ah so its even worse to mess with that setting then. Thought it was the old setting for connections.

12

Untitled
Untitled764×203 3.57 KB

This looks pretty bad…

13

I would disable defraging on that drive you will end up killing it.

14

It’s probably better to put some strain for an hour or two to sort things out than to tire disk more in the long run.

15

Your OS isnt running on the same drive is it?

16

It’s a 6TB external HDD. I have Windows on SSD.

17

Ok good. Also im sure that 6TB drive is an SMR drive as well thats probably why it seems like its getting slower.

18

At some point I couldn’t watch a movie or episode without lag from disk read. That’s when I started to copy everything to internal drives. When I’ll buy a new external disk to have a second copy, then this 6TB drive will be for Storj only.

19

Yeah I can imagine. SMR have good read but if any write is happening a the same time say goodbye to any performance of it. I to had SMR drives I upgraded to CMR drives.

20

Well, I bought the drive before joining Storj, so I did not know, that SMR is a problem. If I’ll buy any more drives for Storj, then I’ll take a closer look at this aspect.