4 Storj Nodes kill my system performance

Node Operators troubleshooting
1

Hi :slight_smile:
Storj has a high usage recently because of the changes.
My CPU Ryzen 5 5600G is utilized between 70-100%. Each node currently needs between 1-4GB RAM. Since I also use other dockers with the system, I wonder if the load can be reduced somehow or if I can optimize something?
When everything is full I wanted to add more HDDs, but my system doesn’t seem to have enough power?

I run 4 Nodes ext4 formatted. I think my drives are SMR? How much would I benefit from CMR?

System:
OS Unraid 6.12.x
Ryzen 5 5600G
32GB RAM
2x 12TB HDD WDC_WD120EMFZ, WDC_WD120EDAZ
2x14TB HDD WDC_WD140EDFZ

2

If you are using the Unraid dashboard to measure CPU utilization, then it also includes iowait, so it’s not really a true good measure of how hard the CPU is actually running taks.

Those drives all look like CMR to me.

You say these are formatted for ext4 which probably means you are using the unassigned drives plugin to do this, which is right because we don’t want to run multiple nodes on the array itself.

The only things I could suggest is to move databases to an SSD if you haven’t already, and to add more RAM.

2 Likes
3

Have you been seeing your drives fill up quicker lately: to account for the extra usage? It sounds like things are working properly: and you’ll be seeing higher payouts to account for it.

If you’d like one or more nodes to go idle for awhile: you can set their STORAGE variable to the space they’re already using (or less) so they’ll stop accepting uploads. Just remember if you’re not growing you’re not increasing the money you make every month :money_mouth_face:

1 Like
4

Move to Windows. My Ryzen 5700G/32GB Windows Server handles 8 nodes easily. :laughing:

5

Please don’t move to windows😱
Windows hogs so much more resources than Linux.
I don’t know how much unraid os uses. But I would use bare metal Linux.
My Debian machine can handle everything without any problem and is so ram and CPU friendly

3 Likes
6

You are right. Its really high now and makes sense. From max 90GB to now 300-600GB.
My internet is fine but my server is crying :joy:

1 Like
7

Yes you are right with your assumption. Is there a guide with DB on SSD?

I am not sure if they are CMR. The asnwers I found are all different :grinning:
The drives are originally WD My Books and Elements

Is there a better tool to measure CPU usage? H Top in terminal?

Thank you!

8

Yes, just top, it should show the IO wait too.

1 Like
9

Thank you I will give that a try and compare it.
Would it help to set the docker log level down to warn? I did not find out how to do this correct.
My docker wont start when I change the config.yaml

CPU usage looks different compared to unraid GUI. Interesting:

image
image1161×119 16.3 KB

10

I do not think so, but you may try to add to the end of the docker run command (after the image name) the command line option --log.level=warn. The same should work for config.yaml in the format log.level: warn, but it is required to be unique - so make sure that you modified the existing option, not added a new one.

I think only top shows an IO Wait.

11

In Unraid it works in the post arguments for the docker. In case someone need this :slight_smile:

I discovered errors I could not see before. I thought I removed any untrusted satellite but there seem to be 2 remaining?

ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "2NDLUW3HNYHMNPHMBKV7CE6UYALTOTP7QAAKCE4Z4XMRFLKFB2HA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 8960, "Remote Address": "109.61.92.78:51254", "error": "untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup us1.storj.io: operation was canceled", "errorVerbose": "untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup us1.storj.io: operation was canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).VerifyOrderLimitSignature:140\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:62\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:618\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:167\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:109\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:157\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-07-02T16:50:54+02:00       WARN    console:service unable to get Satellite URL     {"Process": "storagenode", "Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "error": "console: trust: satellite is untrusted", "errorVerbose": "console: trust: satellite is untrusted\n\tstorj.io/storj/storagenode/trust.init:29\n\truntime.doInit1:6740\n\truntime.doInit:6707\n\truntime.main:249"}
2024-07-02T16:50:54+02:00       WARN    console:service unable to get Satellite URL     {"Process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "error": "console: trust: satellite is untrusted", "errorVerbose": "console: trust: satellite is untrusted\n\tstorj.io/storj/storagenode/trust.init:29\n\truntime.doInit1:6740\n\truntime.doInit:6707\n\truntime.main:249"}
12

You need to remove the untrusted satellites:

1 Like