Problem with docker. 100% cpu usage and nodes go down

Node Operators troubleshooting
,
1

Hello - i have a docker setup in windows 10 home.
I have 8 storj nodes running all on this docker.
I have the following issue:

I can have my machine do amazing for 2 days (ish) - but then all of the sudden my cpu just spikes to 100% usage and all storj containers become unresponsive. The only fix is to restart the computer.
In joblist in windows - it is the process called “vmmem” that uses 100% cpu.

Here is what i have tried:
update docker
dissable filewalker for all nodes
make sure all 8 is running with CMR
added ram to a total of 32GB

My system consists of:
i5 8400
32GB ram
cheap motherboard
good powersupply (be Quiet - brand new)

Is the CPU maybe to weak to run 8 nodes and some dashboards?
Should i upgrade to i7 8700 (i have one laying around).

Also got this error after a restart:

IMG_6298.HEIC
IMG_6298.HEIC3024×4032 3.4 MB

So maybe i should try to reinstall WSL?

This is a big problem - because all 8 nodes gets downtime - and uptime robot does not catch this as port is still open… so if it happends at night - i have no way to know.
Thanks in advance!

2

It’s possible your disk subsystem cannot handle the IO, node(s) is(are) buffering data and run(s) out of ram.

Side note: Why are you using docker on windows? It forces it to run in a VM. Storagenode already has windows version, or you can run Ubuntu version in WSL. This will remove some overhead.

What is the underlying storage?

3

So - you suggest using the build in windows service instead of docker?

Also - i need to configure WSL to use ubuntu? i can give that a try.

Can you ellaborate on “disk subsystem cannot handle io”?
I can see not all my ram is used.

Underlying storage?

4

I just set the default to ubuntu - looking forward to seeing if it helps.

image
image994×414 93.1 KB

5

This is the current cpu usage after setting wsl to ubuntu and restarting the machine:

image
image1200×446 104 KB

How “hard” is it to move all docker nodes to windows services?

6

how much do you have them?

7

Sorry i dont understand? how much data? in total only like 2TB on all 8 nodes.

8

8 nodes that is what asked.

i have more and cpu is much less, and it is i3-10100

Screenshot 2023-08-10 111853
Screenshot 2023-08-10 111853842×752 104 KB

9

Yea- i think i need to change to windows service. Is it hard to change from docker to windows service? i can use your toolbox - but is there anything special i need to do?

10

You aware of 1 core per node?
@Vadim Nooooooooooooo! :shushing_face:

You only have 6 cores but 8 nodes.

The 8700 also 6 cores.but with 12 threads.

Also only one node per hdd?

Aside from the wsl error.

wsl
wsl1200×446 113 KB

Virtualization in bios activated?

11

I am not moved any nodes. Please do not use long paths for identity. I use C:\identityX where X is a number of node

12

1 core per node is made for low power cpu like ARM. I do not use them and not recommend.
only may be if it is 1 node per cpu

13

Node data drives excluded in Windows defender ?

14

req
req1920×1032 226 KB

come on…

15

interrupts
interrupts1200×446 103 KB

the nodes fight already for its core. from docker wich propably uses one core percontainer or smth.

16

because docker is crap(on windows it is defiantly crap, on linux I agree it work much better), I agree with you that it should have 1 core per node.

17

Nevermind, as long as it works :face_with_peeking_eye:

18

Thanks for the answer - yes only 1 node pr HDD - i have 8 HDD’s in the machine.

Virtualization is activated in bios.
I think 1 node per cores is not really needed.

19

make sure the ram works perfectly (also including the old one), using memtest or else.
(multiple processes to scan all ram not just 2gb)

20

So i have 1 stick of brand new 32GB ram. that is whats in the system. the old 16GB is taken out