High Iowait (mostly) on startup

Node Operators
, , ,
1

Hi all,

I have noticed high Iowait (mostly) during boot on my main system. It runs StorJ exclusively and consists of a 3960x with currently 64GB of ram. All DB files are stored on an NVME SSD, which is also the boot device. The nodes run via docker and docker is also installed on the NVME. I have read some posts here about high Iowait but I am still not sure what the exact problem / solution in my case looks like. I know for a fact that the NVME is being overwhelmed with at times close to 10,000 IOps (reads), but I don’t know the exact reason.

CPU Utilization (restarted docker at ~18h):

image
image923×264 38.7 KB

Disk IOps:

image
image919×307 37.6 KB

Is it just the DB files being checked on node startup causing the load or can it also be the Docker root directory as all the containers being run overwhelms the drive?

Could a simple raid 1 with two high performance NVMEs be enough to tackle this if i moved all DB Files over there, or should I go for 4 drives in either raid 1 or raid 5?

Thanks for any advice on this!

2

It’s likely a filewalker. If you have several nodes pointed to the same device - no wonder.
You should go with the one disk per node approach.

3

Each Node is running on its own HDD. Just the DB Files are located on the same NVME Drive.

4

How your HDD is connected?
And why your SSD is overwhelmed if there is no node’s data?

5

HDDs are connected via 6 GB/s JBODS.
Why the SSD is overwhelmed is exactly my problem i am not quite sure. I can observe the high amount of IOps but cannot locate the source. I thought that during startup maybe the DB Files are checked by the node or something like that.

6

none of that.

it is not “overwhelmed”

My databases for one 9.5TB node are on an usb-flashdrive with 60mb/rw bandwith, doing fine.
The 10.000 iops are a slow start for an nvme. they have 230.000-1.550.000 iops in general.

The cpu iowait is high, but what could be the cause?
What kind of ram is your setup using how are the channels/banks distributed? wich ram is it?

in general: for what device is your cpu waiting? i bet its not the nvme. i think it is related to the filewalks wich led you to the nvme. but its not verry likely that an nvme is slow. there are other Bottlenecks.

this could be also an OS or driver problem.

1 Like
7

Interesting. I am using 4x16GB G.Skill Trident Z Neo DDR4-3600

8

I assume xmp 2.0 is enabled via uefi,
or what mainboard and os is it?

Nice choice btw.

9

xmp i would have to check but i am running Ubuntu 20.04.6 LTS on a Gigabyte TRX40 AORUS PRO WIFI mainboard.

10

So the 4 modules should sit in DDR4_A2, DDR4_B2, DDR4_C2, DDR4_D2 slots.
Are they there?

11

is there a specific need to run Ubuntu ?
As there are no Linux Drivers from the manufacturer. (afaik)

What type is the nvme?

12

Yes they are. Windows on a server is not really an option for me and Ubuntu is just my goto when running linux.
The NVME is a Samsung 970 EVO Plus 1tb.

13

Ok, for me you are good to go with further inspection.

It could be, that the HDDs are on the same PCIE lanes as the NVME.?

The nvme should not sit on 1 M.2 Socket 3 (M2P)

the other 2 are directly connected to the cpu. only the max.length is different.

14

its under tweaker-xmp - profile1 (instead of diabled) in advanced mode at the uefi.
If MB refuses to boot after 15 min just change it back.

15

M.2 is installed in the M2M Slot so it should be fine. The server is in a different location but i will be there sometime in the next few days to check the xmp settings.

16

Are there other raid volumes? if not, should it not beeing in ahci mode?

17

No Raid, HBAs are running in IT Mode so each drive shows up individually.

18

I hope you didn’t mean JBOD actually (RAID0, mergefs, LVM/zfs simple volume, etc.), because with a one disk failure the whole node is gone.

So, they are actually a separate drives, I hope.

Usually high IO wait should be related to HDD, not SSD.

19

Most likely.

Do you run an extension card (wich one?) or the mainboard controler?
if mainboard, the raid controler could be disabled, using only the direct connection with the MB-sata ports, eliminating an unneccesary step.

20

Yes, no RAID all individual Disks.