Storagenode container using 100% cpu

Node Operators troubleshooting
,
21

Can you explain a little more about what kind of storage you are using? Model of HDD(s), unRAID setup, is there a cache for that unRAID array?

Your previous message seem to suggest you do use a cache for the data array, but it would make nu sense if there is minimum activity on there if a node is running.

22

I have a array of 9.
2 parity disks of Seagate ironwolf NAS 4TB
Disk 1 Seagate barracuda 4TB xfs (personal use)
Disk 2 Seagate barracuda 4TB xfs (personal use)
Disk 3 Seagate barracuda 4TB xfs (Storj storage use)
Disk 4 Seagate barracuda 3TB xfs (Storj storage use)

Cache Kingston 240GB btrfs (docker.img use)

and 2 unassigned devices

I dont use cache to storj data.
The only thing thats on the cache disk now is docker.img and libvirt.img

23

Well write speed on unraid without a cache aren’t exactly know to be great. You’re sharing the parity disks among all hdds so load on other hdds will impact it as well.
I’m still a little confused by you calling the SSD cache while not actually using it as a cache?

Do you happen to know whether those barracudas are SMR drives? What’s the exact model?

24

It have been said that using cache woth storj is not a good thing, so i only use it to docker.img and to my private fileserver.

The modle name to
seagate ironwolf 4TB: ST4000VN008
seagate barracua 4TB: ST4000DM004
seagate barracuda 3TB: ST3000DM003

But when i copy stuff to my private hdd’s the write speed is 80 MB’s and up without cache.

25

I don’t think the speed of the drive is the issue though because test copying 4000 files that are only 2MB each and see how much stress it causes a hard drive without a cache.

26

I going to try that right now

27

It was first 50MB’s andthen it went to 18MB’s.
But i also tried yesterday to use cache disk with storj… no difference. cpu 100% wa 98-100%
Is it possible that the docker.img is little corrupt?

28

But have you tried to use that same hardware without unraid with storj?

29

When this problem started, i had different hardware. i7-2600 with 16GB ram.
I switched hardware 2 days ago to a ml250 gen5 server and the problem followed.
i7 server is now up and running a new server without any problems.

30

Is the i7 2600 running another unraid with storj on it?

31

Yes. It’s running unraid 6.8.3 with a new storj node

32

But is it a new storj node though with alot less space used and a smaller database?

33

Yes. I started that node 21 hours ago

34

So I have a theory once that node gets alot more data it will do the samething…

35
  1. It’s going to take almost a year to figure that out (it will take a loong time to get 6TB into that node)
  2. This problem happened after a storj node update, database malformed and a unraid upgrade from 6.8.2 to 6.8.3

This happend late 27 mars and my node went down. before that my node work perfectly. No high cpu usage, no wa.
I got the node up and running again 2. april and and right after the container started the cpu and wa immediatley went up.

36

So you think the issue was cause unraid upgraded and the storj node updated? Possible but l think theres more to it with the way unraid runs. I hope that it works better but I still have this feeling it could happen with this node. I wouldnt use unraid for anything serious though but that is just me. Have you ever tried promox?

37

Never heard of promox

38

Oh well its free to use unlike unraid is. But im guessing you already paid for unraid tho.

39

Your issues also started right at the moment the current heavy upload load testing started. I highly doubt it has anything to do with the new version and it probably has a lot more to do with high amounts of uploads.

So… Here’s the bad news.
ST4000DM004 uses SMR technology. And during these recent stress tests, we’ve seen many SNOs run into issues with SMR HDD’s. Even though modern SMR disks use large caches to overcome the shortcomings, there is simply no way an HDD like that can deal competently with continuous random write behavior like storagenodes use. I think you may be able to overcome this with SSD cache, but I’m not entirely sure that won’t just postpone the bottleneck.

I really don’t like that HDD manufacturers are now silently sneaking this tech into some of their HDDs. Be very aware to avoid them if you’re buying new HDDs. It’s definitely not worth the tiny cost savings.

40

But again… When i activate cache disk to storj the problem is still there. Its no activity on the seagate disk’s when the cache is in use.

Is there a way to edit how many requests i can have on that node?