Errors with ubuntu22

[Egaria]

Good morning,

For a few days I have been having these errors, they are causing my I/O to go up and I don’t understand why.
I haven’t touched anything on this VM for 6 months…

Thanks

[Toyoo]

This is a symptom of high I/O, not the cause. You need to find the cause.

[Egaria]

Do you know how I could find the cause?

[Toyoo]

Looking at your logs (both host and VM) and monitoring to check if there are any obvious anomalies would be a great first step. Also, I would try to identify which parts of your setup (physical HDDs, all sorts of block devices like partitions/LVM/RAID, file systems, etc., again both on host and VM side) do observe high I/O, so that you can limit your investigation to just those components.

[Egaria]

I don’t think it’s a problem with the hard drive because my other hard drives have the problem…

[Egaria]

I noticed that these errors occur when the storagenode container increases my I/O, which arrives between 300% and 1000%…
During these heavy loads if I stop the container the I/O reaches a normal threshold. How can I check what the storagenode container is doing?

[Alexey]

You may enable the debug port

Guide to debug my storage node, uplink, s3 gateway, satellite

We have integrated a package called monkit in all storage nodes. It is very usefull for debuging. By default it will open a random port. In order to connect to the debug port you will need to: Add debug.addr: ":5999" to the config file. Now the port is fixed and will not change on every restart. Add -p 127.0.0.1:5999:5999 to the docker run command. Ready for action! You can request the following informations: curl localhost:5999/mon/funcs Find out how long a function like GetExpired need…

and check the report

@littleskunk

curl localhost:5999/mon/ps
What is running at the moment and how long is it running.

You may also use other endpoints such as

@littleskunk

curl localhost:5999/mon/funcs
Find out how long a function like GetExpired needs to execute.