Memory consuption

[Adavan]

Using “collectd” (collecting metrics) -> InfluxDB (database backend) -> Grafana (GUI).

[spewie]

Same issue here too. storagenode consuming all memory.

[Odmin]

@mirnyyy

Btw, guys what are those graphic monitoring tools you using?

Simple and power way, use netdata

Installer:
bash <(curl -Ss https://my-netdata.io/kickstart.sh)

[mirnyyy]

Will check, Thank you.

[BrightSilence]

Issue should be fixed in v0.14.13 according to the changelog here:

Changelog v0.14.13

Changelog v0.14.13 Storage Node: Fix memory leak https://github.com/storj/storj/pull/2553 Note: This fix only removes the memory leak we have added in the latest version. It doesn’t fix the memory leak we had before that. Satellite: Change repair queue order First try to repair everything that is new on the queue and push the retries to the end of the repair queue. https://github.com/storj/storj/pull/2533

If you don’t use watchtower, please update!

[Adavan]

@BrightSilence Thank you, just I readed changelog. Updated, looks better, let see later.

[Odmin]

Thanks @BrightSilence !
Seems like issue fixed, I not see that memory consumption is growing like on previous version.

[BrightSilence]

I appreciate the thanks, but I’m just reporting it. Thank the storjlabs people for fixing it so quickly.

[Odmin]

I can confirm, issue was fixed!

As you can see, after upgrade to new version memory consuption is normal.

[Adavan]

Confirm too, issue fixed.

[Lucas]

My ram is rising again !

[Alexey]

Please, show the rest of htop, where we can see how much used by the storagenode container.
Also, do you have a dashboard running?

[Lucas]

[Lucas]

[Alexey]

As you can see, the storagenode uses only 862M, which is below the 1GB

[Adavan]

@Lucas run command docker stats, looks like:

Here you can see more detail information about containers.

In few posts above I wrote about docker parameter to limit RAM usage (my node have 6GB, but placed 4GB as limit for storagenode).

[Lucas]

Thanks a lot you two !

[spewie]

Memory consumption has reduced greatly. Thank you.

[karacurt]

@Odmin

I strongly reccomend use timeout, docker will waiting only 10sec by default, the it kill all process inside container!
As result, you can destroy your info.db

Would you get some clue what measurement of timeout should be and what the docker command?

[karacurt]

@tyakimov

will add a 3 min timeout, but 3x times now the container stop will not finish at all. that only happens when i step in when it’s crashed already and try to stop it. let see if the container can stop as expected b4 the crash. i am also clearly having all the other issues like the
upload rejected, too many requests {“live requests”: 7} and
ERROR untrusted: unable to get signee: trust:: node not found

got the same problem with “upload rejected, too many requests {“live requests”: 7}”
what the solution or where the direction to dig?