I run 3 nodes in docker images on this machine. 2 of the 3 are working fine. I’ve suddenly begun having an issue with the oldest (and largest) node a couple days ago. It only stays up for about 10 to 15 minutes before crashing with a thread allocation error. Docker restarts it again after about 10 m…

Try setting max-concurrent-requests to be some finite number, maybe 75. For me it stopped the storagenode from spawning too many processes.

If you set the max concurrent requests, it’ll tell you the active count whenever it’s exceeded, for what that’s worth. And if anyone at Storj is paying attention ( @Alexey ?), maybe this is a consequence of too many nodes dropping off after reduced payouts, combined with recent surge in traffic? An…

Yea I saw that, I set it to 300 and still maxing that out but it hasn’t crashed either. Still getting upload rejected at 300, but if it runs that is better than the crashing.

My update: setting max-concurrent-requests to 75 seemed to be a decent workaround, as the node never went more than 15 minutes without running out of threads (or RAM/getting killed by OOM killer when I turned off swap), and I just achieved over an hour. I guess everyone will come to a different opti…

[image] consumerbot797287: runtime/cgo: pthread_create failed: Resource temporarily unavailable [image] jarridgraham: runtime: program exceeds 10000-thread limit fatal error: thread exhaustion You may try to increase it: docker run | Docker Docs I notified the team.

I set max-concurrent-requests to 75 as well, the node has been stable for 3 hours now but its still using 6 gb of RAM and 30-50% CPU

[image] Updates on Test Data Announcements Hello Storj Community, We are excited to announce some changes that we have begun to implement regarding how we are using test data on the network! What is changing We are currently seeing exponential growth in our paid custome…

My issues began on 2024-04-26 at 19:24:17 CST. Since you marked this as a “solution”, I assume that means that your synthetic testing was the trigger? I signed up to share unused storage capacity. If the HDDs are already spinning, might as well put them to use, right? But if it comes at the expense…

[image] consumerbot797287: overburdening my server’s CPU and RAM, Anecdotally, it does not “overburden” cpu or ram. I did not notice any difference. CPU is almost idle. Ram usage stayed the same. I did notice slightly higher power consumption due to higher disk activity, but that’s not unexp…

[image] Alexey: You may try to increase it: docker run | Docker Docs Are you suggesting that your software needs more than 35k+ active threads? I joined Storj to share excess storage, but not to have my server subjected to a DoS attack. Please forgive my bluntness, but I hope that you (and t…

I check back my nodes, some of my nodes with swap was also intermittent down time and autorestart (I disable swap and the problem seem go away so I didn’t bother to check logs): 3a0122da2cd8 storjlabs/storagenode:latest “/entrypoint” 4 weeks ago Up 12 hours bbc8961f0c3e storjlab…

Node goes down/restarts every 10-15 minutes. thread allocation error in logs

Node Operators troubleshooting

jtolio April 30, 2024, 3:49pm 33