Than You i will try.
I think I know what a root cause of this issue, I suspect some application that used Storj network (at this moment I not have proof and screen but I think we will have it very soon)
This connections is not idle, it active but with meager traffic (about 200bps), how to hunting for it:
- Debug port should be enabled
- Check active connections with
curl -s localhost:7777/mon/ps|grep -c "live-request"
- Check real-time traffic and connections with
pktstat -i eth0 -tPF
You will see a lot of little requests from this application to your storage node.
Im not seeing any of the issues with live requests or memory issues and Im running on a Dell optiplex Dual core CPU only 4gigs of ram. Running 2 nodes fine no issues at all.
Sorry, but you screen show nothing about network stack and current limits.
I can see only problem with disk system (iowait).
The root cause is here:
curl -s localhost:7777/mon/ps
Summary
[4748700888146592173] storj.io/storj/pkg/process.root() (elapsed: 64h45m31.500167748s)
[3383465883315009742] storj.io/storj/storagenode.(*Peer).Run() (elapsed: 64h45m30.83699121s)
[5538018895755032875] storj.io/storj/pkg/server.(*Server).Run() (elapsed: 64h45m30.253718604s)
[6341958398299244350] storj.io/storj/private/version/checker.(*Service).Run() (elapsed: 64h45m30.254758287s)
[2974185384575018617] storj.io/storj/storagenode/bandwidth.(*Service).Run() (elapsed: 64h45m30.254904334s)
[8511132905675038257] storj.io/storj/storagenode/collector.(*Service).Run() (elapsed: 64h45m30.255268631s)
[1851594380000276706] storj.io/storj/storagenode/console/consoleserver.(*Server).Run() (elapsed: 64h45m30.255013119s)
[6827246398812925390] storj.io/storj/storagenode/contact.(*Chore).Run() (elapsed: 64h45m30.255323608s)
[8996420906188719297] storj.io/storj/storagenode/gracefulexit.(*Chore).Run() (elapsed: 64h45m30.255741533s)
[7631185901357136866] storj.io/storj/storagenode/monitor.(*Service).Run() (elapsed: 64h45m30.255545066s)
[8025844905161357217] storj.io/storj/storagenode/orders.(*Service).Run() (elapsed: 64h45m30.254762753s)
[2018230878483427311] storj.io/storj/storagenode/pieces.(*CacheService).Run() (elapsed: 64h45m30.255719727s)
[246998074698163298] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 3h47m14.018383938s)
[808293576985534254] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 3h47m14.018355729s)
[521785091921381413] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 3h46m21.516212221s)
[1083080594208752369] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 3h46m21.516182557s)
[669062436094953411] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 8.431945062s)
[1230357938382324367] storj.io/storj/storagenode/piecestore.(*Endpoint).doDownload() (elapsed: 8.431926097s)
[764429092178221933] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 3h46m24.876848042s)
[1325724594465592889] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 3h46m24.876820492s)
[797416985003956236] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 34m55.070350964s)
[1358712487291327192] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 34m55.070308421s)
[1516239066543628054] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 2h33m46.955726435s)
[2077534568830999010] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 2h33m46.95570099s)
[1546741410592644934] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 43m54.13358261s)
[2108036912880015889] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 43m54.133561632s)
[1551909467507690850] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 1.098478514s)
[2113204969795061806] storj.io/storj/storagenode/piecestore.(*Endpoint).doDownload() (elapsed: 1.098471378s)
[1610395447250773944] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 5.958503942s)
[2171690949538144900] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 5.958474786s)
[1616501220280310887] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 1h36m36.016138719s)
[2177796722567681843] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 1h36m36.016103213s)
[2098583474859291032] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 381.584564ms)
[2659878977146661988] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 381.565813ms)
[2185250049658051410] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 3h48m34.410759946s)
[2746545551945422366] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 3h48m34.410729477s)
[2188150076752887458] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 3h47m24.477807525s)
[2749445579040258414] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 3h47m24.477786305s)
[2362076102543029802] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 1h37m43.087708141s)
[2923371604830400758] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 1h37m43.087733813s)
[5408176227680298518] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 1h36m37.65029085s)
[5969471729967669474] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 1h36m37.65025902s)
[5495599478455058312] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 2.369567548s)
[6056894980742429268] storj.io/storj/storagenode/piecestore.(*Endpoint).doDownload() (elapsed: 2.369559602s)
[5527115138518586754] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 2h33m24.691704814s)
[6088410640805957710] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 2h33m24.691677556s)
[5676857471873979690] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 4.297534485s)
[6238152974161350646] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 4.297499465s)
[5880496006158906256] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 3h49m34.05182498s)
[6441791508446277212] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 3h49m34.051802584s)
[5883831071166757414] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 2h33m50.522255303s)
[6445126573454128370] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 2h33m50.522213653s)
[6680124927630765279] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 1h39m56.483976651s)
[7241420429918136235] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 1h39m56.483944341s)
[6783764586697086565] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 3h47m43.18890803s)
[7345060088984457520] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 3h47m43.188877827s)
[7332694023885559859] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 2h46m38.3172274s)
[7893989526172930815] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 2h46m38.317202863s)
[7565348748341966364] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 36m30.787625737s)
[8126644250629337320] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 36m30.787607502s)
[7666611618109824004] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 3h46m27.945566787s)
[8227907120397194959] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 3h46m27.945523072s)
[7708011113735315386] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 2h33m39.647650268s)
[8269306616022686342] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 2h33m39.647630174s)
[7824983073221481574] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 2h33m53.078449001s)
[8386278575508852530] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 2h33m53.078429431s)
[7997829463840645873] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 35m11.243784504s)
[8559124966128016828] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 35m11.243757674s)
[8735106143153638512] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 1h37m30.844820519s)
[73029608586233660] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 1h37m30.844787622s)
[8742438110782417546] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 3h46m58.123137653s)
[80361576215012694] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 3h46m58.123095007s)
[9082067559312996495] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 2h33m59.486956651s)
[419991024745591642] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 2h33m59.486928347s)
pktstat -i eth0 -tPF
docker exec -it storagenode netstat -anp |wc -l
34
docker exec -it storagenode netstat -ap
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 localhost:7778 0.0.0.0:* LISTEN 1/storagenode
tcp 0 0 :::7777 :::* LISTEN 1/storagenode
tcp 0 0 :::28967 :::* LISTEN 1/storagenode
tcp 0 0 :::14002 :::* LISTEN 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:59424 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:52290 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 103.235.67.34.bc.googleusercontent.com:42404 SYN_RECV -
tcp 0 0 41072aba9c23:28967 gitbackup.org:37722 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:47304 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:38850 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:50658 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:55066 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:47386 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 01.worker.gitbackup.org:49602 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 103.235.67.34.bc.googleusercontent.com:40594 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:47866 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:39742 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:49282 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:56470 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:45356 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:48672 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:34022 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:48788 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:41850 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:58918 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:60074 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:48616 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:37262 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:50408 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:36862 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:33008 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:38144 ESTABLISHED 1/storagenode
Now I can say: The gitbackup is not closing connections and use with very low speed (~200bps-400bps) for keep-alive connections!
I do see that though everything else is closing pretty quickly.
None of these really effect my connection its less then 56k modem speeds its very low compared to everything else.
I’m more concerned if there’s a memory leak an lots of I/O and cpu useage as this is a dedicated storj node.
As you can see there is no memory or CPU issues
My concern is another: If an application for storj network will open connections and keeping it alive by days(I saw ~100 connections from gitbackup), all free sockets on storage node side is out soon and storage nodes will drop any TCP connections (not only for storagenode).
I agree that is a bit concerning, Im not even sure as to why it would allow it to hit 100 connections and not disconnect them. Is there a set TTL for these connections or are they just open forever and never close?