Than You i will try.
I think I know what a root cause of this issue, I suspect some application that used Storj network (at this moment I not have proof and screen but I think we will have it very soon)
This connections is not idle, it active but with meager traffic (about 200bps), how to hunting for it:
- Debug port should be enabled
- Check active connections with
curl -s localhost:7777/mon/ps|grep -c "live-request" - Check real-time traffic and connections with
pktstat -i eth0 -tPF
You will see a lot of little requests from this application to your storage node.
Sorry, but you screen show nothing about network stack and current limits.
I can see only problem with disk system (iowait).
The root cause is here:
curl -s localhost:7777/mon/ps
Summary
[4748700888146592173] storj.io/storj/pkg/process.root() (elapsed: 64h45m31.500167748s)
[3383465883315009742] storj.io/storj/storagenode.(*Peer).Run() (elapsed: 64h45m30.83699121s)
[5538018895755032875] storj.io/storj/pkg/server.(*Server).Run() (elapsed: 64h45m30.253718604s)
[6341958398299244350] storj.io/storj/private/version/checker.(*Service).Run() (elapsed: 64h45m30.254758287s)
[2974185384575018617] storj.io/storj/storagenode/bandwidth.(*Service).Run() (elapsed: 64h45m30.254904334s)
[8511132905675038257] storj.io/storj/storagenode/collector.(*Service).Run() (elapsed: 64h45m30.255268631s)
[1851594380000276706] storj.io/storj/storagenode/console/consoleserver.(*Server).Run() (elapsed: 64h45m30.255013119s)
[6827246398812925390] storj.io/storj/storagenode/contact.(*Chore).Run() (elapsed: 64h45m30.255323608s)
[8996420906188719297] storj.io/storj/storagenode/gracefulexit.(*Chore).Run() (elapsed: 64h45m30.255741533s)
[7631185901357136866] storj.io/storj/storagenode/monitor.(*Service).Run() (elapsed: 64h45m30.255545066s)
[8025844905161357217] storj.io/storj/storagenode/orders.(*Service).Run() (elapsed: 64h45m30.254762753s)
[2018230878483427311] storj.io/storj/storagenode/pieces.(*CacheService).Run() (elapsed: 64h45m30.255719727s)
[246998074698163298] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 3h47m14.018383938s)
[808293576985534254] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 3h47m14.018355729s)
[521785091921381413] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 3h46m21.516212221s)
[1083080594208752369] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 3h46m21.516182557s)
[669062436094953411] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 8.431945062s)
[1230357938382324367] storj.io/storj/storagenode/piecestore.(*Endpoint).doDownload() (elapsed: 8.431926097s)
[764429092178221933] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 3h46m24.876848042s)
[1325724594465592889] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 3h46m24.876820492s)
[797416985003956236] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 34m55.070350964s)
[1358712487291327192] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 34m55.070308421s)
[1516239066543628054] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 2h33m46.955726435s)
[2077534568830999010] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 2h33m46.95570099s)
[1546741410592644934] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 43m54.13358261s)
[2108036912880015889] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 43m54.133561632s)
[1551909467507690850] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 1.098478514s)
[2113204969795061806] storj.io/storj/storagenode/piecestore.(*Endpoint).doDownload() (elapsed: 1.098471378s)
[1610395447250773944] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 5.958503942s)
[2171690949538144900] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 5.958474786s)
[1616501220280310887] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 1h36m36.016138719s)
[2177796722567681843] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 1h36m36.016103213s)
[2098583474859291032] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 381.584564ms)
[2659878977146661988] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 381.565813ms)
[2185250049658051410] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 3h48m34.410759946s)
[2746545551945422366] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 3h48m34.410729477s)
[2188150076752887458] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 3h47m24.477807525s)
[2749445579040258414] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 3h47m24.477786305s)
[2362076102543029802] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 1h37m43.087708141s)
[2923371604830400758] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 1h37m43.087733813s)
[5408176227680298518] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 1h36m37.65029085s)
[5969471729967669474] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 1h36m37.65025902s)
[5495599478455058312] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 2.369567548s)
[6056894980742429268] storj.io/storj/storagenode/piecestore.(*Endpoint).doDownload() (elapsed: 2.369559602s)
[5527115138518586754] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 2h33m24.691704814s)
[6088410640805957710] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 2h33m24.691677556s)
[5676857471873979690] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 4.297534485s)
[6238152974161350646] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 4.297499465s)
[5880496006158906256] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 3h49m34.05182498s)
[6441791508446277212] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 3h49m34.051802584s)
[5883831071166757414] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 2h33m50.522255303s)
[6445126573454128370] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 2h33m50.522213653s)
[6680124927630765279] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 1h39m56.483976651s)
[7241420429918136235] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 1h39m56.483944341s)
[6783764586697086565] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 3h47m43.18890803s)
[7345060088984457520] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 3h47m43.188877827s)
[7332694023885559859] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 2h46m38.3172274s)
[7893989526172930815] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 2h46m38.317202863s)
[7565348748341966364] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 36m30.787625737s)
[8126644250629337320] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 36m30.787607502s)
[7666611618109824004] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 3h46m27.945566787s)
[8227907120397194959] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 3h46m27.945523072s)
[7708011113735315386] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 2h33m39.647650268s)
[8269306616022686342] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 2h33m39.647630174s)
[7824983073221481574] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 2h33m53.078449001s)
[8386278575508852530] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 2h33m53.078429431s)
[7997829463840645873] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 35m11.243784504s)
[8559124966128016828] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 35m11.243757674s)
[8735106143153638512] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 1h37m30.844820519s)
[73029608586233660] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 1h37m30.844787622s)
[8742438110782417546] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 3h46m58.123137653s)
[80361576215012694] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 3h46m58.123095007s)
[9082067559312996495] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 2h33m59.486956651s)
[419991024745591642] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 2h33m59.486928347s)
pktstat -i eth0 -tPF
docker exec -it storagenode netstat -anp |wc -l
34
docker exec -it storagenode netstat -ap
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 localhost:7778 0.0.0.0:* LISTEN 1/storagenode
tcp 0 0 :::7777 :::* LISTEN 1/storagenode
tcp 0 0 :::28967 :::* LISTEN 1/storagenode
tcp 0 0 :::14002 :::* LISTEN 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:59424 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:52290 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 103.235.67.34.bc.googleusercontent.com:42404 SYN_RECV -
tcp 0 0 41072aba9c23:28967 gitbackup.org:37722 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:47304 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:38850 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:50658 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:55066 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:47386 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 01.worker.gitbackup.org:49602 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 103.235.67.34.bc.googleusercontent.com:40594 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:47866 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:39742 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:49282 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:56470 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:45356 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:48672 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:34022 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:48788 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:41850 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:58918 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:60074 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:48616 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:37262 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:50408 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:36862 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:33008 ESTABLISHED 1/storagenode
tcp 0 0 41072aba9c23:28967 gitbackup.org:38144 ESTABLISHED 1/storagenode
Now I can say: The gitbackup is not closing connections and use with very low speed (~200bps-400bps) for keep-alive connections!
As you can see there is no memory or CPU issues
My concern is another: If an application for storj network will open connections and keeping it alive by days(I saw ~100 connections from gitbackup), all free sockets on storage node side is out soon and storage nodes will drop any TCP connections (not only for storagenode).
I agree that is a bit concerning, Im not even sure as to why it would allow it to hit 100 connections and not disconnect them. Is there a set TTL for these connections or are they just open forever and never close?