Live connections doesn't closed for hours

Than You i will try.

I think I know what a root cause of this issue, I suspect some application that used Storj network (at this moment I not have proof and screen but I think we will have it very soon)
This connections is not idle, it active but with meager traffic (about 200bps), how to hunting for it:

  1. Debug port should be enabled
  2. Check active connections with curl -s localhost:7777/mon/ps|grep -c "live-request"
  3. Check real-time traffic and connections with pktstat -i eth0 -tPF

You will see a lot of little requests from this application to your storage node.

Im not seeing any of the issues with live requests or memory issues and Im running on a Dell optiplex Dual core CPU only 4gigs of ram. Running 2 nodes fine no issues at all.image

1 Like

Sorry, but you screen show nothing about network stack and current limits.
I can see only problem with disk system (iowait).

The root cause is here:

curl -s localhost:7777/mon/ps

Summary
[4748700888146592173] storj.io/storj/pkg/process.root() (elapsed: 64h45m31.500167748s)
 [3383465883315009742] storj.io/storj/storagenode.(*Peer).Run() (elapsed: 64h45m30.83699121s)
  [5538018895755032875] storj.io/storj/pkg/server.(*Server).Run() (elapsed: 64h45m30.253718604s)
  [6341958398299244350] storj.io/storj/private/version/checker.(*Service).Run() (elapsed: 64h45m30.254758287s)
  [2974185384575018617] storj.io/storj/storagenode/bandwidth.(*Service).Run() (elapsed: 64h45m30.254904334s)
  [8511132905675038257] storj.io/storj/storagenode/collector.(*Service).Run() (elapsed: 64h45m30.255268631s)
  [1851594380000276706] storj.io/storj/storagenode/console/consoleserver.(*Server).Run() (elapsed: 64h45m30.255013119s)
  [6827246398812925390] storj.io/storj/storagenode/contact.(*Chore).Run() (elapsed: 64h45m30.255323608s)
  [8996420906188719297] storj.io/storj/storagenode/gracefulexit.(*Chore).Run() (elapsed: 64h45m30.255741533s)
  [7631185901357136866] storj.io/storj/storagenode/monitor.(*Service).Run() (elapsed: 64h45m30.255545066s)
  [8025844905161357217] storj.io/storj/storagenode/orders.(*Service).Run() (elapsed: 64h45m30.254762753s)
  [2018230878483427311] storj.io/storj/storagenode/pieces.(*CacheService).Run() (elapsed: 64h45m30.255719727s)

[246998074698163298] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 3h47m14.018383938s)
 [808293576985534254] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 3h47m14.018355729s)

[521785091921381413] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 3h46m21.516212221s)
 [1083080594208752369] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 3h46m21.516182557s)

[669062436094953411] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 8.431945062s)
 [1230357938382324367] storj.io/storj/storagenode/piecestore.(*Endpoint).doDownload() (elapsed: 8.431926097s)

[764429092178221933] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 3h46m24.876848042s)
 [1325724594465592889] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 3h46m24.876820492s)

[797416985003956236] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 34m55.070350964s)
 [1358712487291327192] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 34m55.070308421s)

[1516239066543628054] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 2h33m46.955726435s)
 [2077534568830999010] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 2h33m46.95570099s)

[1546741410592644934] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 43m54.13358261s)
 [2108036912880015889] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 43m54.133561632s)

[1551909467507690850] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 1.098478514s)
 [2113204969795061806] storj.io/storj/storagenode/piecestore.(*Endpoint).doDownload() (elapsed: 1.098471378s)

[1610395447250773944] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 5.958503942s)
 [2171690949538144900] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 5.958474786s)

[1616501220280310887] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 1h36m36.016138719s)
 [2177796722567681843] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 1h36m36.016103213s)

[2098583474859291032] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 381.584564ms)
 [2659878977146661988] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 381.565813ms)

[2185250049658051410] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 3h48m34.410759946s)
 [2746545551945422366] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 3h48m34.410729477s)

[2188150076752887458] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 3h47m24.477807525s)
 [2749445579040258414] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 3h47m24.477786305s)

[2362076102543029802] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 1h37m43.087708141s)
 [2923371604830400758] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 1h37m43.087733813s)

[5408176227680298518] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 1h36m37.65029085s)
 [5969471729967669474] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 1h36m37.65025902s)

[5495599478455058312] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 2.369567548s)
 [6056894980742429268] storj.io/storj/storagenode/piecestore.(*Endpoint).doDownload() (elapsed: 2.369559602s)

[5527115138518586754] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 2h33m24.691704814s)
 [6088410640805957710] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 2h33m24.691677556s)

[5676857471873979690] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 4.297534485s)
 [6238152974161350646] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 4.297499465s)

[5880496006158906256] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 3h49m34.05182498s)
 [6441791508446277212] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 3h49m34.051802584s)

[5883831071166757414] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 2h33m50.522255303s)
 [6445126573454128370] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 2h33m50.522213653s)

[6680124927630765279] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 1h39m56.483976651s)
 [7241420429918136235] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 1h39m56.483944341s)

[6783764586697086565] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 3h47m43.18890803s)
 [7345060088984457520] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 3h47m43.188877827s)

[7332694023885559859] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 2h46m38.3172274s)
 [7893989526172930815] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 2h46m38.317202863s)

[7565348748341966364] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 36m30.787625737s)
 [8126644250629337320] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 36m30.787607502s)

[7666611618109824004] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 3h46m27.945566787s)
 [8227907120397194959] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 3h46m27.945523072s)

[7708011113735315386] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 2h33m39.647650268s)
 [8269306616022686342] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 2h33m39.647630174s)

[7824983073221481574] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 2h33m53.078449001s)
 [8386278575508852530] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 2h33m53.078429431s)

[7997829463840645873] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 35m11.243784504s)
 [8559124966128016828] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 35m11.243757674s)

[8735106143153638512] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 1h37m30.844820519s)
 [73029608586233660] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 1h37m30.844787622s)

[8742438110782417546] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 3h46m58.123137653s)
 [80361576215012694] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 3h46m58.123095007s)

[9082067559312996495] storj.io/storj/storagenode/piecestore.live-request() (elapsed: 2h33m59.486956651s)
 [419991024745591642] storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload() (elapsed: 2h33m59.486928347s)

pktstat -i eth0 -tPF

docker exec -it storagenode netstat -anp |wc -l
34

docker exec -it storagenode netstat -ap

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 localhost:7778          0.0.0.0:*               LISTEN      1/storagenode
tcp        0      0 :::7777                 :::*                    LISTEN      1/storagenode
tcp        0      0 :::28967                :::*                    LISTEN      1/storagenode
tcp        0      0 :::14002                :::*                    LISTEN      1/storagenode
tcp        0      0 41072aba9c23:28967      gitbackup.org:59424     ESTABLISHED 1/storagenode
tcp        0      0 41072aba9c23:28967      gitbackup.org:52290     ESTABLISHED 1/storagenode
tcp        0      0 41072aba9c23:28967      103.235.67.34.bc.googleusercontent.com:42404 SYN_RECV    -
tcp        0      0 41072aba9c23:28967      gitbackup.org:37722     ESTABLISHED 1/storagenode
tcp        0      0 41072aba9c23:28967      gitbackup.org:47304     ESTABLISHED 1/storagenode
tcp        0      0 41072aba9c23:28967      gitbackup.org:38850     ESTABLISHED 1/storagenode
tcp        0      0 41072aba9c23:28967      gitbackup.org:50658     ESTABLISHED 1/storagenode
tcp        0      0 41072aba9c23:28967      gitbackup.org:55066     ESTABLISHED 1/storagenode
tcp        0      0 41072aba9c23:28967      gitbackup.org:47386     ESTABLISHED 1/storagenode
tcp        0      0 41072aba9c23:28967      01.worker.gitbackup.org:49602 ESTABLISHED 1/storagenode
tcp        0      0 41072aba9c23:28967      103.235.67.34.bc.googleusercontent.com:40594 ESTABLISHED 1/storagenode
tcp        0      0 41072aba9c23:28967      gitbackup.org:47866     ESTABLISHED 1/storagenode
tcp        0      0 41072aba9c23:28967      gitbackup.org:39742     ESTABLISHED 1/storagenode
tcp        0      0 41072aba9c23:28967      gitbackup.org:49282     ESTABLISHED 1/storagenode
tcp        0      0 41072aba9c23:28967      gitbackup.org:56470     ESTABLISHED 1/storagenode
tcp        0      0 41072aba9c23:28967      gitbackup.org:45356     ESTABLISHED 1/storagenode
tcp        0      0 41072aba9c23:28967      gitbackup.org:48672     ESTABLISHED 1/storagenode
tcp        0      0 41072aba9c23:28967      gitbackup.org:34022     ESTABLISHED 1/storagenode
tcp        0      0 41072aba9c23:28967      gitbackup.org:48788     ESTABLISHED 1/storagenode
tcp        0      0 41072aba9c23:28967      gitbackup.org:41850     ESTABLISHED 1/storagenode
tcp        0      0 41072aba9c23:28967      gitbackup.org:58918     ESTABLISHED 1/storagenode
tcp        0      0 41072aba9c23:28967      gitbackup.org:60074     ESTABLISHED 1/storagenode
tcp        0      0 41072aba9c23:28967      gitbackup.org:48616     ESTABLISHED 1/storagenode
tcp        0      0 41072aba9c23:28967      gitbackup.org:37262     ESTABLISHED 1/storagenode
tcp        0      0 41072aba9c23:28967      gitbackup.org:50408     ESTABLISHED 1/storagenode
tcp        0      0 41072aba9c23:28967      gitbackup.org:36862     ESTABLISHED 1/storagenode
tcp        0      0 41072aba9c23:28967      gitbackup.org:33008     ESTABLISHED 1/storagenode
tcp        0      0 41072aba9c23:28967      gitbackup.org:38144     ESTABLISHED 1/storagenode

Now I can say: The gitbackup is not closing connections and use with very low speed (~200bps-400bps) for keep-alive connections!

1 Like

I do see that though everything else is closing pretty quickly.
None of these really effect my connection its less then 56k modem speeds its very low compared to everything else.
I’m more concerned if there’s a memory leak an lots of I/O and cpu useage as this is a dedicated storj node.


As you can see there is no memory or CPU issues

My concern is another: If an application for storj network will open connections and keeping it alive by days(I saw ~100 connections from gitbackup), all free sockets on storage node side is out soon and storage nodes will drop any TCP connections (not only for storagenode).

I agree that is a bit concerning, Im not even sure as to why it would allow it to hit 100 connections and not disconnect them. Is there a set TTL for these connections or are they just open forever and never close?