Node Connection Tracker

I noticed that there was no way to track how many connections were going through a node at a given time, so I wrote up a tracker. The GitHub repo is at https://github.com/DatOneLefty/storj_connection_tracker.
Screenshot:

If you don’t want to use the shell script, you are able to just download the app.js file and run docker logs --tail 0 storagenode -f 2>&1 | node app
Make sure you have Node.js installed first.

5 Likes

Very cool, How hard would it be to fork this to a one-time run, and it would dump the values as an echo?

I believe it could be possible by passing the last 100 or so records and itll sort through which ones are incomplete, if you want I could add a second file to the repo for it

Please add information about required node.js version.
I have:

apt install nodejs-legacy
node --version
v4.8.2

And have issue with docker logs --tail 0 storagenode -f 2>&1 | node app

Try another way:

apt install nodejs
nodejs --version
v4.8.2

Issue is the same docker logs --tail 0 storagenode -f 2>&1 | nodejs app

and issue here:

I was able to run it with both v8.10.0 and v12.7.0, I’ll update the readme with instructions/
I used this site for v12 installation instructions https://tecadmin.net/install-latest-nodejs-npm-on-ubuntu/

Hi @lholliger
Thanks for your reply!
I think requiremen is Node.js >= 8.xx version, can I ask you add this information to github?
I also found official manual from node.js

I just install latest 12.7 and everything is working fine, thanks!

1 Like

There is actually a debug monitoring tool in your node for something similar already.

You’d have to do a few things to enable it though.

  1. Add the following lines to config.yaml
# debug port for monitoring node
debug.addr: ":7777"
  1. Stop your node docker stop -t 300 storagenode
  2. Remove the container docker rm storagenode
  3. Run the container with an additional port mapping for port 7777. Add -p 7777:7777 to the run command.
  4. Go to http://localhost:7777/mon/ps in your browser (or use LAN IP if your node is not running on the system you are on)

You’ll see something like this:

[2666028377022387705] storj.io/storj/pkg/process.root() (elapsed: 24h38m58.107722787s)
 [2838746615445066551] storj.io/storj/storagenode.(*Peer).Run() (elapsed: 24h38m56.265203481s)
  [5222309624521264138] storj.io/storj/internal/version.(*Service).Run() (elapsed: 24h38m56.265113515s)
  [6327732009848023509] storj.io/storj/pkg/kademlia.(*Kademlia).Run() (elapsed: 24h38m56.265105606s)
  [386070651578182665] storj.io/storj/pkg/server.(*Server).Run() (elapsed: 24h38m56.264795963s)
  [5602302578761964978] storj.io/storj/storagenode/bandwidth.(*Service).Run() (elapsed: 24h38m56.265153627s)
  [6120457294030001515] storj.io/storj/storagenode/collector.(*Service).Run() (elapsed: 24h38m56.265024828s)
  [3944169000771825922] storj.io/storj/storagenode/console/consoleserver.(*Server).Run() (elapsed: 24h38m56.265193051s)
  [247908890550846969] storj.io/storj/storagenode/monitor.(*Service).Run() (elapsed: 24h38m56.265132335s)
  [6155013771425344664] storj.io/storj/storagenode/orders.(*Sender).Run() (elapsed: 24h38m56.265161123s)
  [7985865587838162566] storj.io/storj/storagenode/vouchers.(*Service).Run() (elapsed: 24h38m56.26510918s)

[800340686918948619] storj.io/storj/storagenode/piecestore.(*Endpoint).Download() (elapsed: 18.047139713s)

[1249125853180151259] storj.io/storj/storagenode/piecestore.(*Endpoint).Download() (elapsed: 3.525829232s)

[1940832738073346335] storj.io/storj/storagenode/piecestore.(*Endpoint).Download() (elapsed: 48.338710674s)

[8676266699571489214] storj.io/storj/storagenode/piecestore.(*Endpoint).Download() (elapsed: 7.432933555s)

[834576421544107270] storj.io/storj/storagenode/piecestore.(*Endpoint).Upload() (elapsed: 1.626534122s)

[1629118807420852099] storj.io/storj/storagenode/piecestore.(*Endpoint).Upload() (elapsed: 3.686138703s)

[4807160053819757616] storj.io/storj/storagenode/piecestore.(*Endpoint).Upload() (elapsed: 127.817797ms)

[7916216642536050634] storj.io/storj/storagenode/piecestore.(*Endpoint).Upload() (elapsed: 4.028173994s)
2 Likes

Note: We have plans to get rid of kademlia. Please keep that in mind while developing tools for it :slight_smile:

1 Like

Looks like this was built on logs rather than kademlia. Is there any more info on what will replace kademlia somewhere?

To explain this I have to switch the topic to satellite trust. What happens if a new satellite would like to join the network. All storage nodes have to add it to the trusted satellite list (Opt-In) vs the satellite is untrusted and will get trusted over time without any user interaction. The user can decide to untrust the satellite (Opt-Out).

Kademlia is needed for the Opt-Out approach. For the Opt-In approach we don’t need any kademlia and we don’t need to replace it.

I hope that explains the situation. If not please let me know and I will try to explain it better.

1 Like

It does actually! So all you would need is a routing table of nodes on the satellite and all other nodes (uplinks/storagenodes) simply tell the satellites directly that they exist and where they can be found. Much simpler than I was expecting. Thanks for the quick response.
I thought the opt-in vs opt-out approach was still being discussed. But I guess that decision has been made then?

1 Like

We are working on the details. One option was Opt-In in combination with subscribing to a trusted satellite list. The owner of this list can add and remove satellites and the storage nodes operator will follow without having to update the config file. I like the idea.

I can’t predict what the final descicion will be. My point is to give you all a fair warning before you spend too much time with kademlia.

1 Like

I always wondered why are you keeping Kademlia in v3 if the nodes do not really talk to each other (the uplink is given a list of nodes by the satellite etc).

I guess there can be two types of lists - a list of trusted satellites for the node (opt-in) and a list of nodes that are willing to try working with any satellite (opt-out).

A new satellite would like to join the network. How does it find all storage nodes? Opt-In = the satellite doesn’t need to care because the storage nodes have to add the satellite. Opt-Out = the storage nodes don’t need to care because the satellite has to find them. They need the kademlia network to do so.

1 Like