CLI dashboard for SNOs

[Toyoo]

@s-t-o-r-j-user

@nerdatwork

@s-t-o-r-j-user

Would you also consider implementing comparison of debug counters

It would entail dealing with the log file. This would be a different project altogether.

After reading @Toyoo information above, my initial understanding was that this method does not require access to logs. I admit that until now, I did not take any closer look at this topic. Any additional comments @Toyoo, @Alexey?

It does not require access to logging. It does require enabling debug endpoints though, they are not enabled by default.

[nerdatwork]

@Toyoo

debug endpoints

I don’t see any such data through the debug port. Which command does the trick?

[snorkel]

Maybe are you looking for something like this:
https://forum.storj.io/t/please-enable-tcp-fastopen-on-your-storage-nodes/22886/149?u=snorkel

I don’t have logs enabled (fatal mode), just using the debug port.

[nerdatwork]

 curl localhost:5999/mon/stats
function,name=root,scope=storj.io/private/process current=1.000000
function,name=root,scope=storj.io/private/process highwater=1.000000
function,name=root,scope=storj.io/private/process successes=0.000000
function,name=root,scope=storj.io/private/process errors=0.000000
function,name=root,scope=storj.io/private/process panics=0.000000
function,name=root,scope=storj.io/private/process failures=0.000000
function,name=root,scope=storj.io/private/process total=0.000000
function,name=root,scope=storj.io/private/process delta=0.000000
function_times,kind=success,name=root,scope=storj.io/private/process count=0.000000
function_times,kind=failure,name=root,scope=storj.io/private/process count=0.000000
function,name=(*Client).All,scope=storj.io/storj/private/version/checker current=0.000000
function,name=(*Client).All,scope=storj.io/storj/private/version/checker highwater=1.000000
function,name=(*Client).All,scope=storj.io/storj/private/version/checker successes=265.000000
function,name=(*Client).All,scope=storj.io/storj/private/version/checker errors=0.000000
function,name=(*Client).All,scope=storj.io/storj/private/version/checker panics=0.000000
function,name=(*Client).All,scope=storj.io/storj/private/version/checker failures=0.000000
function,name=(*Client).All,scope=storj.io/storj/private/version/checker total=265.000000
function,name=(*Client).All,scope=storj.io/storj/private/version/checker delta=15.000000
function_times,kind=success,name=(*Client).All,scope=storj.io/storj/private/version/checker count=265.000000
function_times,kind=success,name=(*Client).All,scope=storj.io/storj/private/version/checker sum=224.124067
function_times,kind=success,name=(*Client).All,scope=storj.io/storj/private/version/checker min=0.751037
function_times,kind=success,name=(*Client).All,scope=storj.io/storj/private/version/checker max=1.865896
function_times,kind=success,name=(*Client).All,scope=storj.io/storj/private/version/checker rmin=0.752052
function_times,kind=success,name=(*Client).All,scope=storj.io/storj/private/version/checker ravg=0.864837
function_times,kind=success,name=(*Client).All,scope=storj.io/storj/private/version/checker r10=0.760516
function_times,kind=success,name=(*Client).All,scope=storj.io/storj/private/version/checker r50=0.781230
function_times,kind=success,name=(*Client).All,scope=storj.io/storj/private/version/checker r90=1.056798
function_times,kind=success,name=(*Client).All,scope=storj.io/storj/private/version/checker r99=1.748419
function_times,kind=success,name=(*Client).All,scope=storj.io/storj/private/version/checker rmax=1.865896
function_times,kind=success,name=(*Client).All,scope=storj.io/storj/private/version/checker recent=0.819285
function_times,kind=failure,name=(*Client).All,scope=storj.io/storj/private/version/checker count=0.000000

This is what I see.

[s-t-o-r-j-user]

@nerdatwork

This is what I see.

And what do you think, is there maybe any documentation provided by developers of Storj Inc. on currently discussed subjects?

[Toyoo]

Interesting. I see thousands of lines there.

[s-t-o-r-j-user]

@nerdatwork

This is what I see.

I am confirming my interest, however, due to some reasons I will not be able to take a closer look at this subject probably until Thursday. I checked my notes (did only preliminary reading on this topic) and I found such information (algorithm) provided by @Toyoo, that does look very promising. Hope it helps.

Should I change max-concurrent-requests? >> Note: The implementation has changed. You shouldn't use this setting any longer. Leave it commented out

Can’t upload the script, sorry. But it’s pretty simple. For each line of log: if it’s upload started or download started, increment the counter for the number of current in-flight requests, if it’s uploaded, downloaded, upload cancelled or download cancelled, decrement the above counter. if it’s upload started or download started, memorize the satellite id, piece id and the current value of the above counter, if it’s uploaded or downloaded, retrieve the memorized value of the counter for the g…

[nerdatwork]

@s-t-o-r-j-user

any documentation provided by developers of Storj Inc

Guide to debug my storage node, uplink, s3 gateway, satellite

We have integrated a package called monkit in all storage nodes. It is very usefull for debuging. By default it will open a random port. In order to connect to the debug port you will need to: Add debug.addr: ":5999" to the config file. Now the port is fixed and will not change on every restart. Add -p 127.0.0.1:5999:5999 to the docker run command. Ready for action! You can request the following informations: curl localhost:5999/mon/funcs Find out how long a function like GetExpired need…

[snorkel]

This is just the begining of it, right? Because there should be hundreds of lines.

[nerdatwork]

@snorkel

right?

Nope. Its all I can see after I run the command.

[Toyoo]

This was a different thing.

[littleskunk]

@nerdatwork

This is what I see.

That looks like the storage node updater. They both read the same config file. Make sure to give the updater a different debug port.

[snorkel]

One of my nodes outputs 6072 lines of text from /mon/stats, I save it, open it with Notepad++ and search for the strings I mentioned in the linked post. Form those, I can calculate the success rate and etc. There are many values that I can’t figure out what they represent.
I search only for:

audit_success_count
upload_started_count
upload_cancel_count
upload_failure_count
upload_success_count
download_started_count,action=GET,
download_cancel_count,action=GET,
download_failure_count,action=GET,
download_success_count,action=GET,

Strigs found look like this:

upload_started_count,scope=storj.io/storj/storagenode/piecestore high=696670.000000
upload_started_count,scope=storj.io/storj/storagenode/piecestore low=1.000000
upload_started_count,scope=storj.io/storj/storagenode/piecestore value=696670.000000

upload_cancel_count,scope=storj.io/storj/storagenode/piecestore high=689.000000
upload_cancel_count,scope=storj.io/storj/storagenode/piecestore low=1.000000
upload_cancel_count,scope=storj.io/storj/storagenode/piecestore value=689.000000

upload_failure_count,scope=storj.io/storj/storagenode/piecestore high=7979.000000
upload_failure_count,scope=storj.io/storj/storagenode/piecestore low=1.000000
upload_failure_count,scope=storj.io/storj/storagenode/piecestore value=7979.000000

upload_success_count,scope=storj.io/storj/storagenode/piecestore high=687987.000000
upload_success_count,scope=storj.io/storj/storagenode/piecestore low=1.000000
upload_success_count,scope=storj.io/storj/storagenode/piecestore value=687987.000000

download_started_count,action=GET,scope=storj.io/storj/storagenode/piecestore high=459980.000000
download_started_count,action=GET,scope=storj.io/storj/storagenode/piecestore low=1.000000
download_started_count,action=GET,scope=storj.io/storj/storagenode/piecestore value=459980.000000

download_cancel_count,action=GET,scope=storj.io/storj/storagenode/piecestore high=14204.000000
download_cancel_count,action=GET,scope=storj.io/storj/storagenode/piecestore low=1.000000
download_cancel_count,action=GET,scope=storj.io/storj/storagenode/piecestore value=14204.000000

download_failure_count,action=GET,scope=storj.io/storj/storagenode/piecestore high=3066.000000
download_failure_count,action=GET,scope=storj.io/storj/storagenode/piecestore low=1.000000
download_failure_count,action=GET,scope=storj.io/storj/storagenode/piecestore value=3066.000000

download_success_count,action=GET,scope=storj.io/storj/storagenode/piecestore high=442708.000000
download_success_count,action=GET,scope=storj.io/storj/storagenode/piecestore low=1.000000
download_success_count,action=GET,scope=storj.io/storj/storagenode/piecestore value=442708.000000

[Toyoo]

Out of 696670 attempts to upload a file, 689 were canceled, 7979 failed, and 687987 were successful (from the node’s point of view—not all successful uploads win races). In other words, 689/696670 = 0.09% canceled, 7979/696670 = 1.1% failed, 687987/696670 = 98.8% succeded.

Out of 459980 attempts to download a file, 14204 were canceled, 3066 failed, 442708 were successful.

All counts since the last restart of the node.

Ignore the low and high rows, they’re just artifacts of the code that collects these metrics.

[s-t-o-r-j-user]

Excellent info / explanation @Toyoo and also @littleskunk, @snorkel, @nerdatwork. Many thanks.

I still did not have the opportunity to take a deeper look at this subject, however, I am wondering if you may be willing to confirm:

a) Is my understanding correct that it is not possible to obtain precise information about successful uploads that won races not only from this debug port but also not from storagenode log file?

and

b) Is my understanding correct that those debug stats in terms of success rate are more or less the same as the ones that can be obtained by running storj_success_rate script, I mean more or less the same in a sense that they are exactly the same but the dimensions (like audit, repair etc) are summed?

[Toyoo]

Yes to both. The method of measurement is the same, it’s just that the debug endpoint and logs provide different views to the same information.

[s-t-o-r-j-user]

@nerdatwork, taking into account new info provided above related to success rate and debug port are you maybe considering adding such functionality to your dashboard?

EDIT:
@nerdatwork, I just came with one more question, is it possible to access basic data about account with Etherscan API, it seems that if I got zksync option active I am not able to see the correct balance in the dashboard?

@littleskunk, is there any way to obtain precise information about successful uploads that won races, are you planning maybe such functionality, do you think that such info might be useful when running a node?

[littleskunk]

@s-t-o-r-j-user

@littleskunk, is there any way to obtain precise information about successful uploads that won races, are you planning maybe such functionality, do you think that such info might be useful when running a node?

I am happy with my grafana dashboard.

[s-t-o-r-j-user]

I was not aware that we are communicating though an intermediary. Sorry, I have never been using your grafana dashboard. This CLI approach seems to be much simpler and faster to deploy. My intention behind this question was to obtain information related to the real success rate statistics. And my understanding is that you are the representative of Storj developers, thus the question. Sorry … again.

[littleskunk]

@s-t-o-r-j-user

Sorry, I have never been using your grafana dashboard.

You don’t have to use my grafana dashboard. The point is my grafana dashboard is based on data that the storage node binary exposes itself. If I have a need for some additional graphs (long time ago for success vs failure rate for audit, repair, uploads and downloads) I make some code changes to get it supported by the storage node binary. All I am saying is the data is available for everyone. You can use my grafana dashboard. You can build your own grafana dashboard. You can even write yourself a python script if you don’t want grafana at all. The storage node binary provides all the data you need. If you are missing something we can add that to the binary.