Guide to debug my storage node, uplink, s3 gateway, satellite

[littleskunk]

We have integrated a package called monkit in all storage nodes. It is very usefull for debuging. By default it will open a random port.

In order to connect to the debug port you will need to:

1. Add debug.addr: ":5999" to the config file. Now the port is fixed and will not change on every restart.
2. Add -p 127.0.0.1:5999:5999 to the docker run command.
3. Ready for action!

You can request the following informations:

1. curl localhost:5999/mon/funcs
   Find out how long a function like GetExpired needs to execute.
2. curl localhost:5999/mon/ps
   What is running at the moment and how long is it running.
3. curl localhost:5999/mon/stats
   Statistics.
4. curl localhost:5999/debug/pprof/goroutine?debug=2
   Full stacktrace with everything that is running.
5. curl localhost:5999/mon/trace/svg?regex=Upload > upload.svg
   Wait for the next call of the specific function and create a tracefile.
6. curl localhost:5999/debug/pprof/heap > heap
   Memory leaks.

With v1.5.2 it is possible to change the log level without having to restart the process.

curl -XPUT 'http://127.0.0.1:5999/logging' --header 'Content-Type: text/plain' --data-raw '{"level":"debug"}'

This is just a quick introduction out of my head. The more advanced calls are missing right now. Feel free to add them.

[subwolf]

I tried to research it and I think my head exploded.

[littleskunk]

Anything I can help with?

[Alexey]

Bring up pieces of @subwolf’s head together I think

[subwolf]

Well I found the entry point in /storagenode/pkg/process/debug.go, it’s trying to navigate through https://godoc.org/gopkg.in/spacemonkeygo/monkit.v2 … just, no.

[littleskunk]

@Alexey can you please make my post edit able?

With v1.5.2 it is possible to change the log level without having to restart the process.

curl -XPUT 'http://127.0.0.1:5999/logging' --header 'Content-Type: text/plain' --data-raw '{"level":"debug"}'

[Alexey]

done.
I made it a wiki.

[BrightSilence]

Small followup question, I assume this means it only changes for this session and will revert to what’s in the config file after a reboot? Can you confirm?

[littleskunk]

Yes that is correct.

[bramnet]

Is there currently a debugging method for those using the Windows GUI?

[BrightSilence]

Sure, you just have to do the first step of adding debug.addr: ":5999" to the config.yaml.

I just put the URLs localhost:5999/mon/funcs etc in a browser after that, so you don’t need curl.

[Doom4535]

Do any of you know how to set the debug.addr environment variable through the docker run command instead of via the config.yaml? Something along the lines of -e --debug.addr 5999 ?

[Alexey]

Hello @Doom4535,
Welcome to the forum!

The other way around - you can add it after the name of the container, i.e.

docker run ... storjlabs/storagenode:latest --debug.addr=":5999"

[Doom4535]

Awesome, thank you @Alexey

[snorkel]

In order to use this monithoring tool, do I need to set the log.level to debug? Curently is running with log.level error…

[gpan]

Hi
I’ve followed the guide and got the debug port exported (storage node).
It seems to be working fine but some info is missing.
eg From the curl examples you mention, the 1st (/mon/funcs) only shows 2 functions:
storj.io/private/process.root
storj.io/storj/private/version/checker.(*Client).All

The 5th example (/mon/trace/svg with regex=Download) returns:
Bad Request: regex preselect matches 0 functions
adding &preselect=false, it never completes (no reply) even though I see downloads in the meantime (in the log).

Is there something else that needs to be enabled?
The log level is info if that is relevant.

[littleskunk]

Are you running the storage node updater? That might block the debug port first before the storage node gets it. They both share the same config file. My solution to this problem was to specify the debug port in my systemd service with --debug.addr

[gpan]

Yep, that’s what it was.
I added the --debug.addr (instead of the config file) and it’s now working as expected.
Thanks!