Guide to debug my storage node, uplink, s3 gateway, satellite

We have integrated a package called monkit in all storage nodes. It is very usefull for debuging. By default it will open a random port.

In order to connect to the debug port you will need to:

  1. Add debug.addr: ":5999" to the config file. Now the port is fixed and will not change on every restart.
  2. Add -p 127.0.0.1:5999:5999 to the docker run command.
  3. Ready for action!

You can request the following informations:

  1. curl localhost:5999/mon/funcs
    Find out how long a function like GetExpired needs to execute.
  2. curl localhost:5999/mon/ps
    What is running at the moment and how long is it running.
  3. curl localhost:5999/mon/stats
    Statistics.
  4. curl localhost:5999/debug/pprof/goroutine?debug=2
    Full stacktrace with everything that is running.
  5. curl localhost:5999/mon/trace/svg?regex=Upload > upload.svg
    Wait for the next call of the specific function and create a tracefile.
  6. curl localhost:5999/debug/pprof/heap > heap
    Memory leaks.

With v1.5.2 it is possible to change the log level without having to restart the process.

curl -XPUT 'http://127.0.0.1:5999/logging' --header 'Content-Type: text/plain' --data-raw '{"level":"debug"}'

This is just a quick introduction out of my head. The more advanced calls are missing right now. Feel free to add them.

10 Likes

I tried to research it and I think my head exploded.

1 Like

Anything I can help with?

Bring up pieces of @subwolf’s head together I think

Well I found the entry point in /storagenode/pkg/process/debug.go, it’s trying to navigate through https://godoc.org/gopkg.in/spacemonkeygo/monkit.v2 … just, no.

@Alexey can you please make my post edit able?

With v1.5.2 it is possible to change the log level without having to restart the process.

curl -XPUT 'http://127.0.0.1:5999/logging' --header 'Content-Type: text/plain' --data-raw '{"level":"debug"}'

3 Likes

done.
I made it a wiki.

1 Like

Small followup question, I assume this means it only changes for this session and will revert to what’s in the config file after a reboot? Can you confirm?

Yes that is correct.

1 Like

Is there currently a debugging method for those using the Windows GUI?

Sure, you just have to do the first step of adding debug.addr: ":5999" to the config.yaml.

I just put the URLs localhost:5999/mon/funcs etc in a browser after that, so you don’t need curl.

1 Like