Guide to debug my storage node

We have integrated a package called monkit in all storage nodes. It is very usefull for debuging. By default it will open a random port.

In order to connect to the debug port you will need to:

  1. Add debug.addr: ":7777" to the config file. Now the port is fixed and will not change on every restart.
  2. Add -p 127.0.0.1:7777:7777 to the docker run command.
  3. Ready for action!

You can request the following informations:

  1. curl localhost:7777/mon/funcs
    Find out how long a function like GetExpired needs to execute.
  2. curl localhost:7777/mon/ps
    What is running at the moment and how long is it running.
  3. curl localhost:7777/mon/stats
    Statistics.
  4. curl localhost:7777/debug/pprof/goroutine?debug=2
    Full stacktrace with everything that is running.
  5. curl localhost:7777/mon/trace/svg?regex=Upload > upload.svg
    Wait for the next call of the specific function and create a tracefile.
  6. curl localhost:7777/debug/pprof/heap > heap
    Memory leaks.

This is just a quick introduction out of my head. The more advanced calls are missing right now. Feed free to add them.

7 Likes

I tried to research it and I think my head exploded.

1 Like

Anything I can help with?

Bring up pieces of @subwolf’s head together I think

Well I found the entry point in /storagenode/pkg/process/debug.go, it’s trying to navigate through https://godoc.org/gopkg.in/spacemonkeygo/monkit.v2 … just, no.