When there is a current compaction in progress, is there a way to check on what log file it is currently working on?
Or is it even limited to a specific log file or does it mean pieces of multiple log files get rewritten into a new log file? I am confused.
When compaction start it writes to log campaction start or something like this.
When it end, then compaction complete, or failed.
Just open log and search word compaction, you will find all needed.
but it write all this if you have log level Info
I can find a lot of information in there but it is unclear what exactly that is and if it contains the inforamtion I am looking for.
What are e.g. nset, nexist, Today, NumSet and so many more, just as example.
I mean I can see that compaction has been started, but I cannot see what it is actually doing and I would like to know/see, which log file gets currently processed by compaction taks.
You can usually infer it by checking what open files/directories the storagenode process is using.
i.e. for a process number $PROC run ls -l /proc/$PROC/fd 2> /dev/null and progressively use grep to filter out the noise.
There are also metrics, related to it.
I did not see any metric that showed the filename or path of the log file that is currently getting processed. Which metric is supposed to show that?
Not live. I belive at the beginning it does print out which log IDs it wants to to compact. You could map that out to the file names but the conversion is a bit tricky.
As far as I know it goes through that list and basically copies all the allive content of the log file into a new one. Since the new one will not be full it picks the next log file from the list and continues to copy it over. Like a customer upload. When the new log file is full it creates another one. 5 log files with 20% dead space in it would get compacted down to 4 new log files.
I see. So when I see something like "candidates": [6574, 6076, 3902, 5797, 1272, 6328, 2069, 751, 895, 6426, 5289, 4781, 901, 4876, 3760, 4942, 1688, 5792, 4659, 6662, 1881, 5879, 2122, 5694, 6545, 4386, 5412, 8249, 5404, 5082, 8074, 8579, 8635], "rewrite": [5404, 5412, 6574, 8074, 8249, 8579, 8635] then these are the IDs of the log files and the order how they will be processed?
^ this is the list that will get compatced. The other list is the total amount of candidates and there is some randomness that will pick them more likely the more dead space they have. The bigger the list the more logs will get selected but also it doesn’t spike from no compacts to all compacts all the time. It balances the amount that needs to be compacted between the runs.
I can see these are members of the candidates group. That makes sense.
Does this help in identifying the log files:
"store": "s0", "nset": 7315032, "nexist": 7319170
I don’t know what nset dns nexist is but "store": "s0" seemst to indicate the subfolder at least. Can stbb tell what ID is what file?
It should. Try stbb hashstore logs <storage>/hashstore/<satelliteID>/s0/meta
Hmmm. Almost I think. But I don’t know if I like what I am seeing. I pointed it to the hashtbl-0000000 file in there:
stbb hashstore logs /storagenode/satid/s0/meta/hashtabl-000001
The result:
WARNING: log file 5292 is not found (out of 0 log files)
WARNING: log file 1821 is not found (out of 0 log files)
WARNING: log file 4082 is not found (out of 0 log files)
WARNING: log file 4435 is not found (out of 0 log files)
WARNING: log file 4825 is not found (out of 0 log files)
WARNING: log file 6487 is not found (out of 0 log files)
WARNING: log file 6385 is not found (out of 0 log files)
WARNING: log file 4467 is not found (out of 0 log files)
WARNING: log file 4145 is not found (out of 0 log files)
WARNING: log file 3735 is not found (out of 0 log files)
WARNING: log file 8505 is not found (out of 0 log files)
WARNING: log file 6499 is not found (out of 0 log files)
WARNING: log file 6002 is not found (out of 0 log files)
WARNING: log file 5425 is not found (out of 0 log files)
WARNING: log file 6374 is not found (out of 0 log files)
WARNING: log file 5278 is not found (out of 0 log files)
WARNING: log file 6201 is not found (out of 0 log files)
WARNING: log file 6586 is not found (out of 0 log files)
WARNING: log file 6249 is not found (out of 0 log files)
WARNING: log file 7997 is not found (out of 0 log files)
WARNING: log file 8286 is not found (out of 0 log files)
Either I made a mistake with that or something is broken? Or do I have to put the log ID somewhere?
It should end with just /meta and should print out all the log files. So you would have to find the one you are looking for in that list.
I see.
Then I try stbb hashstore logs /storagenode/hashstore/satid/s0/meta and get
hashstore: unable to read header: read /storagenode/hashstore/satid/s0/meta: is a directory
github.com/elek/stbb/pkg/hashstore.(*Logs).Run:35
reflect.Value.call:586
reflect.Value.Call:369
github.com/alecthomas/kong.callMethod:95
github.com/alecthomas/kong.(*Context).RunNode:755
github.com/alecthomas/kong.(*Context).Run:780
main.main:224
runtime.main:290
github.com/elek/stbb/pkg/hashstore.(*Logs).Run
/home/elek/p/stbb/pkg/hashstore/log.go:37
reflect.Value.call
/usr/lib/go/src/reflect/value.go:586
reflect.Value.Call
/usr/lib/go/src/reflect/value.go:369
github.com/alecthomas/kong.callMethod
/home/elek/go/pkg/mod/github.com/alecthomas/kong@v0.7.1/callbacks.go:95
github.com/alecthomas/kong.(*Context).RunNode
/home/elek/go/pkg/mod/github.com/alecthomas/kong@v0.7.1/context.go:755
github.com/alecthomas/kong.(*Context).Run
/home/elek/go/pkg/mod/github.com/alecthomas/kong@v0.7.1/context.go:780
main.main
/home/elek/p/stbb/main.go:224
runtime.main
/usr/lib/go/src/runtime/proc.go:290
runtime.goexit
/usr/lib/go/src/runtime/asm_amd64.s:1771