Continuing analyzing C-States.
I've built Intel PCM for FreeBSD in the jail
git clone --recursive https://github.com/intel/pcm
mkdir -p pcm/build
cd pcm/build
# Hackery to unfuck cmake in my old crusty temporrary jail, you may not need to do this
ln -s /usr/local/lib/libjsoncpp.so.26 /usr/local/lib/libjsoncpp.so.25
# prepare and build
cmake ..
cmake --build . --parallel
# Copy dependenceis reported by `ldd ./pcm` to the same folder to package for running on the host:
cp /usr/lib/libexecinfo.so.1 .
cp /usr/lib/libc++.so.1 .
cp /lib/libcxxrt.so.1 .
cp /lib/libm.so.5 .
cp /lib/libgcc_s.so.1 .
cp /lib/libthr.so.3 .
cp /lib/libc.so.7 .
cp /lib/libelf.so.2 .
# copy the bin folder the host, and run pointing dynamic linker to the current folder:
LD_LIBRARY_PATH=. ./pcm
to look at what’s actualy going on here:
Core C-state residencies: C0 (active,non-halted): 16.21 %; C1: 31.32 %; C3: 0.00 %; C6: 52.47 %; C7: 0.00 %;
Package C-state residencies: C0: 78.28 %; C2: 19.32 %; C3: 0.00 %; C6: 2.40 %; C7: 0.00 %;
┌────────────────────────────────────────────────────────────────────────────────┐
Core C-state distribution│00000000000001111111111111111111111111666666666666666666666666666666666666666666│
└────────────────────────────────────────────────────────────────────────────────┘
┌────────────────────────────────────────────────────────────────────────────────┐
Package C-state distribution│00000000000000000000000000000000000000000000000000000000000000022222222222222266│
└────────────────────────────────────────────────────────────────────────────────┘
---------------------------------------------------------------------------------------------------------------
Intel(r) QPI traffic estimation in bytes (data and non-data traffic outgoing from CPU/socket through QPI links):
QPI0 QPI1 | QPI0 QPI1
---------------------------------------------------------------------------------------------------------------
SKT 0 112 M 113 M | 0% 0%
SKT 1 120 M 116 M | 0% 0%
---------------------------------------------------------------------------------------------------------------
Total QPI outgoing data and non-data traffic: 462 M
So… CPU cores spend in states lower than C0 over 75% of time, and yet the package is only 18% of time gets to go to sleep. Perhaps QPI contributes to this awakenes.
Suspicion confirmed!
Next step – Yank out second CPU. But before that I need to find out how to put four devices (NIC, HBA, and two SSDs) into three remaining PCI slots… 
LOL. After reading readme, I realized I could just pkg install intel-pcm… 
An interestign line from the pcm:
Package thermal spec power: 135 Watt; Package minimum power: 43 Watt; Package maximum power: 270 Watt;
So, yanking out one CPU I shall save at least 40W! Nice!