LVMcache is a permanent cache, reboots won’t affect it. Though, indeed you point out an important thing—prewarming should be part of your testing procedure as well, in all cases.
When I was doing my benchmarks, I repeated each test sequence 11 times, and dropped the first result. If we can assume that the test sequence reflects regular node operations, then after one sequence we should be able to assume we reach the state of cache that would be there during regular node operations.
On the Storj benchmarking tool: