Script to count time for downloads (uploads) pieces

Pentium100 · November 15, 2021, 12:33pm

Here’s my version:

Pentium100MHz/storjv3-tools/blob/master/piece_times.php

<?php
//usage: cat /path/to/log/file.log | php piece_times.php

function formatDate($timestamp) {
//produces date in a format that Storj v3 node uses in its logs
return sprintf("%s.%dZ", gmdate('Y-m-d\TH:i:s',$timestamp), ($timestamp-floor($timestamp))*1000);
}

$log=array();
exec("cat | sed 's/\x1b\[[0-9;]*m//g' ",$log);
$max=0; $min=PHP_FLOAT_MAX;
foreach ($log as $line) {
        $parts=explode("\t",$line);
        if (isset($parts[4])) {
                $json=json_decode($parts[4],true);
                $action=$parts[3];
                switch ($action) {
                case "download started":
                                if (isset($pieces[$json["Satellite ID"]][$json["Piece ID"]][$json["Action"]])) {
                                        $pieces[$json["Satellite ID"]][$json["Piece ID"]][$json["Action"]]++;

This file has been truncated. show original

Just pipe the log content to it and it will find the fastest and slowest times.

An example:

# cat /storj/node/log/docker.00065.log | grep 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs | grep GET_REPAIR | php piece_times.php 
Slowest: satellite: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, piece: DZSAWMEP4KLTX2CNHAJTFSONAH7XLJNUMYI3AL5AYXAPTZLLJNKA, time: 4.440
Fastest: satellite: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, piece: ZSYW6HKR2WZ7GZI7I3MXU42HH45LNJU6L7COGCSBH2WOJGLI5XHQ, time: 0.000
# cat /storj/node/log/docker.00065.log | grep 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs | grep GET_AUDIT | php piece_times.php       
Slowest: satellite: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, piece: TI2RYSBM3YITTHD65S3MFL35VWS7Z7MK56W6BZC72R6QVTF65QEQ, time: 1.570
Fastest: satellite: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, piece: 2AUWHSLVDL2D5ZCXPOWIQ26UUTBIIMM6IT443PUNJJOHFTEAN2ZQ, time: 0.053

Fastest time for GET_REPAIR is zero because the above mentioned error was immediate.

SGC · November 17, 2021, 11:42am

doesn’t seem to work for me

 cat 2021-11-17-sn0001.log | php piece_times.php
PHP Notice:  Undefined offset: 4 in /bitpool/storjlogs/piece_times.php on line 8
PHP Notice:  Undefined offset: 4 in /bitpool/storjlogs/piece_times.php on line 8

i did however find out why the other one wasn’t showing any errors, because the logging script is breaking for some reason, so i guess that explains why it isn’t showing errors if the logs are incomplete.

i suppose this is why they say not to create ones own logging solutions lol
ffs why can’t stuff just work

Pentium100 · November 17, 2021, 11:53am

Can you send me the log file? I want to check why my script breaks. It works OK for me with my logs.

SGC · November 17, 2021, 12:00pm

sent it over discord had to zip it to be allowed to send it.
should be just a regular log… i just did apt install php in debian

Pentium100 · November 17, 2021, 12:54pm

Your log has debug entries, they do not have the json part with piece ID and so on. I have updated the script.

cat 2021-11-17-sn0001.log | grep GET_AUDIT | php piece_times.php       
Slowest: satellite: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE, piece: DYHHZRSVDDTOSMPPN6BON3F763ERJXHCUUNM2CIIBM5ELA47K3PQ, time: 0.737 start: 2021-11-17T01:59:19.592Z, end: 2021-11-17T01:59:20.329Z
Fastest: satellite: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, piece: IBKDZ6FSTQSDEDWN6MZROSTYQJUBP33E6GWYB3GGBQQ2NM2ATMIA, time: 0.025 start: 2021-11-17T03:48:13.470Z, end: 2021-11-17T03:48:13.496Z

cat 2021-11-17-sn0001.log | grep GET_REPAIR | php piece_times.php      
Slowest: satellite: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE, piece: ZTRJC3Z5IIWZ4TPLEBCLJUZEMM2E4N2WTJ7S57LDPKZK2QIYDGGQ, time: 20.243 start: 2021-11-17T10:52:47.805Z, end: 2021-11-17T10:53:08.48Z
Fastest: satellite: 121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6, piece: EP3UPPFS5YECQJQ4BDZ3WXDSUCINLQCZR233JU4LKVARGPMLF2IQ, time: 0.027 start: 2021-11-17T02:04:34.52Z, end: 2021-11-17T02:04:34.79Z

cat 2021-11-17-sn0001.log | php piece_times.php 
Slowest: satellite: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, piece: 3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ, time: 11544.449 start: 2021-11-17T05:58:33.891Z, end: 2021-11-17T09:10:58.339Z
Fastest: satellite: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, piece: OTA6X3Z565Q3FZ4RPAV3UGYTDKJFOXCMK3MSZKVRYQAJGVKJL5YQ, time: 0.002 start: 2021-11-17T05:36:23.589Z, end: 2021-11-17T05:36:23.592Z

The last one is for a regular download, not audit or repair. It may or may not be true, the script tries to take into account the fact that multiple downloads (or uploads) can happen at the same time.

SGC · November 17, 2021, 1:31pm

Pentium100:

cat 2021-11-17-sn0001.log | php piece_times.php 
Slowest: satellite: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, piece: 3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ, time: 11544.449 start: 2021-11-17T05:58:33.891Z, end: 2021-11-17T09:10:58.339Z
Fastest: satellite: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, piece: OTA6X3Z565Q3FZ4RPAV3UGYTDKJFOXCMK3MSZKVRYQAJGVKJL5YQ, time: 0.002 start: 2021-11-17T05:36:23.589Z, end: 2021-11-17T05:36:23.592Z

that doesn’t seem very representative of reality tho.
both the slowest and the fastest are so extreme the numbers becomes useless
if i check the slowest piece in detail i see its a commonly downloaded piece and thus will most likely be cached, which is inconsistent with the times it gets.

the slowest of 11500 seconds is like 3 hours and 12 minutes

grep 3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ 2021-11-17-sn0001*
2021-11-17T00:13:47.889Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T00:17:29.117Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T00:23:16.927Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T00:23:18.706Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T01:39:00.508Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T01:39:11.496Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T01:39:12.818Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T01:39:20.634Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T01:42:00.748Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T01:44:41.977Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T01:44:42.814Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T01:45:17.034Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T01:45:20.266Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T01:46:41.858Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T01:46:42.953Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T01:46:59.436Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T01:48:07.525Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T01:48:09.107Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T02:01:16.798Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T02:02:15.524Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T02:02:22.090Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T02:02:46.294Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T02:07:38.782Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T02:07:46.271Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T02:12:29.959Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T02:13:38.353Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T02:13:58.011Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T02:14:31.808Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T02:15:40.705Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T02:15:42.059Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T02:15:45.672Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T02:15:46.180Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T02:15:46.403Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T02:15:46.909Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T02:22:48.366Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T02:23:19.299Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T03:03:48.996Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T03:05:54.207Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T03:55:38.586Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T03:56:35.802Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T04:50:55.357Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T05:23:22.161Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T05:23:39.679Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T05:31:31.916Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T05:32:21.661Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T05:58:33.891Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T05:59:54.318Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T09:05:04.600Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T09:10:40.733Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T09:10:58.340Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T09:12:48.636Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T09:16:48.477Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T09:17:10.706Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T09:18:38.604Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T09:19:11.383Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T10:31:35.326Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T10:33:01.826Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T10:56:12.606Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T10:56:51.976Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T11:04:47.547Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T11:04:57.844Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T12:01:25.614Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T12:02:45.633Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}

not saying i got a better way, but that seems more like a customer download speed rather than on my side.

its been downloaded 1563 times this month, so would be cached without a doubt.
and my upload speed is never saturated, not even close… i’m usually at 10% saturation of egress max.
looking at daily max i got a 300mbit spike, but thats about it… and that actually seems to be lan traffic.
grep 3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ 2021-11-*-sn0001.log | wc -l
1563

the only one that looks realistic from my perspective is the audits … which sort of makes sense because it would be done by the satellites… i suppose get repair might be the same… unsure
might also be useful if GET_REPAIR is actually done by the sats also… but not sure the others are worth including

Pentium100 · November 17, 2021, 2:03pm

OK, I have found where I think the bug is:

I wrote a script that simply counts “download started” and “downloaded” lines, keeping a running count of how many transfers for that piece are in parallel, here’s what I got:

2021-11-17T00:13:47.889Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
1
2021-11-17T00:17:29.117Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
0
2021-11-17T00:23:16.927Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
1
2021-11-17T00:23:18.706Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
0
2021-11-17T01:39:00.508Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
1
2021-11-17T01:39:11.496Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2
2021-11-17T01:39:12.818Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
1
2021-11-17T01:39:20.634Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2
2021-11-17T01:42:00.748Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
1
2021-11-17T01:44:41.977Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2
2021-11-17T01:44:42.814Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
1
2021-11-17T01:45:17.034Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
0
2021-11-17T01:45:20.266Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
1
2021-11-17T01:46:41.858Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2
2021-11-17T01:46:42.953Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
1
2021-11-17T01:46:59.436Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
0
2021-11-17T01:48:07.525Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
1
2021-11-17T01:48:09.107Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
0
2021-11-17T02:01:16.798Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
1
2021-11-17T02:02:15.524Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
0
2021-11-17T02:02:22.090Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
1
2021-11-17T02:02:46.294Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
0
2021-11-17T02:07:38.782Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
1
2021-11-17T02:07:46.271Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
0
2021-11-17T02:12:29.959Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
1
2021-11-17T02:13:38.353Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2
2021-11-17T02:13:58.011Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
1
2021-11-17T02:14:31.808Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
0
2021-11-17T02:15:40.705Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
1
2021-11-17T02:15:42.059Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
0
2021-11-17T02:15:45.672Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
1
2021-11-17T02:15:46.180Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2
2021-11-17T02:15:46.403Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
1
2021-11-17T02:15:46.909Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
0
2021-11-17T02:22:48.366Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
-1
2021-11-17T02:23:19.299Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
-2
2021-11-17T03:03:48.996Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
-1
2021-11-17T03:05:54.207Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
-2
2021-11-17T03:55:38.586Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
-1
2021-11-17T03:56:35.802Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
-2
2021-11-17T04:50:55.357Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
-1
2021-11-17T05:23:22.161Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
0
2021-11-17T05:23:39.679Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
-1
2021-11-17T05:31:31.916Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
0
2021-11-17T05:32:21.661Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
-1
2021-11-17T05:58:33.891Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
0
2021-11-17T05:59:54.318Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
-1
2021-11-17T09:05:04.600Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
0
2021-11-17T09:10:40.733Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
1
2021-11-17T09:10:58.340Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
0
2021-11-17T09:12:48.636Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
-1
2021-11-17T09:16:48.477Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
0
2021-11-17T09:17:10.706Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
-1
2021-11-17T09:18:38.604Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
0
2021-11-17T09:19:11.383Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
-1
2021-11-17T10:31:35.326Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
0
2021-11-17T10:33:01.826Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
-1
2021-11-17T10:56:12.606Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
0
2021-11-17T10:56:51.976Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
-1
2021-11-17T11:04:47.547Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
0
2021-11-17T11:04:57.844Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
-1

Either some “download started” lines are missing or (more likely) the downloads started before the beginning of this log file. This is probably tripping my script up. I’ll have to think of something.

SGC · November 17, 2021, 2:09pm

yeah my logs are cut every day, this would ofc not happen for a regular docker log tho i would expect it to be the same with logrotate.

maybe you should make a post dedicated to that script your a making, so people can reference to it easily and so we can troubleshoot / develop it.

i think its a great idea

Pentium100 · November 17, 2021, 2:35pm

It turns out my script ignores the extra “downloaded” lines. I also found the place where something is wrong in your log:

after this:

2021-11-17T02:15:46.403Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T02:15:46.909Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T02:22:48.366Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T02:23:19.299Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}

(where the count goes negative). Let’s assume that at this point there are no downloads in progress.

2021-11-17T03:03:48.996Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T03:05:54.207Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T03:55:38.586Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T03:56:35.802Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}

so far so good.

2021-11-17T04:50:55.357Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T05:23:22.161Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-17T05:23:39.679Z        INFO    piecestore      downloaded      {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}

Two downlaods started, one finished. My script takes the earliest “unifinished” transfer to calculate the time and this looks like the transfer started at 4:50:55 and finished at 5:23:39 with another transfer starting at 05:23:22 (before the first one finished).
There’s one “downloaded” line missing.

If I remove the

2021-11-17T04:50:55.357Z        INFO    piecestore      download started        {"Piece ID": "3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}

line from your long, them my script shows this:

12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S 3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ 221.228 2021-11-17T00:13:47.888Z 2021-11-17T00:17:29.117Z
12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S 3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ 1.779 2021-11-17T00:23:16.927Z 2021-11-17T00:23:18.706Z
12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S 3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ 12.310 2021-11-17T01:39:00.507Z 2021-11-17T01:39:12.818Z
12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S 3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ 169.252 2021-11-17T01:39:11.496Z 2021-11-17T01:42:00.747Z
12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S 3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ 322.180 2021-11-17T01:39:20.634Z 2021-11-17T01:44:42.813Z
12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S 3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ 35.057 2021-11-17T01:44:41.976Z 2021-11-17T01:45:17.033Z
12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S 3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ 82.687 2021-11-17T01:45:20.266Z 2021-11-17T01:46:42.953Z
12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S 3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ 17.578 2021-11-17T01:46:41.858Z 2021-11-17T01:46:59.436Z
12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S 3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ 1.582 2021-11-17T01:48:07.525Z 2021-11-17T01:48:09.107Z
12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S 3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ 58.726 2021-11-17T02:01:16.798Z 2021-11-17T02:02:15.523Z
12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S 3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ 24.204 2021-11-17T02:02:22.089Z 2021-11-17T02:02:46.293Z
12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S 3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ 7.489 2021-11-17T02:07:38.782Z 2021-11-17T02:07:46.270Z
12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S 3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ 88.052 2021-11-17T02:12:29.959Z 2021-11-17T02:13:58.010Z
12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S 3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ 53.455 2021-11-17T02:13:38.352Z 2021-11-17T02:14:31.808Z
12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S 3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ 1.354 2021-11-17T02:15:40.704Z 2021-11-17T02:15:42.059Z
12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S 3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ 0.731 2021-11-17T02:15:45.671Z 2021-11-17T02:15:46.403Z
12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S 3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ 0.729 2021-11-17T02:15:46.180Z 2021-11-17T02:15:46.908Z
12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S 3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ 125.211 2021-11-17T03:03:48.996Z 2021-11-17T03:05:54.207Z
12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S 3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ 57.216 2021-11-17T03:55:38.585Z 2021-11-17T03:56:35.802Z
12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S 3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ 17.518 2021-11-17T05:23:22.161Z 2021-11-17T05:23:39.678Z
12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S 3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ 49.745 2021-11-17T05:31:31.915Z 2021-11-17T05:32:21.661Z
12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S 3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ 80.427 2021-11-17T05:58:33.891Z 2021-11-17T05:59:54.318Z
12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S 3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ 353.740 2021-11-17T09:05:04.599Z 2021-11-17T09:10:58.339Z
12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S 3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ 127.903 2021-11-17T09:10:40.733Z 2021-11-17T09:12:48.635Z
12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S 3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ 22.229 2021-11-17T09:16:48.476Z 2021-11-17T09:17:10.706Z
12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S 3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ 32.779 2021-11-17T09:18:38.604Z 2021-11-17T09:19:11.382Z
12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S 3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ 86.500 2021-11-17T10:31:35.325Z 2021-11-17T10:33:01.825Z
12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S 3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ 39.370 2021-11-17T10:56:12.605Z 2021-11-17T10:56:51.976Z
12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S 3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ 10.297 2021-11-17T11:04:47.546Z 2021-11-17T11:04:57.844Z
Slowest: satellite: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, piece: 3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ, time: 353.740, start: 2021-11-17T09:05:04.599Z, end: 2021-11-17T09:10:58.339Z
Fastest: satellite: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, piece: 3A6JTSG5U77TZ7PZZAQ6WT3RB2OZ2ZBGMHCCXXRZOCAPWO6ICYOQ, time: 0.729, start: 2021-11-17T02:15:46.180Z, end: 2021-11-17T02:15:46.908Z

So yeah, you got a download that started and probably never finished. I should probably add a maximum duration cap in my script, so it produces a warning for such case instead of messing up the calculation.

SGC · November 17, 2021, 3:03pm

often when doing such math one removes the extreme outliers to get a more general real world result.

but yeah this isn’t exactly easy to work with when the log is essentially cut and the downloads of the same piece basically runs parallel over on another.

Pentium100 · November 17, 2021, 3:22pm

Yeah. I wrote this script (well, this is a simplified version of the original script) to have the average times and peak number of concurrent transfers, it operate on a 2 minute portion of the log, so the numbers do not get too big, even in case of something like that.
For checking how long audits and repairs were taking, I just took the original script and simplified it for this use.

Also, I think that the script worked well, even though it reported inflated times. The inflated times signify some sort of a problem, in your case, a download that starts and apparently never ends.

SGC · November 17, 2021, 4:01pm

yeah it seems to work fine, but customer download times might not really reflect useful information for us… i think the audits are most likely a much more useful thing to look at… ofc that forces the question if the audits try to pull the entire piece, because if they don’t that number is also much less useful.

but it does atleast give us a more realistic time since it would be the satellites that are pulling the audits and thus bandwidth issues would be the same for all people running the script rather than a view into their customer download times.

also since audits are much less frequent, makes the script faster to run and since they seems to have short intervals between download and downloaded the chances of them getting split between two different logs is highly unlikely…

the only real downside to using the audits only would be that on new or very new nodes which might get a couple of audits in a day or less they might not be able to get result from the script.

but GET_REPAIR also seems like it outputs useful results.
atleast closer to what seems real world, most likely for the same reason as for the GET_AUDIT they don’t run as often and for as extended periods as customer downloads and thus doesn’t get split across logs or run multiple parallel downloads.

i think you should limit the script to use those two GET_AUDIT and GET_REPAIR, and then you are basically already done… atleast until more feedback rolls in with weird results.

Pentium100 · November 17, 2021, 4:32pm

You can do that by inserting grep before the script (see my examples above). This way you can filter by satellite as well

SGC · November 17, 2021, 4:55pm

good point
still think its a bit advanced for many, maybe a wiki for it on github.
and add a wget line for how to download it from the command line, and tell them to install php

and then recommended using GET_AUDIT

even my longest audit this month was 9 seconds lol
maybe an avg would make it more useful also…, i mean it has to process all the numbers anyways to get the fastest and slowest, getting an avg out also shouldn’t be to much of a hassle.

cat 2021-11-*-sn0001.log | grep GET_AUDIT | php piece_times.php
Slowest: satellite: 12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB, piece: JS3IQXPGAVZE6FLACTRVI3KCRGX2T75S44L5OJDTMGLKUYB2H4UA, time: 9.259, start: 2021-11-09T01:27:49.904Z, end: 2021-11-09T01:27:59.163Z
Fastest: satellite: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, piece: YKM6ODWTUMA5CCKXOXCTXAJWJE2YHRC44U4YHIW43IVZBVALCXHQ, time: 0.025, start: 2021-11-05T01:32:21.055Z, end: 2021-11-05T01:32:21.079Z

might also be nice to have some ranges of whats good and what’s bad… in the wiki, when we start getting more numbers on it.

wget https://raw.githubusercontent.com//Pentium100MHz/storjv3-tools/master/piece_times.php

Pentium100 · November 17, 2021, 7:56pm

The original script had that, I removed it . I can add it back in. I have updated the script to include it. Also, if you uncomment the commented out “printf” statement, you’ll get the durations of every piece.