$ cat /mnt/x/storagenode3/storagenode.log | grep 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE | grep -E "GET_AUDIT" | jq -R '. | split("\t") | (.[4] | fromjson) as $body | {SatelliteID: $body."Satellite ID", ($body."Piece ID"): {(.[0]): .[3]}}' | jq -s 'reduce .[] as $item ({}; . * $item)'
...
"EY6AAGG33LEZK3O7NIM2S3R5VEPXRBIYCKPWYSV5LDWNMHFYF7EA": {
"2021-08-22T22:42:40.601Z": "download started",
"2021-08-22T22:50:28.975Z": "download started",
"2021-08-22T23:04:59.672Z": "download canceled",
"2021-08-22T23:11:19.397Z": "download started",
"2021-08-22T23:15:09.613Z": "downloaded",
"2021-08-22T23:42:01.821Z": "download started",
"2021-08-23T00:04:21.554Z": "downloaded",
"2021-08-23T00:30:52.619Z": "download started",
"2021-08-23T00:35:27.385Z": "downloaded",
"2021-08-23T00:37:23.375Z": "download started",
"2021-08-23T01:32:05.631Z": "download started",
"2021-08-23T01:34:09.934Z": "download started",
"2021-08-23T01:38:56.373Z": "downloaded",
"2021-08-23T01:56:29.461Z": "download started",
"2021-08-23T02:09:35.964Z": "download started",
"2021-08-23T02:10:28.193Z": "download started",
"2021-08-23T02:19:04.072Z": "download started",
"2021-08-23T02:37:11.049Z": "download started",
"2021-08-23T02:40:59.384Z": "downloaded",
"2021-08-23T03:07:47.499Z": "downloaded",
"2021-08-23T03:13:37.528Z": "downloaded",
"2021-08-23T03:25:25.152Z": "downloaded",
"2021-08-23T03:31:02.822Z": "downloaded",
"2021-08-23T03:52:26.973Z": "downloaded",
"2021-08-23T04:01:59.479Z": "downloaded",
"2021-08-23T04:17:16.239Z": "downloaded"
}
In such format it’s now obvious - there was a problem. The interval between “GET_AUDIT” “download started” and “downloaded” should not be greater than 5 minutes for each piece.
And there is problem started:
"LOK6MJKTW7OFFC76SHFV5M7LHGEZSJOG4TQHEO7XYDX2EUOYQ4XA": {
"2021-08-21T20:38:47.156Z": "download started",
"2021-08-21T20:38:49.832Z": "downloaded"
},
"6VLHZ7JZVYDRB3K7ZM55DIRUPOOBSELFX6NXRVTYIALOV4LUL32Q": {
"2021-08-21T20:56:08.230Z": "download started",
"2021-08-21T20:57:07.722Z": "downloaded"
},
"TIEF7N6K7ZQUYMUUHJBM4J6QIB44GPKUTCEAR52TGUYF332YGZTQ": {
"2021-08-21T20:59:45.667Z": "download started",
"2021-08-21T21:04:56.621Z": "downloaded"
},
"A2PLV7FUBDMJDGZP3MY4XYUS36X7OVMSLRXEBZMS2BLCMPBJOEDQ": {
"2021-08-21T21:28:50.623Z": "download started",
"2021-08-21T21:28:54.659Z": "downloaded"
},
Instead of 2 seconds, it gave a piece after 4 minutes. Later longer and longer