I see this now also from time to time on one of my nodes. This issue started with using 1.63 windows version - I never had this before with older versions. All databases are correct (checked). But the file mentioned is really not there - question is why this is happening?
Ah I am using a raid 50 and 60 so plenty of redundancy and error correction - file system etc. is def. OK.
1 Like
Also started from 1.6.3
1 Like
Same here. Rebooted windows and worked normally again.
Again. Hopefully a fix is released soon.
2020-06-12T03:55:34.008+0300 ERROR piecestore:cache error getting current space used calculation: {“error”: “CreateFile h:\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\cp/6j47pubxb6lbrryndy4dbl6xoypfxdgh7ch557im2j26ppmgba.sj1: The system cannot find the file specified.”}
2020-06-12T03:55:34.008+0300 ERROR piecestore:cache error getting current space used calculation: {“error”: “CreateFile h:\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\cp/6j47pubxb6lbrryndy4dbl6xoypfxdgh7ch557im2j26ppmgba.sj1: The system cannot find the file specified.”}
2020-06-12T03:55:34.008+0300 ERROR piecestore:cache error getting current space used calculation: {“error”: “CreateFile h:\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\cp/6j47pubxb6lbrryndy4dbl6xoypfxdgh7ch557im2j26ppmgba.sj1: The system cannot find the file specified.”}
2020-06-12T03:55:34.127+0300 FATAL Unrecoverable error {“error”: “CreateFile h:\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\cp/6j47pubxb6lbrryndy4dbl6xoypfxdgh7ch557im2j26ppmgba.sj1: The system cannot find the file specified.”}
2020-06-12T03:55:34.127+0300 FATAL Unrecoverable error {“error”: “CreateFile h:\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\cp/6j47pubxb6lbrryndy4dbl6xoypfxdgh7ch557im2j26ppmgba.sj1: The system cannot find the file specified.”}
2020-06-11T03:56:12.993+0300 ERROR piecestore:cache error getting current space used calculation: {“error”: “CreateFile h:\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\6q/uyzzwobl3rzewr5fgcih7tsz37lvutpqvha3ikexz7ppyetgea.sj1: The system cannot find the file specified.”}
2020-06-11T03:56:12.993+0300 ERROR piecestore:cache error getting current space used calculation: {“error”: “CreateFile h:\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\6q/uyzzwobl3rzewr5fgcih7tsz37lvutpqvha3ikexz7ppyetgea.sj1: The system cannot find the file specified.”}
2020-06-11T03:56:12.993+0300 ERROR piecestore:cache error getting current space used calculation: {“error”: “CreateFile h:\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\6q/uyzzwobl3rzewr5fgcih7tsz37lvutpqvha3ikexz7ppyetgea.sj1: The system cannot find the file specified.”}
2020-06-11T03:56:13.129+0300 FATAL Unrecoverable error {“error”: “CreateFile h:\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\6q/uyzzwobl3rzewr5fgcih7tsz37lvutpqvha3ikexz7ppyetgea.sj1: The system cannot find the file specified.”}
2020-06-11T03:56:13.129+0300 FATAL Unrecoverable error {“error”: “CreateFile h:\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\6q/uyzzwobl3rzewr5fgcih7tsz37lvutpqvha3ikexz7ppyetgea.sj1: The system cannot find the file specified.”}
2020-06-10T00:03:52.102+0300 ERROR piecestore:cache error getting current space used calculation: {“error”: “CreateFile h:\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\z6/hqtxm3fimqppsyv26kpngd7al74yjhwp4mysdjcmfuc6zhwzaa.sj1: The system cannot find the file specified.”}
2020-06-10T00:03:52.102+0300 ERROR piecestore:cache error getting current space used calculation: {“error”: “CreateFile h:\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\z6/hqtxm3fimqppsyv26kpngd7al74yjhwp4mysdjcmfuc6zhwzaa.sj1: The system cannot find the file specified.”}
2020-06-10T00:03:52.102+0300 ERROR piecestore:cache error getting current space used calculation: {“error”: “CreateFile h:\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\z6/hqtxm3fimqppsyv26kpngd7al74yjhwp4mysdjcmfuc6zhwzaa.sj1: The system cannot find the file specified.”}
2020-06-10T00:03:52.217+0300 FATAL Unrecoverable error {“error”: “CreateFile h:\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\z6/hqtxm3fimqppsyv26kpngd7al74yjhwp4mysdjcmfuc6zhwzaa.sj1: The system cannot find the file specified.”}
2020-06-10T00:03:52.217+0300 FATAL Unrecoverable error {“error”: “CreateFile h:\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\z6/hqtxm3fimqppsyv26kpngd7al74yjhwp4mysdjcmfuc6zhwzaa.sj1: The system cannot find the file specified.”}
for some unknown reason it hapening at same time. @littleskunk can it be investigated?
it started from version 1.6.3
Thank you all for reporting it.
We are stopping the rollout of v1.6.3 and escalate the issue internally to investigate the error.
Thank you for patience.
8 Likes
Great, it crashed again while I was sleeping. 9h of downtime.
does it crash at one specific time all the time?
Not, not any that I can correlate…
Yepp absolutely random. I changed the windows service to restart automatically after 10min as workaround.
1 Like
If only this thing was a service. Perhaps it can be done…
We’re looking into resolving this issue right now.
Feel free to contact me today or Monday if you have any further related issues!
4 Likes
on windows GUI storagenode is service
I have both services restarting automatically, but that won’t solve it. Only after a reboot it recovers. Happened on my 3 Windows nodes, but not on the pi and also not on my 2 Synologys.
Weird, on FreeBSD it just closes, I just have to open it again. Having to reboot is a bigger hassle.
How fast do you restart? try waiting10min.
First try was also immediate restart - but then it just runs into the same issue as you describe. WIth a pause it works on my nodes.
I think the more you pause the less load you will have from satellites, might be related. I’ve had to re-start it twice so far.
Who meet this error - please, enable the debug level for logs (it’s a log.level: option in the "C:\Program Files\Storj\Storage Node\config.yaml") and post one of logs (30-40 strings, when the node is crashed) between three backticks: ``` at the first line and three backticks after the last line.
Do not forget to save the config and restart the storagenode service.
2 Likes
Please, also add the following two options to your configuration file following what @Alexey has commented above.
log.caller: truelog.stack: true
Thanks for your help.
Here you go:
2020-06-11T21:26:54.334+0200 INFO piecestore upload started {"Piece ID": "DGTXPA3Z3PXHMGPPRBJKJQBORRUQQVU6QRBT6C6V2ZQUL7KUQXFA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 194794422358}
2020-06-11T21:26:54.412+0200 INFO piecestore upload started {"Piece ID": "FOY4XHK6O7Z7JXRMBIOF2EMTJYZETS7F4ZTCDDB2PZ3ZX5KJU3VQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 194794422358}
2020-06-11T21:26:54.965+0200 INFO piecestore upload canceled {"Piece ID": "H43C6MAFJM3EOPRCTXCICURWZVBZ7L7LKPJSQ3TOX3RT6TJDBUGA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2020-06-11T21:26:55.389+0200 INFO piecedeleter deleted {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Piece ID": "IAW7UHTUCNBIRSFCRXNXDTPRHQEVHB3QB6IYGUYYIL353WLUYNEA"}
2020-06-11T21:26:55.458+0200 INFO piecedeleter deleted {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Piece ID": "UYW6J6TRTOD4DPMNYNDHT7NJU3G5IWQROVBYYHVCU5TPLJ5KOGQA"}
2020-06-11T21:26:55.544+0200 INFO piecedeleter deleted {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Piece ID": "OBOXK2HHZW3HX7QS536BTPZMQFOBG66EAQGWP6BCTWRXCBDPY4WA"}
2020-06-11T21:26:55.586+0200 INFO piecedeleter deleted {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Piece ID": "XBR6YLHXENRHSHLNKSMIHZVYYHJFM33ZU3TNBIKSCWTZONP3223A"}
2020-06-11T21:26:55.671+0200 INFO piecedeleter deleted {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Piece ID": "6FB6C3BWVSPCX2TKYH6K3QJNJTQCLFOENFTTQJEZRQTI36Y6NJSA"}
2020-06-11T21:26:55.843+0200 INFO piecestore upload started {"Piece ID": "F3NV6F2V7FQDQ556RZS4JSZAW34ABRRVZYKCG2MRBOPCTXKMBUNA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 194794435158}
2020-06-11T21:26:55.968+0200 INFO piecedeleter deleted {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Piece ID": "ARMIGRQD5F2GWH6RILBCYCNQV7XFKBUUBZEWQDGE6KUKEYAW6QZQ"}
2020-06-11T21:26:56.040+0200 INFO piecestore upload started {"Piece ID": "3EGUE6NCDTAO42YDHCHI6NKNEABDNTANJ4KFJWXUJEE2R77HRDCA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 194794437718}
2020-06-11T21:26:56.042+0200 INFO piecedeleter deleted {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Piece ID": "WDVRR7GK5EYSHT76E7FP4AE26GR5XYJ4WIUFFFM54VEU3K2N3YSA"}
2020-06-11T21:26:56.226+0200 INFO piecestore upload canceled {"Piece ID": "FOY4XHK6O7Z7JXRMBIOF2EMTJYZETS7F4ZTCDDB2PZ3ZX5KJU3VQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-06-11T21:26:56.490+0200 INFO piecestore upload started {"Piece ID": "6XH6UIKCHZYJ22FGX6WUVYQIID7R6CYIHCS363NBZ4M4RQPPFWNA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 194789800534}
2020-06-11T21:26:56.508+0200 INFO piecestore upload canceled {"Piece ID": "DGTXPA3Z3PXHMGPPRBJKJQBORRUQQVU6QRBT6C6V2ZQUL7KUQXFA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-06-11T21:26:56.573+0200 INFO piecestore download started {"Piece ID": "2LKFDZFRJMATGMEFDEWLVOWOE7RKLYDTP55INFCH7DCRHC7VUIIQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2020-06-11T21:26:56.628+0200 INFO piecestore upload started {"Piece ID": "SQ37KY426QS6RTNHJUTOX5DTPHPJXAMZ6K6COT2MCAZIBL7VYA4Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 194789800534}
2020-06-11T21:26:56.709+0200 INFO piecestore upload started {"Piece ID": "RXV56EZTP3TDTPEO7Q3QVZVNFJSMN7FGKTJ6H6HK5H7LAUHXMEBQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 194789780822}
2020-06-11T21:26:56.717+0200 INFO piecestore upload canceled {"Piece ID": "SQ37KY426QS6RTNHJUTOX5DTPHPJXAMZ6K6COT2MCAZIBL7VYA4Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT"}
2020-06-11T21:26:57.081+0200 INFO piecestore downloaded {"Piece ID": "2LKFDZFRJMATGMEFDEWLVOWOE7RKLYDTP55INFCH7DCRHC7VUIIQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2020-06-11T21:26:57.383+0200 INFO piecestore upload canceled {"Piece ID": "JKTKZOVSRSVGJHJNFZT3BUNHBVNMLCD6Q43D5LATIS2BAEFNERQQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2020-06-11T21:26:57.499+0200 INFO piecestore upload canceled {"Piece ID": "F3NV6F2V7FQDQ556RZS4JSZAW34ABRRVZYKCG2MRBOPCTXKMBUNA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-06-11T21:26:57.724+0200 INFO piecestore upload started {"Piece ID": "TP42SAPFYJXLNG7YMFZUHDHXO767GRDHZBE3UF6KZRN3EZZ533GQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT", "Available Space": 194787460950}
2020-06-11T21:26:57.821+0200 INFO piecestore upload canceled {"Piece ID": "TP42SAPFYJXLNG7YMFZUHDHXO767GRDHZBE3UF6KZRN3EZZ533GQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT"}
2020-06-11T21:26:57.933+0200 INFO piecestore upload canceled {"Piece ID": "6XH6UIKCHZYJ22FGX6WUVYQIID7R6CYIHCS363NBZ4M4RQPPFWNA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-06-11T21:26:58.114+0200 INFO piecestore download started {"Piece ID": "MGBPLCWCUNM2SCUNI3AHEIJL5TEO5YOSSF3ZWYZLMLPSKU4A5HXQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2020-06-11T21:26:58.135+0200 INFO piecestore upload started {"Piece ID": "D7ZW34ZVPZ5JRP25GLMENBIAX4RKTGYXD73YPH46WNMNKXSTMHWA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 194785125462}
2020-06-11T21:26:58.360+0200 INFO piecestore downloaded {"Piece ID": "EAMQ4OIQ3C6YEO5DKLO2MGIKAW2AXSXJ7AJGBFOO6G7WQGHQ54EQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "GET"}
2020-06-11T21:26:58.421+0200 INFO piecestore downloaded {"Piece ID": "MGBPLCWCUNM2SCUNI3AHEIJL5TEO5YOSSF3ZWYZLMLPSKU4A5HXQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2020-06-11T21:26:58.462+0200 INFO piecestore upload canceled {"Piece ID": "NWSUJ766ERHJPNGLQ4NVVHD6AWWC3HG7ISI5GKGO6T46A27YEBMA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2020-06-11T21:26:58.493+0200 INFO piecestore upload canceled {"Piece ID": "RXV56EZTP3TDTPEO7Q3QVZVNFJSMN7FGKTJ6H6HK5H7LAUHXMEBQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-06-11T21:26:59.277+0200 INFO piecestore uploaded {"Piece ID": "D7ZW34ZVPZ5JRP25GLMENBIAX4RKTGYXD73YPH46WNMNKXSTMHWA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-06-11T21:26:59.387+0200 INFO piecestore uploaded {"Piece ID": "3EGUE6NCDTAO42YDHCHI6NKNEABDNTANJ4KFJWXUJEE2R77HRDCA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2020-06-11T21:26:59.508+0200 INFO piecestore upload started {"Piece ID": "BBKP7L4UDIPFTYYZ2X57NXEBMIZTNZ23XCFPV56SCY65QYIW2DRQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 194775845974}
2020-06-11T21:26:59.529+0200 INFO piecestore upload started {"Piece ID": "DILJWA3FMJFB7RBY7QOGO7OGNQK25NKWHYUVTX7ZJOUWL7DSKLGA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 194775845974}
2020-06-11T21:27:00.152+0200 INFO piecestore upload started {"Piece ID": "QTSAJHHE75FOCT4GME4TW2C744VO7JOLCUIKLGWYM7CBWREKVQPQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 194775845974}
2020-06-11T21:27:00.174+0200 INFO piecestore upload started {"Piece ID": "Z3EZLZTGXAC4MNQ42WZJ2TO3PD7FUV4YGIFOWQ4KO5AMHHQQBVHQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 194775845974}
2020-06-11T21:27:00.728+0200 INFO piecedeleter deleted {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Piece ID": "J7UUMH2FG35EJJWHE2YYTWAZ6PNKNPNSJL2ZDDRTPTV5O6QE4X5A"}
2020-06-11T21:27:00.808+0200 INFO piecedeleter deleted {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Piece ID": "NVPULFQ643NUOGUD547XZEPKEYJLY7GCQCDVGXIAQZEGWFSA6LUA"}
2020-06-11T21:27:00.882+0200 INFO piecedeleter deleted {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Piece ID": "DN5KBB26OSZ5EAPPTRETMH6ZS2ZIK2J7PXPAT26HATUDCOBZ3TYQ"}
2020-06-11T21:27:00.928+0200 INFO piecestore upload started {"Piece ID": "CVCEDUBHREYMZQZSV6KICZRNEEYREDUGSTYGFR5R64BUBZU2GNBA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 194773533782}
2020-06-11T21:27:00.978+0200 INFO piecestore upload canceled {"Piece ID": "BBKP7L4UDIPFTYYZ2X57NXEBMIZTNZ23XCFPV56SCY65QYIW2DRQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-06-11T21:27:00.980+0200 INFO piecedeleter deleted {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Piece ID": "TJZLPGJO7TFTAKVXTAJLD6HNUN32X3SAC22ACT25IL3A7FRCYT7A"}
2020-06-11T21:27:00.992+0200 INFO piecestore download started {"Piece ID": "TWJ6N6AEZFOIFVUF243ZJWCYYCHZ3GNFZDG4XL6JHOI74HFXNDSQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2020-06-11T21:27:01.056+0200 INFO piecedeleter deleted {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Piece ID": "SRFA3DXTBIKVKZ2H4LO2YQE73K7DSQK6HI43ESJECD5C4EWTHOCA"}
2020-06-11T21:27:01.166+0200 INFO piecestore upload started {"Piece ID": "LBF3HCU5VEDR24U5VRBFOAOKR3O6R233RXXMWXZRJ4TZWYUHOC2Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 194773538902}
2020-06-11T21:27:01.198+0200 INFO piecestore upload started {"Piece ID": "6PIGNZ5SE4DTR4M3EEGTQMGM4Q4GFAFC5PJKSUUDCAQ7QNXYNRPA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 194773538902}
2020-06-11T21:27:01.310+0200 INFO piecestore upload canceled {"Piece ID": "CVCEDUBHREYMZQZSV6KICZRNEEYREDUGSTYGFR5R64BUBZU2GNBA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2020-06-11T21:27:01.397+0200 INFO piecestore downloaded {"Piece ID": "TWJ6N6AEZFOIFVUF243ZJWCYYCHZ3GNFZDG4XL6JHOI74HFXNDSQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2020-06-11T21:27:01.421+0200 INFO piecestore downloaded {"Piece ID": "SYDNCW7M3D6H7XDJT73LYJLAINPCMYOLEWWEH74OOAPJBTSHECBA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET"}
2020-06-11T21:27:01.568+0200 INFO piecestore upload started {"Piece ID": "NPCMZVIMVZTK4LRAA4V7RA6SF77OU57QB4LJF3JJVO6CCEZ6WAFQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 194773183830}
2020-06-11T21:27:02.209+0200 INFO piecestore upload canceled {"Piece ID": "Z3EZLZTGXAC4MNQ42WZJ2TO3PD7FUV4YGIFOWQ4KO5AMHHQQBVHQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-06-11T21:27:02.215+0200 INFO piecestore upload started {"Piece ID": "LUGK6UXLB4HRYBGCR7NXMSTEZS6RZK6B6DPMN2SWASYDLABRVJ2A", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 194770863958}
2020-06-11T21:27:02.280+0200 INFO piecestore download started {"Piece ID": "WMJAG7VXM2X2JASR3CS4IT3PEXGNUE47N27W6H5FZWUHQYPAXKUA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2020-06-11T21:27:02.335+0200 INFO piecestore upload started {"Piece ID": "R2BW7EGDYRISAKY5PIAEWGCF5QPZPCCZBLCMOTV46W2M2JRQKVQA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 194770863958}
2020-06-11T21:27:02.384+0200 INFO piecestore download started {"Piece ID": "L7NV37BQW3EVV732MOYFVPO2JJYIP3HA5DKBQDDER7U4FB5BGMKQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET"}
2020-06-11T21:27:02.392+0200 INFO piecedeleter deleted {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Piece ID": "GEA7FFFYVCAJJWHUJLIQYNWVEBZIYFZLXMPMGJIZOHMI3DJ4XMZA"}
2020-06-11T21:27:02.465+0200 INFO piecedeleter deleted {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Piece ID": "5PHDULABQ4AUHGC6DBLKYBEJSH3TPVTHNSK5CYQI5HCTWRIRRMNA"}
2020-06-11T21:27:02.532+0200 INFO piecedeleter deleted {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Piece ID": "QRXITRP5UU2WEL5KGNOPQCBV6AOQTICA7FIMV6UCAYPVACRWKPUA"}
2020-06-11T21:27:02.898+0200 INFO piecestore upload started {"Piece ID": "LXTHEA7LOTWICBM4FRLI7IOENHJ7B5UPSYCSRRSMKLKF5WVEHNXQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 194770871638}
2020-06-11T21:27:02.937+0200 INFO piecestore uploaded {"Piece ID": "LXTHEA7LOTWICBM4FRLI7IOENHJ7B5UPSYCSRRSMKLKF5WVEHNXQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT"}
2020-06-11T21:27:03.120+0200 INFO piecestore download started {"Piece ID": "U2N66PVS2R2T4CO6ZP567WNWDX2ONYHESDCLBUNGBD4KJZESZQDQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2020-06-11T21:27:03.284+0200 INFO piecestore downloaded {"Piece ID": "U2N66PVS2R2T4CO6ZP567WNWDX2ONYHESDCLBUNGBD4KJZESZQDQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2020-06-11T21:27:03.348+0200 INFO piecestore uploaded {"Piece ID": "DILJWA3FMJFB7RBY7QOGO7OGNQK25NKWHYUVTX7ZJOUWL7DSKLGA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2020-06-11T21:27:03.385+0200 INFO piecestore upload canceled {"Piece ID": "NPCMZVIMVZTK4LRAA4V7RA6SF77OU57QB4LJF3JJVO6CCEZ6WAFQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-06-11T21:27:03.508+0200 INFO piecestore upload canceled {"Piece ID": "6PIGNZ5SE4DTR4M3EEGTQMGM4Q4GFAFC5PJKSUUDCAQ7QNXYNRPA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-06-11T21:27:03.932+0200 ERROR piecestore:cache error getting current space used calculation: {"error": "lstat /Storj/Storj-v3-data/storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/7o/ejeq4pyvvikje2o7nexhhxwt47gi5ij6dvc76w5y6eu57qop2a.sj1: no such file or directory; lstat /Storj/Storj-v3-data/storage/blobs/qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa/c2/g3n2rbju2b5ukbq7y5l54xj2bdip3znzk45lm4aiepamg3mcwa.sj1: no such file or directory", "errorVerbose": "group:\n--- lstat /Storj/Storj-v3-data/storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/7o/ejeq4pyvvikje2o7nexhhxwt47gi5ij6dvc76w5y6eu57qop2a.sj1: no such file or directory\n--- lstat /Storj/Storj-v3-data/storage/blobs/qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa/c2/g3n2rbju2b5ukbq7y5l54xj2bdip3znzk45lm4aiepamg3mcwa.sj1: no such file or directory"}
2020-06-11T21:27:03.934+0200 INFO piecestore upload canceled {"Piece ID": "LUGK6UXLB4HRYBGCR7NXMSTEZS6RZK6B6DPMN2SWASYDLABRVJ2A", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-06-11T21:27:03.934+0200 INFO piecestore download canceled {"Piece ID": "L7NV37BQW3EVV732MOYFVPO2JJYIP3HA5DKBQDDER7U4FB5BGMKQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET"}
2020-06-11T21:27:03.935+0200 INFO piecestore download canceled {"Piece ID": "WMJAG7VXM2X2JASR3CS4IT3PEXGNUE47N27W6H5FZWUHQYPAXKUA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2020-06-11T21:27:03.935+0200 INFO piecestore upload canceled {"Piece ID": "R2BW7EGDYRISAKY5PIAEWGCF5QPZPCCZBLCMOTV46W2M2JRQKVQA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-06-11T21:27:03.943+0200 INFO piecestore upload canceled {"Piece ID": "LBF3HCU5VEDR24U5VRBFOAOKR3O6R233RXXMWXZRJ4TZWYUHOC2Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2020-06-11T21:27:03.944+0200 INFO piecestore upload canceled {"Piece ID": "QTSAJHHE75FOCT4GME4TW2C744VO7JOLCUIKLGWYM7CBWREKVQPQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2020-06-11T21:27:04.763+0200 FATAL Unrecoverable error {"error": "lstat /Storj/Storj-v3-data/storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/7o/ejeq4pyvvikje2o7nexhhxwt47gi5ij6dvc76w5y6eu57qop2a.sj1: no such file or directory; lstat /Storj/Storj-v3-data/storage/blobs/qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa/c2/g3n2rbju2b5ukbq7y5l54xj2bdip3znzk45lm4aiepamg3mcwa.sj1: no such file or directory", "errorVerbose": "group:\n--- lstat /Storj/Storj-v3-data/storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/7o/ejeq4pyvvikje2o7nexhhxwt47gi5ij6dvc76w5y6eu57qop2a.sj1: no such file or directory\n--- lstat /Storj/Storj-v3-data/storage/blobs/qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa/c2/g3n2rbju2b5ukbq7y5l54xj2bdip3znzk45lm4aiepamg3mcwa.sj1: no such file or directory"}