Who meet this error - please, enable the debug level for logs (it’s a log.level: option in the "C:\Program Files\Storj\Storage Node\config.yaml") and post one of logs (30-40 strings, when the node is crashed) between three backticks: ``` at the first line and three backticks after the last line.
Do not forget to save the config and restart the storagenode service.
Please, also add the following two options to your configuration file following what @Alexey has commented above.
log.caller: truelog.stack: trueThanks for your help.
Here you go:
2020-06-11T21:26:54.334+0200 INFO piecestore upload started {"Piece ID": "DGTXPA3Z3PXHMGPPRBJKJQBORRUQQVU6QRBT6C6V2ZQUL7KUQXFA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 194794422358}
2020-06-11T21:26:54.412+0200 INFO piecestore upload started {"Piece ID": "FOY4XHK6O7Z7JXRMBIOF2EMTJYZETS7F4ZTCDDB2PZ3ZX5KJU3VQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 194794422358}
2020-06-11T21:26:54.965+0200 INFO piecestore upload canceled {"Piece ID": "H43C6MAFJM3EOPRCTXCICURWZVBZ7L7LKPJSQ3TOX3RT6TJDBUGA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2020-06-11T21:26:55.389+0200 INFO piecedeleter deleted {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Piece ID": "IAW7UHTUCNBIRSFCRXNXDTPRHQEVHB3QB6IYGUYYIL353WLUYNEA"}
2020-06-11T21:26:55.458+0200 INFO piecedeleter deleted {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Piece ID": "UYW6J6TRTOD4DPMNYNDHT7NJU3G5IWQROVBYYHVCU5TPLJ5KOGQA"}
2020-06-11T21:26:55.544+0200 INFO piecedeleter deleted {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Piece ID": "OBOXK2HHZW3HX7QS536BTPZMQFOBG66EAQGWP6BCTWRXCBDPY4WA"}
2020-06-11T21:26:55.586+0200 INFO piecedeleter deleted {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Piece ID": "XBR6YLHXENRHSHLNKSMIHZVYYHJFM33ZU3TNBIKSCWTZONP3223A"}
2020-06-11T21:26:55.671+0200 INFO piecedeleter deleted {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Piece ID": "6FB6C3BWVSPCX2TKYH6K3QJNJTQCLFOENFTTQJEZRQTI36Y6NJSA"}
2020-06-11T21:26:55.843+0200 INFO piecestore upload started {"Piece ID": "F3NV6F2V7FQDQ556RZS4JSZAW34ABRRVZYKCG2MRBOPCTXKMBUNA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 194794435158}
2020-06-11T21:26:55.968+0200 INFO piecedeleter deleted {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Piece ID": "ARMIGRQD5F2GWH6RILBCYCNQV7XFKBUUBZEWQDGE6KUKEYAW6QZQ"}
2020-06-11T21:26:56.040+0200 INFO piecestore upload started {"Piece ID": "3EGUE6NCDTAO42YDHCHI6NKNEABDNTANJ4KFJWXUJEE2R77HRDCA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 194794437718}
2020-06-11T21:26:56.042+0200 INFO piecedeleter deleted {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Piece ID": "WDVRR7GK5EYSHT76E7FP4AE26GR5XYJ4WIUFFFM54VEU3K2N3YSA"}
2020-06-11T21:26:56.226+0200 INFO piecestore upload canceled {"Piece ID": "FOY4XHK6O7Z7JXRMBIOF2EMTJYZETS7F4ZTCDDB2PZ3ZX5KJU3VQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-06-11T21:26:56.490+0200 INFO piecestore upload started {"Piece ID": "6XH6UIKCHZYJ22FGX6WUVYQIID7R6CYIHCS363NBZ4M4RQPPFWNA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 194789800534}
2020-06-11T21:26:56.508+0200 INFO piecestore upload canceled {"Piece ID": "DGTXPA3Z3PXHMGPPRBJKJQBORRUQQVU6QRBT6C6V2ZQUL7KUQXFA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-06-11T21:26:56.573+0200 INFO piecestore download started {"Piece ID": "2LKFDZFRJMATGMEFDEWLVOWOE7RKLYDTP55INFCH7DCRHC7VUIIQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2020-06-11T21:26:56.628+0200 INFO piecestore upload started {"Piece ID": "SQ37KY426QS6RTNHJUTOX5DTPHPJXAMZ6K6COT2MCAZIBL7VYA4Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 194789800534}
2020-06-11T21:26:56.709+0200 INFO piecestore upload started {"Piece ID": "RXV56EZTP3TDTPEO7Q3QVZVNFJSMN7FGKTJ6H6HK5H7LAUHXMEBQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 194789780822}
2020-06-11T21:26:56.717+0200 INFO piecestore upload canceled {"Piece ID": "SQ37KY426QS6RTNHJUTOX5DTPHPJXAMZ6K6COT2MCAZIBL7VYA4Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT"}
2020-06-11T21:26:57.081+0200 INFO piecestore downloaded {"Piece ID": "2LKFDZFRJMATGMEFDEWLVOWOE7RKLYDTP55INFCH7DCRHC7VUIIQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2020-06-11T21:26:57.383+0200 INFO piecestore upload canceled {"Piece ID": "JKTKZOVSRSVGJHJNFZT3BUNHBVNMLCD6Q43D5LATIS2BAEFNERQQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2020-06-11T21:26:57.499+0200 INFO piecestore upload canceled {"Piece ID": "F3NV6F2V7FQDQ556RZS4JSZAW34ABRRVZYKCG2MRBOPCTXKMBUNA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-06-11T21:26:57.724+0200 INFO piecestore upload started {"Piece ID": "TP42SAPFYJXLNG7YMFZUHDHXO767GRDHZBE3UF6KZRN3EZZ533GQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT", "Available Space": 194787460950}
2020-06-11T21:26:57.821+0200 INFO piecestore upload canceled {"Piece ID": "TP42SAPFYJXLNG7YMFZUHDHXO767GRDHZBE3UF6KZRN3EZZ533GQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT"}
2020-06-11T21:26:57.933+0200 INFO piecestore upload canceled {"Piece ID": "6XH6UIKCHZYJ22FGX6WUVYQIID7R6CYIHCS363NBZ4M4RQPPFWNA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-06-11T21:26:58.114+0200 INFO piecestore download started {"Piece ID": "MGBPLCWCUNM2SCUNI3AHEIJL5TEO5YOSSF3ZWYZLMLPSKU4A5HXQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2020-06-11T21:26:58.135+0200 INFO piecestore upload started {"Piece ID": "D7ZW34ZVPZ5JRP25GLMENBIAX4RKTGYXD73YPH46WNMNKXSTMHWA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 194785125462}
2020-06-11T21:26:58.360+0200 INFO piecestore downloaded {"Piece ID": "EAMQ4OIQ3C6YEO5DKLO2MGIKAW2AXSXJ7AJGBFOO6G7WQGHQ54EQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "GET"}
2020-06-11T21:26:58.421+0200 INFO piecestore downloaded {"Piece ID": "MGBPLCWCUNM2SCUNI3AHEIJL5TEO5YOSSF3ZWYZLMLPSKU4A5HXQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2020-06-11T21:26:58.462+0200 INFO piecestore upload canceled {"Piece ID": "NWSUJ766ERHJPNGLQ4NVVHD6AWWC3HG7ISI5GKGO6T46A27YEBMA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2020-06-11T21:26:58.493+0200 INFO piecestore upload canceled {"Piece ID": "RXV56EZTP3TDTPEO7Q3QVZVNFJSMN7FGKTJ6H6HK5H7LAUHXMEBQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-06-11T21:26:59.277+0200 INFO piecestore uploaded {"Piece ID": "D7ZW34ZVPZ5JRP25GLMENBIAX4RKTGYXD73YPH46WNMNKXSTMHWA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-06-11T21:26:59.387+0200 INFO piecestore uploaded {"Piece ID": "3EGUE6NCDTAO42YDHCHI6NKNEABDNTANJ4KFJWXUJEE2R77HRDCA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2020-06-11T21:26:59.508+0200 INFO piecestore upload started {"Piece ID": "BBKP7L4UDIPFTYYZ2X57NXEBMIZTNZ23XCFPV56SCY65QYIW2DRQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 194775845974}
2020-06-11T21:26:59.529+0200 INFO piecestore upload started {"Piece ID": "DILJWA3FMJFB7RBY7QOGO7OGNQK25NKWHYUVTX7ZJOUWL7DSKLGA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 194775845974}
2020-06-11T21:27:00.152+0200 INFO piecestore upload started {"Piece ID": "QTSAJHHE75FOCT4GME4TW2C744VO7JOLCUIKLGWYM7CBWREKVQPQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 194775845974}
2020-06-11T21:27:00.174+0200 INFO piecestore upload started {"Piece ID": "Z3EZLZTGXAC4MNQ42WZJ2TO3PD7FUV4YGIFOWQ4KO5AMHHQQBVHQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 194775845974}
2020-06-11T21:27:00.728+0200 INFO piecedeleter deleted {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Piece ID": "J7UUMH2FG35EJJWHE2YYTWAZ6PNKNPNSJL2ZDDRTPTV5O6QE4X5A"}
2020-06-11T21:27:00.808+0200 INFO piecedeleter deleted {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Piece ID": "NVPULFQ643NUOGUD547XZEPKEYJLY7GCQCDVGXIAQZEGWFSA6LUA"}
2020-06-11T21:27:00.882+0200 INFO piecedeleter deleted {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Piece ID": "DN5KBB26OSZ5EAPPTRETMH6ZS2ZIK2J7PXPAT26HATUDCOBZ3TYQ"}
2020-06-11T21:27:00.928+0200 INFO piecestore upload started {"Piece ID": "CVCEDUBHREYMZQZSV6KICZRNEEYREDUGSTYGFR5R64BUBZU2GNBA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 194773533782}
2020-06-11T21:27:00.978+0200 INFO piecestore upload canceled {"Piece ID": "BBKP7L4UDIPFTYYZ2X57NXEBMIZTNZ23XCFPV56SCY65QYIW2DRQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-06-11T21:27:00.980+0200 INFO piecedeleter deleted {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Piece ID": "TJZLPGJO7TFTAKVXTAJLD6HNUN32X3SAC22ACT25IL3A7FRCYT7A"}
2020-06-11T21:27:00.992+0200 INFO piecestore download started {"Piece ID": "TWJ6N6AEZFOIFVUF243ZJWCYYCHZ3GNFZDG4XL6JHOI74HFXNDSQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2020-06-11T21:27:01.056+0200 INFO piecedeleter deleted {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Piece ID": "SRFA3DXTBIKVKZ2H4LO2YQE73K7DSQK6HI43ESJECD5C4EWTHOCA"}
2020-06-11T21:27:01.166+0200 INFO piecestore upload started {"Piece ID": "LBF3HCU5VEDR24U5VRBFOAOKR3O6R233RXXMWXZRJ4TZWYUHOC2Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 194773538902}
2020-06-11T21:27:01.198+0200 INFO piecestore upload started {"Piece ID": "6PIGNZ5SE4DTR4M3EEGTQMGM4Q4GFAFC5PJKSUUDCAQ7QNXYNRPA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 194773538902}
2020-06-11T21:27:01.310+0200 INFO piecestore upload canceled {"Piece ID": "CVCEDUBHREYMZQZSV6KICZRNEEYREDUGSTYGFR5R64BUBZU2GNBA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2020-06-11T21:27:01.397+0200 INFO piecestore downloaded {"Piece ID": "TWJ6N6AEZFOIFVUF243ZJWCYYCHZ3GNFZDG4XL6JHOI74HFXNDSQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2020-06-11T21:27:01.421+0200 INFO piecestore downloaded {"Piece ID": "SYDNCW7M3D6H7XDJT73LYJLAINPCMYOLEWWEH74OOAPJBTSHECBA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET"}
2020-06-11T21:27:01.568+0200 INFO piecestore upload started {"Piece ID": "NPCMZVIMVZTK4LRAA4V7RA6SF77OU57QB4LJF3JJVO6CCEZ6WAFQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 194773183830}
2020-06-11T21:27:02.209+0200 INFO piecestore upload canceled {"Piece ID": "Z3EZLZTGXAC4MNQ42WZJ2TO3PD7FUV4YGIFOWQ4KO5AMHHQQBVHQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-06-11T21:27:02.215+0200 INFO piecestore upload started {"Piece ID": "LUGK6UXLB4HRYBGCR7NXMSTEZS6RZK6B6DPMN2SWASYDLABRVJ2A", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 194770863958}
2020-06-11T21:27:02.280+0200 INFO piecestore download started {"Piece ID": "WMJAG7VXM2X2JASR3CS4IT3PEXGNUE47N27W6H5FZWUHQYPAXKUA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2020-06-11T21:27:02.335+0200 INFO piecestore upload started {"Piece ID": "R2BW7EGDYRISAKY5PIAEWGCF5QPZPCCZBLCMOTV46W2M2JRQKVQA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 194770863958}
2020-06-11T21:27:02.384+0200 INFO piecestore download started {"Piece ID": "L7NV37BQW3EVV732MOYFVPO2JJYIP3HA5DKBQDDER7U4FB5BGMKQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET"}
2020-06-11T21:27:02.392+0200 INFO piecedeleter deleted {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Piece ID": "GEA7FFFYVCAJJWHUJLIQYNWVEBZIYFZLXMPMGJIZOHMI3DJ4XMZA"}
2020-06-11T21:27:02.465+0200 INFO piecedeleter deleted {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Piece ID": "5PHDULABQ4AUHGC6DBLKYBEJSH3TPVTHNSK5CYQI5HCTWRIRRMNA"}
2020-06-11T21:27:02.532+0200 INFO piecedeleter deleted {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Piece ID": "QRXITRP5UU2WEL5KGNOPQCBV6AOQTICA7FIMV6UCAYPVACRWKPUA"}
2020-06-11T21:27:02.898+0200 INFO piecestore upload started {"Piece ID": "LXTHEA7LOTWICBM4FRLI7IOENHJ7B5UPSYCSRRSMKLKF5WVEHNXQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 194770871638}
2020-06-11T21:27:02.937+0200 INFO piecestore uploaded {"Piece ID": "LXTHEA7LOTWICBM4FRLI7IOENHJ7B5UPSYCSRRSMKLKF5WVEHNXQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT"}
2020-06-11T21:27:03.120+0200 INFO piecestore download started {"Piece ID": "U2N66PVS2R2T4CO6ZP567WNWDX2ONYHESDCLBUNGBD4KJZESZQDQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2020-06-11T21:27:03.284+0200 INFO piecestore downloaded {"Piece ID": "U2N66PVS2R2T4CO6ZP567WNWDX2ONYHESDCLBUNGBD4KJZESZQDQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2020-06-11T21:27:03.348+0200 INFO piecestore uploaded {"Piece ID": "DILJWA3FMJFB7RBY7QOGO7OGNQK25NKWHYUVTX7ZJOUWL7DSKLGA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2020-06-11T21:27:03.385+0200 INFO piecestore upload canceled {"Piece ID": "NPCMZVIMVZTK4LRAA4V7RA6SF77OU57QB4LJF3JJVO6CCEZ6WAFQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-06-11T21:27:03.508+0200 INFO piecestore upload canceled {"Piece ID": "6PIGNZ5SE4DTR4M3EEGTQMGM4Q4GFAFC5PJKSUUDCAQ7QNXYNRPA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-06-11T21:27:03.932+0200 ERROR piecestore:cache error getting current space used calculation: {"error": "lstat /Storj/Storj-v3-data/storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/7o/ejeq4pyvvikje2o7nexhhxwt47gi5ij6dvc76w5y6eu57qop2a.sj1: no such file or directory; lstat /Storj/Storj-v3-data/storage/blobs/qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa/c2/g3n2rbju2b5ukbq7y5l54xj2bdip3znzk45lm4aiepamg3mcwa.sj1: no such file or directory", "errorVerbose": "group:\n--- lstat /Storj/Storj-v3-data/storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/7o/ejeq4pyvvikje2o7nexhhxwt47gi5ij6dvc76w5y6eu57qop2a.sj1: no such file or directory\n--- lstat /Storj/Storj-v3-data/storage/blobs/qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa/c2/g3n2rbju2b5ukbq7y5l54xj2bdip3znzk45lm4aiepamg3mcwa.sj1: no such file or directory"}
2020-06-11T21:27:03.934+0200 INFO piecestore upload canceled {"Piece ID": "LUGK6UXLB4HRYBGCR7NXMSTEZS6RZK6B6DPMN2SWASYDLABRVJ2A", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-06-11T21:27:03.934+0200 INFO piecestore download canceled {"Piece ID": "L7NV37BQW3EVV732MOYFVPO2JJYIP3HA5DKBQDDER7U4FB5BGMKQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET"}
2020-06-11T21:27:03.935+0200 INFO piecestore download canceled {"Piece ID": "WMJAG7VXM2X2JASR3CS4IT3PEXGNUE47N27W6H5FZWUHQYPAXKUA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2020-06-11T21:27:03.935+0200 INFO piecestore upload canceled {"Piece ID": "R2BW7EGDYRISAKY5PIAEWGCF5QPZPCCZBLCMOTV46W2M2JRQKVQA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-06-11T21:27:03.943+0200 INFO piecestore upload canceled {"Piece ID": "LBF3HCU5VEDR24U5VRBFOAOKR3O6R233RXXMWXZRJ4TZWYUHOC2Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2020-06-11T21:27:03.944+0200 INFO piecestore upload canceled {"Piece ID": "QTSAJHHE75FOCT4GME4TW2C744VO7JOLCUIKLGWYM7CBWREKVQPQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2020-06-11T21:27:04.763+0200 FATAL Unrecoverable error {"error": "lstat /Storj/Storj-v3-data/storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/7o/ejeq4pyvvikje2o7nexhhxwt47gi5ij6dvc76w5y6eu57qop2a.sj1: no such file or directory; lstat /Storj/Storj-v3-data/storage/blobs/qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa/c2/g3n2rbju2b5ukbq7y5l54xj2bdip3znzk45lm4aiepamg3mcwa.sj1: no such file or directory", "errorVerbose": "group:\n--- lstat /Storj/Storj-v3-data/storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/7o/ejeq4pyvvikje2o7nexhhxwt47gi5ij6dvc76w5y6eu57qop2a.sj1: no such file or directory\n--- lstat /Storj/Storj-v3-data/storage/blobs/qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa/c2/g3n2rbju2b5ukbq7y5l54xj2bdip3znzk45lm4aiepamg3mcwa.sj1: no such file or directory"}
These options should be enabled in the config.yaml:
log.level: debug
log.caller: true
log.stack: true
Save the config and restart the node (elevated Powershell):
Restart-Service storagenode
After that please, post a log with errors after the node is fail.
log.level: debug was set at the time of the original crash. I have set the other two but another crash is yet to happen. It is quite a drawback not being able to change this retroactively in some fashion.
You can, but the debug port should be accessible:
By retroactively I meant to be able to change log level of old logs. But I guess something like that has the least priority among tasks at the moment.
Good to know you can change it without restarting at least though.
It’s a text file on time when the event is happen.
That would be nice to have a time machine, but it is not invented yet
Some software allows you to select with how much detail you want to see the log, I assume they keep everything.
Well here’s a closes thing to it then:
2020-06-19T04:31:47.122+0200 INFO piecedeleter pieces/deleter.go:144 deleted {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "5GZAOYN2SNCTGU3KKHBG42O5BS625EAXOTXTCPODSSNZY7DDVXFA"}
2020-06-19T04:31:47.244+0200 INFO piecedeleter pieces/deleter.go:144 deleted {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "UDD3FSC3G4KBOSXOLSK3XQRA2IPQHPZXUEHSRVAK32FUBQAAOLMQ"}
2020-06-19T04:31:48.308+0200 INFO piecedeleter pieces/deleter.go:144 deleted {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "JKDKG2LVTLUZTDVOCISQGFVMWNM4EHV4HKMFV7UICEOQSLID4GDQ"}
2020-06-19T04:31:48.655+0200 INFO piecedeleter pieces/deleter.go:144 deleted {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "IW5L6BHZXKL3KIM2ZFHFODBZKTW4JKJOP54JGLZNNZKKU53AFI3Q"}
2020-06-19T04:31:48.988+0200 ERROR collector collector/service.go:101 unable to delete piece {"Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "QZYJXKQ37NYKAPMDZUAUMIUXVZHHJMSXLFKJBW4KWKOCRPZK5FZQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:96\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:238\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:281\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:95\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:55\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/storj/storagenode/collector.(*Service).Run:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func1:56\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
storj.io/storj/storagenode/collector.(*Service).Collect
/go/src/storj.io/storj/storagenode/collector/service.go:101
storj.io/storj/storagenode/collector.(*Service).Run.func1
/go/src/storj.io/storj/storagenode/collector/service.go:55
storj.io/common/sync2.(*Cycle).Run
/go/pkg/mod/storj.io/common@v0.0.0-20200519171747-3ff8acf78c46/sync2/cycle.go:152
storj.io/storj/storagenode/collector.(*Service).Run
/go/src/storj.io/storj/storagenode/collector/service.go:51
storj.io/storj/private/lifecycle.(*Group).Run.func1
/go/src/storj.io/storj/private/lifecycle/group.go:56
golang.org/x/sync/errgroup.(*Group).Go.func1
/go/pkg/mod/golang.org/x/sync@v0.0.0-20200317015054-43a5402ce75a/errgroup/errgroup.go:57
2020-06-19T04:31:49.571+0200 INFO piecedeleter pieces/deleter.go:144 deleted {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "S72CN3NH46Z2HI4AQICO2246RSIZI27R24VQJOIDNUZOWNLAELYQ"}
2020-06-19T04:31:51.294+0200 INFO piecestore piecestore/endpoint.go:503 downloaded {"Piece ID": "TFTI6CXDVZQJLM7R2EMVPIMJGYDFL5RX65QDRLZ6K2UTAX3BG6QQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET"}
2020-06-19T04:31:52.642+0200 INFO piecedeleter pieces/deleter.go:144 deleted {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "LX6IDYZH5HFPA4HFNY7NCABTKLJYAJIAFXT7T6X2AG27M5RWYMRA"}
2020-06-19T04:31:52.771+0200 INFO piecedeleter pieces/deleter.go:144 deleted {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "H4K5M4FBRV3ZJD45ZCPA4YCR5HWUTFI3EPE4TPGIGQRAYRQN5HIQ"}
2020-06-19T04:31:53.889+0200 INFO piecedeleter pieces/deleter.go:144 deleted {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "NJOQY4AZZJYC5IPIXEMRHMVRMVUA7YIZ45UHEXOBAJHPMM45NO2Q"}
2020-06-19T04:31:54.346+0200 INFO piecedeleter pieces/deleter.go:144 deleted {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "ESUKDPNOAXITUHBHDD562Y6CMXQRLBFILBEGMK7VFOOBGMY3V2XQ"}
2020-06-19T04:31:55.443+0200 INFO piecedeleter pieces/deleter.go:144 deleted {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "OY2A7UY5SIZTRHPJ5DCPA5TQZTSPLP47FDGRZYU2VMBDNEHRWY2Q"}
2020-06-19T04:31:56.072+0200 INFO piecedeleter pieces/deleter.go:144 deleted {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "2NXRKR3UAB2KW6WUHRXS3OOSU6LG6Z7QCDMX6P2JJXJ4X7N6DFJA"}
2020-06-19T04:31:56.347+0200 INFO piecedeleter pieces/deleter.go:144 deleted {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "YGVVLFDFYUHEWLPAKS3QPHVGG7O7ECDESFD7UEMXUM6HIZHVRU7Q"}
2020-06-19T04:31:57.741+0200 INFO piecedeleter pieces/deleter.go:144 deleted {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "PCWFT24PJS4NYDQ2W5G3TB3QCO3ZUNZA57VEUWMXSB4GQCOMTYGA"}
2020-06-19T04:31:57.925+0200 INFO piecedeleter pieces/deleter.go:144 deleted {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "2GCZAH6OLOYEUQPFLR4VAUZ4V2D4FB7O7U3NSOM7NKU2J65MJ6LQ"}
2020-06-19T04:31:58.533+0200 INFO piecestore piecestore/endpoint.go:465 download started {"Piece ID": "EQGG2RD5OC2SHJ4U6CCGAPRYZKVVTQNNQQLZNEWLD3JBUHDS3OSQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR"}
2020-06-19T04:31:58.739+0200 INFO piecestore piecestore/endpoint.go:503 downloaded {"Piece ID": "EQGG2RD5OC2SHJ4U6CCGAPRYZKVVTQNNQQLZNEWLD3JBUHDS3OSQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR"}
2020-06-19T04:31:58.943+0200 INFO piecedeleter pieces/deleter.go:144 deleted {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "SGQDMUVHOCJ2PDGZRSUHREDJ3ONXR3IEAJAKNDFXZRE5AT5NPAPQ"}
2020-06-19T04:31:58.955+0200 INFO piecedeleter pieces/deleter.go:144 deleted {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "ZGUQD74EXQNWJ3MO2TXQMKOV6QOGZTF5QUGINBK5HGWVZRRPPTTQ"}
2020-06-19T04:31:59.789+0200 INFO piecedeleter pieces/deleter.go:144 deleted {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "O6D7FCUFOYYWRWWQ6XJPROZU3FDEBG56UW2IECWSJVBLNS5ZUQHQ"}
2020-06-19T04:32:01.180+0200 INFO piecestore piecestore/endpoint.go:465 download started {"Piece ID": "N5KDX7Q24G7ZJZJUWTLERKGN7REX7OKIWQJXF3TGWBX7PEB35NUQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_REPAIR"}
2020-06-19T04:32:02.527+0200 INFO piecedeleter pieces/deleter.go:144 deleted {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "RS22O3KAOTIWXUBS3UGSS3IZPHCYT6UN6RC6YEG7DOLSW3TEYVWQ"}
2020-06-19T04:32:04.629+0200 INFO piecedeleter pieces/deleter.go:144 deleted {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "Q2IE57HQDOA2YEHITR6LJNQM4FUMAMVIFW5RK5CV7CXET72EEVZQ"}
2020-06-19T04:32:05.650+0200 INFO piecedeleter pieces/deleter.go:144 deleted {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "SGRMNNU32HXGX3C5H5OCDG2OVJKB6N2NE6XJ46KTB4EAGZFMNNFA"}
2020-06-19T04:32:06.286+0200 INFO piecedeleter pieces/deleter.go:144 deleted {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "A2GHWLRDNBDT3537JMPOKPQD62NCHXL33QLYWDHKE6SP32LD5KOQ"}
2020-06-19T04:32:07.112+0200 INFO piecestore piecestore/endpoint.go:465 download started {"Piece ID": "JI6T2CRCYNYULUVJQN2GNNVXUKTB6VMLE32RMFMIRWVBFSUTCPRA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET"}
2020-06-19T04:32:07.126+0200 INFO piecedeleter pieces/deleter.go:144 deleted {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "77GWE7K6GLGIQGZCZ4DDQEHZU6HHW6B7WQ264BMCTISCO4SUV6AA"}
2020-06-19T04:32:07.155+0200 INFO piecedeleter pieces/deleter.go:144 deleted {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "26AM4BFCB4LMLIPIH2DP645DIQK3A2MPFWWELRUJMRQAMC3FXPEQ"}
2020-06-19T04:32:07.337+0200 INFO piecestore piecestore/endpoint.go:503 downloaded {"Piece ID": "JI6T2CRCYNYULUVJQN2GNNVXUKTB6VMLE32RMFMIRWVBFSUTCPRA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET"}
2020-06-19T04:32:07.486+0200 INFO piecedeleter pieces/deleter.go:144 deleted {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "QI6X2G63BZUI5OGISSATH2HUTUVY3ZU6AERSYG5D3BABJ6RTH74Q"}
2020-06-19T04:32:07.657+0200 INFO piecedeleter pieces/deleter.go:144 deleted {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "FOQJOHJ4A2IW7U23J7N45CCX4LEMQBJEYARGQQ4B5LZ2YECXB3MA"}
2020-06-19T04:32:09.272+0200 INFO piecestore piecestore/endpoint.go:465 download started {"Piece ID": "7LJSTQBA4QUIV22GYQ3PQ7SGYL33RQ3IKM3LFBJB524DHVFHH5FA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET_REPAIR"}
2020-06-19T04:32:09.668+0200 INFO piecestore piecestore/endpoint.go:503 downloaded {"Piece ID": "7LJSTQBA4QUIV22GYQ3PQ7SGYL33RQ3IKM3LFBJB524DHVFHH5FA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET_REPAIR"}
2020-06-19T04:32:10.162+0200 INFO piecedeleter pieces/deleter.go:144 deleted {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "KLY4E6KTWJW7H5VXX4TW5CDDDFYNSJ3WULN5CAEKGOC7FN5VOBRQ"}
2020-06-19T04:32:10.182+0200 INFO piecedeleter pieces/deleter.go:144 deleted {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "AEVQOQ6INDOHANMOV7DGTQWUB7GTDTZ4RNSL2C7RM5NLNA45Y4HQ"}
2020-06-19T04:32:11.009+0200 INFO piecedeleter pieces/deleter.go:144 deleted {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "R3Z5XHBJCJBV2NXXDA2M7T2DXIZVDSOFC6RFH4HG2MQ7NW5U6FQA"}
2020-06-19T04:32:11.513+0200 INFO piecedeleter pieces/deleter.go:144 deleted {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "2USJVDMX546RZLTHHJ76JMXSNZK74OXNW66LBIJRI3UPVANZKB2A"}
2020-06-19T04:32:12.154+0200 INFO piecestore piecestore/endpoint.go:503 downloaded {"Piece ID": "N5KDX7Q24G7ZJZJUWTLERKGN7REX7OKIWQJXF3TGWBX7PEB35NUQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_REPAIR"}
2020-06-19T04:32:13.718+0200 ERROR piecestore:cache pieces/cache.go:56 error getting current space used calculation: {"error": "lstat /Storj/Storj-v3-data/storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/s5/idqsy3x4etubzbxox5ovj253zqswlitqgvz5gycomvdj5xxysq.sj1: no such file or directory"}
storj.io/storj/storagenode/pieces.(*CacheService).Run
/go/src/storj.io/storj/storagenode/pieces/cache.go:56
storj.io/storj/private/lifecycle.(*Group).Run.func1
/go/src/storj.io/storj/private/lifecycle/group.go:56
golang.org/x/sync/errgroup.(*Group).Go.func1
/go/pkg/mod/golang.org/x/sync@v0.0.0-20200317015054-43a5402ce75a/errgroup/errgroup.go:57
2020-06-19T04:32:14.338+0200 FATAL process/exec_conf.go:396 Unrecoverable error {"error": "lstat /Storj/Storj-v3-data/storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/s5/idqsy3x4etubzbxox5ovj253zqswlitqgvz5gycomvdj5xxysq.sj1: no such file or directory"}
storj.io/private/process.cleanup.func1
/go/pkg/mod/storj.io/private@v0.0.0-20200504130741-565a173f6d33/process/exec_conf.go:396
github.com/spf13/cobra.(*Command).execute
/go/pkg/mod/github.com/spf13/cobra@v0.0.6/command.go:840
github.com/spf13/cobra.(*Command).ExecuteC
/go/pkg/mod/github.com/spf13/cobra@v0.0.6/command.go:945
github.com/spf13/cobra.(*Command).Execute
/go/pkg/mod/github.com/spf13/cobra@v0.0.6/command.go:885
storj.io/private/process.ExecWithCustomConfig
/go/pkg/mod/storj.io/private@v0.0.0-20200504130741-565a173f6d33/process/exec_conf.go:88
storj.io/private/process.ExecCustomDebug
/go/pkg/mod/storj.io/private@v0.0.0-20200504130741-565a173f6d33/process/exec_conf.go:70
main.main
/go/src/storj.io/storj/cmd/storagenode/main.go:320
runtime.main
/usr/local/go/src/runtime/proc.go:203
Ours too. You enabled extended log and now we can see more information.
I do not aware of any service which can automagically add missed information to the text log back to the past.
This is the same, as requesting to show a video from the Polaroid photo.
You are able to do so, if you start a record of video before the time of the making a Polaroid photo.
The same for logging - you can receive an extended log, if you enable the extended logging long before it is needed.
Thanks for providing the log.
Do you know how long in average your node takes in average to crash since it’s started?
Can you also check if your node has the file blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/s5/idqsy3x4etubzbxox5ovj253zqswlitqgvz5gycomvdj5xxysq.sj1 in the data directory?
Thanks
I’ve seen some where you can select a level with which you want to view the log, I assume they keep the whole log and then filter it later. Is there any benefit to logging higher priority events only?
When I first installed 1.6.3 it manifested 3 times in about a day, then it didn’t happen again until you asked me to change logging parameters, almost a week later, once I restarted, it crashed again within hours. Doesn’t seem temporal but I don’t know.
It is not there.
By the way, I occasionally see file not found/doesnt exist errors in the log, it has been happening for a long time, and I do not think there is any data loss, all the audits are at 100% and it’s running on ZFS. Not sure if this is some artefact in way system works or a bug somewhere.
There is always a performance impact. The more you log the more computation that machine needs. Also logging the stack and caller requires more resources.
It may not impact that much in your machine, but it has more impact in low power machines.
Thanks for your replies.
I’ve found a place where we should skip that error and not interrupt the execution because it could happen that the file gets deleted before we call the Lstat function.
The changes are in review considering that today’s is an official holidays, so I expect that it will be reviewed next week and perhaps some back and forth with them.
Let’s see if my mates see that change with good eyes and it get merged and rolled out next week.
On the other hand, other errors reported in this post were pointing to another problem that your one don’t have; both are related and caused the same crash, but I couldn’t get root cause of that one because your’s didn’t show it.
I see, thanks for your reply!
The other one is the createFile error? I will keep an eye out for that one and report if it’s different from lstat one already reported here then.
I will leave the extra logging on, it doesn’t bother me, in case I run into any bottlenecks I can always shut it off. The only other problem at the moment is big overstep of allotted space, mentioned in other thread, tell me if there’s anything I can do to help debug that.
Any purposed fix? I’m crashing on Windows GUI Server 2019 every 12 hours or so now. Per the logs it looks like Piece Deleter removed the blob a few hours before the error.
2020-06-22T00:06:55.779-0400 FATAL Unrecoverable error {“error”: “CreateFile G:\Storj Node\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\6g/itvzk3s6o5csohh5shpp3hcvsi6dstfhbdeqmyv5yllnaesfua.sj1: The system cannot find the file specified.”}
2020-06-21T22:49:59.016-0400 INFO piecedeleter deleted {“Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Piece ID”: “6GITVZK3S6O5CSOHH5SHPP3HCVSI6DSTFHBDEQMYV5YLLNAESFUA”}
@llamamanc We were promised a fix is coming soon! Oh it looks like you have the second type of error. Please turn all logging on on your SN and post the fatal log message when it happens again.
Changes applied, I’ll let you know what I find. I noticed another error just before the fatal error about current space used calculation.
2020-06-22T00:06:55.651-0400 ERROR piecestore:cache error getting current space used calculation: {“error”: “CreateFile G:\Storj Node\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\6g/itvzk3s6o5csohh5shpp3hcvsi6dstfhbdeqmyv5yllnaesfua.sj1: The system cannot find the file specified.”}
2020-06-22T00:06:55.659-0400 INFO piecestore upload canceled {“Piece ID”: “EUCMFHPAHP2264LOVV3O6HD5RDRCZEGTT5RXWUBTCAMDGTBTLZ6A”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT_REPAIR”}
2020-06-22T00:06:55.664-0400 INFO piecestore upload canceled {“Piece ID”: “B6EOW5DF7CNKWTBG7UCOWW6O4NZIB6JXBDM6LLENUH6AFDMXEIFA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”}
2020-06-22T00:06:55.779-0400 FATAL Unrecoverable error {“error”: “CreateFile G:\Storj Node\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\6g/itvzk3s6o5csohh5shpp3hcvsi6dstfhbdeqmyv5yllnaesfua.sj1: The system cannot find the file specified.”}
Right, that’s it, there’s an error message followed by a fatal message for the same thing.
Is there any ETA for this to continue release (of the next version that fixes the issues, I suppose), or any place we can see the status? I could not find any relevant PR/issue on GitHub 
I suppose it is not easy to give a date, I just wanted to know if you think it would be a matter of days or weeks - I have a Windows node waiting for update to be on the same page as linux to migrate it 
Storj core devs are doing an amazing task building this platform - my intention is not to make pressure on this at all <3