Fatal message on SN

Storgeez · June 11, 2020, 7:54pm

Got a fatal log message on my node, it also terminated:

2020-06-11T21:27:04.763+0200 FATAL Unrecoverable error {“error”: “lstat /Storj/Storj-v3-data/storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/7o/ejeq4pyvvikje2o7nexhhxwt47gi5ij6dvc76w5y6eu57qop2a.sj1: no such file or directory; lstat /Storj/Storj-v3-data/storage/blobs/qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa/c2/g3n2rbju2b5ukbq7y5l54xj2bdip3znzk45lm4aiepamg3mcwa.sj1: no such file or directory”, “errorVerbose”: “group:\n— lstat /Storj/Storj-v3-data/storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/7o/ejeq4pyvvikje2o7nexhhxwt47gi5ij6dvc76w5y6eu57qop2a.sj1: no such file or directory\n— lstat /Storj/Storj-v3-data/storage/blobs/qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa/c2/g3n2rbju2b5ukbq7y5l54xj2bdip3znzk45lm4aiepamg3mcwa.sj1: no such file or directory”}

And before that I see:

2020-06-11T21:27:03.932+0200 ERROR piecestore:cache error getting current space used calculation: {“error”: “lstat /Storj/Storj-v3-data/storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/7o/ejeq4pyvvikje2o7nexhhxwt47gi5ij6dvc76w5y6eu57qop2a.sj1: no such file or directory; lstat /Storj/Storj-v3-data/storage/blobs/qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa/c2/g3n2rbju2b5ukbq7y5l54xj2bdip3znzk45lm4aiepamg3mcwa.sj1: no such file or directory”, “errorVerbose”: “group:\n— lstat /Storj/Storj-v3-data/storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/7o/ejeq4pyvvikje2o7nexhhxwt47gi5ij6dvc76w5y6eu57qop2a.sj1: no such file or directory\n— lstat /Storj/Storj-v3-data/storage/blobs/qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa/c2/g3n2rbju2b5ukbq7y5l54xj2bdip3znzk45lm4aiepamg3mcwa.sj1: no such file or directory”}

It tried to do the same action on same files 2x4 times from what I understand and obviously got a consistent result - looks like it didn’t like that.

I started it again, it seems to be working normally.
What is this?

Vadim · June 11, 2020, 8:07pm

I have same eerors ocured discribed here.

Storgeez · June 11, 2020, 8:22pm

From my uneducated look at the message it looks as if not finding a file is a reason to crash.
It is pretty much impossible that something is corrupted from filesystem side as it’s running on a mirror, but I’m running a check anyway. It is possible that the file is missing/shouldn’t be there. The main problem is the crash though.

endurance · June 11, 2020, 9:03pm

I see this now also from time to time on one of my nodes. This issue started with using 1.63 windows version - I never had this before with older versions. All databases are correct (checked). But the file mentioned is really not there - question is why this is happening?
Ah I am using a raid 50 and 60 so plenty of redundancy and error correction - file system etc. is def. OK.

Vadim · June 11, 2020, 9:13pm

Also started from 1.6.3

naxbc · June 11, 2020, 9:38pm

Same here. Rebooted windows and worked normally again.

Storgeez · June 12, 2020, 12:17am

Again. Hopefully a fix is released soon.

Vadim · June 12, 2020, 7:51am

2020-06-12T03:55:34.008+0300 ERROR piecestore:cache error getting current space used calculation: {“error”: “CreateFile h:\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\cp/6j47pubxb6lbrryndy4dbl6xoypfxdgh7ch557im2j26ppmgba.sj1: The system cannot find the file specified.”}
2020-06-12T03:55:34.008+0300 ERROR piecestore:cache error getting current space used calculation: {“error”: “CreateFile h:\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\cp/6j47pubxb6lbrryndy4dbl6xoypfxdgh7ch557im2j26ppmgba.sj1: The system cannot find the file specified.”}
2020-06-12T03:55:34.008+0300 ERROR piecestore:cache error getting current space used calculation: {“error”: “CreateFile h:\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\cp/6j47pubxb6lbrryndy4dbl6xoypfxdgh7ch557im2j26ppmgba.sj1: The system cannot find the file specified.”}
2020-06-12T03:55:34.127+0300 FATAL Unrecoverable error {“error”: “CreateFile h:\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\cp/6j47pubxb6lbrryndy4dbl6xoypfxdgh7ch557im2j26ppmgba.sj1: The system cannot find the file specified.”}
2020-06-12T03:55:34.127+0300 FATAL Unrecoverable error {“error”: “CreateFile h:\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\cp/6j47pubxb6lbrryndy4dbl6xoypfxdgh7ch557im2j26ppmgba.sj1: The system cannot find the file specified.”}

2020-06-11T03:56:12.993+0300 ERROR piecestore:cache error getting current space used calculation: {“error”: “CreateFile h:\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\6q/uyzzwobl3rzewr5fgcih7tsz37lvutpqvha3ikexz7ppyetgea.sj1: The system cannot find the file specified.”}
2020-06-11T03:56:12.993+0300 ERROR piecestore:cache error getting current space used calculation: {“error”: “CreateFile h:\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\6q/uyzzwobl3rzewr5fgcih7tsz37lvutpqvha3ikexz7ppyetgea.sj1: The system cannot find the file specified.”}
2020-06-11T03:56:12.993+0300 ERROR piecestore:cache error getting current space used calculation: {“error”: “CreateFile h:\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\6q/uyzzwobl3rzewr5fgcih7tsz37lvutpqvha3ikexz7ppyetgea.sj1: The system cannot find the file specified.”}
2020-06-11T03:56:13.129+0300 FATAL Unrecoverable error {“error”: “CreateFile h:\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\6q/uyzzwobl3rzewr5fgcih7tsz37lvutpqvha3ikexz7ppyetgea.sj1: The system cannot find the file specified.”}
2020-06-11T03:56:13.129+0300 FATAL Unrecoverable error {“error”: “CreateFile h:\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\6q/uyzzwobl3rzewr5fgcih7tsz37lvutpqvha3ikexz7ppyetgea.sj1: The system cannot find the file specified.”}

2020-06-10T00:03:52.102+0300 ERROR piecestore:cache error getting current space used calculation: {“error”: “CreateFile h:\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\z6/hqtxm3fimqppsyv26kpngd7al74yjhwp4mysdjcmfuc6zhwzaa.sj1: The system cannot find the file specified.”}
2020-06-10T00:03:52.102+0300 ERROR piecestore:cache error getting current space used calculation: {“error”: “CreateFile h:\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\z6/hqtxm3fimqppsyv26kpngd7al74yjhwp4mysdjcmfuc6zhwzaa.sj1: The system cannot find the file specified.”}
2020-06-10T00:03:52.102+0300 ERROR piecestore:cache error getting current space used calculation: {“error”: “CreateFile h:\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\z6/hqtxm3fimqppsyv26kpngd7al74yjhwp4mysdjcmfuc6zhwzaa.sj1: The system cannot find the file specified.”}
2020-06-10T00:03:52.217+0300 FATAL Unrecoverable error {“error”: “CreateFile h:\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\z6/hqtxm3fimqppsyv26kpngd7al74yjhwp4mysdjcmfuc6zhwzaa.sj1: The system cannot find the file specified.”}
2020-06-10T00:03:52.217+0300 FATAL Unrecoverable error {“error”: “CreateFile h:\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\z6/hqtxm3fimqppsyv26kpngd7al74yjhwp4mysdjcmfuc6zhwzaa.sj1: The system cannot find the file specified.”}

for some unknown reason it hapening at same time. @littleskunk can it be investigated?
it started from version 1.6.3

ifraixedes · June 12, 2020, 10:34am

Thank you all for reporting it.

We are stopping the rollout of v1.6.3 and escalate the issue internally to investigate the error.

Thank you for patience.

Storgeez · June 12, 2020, 1:56pm

Great, it crashed again while I was sleeping. 9h of downtime.

Vadim · June 12, 2020, 2:29pm

does it crash at one specific time all the time?

Storgeez · June 12, 2020, 2:32pm

Not, not any that I can correlate…

endurance · June 12, 2020, 4:01pm

Yepp absolutely random. I changed the windows service to restart automatically after 10min as workaround.

Storgeez · June 12, 2020, 4:29pm

If only this thing was a service. Perhaps it can be done…

monty · June 12, 2020, 4:29pm

We’re looking into resolving this issue right now.
Feel free to contact me today or Monday if you have any further related issues!

Vadim · June 12, 2020, 4:34pm

on windows GUI storagenode is service

naxbc · June 13, 2020, 12:56am

I have both services restarting automatically, but that won’t solve it. Only after a reboot it recovers. Happened on my 3 Windows nodes, but not on the pi and also not on my 2 Synologys.

Storgeez · June 13, 2020, 1:24am

Weird, on FreeBSD it just closes, I just have to open it again. Having to reboot is a bigger hassle.

endurance · June 13, 2020, 6:11pm

How fast do you restart? try waiting10min.
First try was also immediate restart - but then it just runs into the same issue as you describe. WIth a pause it works on my nodes.

Storgeez · June 13, 2020, 6:34pm

I think the more you pause the less load you will have from satellites, might be related. I’ve had to re-start it twice so far.