Power outage wiped my node config, then more woes came

Node Operators troubleshooting
1

Hey all, Windows GUI node operator on NoIP ddns here with some recent issues I can’t seem to wrap my head around.

A power outage recently knocked out my desktop and the node along with it. When I first powered the computer back on and tried to get the node dashboard to pull up, I received the message “localhost refused to connect.” Trying to start the node from elevated PowerShell did nothing and did not return any error messages.

After doing some digging, I was able to figure out that while the config.yaml file in C:\Storj\ was fine and had my right info, a second copy of it in C:\Users(My account)\AppData\Roaming\Storj\StorageNode was completely wiped. I replaced the config file inside the AppData folder and my Storj node got back up and running, temporarily. Strangely, I had to use Command Prompt instead of PowerShell to get it running.

Fast forward to today, and I came in to see that my node dashboard was on a continuous looping load. I checked in Task Manager and the node was eating all of my memory, as in 96%. I went to check my external hard drive I used for Storj and it appeared as though the hard drive malfunctioned and its data couldn’t be accessed. Windows Explorer froze when I tried to click into it.

I had to perform a hard reset on the desktop, and when I got things running again, the “localhost refused to connect” message returned. My hard drive is now accessible again though, and even stranger is that my config.yaml files are identical in both locations now.

I was looking for a time to try and expand the amount of storage my node was using, and changed both config files to 3.00 TB instead of 2.50 TB. Changing the storage in config back to 2.50 TB did nothing and localhost still refuses to connect.

Nothing appears to get this node back online. I tried restarting from elevated PowerShell, nothing happened. I tried running storagenode.exe commands in admin Command Prompt, and config, dashboard, run and setup do nothing. Diag does give results from 6 satellites.

What should I do at this point?

2

I Would check your logs to see if it will give you any info of what is going on. Espically knowing your using an external usb drive a power outage can and will corrupts data, And or change the Drive letter.

3

The standard path to the config.yaml for the Windows GUI node is "C:\Program Files\Storj\Storage Node\config.yaml", but if you installed the node to the manually changed path, it should place config.yaml to the installation dir (C:\Storj I suppose).

However, if you run setup from the command line, the default path for the config.yaml is %APPDATA%\Storj\Storagenode\config.yaml (seems you did that), and it doesn’t exist if you used a GUI installer.

Please, make sure that the drive letter is updated to the current one. As @deathlessdd said, Windows could change the drive letter when something is goes wrong with disk (as in your case).
Please, show result of the command (PowerShell):

(Get-WmiObject Win32_Service -Filter "name = 'storagenode'").PathName

And

Get-PSDrive

Then

$path = (Get-WmiObject Win32_Service -Filter "name = 'storagenode'" | Select-Object PathName | sls '--config-dir "(.*)\\').Matches.Groups[1].Value;
sls "storage.path" ${path}config.yaml;
sls "identity" ${path}config.yaml;

Then

$logFile = (sls "log.output: winfile:///(.*)" ${path}config.yaml).Matches.Groups[1].Value

and finally

Get-Content -Tail 20 $logFile

Please, place the returned text between two new lines with three backticks in the answer, like this:

```
put text here
```
4

@Alexey and @deathlessdd - Thank you both for the assistance. I did a little more tinkering and somehow got the node back up and running, though I’m not really sure how. The dashboard pulls up now and I’m showing all of my previous satellites online.

I think when I installed the node, it was via the GUI method for Windows. I believe I changed the install directory away from Program FIles to avoid administrative access issues to just C:\Storj. I’m not really sure how the APPDATA directory came to exist, but I’m wondering if my using the setup command while trying to get the node back up and running in Command Prompt created it.

I’m going to follow up when I get home and go through the steps Alexey provided to confirm the drive letter is correct, and will post the results afterwards. Again, thank you both for replying to help!

(P.S. I have my computer and the external hard drive’s power connected to a UPS, does that help protect against external drive data corruption from outages?)

1 Like
5

Yes, it is, if you did not specify the --config-dir with storagenode setup, see storagenode setup --help | findstr /c:"--config-dir"

Only if it’s managed (has the cable connected to the PC and you have an application to shutdown the PC in case of power outage). Non-managed UPS is useful too, if the power outage is not longer than a few minutes (maybe longer depends on UPS capacity).

6

@Alexey and @deathlessdd - New development in this case, from a different kind of outage it seems.

So I was able to get things back up and running without issue for another 2 months, and then apparently on 10/11/2021 and 10/15/2021 something went wrong. I was hit with an internet outage yesterday so I went to check on my node, which was showing as Offline. For some reason, I didn’t receive a suspension email until 10 days later, today, but now I’ve gotten the email notice that I’ve been suspended on three different satellites. Checking on my node, it’s saying I was suspended on 5 nodes on 10/11, and then the last one on 10/15. My online score for all 6 of them is at 30% or below, so this can’t have happened overnight.

I’m trying to access my log file to see what went wrong so I can fix, but the text file is 7 gigabytes large and Notepad is saying it’s too big to open. Is there another editor that would work instead?

Also worth noting, after restarting my computer the node is now showing as Online again. Would this be it running in suspension mode as described here?

7

Created an issue [storagenode] The email about suspension is coming only after check-in to satellite · Issue #4235 · storj/storj · GitHub

You can use PowerShell or cmd and CLI commands to search for needed strings.
For PowerShell:

sls "ERROR" "C:\Program Files\Storj\Storage Node\storagenode.log" | select -last 10

For cmd.exe (but there is no analogue of tail (bash) or select -last 10 (PowerShell))

findstr /c:"ERROR" "C:\Program Files\Storj\Storage Node\storagenode.log"

Or use WSL Ubuntu and use Linux commands from there. All disks will be automatically mounted under /mnt
Another way is to install msys2 or cygwin to use Linux commands too.

See also

However, I would recommend to stop the service, archive the log and delete the source log, then start the service. This way you would not lost the previous logs and can keep it small.
You can also use a logrotate for Windows or some scripting like GitHub - AlexeyALeonov/storagenode_win_log_rotate: Windows GUI storagenode log rotation script (doesn't compatible with logrotate) (you need to modify the script for your case) and run it as an Administrator (requires rights to start/stop the service).

8

@Alexey thank you for all of your help on this. Here is what I was able to gather from the logs.

It looks as if there was a problem that surfaced on 9/28, then a fatal error on 9/29, and then the logs jump to 10/20 for some reason. I am pasting the noteworthy portions of the logs here. The node operated normally with uploads and downloads on 9/28 for several hours, and 9/29 as well I think.

From what I can tell (which isn’t much) it looks as if initially there was some error with my ddns setup on 9/28?

9/28

2021-09-28T12:18:50.857-0500	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "Error": "contact: failed to dial storage node (ID: 12LuCEGg5zPEtkntnftNfRRbcBNmjvJw6iK51hV2bTozPNxMwzE) at address markfeuerborn.ddns.net:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-09-28T12:18:50.860-0500	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 12LuCEGg5zPEtkntnftNfRRbcBNmjvJw6iK51hV2bTozPNxMwzE) at address markfeuerborn.ddns.net:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-09-28T12:18:51.327-0500	INFO	piecedeleter	delete piece sent to trash	{"Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "NAV2AYGMI3MD2AIZD5W7XQ6U3BINZIFV42M7PJAEJOFCTFW4LH2A"}
2021-09-28T12:18:51.493-0500	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Error": "contact: failed to dial storage node (ID: 12LuCEGg5zPEtkntnftNfRRbcBNmjvJw6iK51hV2bTozPNxMwzE) at address markfeuerborn.ddns.net:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-09-28T12:18:51.592-0500	INFO	piecedeleter	delete piece sent to trash	{"Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "3LVNVJTL2IXRSGO6IWXQ7QGQLAQWQ5BFHJCYLAEADQCCWJTOILHQ"}
2021-09-28T12:18:51.630-0500	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Error": "contact: failed to dial storage node (ID: 12LuCEGg5zPEtkntnftNfRRbcBNmjvJw6iK51hV2bTozPNxMwzE) at address markfeuerborn.ddns.net:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-09-28T12:18:51.633-0500	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Error": "contact: failed to dial storage node (ID: 12LuCEGg5zPEtkntnftNfRRbcBNmjvJw6iK51hV2bTozPNxMwzE) at address markfeuerborn.ddns.net:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-09-28T12:18:51.783-0500	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Error": "contact: failed to dial storage node (ID: 12LuCEGg5zPEtkntnftNfRRbcBNmjvJw6iK51hV2bTozPNxMwzE) at address markfeuerborn.ddns.net:28967 using QUIC: rpc: quic: timeout: no recent network activity"}

9/29

2021-09-29T00:23:12.698-0500	ERROR	services	unexpected shutdown of a runner	{"name": "trust", "error": "satellitesdb: database is locked", "errorVerbose": "satellitesdb: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*satellitesDB).SetAddress:40\n\tstorj.io/storj/storagenode/trust.(*Pool).Run:124\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2021-09-29T00:23:12.741-0500	INFO	piecestore	upload started	{"Piece ID": "WGSUL3YGZYB6DE5PLKK2ZH455CNE3MTNLOHQWZYZNICPACCG3KDA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 30959918592}
2021-09-29T00:23:12.742-0500	INFO	piecestore	downloaded	{"Piece ID": "DOLQUYJRWCR6L7ZR7NG37GDXLXJXUKQMVWG5P6X7VCRYIEB6IQBA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-09-29T00:23:12.742-0500	INFO	piecestore	downloaded	{"Piece ID": "TJAWCUJVYKUNJI65Y7C5ONTZXVFOQKDNQMILA6M3CUOZS6TYTKUQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-09-29T00:23:12.786-0500	INFO	piecestore	downloaded	{"Piece ID": "76FD2HJUFRQVTIPUAPQ43XE4Y6FSFCR2LMU2HSPCCARVUQYOFNCA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-09-29T00:23:12.868-0500	INFO	piecestore	downloaded	{"Piece ID": "JCPW7PBHJYXWQQHH22HMYSSVYOUEHS4LUGORG77XCZBOVMIP2S4A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-09-29T00:23:13.345-0500	ERROR	piecestore	error sending hash and order limit	{"error": "context canceled"}
2021-09-29T00:23:13.376-0500	ERROR	gracefulexit:chore	error retrieving satellites.	{"error": "satellitesdb: context canceled", "errorVerbose": "satellitesdb: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*satellitesDB).ListGracefulExits:149\n\tstorj.io/storj/storagenode/gracefulexit.(*service).ListPendingExits:89\n\tstorj.io/storj/storagenode/gracefulexit.(*Chore).Run.func1:53\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/storj/storagenode/gracefulexit.(*Chore).Run:50\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2021-09-29T00:23:13.396-0500	INFO	piecestore	download canceled	{"Piece ID": "UU5OKUK5LR5V4SNFHEO3VOMWV6DZ5C6O5P722EOXZPGCEPYC6ZZQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_REPAIR"}
2021-09-29T00:23:13.445-0500	ERROR	piecestore	error sending hash and order limit	{"error": "context canceled"}
2021-09-29T00:23:13.476-0500	INFO	piecestore	downloaded	{"Piece ID": "3YH3LCVOTAM2LSYKQLI7CG3TEUJKHZUYBXAGERUQTIGXEERRNXWA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-09-29T00:23:13.479-0500	INFO	piecestore	download canceled	{"Piece ID": "DQ7Z2S6SMJWQ2MXTNLPFKUD7HJ4CZBIHPF4V767GNLQIUDXFDIOQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "GET_REPAIR"}
2021-09-29T00:23:13.525-0500	WARN	retain	failed to delete piece	{"Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "52SNOQWJAY52HGZDTKRKJSN3KO7FB4AHA5V7UMXBG4WKPT56JJCQ", "error": "pieces error: pieceexpirationdb: context canceled", "errorVerbose": "pieces error: pieceexpirationdb: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceExpirationDB).Trash:112\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:354\n\tstorj.io/storj/storagenode/retain.(*Service).trash:441\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces.func1:406\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces.func1:509\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:797\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:284\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces:375\n\tstorj.io/storj/storagenode/retain.(*Service).Run.func2:222\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2021-09-29T00:23:13.544-0500	INFO	piecestore	downloaded	{"Piece ID": "6KFEUPMPDOEOLU77GWP6UE24AH6DPBFCRZMANMH2YA5E2FCLRVPQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "GET_REPAIR"}
2021-09-29T00:23:13.600-0500	INFO	piecestore	downloaded	{"Piece ID": "BET6LM2SOFRXPOVJQM3R2O32RDKDBV4US7YMU6M7KYJYIK7IBUFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-09-29T00:23:13.603-0500	ERROR	retain	retain pieces failed	{"error": "retain: context canceled", "errorVerbose": "retain: context canceled\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces:426\n\tstorj.io/storj/storagenode/retain.(*Service).Run.func2:222\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2021-09-29T00:23:13.691-0500	ERROR	piecedeleter	could not send delete piece to trash	{"Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "Z6U4GPQEZK6EQHMF7P5EXSGKD5MZXHCIP7BIXSY5EYFWMEG5T4BQ", "error": "pieces error: pieceexpirationdb: context canceled", "errorVerbose": "pieces error: pieceexpirationdb: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceExpirationDB).Trash:112\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:354\n\tstorj.io/storj/storagenode/pieces.(*Deleter).deleteOrTrash:185\n\tstorj.io/storj/storagenode/pieces.(*Deleter).work:135\n\tstorj.io/storj/storagenode/pieces.(*Deleter).Run.func1:72\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2021-09-29T00:23:13.850-0500	ERROR	piecedeleter	could not send delete piece to trash	{"Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "SN7BSM2P43COA5W6XCBQ4H6FD7M5NOT33AL5YJSWWEL4C5XX66TA", "error": "pieces error: pieceexpirationdb: context canceled", "errorVerbose": "pieces error: pieceexpirationdb: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceExpirationDB).Trash:112\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:354\n\tstorj.io/storj/storagenode/pieces.(*Deleter).deleteOrTrash:185\n\tstorj.io/storj/storagenode/pieces.(*Deleter).work:135\n\tstorj.io/storj/storagenode/pieces.(*Deleter).Run.func1:72\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2021-09-29T00:23:14.267-0500	INFO	piecestore	upload canceled	{"Piece ID": "37XANSICRI4FVIDWTYVKHUZL7S6G2RIQPTLBX3PLKTR3TEQ3AX5A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 0}
2021-09-29T00:23:14.267-0500	INFO	piecestore	upload canceled	{"Piece ID": "UUIUS5T3LAQIVVDRO6JLA4V2MSSC2GCZBO2QQIVTRQK4C5KABF7A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 0}
2021-09-29T00:23:14.267-0500	INFO	piecestore	upload canceled	{"Piece ID": "DKX5CUPIQKTSX63M7UAWTDGTGFBN74YQ2FE6LFDV7M7ZOUA72FBQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT_REPAIR", "Size": 0}
2021-09-29T00:23:14.267-0500	INFO	piecestore	upload canceled	{"Piece ID": "RQ4F5HKEDLGSEXXTNHGLSGGQM4BZTZCO6AJAELZ2UR4IMFON4IMQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 0}
2021-09-29T00:23:14.267-0500	INFO	piecestore	upload canceled	{"Piece ID": "M6TPCFYYLCRKMUSEZHEULPAB62EU3XNOFVRKWAYLMVZMMUDJU4DQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 0}
2021-09-29T00:23:14.267-0500	INFO	piecestore	upload canceled	{"Piece ID": "62NQXA2MISHP6WX7L3C2O7OGY6SMVMWAU5C7TZATOYMGX3OKHBMQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 0}
2021-09-29T00:23:14.267-0500	INFO	piecestore	upload canceled	{"Piece ID": "PQFSGAW2OH7ELFWPLFEDOF7WSCDOKFK2MEMUHOHIZYEIQKVDZTXQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 0}
2021-09-29T00:23:14.268-0500	INFO	piecestore	upload canceled	{"Piece ID": "RIJF5VKUAU5FJ52BFY6MY42GSXUFYNSYJZAKC2ATT4SCBQ4MOMNA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 0}
2021-09-29T00:23:14.268-0500	INFO	piecestore	upload canceled	{"Piece ID": "IU5Y7GZJHAXQFZU5E4APNPFL5L5OZANH5F7DJVTZ2GO3XRPKD4FQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 0}
2021-09-29T00:23:14.268-0500	INFO	piecestore	upload canceled	{"Piece ID": "PY3I32I44YYHQAG5UUXS7CI3B6XM3CHJZUXZR6W6A4R57KXBR67A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 0}
2021-09-29T00:23:14.268-0500	INFO	piecestore	upload canceled	{"Piece ID": "37ZXKZURBGYQTOYWCRZKBPRE6423MPCFGVR5R4ZRBOTVL67TKKFA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Size": 0}
2021-09-29T00:23:14.268-0500	INFO	piecestore	upload canceled	{"Piece ID": "5WSSSH5SGJ6BV4ALCA7A62WYDEFINQ5VMDYY4UWFXUET4LNRC6NQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 0}
2021-09-29T00:23:14.268-0500	INFO	piecestore	upload canceled	{"Piece ID": "PS7FUAWVGA642B6KP7EA4GB4ALRSOMQCHVERK7YQOUYTMQX4QHMQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 0}
2021-09-29T00:23:14.268-0500	INFO	piecestore	upload canceled	{"Piece ID": "E5GR3PJ2CLGFQD6LIKYK2XRQTHUJQFMTY4KT4ONO7K6Z5KQQU6XQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 0}
2021-09-29T00:23:14.268-0500	INFO	piecestore	upload canceled	{"Piece ID": "A5AVBNV2O6TTQ3JO47DFLPFLK7MU353IL2UQGBA2LONNQBISQSYQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 0}
2021-09-29T00:23:14.269-0500	INFO	piecestore	upload canceled	{"Piece ID": "TFIFV5AI5WLC4HU4FBB3LHQ4333URGSWYRJPNRXZ262F25LYGS4Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 0}
2021-09-29T00:23:14.269-0500	INFO	piecestore	upload canceled	{"Piece ID": "HOJQQGWVHIRAUBOV62OGSPHWZSTE766FUHJWXFUDKGMUEW4DR3SQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 0}
2021-09-29T00:23:14.271-0500	INFO	piecestore	upload canceled	{"Piece ID": "GVLHVSTVPV3DOGWUZNUMGR6HBVLGFM426NFGLFZ7GGUYA5CKD6JQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 0}
2021-09-29T00:23:14.331-0500	INFO	piecestore	uploaded	{"Piece ID": "UW4FNJAVKCHDDKH4D36P3PEJCVXC7B2OA3RCUT4OPEFFU2GLQXLA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 5632}
2021-09-29T00:23:14.381-0500	ERROR	retain	retain pieces failed	{"error": "retain: context canceled", "errorVerbose": "retain: context canceled\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces:426\n\tstorj.io/storj/storagenode/retain.(*Service).Run.func2:222\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2021-09-29T00:23:14.428-0500	INFO	piecestore	uploaded	{"Piece ID": "2D4WLD3JL2WW2VP47RCEFWZMXPYXWRBIJBLPCALJ2XOWT4KRB5SQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 3584}
2021-09-29T00:23:14.457-0500	INFO	piecestore	uploaded	{"Piece ID": "GBY4M7D4M2T4RK2CRHIZQKVYTS63SSBYTCNT6KP7SPOOJONARKLA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 2560}

2021-09-29T00:23:16.443-0500	FATAL	Unrecoverable error	{"error": "satellitesdb: database is locked", "errorVerbose": "satellitesdb: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*satellitesDB).SetAddress:40\n\tstorj.io/storj/storagenode/trust.(*Pool).Run:124\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2021-10-20T18:10:08.454-0500	INFO	Configuration loaded	{"Location": "C:\\Users\\Greg\\AppData\\Roaming\\Storj\\Storagenode\\config.yaml"}

10/20

2021-10-20T18:10:12.530-0500	INFO	Operator email	{"Address": "XXXX"}
2021-10-20T18:10:12.530-0500	INFO	Operator wallet	{"Address": "XXXX"}
2021-10-20T18:10:32.642-0500	INFO	Telemetry enabled	{"instance ID": "12LuCEGg5zPEtkntnftNfRRbcBNmjvJw6iK51hV2bTozPNxMwzE"}
2021-10-20T18:10:33.271-0500	INFO	db.migration	Database Version	{"version": 53}
2021-10-20T18:10:37.989-0500	INFO	preflight:localtime	start checking local system clock with trusted satellites' system clock.
2021-10-20T18:10:39.040-0500	INFO	preflight:localtime	local system clock is in sync with trusted satellites' system clock.
2021-10-20T18:10:39.385-0500	INFO	Node 12LuCEGg5zPEtkntnftNfRRbcBNmjvJw6iK51hV2bTozPNxMwzE started
2021-10-20T18:10:39.385-0500	INFO	Public server started on [::]:28967
2021-10-20T18:10:39.385-0500	INFO	Private server started on 127.0.0.1:7778
2021-10-20T18:10:39.707-0500	INFO	bandwidth	Performing bandwidth usage rollups
2021-10-20T18:10:39.722-0500	INFO	trust	Scheduling next refresh	{"after": "7h17m11.580878414s"}
2021-10-20T18:10:48.895-0500	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 12LuCEGg5zPEtkntnftNfRRbcBNmjvJw6iK51hV2bTozPNxMwzE) at address markfeuerborn.ddns.net:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-10-20T18:10:48.918-0500	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "Error": "contact: failed to dial storage node (ID: 12LuCEGg5zPEtkntnftNfRRbcBNmjvJw6iK51hV2bTozPNxMwzE) at address markfeuerborn.ddns.net:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-10-20T18:10:49.082-0500	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Error": "contact: failed to dial storage node (ID: 12LuCEGg5zPEtkntnftNfRRbcBNmjvJw6iK51hV2bTozPNxMwzE) at address markfeuerborn.ddns.net:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-10-20T18:10:49.918-0500	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Error": "contact: failed to dial storage node (ID: 12LuCEGg5zPEtkntnftNfRRbcBNmjvJw6iK51hV2bTozPNxMwzE) at address markfeuerborn.ddns.net:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-10-20T18:10:50.926-0500	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Error": "contact: failed to dial storage node (ID: 12LuCEGg5zPEtkntnftNfRRbcBNmjvJw6iK51hV2bTozPNxMwzE) at address markfeuerborn.ddns.net:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-10-20T18:10:51.358-0500	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Error": "contact: failed to dial storage node (ID: 12LuCEGg5zPEtkntnftNfRRbcBNmjvJw6iK51hV2bTozPNxMwzE) at address markfeuerborn.ddns.net:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-10-20T18:10:51.952-0500	INFO	orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6	sending	{"count": 34}
2021-10-20T18:10:51.955-0500	INFO	orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S	sending	{"count": 5446}
2021-10-20T18:10:51.969-0500	INFO	orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs	sending	{"count": 561}
2021-10-20T18:10:51.999-0500	INFO	orders.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE	sending	{"count": 62}
2021-10-20T18:10:52.005-0500	INFO	orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB	sending	{"count": 121}
2021-10-20T18:10:52.010-0500	INFO	orders.12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo	sending	{"count": 3}
2021-10-20T18:10:52.621-0500	INFO	orders.12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo	finished
2021-10-20T18:10:52.787-0500	INFO	orders.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE	finished
2021-10-20T18:10:53.146-0500	INFO	orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6	finished
2021-10-20T18:10:53.158-0500	INFO	orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB	finished
2021-10-20T18:10:53.900-0500	INFO	orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S	finished
2021-10-20T18:10:53.984-0500	INFO	orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs	finished
2021-10-20T18:10:55.076-0500	INFO	collector	delete expired	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "C2GY3JXHG6GBCH26EDST2QCGSUU76WMKPY5Z3MLEB3LZ7OH2ANBA"}
2021-10-20T18:10:55.252-0500	INFO	orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S	sending	{"count": 1877}
2021-10-20T18:10:55.252-0500	INFO	orders.12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo	sending	{"count": 2}
2021-10-20T18:10:55.260-0500	INFO	orders.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE	sending	{"count": 12}
2021-10-20T18:10:55.260-0500	INFO	orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs	sending	{"count": 164}
2021-10-20T18:10:55.292-0500	INFO	orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB	sending	{"count": 47}
2021-10-20T18:10:55.300-0500	INFO	orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6	sending	{"count": 33}
2021-10-20T18:10:55.505-0500	INFO	orders.12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo	finished
2021-10-20T18:10:55.545-0500	INFO	orders.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE	finished
2021-10-20T18:10:55.705-0500	INFO	collector	delete expired	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "VGWFVCJJDXLILHRNQYUHAYRY3ZWDR5MRVLP4EHDUD5PYU3O6CCSA"}
2021-10-20T18:10:55.880-0500	INFO	orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S	finished
2021-10-20T18:10:56.003-0500	INFO	orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB	finished
2021-10-20T18:10:56.091-0500	INFO	orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs	finished
2021-10-20T18:10:56.267-0500	INFO	collector	delete expired	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "U4LZ3K3L6HHK2L6C6VU47PSIJRGBB733OGW6HNWMX2ECGGZEO4IQ"}
2021-10-20T18:10:56.275-0500	INFO	orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6	finished
9

Unlikely. If you have had problems with DDNS, you will get “ping satellite failed” errors with your NodeID and your current public IP/DDNS with explanation why is it failed.

is mean that you need to forward UDP port too, and allow it in the firewall

The FATAL error is the reason for node restart, as result your dashboard could not respond too - it requires an access to databases, which was locked.

The database could be locked because your disk become slow and cannot keep up the load. The restart usually helps (and the node did it itself in your case).
I would like to suggest to check your disk on errors with chkdsk /f when the node is stopped.

10

@Alexey thank you for the insight on this. I am running chkdsk right now on that hard drive, and just ensured my router is allowing both TCP and UDP protocol. For Windows Firewall settings, is there a way to ensure on there that it allows both TCP and UDP simultaneously, or should I just create a second inbound rule for UDP that matches the TCP properties?

11

Yes, this is the only way to do so.

12

Thanks for all of your help @Alexey - After adding the UDP rule and running chkdsk, I restarted my node and it appears to be running stable now. My online score for all the satellites is still dangerously low at 19%, but it appears to be going back up since that is higher than it was! Hoping that it will shoot back up quickly for the next audit.

1 Like