Hi All,
I’m trying to migrate from Ubuntu Docker to Windows GUI, I have followed the guide and copied over the identity and storage folders / data and ran the install and pointed to the correct locations - all looks good. But the storage node service keeps stopping and i’m unable to get to the dashboard, did i do something wrong ?
How best to diagnose this issue?
Thanks !
Hello @brad-pea,
Welcome to the forum!
Please, show 20 lines from the log: https://documentation.storj.io/resources/faq/check-logs
Thanks !
So I did some digging around whilst waiting for the reply and if i point it to another folder on the same drive (though i am 99% sure the docker was not using that folder) the service will remain up, but i still cannot access the dashboard - it does not load …
Heres the 20 lines from the folder i believe to be correct … but I see there is a storage folder within it so this is outside of the storage folder…
2020-03-13T00:25:35.174Z INFO Configuration loaded from: C:\Program Files\Storj\Storage Node\config.yaml
2020-03-13T00:25:35.199Z INFO Operator email: *********
2020-03-13T00:25:35.199Z INFO operator wallet: *********
2020-03-13T00:25:36.012Z INFO version running on version v0.34.6
2020-03-13T00:25:37.382Z INFO db.migration Database Version {"version": 31}
2020-03-13T00:25:40.140Z INFO preflight:localtime start checking local system clock with trusted satellites' system clock.
2020-03-13T00:25:40.990Z INFO preflight:localtime local system clock is in sync with trusted satellites' system clock.
2020-03-13T00:25:40.990Z INFO Node 12o9pkuq3KfHQTQmiJrTq5nvpJEmYtkpTuKCC3XjQc3GTVCZDcX started
2020-03-13T00:25:40.990Z INFO Public server started on [::]:28967
2020-03-13T00:25:40.990Z INFO Private server started on 127.0.0.1:7778
2020-03-13T00:25:40.990Z INFO bandwidth Performing bandwidth usage rollups
2020-03-13T00:25:40.990Z INFO trust Scheduling next refresh {"after": "6h11m23.016412979s"}
2020-03-13T00:25:41.110Z INFO version running on version v0.34.6
2020-03-13T00:25:41.150Z ERROR piecestore:cache error persisting cache totals to the database: {"error": "piece space used error: context canceled", "errorVerbose": "piece space used error: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceSpaceUsedDB).UpdatePieceTotals:174\n\tstorj.io/storj/storagenode/pieces.(*CacheService).PersistCacheTotals:100\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:85\n\tstorj.io/common/sync2.(*Cycle).Run:87\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:80\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func1:56\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}and this is inside the storage folder .
2020-03-13T00:26:59.004Z INFO Configuration loaded from: C:\Program Files\Storj\Storage Node\config.yaml
2020-03-13T00:26:59.036Z INFO Operator email: *********
2020-03-13T00:26:59.036Z INFO operator wallet: *********
2020-03-13T00:27:00.053Z INFO version running on version v0.34.6
2020-03-13T00:27:01.473Z INFO db.migration Database Version {"version": 31}
2020-03-13T00:27:04.198Z INFO preflight:localtime start checking local system clock with trusted satellites' system clock.
2020-03-13T00:27:05.043Z INFO preflight:localtime local system clock is in sync with trusted satellites' system clock.
2020-03-13T00:27:05.043Z INFO trust Scheduling next refresh {"after": "4h28m31.193329243s"}
2020-03-13T00:27:05.043Z INFO Node 12o9pkuq3KfHQTQmiJrTq5nvpJEmYtkpTuKCC3XjQc3GTVCZDcX started
2020-03-13T00:27:05.043Z INFO Public server started on [::]:28967
2020-03-13T00:27:05.043Z INFO Private server started on 127.0.0.1:7778
2020-03-13T00:27:05.043Z INFO bandwidth Performing bandwidth usage rollups
2020-03-13T00:27:05.161Z INFO version running on version v0.34.6
2020-03-13T00:27:05.209Z INFO piecestore download started {"Piece ID": "VGZEN6DOX4M5SKRG52SVGWGD7XPGRVIC6UAGYZNNKUAYV2E6IZLQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET"}
2020-03-13T00:27:05.362Z ERROR collector error during collecting pieces: {"error": "context canceled"}
2020-03-13T00:27:05.875Z ERROR pieces:trash emptying trash failed {"error": "pieces error: filestore error: context canceled", "errorVerbose": "pieces error: filestore error: context canceled\n\tstorj.io/storj/storage/filestore.(*blobStore).EmptyTrash:127\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).EmptyTrash:309\n\tstorj.io/storj/storagenode/pieces.(*Store).EmptyTrash:329\n\tstorj.io/storj/storagenode/pieces.(*TrashChore).Run.func1:51\n\tstorj.io/common/sync2.(*Cycle).Run:87\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-03-13T00:27:05.901Z ERROR pieces:trash emptying trash failed {"error": "pieces error: filestore error: context canceled", "errorVerbose": "pieces error: filestore error: context canceled\n\tstorj.io/storj/storage/filestore.(*blobStore).EmptyTrash:127\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).EmptyTrash:309\n\tstorj.io/storj/storagenode/pieces.(*Store).EmptyTrash:329\n\tstorj.io/storj/storagenode/pieces.(*TrashChore).Run.func1:51\n\tstorj.io/common/sync2.(*Cycle).Run:87\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-03-13T00:27:05.927Z ERROR pieces:trash emptying trash failed {"error": "pieces error: filestore error: context canceled", "errorVerbose": "pieces error: filestore error: context canceled\n\tstorj.io/storj/storage/filestore.(*blobStore).EmptyTrash:127\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).EmptyTrash:309\n\tstorj.io/storj/storagenode/pieces.(*Store).EmptyTrash:329\n\tstorj.io/storj/storagenode/pieces.(*TrashChore).Run.func1:51\n\tstorj.io/common/sync2.(*Cycle).Run:87\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-03-13T00:27:05.953Z ERROR pieces:trash emptying trash failed {"error": "pieces error: filestore error: context canceled", "errorVerbose": "pieces error: filestore error: context canceled\n\tstorj.io/storj/storage/filestore.(*blobStore).EmptyTrash:127\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).EmptyTrash:309\n\tstorj.io/storj/storagenode/pieces.(*Store).EmptyTrash:329\n\tstorj.io/storj/storagenode/pieces.(*TrashChore).Run.func1:51\n\tstorj.io/common/sync2.(*Cycle).Run:87\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-03-13T00:27:05.954Z ERROR pieces:trash emptying trash failed {"error": "pieces error: filestore error: context canceled", "errorVerbose": "pieces error: filestore error: context canceled\n\tstorj.io/storj/storage/filestore.(*blobStore).EmptyTrash:127\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).EmptyTrash:309\n\tstorj.io/storj/storagenode/pieces.(*Store).EmptyTrash:329\n\tstorj.io/storj/storagenode/pieces.(*TrashChore).Run.func1:51\n\tstorj.io/common/sync2.(*Cycle).Run:87\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-03-13T00:27:07.571Z ERROR piecestore:cache error getting current space used calculation: {"error": "context canceled; context canceled; context canceled; context canceled; context canceled", "errorVerbose": "group:\n--- context canceled\n--- context canceled\n--- context canceled\n--- context canceled\n--- context canceAnd heres the same lines from the folder i am not sure is the correct, but the service stays up …
2020-03-13T00:28:22.693Z INFO Configuration loaded from: C:\Program Files\Storj\Storage Node\config.yaml
2020-03-13T00:28:22.730Z INFO Operator email: ******
2020-03-13T00:28:22.730Z INFO operator wallet: ******
2020-03-13T00:28:23.656Z INFO version running on version v0.34.6
2020-03-13T00:28:23.925Z INFO db.migration.7 Drop certificate table.
Thanks.
Please, do not experiment with a different folders locations, your node will be disqualified very quick.
Stop the storagenode service.
If you migrated from a docker, your storage.path: option in the "%ProgramFiles%\Storj\Storage Node\config.yaml" should point to the folder ended with storage, like this:
storage.path: X:\storagenode\storage\
Please, provide a way how you migrated from the Ubuntu Docker to the Windows GUI.
Also, please specify the path, which you have used for the storage on the windows PC.
I’ll give you commands based on this information.
Hi Alexey,
Understood - OK so the procedure I did was …
Manually copy off the identity folder from Ubuntu and copy into the documents on Windows GUI…
Run the GUI installer filling in the same details - payout address, email etc … I also pointed the identity to the newly copied folder…
I then reconnected the drive to the Windows machine (This was added as an iscsi drive on Ubuntu, and is iscsi again on windows) making sure to point the gui to the same folder structure, IE: F:\DATA\storage
I appreciate you push towards local storage only, but for now this used to work fine and hopefully will continue to work…
I would believe the settings to be correct based on the above. Please let me know if you think of anything I may have done wrong or what to check next.
Thanks !
Can explain these errors:
I may have found the underlying issue… on Ubuntu the iSCSI volume was formatted as EXT4, when I’ve moved over to Windows this cannot natively read that FS so I installed ext2 volume manager and I was able to access.
I thought that Storage Node would be find reading and writing based on this but I am guessing not now ? The plan is migrate the data over to an NTFS volume and we should be back in business - would you agree?
The Windows will not work normally with ext4 either.
You must use the NTFS drive and migrate the data to there
Currently copying data over to an NTFS drive now…
Feeling confident all will be well after this.
I assume I will be able to access the Dashboard from other machines on the same network like I could when on Ubuntu Docker ? Do i need to allow / change anything?
Yes, you should uncomment this line in the "%ProgramFiles%\Storj\Storage Node\config.yaml":
# console.address: 127.0.0.1:14002
And replace the 127.0.0.1 to 0.0.0.0, save the config and restart the storagenode service either from the Services applet or from the elevated Powershell:
Restart-Service storagenode
Ah - yes that makes sense, thanks I will do that once the data has copied and we are up and running.
Cheers.
Hi
So all the data seemed to copy over fine and the service is now running and i can access the dashboard.
Though one thing i see is the bandwidth used this month and disk space used this month are at 0 and have been for some while ?
Also in the logs every so often i see …
2020-03-17T23:08:45.994Z ERROR console:endpoint api handler server error {"status code": 500, "error": "storagenode console web error: storage node dashboard service error: bandwidthdb error: database disk image is malformed", "errorVerbose": "storagenode console web error: storage node dashboard service error: bandwidthdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).getDailyUsageRollups:435\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).GetDailyRollups:342\n\tstorj.io/storj/storagenode/console.(*Service).GetAllSatellitesData:268\n\tstorj.io/storj/storagenode/console/consoleserver.(*Server).satellitesHandler:147\n\tnet/http.HandlerFunc.ServeHTTP:2007\n\tgithub.com/gorilla/mux.(*Router).ServeHTTP:212\n\tnet/http.serverHandler.ServeHTTP:2802\n\tnet/http.(*conn).serve:1890"}
Could that be to do with the anything?
Thanks
Please, immediately stop the node!
Give me result of this command (Powershell):
ls (sls "storage.path: (.*)" "C:\Program Files\Storj\Storage Node\config.yaml").Matches.Groups[1].Value
database disk image is malformed
Is happened because you didn’t rsync (or robocopy) the last time after the source node is stopped.
Please, check your databases and fix them: https://support.storj.io/hc/en-us/articles/360029309111-How-to-fix-a-database-disk-image-is-malformed-
Hi
I ran robocopy from the source and destination to ensure nothing was missed.
Heres the output of the command
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 13/03/2020 18:58 blobs
d----- 17/03/2020 23:34 garbage
d----- 17/03/2020 21:22 temp
d----- 17/03/2020 19:56 trash
-a---- 17/03/2020 23:04 5849088 bandwidth.db
-a---- 17/03/2020 23:04 32768 bandwidth.db-shm
-a---- 17/03/2020 23:05 41232 bandwidth.db-wal
-a---- 17/03/2020 23:04 16384 info.db
-a---- 17/03/2020 23:04 32768 info.db-shm
-a---- 17/03/2020 23:04 32 info.db-wal
-a---- 17/03/2020 23:04 24576 notifications.db
-a---- 17/03/2020 23:04 32768 notifications.db-shm
-a---- 17/03/2020 23:04 32 notifications.db-wal
-a---- 17/03/2020 23:04 111591424 orders.db
-a---- 17/03/2020 23:04 32768 orders.db-shm
-a---- 17/03/2020 23:33 3015872 orders.db-wal
-a---- 10/03/2020 17:47 3076096 pieceinfo.db
-a---- 17/03/2020 23:05 32768 pieceinfo.db-shm
-a---- 17/03/2020 23:00 82432 pieceinfo.db-wal
-a---- 17/03/2020 23:04 65536 piece_expiration.db
-a---- 17/03/2020 23:04 32768 piece_expiration.db-shm
-a---- 17/03/2020 23:05 32 piece_expiration.db-wal
-a---- 17/03/2020 23:04 24576 piece_spaced_used.db
-a---- 17/03/2020 23:04 32768 piece_spaced_used.db-shm
-a---- 17/03/2020 23:05 32 piece_spaced_used.db-wal
-a---- 17/03/2020 23:04 20480 reputation.db
-a---- 17/03/2020 23:05 32768 reputation.db-shm
-a---- 17/03/2020 23:05 32 reputation.db-wal
-a---- 17/03/2020 23:04 32768 satellites.db
-a---- 17/03/2020 23:04 32768 satellites.db-shm
-a---- 17/03/2020 23:05 32 satellites.db-wal
-a---- 17/03/2020 23:04 159744 storage_usage.db
-a---- 17/03/2020 23:05 32768 storage_usage.db-shm
-a---- 17/03/2020 23:05 32 storage_usage.db-wal
-a---- 17/03/2020 23:31 27852800 used_serial.db
-a---- 17/03/2020 23:17 65536 used_serial.db-shm
-a---- 17/03/2020 23:33 25169112 used_serial.db-wal
Looking at the page you linked it references docker but I am using gui ?
Please advise
Thanks
it’s generic. You can install the native sqlite3 or use a docker. You even can install it in the WSL+Ubuntu.
Hi Alexey,
Just want to say thanks for all your help on this one.
I was able to find the corrupt database and issue the repairs as per the document you linked.
Dashboard is now reporting bandwidth and disk space usage and the node is operational all without any data loss !
Hopefully there is no further downtime now and all runs well as I hopefully have plenty of space to offer.
Thanks again.