Migrating from Ubuntu Docker to Windows GUI - Service Stopping

Hi All,

I’m trying to migrate from Ubuntu Docker to Windows GUI, I have followed the guide and copied over the identity and storage folders / data and ran the install and pointed to the correct locations - all looks good. But the storage node service keeps stopping and i’m unable to get to the dashboard, did i do something wrong ?

How best to diagnose this issue?

Thanks !

Hello @brad-pea,
Welcome to the forum!

Please, show 20 lines from the log: https://documentation.storj.io/resources/faq/check-logs

Thanks !

So I did some digging around whilst waiting for the reply and if i point it to another folder on the same drive (though i am 99% sure the docker was not using that folder) the service will remain up, but i still cannot access the dashboard - it does not load …

Heres the 20 lines from the folder i believe to be correct … but I see there is a storage folder within it so this is outside of the storage folder…

2020-03-13T00:25:35.174Z        INFO    Configuration loaded from: C:\Program Files\Storj\Storage Node\config.yaml
2020-03-13T00:25:35.199Z        INFO    Operator email: *********
2020-03-13T00:25:35.199Z        INFO    operator wallet: *********
2020-03-13T00:25:36.012Z        INFO    version running on version v0.34.6
2020-03-13T00:25:37.382Z        INFO    db.migration    Database Version        {"version": 31}
2020-03-13T00:25:40.140Z        INFO    preflight:localtime     start checking local system clock with trusted satellites' system clock.
2020-03-13T00:25:40.990Z        INFO    preflight:localtime     local system clock is in sync with trusted satellites' system clock.
2020-03-13T00:25:40.990Z        INFO    Node 12o9pkuq3KfHQTQmiJrTq5nvpJEmYtkpTuKCC3XjQc3GTVCZDcX started
2020-03-13T00:25:40.990Z        INFO    Public server started on [::]:28967
2020-03-13T00:25:40.990Z        INFO    Private server started on 127.0.0.1:7778
2020-03-13T00:25:40.990Z        INFO    bandwidth       Performing bandwidth usage rollups
2020-03-13T00:25:40.990Z        INFO    trust   Scheduling next refresh {"after": "6h11m23.016412979s"}
2020-03-13T00:25:41.110Z        INFO    version running on version v0.34.6
2020-03-13T00:25:41.150Z        ERROR   piecestore:cache        error persisting cache totals to the database:  {"error": "piece space used error: context canceled", "errorVerbose": "piece space used error: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceSpaceUsedDB).UpdatePieceTotals:174\n\tstorj.io/storj/storagenode/pieces.(*CacheService).PersistCacheTotals:100\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:85\n\tstorj.io/common/sync2.(*Cycle).Run:87\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:80\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func1:56\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

and this is inside the storage folder .

2020-03-13T00:26:59.004Z        INFO    Configuration loaded from: C:\Program Files\Storj\Storage Node\config.yaml
2020-03-13T00:26:59.036Z        INFO    Operator email: *********
2020-03-13T00:26:59.036Z        INFO    operator wallet: *********
2020-03-13T00:27:00.053Z        INFO    version running on version v0.34.6
2020-03-13T00:27:01.473Z        INFO    db.migration    Database Version        {"version": 31}
2020-03-13T00:27:04.198Z        INFO    preflight:localtime     start checking local system clock with trusted satellites' system clock.
2020-03-13T00:27:05.043Z        INFO    preflight:localtime     local system clock is in sync with trusted satellites' system clock.
2020-03-13T00:27:05.043Z        INFO    trust   Scheduling next refresh {"after": "4h28m31.193329243s"}
2020-03-13T00:27:05.043Z        INFO    Node 12o9pkuq3KfHQTQmiJrTq5nvpJEmYtkpTuKCC3XjQc3GTVCZDcX started
2020-03-13T00:27:05.043Z        INFO    Public server started on [::]:28967
2020-03-13T00:27:05.043Z        INFO    Private server started on 127.0.0.1:7778
2020-03-13T00:27:05.043Z        INFO    bandwidth       Performing bandwidth usage rollups
2020-03-13T00:27:05.161Z        INFO    version running on version v0.34.6
2020-03-13T00:27:05.209Z        INFO    piecestore      download started        {"Piece ID": "VGZEN6DOX4M5SKRG52SVGWGD7XPGRVIC6UAGYZNNKUAYV2E6IZLQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET"}
2020-03-13T00:27:05.362Z        ERROR   collector       error during collecting pieces:         {"error": "context canceled"}
2020-03-13T00:27:05.875Z        ERROR   pieces:trash    emptying trash failed   {"error": "pieces error: filestore error: context canceled", "errorVerbose": "pieces error: filestore error: context canceled\n\tstorj.io/storj/storage/filestore.(*blobStore).EmptyTrash:127\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).EmptyTrash:309\n\tstorj.io/storj/storagenode/pieces.(*Store).EmptyTrash:329\n\tstorj.io/storj/storagenode/pieces.(*TrashChore).Run.func1:51\n\tstorj.io/common/sync2.(*Cycle).Run:87\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-03-13T00:27:05.901Z        ERROR   pieces:trash    emptying trash failed   {"error": "pieces error: filestore error: context canceled", "errorVerbose": "pieces error: filestore error: context canceled\n\tstorj.io/storj/storage/filestore.(*blobStore).EmptyTrash:127\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).EmptyTrash:309\n\tstorj.io/storj/storagenode/pieces.(*Store).EmptyTrash:329\n\tstorj.io/storj/storagenode/pieces.(*TrashChore).Run.func1:51\n\tstorj.io/common/sync2.(*Cycle).Run:87\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-03-13T00:27:05.927Z        ERROR   pieces:trash    emptying trash failed   {"error": "pieces error: filestore error: context canceled", "errorVerbose": "pieces error: filestore error: context canceled\n\tstorj.io/storj/storage/filestore.(*blobStore).EmptyTrash:127\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).EmptyTrash:309\n\tstorj.io/storj/storagenode/pieces.(*Store).EmptyTrash:329\n\tstorj.io/storj/storagenode/pieces.(*TrashChore).Run.func1:51\n\tstorj.io/common/sync2.(*Cycle).Run:87\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-03-13T00:27:05.953Z        ERROR   pieces:trash    emptying trash failed   {"error": "pieces error: filestore error: context canceled", "errorVerbose": "pieces error: filestore error: context canceled\n\tstorj.io/storj/storage/filestore.(*blobStore).EmptyTrash:127\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).EmptyTrash:309\n\tstorj.io/storj/storagenode/pieces.(*Store).EmptyTrash:329\n\tstorj.io/storj/storagenode/pieces.(*TrashChore).Run.func1:51\n\tstorj.io/common/sync2.(*Cycle).Run:87\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-03-13T00:27:05.954Z        ERROR   pieces:trash    emptying trash failed   {"error": "pieces error: filestore error: context canceled", "errorVerbose": "pieces error: filestore error: context canceled\n\tstorj.io/storj/storage/filestore.(*blobStore).EmptyTrash:127\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).EmptyTrash:309\n\tstorj.io/storj/storagenode/pieces.(*Store).EmptyTrash:329\n\tstorj.io/storj/storagenode/pieces.(*TrashChore).Run.func1:51\n\tstorj.io/common/sync2.(*Cycle).Run:87\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-03-13T00:27:07.571Z        ERROR   piecestore:cache        error getting current space used calculation:   {"error": "context canceled; context canceled; context canceled; context canceled; context canceled", "errorVerbose": "group:\n--- context canceled\n--- context canceled\n--- context canceled\n--- context canceled\n--- context cance

And heres the same lines from the folder i am not sure is the correct, but the service stays up …

2020-03-13T00:28:22.693Z        INFO    Configuration loaded from: C:\Program Files\Storj\Storage Node\config.yaml
2020-03-13T00:28:22.730Z        INFO    Operator email: ******
2020-03-13T00:28:22.730Z        INFO    operator wallet: ******
2020-03-13T00:28:23.656Z        INFO    version running on version v0.34.6
2020-03-13T00:28:23.925Z        INFO    db.migration.7  Drop certificate table.

Thanks.

Please, do not experiment with a different folders locations, your node will be disqualified very quick.
Stop the storagenode service.
If you migrated from a docker, your storage.path: option in the "%ProgramFiles%\Storj\Storage Node\config.yaml" should point to the folder ended with storage, like this:

storage.path: X:\storagenode\storage\

Please, provide a way how you migrated from the Ubuntu Docker to the Windows GUI.
Also, please specify the path, which you have used for the storage on the windows PC.

I’ll give you commands based on this information.

Hi Alexey,

Understood - OK so the procedure I did was …

Manually copy off the identity folder from Ubuntu and copy into the documents on Windows GUI…

Run the GUI installer filling in the same details - payout address, email etc … I also pointed the identity to the newly copied folder…

I then reconnected the drive to the Windows machine (This was added as an iscsi drive on Ubuntu, and is iscsi again on windows) making sure to point the gui to the same folder structure, IE: F:\DATA\storage

I appreciate you push towards local storage only, but for now this used to work fine and hopefully will continue to work…

I would believe the settings to be correct based on the above. Please let me know if you think of anything I may have done wrong or what to check next.

Thanks !

Can explain these errors:

I may have found the underlying issue… on Ubuntu the iSCSI volume was formatted as EXT4, when I’ve moved over to Windows this cannot natively read that FS so I installed ext2 volume manager and I was able to access.

I thought that Storage Node would be find reading and writing based on this but I am guessing not now ? The plan is migrate the data over to an NTFS volume and we should be back in business - would you agree?

The Windows will not work normally with ext4 either.
You must use the NTFS drive and migrate the data to there

Currently copying data over to an NTFS drive now…

Feeling confident all will be well after this.

I assume I will be able to access the Dashboard from other machines on the same network like I could when on Ubuntu Docker ? Do i need to allow / change anything?

Yes, you should uncomment this line in the "%ProgramFiles%\Storj\Storage Node\config.yaml":

# console.address: 127.0.0.1:14002

And replace the 127.0.0.1 to 0.0.0.0, save the config and restart the storagenode service either from the Services applet or from the elevated Powershell:

Restart-Service storagenode

Ah - yes that makes sense, thanks I will do that once the data has copied and we are up and running.

Cheers.

Hi

So all the data seemed to copy over fine and the service is now running and i can access the dashboard.

Though one thing i see is the bandwidth used this month and disk space used this month are at 0 and have been for some while ?

Also in the logs every so often i see …

2020-03-17T23:08:45.994Z        ERROR   console:endpoint        api handler server error        {"status code": 500, "error": "storagenode console web error: storage node dashboard service error: bandwidthdb error: database disk image is malformed", "errorVerbose": "storagenode console web error: storage node dashboard service error: bandwidthdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).getDailyUsageRollups:435\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).GetDailyRollups:342\n\tstorj.io/storj/storagenode/console.(*Service).GetAllSatellitesData:268\n\tstorj.io/storj/storagenode/console/consoleserver.(*Server).satellitesHandler:147\n\tnet/http.HandlerFunc.ServeHTTP:2007\n\tgithub.com/gorilla/mux.(*Router).ServeHTTP:212\n\tnet/http.serverHandler.ServeHTTP:2802\n\tnet/http.(*conn).serve:1890"}

Could that be to do with the anything?

Thanks

Please, immediately stop the node!
Give me result of this command (Powershell):

 ls (sls "storage.path: (.*)" "C:\Program Files\Storj\Storage Node\config.yaml").Matches.Groups[1].Value

database disk image is malformed

Is happened because you didn’t rsync (or robocopy) the last time after the source node is stopped.
Please, check your databases and fix them: https://support.storj.io/hc/en-us/articles/360029309111-How-to-fix-a-database-disk-image-is-malformed-

Hi

I ran robocopy from the source and destination to ensure nothing was missed.

Heres the output of the command

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----       13/03/2020     18:58                blobs
d-----       17/03/2020     23:34                garbage
d-----       17/03/2020     21:22                temp
d-----       17/03/2020     19:56                trash
-a----       17/03/2020     23:04        5849088 bandwidth.db
-a----       17/03/2020     23:04          32768 bandwidth.db-shm
-a----       17/03/2020     23:05          41232 bandwidth.db-wal
-a----       17/03/2020     23:04          16384 info.db
-a----       17/03/2020     23:04          32768 info.db-shm
-a----       17/03/2020     23:04             32 info.db-wal
-a----       17/03/2020     23:04          24576 notifications.db
-a----       17/03/2020     23:04          32768 notifications.db-shm
-a----       17/03/2020     23:04             32 notifications.db-wal
-a----       17/03/2020     23:04      111591424 orders.db
-a----       17/03/2020     23:04          32768 orders.db-shm
-a----       17/03/2020     23:33        3015872 orders.db-wal
-a----       10/03/2020     17:47        3076096 pieceinfo.db
-a----       17/03/2020     23:05          32768 pieceinfo.db-shm
-a----       17/03/2020     23:00          82432 pieceinfo.db-wal
-a----       17/03/2020     23:04          65536 piece_expiration.db
-a----       17/03/2020     23:04          32768 piece_expiration.db-shm
-a----       17/03/2020     23:05             32 piece_expiration.db-wal
-a----       17/03/2020     23:04          24576 piece_spaced_used.db
-a----       17/03/2020     23:04          32768 piece_spaced_used.db-shm
-a----       17/03/2020     23:05             32 piece_spaced_used.db-wal
-a----       17/03/2020     23:04          20480 reputation.db
-a----       17/03/2020     23:05          32768 reputation.db-shm
-a----       17/03/2020     23:05             32 reputation.db-wal
-a----       17/03/2020     23:04          32768 satellites.db
-a----       17/03/2020     23:04          32768 satellites.db-shm
-a----       17/03/2020     23:05             32 satellites.db-wal
-a----       17/03/2020     23:04         159744 storage_usage.db
-a----       17/03/2020     23:05          32768 storage_usage.db-shm
-a----       17/03/2020     23:05             32 storage_usage.db-wal
-a----       17/03/2020     23:31       27852800 used_serial.db
-a----       17/03/2020     23:17          65536 used_serial.db-shm
-a----       17/03/2020     23:33       25169112 used_serial.db-wal

Looking at the page you linked it references docker but I am using gui ?

Please advise

Thanks

it’s generic. You can install the native sqlite3 or use a docker. You even can install it in the WSL+Ubuntu.

Hi Alexey,

Just want to say thanks for all your help on this one.

I was able to find the corrupt database and issue the repairs as per the document you linked.

Dashboard is now reporting bandwidth and disk space usage and the node is operational all without any data loss !

Hopefully there is no further downtime now and all runs well as I hopefully have plenty of space to offer.

Thanks again.

3 Likes