Web interface not showing any information

smoke_nksp · August 27, 2024, 4:15am

after electrical failure and disk check, there was a problem with trust-cache.json, deleted it, created a new one and noda worked, but the web panel does not show anything, anyone know what to do?
logs seems ok.
Please advise.

Alexey · August 27, 2024, 7:27am

Make sure that the service is running:
PowerShell:

Get-Service storagenode*

Please also check the logs for Unrecoverable and/or FATAL errors:

sls "Unrecoverable|FATAL" "$env:ProgramFiles\Storj\Storage Node\storagenode.log" | select -last 10

Also, please check your databases:

If they are ok, and service is running and there is no a FATAL error, then search for “ping satellite failed” but “rate”:

sls "ping satellite failed" "$env:ProgramFiles\Storj\Storage Node\storagenode.log" | sls -NotMatch "rate" | select -last 10

smoke_nksp · August 27, 2024, 5:25pm

Get-Service storagenode*

Status   Name               DisplayName
------   ----               -----------
Running  storagenode        Storj V3 Storage Node
Running  storagenode-upd... Storj V3 Storage Node Updater

sls "Unrecoverable|FATAL" "D:\storjnode\storagenode.log" | select -last 10

D:\storjnode\storagenode.log:215241:2024-08-26T20:47:57+03:00   INFO    piecestore      upload started  {"Piece ID": "TY5HSPP4PDN4
XDJC7LW3V7ZUKQZJE6A42BZU5FATALLDKAWPYNTQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Acti
on": "PUT", "Remote Address": "79.127.226.101:47236", "Available Space": 179399966153}
D:\storjnode\storagenode.log:215252:2024-08-26T20:47:57+03:00   INFO    piecestore      uploaded        {"Piece ID": "TY5HSPP4PDN4XDJC7L
W3V7ZUKQZJE6A42BZU5FATALLDKAWPYNTQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "
PUT", "Remote Address": "79.127.226.101:47236", "Size": 2560}

database check:

bandwidth.db ok
garbage_collection_filewalker_progress.db ok
heldamount.db ok
info.db ok
notifications.db ok
orders.db ok
pieceinfo.db ok
piece_expiration.db *** in database main *** Page 9898: never used Page 11279: never used Page 12401: never used Page 12597: never used Page 12706: never used Page 12910: never used Page 12922: never used Page 12930: never used Page 13431: never used Page 13778: never used Page 14911: never used Page 15386: never used Page 15415: never used Page 15428: never used Page 15930: never used Page 16571: never used Page 17407: never used Page 17720: never used Page 18228: never used Page 19040: never used Page 21627: never used Page 23717: never used Page 23819: never used Page 23941: never used Page 25679: never used Page 25855: never used Page 25964: never used Page 26605: never used Page 26761: never used Page 27030: never used Page 29206: never used Page 30332: never used Page 30442: never used Page 40360: never used Page 40643: never used Page 41280: never used Page 41427: never used Page 42210: never used Page 42242: never used Page 42423: never used Page 42546: never used Page 43174: never used Page 49075: never used Page 49964: never used Page 54844: never used Page 56098: never used Page 57784: never used Page 58411: never used Page 58614: never used Page 58884: never used Page 59197: never used Page 59519: never used Page 59880: never used Page 60021: never used Page 60086: never used Page 60097: never used Page 60209: never used Page 60262: never used Page 60319: never used Page 61046: never used Page 61055: never used Page 61157: never used Page 61268: never used Page 61315: never used Page 62566: never used Page 66237: never used Page 66977: never used Page 67284: never used Page 67353: never used
piece_spaced_used.db ok
pricing.db ok
Error: stepping, database disk image is malformed (11)
reputation.db *** in database main *** Tree 2 page 2 cell 0: 2nd reference to page 8 wrong # of entries in index sqlite_autoindex_reputation_1
Error: in prepare, file is not a database (26)
revocations.db
satellites.db ok
secret.db ok
storage_usage.db ok
used_serial.db ok
used_space_per_prefix.db ok

sls "ping satellite failed" "D:\storjnode\storagenode.log" | sls -NotMatch "rate" | select -last 10

D:\storjnode\storagenode.log:906323:2024-08-27T08:08:59+03:00   ERROR   contact:service ping satellite failed   {"Satellite ID": "12EayRS2V1kEs
WESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "attempts": 1, "error": "ping satellite: write tcp 192.168.0.106:64225->34.150.199.48:7777: wsase
nd: An existing connection was forcibly closed by the remote host.", "errorVerbose": "ping satellite: write tcp 192.168.0.106:64225->34.15
0.199.48:7777: wsasend: An existing connection was forcibly closed by the remote host.\n\tstorj.io/drpc/drpcstream.(*Stream).sendPacketLoc
ked:336\n\tstorj.io/drpc/drpcstream.(*Stream).Close:594\n\tstorj.io/drpc/drpcconn.(*Conn).Invoke.func1:118\n\tstorj.io/drpc/drpcconn.(*Con
n).Invoke:134\n\tstorj.io/common/rpc/rpcpool.(*poolConn).Invoke.func1:126\n\tstorj.io/common/rpc/rpcpool.stackAnnotated:43\n\tstorj.io/com
mon/rpc/rpcpool.stackAnnotate:79\n\tstorj.io/common/rpc/rpcpool.(*poolConn).Invoke:125\n\tstorj.io/common/rpc/rpctracing.(*TracingWrapper)
.Invoke:32\n\tstorj.io/common/experiment.(*Wrapper).Invoke:28\n\tstorj.io/common/pb.(*drpcNodeClient).CheckIn:137\n\tstorj.io/storj/storag
enode/contact.(*Service).pingSatelliteOnce:190\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:156\n\tstorj.io/storj/storag
enode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\
tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
D:\storjnode\storagenode.log:1058870:2024-08-27T11:08:58+03:00  ERROR   contact:service ping satellite failed   {"Satellite ID": "12EayRS2V1kE
sWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "attempts": 1, "error": "ping satellite: manager closed: closed: read tcp 192.168.0.106:53082->3
4.150.199.48:7777: wsarecv: An existing connection was forcibly closed by the remote host.", "errorVerbose": "ping satellite: manager clos
ed: closed: read tcp 192.168.0.106:53082->34.150.199.48:7777: wsarecv: An existing connection was forcibly closed by the remote host.\n\ts
torj.io/drpc/drpcmanager.(*Manager).manageReader:233"}
D:\storjnode\storagenode.log:1230453:2024-08-27T15:08:59+03:00  ERROR   contact:service ping satellite failed   {"Satellite ID": "12EayRS2V1kE
sWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "attempts": 1, "error": "ping satellite: manager closed: closed: read tcp 192.168.0.106:58979->3
4.150.199.48:7777: wsarecv: An existing connection was forcibly closed by the remote host.", "errorVerbose": "ping satellite: manager clos
ed: closed: read tcp 192.168.0.106:58979->34.150.199.48:7777: wsarecv: An existing connection was forcibly closed by the remote host.\n\ts
torj.io/drpc/drpcmanager.(*Manager).manageReader:233"}

as i understand here some problem in an database? i don’t understand which one? reputation.db maybe or some before?

Alexey · August 28, 2024, 3:26am

Yes, which are not “ok”:

smoke_nksp · August 28, 2024, 4:31am

ok, i do db restore, but node can’t start

2024-08-28T07:28:59+03:00	INFO	db.migration.26	Add Trash column to pieceExpirationDB
2024-08-28T07:28:59+03:00	INFO	db.migration.34	Add suspended field to satellites db
2024-08-28T07:28:59+03:00	INFO	db.migration.36	Add joined_at field to satellites db
2024-08-28T07:28:59+03:00	ERROR	failure during run	{"error": "Error migrating tables for database on storagenode: migrate: v36: duplicate column name: joined_at\n\tstorj.io/storj/private/migrate.(*Migration).Run:212\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:432\n\tmain.cmdRun:100\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "errorVerbose": "Error migrating tables for database on storagenode: migrate: v36: duplicate column name: joined_at\n\tstorj.io/storj/private/migrate.(*Migration).Run:212\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:432\n\tmain.cmdRun:100\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78\n\tmain.cmdRun:102\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-08-28T07:28:59+03:00	FATAL	Unrecoverable error	{"error": "Error migrating tables for database on storagenode: migrate: v36: duplicate column name: joined_at\n\tstorj.io/storj/private/migrate.(*Migration).Run:212\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:432\n\tmain.cmdRun:100\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "errorVerbose": "Error migrating tables for database on storagenode: migrate: v36: duplicate column name: joined_at\n\tstorj.io/storj/private/migrate.(*Migration).Run:212\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:432\n\tmain.cmdRun:100\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78\n\tmain.cmdRun:102\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}

what next?

Alexey · August 28, 2024, 4:39am

Seems the database reputation.db was corrupted too much, unfortunately you need to re-create this database from scratch:

smoke_nksp · August 28, 2024, 5:04am

thanks, from scratch node started and online, bandwish and disk space also ok.
but suspension and audit all red, if it ok?

smoke_nksp · August 28, 2024, 5:06am

ah ok, NVM, it grows up. thank you very much!

Alexey · August 28, 2024, 5:54am

Do you mean that you deleted all data instead of the one database?

smoke_nksp · August 28, 2024, 8:29am

i follow instuctions, make backup all db files, start node, wait a little bit, stop node, move all db files except reputation.db

Alexey · August 29, 2024, 4:11am

Thank you, I was just concerned when you said

I immediately interpreted this as you recreating the node from scratch.
Then I guess everything should be fine.

christos · November 13, 2024, 9:36am

Apparently I have a similar problem, but without any corrupted db whatsoever.
bandwidth.db ok
garbage_collection_filewalker_progress.db ok
heldamount.db ok
info.db ok
notifications.db ok
orders.db ok
pieceinfo.db ok
piece_expiration.db ok
piece_spaced_used.db ok
pricing.db ok
reputation.db ok
satellites.db ok
secret.db ok
storage_usage.db ok
used_serial.db ok
used_space_per_prefix.db ok

“Unrecoverable|FATAL” and “ping satellite failed” searched gave no results.

christos · November 13, 2024, 11:26am

As it turned out, searching the logs 1 week ago, showed some fatal error.
C:\Program Files\Storj\Storage Node\storagenode_20241107.log:3939110:2024-11-07T05:00:01+03:00 FATAL Unrecoverable error {“error”: “debug: listener closed”, “errorVerbose”: “debug: listener closed\n\tstorj.io/drpc/drpcmigrate.init:17\n\truntime.doInit:6527\n\truntime.doInit:6504\n\truntime.doInit:6504\n\truntime.do
Init:6504\n\truntime.main:233”

Apparently, after leaving the dashboard open for several minutes (or maybe hours? I didn’t really check it), it shows the data properly, but it’s really annoying. My other node shows dashboard properly, with no delays.

Update: Ok, just a couple of minutes, but still, I can’t get why the difference.

Vasabi · November 13, 2024, 1:36pm

у вас не включен storage2.monitor.dedicated-disk: true ?
начиная с версии 1.115.x у меня на Windows OS возникла такая проблема, что веб интерфейс не загружается, если оставить ее открытой на некоторое время то она начинает все отображать.
Помогло отключение # storage2.monitor.dedicated-disk: true

Alexey · November 15, 2024, 8:14am

This is related to the fact that this feature is experimental, not all code paths are updated, so the dashboard will show a wrong stat.

на русском

Это связано с тем, что эта фича - экспериментальная, не все места обновлены, чтобы информация отображалась корректно.

Alexey · November 15, 2024, 8:14am

The dashboard doesn’t automatically refresh. You need to refresh it manually.

christos · November 18, 2024, 4:32pm

Well, that’s the point. Even if I manually refresh, that round thing will keep rotating, but either it takes too long or it won’t refresh at all.

Alexey · November 19, 2024, 7:04am

Then you need to check - is the node still running and accessible.
You may use its port for a quick check: http://localhost:28967 (or http://local_ip:28967), it should response.

christos · November 19, 2024, 10:38am

I use port 28974 and it’s accessible through the Internet, using YouGetSignal. Shall I doubt and check it specifically through LAN?

Alexey · November 20, 2024, 6:46am

You need to try to open it as a web page, if it would respond, then the node is at least functioning and can accept requests.