Node Dashboard: Online Status and QUIC Functionality OK, but Charts, Suspension, and Audit Features Unavailable

Node Operators troubleshooting
,
1

I’ve encountered warning logs indicating “unable to get Satellite URL | console: trust: satellite is untrusted.” I’m unsure if this is relevant to the issue.

Furthermore, this morning, I received an email stating “Your Node has gone offline,” despite the traffic still moving. To investigate, I checked the status of port 28967 using Open Port Check Tool - Test Port Forwarding on Your Router, and the port appears to be open.

Here is a snippet of relevant logs:

2024-01-14T07:25:47Z	WARN	console:service	unable to get Satellite URL	{"process": "storagenode", "Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "error": "console: trust: satellite is untrusted", "errorVerbose": "console: trust: satellite is untrusted\n\tstorj.io/storj/storagenode/trust.init:29\n\truntime.doInit1:6740\n\truntime.doInit:6707\n\truntime.main:249"}
2024-01-14T07:25:47Z	WARN	console:service	unable to get Satellite URL	{"process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "error": "console: trust: satellite is untrusted", "errorVerbose": "console: trust: satellite is untrusted\n\tstorj.io/storj/storagenode/trust.init:29\n\truntime.doInit1:6740\n\truntime.doInit:6707\n\truntime.main:249"}
2024-01-14T07:25:50Z	INFO	piecestore	upload started	{"process": "storagenode", "Piece ID": "6CCNLVORFDKD5MASAOXSURAAIHQB6R7BQFPFMQ3NEKXBV2G456WA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 4461839535616, "Remote Address": "10.69.69.201:55579"}
2024-01-14T07:25:50Z	INFO	piecestore	upload started	{"process": "storagenode", "Piece ID": "LPHQZ4OJYRDH4NK5S3S7O4EC4FGIKH25OPD5HNG2BLLIEIXJVZ7Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 4461837195520, "Remote Address": "10.69.69.201:15261"}
2024-01-14T07:25:50Z	INFO	piecestore	upload started	{"process": "storagenode", "Piece ID": "UXYCR67UQVTKKUPFSBO4CEDARPNJLZLFVW6ZXGW2D3ACDUSHHY7Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 4461837158400, "Remote Address": "10.69.69.201:32021"}
2024-01-14T07:25:51Z	INFO	piecestore	upload started	{"process": "storagenode", "Piece ID": "2SG5TNIWGNSR2FQB5FFWCCP36TQS5FZZ2BTFFTZAGWXD5C6GUGOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 4461837158400, "Remote Address": "10.69.69.201:53256"}
2024-01-14T07:25:53Z	INFO	piecestore	download started	{"process": "storagenode", "Piece ID": "SNKVDSAQJUUP6Y3WFE625D3HLZR3QUP5ICMSKC5KTIKH4LMGVTDA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 17408, "Remote Address": "10.69.69.201:13090"}
2024-01-14T07:25:55Z	INFO	reputation:service	node scores updated	{"process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Total Audits": 5689, "Successful Audits": 5583, "Audit Score": 1, "Online Score": 0.9516714766714766, "Suspension Score": 0.971559995386177, "Audit Score Delta": 0, "Online Score Delta": 0, "Suspension Score Delta": 0}
2024-01-14T07:25:55Z	INFO	reputation:service	node scores updated	{"process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Total Audits": 66783, "Successful Audits": 65369, "Audit Score": 1, "Online Score": 0.9391022700055313, "Suspension Score": 0.9336818204931208, "Audit Score Delta": 0, "Online Score Delta": 0, "Suspension Score Delta": 0.011032080888955909}

Screenshot 2567-01-14 at 14.34.46
Screenshot 2567-01-14 at 14.34.46726×1070 57.6 KB

2

You need to forget these satellites: How To Forget Untrusted Satellites

Please check your databases:

3 
./storagenode  forget-satellite --all-untrusted --config-dir ./config/ --identity-dir ./identity/
2024-01-14T08:34:05Z    INFO    Configuration loaded    {"process": "storagenode", "Location": "/app/config/config.yaml"}
2024-01-14T08:34:05Z    INFO    Anonymized tracing enabled      {"process": "storagenode"}
2024-01-14T08:34:05Z    INFO    Identity loaded.        {"process": "storagenode", "Node ID": "1wDLe5VY7fzQrq8xBFFMwz2h17E5EKF6jPTvsA1V2paTXJBmW2"}
2024-01-14T08:34:06Z    INFO    Removing satellite from trust cache.    {"process": "storagenode", "satelliteID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB"}
2024-01-14T08:34:06Z    INFO    Cleaning up satellite data.     {"process": "storagenode", "satelliteID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB"}
2024-01-14T08:40:43Z    INFO    Cleaning up the trash.  {"process": "storagenode", "satelliteID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB"}
2024-01-14T08:41:48Z    INFO    Removing satellite info from reputation DB.     {"process": "storagenode", "satelliteID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB"}
2024-01-14T08:41:48Z    INFO    Removing satellite v0 pieces if any.    {"process": "storagenode", "satelliteID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB"}
2024-01-14T08:41:48Z    INFO    Removing satellite from satellites DB.  {"process": "storagenode", "satelliteID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB"}
2024-01-14T08:41:48Z    INFO    Removing satellite from trust cache.    {"process": "storagenode", "satelliteID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo"}
2024-01-14T08:41:48Z    INFO    Cleaning up satellite data.     {"process": "storagenode", "satelliteID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo"}
2024-01-14T08:42:53Z    INFO    Cleaning up the trash.  {"process": "storagenode", "satelliteID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo"}
2024-01-14T08:43:44Z    INFO    Removing satellite info from reputation DB.     {"process": "storagenode", "satelliteID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo"}
2024-01-14T08:43:44Z    INFO    Removing satellite v0 pieces if any.    {"process": "storagenode", "satelliteID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo"}
2024-01-14T08:43:44Z    INFO    Removing satellite from satellites DB.  {"process": "storagenode", "satelliteID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo"}

find . -maxdepth 1 -iname "*.db" -print0-exec sqlite3 '{}' 'PRAGMA integrity_check;' ';'
./satellites.dbok
./secret.dbok
./piece_spaced_used.dbok
./piece_expiration.dbok
./info.dbok
./used_serial.dbok
./pieceinfo.dbok
./heldamount.dbok

Following the node restart, the chart and audit features remain inaccessible; however, the warning logs have been resolved. Despite this, I have not yet received the “Node online” email.

4

It appears that these requests continue to experience timeouts, and the reason behind this issue is currently unclear.

Screenshot 2567-01-14 at 16.19.08
Screenshot 2567-01-14 at 16.19.08878×210 22.1 KB

5

You have a short list of databases, seems some databases are missing.
Please, recreate them:
https://support.storj.io/hc/en-us/articles/4403032417044-How-to-fix-database-file-is-not-a-database-error
To forget decommissioned satellites you need to use a different command, unless you explicitly added these satellites to the untrusted list (as explained in the article):

But before that you need to recreate missing databases.

6

Oh, I think I did not copy all result from terminal, here the result

find . -maxdepth 1 -iname "*.db" -print0-exec sqlite3 '{}' 'PRAGMA integrity_check;' ';'
./satellites.dbok
./secret.dbok
./piece_spaced_used.dbok
./piece_expiration.dbok
./info.dbok
./used_serial.dbok
./pieceinfo.dbok
./heldamount.dbok
./pricing.dbok
./notifications.dbok
./reputation.dbok
./orders.dbok
./bandwidth.dbok
./storage_usage.dbok

but I will try that again

move *.db to backup then start node

2024-01-14T10:04:35Z	INFO	db	database does not exist	{"process": "storagenode", "database": "info"}
2024-01-14T10:04:35Z	INFO	db	database does not exist	{"process": "storagenode", "database": "bandwidth"}
2024-01-14T10:04:35Z	INFO	db	database does not exist	{"process": "storagenode", "database": "orders"}
2024-01-14T10:04:35Z	INFO	db	database does not exist	{"process": "storagenode", "database": "piece_expiration"}
2024-01-14T10:04:35Z	INFO	db	database does not exist	{"process": "storagenode", "database": "pieceinfo"}
2024-01-14T10:04:35Z	INFO	db	database does not exist	{"process": "storagenode", "database": "piece_spaced_used"}
2024-01-14T10:04:35Z	INFO	db	database does not exist	{"process": "storagenode", "database": "reputation"}
2024-01-14T10:04:35Z	INFO	db	database does not exist	{"process": "storagenode", "database": "storage_usage"}
2024-01-14T10:04:35Z	INFO	db	database does not exist	{"process": "storagenode", "database": "used_serial"}
2024-01-14T10:04:35Z	INFO	db	database does not exist	{"process": "storagenode", "database": "satellites"}
2024-01-14T10:04:35Z	INFO	db	database does not exist	{"process": "storagenode", "database": "notifications"}
2024-01-14T10:04:35Z	INFO	db	database does not exist	{"process": "storagenode", "database": "heldamount"}
2024-01-14T10:04:35Z	INFO	db	database does not exist	{"process": "storagenode", "database": "pricing"}
2024-01-14T10:04:35Z	INFO	db	database does not exist	{"process": "storagenode", "database": "secret"}

and it create new empty database

info.db-shm              piece_expiration.db-wal  reputation.db            storage_usage.db
bandwidth.db             info.db-wal              piece_spaced_used.db     reputation.db-shm        storage_usage.db-shm
bandwidth.db-shm         notifications.db         piece_spaced_used.db-shm reputation.db-wal        storage_usage.db-wal
bandwidth.db-wal         notifications.db-shm     piece_spaced_used.db-wal satellites.db            temp
blobs                    notifications.db-wal     pieceinfo.db             satellites.db-shm        trash
garbage                  orders.db                pieceinfo.db-shm         satellites.db-wal        used_serial.db
heldamount.db            orders.db-shm            pieceinfo.db-wal         secret.db                used_serial.db-shm
heldamount.db-shm        orders.db-wal            pricing.db               secret.db-shm            used_serial.db-wal
heldamount.db-wal        piece_expiration.db      pricing.db-shm           secret.db-wal
info.db                  piece_expiration.db-shm  pricing.db-wal           storage-dir-verification

Then I replace database from backup to config dir

and start node, still those stat still not showing

I also found some error logs

1-14T10:05:14Z	INFO	piecestore	download started	{"process": "storagenode", "Piece ID": "OZXIAYQIISA7V5ZEXQOZRTQQ5TDHFRYCFLBD57WJNTTMDNZADNJQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Offset": 1132288, "Size": 4864, "Remote Address": "10.69.69.201:16365"}
2024-01-14T10:05:14Z	ERROR	piecestore	download failed	{"process": "storagenode", "Piece ID": "OZXIAYQIISA7V5ZEXQOZRTQQ5TDHFRYCFLBD57WJNTTMDNZADNJQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Remote Address": "10.69.69.201:16365", "error": "untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup eu1.storj.io: operation was canceled", "errorVerbose": "untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup eu1.storj.io: operation was canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).VerifyOrderLimitSignature:140\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:62\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:616\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:251\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-01-14T10:05:14Z	INFO	piecestore	download started	{"process": "storagenode", "Piece ID": "64Y4KKRKLOKMGI5PKOQYSQUBYC3FV4E2LJVHQR3GWEPA62BX5NTA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 13824, "Remote Address": "10.69.69.201:34699"}
2024-01-14T10:05:14Z	INFO	piecestore	download started	{"process": "storagenode", "Piece ID": "BYXZDYA47S35KPRLOV5VBUEAG26SHSQ6FJY5SSYO2ARUYH2SYXTA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Offset": 728576, "Size": 4864, "Remote Address": "10.69.69.201:8176"}
2024-01-14T10:05:14Z	INFO	piecestore	download started	{"process": "storagenode", "Piece ID": "FPU2YWGUJZTZS7MZLYXSJ5BU2C3ZULNJTLNSHLVLEW46A3RGX6BA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 14080, "Remote Address": "10.69.69.201:49801"}
2024-01-14T10:05:14Z	INFO	piecestore	download started	{"process": "storagenode", "Piece ID": "N7UY4N4GNV7ZLS6645XVFRXGPCFTIYH2YDE2NNA4RWACCUVIRY3A", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Offset": 0, "Size": 256, "Remote Address": "10.69.69.201:16671"}
2024-01-14T10:05:14Z	INFO	piecestore	download started	{"process": "storagenode", "Piece ID": "AMNG3MZ3NPQVA3DVT37YFB6PXMRDTXBU346ZMIA3Y6DFU7UCDTCA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 39680, "Remote Address": "10.69.69.201:44296"}
2024-01-14T10:05:14Z	INFO	piecestore	download started	{"process": "storagenode", "Piece ID": "TTN55XNNMDZII7O642PAPQ5KPLNTXYX24H3A3RDCBZN37PLGO7BQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 7680, "Remote Address": "10.69.69.201:7736"}
2024-01-14T10:05:14Z	INFO	piecestore	download started	{"process": "storagenode", "Piece ID": "5AH6BFL4K4AODW4AWJ7EDVXW575QQZ7GSP3XHFOCIXOAB5BAMR6A", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET", "Offset": 0, "Size": 7680, "Remote Address": "10.69.69.201:5478"}
2024-01-14T10:05:14Z	ERROR	piecestore	download failed	{"process": "storagenode", "Piece ID": "5AH6BFL4K4AODW4AWJ7EDVXW575QQZ7GSP3XHFOCIXOAB5BAMR6A", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET", "Remote Address": "10.69.69.201:5478", "error": "untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup ap1.storj.io: operation was canceled", "errorVerbose": "untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup ap1.storj.io: operation was canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).VerifyOrderLimitSignature:140\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:62\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:616\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:251\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-01-14T10:05:14Z	INFO	piecestore	download started	{"process": "storagenode", "Piece ID": "SP2NPMACSLHO6BXMCQHSW33A6T5AXZGCLBJ7UVGCZIIL3N4N3OJA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Offset": 0, "Size": 41472, "Remote Address": "10.69.69.201:9963"}
2024-01-14T10:05:14Z	INFO	piecestore	download started	{"process": "storagenode", "Piece ID": "PYBTNSF6NICBOPMPXMGBCHSMEXEANYRM42OVQJDJZFPAHD7FS5KA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Offset": 0, "Size": 256, "Remote Address": "10.69.69.201:62568"}
2024-01-14T10:05:14Z	INFO	lazyfilewalker.used-space-filewalker	starting subprocess	{"process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-01-14T10:05:14Z	INFO	lazyfilewalker.used-space-filewalker	subprocess started	{"process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-01-14T10:05:14Z	ERROR	piecestore	download failed	{"process": "storagenode", "Piece ID": "BYXZDYA47S35KPRLOV5VBUEAG26SHSQ6FJY5SSYO2ARUYH2SYXTA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Remote Address": "10.69.69.201:8176", "error": "trust: rpc: tcp connector failed: rpc: context canceled", "errorVerbose": "trust: rpc: tcp connector failed: rpc: context canceled\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2024-01-14T10:05:14Z	ERROR	piecestore	download failed	{"process": "storagenode", "Piece ID": "64Y4KKRKLOKMGI5PKOQYSQUBYC3FV4E2LJVHQR3GWEPA62BX5NTA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Remote Address": "10.69.69.201:34699", "error": "trust: rpc: tcp connector failed: rpc: context canceled", "errorVerbose": "trust: rpc: tcp connector failed: rpc: context canceled\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2024-01-14T10:05:14Z	ERROR	piecestore	download failed	{"process": "storagenode", "Piece ID": "FPU2YWGUJZTZS7MZLYXSJ5BU2C3ZULNJTLNSHLVLEW46A3RGX6BA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Remote Address": "10.69.69.201:49801", "error": "untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup us1.storj.io: operation was canceled", "errorVerbose": "untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup us1.storj.io: operation was canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).VerifyOrderLimitSignature:140\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:62\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:616\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:251\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-01-14T10:05:14Z	ERROR	piecestore	download failed	{"process": "storagenode", "Piece ID": "N7UY4N4GNV7ZLS6645XVFRXGPCFTIYH2YDE2NNA4RWACCUVIRY3A", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Remote Address": "10.69.69.201:16671", "error": "trust: rpc: tcp connector failed: rpc: dial tcp 104.199.30.73:7777: operation was canceled", "errorVerbose": "trust: rpc: tcp connector failed: rpc: dial tcp 104.199.30.73:7777: operation was canceled\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2024-01-14T10:05:14Z	ERROR	piecestore	download failed	{"process": "storagenode", "Piece ID": "SP2NPMACSLHO6BXMCQHSW33A6T5AXZGCLBJ7UVGCZIIL3N4N3OJA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Remote Address": "10.69.69.201:9963", "error": "untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup eu1.storj.io: operation was canceled", "errorVerbose": "untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup eu1.storj.io: operation was canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).VerifyOrderLimitSignature:140\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:62\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:616\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:251\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-01-14T10:05:14Z	ERROR	piecestore	download failed	{"process": "storagenode", "Piece ID": "PYBTNSF6NICBOPMPXMGBCHSMEXEANYRM42OVQJDJZFPAHD7FS5KA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Remote Address": "10.69.69.201:62568", "error": "untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup eu1.storj.io: operation was canceled", "errorVerbose": "untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup eu1.storj.io: operation was canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).VerifyOrderLimitSignature:140\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:62\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:616\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:251\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-01-14T10:05:15Z	INFO	piecestore	upload started	{"process": "storagenode", "Piece ID": "7MCPC523XTEGL7MJCHWMIG7ETSC75L4MU3BU326OJ7LDIVI757FA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 4597066366976, "Remote Address": "10.69.69.201:19041"}
2024-01-14T10:05:15Z	INFO	piecestore	upload started	{"process": "storagenode", "Piece ID": "GUD5SEZ5HK6NYPKSHOKNBCVLOM4C2BWSKTQXQ3QVEK2RPJTHHLLA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 4597066366976, "Remote Address": "10.69.69.201:31506"}
2024-01-14T10:05:15Z	INFO	piecestore	upload started	{"process": "storagenode", "Piece ID": "LAYRUVGZHKQHVYZLYMIYIIBXRCPEEXJGJHDPYZFGIZWJJATG753A", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 4597066366976, "Remote Address": "10.69.69.201:46009"}
2024-01-14T10:05:15Z	INFO	piecestore	upload started	{"process": "storagenode", "Piece ID": "EHXNAD4AP2USKRXLKOVIYWFCGC3DN6IGSZXUU75KLZA2U4FGJJPA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 4597066366976, "Remote Address": "10.69.69.201:37266"}
2024-01-14T10:05:15Z	ERROR	piecestore	download failed	{"process": "storagenode", "Piece ID": "AMNG3MZ3NPQVA3DVT37YFB6PXMRDTXBU346ZMIA3Y6DFU7UCDTCA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Remote Address": "10.69.69.201:44296", "error": "untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup us1.storj.io: operation was canceled", "errorVerbose": "untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup us1.storj.io: operation was canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).VerifyOrderLimitSignature:140\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:62\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:616\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:251\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-01-14T10:05:15Z	ERROR	piecestore	download failed	{"process": "storagenode", "Piece ID": "TTN55XNNMDZII7O642PAPQ5KPLNTXYX24H3A3RDCBZN37PLGO7BQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Remote Address": "10.69.69.201:7736", "error": "untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup us1.storj.io: operation was canceled", "errorVerbose": "untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup us1.storj.io: operation was canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).VerifyOrderLimitSignature:140\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:62\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:616\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:251\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-01-14T10:05:15Z	INFO	piecestore	download started	{"process": "storagenode", "Piece ID": "N7UY4N4GNV7ZLS6645XVFRXGPCFTIYH2YDE2NNA4RWACCUVIRY3A", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Offset": 0, "Size": 256, "Remote Address": "10.69.69.201:16671"}


root@storagenode-8569cd844f-mc9fs:/app# nslookup eu1.storj.io
Server:         10.152.183.10
Address:        10.152.183.10#53

Non-authoritative answer:
Name:   eu1.storj.io
Address: 104.199.30.73
Name:   eu1.storj.io
Address: 34.159.134.91

root@storagenode-8569cd844f-mc9fs:/app# nslookup eu1.storj.io 8.8.8.8
Server:         8.8.8.8
Address:        8.8.8.8#53

Non-authoritative answer:
Name:   eu1.storj.io
Address: 104.199.30.73
Name:   eu1.storj.io
Address: 34.159.134.91

root@storagenode-8569cd844f-mc9fs:/app# nslookup storj.nv4re.com 8.8.8.8
Server:         8.8.8.8
Address:        8.8.8.8#53

Non-authoritative answer:
storj.nv4re.com canonical name = nv4re.com.
Name:   nv4re.com
Address: 124.121.84.224
7

Then you need either move databases to SSD: How to move DB’s to SSD on Docker or wait more time, if the disk is slow, it could take several minutes to load a dashboard. You can also remove cache for this address, the browser should load latest version of the dashboard.

the usual “context canceled” errors - your node was slower than others for this customer.
Your node cannot win all races - it cannot be close to every customer in the world, so this is normal.

1 Like