Hi Alexey, yep, all is clean as per the other tests I have done. Shall I then let it run even if I see those errors?
UPDATE: I rebuilt version v1.39.6 as per your instructions in Operating a node on a 32bit system and I see the pieces coming!!!
Hi Alexey, yep, all is clean as per the other tests I have done. Shall I then let it run even if I see those errors?
UPDATE: I rebuilt version v1.39.6 as per your instructions in Operating a node on a 32bit system and I see the pieces coming!!!
UPDATE: updated to v1.40.1 and it works as well
One question: if I use a file to store the logs, is there a config to make it rolling?
No. You should use something like logrotate to configure that:
1 Like
Hi all,
First : I’m French, and my English is a little “average”.
I Explain my problem, i’m running a Storj node since less one year (nine months precisly).
Et je rencontre quelques problèmes depuis plus de 4 ou 5 months or so : My logs contain these informations recursivly :
2022-01-28T00:01:14.462+0100 ERROR contact:service ping satellite failed {"Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "attempts": 1, "error": "ping satellite: failed to dial storage node (ID: 12MTuu3S4QwvphTJZ88qjKyFqXZ7Ajayt6JFBQSY7JFw65koP9D) at address home.sepu-design.com:28967: rpc: dial tcp 77.129.145.66:28967: i/o timeout", "errorVerbose": "ping satellite: failed to dial storage node (ID: 12MTuu3S4QwvphTJZ88qjKyFqXZ7Ajayt6JFBQSY7JFw65koP9D) at address home.sepu-design.com:28967: rpc: dial tcp 77.129.145.66:28967: i/o timeout\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2022-01-28T00:01:35.686+0100 ERROR contact:service ping satellite failed {"Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "attempts": 2, "error": "ping satellite: failed to dial storage node (ID: 12MTuu3S4QwvphTJZ88qjKyFqXZ7Ajayt6JFBQSY7JFw65koP9D) at address home.sepu-design.com:28967: rpc: dial tcp 77.129.145.66:28967: i/o timeout", "errorVerbose": "ping satellite: failed to dial storage node (ID: 12MTuu3S4QwvphTJZ88qjKyFqXZ7Ajayt6JFBQSY7JFw65koP9D) at address home.sepu-design.com:28967: rpc: dial tcp 77.129.145.66:28967: i/o timeout\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2022-01-28T00:01:37.850+0100 ERROR contact:service ping satellite failed {"Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "attempts": 3, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:138\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2022-01-28T00:02:02.062+0100 ERROR contact:service ping satellite failed {"Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "attempts": 4, "error": "ping satellite: failed to dial storage node (ID: 12MTuu3S4QwvphTJZ88qjKyFqXZ7Ajayt6JFBQSY7JFw65koP9D) at address home.sepu-design.com:28967: rpc: dial tcp 77.129.145.66:28967: i/o timeout", "errorVerbose": "ping satellite: failed to dial storage node (ID: 12MTuu3S4QwvphTJZ88qjKyFqXZ7Ajayt6JFBQSY7JFw65koP9D) at address home.sepu-design.com:28967: rpc: dial tcp 77.129.145.66:28967: i/o timeout\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
...
2022-01-28T08:13:46.117+0100 ERROR contact:service ping satellite failed {"Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "attempts": 6, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:138\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2022-01-28T08:14:18.313+0100 ERROR contact:service ping satellite failed {"Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "attempts": 7, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:138\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2022-01-28T08:15:42.565+0100 ERROR contact:service ping satellite failed {"Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "attempts": 8, "error": "ping satellite: failed to dial storage node (ID: 12MTuu3S4QwvphTJZ88qjKyFqXZ7Ajayt6JFBQSY7JFw65koP9D) at address home.sepu-design.com:28967: rpc: dial tcp 77.129.145.66:28967: i/o timeout", "errorVerbose": "ping satellite: failed to dial storage node (ID: 12MTuu3S4QwvphTJZ88qjKyFqXZ7Ajayt6JFBQSY7JFw65koP9D) at address home.sepu-design.com:28967: rpc: dial tcp 77.129.145.66:28967: i/o timeout\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2022-01-28T08:17:50.766+0100 ERROR contact:service ping satellite failed {"Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "attempts": 9, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:138\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2022-01-28T08:22:06.973+0100 ERROR contact:service ping satellite failed {"Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "attempts": 10, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:138\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
(I have masked the “INFO” logs wich is not util)
All node can contact me with no issue minus this 2 nodes. (europe-north-1.tardigrade.io and ap1.storj.io)
The stats of my node for one month is clearly “good” but can to be better with this 2 missing nodes.
screenshoot :
I’m monitoring my internet connection, and it’s up and appear to not shut down since several months. Node undulated, firewall all open for the node, Identity is ok (6 files, response ok) ports open, this situation is unexplainable. can you help me ?
Your node is offline at least for the one satellite. This could happen if your router, firewall or your ISP are blocking traffic to/from the node.
Please, check your firewall rules:
- outbound: should not be limited and should not have any rules. If it has - you need to add a rule to allow any traffic from your node to any destination.
- inbound: should allow TCP/UDP traffic to your node (local IP inside your LAN and node’s port, by default 28967) from any source. You should not specify in the source any port or IP, there should be literally any.
- outbound: should not be limited and should not have any rules. If it has - you need to add a rule to allow any traffic from your node to any destination.
No blocking Rules. The node is whitelisted on my firewall.
- inbound: should allow TCP/UDP traffic to your node (local IP inside your LAN and node’s port, by default 28967) from any source. You should not specify in the source any port or IP, there should be literally any.
I’ve checked my Firewall logs, and no errors tags, no blocking tags from theses nodes.
And this is the “traffic” related to ap1 from the month start :
Since of 6th January, no inbound traffic, and sometimes, a little outbound traffic.
my node was disqualified since : Wed, 17 Nov 2021 13:37:01 GMT on europe-north-1 with no reason.
no news about that ?
What news do you expect? If your node is disqualified - it can be only for lost files. This is unrecoverable state, the disqualified node will remain disqualified.
Please, note - every satellite make an own decision regading node, so if it’s disqualified not on all satellites - the remained will still pay you.
If your node is not disqualified - then the online score will recover after 30 days online. So, if your node is still offline - you need to fix this issue.
I know that. But, why my node is diqualified ? It’s non-logic, all other node is ok, and one is not. Why ?
Thank for your support, and have a Nice Day !
Because of missed, not available or corrupted files I can guess.
Please, use this article to figure that out exactly: https://support.storj.io/hc/en-us/articles/4403035941780-Why-is-my-node-disqualified-
I checked, no corrupted files… i don’t receive files from theses node… Never.
Did you search for the errors? Or your logs are lost?
Please, provide the last 3 lines for each check from the article.
First command
sls “GET_AUDIT|GET_REPAIR” “C:\Program Files\Storj\Storage Node\storagenode.log” | sls failed
Get this return (just one, no more) :
storagenode.log:657085:2022-01-29T07:26:10.633+0100 ERROR piecestore download
failed {“Piece ID”: “AWY4NW6BNZ2RS56PU4ANO7OVAZJPZGTLOA5E2Q4YEZIA3KDYE2QQ”, “Satellite ID”:
“12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET_REPAIR”, “error”: “file does not exist”,
“errorVerbose”: “file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:73\n\tstorj.io/storj/storagenode/piecestore.
(*Endpoint).Download:545\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mu
x).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleR
PC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.
io/drpc/drpcctx.(*Tracker).track:52”}
And the next commands :
(sls “GET_AUDIT|GET_REPAIR” “C:\Program Files\Storj\Storage Node\storagenode.log” | sls started).Count
(sls “GET_AUDIT|GET_REPAIR” “C:\Program Files\Storj\Storage Node\storagenode.log” | sls downloaded).Count
Number of audit Started : 2722 (from 01/01/2022) > 51150 (since starting the node to 01/01/2022)
Number of audit Finished: 2721 (from 01/01/2022) > 50986 (since starting the node to 01/01/2022)
Ratio of started to finish equal : 99,69% for all time
This only prove that your node lost files.
Seems it also have corrupted files (this type is not logged on node’s side).
I would like to suggest to check time between audit started and audit finished for each piece audited by that satellite.
To use this script under Windows you need to install WSL and Ubuntu. Your disks will appear under /mnt automatically. You also need to install jq
sudo apt update
sudo apt install jq -y
Uptime doesn’t matter too much, more important audit score and online score.
If the audit score fall below 60%, your node become disqualified.
If the online score is 0% for a month (your node is offline for more than 30 days), it will be disqualified.
Hi Alexey, how are you today. Last update. I’ve changed nothing on my node, and… ap1 seem’s to be able to discuss with my node normaly since few days. Online percentage grow up, and dl/ul grow up too. (it falled to 0% and grow up after that)
This is… strange, no ?
The online account is calculated for a 30-day window. Your node moves in time out of the window where the offline event occurred and slowly recovers, as intended.
Of course, this could only be possible when the problem was fixed.
The same Problem in 1.77.3 . Whats wrong?
Is this helping:
2023-04-25T20:06:16.086+0200 ERROR contact:service ping satellite failed {“Satellite ID”: “12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo”, “attempts”: 1, “error”: “ping satellite: check-in ratelimit: node rate limited by id”, “errorVerbose”: “ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:143\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:102\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”}
Windows-GUI-Installation.
Thanks and a nice evening!
you have other error, satellite cant connect your node. so you have connectivity problem
2 Likes