Moved from Windows to Synology and ALMOST have it going

Got almost done but these errors persist. Does this mean maybe that the ports are closed on the firewall?

2021-05-27T22:07:47.958824664Z 2021-05-27T22:07:47.958Z ERROR contact:service ping satellite failed {“Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “attempts”: 4, “error”: “ping satellite error: check-in ratelimit: node rate limited by id”, “errorVerbose”: “ping satellite error: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:138\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

Well, I’m pretty sure it has nothing to do with the ports not being open behind the firewall. When I start the node, I check the port and it opens up just fine from the outside. @Alexey can you take a look at the error and see what you think? I’ve found several similar problems but haven’t been successful finding this one. @BrightSilence Can you take a look at this? Took me 5 days to transfer the data and I really need to get this thing going again or turn the original back on again…

Any other folks have idears @baker maybe?

I haven’t seen this error before. The language to me seems like it is a restriction on the satellite side, but maybe your node is flooding the sat with ping attempts. (All speculation on my part). Is it preventing your node from running? Do you see this on all satellites? If not, I would be inclined to let it run to at least keep your uptime with the other satellites. Or perhaps it will eventually sort itself out. Is the node showing as online in the dashboard?

This happens for all satellites and the dash says the node is offline… I’ll let it run for a bit and see if anything changes.

This is a long shot, but the original node is definitely not running any more, correct? I’m sure you thought of this, but maybe it got restarted by accident?

Noper, it’s been off for 5 days now.

DNS is working, ports are open, identity says its OK… Just don’t know what else to look at…

I can tell you for sure, I’m leaning towards running a few smaller node instances than having large sized ones… Moving the data around is difficult at the 10+ TB level

Well, I turned the old node back on and it logged a few similar errors and then started on it’s merry way to catching up… This I’ll have to try again another day but maybe use rsync even if it’s so slow. I have another node I’m attempting the same migration with using Resilio which syncs really fast. That particular node is almost 16tb so I’m really curious how it’s going to work if at all…

Can you post a little more of the log? The node checking in too often could be caused by it restarting constantly due to another fatal error. I think you posted a symptom, not the cause.

1 Like

This is from the start of the log and it just keeps repeating

2021-05-28T02:18:16.514023513Z 2021-05-28T02:18:16.513Z INFO Operator email {“Address”: “xxxxxx@protonmail.com”}
2021-05-28T02:18:16.514736736Z 2021-05-28T02:18:16.514Z INFO Operator wallet {“Address”: “xxxxxxxxx”}
2021-05-28T02:18:16.870953033Z 2021-05-28T02:18:16.870Z INFO Telemetry enabled {“instance ID”: “1qNB4xxxxxxxxx5eYVoi2NhZwTtBmXipqfVGXxVLQS”}
2021-05-28T02:18:16.969163403Z 2021-05-28T02:18:16.968Z INFO db.migration Database Version {“version”: 51}
2021-05-28T02:18:17.522623690Z 2021-05-28T02:18:17.522Z INFO preflight:localtime start checking local system clock with trusted satellites’ system clock.
2021-05-28T02:18:18.035557359Z 2021-05-28T02:18:18.035Z INFO preflight:localtime local system clock is in sync with trusted satellites’ system clock.
2021-05-28T02:18:18.036054929Z 2021-05-28T02:18:18.035Z INFO bandwidth Performing bandwidth usage rollups
2021-05-28T02:18:18.036821853Z 2021-05-28T02:18:18.036Z INFO Node 1qNB46eQGPxxxxxxxxxxoi2NhZwTtBmXipqfVGXxVLQS started
2021-05-28T02:18:18.036941217Z 2021-05-28T02:18:18.036Z INFO Public server started on [::]:7777
2021-05-28T02:18:18.037028193Z 2021-05-28T02:18:18.036Z INFO Private server started on 127.0.0.1:7778
2021-05-28T02:18:18.037908631Z 2021-05-28T02:18:18.037Z INFO trust Scheduling next refresh {“after”: “4h7m21.112701333s”}
2021-05-28T02:18:18.232711249Z 2021-05-28T02:18:18.232Z ERROR contact:service ping satellite failed {“Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “attempts”: 1, “error”: “ping satellite error: check-in ratelimit: node rate limited by id”, “errorVerbose”: “ping satellite error: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:138\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-05-28T02:18:18.456118244Z 2021-05-28T02:18:18.455Z ERROR contact:service ping satellite failed {“Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “attempts”: 1, “error”: “ping satellite error: check-in ratelimit: node rate limited by id”, “errorVerbose”: “ping satellite error: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:138\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-05-28T02:18:18.459795549Z 2021-05-28T02:18:18.459Z ERROR contact:service ping satellite failed {“Satellite ID”: “12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo”, “attempts”: 1, “error”: “ping satellite error: failed to dial storage node (ID: 1qNB46eQGPC8y1XUenf4AQ5eYVoi2NhZwTtBmXipqfVGXxVLQS) at address 76.214.xxx.xx:28967: rpc: read tcp 10.102.7.4:36318->76.214.xxx.xx:28967: read: connection reset by peer”, “errorVerbose”: “ping satellite error: failed to dial storage node (ID: 1qNB46eQGPC8y1XUenf4AQ5eYVoi2NhZwTtBmXipqfVGXxVLQS) at address 76.214.xxx.xx:28967: rpc: read tcp 10.102.7.4:36318->76.214.xxx.xx:28967: read: connection reset by peer\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-05-28T02:18:18.601552847Z 2021-05-28T02:18:18.601Z ERROR contact:service ping satellite failed {“Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “attempts”: 1, “error”: “ping satellite error: check-in ratelimit: node rate limited by id”, “errorVerbose”: “ping satellite error: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:138\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-05-28T02:18:18.620515693Z 2021-05-28T02:18:18.620Z ERROR contact:service ping satellite failed {“Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “attempts”: 1, “error”: “ping satellite error: check-in ratelimit: node rate limited by id”, “errorVerbose”: “ping satellite error: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:138\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-05-28T02:18:19.773538546Z 2021-05-28T02:18:19.773Z ERROR contact:service ping satellite failed {“Satellite ID”: “12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo”, “attempts”: 2, “error”: “ping satellite error: failed to dial storage node (ID: 1qNB46eQGPC8y1XUenf4AQ5eYVoi2NhZwTtBmXipqfVGXxVLQS) at address 76.214.xxx.xx:28967: rpc: EOF”, “errorVerbose”: “ping satellite error: failed to dial storage node (ID: 1qNB46eQGPC8y1XUenf4AQ5eYVoi2NhZwTtBmXipqfVGXxVLQS) at address 76.214.xxx.xx:28967: rpc: EOF\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-05-28T02:18:19.983825608Z 2021-05-28T02:18:19.983Z ERROR contact:service ping satellite failed {“Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “attempts”: 2, “error”: “ping satellite error: failed to dial storage node (ID: 1qNB46eQGPC8y1XUenf4AQ5eYVoi2NhZwTtBmXipqfVGXxVLQS) at address 76.214.xxx.xx:28967: rpc: EOF”, “errorVerbose”: “ping satellite error: failed to dial storage node (ID: 1qNB46eQGPC8y1XUenf4AQ5eYVoi2NhZwTtBmXipqfVGXxVLQS) at address 76.214.xxx.xx:28967: rpc: EOF\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-05-28T02:18:20.217523570Z 2021-05-28T02:18:20.217Z ERROR contact:service ping satellite failed {“Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “attempts”: 2, “error”: “ping satellite error: failed to dial storage node (ID: 1qNB46eQGPC8y1XUenf4AQ5eYVoi2NhZwTtBmXipqfVGXxVLQS) at address 76.214.xxx.xx:28967: rpc: EOF”, “errorVerbose”: “ping satellite error: failed to dial storage node (ID: 1qNB46eQGPC8y1XUenf4AQ5eYVoi2NhZwTtBmXipqfVGXxVLQS) at address 76.214.xxx.xx:28967: rpc: EOF\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-05-28T02:18:21.246125097Z 2021-05-28T02:18:21.245Z ERROR contact:service ping satellite failed {“Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “attempts”: 2, “error”: “ping satellite error: failed to dial storage node (ID: 1qNB46eQGPC8y1XUenf4AQ5eYVoi2NhZwTtBmXipqfVGXxVLQS) at address 76.214.xxx.xx:28967: rpc: read tcp 10.108.1.18:43766->76.214.xxx.xx:28967: read: connection reset by peer”, “errorVerbose”: “ping satellite error: failed to dial storage node (ID: 1qNB46eQGPC8y1XUenf4AQ5eYVoi2NhZwTtBmXipqfVGXxVLQS) at address 76.214.xxx.xx:28967: rpc: read tcp 10.108.1.18:43766->76.214.xxx.xx:28967: read: connection reset by peer\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-05-28T02:18:22.121204963Z 2021-05-28T02:18:22.120Z ERROR contact:service ping satellite failed {“Satellite ID”: “12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo”, “attempts”: 3, “error”: “ping satellite error: failed to dial storage node (ID: 1qNB46eQGPC8y1XUenf4AQ5eYVoi2NhZwTtBmXipqfVGXxVLQS) at address 76.214.xxx.xx:28967: rpc: EOF”, “errorVerbose”: “ping satellite error: failed to dial storage node (ID: 1qNB46eQGPC8y1XUenf4AQ5eYVoi2NhZwTtBmXipqfVGXxVLQS) at address 76.214.xxx.xx:28967: rpc: EOF\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-05-28T02:18:22.153066503Z 2021-05-28T02:18:22.152Z ERROR contact:service ping satellite failed {“Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “attempts”: 3, “error”: “ping satellite error: check-in ratelimit: node rate limited by id”, “errorVerbose”: “ping satellite error: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:138\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-05-28T02:18:22.631045910Z 2021-05-28T02:18:22.630Z ERROR contact:service ping satellite failed {“Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “attempts”: 3, “error”: “ping satellite error: check-in ratelimit: node rate limited by id”, “errorVerbose”: “ping satellite error: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:138\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-05-28T02:18:22.676249624Z 2021-05-28T02:18:22.675Z ERROR contact:service ping satellite failed {“Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “attempts”: 1, “error”: “ping satellite error: failed to dial storage node (ID: 1qNB46eQGPC8y1XUenf4AQ5eYVoi2NhZwTtBmXipqfVGXxVLQS) at address 76.214.xxx.xx:28967: rpc: EOF”, “errorVerbose”: “ping satellite error: failed to dial storage node (ID: 1qNB46eQGPC8y1XUenf4AQ5eYVoi2NhZwTtBmXipqfVGXxVLQS) at address 76.214.xxx.xx:28967: rpc: EOF\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-05-28T02:18:23.733078099Z 2021-05-28T02:18:23.732Z ERROR contact:service ping satellite failed {“Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “attempts”: 2, “error”: “ping satellite error: failed to dial storage node (ID: 1qNB46eQGPC8y1XUenf4AQ5eYVoi2NhZwTtBmXipqfVGXxVLQS) at address 76.214.xxx.xx:28967: rpc: EOF”, “errorVerbose”: “ping satellite error: failed to dial storage node (ID: 1qNB46eQGPC8y1XUenf4AQ5eYVoi2NhZwTtBmXipqfVGXxVLQS) at address 76.214.xxx.xx:28967: rpc: EOF\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-05-28T02:18:23.734079364Z 2021-05-28T02:18:23.733Z ERROR contact:service ping satellite failed {“Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “attempts”: 3, “error”: “ping satellite error: check-in ratelimit: node rate limited by id”, “errorVerbose”: “ping satellite error: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:138\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-05-28T02:18:26.309281926Z 2021-05-28T02:18:26.308Z ERROR contact:service ping satellite failed {“Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “attempts”: 4, “error”: “ping satellite error: check-in ratelimit: node rate limited by id”, “errorVerbose”: "ping satellite error: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:138\n\tst

This suggest that your external address is not accessible, either because it’s changed but your DDNS is not updated or you use an IP directly and now you need to update it.
The other possibility is your local IP has changed but port forwarding rule is not updated to use it. You can configure the static local IP either on your router (recommended) or directly on your Synology (not recommended and I’m not sure that Synology will allow to do so).

Hi @Alexey and thanks for giving me a hand. I had changed the internal IP address on the existing rule in the firewall to the new machines new internal IP. When that didn’t work I deleted the rule entirely and made a new one. When the node started, I used a port checker and 28967 opened to the net. Turn off the node, and the port closed as expected. After throwing in the towel and turning the old node back on, I changed only the internal IP address on the rule back to the old and the node started working. So I know the rule is behaving as expected.

The only thing I could think of is a permissions issue or the Synology is blocking the port internally. Your thoughts?

Then the port would be closed too. But maybe Synology has some firewall too and you should allow an incoming connections to the node’s port and allow any outgoing connections from the Synology.
The error points to some routing problems or working firewall like here:

The error like

was implemented recently to reduce ddos’ing from incorrectly configured nodes like when the firewall allows outgoing connections to ping the satellite, but unable to receive an incoming ping because incoming traffic is blocked by source for example (you should not specify source port/address in the firewall rule, you should specify only target port and IP (if needed)).
For outgoing connections the source and target should be any (i.e. no blocking of any outgoing traffic).

1 Like

Hi @Alexey, @BrightSilence , and @baker ,I finally figured out where I went wrong and I thought I’d share. I’ve setup around 20 nodes over the years and not too long ago there was the change installing Docker. I didn’t do the new step (docker run --rm -e SETUP=“true” ) and of course it failed starting. Just goes to show you, RTFM every now and then… LOL.

Thanks again as always for helping us boneheads keep things going.

4 Likes