“error”: “untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup us1.storj.io: operation was canceled

[shaltay]

Здравствуйте. У меня подобная проблема. Ни с того ни с сего начала перезагружаться одна нода. До этого год с лишним работала нормально.
Пишет ошибку
2023-05-23T08:33:30.786+0500 ERROR piecestore download failed {“Process”: “storagenode”, “Piece ID”: “J2BDPTOAM2ZYION6MZL42AJSNO7HMZMXYPHLQG2HFK5LSTMVAGEA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Remote Address”: “172.18.0.1:48722”, “error”: “untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup us1.storj.io: operation was canceled”, “errorVerbose”: “untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup us1.storj.io: operation was canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).VerifyOrderLimitSignature:140\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:62\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:599\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:251\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35”}
Нода работает под Docker на Synology.

[Alexey]

Здравствуйте @shaltay!
Очень похоже на проблемы с сетью

@shaltay

tcp connector failed: rpc: dial tcp: lookup us1.storj.io: operation was canceled

видимо есть подобные ошибки и для других сателлитов.
Попробуйте сконфигурировать другой DNS сервер в DHCP секции вашего роутера, например, 8.8.8.8. Правда это сработает только после перезагрузки Synology или отключения/включения ethernet кабеля в Synology.
Другой вариант, попробовать сконфигурировать другой DNS прямо на Synology.
Или самый простой - добавьте опцию --dns 8.8.8.8 в вашу команду docker run до имени образа (потому что это опция docker, а не узла).

[shaltay]

Другие ноды работают. Но на всякий прописал принудительно гугловский и ip4 и ip6 dns адреса.
Логи с проблемной:

[Alexey]

Ошибки ratelimit следствие предыдущих неудачных попыток зарегистрироваться на сателлите, либо потому что внешний адрес и порт недоступны, либо была какая-то ошибка при регистрации. В приведённом скриншоте предыдущих ошибок не видно.

• Проверьте, пожалуйста, identity для этого узла: Step 5. Create an Identity - Storj Node Operator Docs
• также убедитесь, что в опции ADDRESS указан актуальный адрес и если это DDNS hostname, что он ресолвится в публичный IP, совпадающий с WAN IP на вашем роутере:

nslookup your.ddns.hostname 8.8.8.8

должен вернуть такой же IP как на Open Port Check Tool - Test Port Forwarding on Your Router и совпадающий с WAN IP на роутере.

P.S. Фото текста не так полезно, как сам текст , пожалуйста, в следующий раз скопируйте текст и вставьте его тут между двумя новыми строчками с тремя backticks, вот так:

```
текст логов
```

приватную информацию можно замаскировать или удалить.

[shaltay]

ash-4.4# nslookup ***.ru 8.8.8.8
Server: 8.8.8.8
Address: 8.8.8.8#53

Non-authoritative answer:
Name: ***.ru
Address: ..170.140

ash-4.4# nslookup ***.synology.me 8.8.8.8
Server: 8.8.8.8
Address: 8.8.8.8#53

Non-authoritative answer:
Name: ***.synology.me
Address: ..170.140
Name: ***.synology.me
Address: ::902b:ca7:211:32ff:feea:af63

Файлы находятся в папке ноды, нуть прописан правильно:

ash-4.4# grep -c BEGIN ca.cert
2

ash-4.4# grep -c BEGIN identity.cert
3

2023-05-23 11:46:38,554 INFO spawned: 'storagenode-updater' with pid 64
2023-05-23T11:46:38.565+0500	INFO	Configuration loaded	{"Process": "storagenode-updater", "Location": "/app/config/config.yaml"}
2023-05-23T11:46:38.565+0500	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "operator.wallet"}
2023-05-23T11:46:38.565+0500	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "operator.email"}
2023-05-23T11:46:38.565+0500	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "server.address"}
2023-05-23T11:46:38.565+0500	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "server.private-address"}
2023-05-23T11:46:38.565+0500	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "operator.wallet-features"}
2023-05-23T11:46:38.565+0500	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "storage.allocated-bandwidth"}
2023-05-23T11:46:38.565+0500	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "contact.external-address"}
2023-05-23T11:46:38.565+0500	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "storage.allocated-disk-space"}
2023-05-23T11:46:38.566+0500	INFO	Anonymized tracing enabled	{"Process": "storagenode-updater"}
2023-05-23T11:46:38.566+0500	INFO	Running on version	{"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.78.3"}
2023-05-23T11:46:38.566+0500	INFO	Downloading versions.	{"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2023-05-23T11:46:38.606+0500	INFO	Configuration loaded	{"Process": "storagenode", "Location": "/app/config/config.yaml"}
2023-05-23T11:46:38.606+0500	INFO	Anonymized tracing enabled	{"Process": "storagenode"}
2023-05-23T11:46:38.607+0500	INFO	Operator email	{"Process": "storagenode", "Address": "**********************"}
2023-05-23T11:46:38.607+0500	INFO	Operator wallet	{"Process": "storagenode", "Address": "0x****************************"}
2023-05-23T11:46:39.086+0500	INFO	server	kernel support for tcp fast open unknown	{"Process": "storagenode"}
2023-05-23T11:46:39.173+0500	INFO	Current binary version	{"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.78.3"}
2023-05-23T11:46:39.173+0500	INFO	Version is up to date	{"Process": "storagenode-updater", "Service": "storagenode"}
2023-05-23T11:46:39.180+0500	INFO	Current binary version	{"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.78.3"}
2023-05-23T11:46:39.180+0500	INFO	Version is up to date	{"Process": "storagenode-updater", "Service": "storagenode-updater"}
2023-05-23T11:46:39.840+0500	INFO	Telemetry enabled	{"Process": "storagenode", "instance ID": "1MBwuz3R4AAXfhvfHFFLjtPyUFetYJnHcc94PZEYQN6kEJqTW4"}
2023-05-23T11:46:39.840+0500	INFO	Event collection enabled	{"Process": "storagenode", "instance ID": "1MBwuz3R4AAXfhvfHFFLjtPyUFetYJnHcc94PZEYQN6kEJqTW4"}
2023-05-23 11:46:39,840 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-05-23 11:46:39,840 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-05-23 11:46:39,840 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-05-23T11:46:39.863+0500	INFO	db.migration	Database Version	{"Process": "storagenode", "version": 54}
2023-05-23T11:46:40.726+0500	INFO	preflight:localtime	start checking local system clock with trusted satellites' system clock.	{"Process": "storagenode"}
2023-05-23T11:46:41.740+0500	INFO	preflight:localtime	local system clock is in sync with trusted satellites' system clock.	{"Process": "storagenode"}
2023-05-23T11:46:41.741+0500	INFO	Node 1MBwuz3R4AAXfhvfHFFLjtPyUFetYJnHcc94PZEYQN6kEJqTW4 started	{"Process": "storagenode"}
2023-05-23T11:46:41.741+0500	INFO	Public server started on [::]:28967	{"Process": "storagenode"}
2023-05-23T11:46:41.741+0500	INFO	trust	Scheduling next refresh	{"Process": "storagenode", "after": "4h57m7.284417984s"}
2023-05-23T11:46:41.741+0500	INFO	Private server started on 127.0.0.1:7778	{"Process": "storagenode"}
2023-05-23T11:46:41.741+0500	INFO	bandwidth	Performing bandwidth usage rollups	{"Process": "storagenode"}
2023-05-23T11:46:41.741+0500	INFO	pieces:trash	emptying trash started	{"Process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB"}
2023-05-23T11:46:41.742+0500	INFO	piecestore	download started	{"Process": "storagenode", "Piece ID": "TLTAUVI33MDRWCRE2RLIGWAR4OLNUHLTGLR5OHRAJICPQV34YT7Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 97536, "Remote Address": "172.18.0.1:54562"}
2023-05-23T11:46:41.743+0500	ERROR	piecestore	download failed	{"Process": "storagenode", "Piece ID": "TLTAUVI33MDRWCRE2RLIGWAR4OLNUHLTGLR5OHRAJICPQV34YT7Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Remote Address": "172.18.0.1:54562", "error": "untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup us1.storj.io: operation was canceled", "errorVerbose": "untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup us1.storj.io: operation was canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).VerifyOrderLimitSignature:140\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:62\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:599\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:251\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2023-05-23T11:46:41.797+0500	INFO	pieces:trash	emptying trash started	{"Process": "storagenode", "Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo"}
2023-05-23T11:46:41.817+0500	INFO	pieces:trash	emptying trash started	{"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2023-05-23T11:46:41.905+0500	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "attempts": 1, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:143\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:102\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-05-23T11:46:42.438+0500	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 1, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:143\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:102\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-05-23T11:46:42.758+0500	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "attempts": 1, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:143\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:102\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-05-23T11:46:42.941+0500	INFO	pieces:trash	emptying trash started	{"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2023-05-23T11:46:43.013+0500	INFO	pieces:trash	emptying trash started	{"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2023-05-23T11:46:43.070+0500	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "attempts": 2, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:143\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:102\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-05-23T11:46:43.198+0500	INFO	pieces:trash	emptying trash started	{"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2023-05-23T11:46:44.133+0500	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 2, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:143\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:102\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-05-23T11:46:44.761+0500	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "attempts": 2, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:143\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:102\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-05-23T11:46:45.241+0500	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "attempts": 3, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:143\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:102\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-05-23T11:46:46.813+0500	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 3, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:143\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:102\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-05-23T11:46:47.765+0500	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "attempts": 3, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:143\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:102\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-05-23T11:46:49.400+0500	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "attempts": 4, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:143\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:102\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-05-23T11:46:51.502+0500	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 4, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:143\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:102\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-05-23T11:46:52.777+0500	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "attempts": 4, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:143\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:102\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-05-23T11:46:53.250+0500	INFO	piecestore	download started	{"Process": "storagenode", "Piece ID": "2YEVN7KA4GYVK66ADCWKGMFXQ7ERT5IKY36HKOUULA5G22RLTY5A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 2100992, "Size": 2560, "Remote Address": "172.18.0.1:54688"}
2023-05-23T11:46:53.329+0500	ERROR	piecestore	download failed	{"Process": "storagenode", "Piece ID": "2YEVN7KA4GYVK66ADCWKGMFXQ7ERT5IKY36HKOUULA5G22RLTY5A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Remote Address": "172.18.0.1:54688", "error": "trust: rpc: tcp connector failed: rpc: dial tcp 34.150.199.48:7777: operation was canceled", "errorVerbose": "trust: rpc: tcp connector failed: rpc: dial tcp 34.150.199.48:7777: operation was canceled\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2023-05-23T11:46:57.561+0500	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "attempts": 5, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:143\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:102\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-05-23T11:47:00.192+0500	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 5, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:143\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:102\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-05-23T11:47:00.292+0500	INFO	piecestore	download started	{"Process": "storagenode", "Piece ID": "OHTKBB7U2XM6JX2SKSI2JXTKEHA5OU6M7N6M55WVS255NMYSMBRQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Offset": 2216448, "Size": 2304, "Remote Address": "172.18.0.1:54722"}
2023-05-23T11:47:00.322+0500	INFO	piecestore	download started	{"Process": "storagenode", "Piece ID": "5R7JLT4XNHIOWEJZCUXS3XCCT3ZURBA5GEWJXGAFIUA3VSQTORMQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Offset": 1837056, "Size": 2560, "Remote Address": "172.18.0.1:54724"}
2023-05-23T11:47:00.412+0500	ERROR	piecestore	download failed	{"Process": "storagenode", "Piece ID": "OHTKBB7U2XM6JX2SKSI2JXTKEHA5OU6M7N6M55WVS255NMYSMBRQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Remote Address": "172.18.0.1:54722", "error": "trust: rpc: tcp connector failed: rpc: context canceled", "errorVerbose": "trust: rpc: tcp connector failed: rpc: context canceled\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2023-05-23T11:47:00.451+0500	ERROR	piecestore	download failed	{"Process": "storagenode", "Piece ID": "5R7JLT4XNHIOWEJZCUXS3XCCT3ZURBA5GEWJXGAFIUA3VSQTORMQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Remote Address": "172.18.0.1:54724", "error": "untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup eu1.storj.io: operation was canceled", "errorVerbose": "untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup eu1.storj.io: operation was canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).VerifyOrderLimitSignature:140\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:62\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:599\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:251\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2023-05-23T11:47:01.198+0500	INFO	piecestore	download started	{"Process": "storagenode", "Piece ID": "WY7WOZES2L3WAJULSLBO6WR4NWPROCPXNVBE25INOVMJ2V7NHXMA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Offset": 651008, "Size": 2560, "Remote Address": "172.18.0.1:54724"}
2023-05-23T11:47:01.349+0500	INFO	piecestore	download started	{"Process": "storagenode", "Piece ID": "BCBZODLRKFM46INHVUNNPS6IUK52HOEJJ75FQR7YNRXTU7XIKD7Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 6144, "Remote Address": "172.18.0.1:54734"}
2023-05-23T11:47:01.364+0500	ERROR	piecestore	download failed	{"Process": "storagenode", "Piece ID": "WY7WOZES2L3WAJULSLBO6WR4NWPROCPXNVBE25INOVMJ2V7NHXMA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Remote Address": "172.18.0.1:54724", "error": "trust: rpc: tcp connector failed: rpc: context canceled", "errorVerbose": "trust: rpc: tcp connector failed: rpc: context canceled\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2023-05-23T11:47:01.448+0500	ERROR	piecestore	download failed	{"Process": "storagenode", "Piece ID": "BCBZODLRKFM46INHVUNNPS6IUK52HOEJJ75FQR7YNRXTU7XIKD7Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Remote Address": "172.18.0.1:54734", "error": "trust: rpc: tcp connector failed: rpc: dial tcp 34.172.100.72:7777: operation was canceled", "errorVerbose": "trust: rpc: tcp connector failed: rpc: dial tcp 34.172.100.72:7777: operation was canceled\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2023-05-23T11:47:05.424+0500	INFO	piecestore	download started	{"Process": "storagenode", "Piece ID": "7YSGGXYVMAEGVKGEAYYCYM54XFYIUHQRCOZB5CUE4S7Y55CG3UOA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "Offset": 0, "Size": 1536, "Remote Address": "172.18.0.1:54748"}
2023-05-23T11:47:05.974+0500	INFO	piecestore	downloaded	{"Process": "storagenode", "Piece ID": "7YSGGXYVMAEGVKGEAYYCYM54XFYIUHQRCOZB5CUE4S7Y55CG3UOA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "Offset": 0, "Size": 1536, "Remote Address": "172.18.0.1:54748"}
2023-05-23T11:47:13.720+0500	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "attempts": 6, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:143\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:102\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-05-23T11:47:16.256+0500	INFO	piecestore	download started	{"Process": "storagenode", "Piece ID": "MG4RKY6OUEOJNWDTAWJTJ4ZFIHD3DHIYICREPQ3UU3TZUUORRTJA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_REPAIR", "Offset": 0, "Size": 1536, "Remote Address": "172.18.0.1:54802"}
2023-05-23T11:47:16.603+0500	INFO	piecestore	downloaded	{"Process": "storagenode", "Piece ID": "MG4RKY6OUEOJNWDTAWJTJ4ZFIHD3DHIYICREPQ3UU3TZUUORRTJA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_REPAIR", "Offset": 0, "Size": 1536, "Remote Address": "172.18.0.1:54802"}
2023-05-23T11:47:16.875+0500	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 6, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:143\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:102\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-05-23T11:47:23.736+0500	INFO	piecestore	download started	{"Process": "storagenode", "Piece ID": "FU5KXMN5K7PNSUP26P3QBMGO6U5BTR4E3TUYN6VF3FGQNYR56NAA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT", "Offset": 398592, "Size": 256, "Remote Address": "172.18.0.1:54830"}
2023-05-23T11:47:24.506+0500	INFO	piecestore	downloaded	{"Process": "storagenode", "Piece ID": "FU5KXMN5K7PNSUP26P3QBMGO6U5BTR4E3TUYN6VF3FGQNYR56NAA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT", "Offset": 398592, "Size": 256, "Remote Address": "172.18.0.1:54830"}

[shaltay]

И так по кругу перегружаясь.

2023-05-23T11:48:38.017+0500 ERROR contact:service ping satellite failed {"Process": "storagenode", "Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "attempts": 5, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:143\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:102\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"} stdout

11:48:38 2023-05-23T11:48:38.170+0500 ERROR contact:service ping satellite failed {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "attempts": 5, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:143\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:102\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"} stdout

11:48:40 2023-05-23T11:48:40.448+0500 ERROR contact:service ping satellite failed {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "attempts": 5, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:143\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:102\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"} stdout

11:48:44 2023-05-23T11:48:44.808+0500 INFO piecestore download started {"Process": "storagenode", "Piece ID": "IBIHGJT3MV5XS6F5MEKPJHW4BYQCA27NBEP6VACFPRJFG773NTIQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "Offset": 0, "Size": 28160, "Remote Address": "172.18.0.1:55392"} stdout

11:48:45 2023-05-23T11:48:45.260+0500 INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "IBIHGJT3MV5XS6F5MEKPJHW4BYQCA27NBEP6VACFPRJFG773NTIQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "Offset": 0, "Size": 28160, "Remote Address": "172.18.0.1:55392"} stdout

11:48:47 2023-05-23T11:48:47.591+0500 INFO piecestore download started {"Process": "storagenode", "Piece ID": "2UGFC4NRRXDNRFNGLTHRXNYN7CJ2WX5BLHQT3AUWX4YSHFOKFVZA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 851968, "Size": 2560, "Remote Address": "172.18.0.1:55408"} stdout

11:48:47 2023-05-23T11:48:47.802+0500 INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "2UGFC4NRRXDNRFNGLTHRXNYN7CJ2WX5BLHQT3AUWX4YSHFOKFVZA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 851968, "Size": 2560, "Remote Address": "172.18.0.1:55408"} stdout

11:48:54 2023-05-23T11:48:54.547+0500 ERROR contact:service ping satellite failed {"Process": "storagenode", "Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "attempts": 6, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:143\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:102\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"} stdout

11:48:54 2023-05-23T11:48:54.737+0500 ERROR contact:service ping satellite failed {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "attempts": 6, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:143\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:102\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"} stdout

11:48:55 2023-05-23T11:48:55.904+0500 INFO piecestore download started {"Process": "storagenode", "Piece ID": "QDN6A6LHWX5STP44NQW4L4ESSKAJU7IIDLF677RYCPBQTWGAWUEA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 1303552, "Size": 2560, "Remote Address": "172.18.0.1:55438"} stdout

11:48:56 2023-05-23T11:48:56.097+0500 INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "QDN6A6LHWX5STP44NQW4L4ESSKAJU7IIDLF677RYCPBQTWGAWUEA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 1303552, "Size": 2560, "Remote Address": "172.18.0.1:55438"} stdout

11:48:57 2023-05-23T11:48:57.452+0500 ERROR contact:service ping satellite failed {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "attempts": 6, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:143\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:102\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"} stdout

11:48:59 2023-05-23T11:48:59.092+0500 INFO piecestore download started {"Process": "storagenode", "Piece ID": "MML6QXY7V5JBGH5Y6YP6XCDMFV4VPCJTF6IXXVVDQYNFJMGYY2MQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "Offset": 0, "Size": 4352, "Remote Address": "172.18.0.1:55448"} stdout

11:48:59 2023-05-23T11:48:59.394+0500 INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "MML6QXY7V5JBGH5Y6YP6XCDMFV4VPCJTF6IXXVVDQYNFJMGYY2MQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "Offset": 0, "Size": 4352, "Remote Address": "172.18.0.1:55448"} stdout

11:49:00 2023-05-23T11:49:00.445+0500 INFO piecestore download started {"Process": "storagenode", "Piece ID": "CZOVMT5G73KB6WWLVSMWJJVRZOS5O2WSK6XB5VYLDEFLFJOXANCA", "Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "Action": "GET_AUDIT", "Offset": 307200, "Size": 256, "Remote Address": "172.18.0.1:55456"} stdout

11:49:01 2023-05-23T11:49:01.086+0500 INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "CZOVMT5G73KB6WWLVSMWJJVRZOS5O2WSK6XB5VYLDEFLFJOXANCA", "Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "Action": "GET_AUDIT", "Offset": 307200, "Size": 256, "Remote Address": "172.18.0.1:55456"} stdout

11:49:05 2023-05-23T11:49:05.189+0500 INFO piecestore download started {"Process": "storagenode", "Piece ID": "L4VIWTPZPENX5SQ64KGT6HHIJ43K7TEFQPO5VU4CEKB2376ZVMIA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_REPAIR", "Offset": 0, "Size": 5120, "Remote Address": "172.18.0.1:55480"} stdout

11:49:05 2023-05-23T11:49:05.327+0500 INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "L4VIWTPZPENX5SQ64KGT6HHIJ43K7TEFQPO5VU4CEKB2376ZVMIA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_REPAIR", "Offset": 0, "Size": 5120, "Remote Address": "172.18.0.1:55480"} stdout

11:49:05 2023-05-23T11:49:05.846+0500 INFO piecestore download started {"Process": "storagenode", "Piece ID": "IF5V3LQBHYCKSMOYQF7TC3FZPANGCIWONIHI2TC6IMVPBZDTZ2QQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET_REPAIR", "Offset": 0, "Size": 76288, "Remote Address": "172.18.0.1:55486"} stdout

11:49:06 2023-05-23T11:49:06.872+0500 INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "IF5V3LQBHYCKSMOYQF7TC3FZPANGCIWONIHI2TC6IMVPBZDTZ2QQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET_REPAIR", "Offset": 0, "Size": 76288, "Remote Address": "172.18.0.1:55486"} stdout

11:49:07 2023-05-23T11:49:07.134+0500 ERROR piecestore:cache error getting current used space: {"Process": "storagenode", "error": "unrecoverable error accessing data on the storage file system (path=config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/2a2i6cribujsomsow5c45wr4p5rqdtjoti7dyciultzapzjqia.sj1; error=lstat config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/2a2i6cribujsomsow5c45wr4p5rqdtjoti7dyciultzapzjqia.sj1: input/output error). This is most likely due to disk bad sectors or a corrupted file system. Check your disk for bad sectors and integrity"} stdout

11:49:07 2023-05-23T11:49:07.134+0500 ERROR services unexpected shutdown of a runner {"Process": "storagenode", "name": "piecestore:cache", "error": "unrecoverable error accessing data on the storage file system (path=config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/2a2i6cribujsomsow5c45wr4p5rqdtjoti7dyciultzapzjqia.sj1; error=lstat config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/2a2i6cribujsomsow5c45wr4p5rqdtjoti7dyciultzapzjqia.sj1: input/output error). This is most likely due to disk bad sectors or a corrupted file system. Check your disk for bad sectors and integrity"} stdout

11:49:07 2023-05-23T11:49:07.134+0500 INFO contact:service context cancelled {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"} stdout

11:49:07 2023-05-23T11:49:07.134+0500 INFO contact:service context cancelled {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"} stdout

11:49:07 2023-05-23T11:49:07.134+0500 INFO contact:service context cancelled {"Process": "storagenode", "Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo"} stdout

11:49:08 Error: unrecoverable error accessing data on the storage file system (path=config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/2a2i6cribujsomsow5c45wr4p5rqdtjoti7dyciultzapzjqia.sj1; error=lstat config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/2a2i6cribujsomsow5c45wr4p5rqdtjoti7dyciultzapzjqia.sj1: input/output error). This is most likely due to disk bad sectors or a corrupted file system. Check your disk for bad sectors and integrity stdout

11:49:08 2023-05-23 11:49:08,201 INFO exited: storagenode (exit status 1; not expected) stdout

11:49:09 2023-05-23 11:49:09,204 INFO spawned: 'storagenode' with pid 54 stdout

11:49:09 2023-05-23 11:49:09,204 WARN received SIGQUIT indicating exit request stdout

11:49:09 2023-05-23 11:49:09,205 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die stdout

11:49:09 2023-05-23T11:49:09.205+0500 INFO Got a signal from the OS: "terminated" {"Process": "storagenode-updater"} stdout

11:49:09 2023-05-23 11:49:09,206 INFO stopped: storagenode-updater (exit status 0) stdout

11:49:09 2023-05-23T11:49:09.258+0500 INFO Configuration loaded {"Process": "storagenode", "Location": "/app/config/config.yaml"} stdout

11:49:09 2023-05-23T11:49:09.258+0500 INFO Anonymized tracing enabled {"Process": "storagenode"} stdout

11:49:09 2023-05-23T11:49:09.259+0500 INFO Operator email {"Process": "storagenode", "Address": "asperfilev@gmail.com"} stdout

11:49:09 2023-05-23T11:49:09.259+0500 INFO Operator wallet {"Process": "storagenode", "Address": "0x**************************************"} stdout

11:49:10 2023-05-23T11:49:10.057+0500 INFO server kernel support for tcp fast open unknown {"Process": "storagenode"} stdout

11:49:10 2023-05-23T11:49:10.815+0500 INFO Telemetry enabled {"Process": "storagenode", "instance ID": "1MBwuz3R4AAXfhvfHFFLjtPyUFetYJnHcc94PZEYQN6kEJqTW4"} stdout

11:49:11 2023-05-23T11:49:10.815+0500 INFO Event collection enabled {"Process": "storagenode", "instance ID": "1MBwuz3R4AAXfhvfHFFLjtPyUFetYJnHcc94PZEYQN6kEJqTW4"} stdout

11:49:11 2023-05-23T11:49:10.839+0500 INFO db.migration Database Version {"Process": "storagenode", "version": 54} stdout

11:49:11 2023-05-23T11:49:11.987+0500 INFO preflight:localtime start checking local system clock with trusted satellites' system clock. {"Process": "storagenode"} stdout

11:49:12 2023-05-23 11:49:12,989 INFO waiting for storagenode, processes-exit-eventlistener to die stdout

11:49:13 2023-05-23T11:49:12.990+0500 INFO preflight:localtime local system clock is in sync with trusted satellites' system clock. {"Process": "storagenode"} stdout

11:49:13 2023-05-23T11:49:12.990+0500 INFO trust Scheduling next refresh {"Process": "storagenode", "after": "6h11m31.570351252s"} stdout

11:49:13 2023-05-23T11:49:12.990+0500 INFO Node 1MBwuz3R4AAXfhvfHFFLjtPyUFetYJnHcc94PZEYQN6kEJqTW4 started {"Process": "storagenode"} stdout

11:49:13 2023-05-23T11:49:12.990+0500 INFO Public server started on [::]:28967 {"Process": "storagenode"} stdout

11:49:13 2023-05-23T11:49:12.990+0500 INFO bandwidth Performing bandwidth usage rollups {"Process": "storagenode"} stdout

11:49:13 2023-05-23T11:49:12.990+0500 INFO Private server started on 127.0.0.1:7778 {"Process": "storagenode"} stdout

11:49:13 2023-05-23T11:49:12.990+0500 INFO pieces:trash emptying trash started {"Process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB"} stdout

11:49:13 2023-05-23T11:49:13.019+0500 INFO pieces:trash emptying trash started {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"} stdout

11:49:13 2023-05-23T11:49:13.047+0500 INFO pieces:trash emptying trash started {"Process": "storagenode", "Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo"} stdout

11:49:13 2023-05-23T11:49:13.064+0500 INFO pieces:trash emptying trash started {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"} stdout

11:49:13 2023-05-23T11:49:13.166+0500 ERROR contact:service ping satellite failed {"Process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "attempts": 1, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:143\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:102\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"} stdout

11:49:13 2023-05-23T11:49:13.244+0500 INFO pieces:trash emptying trash started {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"} stdout

11:49:13 2023-05-23T11:49:13.301+0500 INFO pieces:trash emptying trash started {"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"} stdout

11:49:13 2023-05-23T11:49:13.518+0500 ERROR contact:service ping satellite failed {"Process": "storagenode", "Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "attempts": 1, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:143\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:102\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"} stdout

11:49:13 2023-05-23T11:49:13.681+0500 ERROR contact:service ping satellite failed {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 1, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:143\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:102\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"} stdout

11:49:13 2023-05-23T11:49:13.998+0500 ERROR contact:service ping satellite failed {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "attempts": 1, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:143\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:102\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"} stdout

11:49:15 2023-05-23T11:49:15.037+0500 ERROR contact:service ping satellite failed {"Process": "storagenode", "Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "attempts": 2, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:143\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:102\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"} stdout

11:49:15 2023-05-23T11:49:15.363+0500 ERROR contact:service ping satellite failed {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 2, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:143\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:102\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"} stdout

11:49:16 2023-05-23T11:49:16.002+0500 ERROR contact:service ping satellite failed {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "attempts": 2, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:143\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:102\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"} stdout

11:49:16 2023-05-23 11:49:16,002 INFO waiting for storagenode, processes-exit-eventlistener to die stdout

11:49:17 2023-05-23T11:49:17.091+0500 INFO piecestore download started {"Process": "storagenode", "Piece ID": "FGZC55POSGD2YC2AKP323SIG6YNPPO3VOE3QL57LECFKJ5KXMT5A", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "GET_AUDIT", "Offset": 1552384, "Size": 256, "Remote Address": "172.18.0.1:55600"} stdout

11:49:18 2023-05-23T11:49:17.344+0500 INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "FGZC55POSGD2YC2AKP323SIG6YNPPO3VOE3QL57LECFKJ5KXMT5A", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "GET_AUDIT", "Offset": 1552384, "Size": 256, "Remote Address": "172.18.0.1:55600"} stdout

11:49:18 2023-05-23T11:49:17.564+0500 ERROR contact:service ping satellite failed {"Process": "storagenode", "Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "attempts": 3, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:143\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:102\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"} stdout

11:49:19 2023-05-23T11:49:19.018+0500 ERROR contact:service ping satellite failed {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "attempts": 3, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:143\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:102\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"} stdout

11:49:19 2023-05-23 11:49:19,018 INFO waiting for storagenode, processes-exit-eventlistener to die stdout

11:49:20 2023-05-23 11:49:20,020 WARN killing 'storagenode' (54) with SIGKILL stdout

11:49:20 2023-05-23 11:49:20,022 INFO stopped: storagenode (terminated by SIGKILL) stdout

11:49:20 2023-05-23 11:49:20,023 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM) stdout

11:49:45 2023-05-23 11:49:45,494 INFO Set uid to user 0 succeeded stdout

11:49:45 2023-05-23 11:49:45,517 INFO RPC interface 'supervisor' initialized stdout

11:49:45 2023-05-23 11:49:45,518 INFO supervisord started with pid 1 stdout

11:49:46 2023-05-23 11:49:46,520 INFO spawned: 'processes-exit-eventlistener' with pid 11 stdout

11:49:46 2023-05-23 11:49:46,522 INFO spawned: 'storagenode' with pid 12 stdout

11:49:46 2023-05-23 11:49:46,523 INFO spawned: 'storagenode-updater' with pid 13 stdout

11:49:46 2023-05-23T11:49:46.577+0500 INFO Configuration loaded {"Process": "storagenode", "Location": "/app/config/config.yaml"} stdout

11:49:46 2023-05-23T11:49:46.577+0500 INFO Anonymized tracing enabled {"Process": "storagenode"} stdout

11:49:46 2023-05-23T11:49:46.578+0500 INFO Operator email {"Process": "storagenode", "Address": "asperfilev@gmail.com"} stdout

11:49:46 2023-05-23T11:49:46.578+0500 INFO Operator wallet {"Process": "storagenode", "Address": "0x**********************"} stdout

11:49:46 2023-05-23T11:49:46.755+0500 INFO Configuration loaded {"Process": "storagenode-updater", "Location": "/app/config/config.yaml"}

[shaltay]

Отключил IP6 посмотрим.
Появилась ещё интересная строчка в логах.
Error: unrecoverable error accessing data on the storage file system (path=config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/2a2i6cribujsomsow5c45wr4p5rqdtjoti7dyciultzapzjqia.sj1; error=lstat config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/2a2i6cribujsomsow5c45wr4p5rqdtjoti7dyciultzapzjqia.sj1: input/output error). This is most likely due to disk bad sectors or a corrupted file system. Check your disk for bad sectors and integrity

[Alexey]

@shaltay

11:49:08 Error: unrecoverable error accessing data on the storage file system (path=config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/2a2i6cribujsomsow5c45wr4p5rqdtjoti7dyciultzapzjqia.sj1; error=lstat config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/2a2i6cribujsomsow5c45wr4p5rqdtjoti7dyciultzapzjqia.sj1: input/output error

Вам нужно проверить диск на ошибки и исправить их. Возможно identity тоже повреждена.
Если после исправления ошибок на диске проблема сохранится, тогда продолжим:

Есть хоть одна ошибка ping satellite failed но без ratelimit? Потому что ratelimit это последствие частых попыток регистрации с ошибкой (неправильный внешний адрес например), а не причина.
Я бы порекомендовал попробовать использовать просто публичный IP. Возможно сателлит не может надёжно ресолвить предоставленный hostname в IP и использует предыдущий из кэша, а он сменился. Или пробует использовать IPv6, а он у вас не настроен на этом хосте или не разрешены порты узла для IPv6 UDP/TCP.

У вас сохранился бэкап этой identity? Если да, сохраните текущую и восстановите из бэкапа, только убедитесь, что она та же самая, например так:

RaspberryPi crash

./identity certificate-authority id --ca.cert-path /mnt/storagenode2/identity/ca.cert It will print a NodeID. Then you need to search in your logs, what was the NodeID before. Please also SKIP a SETUP step, it must not be executed again for a worked node, otherwise you can destroy it. no, this token is one-time only and can be used only once to sign any identity, after that it has no use. It also doesn’t contain any info related to your node (even to used email address, it’s just used for …

Если она была не подписана - подпишите новым токеном авторизации и используйте эту версию.
Если бэкапа нет, попробуйте сделать так:

1. Остановите и удалите контейнер storagenode
2. Сделайте резервную копию папки identity
3. удалите из текущей папки с identity файлы identity.cert и ca.cert
4. Скопируйте файл вида identity.1560971067.cert как identity.cert, а файл вида ca.1560971067.cert как ca.cert (цифры у вас будут свои - это дата в Unix формате, когда вы подписывали identity в первый раз).
5. Подпишите эту identity новым ключом авторизации
6. Проверьте, что NodeID не изменился, таким запросом: RaspberryPi crash - #5 by Alexey
7. Если NodeID тот же - стартуйте узел и понаблюдайте за логами.

[shaltay]

Стал проверять по мануалу: Алгоритм починки BTRFS — NixTux
И проверка идёт уже 3 день, нумаю что ноде кабзда. Надо было синхронизировать на отдельный диск , а после проверять уже проблемный диск.

 unresolved ref dir 707568 index 0 namelen 54 name x3ol5nwdpjmtccmq5vb2tcylv2umdebxeqmo6ehjbt7zeszysa.sj1 filetype 1 errors 6, no dir index, no inode ref
Trying to rebuild inode:3589695
root 257 inode 3589695 errors 2001, no inode item, link count wrong
        unresolved ref dir 52472 index 0 namelen 54 name jzo77nuduenrx7corf725yf37ktrlbg4zzo6qzpgsmuoveio7a.sj1 filetype 1 errors 6, no dir index, no inode ref
Trying to rebuild inode:3589740
root 257 inode 3589740 errors 2001, no inode item, link count wrong
        unresolved ref dir 432523 index 0 namelen 54 name uuorhcacji7opgohxbqd7654e6rj55h2hbqeuhj2trspwhan4a.sj1 filetype 1 errors 6, no dir index, no inode ref
Trying to rebuild inode:3589742
root 257 inode 3589742 errors 2001, no inode item, link count wrong
        unresolved ref dir 432266 index 0 namelen 54 name pm5pc6rvdmne2znxpwq5iwe2ckahaz4byzfrp5p2qqtxnsmtaq.sj1 filetype 1 errors 6, no dir index, no inode ref
Trying to rebuild inode:3589748
root 257 inode 3589748 errors 2001, no inode item, link count wrong
        unresolved ref dir 1248306 index 0 namelen 54 name jdlkda5bq2lilfefuinrdly5eacssxykuoupc3wgbhk7pc5fxq.sj1 filetype 1 errors 6, no dir index, no inode ref
Trying to rebuild inode:3589751
root 257 inode 3589751 errors 2001, no inode item, link count wrong
        unresolved ref dir 432976 index 0 namelen 54 name o57hguuggf6dkbyrvk5k66dsebfmjwuxpstgvlriujb3jslb7q.sj1 filetype 1 errors 6, no dir index, no inode ref
Trying to rebuild inode:3589753
root 257 inode 3589753 errors 2001, no inode item, link count wrong
        unresolved ref dir 707923 index 0 namelen 54 name z5rbimjjx3rhj5kkss4bwqdwwwt22mzmn6j2l47rb36qxqjhza.sj1 filetype 1 errors 6, no dir index, no inode ref
Trying to rebuild inode:3589772
root 257 inode 3589772 errors 2001, no inode item, link count wrong
        unresolved ref dir 707701 index 0 namelen 54 name ol7v3cio5bk2oy5t33srntee45fsqr4kaifs4ae624hejmhioa.sj1 filetype 1 errors 6, no dir index, no inode ref
Trying to rebuild inode:3589807
root 257 inode 3589807 errors 2001, no inode item, link count wrong
        unresolved ref dir 707670 index 0 namelen 54 name tasotf337zfexhdbfwj4mnhmvre6vidpx3l7c6onwufuq3olda.sj1 filetype 1 errors 6, no dir index, no inode ref
Trying to rebuild inode:3589816
root 257 inode 3589816 errors 2001, no inode item, link count wrong
        unresolved ref dir 707727 index 0 namelen 54 name hkjetik7ceesza4ja6yw7zz6hxzbyd7k7mbfbgfgjeqqqawmna.sj1 filetype 1 errors 6, no dir index, no inode ref
Trying to rebuild inode:3589825

[Alexey]

Ох. BTRFS, понятно. Оно не готово к нормальному использованию, к сожалению. Лучше тогда zfs, если хочется попробовать что-то новое. Для узлов лучше всего ext4, LVM полезно иметь тоже. Не сильно много гибкости, но бывает полезно, особенно когда хочется переехать на другой диск без downtime.

[shaltay]

Подставил рядом другой диск, слил с проблемного инфу по мануалу. How do I migrate my node to a new device? - Storj Node Operator Docs

ash-4.4# rsync -aP --delete /volume2/storj/data /volumeUSB1/usbshare/node1
sending incremental file list
deleting data/revocations.db
data/
data/trust-cache.json
          1,374 100%    0.00kB/s    0:00:00 (total: 0%) (xfr#1, ir-chk=1309/1312)
deleting data/storage/used_serial.db
deleting data/storage/storage_usage.db
deleting data/storage/secret.db
deleting data/storage/satellites.db
deleting data/storage/reputation.db
deleting data/storage/pricing.db
deleting data/storage/pieceinfo.db
deleting data/storage/piece_spaced_used.db
deleting data/storage/piece_expiration.db
deleting data/storage/orders.db
deleting data/storage/notifications.db
deleting data/storage/info.db
deleting data/storage/heldamount.db
deleting data/storage/bandwidth.db
data/orders/unsent/unsent-orders-121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6-1685700000000000000.v1
          3,354 100%  105.66kB/s    0:00:00 (total: 0%) (xfr#2, ir-chk=1045/1312)
data/orders/unsent/unsent-orders-12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo-1685700000000000000.v1
          7,157 100%  174.73kB/s    0:00:00 (total: 0%) (xfr#3, ir-chk=1037/1312)
data/orders/unsent/unsent-orders-1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE-1685700000000000000.v1
          4,310 100%  102.66kB/s    0:00:00 (total: 0%) (xfr#4, ir-chk=1035/1312)
data/storage/
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/2a2i6cribujsomsow5c45wr4p5rqdtjoti7dyciultzapzjqia.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/2r6kdpqrxswwblvtyj5agy5lzrgj7q2wrhew4vwuzcb6rrgmxq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/34eo7q6oouoghnxgdnnr3pgca6d2ewwpwmyfgphe7nfh7fksla.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/3kpz4hawrqg2torskefmkggnjjv6bryz3j62cx3gquog75chba.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/4hfrprvi6cr2ktlpgbwel6olhhzduogqo6ubtv77ah6y77yz6a.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/5frjuo32txrsugohkp45x2g7boq2gjuh6caxq7enlcrpuqodza.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/632bljgzcq5nhlihniget4eyutw554lblv4usiaq2mmatpuhfa.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/657goqli7moa26byzyxke4ufb56q3kmhyeiemmq2wvcp37z27a.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/acyfdw63ft55kv4x5brqc5dtoegm57njrapiv6w2bib3vnxmxq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/bhl4blnqpvnindp3cmzypwzpuvcjetla2nuk4a4s6idtq727rq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/d44uyjgg5ffzbasqc72scan7zukc6w5o4ifo33qkk5f5i4x5vq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/dbbawmppokaajppecbaxprlmrcwnwi4ksktfgsxnzrvek7mypa.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/fp4gvmwww6oxrpzjc76gokdv7uihyub4zoy5avyfmjiu32riha.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/fxx4terxs6yaahmj4272lkhsufytga5xhu7fwvfi3naiiudahq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/gs76u6ao22s5diza64vaxbxncp3agikki5ttf3wgi5u4tfa6vq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/ikoswl5s36mubiayp435b344xwfuzxn2puygkzpy6sigjkvnla.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/il5qdksjc4gcrna4hlf3frf627z577qjyw2yfrsgkea3hysf2a.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/jk6eccyzetwr7ev64st7izvzmohhd7vfc6w4wvt26vxlisa3dq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/jsmtb76wexfsx5aizmvhux6gtl57n4mv4s2b3efg5slmv6cupq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/krs6drargyq4bzx7ouspidbioes3l7aobkdshto7qoyyahktxa.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/kt3emfwo77tzomwojkbsddc7uhd3zdpzxt367k2eqjfesrmjhq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/luecauggtypbsh77ep2mdfnpfk2zodtgi3iemw7klpsk4ll2ra.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/mgtidzwmoahvumcl6fdifmdxcxwoyunzkanidulhwtf42jw7tq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/n6mnrk4ndbhovvnjdo5glopfc55xllzhlay2bqoepmbqwluqeq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/nbhtstaffuhcz5scopqegro422jp44vcuxmt5md7bsazfbgpgq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/o7ur6a5lnhxfbeknwmfvopqgllgwpjzw4k56fmaknqa5pyab2a.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/ojvnjnww2mtyg3klbclznkg6jwbh4mmfh6yq4hsaxon7ufyiua.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/qottjrn4jzhwgcpyqdwuu2q6wvc355rm27dsqbhusjczp2ofwa.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/qscqkbdj5gvd6lbwqbybxs4ulubwtjbow3vxgslplsfrt7yusa.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/qtythjaid6ivg6gdd3bwrywxxpgct2j2v3yfzs7aydshxczd3q.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/rgy3ykxpdgmpwfvr7fihpqtvdsqu43nquoeu5n7xqazp64sy5a.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/rujf74p7gjp4gygizlkurctravxkwii2cvdz2rtd5hijyzlfpq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/se57odxg5dzpl2dlvitfrymigamy7ortxne7no7q4xdiogct4q.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/sgfuos74one5nefnwhixylpcznk4gngcr55toqsa7gqyar77wa.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/tyyhwim67imka3ctichc3z2v6u45ybjfwtfzxqbdww2li7u23q.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/uczeabm6w3jzlnxvqhuckhalzeie5nvjwdh6bcaausr45oxtba.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/uxpg5rhbcfpkklz5mg5j2fr2wcjmdne2zkt7agqv4iwbdhzbaq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/vi4hygtfsvvrp2nbguymjgyw645jizujiozx2ldgy6kkwk5drq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/ygcfca2zin346qozmzsz4xiat34t73vvmclbuotvoklljllxca.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/zkghj6fnrdgw4roreugwbqnjpfsxeag35csftdx7honfdc365a.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/qoeyy2bonwu7m7xevcgvsvakdjugpnm5n4ieaipnuy6q6vixuq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/oygwfvrgqubzziy6ovxhuqpx4ygluea3bs7xfewzn6673rmsfa.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/xhx6nmh3b6ciit2lycu2j5mo5ewcg2rxcysg44enxapyn3rlia.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/d7eyrnpvfxcmmwbpabqb42nkkmxzrb6cnzxqhwa6nenwryiusq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/fohpxhruzi7pwqxkxrasbegykxevhibqffgtnx7tsuhxeswjea.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/mgyns55uq4iir3rkd4xxmerbbkp4xcxe7okuljkt6uztizp3rq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/7mr6lxyfjqy4swnukd3gsmr3xtt66w2qgx5wqyc3jb6zp3pdyq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/fae2cagrqfte5iv4niiywveqavbtkg55kcdnplmppaygwm7bkq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/yx5upwh4zwdhrrhk4nuiaqruresjz7numoekxussuryeko6beq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/4idsbggwcl47k3ulxscqbmcvfm7l5xql63ms73rcoie7nvuwva.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/ssf7v5rew3zqs4qzkxnbizy7lauplq6a7ga7mrnbgm3vq2klga.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/gldmerwop366cvbkowzm5kav4tcddlsnt7akwrok46ndowdyka.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/cp3iynla3ftlibkrn2o7jjdhof3pknf7sp7stt75vxt4jcif3a.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/jgjn6escjszhrndvfpo3yghqdzu3m2ihquarisgditldu6cfeq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/45qbtlcesqoy2zsqxft2sjs7jw6uqvwhdyzytkkvfxkuzldl2a.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/b4k4ewknsshsgb7xesb5logqhof3eajwjxqok352gja3ckgynq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/mfaykss2ksge3mgr6kwnnwehc7do424xdac7timctxulhxmnua.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/otykbyz6c7w6655m7rkh56fvfxdsle7tbbpnr5kwpvy5obtf7q.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/qndsvjwe6c5v2k654szguwcbmhsvxvabvsnmhxmqpgz4ogf2ua.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/2vudlkj2so24rbyytk6bz7naeikzudi5r3hmnf4u6c2lnaf2cq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/ryndk3wb7ja6iwtwarl2fe55aqa64h2ll5qnzmr7o5skg5ebna.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/zgwzn2cmcwuw37khcsaj3u7vrbmqsuv22zmqyv2jzxncjtzncq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/wyesxsdq6x6s6lo2bbjx4onxfhmtm4fzryma75yivkxqslp6aq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/j6qiihpfnhmmsyeneremdthjpvofebtb6x2okr56uciopgtpmq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/kqgxxr7apf3bv6zyus6h6eywkygalwdzxbsgmceqxxxjd5tlzq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/bl27m7dflu6q42f36exi5x7kzrc2m4soyzt54l6bx4zokmpcmq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/mj22bdyrq4ee3negxjxur6xvtfikdisxewtoucixdx5evto7wa.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/36pag5dmsqbydxa6bwxej2l3gvkgcpapl3cm7vnbr7yctrqela.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/aycdmkk66ebe3pts7cksgb7jtarkxbouefyaiggosd4q6iqpsa.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/ahz7bk7zvliabix6wlwzdtbnzfbrq2rthkbgyy32rbbjopmw4a.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/qsoi3ty3roxgtdvd5f7iwkvtl7nmrhf5qooeoj5wkkjhyx726a.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/d5jqqsjlbdnxqrvbpp4gnca6fyptkl7xjb5zutuvqzv6ab52ha.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/aqifsiedgfsaav4b3yw3byjufelkdgzz2ay4v2oz2qx4hy27tq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/to22chfesvqkmzqu54dmiqa6oj6rxakym6ipje4beorut4y6ka.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/5elsxfphs3uier4dxdtnwsk7fzozihxuzetpddjqvgpxmoicoa.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/tjsuctjn6gao5rjfzqucv2yx4csk7arkaqy573saah3naz46yq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/tygergm5lbfz54zjqvczc2msbbzimuyuu5jbo6vauowy7unzma.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/leekj5aoax27ae4gcbf53rkfz5bpmv6h6gkjborxb556kvtooq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/euo2bwwxsgbe2pylvbonggcwnkdbkte7fb5ehbeww442vkpckq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/u4swyfhgekl25kf4kq4wlp2vf2zyff77x2hw47fiyqhw5xeiwa.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/zj6x7lmhwe2zvvcyielrbbn3lg4vctjmppqrmyi2y67dwuwfqq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/76dtlabtx66ypehiw3cpgq2wil6bqxlrhu7t5jz3w3szqzo5yq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/ncst6qv6hv2quem7i7vq5bticy7t4ybwql744fqkwgy3f7kjra.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/tbiap4z35o6actffmup6ckjgufexcf2d5sxq6ls6lhbc55w4ka.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/ghbnrpyuaegr55e6py5j3l4pa6c4ydp6sonmmom3qcqftrmyeq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/xrdvq6tz25xct5kczk57niyfxxqpmkhq6qnf5pqru53innmjsa.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/4vzoe7w5t6dbu4dvwvruoxmlihnj47omuv5dxemawpsppq2zxq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/v5c3ojxywsielp3jvgblo3q6uzro3ftirckbj4hyamhn2p5d5a.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/246vwped3mxwngygmhgw225q5xeim4aari6ibmks7cj2455mya.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/7kgkpexm6sa6twtsgz5qugibalxz6z7j5o7yjejox7en2f6aeq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/nkcnpm36e2p7pm6i3yvzkwobtyvk64h3eaiegrvze3nssadpvq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/nmwainpx4tzuztmwv3e7nnbnqoja4oyyru5xm6g2aj7xnb6fjq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/o3hcyipcp2ctcwlxwbyijrptwbawgfxm3fcsmzwlzze6vuoncq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/wigffuulbznjsqzwcuaatqduvruk2t5goo2xi2s7vgzj47taqa.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/nqmiz2ae2qdlpysldyacsjctkfyposkynuyw5vdhyhb4xxsjda.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/w75gukqrbinbgfumlz3eyztiedvvjk4aaarcm6vc6ajsxejgaq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/zhmmvs7ke5uvwr2kaxnytpcslsxxiufivvkz33f2mzyc7oufha.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/et53akntyim5cvjzf7sysgsxd3likynr2coynhjqgqkmswfuba.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/bab4eooynpml36g6kxnfngrov3xktr23pwphbmpqzf4nvpgdjq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/vhzghicvmj4oblkpydzrp2meimuq4rbwg53gudmvmvsgnkidna.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/2kiwogy4mjjrgmglnaocfswqwsroshaqfmpxxmj5g3uuy5htwa.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/inond4fzd56mbqxdzxxi7hjudgn22hwi2uoxtk6hu6hkpggtrq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/mv73cfzzfwteitn6t4m3gqxbivf4viru7e5ig34uyaczwggtpa.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/ubmedp42lqvtvkujmu622grw5eosmml4eihvleiyy6rz76jdza.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/eymphmuq7oijt7x3tlqnqlzo5nokpvzhqza3cs3rmwwnbxrzmq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/x2l73sllvwgljzseaj4fjj22v2xbodemh2wiaxifzhhkobthza.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/2b23a6z74beaqwz7olbkvxgfx6l2ukfn7ain747aohgeajeana.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/xzzo6uumwunp7aparbwo6wpx6vaazix4j3zc5cqymdpxchmkrq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/3ddpmdr3kd4qzliqdzecnjd6mpfsq5ncb6id6gsqajn2iprcxa.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/trlsnsjzssaci6hrpirkfxqexcpdfdq3i2a5puc4oyvcnakihq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/ojw5yz4pjsdsro6gudu7e5v4vpzs5iv43dpgr4dplxqukfuleq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/zo5qmgqb6ajhd3jmyef6e63cgzxnnrurrdpuea5hjl64mrechq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/7qojgibm4yupj26l3ymliqk3zkhoy7q3gvyqeyoh6f25jnsknq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/5jzn4d5kdqxyurqa5kxllvj5agvbltl4kv7gddenrctyz6ebtq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/xkksbrh3xp3z376vfi3yw3nlxaou4jh7sqktq4xkjoyh36sm6a.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/4j33z4gzd2ple76igc4d2xuu4vihuoysf7vvr36uwquu6kiatq.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/kh7ltku6qokygdae3rzwi4ahi5v6vztt3dieigz2vhr32pgz7q.sj1") failed: Input/output error (5)
rsync: readlink_stat("/volume2/storj/data/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/3o/sdxloiifspfhgyehgdorugfhw33tze7bkm3thendduezsl5h3a.sj1") failed: Input/output error (5)
IO error encountered -- skipping file deletion
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1462) [sender=3.1.2]

Запустил на новом диске ноду. Нода запустиласть, но 2 TB данных не стало. (
Пока работает.

[shaltay]

Как так. Нода вроде работает, но на емейл прилетает письмо что она дисквалифицированна на US1.

[shaltay]

Вобщем дисквалифицировали ноду по всем сателитам.
Вопрос, если я сделаю identity с новым ключом авторизации, я могу запустить ноду со старыми данными и новым ключом?

[Alexey]

5% потерянных данных неминуемо дисквалифицирует узел.

@shaltay

Как так. Нода вроде работает, но на емейл прилетает письмо что она дисквалифицированна на US1.

да, к сожалению, потеря данных ведёт к необратимой дисквалификации.

@shaltay

Вопрос, если я сделаю identity с новым ключом авторизации, я могу запустить ноду со старыми данными и новым ключом?

это будет та же самая дисквалифицированная identity.

Если вы хотите запустить новый узел, тогда надо сгенерировать новую identity (а не скопировать старую), подписать её новым токеном авторизации и запустить узел с чистым хранилищем (старые данные не будут признаны новой identity и будут медленно удаляться в течение нескольких недель с помощью Garbage Collector, нет смысла растягивать “удовольствие”, они не будут оплачиваться).

[shaltay]

Понял. Спасибо. Следующие ноды будут на EXT4.

[Alexey]

К сожалению, BTRFS пока что production-unready, test purposes only.
Пока что неплохо себя зарекомендовали “родные” файловые системы (ext4 in Linux, NTFS in Windows), после настройки zfs тоже становится более-менее (скорее менее в плане производительности) пригодной.