Anything that can be done , to make Storj a little more robust?

did it help to resolve an offline issue?

No, i think its not even completely booting up, now.

Perhaps you cleared too much. I think you may try to reinstall docker, it should restore missing rules.

Yep tried reinstalling docker, once with a simple apt reinstall
Then a full apt remove, then apt install.

It didnā€™t help

Then just for shits and giggles i apt reinstall iptables, and i think that cleaned it up a bit

[storjnode]:</home/mike/> sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy DROP)
target     prot opt source               destination
DOCKER-USER  all  --  anywhere             anywhere
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain DOCKER (1 references)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             172.17.0.3           tcp dpt:28967
ACCEPT     udp  --  anywhere             172.17.0.3           udp dpt:28967
ACCEPT     tcp  --  anywhere             172.17.0.3           tcp dpt:14002

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain DOCKER-USER (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere
[storjnode]:</home/mike/>

Buttttt it hasnā€™t helped, tail 20 of the log:

2023-03-20T08:20:35.035Z        ERROR   preflight:localtime     unable to get satellite system time     {"Process": "storagenode", "Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "error": "rpc: tcp connector failed: rpc: dial tcp: lookup us2.storj.io on 8.8.4.4:53: read udp 172.17.0.3:44312->8.8.4.4:53: i/o timeout", "errorVerbose": "rpc: tcp connector failed: rpc: dial tcp: lookup us2.storj.io on 8.8.4.4:53: read udp 172.17.0.3:44312->8.8.4.4:53: i/o timeout\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:189"}
2023-03-20T08:20:35.035Z        ERROR   preflight:localtime     unable to get satellite system time     {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "error": "rpc: tcp connector failed: rpc: dial tcp: lookup saltlake.tardigrade.io on 8.8.4.4:53: read udp 172.17.0.3:35758->8.8.4.4:53: i/o timeout", "errorVerbose": "rpc: tcp connector failed: rpc: dial tcp: lookup saltlake.tardigrade.io on 8.8.4.4:53: read udp 172.17.0.3:35758->8.8.4.4:53: i/o timeout\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:189"}
2023-03-20T08:20:35.035Z        ERROR   preflight:localtime     unable to get satellite system time     {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "error": "rpc: tcp connector failed: rpc: dial tcp: lookup ap1.storj.io on 8.8.4.4:53: read udp 172.17.0.3:59892->8.8.4.4:53: i/o timeout", "errorVerbose": "rpc: tcp connector failed: rpc: dial tcp: lookup ap1.storj.io on 8.8.4.4:53: read udp 172.17.0.3:59892->8.8.4.4:53: i/o timeout\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:189"}
2023-03-20T08:20:35.036Z        ERROR   preflight:localtime     unable to get satellite system time     {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "error": "rpc: tcp connector failed: rpc: dial tcp: lookup us1.storj.io on 8.8.4.4:53: read udp 172.17.0.3:43120->8.8.4.4:53: i/o timeout", "errorVerbose": "rpc: tcp connector failed: rpc: dial tcp: lookup us1.storj.io on 8.8.4.4:53: read udp 172.17.0.3:43120->8.8.4.4:53: i/o timeout\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:189"}
2023-03-20T08:20:35.036Z        ERROR   preflight:localtime     unable to get satellite system time     {"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "error": "rpc: tcp connector failed: rpc: dial tcp: lookup eu1.storj.io on 8.8.4.4:53: read udp 172.17.0.3:47471->8.8.4.4:53: i/o timeout", "errorVerbose": "rpc: tcp connector failed: rpc: dial tcp: lookup eu1.storj.io on 8.8.4.4:53: read udp 172.17.0.3:47471->8.8.4.4:53: i/o timeout\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:189"}
2023-03-20T08:20:35.036Z        ERROR   Failed preflight check. {"Process": "storagenode", "error": "system clock is out of sync: system clock is out of sync with all trusted satellites", "errorVerbose": "system clock is out of sync: system clock is out of sync with all trusted satellites\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).Check:96\n\tstorj.io/storj/storagenode.(*Peer).Run:885\n\tmain.cmdRun:251\n\tstorj.io/private/process.cleanup.func1.4:377\n\tstorj.io/private/process.cleanup.func1:395\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:92\n\tmain.main:478\n\truntime.main:250"}
Error: system clock is out of sync: system clock is out of sync with all trusted satellites
2023-03-20 08:20:35,060 INFO exited: storagenode (exit status 1; not expected)
2023-03-20 08:20:36,064 INFO spawned: 'storagenode' with pid 26
2023-03-20 08:20:36,065 WARN received SIGQUIT indicating exit request
2023-03-20 08:20:36,066 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2023-03-20T08:20:36.066Z        INFO    Got a signal from the OS: "terminated" {"Process": "storagenode-updater"}
2023-03-20 08:20:36,067 INFO stopped: storagenode-updater (exit status 0)
2023-03-20T08:20:36.124Z        INFO    Configuration loaded    {"Process": "storagenode", "Location": "/app/config/config.yaml"}
2023-03-20T08:20:36.125Z        INFO    Anonymized tracing enabled      {"Process": "storagenode"}
2023-03-20T08:20:36.127Z        INFO    Operator email  {"Process": "storagenode", "Address": "mo...om"}
2023-03-20T08:20:36.127Z        INFO    Operator wallet {"Process": "storagenode", "Address": "0x38...4b5"}
2023-03-20T08:20:36.139Z        INFO    server  kernel support for server-side tcp fast open remains disabled.  {"Process": "storagenode"}
2023-03-20T08:20:36.139Z        INFO    server  enable with: sysctl -w net.ipv4.tcp_fastopen=3  {"Process": "storagenode"}
2023-03-20 08:20:39,144 INFO waiting for storagenode, processes-exit-eventlistener to die
[storjnode]:</home/mike/>

and cmdline dashboard:

2023-03-20T08:22:17.936Z        INFO    Configuration loaded    {"Process": "storagenode", "Location": "/app/config/config.yaml"}
2023-03-20T08:22:17.939Z        INFO    Anonymized tracing enabled      {"Process": "storagenode"}
2023-03-20T08:22:17.943Z        INFO    Identity loaded.        {"Process": "storagenode", "Node ID": "12c8JB6mnmQhDgh2fbHC9f5bp8ByqM1pDsyzFkzVsQQaEvg6T7u"}
Error: manager closed: read tcp 127.0.0.1:54184->127.0.0.1:7778: read: connection reset by peer
[storjnode]:</home/mike/>

the network still doesnā€™t work.
Is it possible that your VM has not unique IP in your network or it did not match allowed subnet for your modem (router)? And could you try to change the DNS server for it to 8.8.8.8 for example instead of 8.8.4.4?

1 Like

Yep , iā€™m sure i havenā€™t duplicated the IP address, as i said its static, and i donā€™t have that many static addresses.

the DNS entries i originally had was what is given out by my cablemodem.
I have changed them a couple of times, and now just 8.8.8.8

It just puzles me, how it can figure out the address to download the updater, but apparently nothing else

downloading storagenode-updater
--2023-03-21 07:15:18--  https://version.storj.io/processes/storagenode-updater/minimum/url?os=linux&arch=amd64
Resolving version.storj.io (version.storj.io)... 34.173.164.90
Connecting to version.storj.io (version.storj.io)|34.173.164.90|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 92 [text/plain]
Saving to: 'STDOUT'

     0K                                                       100%  124M=0s

2023-03-21 07:15:19 (124 MB/s) - written to stdout [92/92]

--2023-03-21 07:15:19--  https://github.com/storj/storj/releases/download/v1.74.1/storagenode-updater_linux_amd64.zip
Resolving github.com (github.com)... 140.82.113.3
Connecting to github.com (github.com)|140.82.113.3|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/a0d3da67-436e-4d47-b00a-d9831ec51589?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230321%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230321T071520Z&X-Amz-Expires=300&X-Amz-Signature=a15db5ce6cb52408395b87140ab8d08fbab30ef52f18009f2dd6fdcda84fb6f0&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_amd64.zip&response-content-type=application%2Foctet-stream [following]
--2023-03-21 07:15:19--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/a0d3da67-436e-4d47-b00a-d9831ec51589?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230321%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230321T071520Z&X-Amz-Expires=300&X-Amz-Signature=a15db5ce6cb52408395b87140ab8d08fbab30ef52f18009f2dd6fdcda84fb6f0&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_amd64.zip&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.108.133, 185.199.111.133, 185.199.110.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 9141974 (8.7M) [application/octet-stream]
Saving to: '/tmp/storagenode-updater.zip'

Wouldnā€™t you think DNS resolution would wwork all the time, or none of the time ?

How about we just wipe it out and rebuild it, this doesnā€™t seem to be getting me anywhere ?

1 Like

plus we could have had it rebuilt within a couple of hours.

1 Like

one time i had a duplicated mac address, because i had cloned a vm.

I never clone, and just usually deploy templates, and i would think vmware handles that correctly, otherwise spinning up vms would be a nightmare.

i use mac addr for dhcp allocation.
its possible the mistake was mine because the vm needed the same ip for port forwarding.

was a bit of a pain to troubleshoot

I probably might prefer that, but my cablemodem doesnā€™t have that feature.
One way i try to get around that is using avahi-daemon, then you just use the name ind it discovers.

i stopped using my commercial router and now run pfsenseā€¦ its been great.
tho today i might have used opnsense instead, but they are very much the sameā€¦

pfsence is very good thing, but for people who know what they do. normal user cant handle it.

lol, i used to be a network professional, now i just want the easiest way to use it.

Perhaps this would be easier, just make sure that your identity and data in safe (please do not wipe them with VM :slight_smile:), and do not forget about databases if you moved them.

identity is on the system drive ā€œsda2ā€, the ā€œ/dataā€ is on sdb1, the larger drive for Storj.

Itā€™s always better to keep your identity together with data, they are useless without each other anyway.

So did i miss a step somewhere? Why doesnā€™t ā€˜identity createā€™ just place them there ?

1 Like

Because you create the identity before you setup the node.

1 Like