Storage Node Preflight Checks

Product Discussions STORJLINGS
21

How can i add this check on linux docker?
preflight.database-check: true

22

Check your data dir that mounted to destination=/app/config in your docker run string. You can find
config.yaml here, just add preflight.database-check: true to the end of this file.

23

so if preflight.database-check: is not present in the ymal is not present just add it

24

Should be there if you updated to 30.5

25

And donā€™t forget to restart the container using docker restart -t 300 storagenode for the settings to take effect.

Yes, just add it to the config file if not present and restart.

26

thanks add it no messages in the log after restart so I will take that as a positive thing

27

You should see

2020-01-23T14:57:06.987-0500 INFO preflight:localtime start checking local system clock with trusted satellitesā€™ system clock.
2020-01-23T14:57:07.869-0500 INFO preflight:localtime local system clock is in sync with trusted satellitesā€™ system clock.

28

yes I see those two messages but nothing about preflight for the db

29

I think these are the only messages you get.

30

well Iā€™m going take the lack of no malformed db as positive and move on thanks

31

The update appears to have removed my operator wallet, and reset my allocated bandwidth, and allocated disk space. It left some of my commented out lines and did not add preflight check. Although the node appears to be running fine somehow. I suspect on next restart it will break itself.

Edit. NVM:. Thatā€™s in my docker run command. So wallet and space are ok, however it did not add preflight check to yaml, but it did do a preflight check :joy:

32

I still have entries in my ymal referencing kademlia and grpc

33

here the post looks like node not start at all after update

34

4 posts were split to a new topic: Both nodes remain OFFLINE

36

@littleskunk

I got one issue, I think it on the satellite side:

 2020-01-24T07:45:27.539635760Z 2020-01-24T07:45:27.539Z INFO    preflight:localtime     start checking local system clock with trusted satellites' system clock.
2020-01-24T07:45:47.549544205Z 2020-01-24T07:45:47.547Z ERROR   preflight:localtime     unable to get satellite system time     {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "error": "rpccompat: context deadline exceeded", "errorVerbose": "rpccompat: context deadline exceeded\n\tstorj.io/common/rpc.Dialer.dialTransport:242\n\tstorj.io/common/rpc.Dialer.dial:219\n\tstorj.io/common/rpc.Dialer.DialAddressID:138\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).getSatelliteTime:110\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).Check.func1:67\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-01-24T07:45:47.549547401Z 2020-01-24T07:45:47.548Z INFO    preflight:localtime     local system clock is in sync with trusted satellites' system clock.
38

@littleskunk
I restart storage node again (after waiting 10min.)
and see it:

2020-01-24T08:00:30.055558893Z 2020-01-24T08:00:30.055Z INFO    preflight:localtime     start checking local system clock with trusted satellites' system clock.
2020-01-24T08:00:50.057901655Z 2020-01-24T08:00:50.057Z ERROR   preflight:localtime     unable to get satellite system time     {"Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "error": "rpccompat: context deadline exceeded", "errorVerbose": "rpccompat: context deadline exceeded\n\tstorj.io/common/rpc.Dialer.dialTransport:242\n\tstorj.io/common/rpc.Dialer.dial:219\n\tstorj.io/common/rpc.Dialer.DialAddressID:138\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).getSatelliteTime:110\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).Check.func1:67\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-01-24T08:00:50.057906559Z 2020-01-24T08:00:50.057Z INFO    preflight:localtime     local system clock is in sync with trusted satellites' system clock.

Another issues that I see:

2020-01-24T10:38:21.379583492Z 2020-01-24T10:38:21.379Z ERROR   nodestats:cache Get disk space usage query failed       {"error": "node stats service error: unable to connect to the satellite 121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6: rpccompat: context deadline exceeded", "errorVerbose": "node stats service error: unable to connect to the satellite 121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6: rpccompat: context deadline exceeded\n\tstorj.io/storj/storagenode/nodestats.(*Service).dial:129\n\tstorj.io/storj/storagenode/nodestats.(*Service).GetDailyStorageUsage:104\n\tstorj.io/storj/storagenode/nodestats.(*Cache).CacheSpaceUsage.func1:130\n\tstorj.io/storj/storagenode/nodestats.(*Cache).satelliteLoop:170\n\tstorj.io/storj/storagenode/nodestats.(*Cache).CacheSpaceUsage:129\n\tstorj.io/storj/storagenode/nodestats.(*Cache).Run.func2:91\n\tstorj.io/common/sync2.(*Cycle).Run:87\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

next one:
2020-01-24T08:16:20.058561212Z 2020-01-24T08:16:20.058Z ERROR version Failed to do periodic version check: version control client error: Get https://version.storj.io: dial tcp 34.69.12.165:443: i/o timeout

but I did not have an issue if try to get it manualy:

Summary

curl https://version.storj.io

{"Satellite":{"major":0,"minor":29,"patch":0},"Storagenode":{"major":0,"minor":29,"patch":0},"Uplink":{"major":0,"minor":26,"patch":0},"Gateway":{"major":0,"minor":26,"patch":0},"Identity":{"major":0,"minor":28,"patch":0},"processes":{"satellite":{"minimum":{"version":"v0.0.1","url":""},"suggested":{"version":"v0.0.1","url":""},"rollout":{"seed":"","cursor":""}},"storagenode":{"minimum":{"version":"v0.0.1","url":""},"suggested":{"version":"0.30.5","url":"https://github.com/storj/storj/releases/download/v0.30.5/storagenode_{os}_{arch}.exe.zip"},"rollout":{"seed":"41304fbfefbf779f672399fd7a3c01745d6e1226e9dfd9b76090c8c85e262386","cursor":"ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff"}},"storagenode-updater":{"minimum":{"version":"v0.0.1","url":""},"suggested":{"version":"0.30.5","url":"https://github.com/storj/storj/releases/download/v0.30.5/storagenode-updater_{os}_{arch}.exe.zip"},"rollout":{"seed":"9178fcdc3fec42821738287ef3a12947df5d82af0c17fe7d05e251a6d149455e","cursor":"ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff"}},"uplink":{"minimum":{"version":"v0.0.1","url":""},"suggested":{"version":"v0.0.1","url":""},"rollout":{"seed":"","cursor":""}},"gateway":{"minimum":{"version":"v0.0.1","url":""},"suggested":{"version":"v0.0.1","url":""},"rollout":{"seed":"","cursor":""}},"identity":{"minimum":{"version":"v0.0.1","url":""},"suggested":{"version":"v0.0.1","url":""},"rollout":{"seed":"","cursor":""}}}}

Seems like we should increase all timeouts to get data from satellite and version check services.

PS. This is my storage node from far location, about 140ms latency to us-central-1 satellite.

39

Implemented on my 5 nodes with no issue to report :wink:
Should I remove the setting from config.yaml, now that itā€™s tested, or can I just leave it there?

40

The setting wasnā€™t in any of my nodes after the update. Had to add it manually.

1 Like
41

Same here. It will only be implemented forcibly on the next release, I think.

1 Like
42

Yes, but likely as a default if the setting isnā€™t in the config.yaml. I donā€™t expect it will be added to the file automatically. Until very recently I still had a very old version of that file and decided to rename it and let the node recreate it. Iā€™m pretty sure thatā€™s the only way to get all new/recent settings in there and clean up the deprecated ones. This will obviously overwrite all existing settings as well and return it to default. It would be nice to have a default version that will be automatically updated with new defaults by the software and one with only user adjusted settings to overwrite that default. But thatā€™s a different topic.

6 Likes