Is anybody having issues with v1.41 and 1.42?

Hello,
today I have built and installed v1.42.4. Everything seems to be good, but in the logs I see a huge amount of ping errors like

2021-11-13T13:48:56.257+0100 ERROR contact:service ping satellite failed {"Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "attempts": 8, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:138\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

Really a lot! If I switch back to v1.40.4 things go back to normal and the log is clean and full of upload. Note that I am not using docker and starting STORJ as a service on a 32bit ubuntu machine. I just change a symlink to the different versions to switch between them.

The same thing happened with v1.41.2. If that normal?
For now I am holding the upgrade, but I am seeking for your advice: should I upgrade and give it time or will this affect reputation?

Thanks in advance,
Stefano

The reason is that your node is trying to check-in too often, usually this is a consequence of closed port, because either external or internal IP got changed.

Please use this check-list to troubleshoot the offline issue:

A can only add one more check:

Hi Alexey,
thanks as usual for your prompt support, but I am afraid it is not any of those cases. As I said I just shutdown the old version and bring up the new one, everything else stays the same. And in fact I receive some files too… but a lot of satellite errors. The node is reported online from the console and of course I was able to ping it from pingdom.

For more details, this is what I do (which is affected by the errors):

sudo systemctl stop storagenode
ln -s storagenode-v1.42.4 storagenode
sudo systemctl start storagenode

To revert (not affected by the error):

sudo systemctl stop storagenode
ln -s storagenode-v1.40.4 storagenode
sudo systemctl start storagenode

The versions:

./storagenode.v1.40.4 version
2021-11-13T17:23:31.849+0100	INFO	Configuration loaded	{"Location": "/home/storj/.local/share/storj/storagenode/config.yaml"}
Release build
Version: v1.40.4
Build timestamp: 09 Oct 21 15:00 CEST
Git commit: 9cb33b0ab79b044c6da6b9d38b2dde6440dad781

./storagenode-v1.42.4 version
2021-11-13T17:24:06.888+0100	INFO	process/exec_conf.go:284	Configuration loaded	{"Location": "/home/storj/.local/share/storj/storagenode/config.yaml"}
2021-11-13T17:24:06.888+0100	DEBUG	process/tracing.go:72	Anonymized tracing disabled
Development build
Version: v1.42.4
Build timestamp: 13 Nov 21 10:57 CET
Git commit: 0265c10abb98c5dbe10aade180d272a3ae42f5f0-dirty

Hope this helps.
Ste

If it’s online after the update - then let it run. I do not have any such errors.

Hi, I need to come back on this thread, because if I run v1.40.4 I do not have uploads any more (but no errors)… if I upgrade I have lots of errors and still no uploads, plus I am concerned it can affect node reputation.

Now, I think I have enough evidence the node is online and reachable, I can provide it again or in more details if needed.

I can easily reproduce the issue switching between versions, therefore I am starting to think there might be something different in the configuration that is needed since v1.41? can anyone post the config of a running v1.44.2 (or whatever is the latest version)?

Many thanks in advance.

Hi Stefano,

If you are running v1.40.4 you are not going to get any uploads since any node running 3 or more minor versions behind will not get uploads. If you are more than 3 minor versions behind your node is at risk of being disqualified.

You really shouldn’t downgrade a node after updating as databases can change and may not be backward compatible. It is very unlikely that this is a config file issue since the node will just fallback to defaults if the specific config option isn’t in the file. In fact, most are commented out of the config file since the node is designed to work with the default options.

If you are still getting the rate limit error, you could try resetting all of your network equipment. Sometimes routers/modems need a reboot to clear NAT errors, especially ISP provided equipment. How long of a time period did you let the node run and continually see the rate limit error? I have seen this before on my node, but it always sorts itself out eventually. Just keep an eye on your online/audit/suspension scores and see if they are remaining steady.

Hi,

If you are running v1.40.4 you are not going to get any uploads since any node running 3 or more minor versions behind will not get uploads. If you are more than 3 minor versions behind your node is at risk of being disqualified.

ok, that’s explain well why I was not having uploads prior the upgrade.

If you are still getting the rate limit error, you could try resetting all of your network equipment. Sometimes routers/modems need a reboot to clear NAT errors, especially ISP provided equipment. How long of a time period did you let the node run and continually see the rate limit error? I have seen this before on my node, but it always sorts itself out eventually. Just keep an eye on your online/audit/suspension scores and see if they are remaining steady.

I kept it like that all night yesterday and it is all day now today.
I restarted the router, but still the same.

I really think there is something quite bad with my node which is not networking: now I am on the latest version downloaded from git (1.44.1) and I am still at 0B ingress and lot of errors. I can prove that the moment I run v 1.40, the errors go away. I can provide the logs if needed. If it was a networking problem, shouldn’t it be there regardless the version?

The node selection will only upload to storage nodes running the release version.

2 Likes

good point! what’s the latest production release version? I’ve built v1.41.2, v1.42.4, v1.43.1, v1.43.2, v1.441 and all seem to be development versions

ok, I’ve got the information through storj support and more research on the forum.

curl -s https://version.storj.io | jq .processes.storagenode.minimum.version
“1.43.1”

So I’ve rebuilt from source checking out v1.43.1:


./storagenode version
2021-12-01T23:29:23.218+0100	INFO	process/exec_conf.go:284	Configuration loaded	{"Location": "/home/storj/.local/share/storj/storagenode/config.yaml"}
2021-12-01T23:29:23.219+0100	DEBUG	process/tracing.go:72	Anonymized tracing disabled
Development build
Version: v1.43.1
Build timestamp: 01 Dec 21 23:22 CET
Git commit: 163a045ef9496dead5f68815911b8bd866ba0463-dirty

it looks like it builds still a development version. how can I build a production version? (please note I have built v1.40.4 as well)

Just remove the code source modification you applied on top.

I have not applied any modification… but let me try to start from scratch (rm, git clone, …)

This is the script I am following, provided by Alexey

git clone https://github.com/storj/storj.git
cd storj
git checkout <version>

cd web/storagenode && npm install && npm run build
cd ../..
go-bindata -prefix web/storagenode/ -fs -o storagenode/console/consoleassets/bindata.resource.go -pkg consoleassets web/storagenode/dist/... web/storagenode/static/...
/usr/bin/env echo -e '\nfunc init() { FileSystem = AssetFile() }' >> storagenode/console/consoleassets/bindata.resource.go
gofmt -w -s storagenode/console/consoleassets/bindata.resource.go
./scripts/release.sh install ./cmd/storagenode

Wrong order. The first commands will modify the source code and the last command will understand that as a dirty state.

oh… can you please specify better which order should I follow?

This is actually consistent with Build Storj from Source for arch armel/ARMv5 builderror: "build constraints exclude all Go files" - #2 by Alexey

no way, I have done it from scratch but it still buiilds a development binary. here the log:

storj@storj32:~/STORJ$ rm -rf storj
storj@storj32:~/STORJ$ git clone https://github.com/storj/storj.git
Cloning into 'storj'...
remote: Enumerating objects: 94340, done.
remote: Counting objects: 100% (5652/5652), done.
remote: Compressing objects: 100% (2102/2102), done.
remote: Total 94340 (delta 3673), reused 5119 (delta 3456), pack-reused 88688
Receiving objects: 100% (94340/94340), 78.53 MiB | 3.33 MiB/s, done.
Resolving deltas: 100% (66762/66762), done.
Checking out files: 100% (2214/2214), done.
storj@storj32:~/STORJ$ cp -r storj storj.new
storj@storj32:~/STORJ$ cd storj
storj@storj32:~/STORJ/storj$ cd web/storagenode && npm install && npm run build && cd ../..
npm WARN deprecated request-promise-native@1.0.9: request-promise-native has been deprecated because it extends the now deprecated request package, see https://github.com/request/request/issues/3142
npm WARN deprecated @hapi/topo@3.1.6: This version has been deprecated and is no longer supported or maintained
npm WARN deprecated @hapi/bourne@1.3.2: This version has been deprecated and is no longer supported or maintained
npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated
npm WARN deprecated har-validator@5.1.5: this library is no longer supported
npm WARN deprecated eslint-loader@2.2.1: This loader has been deprecated. Please use eslint-webpack-plugin
npm WARN deprecated resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated
npm WARN deprecated left-pad@1.3.0: use String.prototype.padStart()
npm WARN deprecated sane@4.1.0: some dependency vulnerabilities fixed, support for node < 10 dropped, and newer ECMAScript syntax/features added
npm WARN deprecated chokidar@2.1.8: Chokidar 2 will break on node v14+. Upgrade to chokidar 3 with 15x less dependencies.
npm WARN deprecated chokidar@2.1.8: Chokidar 2 will break on node v14+. Upgrade to chokidar 3 with 15x less dependencies.
npm WARN deprecated querystring@0.2.0: The querystring API is considered Legacy. new code should use the URLSearchParams API instead.
npm WARN deprecated html-webpack-plugin@3.2.0: 3.x is no longer supported
npm WARN deprecated babel-eslint@10.1.0: babel-eslint is now @babel/eslint-parser. This package will no longer receive updates.
npm WARN deprecated @hapi/address@2.1.4: Moved to 'npm install @sideway/address'
npm WARN deprecated uuid@3.4.0: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated @hapi/hoek@8.5.1: This version has been deprecated and is no longer supported or maintained
npm WARN deprecated @hapi/joi@15.1.1: Switch to 'npm install joi'
npm WARN deprecated core-js@2.6.12: core-js@<3.3 is no longer maintained and not recommended for usage due to the number of issues. Because of the V8 engine whims, feature detection in old core-js versions could cause a slowdown up to 100x even if nothing is polyfilled. Please, upgrade your dependencies to the actual version of core-js.

added 2293 packages, and audited 2295 packages in 3m

141 packages are looking for funding
  run `npm fund` for details

65 vulnerabilities (51 moderate, 14 high)

To address issues that do not require attention, run:
  npm audit fix

To address all issues possible (including breaking changes), run:
  npm audit fix --force

Some issues need review, and may require choosing
a different dependency.

Run `npm audit` for details.

> storj-storagenode@0.1.0 build
> vue-cli-service build


⠙  Building for production...Starting type checking service...
Using 1 worker with 2048MB memory limit
 WARNING  Compiled with 3 warnings                                                                         12:18:23 AM

 warning  

asset size limit: The following asset(s) exceed the recommended size limit (244 KiB).
This can impact web performance.
Assets: 
  fonts/font_medium.4030a28e.ttf (254 KiB)
  fonts/font_regular.b396b059.ttf (251 KiB)
  fonts/font_bold.97decd2b.ttf (254 KiB)
  js/app_f8637eb3bbfd8a2365a9.js (356 KiB)
  js/vendors_f8637eb3bbfd8a2365a9.js (599 KiB)

 warning  

entrypoint size limit: The following entrypoint(s) combined asset size exceeds the recommended limit (244 KiB). This can impact web performance.
Entrypoints:
  app (1.05 MiB)
      js/vendors_f8637eb3bbfd8a2365a9.js
      css/app.4ce344e8.css
      js/app_f8637eb3bbfd8a2365a9.js


 warning  

webpack performance recommendations: 
You can limit the size of your bundles by using import() or require.ensure to lazy load some parts of your application.
For more info visit https://webpack.js.org/guides/code-splitting/

  File                                       Size                               Gzipped

  dist/js/vendors_f8637eb3bbfd8a2365a9.js    598.79 KiB                         176.35 KiB
  dist/js/app_f8637eb3bbfd8a2365a9.js        356.26 KiB                         59.36 KiB
  dist/css/app.4ce344e8.css                  118.06 KiB                         19.39 KiB

  Images and other types of assets omitted.

 DONE  Build complete. The dist directory is ready to be deployed.
 INFO  Check out deployment instructions at https://cli.vuejs.org/guide/deployment.html
      
storj@storj32:~/STORJ/storj$ go get github.com/go-bindata/go-bindata/go-bindata
go: downloading github.com/go-bindata/go-bindata v3.1.2+incompatible
go get: installing executables with 'go get' in module mode is deprecated.
	To adjust and download dependencies of the current module, use 'go get -d'.
	To install using requirements of the current module, use 'go install'.
	To install ignoring the current module, use 'go install' with a version,
	like 'go install example.com/cmd@latest'.
	For more information, see https://golang.org/doc/go-get-install-deprecation
	or run 'go help get' or 'go help install'.
go get: added github.com/go-bindata/go-bindata v3.1.2+incompatible
storj@storj32:~/STORJ/storj$ go-bindata -prefix web/storagenode/ -fs -o storagenode/console/consoleassets/bindata.resource.go -pkg consoleassets web/storagenode/dist/... web/storagenode/static/...
storj@storj32:~/STORJ/storj$ /usr/bin/env echo -e '\nfunc init() { FileSystem = AssetFile() }' >> storagenode/console/consoleassets/bindata.resource.go
storj@storj32:~/STORJ/storj$ gofmt -w -s storagenode/console/consoleassets/bindata.resource.go
storj@storj32:~/STORJ/storj$ ./scripts/release.sh install ./cmd/storagenode
Build timestamp: 1638401160
Git commit: b3cea3d1b675d8c43be55ed01633bf131bf8a5e7-dirty
Tagged version: fatal: no tag exactly matches 'b3cea3d1b675d8c43be55ed01633bf131bf8a5e7'
Running go install ./cmd/storagenode
storj@storj32:~/STORJ/storj$ cd
storj@storj32:~$ cd go/bin/
storj@storj32:~/go/bin$ ./storagenode version
2021-12-02T00:27:30.089+0100	INFO	process/exec_conf.go:284	Configuration loaded	{"Location": "/home/storj/.local/share/storj/storagenode/config.yaml"}
2021-12-02T00:27:30.089+0100	DEBUG	process/tracing.go:72	Anonymized tracing disabled
Development build
Build timestamp: 02 Dec 21 00:26 CET
Git commit: b3cea3d1b675d8c43be55ed01633bf131bf8a5e7-dirty

As far as I see, the dirty flag depends on the status of git diff --stat.

Can you please share the output of this command? You may have some unnecessary temporary file in your dir…

git diff --stat 
 go.mod                            | 1 +
 go.sum                            | 2 ++
 web/storagenode/package-lock.json | 2 +-
 3 files changed, 4 insertions(+), 1 deletion(-)
```Preformatted text

Checkout all those changes before running ./scripts/release.sh install ./cmd/storagenode

That’s what I did for getting a build version.

By the way, I don’t see the checkout to the specific version, if you want to build the v1.43.1 then after cloning the repository do git checkout v1.43.1.

Thanks! I’ll try.

You are right on the missing checkout. I cleaned the output ant canceled to many rows. :slight_smile:

I’ll redo everything from scratch with your suggestion.