No Ingress on one of my nodes since 5/12

[Sokar]

I’ve noticed that one of my two nodes stopped getting any ingress since 5/12 (which might correlate when I got 1.29.3 but not sure). There is periods when ingress returns but than quickly goes away. I still have space available and I’m running version 1.29.3 which I’ve verified is the latest version. I have a second node (in a completely different location) which is running the same version and has not run into this same issue. I am still getting Egress and the node is still stating online with a perfect audit score. I had not changed anything on this node so was surprised to see the ingress drop off so dramatically.

Looking at the logs the only errors I see could be related as it seems like it fails to get storage stats from the nodestats. But not certain why this would have started all of a sudden.

2021-05-16T14:08:57.810Z        ERROR   nodestats:cache Get disk space usage query failed       {"error": "node stats service error: context canceled", "errorVerbose": "node stats service error: context canceled\n\tstorj.io/storj/storagenode/nodestats.(*Service).GetDailyStorageUsage:114\n\tstorj.io/storj/storagenode/nodestats.(*Cache).CacheSpaceUsage.func1:177\n\tstorj.io/storj/storagenode/nodestats.(*Cache).satelliteLoop:261\n\tstorj.io/storj/storagenode/nodestats.(*Cache).CacheSpaceUsage:176\n\tstorj.io/storj/storagenode/nodestats.(*Cache).Run.func3:132\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2021-05-16T14:09:28.206Z        ERROR   nodestats:cache Get stats query failed  {"error": "node stats service error: context canceled", "errorVerbose": "node stats service error: context canceled\n\tstorj.io/storj/storagenode/nodestats.(*Service).GetReputationStats:74\n\tstorj.io/storj/storagenode/nodestats.(*Cache).CacheReputationStats.func1:154\n\tstorj.io/storj/storagenode/nodestats.(*Cache).satelliteLoop:261\n\tstorj.io/storj/storagenode/nodestats.(*Cache).CacheReputationStats:153\n\tstorj.io/storj/storagenode/nodestats.(*Cache).Run.func2:120\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

[Pac]

Hi @Sokar and welcome to the forum!

@Sokar

Get disk space usage query failed

That’s a bit concerning, maybe something’s up with your disk.

You should probably run a fullscan to ensure it’s fine and is not starting to fail.

If you’re on linux, maybe this thread can give you some hints on what to do to check the disk if needed:

Node crashed and restarted after some "error getting current used space", "lstat […] bad message" and "Unrecoverable error"

Hi there I noticed one of my node was suspiciously very active, and found out that was because it crashed at some point and restarted: the disk activity I’m seing is simply the filewalker rebrowsing the whole disk. Here are the few log lines around when it crashed and abruptluy restarted: [...] 2021-03-30T17:15:21.127Z INFO piecestore download started {"Piece ID": "BSBUCI6M2C3AKHRSJ3JHM4PFVND55RNJA7XUR4QEEVUZPYVS62DA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqk…

[teicheld]

I get also no ingress data since a while.
checked log and but haven’t seen something suspicious: The “uploaded” entries just disappear.
Download and delete entries remain until now.
Status and QUIC are green, but online percentage goes down.
I am running v1.47.3 with autoupdate.
Tried good old restart of both: software and hardware, but with no effect.
DynDNS looks good

[peem]

“storagenode”:{“minimum”:{“version”:“1.50.4”

https://version.storj.io/

Changelog v1.50.4

The changelog can be found here: Release v1.50.4 · storj/storj · GitHub For docker nodes Usually, we ask docker node operators to not update to a new version too early because of the risks. This time we need as many nodes as possible updating early to 5f2777af9-v1.50.4-go1.17.5 The reason for this is that the new docker image contains the Linux updater. We haven’t been able to test it on all operating systems and architectures. So this is the last chance to find and fix potential bugs in time …

[teicheld]

Thanks for the quick answer.
I am now struggeling with the config file:

I was using docker on a raspberry pi.
I have to switch to the binary version or wait, do I get this right?
I am trying to use the binary, but it refuses to load the operator.* variables of the config, although I set the values: operator.wallet: "0x19d106b9f7e7836d4848fab0146131a6e4a88a2a"
I get the following feedback, if I try to run ./storagenode run :
2022-03-30T21:05:25.390+0100 INFO Configuration loaded {“Location”: “/home/pi/.local/share/storj/storagenode/config.yaml”}
2022-03-30T21:05:25.397+0100 WARN Operator email address isn’t specified.
2022-03-30T21:05:25.397+0100 ERROR Invalid configuration. {“error”: “operator wallet address isn’t specified”}
Error: operator wallet address isn’t specified
.

[Alexey]

If you use docker, you should specify all parameters in docker run command, not in config.yaml.
See Storage Node | Storj Docs

Please update your node manually: Software Updates | Storj Docs, then configure automatic updates.

If you run a 32 bit OS, then you should wait for the next version, it should be with arm32 support.
But now please update to at least 1.49.5: 6e7a89161-v1.49.5-go1.17.5

[teicheld]

Thanks for all your help: My node is online again!

What happend:

1. I was running docker on arm32 and was excluded because of an old version.
2. Autoupdate was not reacting and manual update from the instructions was not working because of non matching architecture.
3. So I tried to run a newer version from binary without docker and edited the config.yaml. From this point my node was not even reachable anymore.

I solved it through switching to 64bit architecture, deleting the config.yaml and setting up the docker node again. Took me quite some time to identify the problem but I am happy again. Nice software!