Two days ago (the 25th) I had a network outage affecting two nodes (both of them being in different /24s), and I had to replace a piece of network equipment. During the replacement, inbound UDP traffic between the nodes got confused (TCP was unaffected), resulting in one node getting the UDP-IN traffic from the other. This resulted in log messages of one node telling me that I received data with an invalid node ID from the satellites.
The combined outage (not being reachable and traffic being confused) was ~2-3 hours. After this, this was fixed.
Since then, I am not getting any inboard traffic anymore, only egress. A third node on an unrelated network but registered to the same wallet and email has also stopped receiving inbound data.
Is there a chance my email/wallet has been blacklisted due to the network traffic looking tampered and thus classified as an attack? My setup is fairly non-standard, in the sense that I have full control over my internet routing, but this does come with the consequence that if I make a mistake it affects a large part of my IP space and traffic.
How do I fix this, or how long will it take to clear?
Egress remains stable, ingress is a flat zero. Audit traffic is happening.
This happened after an outage that could be considered malicious use because of the way network traffic got misrouted so I’m curious if my wallet/email got blacklisted.
Last I checked ingress will stop if your 2 versions behind the current version. Which is why theres an auto updater built into docker now so You dont get behind. So not sure this one hasnt updated yet…
–Oh its because its not docker, sorry yes update your node asap.–
I would recommend using DOCKER to keep your node up to date, unless you know how to code, in which case you need to be reviewing JSON data within version.storj.io to find the latest released or can be found via GITHUB too
The version mismatch was indeed what solved it - thanks a lot! One thing which is really annoying is that the GH repository lists 1.68.2 as the latest version, so I was being under the impression I was up to date. I did not expect pre-release versions to count here.
Docker is not an option here in this particular setup; the complexity of my environment is beyond most storage node operators - I’m ‘the mainframe dude’ from r/homelab. I run storj primarily as a learning experience, and because the infrastructure here is running anyway, and extra CPU cycles being consumed add little to no extra overhead for me.
Yeah, I was keeping an eye on the GH releases, but it turned out that this wasn’t the right thing to monitor. I keep a bunch of stuff up to date by monitoring the GH repos, so that’s why I heaped it with that.
Yeah I wouldnt look only at github thats not the latest version out. Always check https://version.storj.io/ it has the latest version and the min version you can have. It wasnt even a pre-released version because the pre released is much later.
Next release is already out which is v1.71.1
Next to the storagenode binary there is a storagenode-updater binary. Run it alongside the storage node, as a service, it shall keep your storagenode up to date. (Except on FreeBSD, where it fails to restart the storagenode following update)
It’s not a matter of just pulling the newest as reported by the version URL; storagenode-updater gradually deploys the new release to nodes according to their node id and filter, to prevent bad update from nuking entire network.
I did the same thing. I have only been updating to “Latest” versions and not the pre-releases. Just figured that made sense. Apparently that was my bad for assuming pre-releases were not the same as latest releases. Can someone explain this one to me? How a pre-release is considered more up to date or stable than the designated latest release? Just curious because I just thought it was good practice to only use the “latest” and / or “stable” versions of software in a production environment. Pre-release makes it sound like it’s potentially unstable.
