Sure, so if something is not done correctly by STORJlabs, what am I supposed to do? Should all SNOs just leave instead of demanding a change?
So operating within the boundaries defined by STORJlabs is discouraged? wth?
I mean, driving 54km/h in a 50km/h might be discouraged too but is accepted behaviour within the defined boundaries meaning I can’t be punished for that.
But if I get punished for driving 54km/h, it would suddenly be outside the defined boundaries.
So if you punish my node for being too old although it is inside the defined boundaries, it suddenly is not inside the defined boundaries because you suddenly have 2 different boundaries.
How is anyone to know about that? How can an SNO be sure he is operating inside the correct boundaries if you don’t lay them out correctly?
You would get punished for driving 54km/h technically, it’s just within some boundaries of accuracy. That comparison does not fit well.
It’s not a requirement to be on the latest version while operating a node. We do provide tooling to help you ensure you Are and thus get the best experience. This is basically an measurement of trust or in other words how well you follow the guidelines to be updated.
Besides all that, you can see that the storagenode section on https://Version.Storj.io has two minimum versions defined. One is the global minimum version, the other one the recommended minimum.
If it’s not a requirement, why do I get punished for it?
Guidelines are not rules. And if they are not well defined, how can I follow them properly? Nowhere does it say “there is a minimum allowed version you can run and there is a minimum version you need to have to get ingress”.
I have a SNO board showing the minimum version. I have to rely on that information. If I have to check 20 different websites and forum posts to know if my SNOboard information is correct, you are doing something wrong.
“recommended minimum” is a recommendation. Punishment for not following a recommendation while not violating rules is just wrong. You should change your wording then.
As you can see multiple SNOs ran into the problem of not receiving any ingress anymore and nobody knew why. Only through trial and error and investigation we found the problem. And we were not on 1.9.0, we were on 1.12.3 which is a very recent version! So there is an obvious lack of information about a new boundry not shown on the dashboard or the guidelines.
I respectfully disagree with almost everything you said. We have official guidelines how to keep your node up to date and if one decides to act against those, then you are on new territory.
Also it’s not a punishment, rather than warning. You will still get the same amount of downloads as before. We just do not trust the node enough any longer to store new data until it catches back up. It’s simply a precaution.
The windows updater automatically respects our encouraged version, same does the watchtower route. Going manual here is discouraged and ends with warnings.
Would be great if it could also be added to the SNOboard so that it doesn’t say “Running the minimal allowed version: 1.9.0” but something like:
“Running the minimal allowed version: 1.9.0, Recommended minimal allowed version: 1.12.3”
Just to iterate over the intention again:
We do not want to make storagenodes unhappy by not giving them data! Our only intention here is to ensure, we can keep making improvements on our protocol and code base without being stuck as a big chunk of nodes does not update regularily or in our cadence.
This percentage is in the neighborhood of 10-20% from all active nodes on the network.
At that point, when deciding to increase our minimum version from v1.9 to lets say v1.12, then we would need to start repairing quite a lot of data, which is making the “outdated” nodes even more unhappy if they then decide to upgrade and notice tons of deletes.
This is a fairly tricky balancing act between development speed and network stability, which directly affects the storagenode income (and for the most the outdated nodes).
As an highlight we are very close to having the linux updater be available (which can be run natively and will be ran directly in the docker image as well). This helps many of our unique and special setups i have seen in this forum so far.
Our improvement story does not end here, we are working on a native debian package than can be installed via apt, yum, etc.
TLDR:Please do not see our second measurement as requirement, rather than safety net before a node will entirely lose its data and thus potential payout, if all data has been repaired of which the given node held pieces.
Thanks for summing that up again.
I can absolutely follow your intentions, they are very reasonable.
My only problem was with the lack of information about the “minimum recommend version” and the consequences of falling below that version. (which never happened to me before as I typically update within a week after rollout but there were many new versions within a very short time)
Looking forward to the linux updater even though I must admit that I’m a bit careful about any kind of automatic update as new features could cause problems at node startup which I therefore prefer to monitor instead of having it update automatically.
I can recommend to suggest features that the updater could inform you about the next update being available and when it would deploy it, one doesnt interfer.
Even if i dont want to say it, but its similar to what Windows Update does.
We all need to work on this together and without proper feedback around updater tools, we are flying blind
Sounds good. It would certainly help if I can choose the time and weekdays it can update, so that I would be available during those hours. How would the notification on linux work though? Through the SNOboard? Email?
i will do that, i will need to start to compile lists of good feature requests…
think i got 3 now that i haven’t gotten around to throwing on the board, including this new one…
The provided tooling goes against security best practices which is why I will not use the current iteration of tooling. As others have said, making sure I’m available to deal with problems is also a concern.
Is there any work being done to come up with update mechanisms that don’t require root access to the container host?
@stefanbenten my node has been on v1.13.3 since 4th october. Due to the satellite termination, I’ve also had more than 50% of the space freed. I have not been receiving any new data at all, besides a burst of 10-20GB in one day on 5th after the node restart. I’ve had less than 500MB of ingress per day since then. I’m running a 15+ months old node.
At first, I though the issue might be related to the huge amount of trash, but now more than a week has passed since the auto-delete of trash at end of the 1st week of October. Still no data.
You seem to be doing some rather counter-productive changes for your SNOs literally a month before Filecoin is launched. Keep it up and you might see a huge exodus away from storj.
I’m at a loos of seeing any improvements you’re making by getting operators on board to carry out upgrades sooner, so that there are no negative effects to the network.
The proposed solution of this thread states that the issue disappears after your node has been upgraded to the desired latest version, however that simply isn’t the case
edit: Keep in mind the overhead of running filecoin is much higher than being a storj SNO, that would immediately translate to having your best SNOs depart and only the normies staying behind.
The topic states that if you run a 1.12.3 or older you will have literally zero ingress.
500MB ingress doesn’t looks like a zero, so it’s not the case.
The age of the node does not guarantee the increased ingress, only customers’ activity.
Please, check your logs for errors, especially related to databases.
If there is no errors and your node not suspended and not disqualified, it’s working fine, especially if you see a regular traffic, not only audit and repair.
As @Alexey just responded, there is no fixed inbound traffic if you are on a certain version either.
There is simply a smaller amount of data being uploaded and a bigger amount being downloaded.
My nodes have 75% of its requests as GET requests for example.
If you are on at least the Version thats on processes.storagenode.minimum on https://version.storj.io, then you are all set.
Please do not assume that customers will be uploading data all day and even mean we are doing the exact opposite of acquire new ones without knowing their usage pattern.
In advance please also don’t forget that with every day dozens of nodes join the network, making the load be spread more across them.
