Its hard to balance the email stuff 
unfortunately in the past when we have sent out emails about software updates we get people unsubscribing from our emails entirely which stinks because then when we need to notify them of something truly urgent they dont get an email. The approach we have been taking lately is to try and be more selective and send the email our specifically to the people who are affected. For this change we sent an email out to about 200 nodes because those were the nodes that would be affected by the change. I like the idea posted in this thread about making it a notification in the dashboard.
3 Likes
In that case the dashboard notifications seem good too.
Typically companies let you select what kind of emails you want to receive: news, important notifications about the node, etc.
But I understand that this has to be implemented too and is probably a lot of effort for “just a few information”.
2 Likes
Update can fail, leaving the node offline. Downtime is/will be punished, which means that the node should be offline as little as possible.
Automatic update most likely would run at a time when I am no there to fix whatever problem caused the node to be offline (asleep, away etc). More importantly, there is no way to test it unless I wanted to run the automatic update software, wait 12 hours after a new version is released and then watch the log for 60 hours.
Which means that automatic updates are no-go for me. I don’t want to have less than 1h of downtime in a month only to get 9 hours of downtime because of failed update. Waking up in the middle of the night might be acceptable for hardware failure etc, but not acceptable for something preventable - like an update.
So, I have been manually updating my node all the time. I update when I know I’ll be there in the hour after the update to try to fix whatever problem would come up.
However, I would like to have an emergency option - if I forget or am unable to update my node long enough so it becomes too old and stops receiving data, I want it to be updated right then, because if the update script waits 72 hours after that, an even newer version could be released, leading to DQ.
And I completely understand not wanting everyone to update at the same time, but, IMO, the uptime requirement means that I have to be present while my node updates.
3 Likes
Heh… A major task that Storj attempts to do is to coördinate a horde of operators, each having different preferences. This is a hard problem. For the great engineers that work at Storj it might even be more difficult than just coördinating a horde of machines: machines don’t complain, machines don’t make silly mistakes, machines are less diverse, machines follow the orders much more precisely. Failing to properly communicate “just a few information” is actually a risk to the health of the whole network, as evidenced recently by the Windows upgrade problem or lack of prompt upgrades of storage nodes.
Given this premise I see two ways of performing this task. Storj should probably either clearly state the “official” channel of communication for important notifications, or attempt to reach out to operators over channels operators prefer. Maybe both, but I suspect doing so will take precious resources from the Storj team, and I believe doing just one of them properly should be enough.
Failing to do so will at best result in Storj wasting operators’ time, at worst risking that operators leave the network.
From my point of view, Storj fails to do the former—if Storj wants to do the former, they should make a clear statement in the documentation, probably right in the requirements section before the installation procedure, together with consequences for neglecting to do so, e.g.: «To properly operate storage nodes accepting data from the Tardigrade.io satellites, you are required to follow the announcements tag on the Storj forum. This is the place that notifies storage node operators about important events within the Storj network, such as necessary maintenance procedures or temporary workarounds for software problems. Failure to do so may result in nodes not receiving data, becoming suspended or disqualified from the network». It might involve emails, forums, dashboards, logs, whatever — as long as the requirements are clear.
From my point of view, Storj also fails to do the latter—some announcements are made only on the forum, some apparently over email, some in the dashboard, some in the logs. If a SNO doesn’t observe all of these, they might miss an important announcement, like in the case of recent Windows upgrade bug. For example, I don’t visit the dashboards—I have 10 nodes now, it would simply take too much time to go over all of them, wait until it loads with the unpleasant loading animation, etc. Some don’t follow the forum regularly, as it is clearly seen by the number of people asking what are those errors from the Stefan’s satellite. And so on… Another factor is having the to check all those sources of information takes a lot of operators’ time.
Given that Storj requires from SNOs to operate their nodes at a certain standard, SNOs have the right to require from Storj some standard of communication.
So, I’d ask a question: what is the actual Storj’s plan for Storj-to-SNOs communication?
6 Likes
Most of what I think has been said above.
As always, a new path to disqualification is fine as long as we get notified way before it definitively kills our node.
That’s basically the only thing I hate about being an SNO: there are still situations where you could wake up in the morning and discover your hard work for months has gone out the window, for good.
This version new hard requirement is another way to DQ: we do need a way to configure nodes so they can notify us via our preferred chanel.
Because otherwise, I could have my node disqualified simply because I forgot to restart the watchtower after maintenance…
4 Likes
If it would help - the update requirements has been added to the documentation a while ago: https://documentation.storj.io/before-you-begin/prerequisites#keep-your-node-updated
2 Likes
Just wantd to chime in that i love emails. Please enable as many email notifications about suspensions, dq, old version, etc.
It makes it easy to find a problem if my normal monitoring tools are missing something.
Thanks.
5 Likes
maybe we could get some configuration options on our account on the forum which defined our preferences for email and such… so those that doesn’t mind getting emails for when there is critical information related to storagenode operation which we should account for…
1 Like
and how would that help SNOs that are not on the forum?
well they have to create an account somewhere when they make a node i seem to remember… then they can put it there… just so people can choose what they want… i duno, just thought that might be a way most could accept.
you don’t have to make an account to be a SNO. but you provide an email in your docker run command (whether it is fake or real…)
You have to provide an email address for the authorization token, unless that is no longer needed.
1 Like
so make it so people have to make an account so they can configure their preferences… i mean it’s cool that one doesn’t have to make an account to get paid… but really one basically have to make an account everywhere else online, why should this be any different…
and i kinda like that i can have one spot for all the information i need to take into account, which would also make sense for people much less active in regard to their nodes…
and the plus side would be if thats the way people get their critical information i could be setup in a nice easy to keep track of way…
If you specified a correct email address in the storagenode, you will receive an important notifications to there
2 Likes
maybe it’s because i keep my node up to date lately… have gotten emails a few times about various stuff
usually if my node version is lagging behind a bit…
It’s not clear to me what notifications we receive, how long after an event, and what events don’t trigger notifications.
I’ve been offline for 24h lately because of an issue with my ISP while I was away. I knew about it thanks to my uptimerobot monitoring. Never got an email from Storj for this event.
But I did get an email notification when one of my nodes got suspended a few months back.
Is this explained/detailed somewhere ( what causes an email notification or not)?
2 Likes
I think email notifications are crucial for a majority of SNO.
4 Likes
At the moment only emails regarding outdated software, suspension and disqualification.
The “offline” emails could be activated back when the suspension and disqualification for downtime would be enabled.
6 Likes
it is bullshit to get notification to email. most people even dont use email. personality im checking email 1 time per month or per 2 month i have busy days, all notification shut appear in node dashboard not send vea email. plus 50% people not use forums is not point to read forums if node running 24/7 without trouble, second storj is not finished project missing many stuff in dashboard,like notifications,like add second node to dash for example i have 20x nodes and all i need to check individual or is online it shut be in one page, second is missing stuff extend hdd as most person on beginning use small capacity hdd and later it trouble if you wanna move to large capacity hdd. also payment system is terrible with holding profit it suppose to be 100% if full month node running fine. if have any stops take fees for offline node or lost data end of month. like most data storage take fees for not keep promise run node, also with offline node couple days ago some idiots cut a tree on street and snap power and internet cables on friday till monday evening all my nodes stay offline and it was no my fault and is no report emails chats or ever about his situation and now all my nodes are suspended
1 Like