Hmm maybe Now I’m not sure
Yeah, but disqualification email is kinda late, no?
I got warned by email many months back when one of my nodes got suspended.
@naxbc I guess you already double checked the spelling of your email adress configured for your node.
If it’s valid, it feels like it’d be worth reaching out to the support team so they can investiagte and fix that.
Or ping Jocelyn about that on the topic @kevink linked above.
StorjLabs seem to go easy on emails and that’s fine, but urgent alerts like a suspension should definitely trigger an email to the SNO, in my opinion.
Yes I’m pretty sure regarding my email spelling.
Dispite it was a failure on my volume, it should trigger an email regarding the suspension in any case.
I’ve already sent @Alexey my NodeID is and deleted my node. It’s lost, on my fault, or better HDD fault.
Lost 300$ in Held Amount, but it paid for itself as it was my oldest node ever since StorjV3 day 1.
Oh well…I’ll keep the other 4 running and just sell this one.
Yes I forwarded this information to the team and they looking for problems with provider. I hope they would fix an issue.
Yippee Ki Yay, finally today I received an email with “dqed on the satellite” but that was in november 2020
That solves it then…
i have a question. whenever you leave the network with graceful exit, is the node after the exit still able to get new data and is vetted?
If you apply for a new node, yes, but you cannot use the old cert files.
Those are dead.
Yes you can, because if you still running your node, it could register on a new satellite us2.
Since it’s new, your node is not eligible to exit from it. And your node could be vetted fast, because it has not a big amount of data.
But it will not register on the exited satellites.
Thus @naxbc is right too - you need to start clean with a new identity, signed with a new authorization token and empty storage.
Let me revive this topic as it seems the issue is still around:
My nodes got suspended 25 days ago according to their dashboards. Before I noticed anything, they all got disqualified something like 10 days ago I would guess… I’m not sure, but I noticed too late that I had a DNS issue with one sat’ only. Not really complaining about that, the satellite couldn’t reach my nodes… so yeah.
But I did not get any kind of e-mail from StorjLabs, until today, where I received all of a sudden:
- 5 e-mails telling me 5 of my nodes got suspended on that sat’ (and only 5 of my nodes’ dashboard show this notification indeed… ?!)
- 3 e-mails telling me 3 of my nodes got disqualified on that sat’ (whereas all of my 8 nodes show that they are disqualified)
So… what’s going on?!
I’m really not sure what’s up on StorjLabs’ side but e-mail notifications still seem to be pretty broken
I thought they had been simply deactivated, but apparently that’s not the case.
Node emails has been an issue we’ve been working on improving for a while. Before I get to the good news, let me outline how these emails are currently sent (spoiler alert: it’s bad)
- We have a “dataflow” - basically, a query that runs on an interval (I think every hour?) - which gets the “reputation” of every node (in the same query!), where “reputation” is a string such as “offline”, “online”, “suspended”, “disqualified”, etc…
- The results of this query (node email → reputation and other info) are sent to our email service customer.io
- customer.io groups nodes into what it calls “segments”. So there is an “offline” segment, a “disqualified” segment, etc…
- we have workflows set up in customer.io so that when a node transitions from one “segment” to another, it will trigger an email. So for example, if your node moved from “online” to “suspended”, a “suspended” email would be triggered. If your node remains in that segment for enough time, a repeat email may be sent.
Anyways, this is bad for a variety of reasons, but the main things I don’t like about it are
- Updates are dependent on a massive query to get info about all nodes, and it takes a long time to query and send all this information to customer.io
- The query runs on an interval, so node emails are isolated from the actual event occurrences
- It’s overly technically complex and confusing, even to people who work at Storj
For further details, see discussion on this post for the alternate design.
All that is to say we have been working on a new and improved system for triggering and sending node emails that will tie the emails much more tightly to the actual “reputation change” event. So the email would be added to the send queue as soon as a node goes from healthy to suspended, for example.
The work for this is almost done, pending a handful of remaining tasks. If you want to track the milestone on the Storj Product Roadmap, see this issue: Send Node Alerts from Satellite Service for timely email arrivals · Issue #43 · storj/roadmap · GitHub
We are trying to get this out asap, but my current estimation is that it will probably take about a month before this new email system is properly QA’d and ready for production.
All that said, I am very surprised at how delayed your emails were (25 days!)
Off the top of my head, I don’t know what would explain that. Mainly what I wanted to convey in this post is that we are well-aware that storagenode emails are Very Not Good Right Now, and we are at the cusp of being able to switch to a much better system.
So I guess that the state is got updated only on check-in.
Ah yes, that rings a bell @Alexey.
@moby Considering I fixed my DNS issue 4 days ago, my nodes could finally check-in to the satellite that could not be reached (Saltlake). That would mean the e-mails it sent only then got delayed by 3-4 days “only” before getting sent then.
But still, satellites should obviously not rely on Nodes’ “check-ins” for triggering anything as this fails at alerting SNOs when they are offline, which is probably the most common issue.
It feels to me like it should have been prioritized a bit higher as many SNOs probably lost their Nodes (or part of them) in an unfair way in the past 1.5 years or so, in the meantime
But! I’m glad this is being worked on
@moby Thanks for all the details.
And maybe in the future, you also send emails with the storagenode tokens. I don’t know if they worked in the past, but in the last 2 years, I started 9 nodes, and never received the tokens in email.
They worked before, but it was unreliable delivery - most of spam filters do not like tokens
Just wanted to follow up that the new node emails system is enabled in production now
Please reply on the post I linked if you notice any issues. Or create a new post. Whatever works
UpTimeRobot monitoring seems to be doing a good job so far, but email notifs is a plus, no question.
maybe using a 3rd party email delivery service is better?
That’s the plan, see the thread above.
And you may participate in
However, you have several alternatives, for example: