Node suspended - no email?

Guys, this makes no sense, my 2 year old node was suspended and not even an email stating that?

3 Likes

Nothing new, from my 4 nodes I get 1-2 maintenance mails from time to time but none in case of problems, no matter if gmail or outlook :frowning:

Check your logs, what does the next bit of the dashboard show with the Suspension and Online numbers?

That’s not the issue.
The problem is not the why it happened, it’s a mail trigger to the operator warning the suspension!!

1 Like

You’re absolutely right. Sending an email should be the first thing the system does when a node is suspended.

For some reason Storj seems to dislike email communication with SNO’s :confused:

Maybe add a suggestion? Ideas & Suggestions - Storj Community Forum (official)

2 Likes

They do send emails in that case but maybe something is broken (at least for some nodes)?

1 Like

I’ve seen emails about disqualification. Are you sure there’s a notification about suspension as well?

Hmm maybe :smiley: Now I’m not sure

1 Like

Yeah, but disqualification email is kinda late, no? :wink:

3 Likes

I got warned by email many months back when one of my nodes got suspended.

@naxbc I guess you already double checked the spelling of your email adress configured for your node.
If it’s valid, it feels like it’d be worth reaching out to the support team so they can investiagte and fix that.

Or ping Jocelyn about that on the topic @kevink linked above.

StorjLabs seem to go easy on emails and that’s fine, but urgent alerts like a suspension should definitely trigger an email to the SNO, in my opinion.

1 Like

Thanks man.
Yes I’m pretty sure regarding my email spelling.
Dispite it was a failure on my volume, it should trigger an email regarding the suspension in any case.
I’ve already sent @Alexey my NodeID is and deleted my node. It’s lost, on my fault, or better HDD fault.
Lost 300$ in Held Amount, but it paid for itself as it was my oldest node ever since StorjV3 day 1.
Oh well…I’ll keep the other 4 running and just sell this one.

Yes I forwarded this information to the team and they looking for problems with provider. I hope they would fix an issue.

6 Likes

Yippee Ki Yay, finally today I received an email with “dqed on the satellite” but that was in november 2020 :+1: :sweat_smile:

2 Likes

:joy:
That solves it then…

1 Like

i have a question. whenever you leave the network with graceful exit, is the node after the exit still able to get new data and is vetted?

If you apply for a new node, yes, but you cannot use the old cert files.
Those are dead.

Yes you can, because if you still running your node, it could register on a new satellite us2.

Since it’s new, your node is not eligible to exit from it. And your node could be vetted fast, because it has not a big amount of data.
But it will not register on the exited satellites.

Thus @naxbc is right too - you need to start clean with a new identity, signed with a new authorization token and empty storage.

Let me revive this topic as it seems the issue is still around:

My nodes got suspended 25 days ago according to their dashboards. Before I noticed anything, they all got disqualified something like 10 days ago I would guess… I’m not sure, but I noticed too late that I had a DNS issue with one sat’ only. Not really complaining about that, the satellite couldn’t reach my nodes… so yeah.

But I did not get any kind of e-mail from StorjLabs, until today, where I received all of a sudden:

  • 5 e-mails telling me 5 of my nodes got suspended on that sat’ (and only 5 of my nodes’ dashboard show this notification indeed… ?!)
  • 3 e-mails telling me 3 of my nodes got disqualified on that sat’ (whereas all of my 8 nodes show that they are disqualified)

So… what’s going on?!
I’m really not sure what’s up on StorjLabs’ side but e-mail notifications still seem to be pretty broken :confused:
I thought they had been simply deactivated, but apparently that’s not the case.

Hey Pac,

Node emails has been an issue we’ve been working on improving for a while. Before I get to the good news, let me outline how these emails are currently sent (spoiler alert: it’s bad)

  • We have a “dataflow” - basically, a query that runs on an interval (I think every hour?) - which gets the “reputation” of every node (in the same query!), where “reputation” is a string such as “offline”, “online”, “suspended”, “disqualified”, etc…
  • The results of this query (node email → reputation and other info) are sent to our email service customer.io
  • customer.io groups nodes into what it calls “segments”. So there is an “offline” segment, a “disqualified” segment, etc…
  • we have workflows set up in customer.io so that when a node transitions from one “segment” to another, it will trigger an email. So for example, if your node moved from “online” to “suspended”, a “suspended” email would be triggered. If your node remains in that segment for enough time, a repeat email may be sent.

Anyways, this is bad for a variety of reasons, but the main things I don’t like about it are

  • Updates are dependent on a massive query to get info about all nodes, and it takes a long time to query and send all this information to customer.io
  • The query runs on an interval, so node emails are isolated from the actual event occurrences
  • It’s overly technically complex and confusing, even to people who work at Storj :sweat_smile:

For further details, see discussion on this post for the alternate design.

All that is to say we have been working on a new and improved system for triggering and sending node emails that will tie the emails much more tightly to the actual “reputation change” event. So the email would be added to the send queue as soon as a node goes from healthy to suspended, for example.

The work for this is almost done, pending a handful of remaining tasks. If you want to track the milestone on the Storj Product Roadmap, see this issue: Send Node Alerts from Satellite Service for timely email arrivals · Issue #43 · storj/roadmap · GitHub

We are trying to get this out asap, but my current estimation is that it will probably take about a month before this new email system is properly QA’d and ready for production.

All that said, I am very surprised at how delayed your emails were (25 days!)
Off the top of my head, I don’t know what would explain that. Mainly what I wanted to convey in this post is that we are well-aware that storagenode emails are Very Not Good Right Now, and we are at the cusp of being able to switch to a much better system.

6 Likes

So I guess that the state is got updated only on check-in.

1 Like