Hey Pac,
Node emails has been an issue we’ve been working on improving for a while. Before I get to the good news, let me outline how these emails are currently sent (spoiler alert: it’s bad)
- We have a “dataflow” - basically, a query that runs on an interval (I think every hour?) - which gets the “reputation” of every node (in the same query!), where “reputation” is a string such as “offline”, “online”, “suspended”, “disqualified”, etc…
- The results of this query (node email → reputation and other info) are sent to our email service customer.io
- customer.io groups nodes into what it calls “segments”. So there is an “offline” segment, a “disqualified” segment, etc…
- we have workflows set up in customer.io so that when a node transitions from one “segment” to another, it will trigger an email. So for example, if your node moved from “online” to “suspended”, a “suspended” email would be triggered. If your node remains in that segment for enough time, a repeat email may be sent.
Anyways, this is bad for a variety of reasons, but the main things I don’t like about it are
- Updates are dependent on a massive query to get info about all nodes, and it takes a long time to query and send all this information to customer.io
- The query runs on an interval, so node emails are isolated from the actual event occurrences
- It’s overly technically complex and confusing, even to people who work at Storj
For further details, see discussion on this post for the alternate design.
All that is to say we have been working on a new and improved system for triggering and sending node emails that will tie the emails much more tightly to the actual “reputation change” event. So the email would be added to the send queue as soon as a node goes from healthy to suspended, for example.
The work for this is almost done, pending a handful of remaining tasks. If you want to track the milestone on the Storj Product Roadmap, see this issue: Send Node Alerts from Satellite Service for timely email arrivals · Issue #43 · storj/roadmap · GitHub
We are trying to get this out asap, but my current estimation is that it will probably take about a month before this new email system is properly QA’d and ready for production.
All that said, I am very surprised at how delayed your emails were (25 days!)
Off the top of my head, I don’t know what would explain that. Mainly what I wanted to convey in this post is that we are well-aware that storagenode emails are Very Not Good Right Now, and we are at the cusp of being able to switch to a much better system.