I am hosting a total of 8 nodes atm and my only problem is the monitoring of the health and performance of the nodes.
As I guess anyone, I worry if the node is online and if there are any errors, also I want to be able to check all my nodes with one link instead of going through 8, and accessibility from the web, as the dashboard as to what I understand is not OK to be public.
I am considering building a storage node monitoring service where the node data is monitored and alarms are sent to an Android app. With some aggregated statistics for all the nodes.
My question is if anyone knows for such a service or something that can help in general?
Also I will be very happy to hear more ideas that I can integrate into the project, for now, I am also considering integrating the more precise earnings calculator Google sheet logic as I would love to see a prediction for multiple nodes, along with the statistics and alarms.
Regarding fast about SN going offline you can take a look at https://uptimerobot.com/. It can check your SN port and send you a notification via multiple channels like android or SMS.
Currently, one of Storj teams is working to provide a dashboard for multiple nodes. Work is not very advanced yet so there is still time to provide ideas, comments, or concerns. You can also follow design document for this purpose which is under construction right now https://review.dev.storj.io/c/storj/storj/+/2289.
This method requires several open ports and has many security implications. It’s not recommended for home networks. However, if you have a domain, you can rent a very small IaaS server to front the traffic and provide some isolation.
I haven’t used this process in a long time, but I can see if I can find an example in my pile of old files.
EDIT:
I should also point out that it’s possible to use IPFS pubsub for a server-less notification system:
I’ve recently been experimenting with pubsub. However, it’s extremely slow unless one is running a node that has direct internet connection.
Email messaging systems are quite easy to configure if one is willing to run a small mail server… which isn’t as complicated as it sounds. I’ve been running several email servers for more than a decade with very few problems.
It used to be easy… just create a “free” gmail account and have your notification script dial in via IMAP. However, I think gmail shut down the “less-secure” authentication methods.
@michaln Thank you this will solve the uptime monitoring problem. And thank you for pointing out the StorJ official multi-node dashboard.
@anon27637763 Thank you this was very interesting reads, to date I was just looking at Kafka and RabbitMQ for such purposes (of course with quite some coding on top). And it looks like a variable option for the development of a large scale notification service.
From my point of view, the StorJ should not be prioritizing very high the multi-node dashboard as the core is more of a priority and data on the network. And I see the document does cover some of the needs however some stay uncovered (like predictions and logs reading, such I do not think should be prioritized at this point from the StorJ team). The priority is why I see it as an opportunity to make such software as a third party, and essentially help the framework grow without loading the core team.
In the end, I am hoping to create a service that will work with the SNs and provide all the knowledge that is now available in this forum and from various scripts and google sheets.
At current my idea for design is to run a container for every node which goal will be to read the logs from the node, and the current UI API (as there is no other at the moment), and send the data to a web service over HTTPS. For UI, for now, I am considering an Android app as the goal of the service is data when you are not at home, so it seems most appropriate, and it opens the door for the notifications. For the local monitor I think GO will be the best fit as the container will become very small, for the web service, for now, I am looking at Java/Kotlin as the BE code and I am still looking for a good DB solution that can store the statistics data in a good and queriable format, there I prefer if I can store the data on the StorJ network and essentially load the network a bit more. I think this should be possible with Hadoop, but I have more to read on the topic.
With the above, I think the features of success rate detection, usage monitoring, offline detection, income prognoses, and failure detection (from the logs) can be made in a secure manner, without exposing ports of a home network. I am also thinking of a set of hints, for common scenarios like 2 nodes on the same network that are both not full for example, but that will be in the future.
Sorry for the long post, and thank you for your responses.