Just a word
I’ve been using Storj as a Storage Node Operator since about 2 years now.
Few months ago, I had this project of moving to another country. Because of a lot of constraints (including Covid-19 restrictions), this move would have impact the availability of my node and would have probably lead to disqualification of all my nodes. Indeed, for a such move, you have to think about how you will do port forwarding, how you will bring your physical storage, etc.
After all time invested in those nodes and since they are all running well for a quite some time, I didn’t want to gracefully exit, neither leaving my held amount.
That’s why I started to think about how to keep my nodes during this move.
I’d like to share this experience with all Storj community.
I hope some of you will find some good advice. If others think this was not the best approach, at least it may lead to interesting discussions.
Do not consider it as a handbook: some keypoints may be missing, some risks may be underestimated, some tools or operations may not work in your situation and since we are talking about a quite new service, some advice/information may be outdated.
This guide is just a detailed process of what worked for me. I hope it will do it for your as well.
So I hope by sharing my own experience, it will help you to decide or to accomplish a migration in a smooth way.
Have fun!
Being a SNO requires you to invest some time…
Since I heard about Storj, I spent non negligible effort to understand, setup and optimize my node architecture. But most importantly: I learned a lot.
Of course, I also earned a significant amount of money. I mean that, considering the capital I put in it (essentially in buying HDDs) and the operational costs (mostly electricity), I earned some money even if this didn’t make me rich. It is worth it.
Of course, being a SNO is not always a “setup and forget” process.
First you have to make sure you choose the right hardware for a good Return On Investment (ROI). Something reliable and durable, not too expensive though.
But then, you have to monitor, operate, automate, upgrade, replace (hardware), optimize, etc. In other words, among the most important questions, here are some I asked myself: How can I make sure everything is fine with my nodes? How should I know about it? When should I know about a failure (little hint: “as soon as the failure occurs” is generally too late)? What should I do in case of a failure? Do I have a “backup plan”? How can I automate some basic operations such as storing/rolling logs? How can I automate creation of new nodes? How can I secure my hardware and software upgrades (OS upgrades, software upgrades, HDD replacements, etc.)? How can I make my architecture more reliable? more cost-efficient? etc.
And it seems fair because, after all, real customers are paying for the service we are offering as SNO. Not considering some basic questions may lead to Disqualification (DQ).
Among indicators to follow, one the most important is the node availability.
… That is why I didn’t want to throw it all because of moving to another country
That is why, when I had to move from Europe to Canada (a life project we had for a long time), I was a little reluctant about stopping my 5 nodes. At that time of moving, a node could be disqualified if it was offline too long (more than 30 days).
In my case, I couldn’t travel with my server (hosting my nodes), had to send my big stuffs in a container (and expect to get them back in 1 month or 2) and to respect Covid-19 safety measures (hotel + quarantine).
So I had 2 options:
- Perform a graceful exit, i.e. saving the money earned and start again from 0 when arrived in Canada.
- Find a way to keep them
I finally chose the option 2. It was risky but definitely worth it. Though it is not true in every case.
Indeed, in order to choose wiesely before migrating a node to a new location far away, there are some questions I had to consider and I really recommend you to do so.
- What is the held amount of my node?
- What is the age of my node?
- What are the average earnings (on the current location)?
- What is the expected average earnings (on the new location)? This one may be difficult to estimate but you have several ways to do it:
- consider it will be the same
- extrapolate the current earnings, based on the current and expected bandwidths
- ask to other SNOs
No option is perfect but it may help you decide anyway.
These figures are essential because migrating a node is not a risk-free operation. If your node is offline for too long or if data are corrupted during the process, you may be disqualified and lose your held amount. That is why you have to evaluate whether the migration is worth it.
General approach I followed
So let me give you some figures about the context:
- 30 days of unavailability: without doing anything, it is fair to estimate that my nodes would be offline for 30 days at least
- About $30 / month for all my nodes
- Quite “old” nodes at that time: All of them are all over 10 months old (so no more amount is held on monthly earnings). The oldest one was 20 months.
- About 80$ in total held amont
- Expected earnings: I expected to earn at least the same amount each month.
And now let’s recap the constraints:
- I can’t take the whole server with me (a HP microserver Gen8): it is too heavy and too bulky to put it in my luggage
- During about 3 weeks, I will have to rely on untrusty Internet connections or at least connections that I may not able to manage myself (airbnb internet connection, Hotel Wifi). It means 2 things:
-
- I will not be able to forward the required ports
-
- I may be disconnected without knowing why and without being able to do anything to restore the connection
-
I ruled out several options:
- Migrate nodes on a Public Cloud infrastructure (Google, AWS, Azure or anything else): even with the “free tier”, it was way too expensive. Even for 1 month, mostly because of the “retrieval fees” (network fees you pay when you transfer data from the Cloud to the Internet). Because even if you try to “cap” the bandwidth of your nodes (plus, this may disqualify your node), keep in mind that will have to send to the Internet AT LEAST the same amount of data as your node is storing.
- Ask a friend: because I don’t want to bother them with it.
Considering those inputs, I decided to keep the nodes with me. In any time.
So the general approach was to:
- Copy all the data of my nodes to external HDDs
- I couldn’t just “take” the internal HDDs of my server because they are formated in a way that my laptop is not able to read it.
- Setup nodes on my Linux laptop, that I will have with me all the time (in hotels and airbnbs)
- Setup a specific network configuration in order to make sure my nodes will be reachable, even when connected on a Public network
- Connect my nodes as much as possible
- Migrate the nodes back to the server when I’ll receive it
This may seem quite simple.
But the difficulty was the preparation. I needed a “plug and play” solution. I wouldn’t have time to troubleshoot if there was an issue. That is why I needed to prepare all what could be prepared before and automate all what could be automated.
Step by step migration
Here is the Step-by-step process I followed.
1 week before
- OS/HW Requirements
To prepare my “temporary” storage (where nodes will store their data during the move), I used external USB HDDs.
Of course, you can use internal storage of your laptop. Interesting thing is that you don’t have to use the “full target” capacity of your node (specified with -e STORAGE ="" in the docker run command, and displayed in the node dashboard). You can just consider the amount of current space used by the node. When starting this node on the laptop (docker run command), you will just have to change the storage environment variable to cap the storage capacity to the current usage (or a little bit more, it’s safer) and your node will be considered as full and will not use any more space. Plus, this has the advantage of reducing I/O and then reducing risk of disk failure.
In my case, I just had 3 external USB HDDs for my 4 nodes. So I decided that it could be good enough to store 2 nodes data on a same disk.
Of course, it is risky since 1 hardware failure would result in losing 2 nodes. Plus, I/O would be more intensive on the same disk since it is used by 2 nodes. But I accepted the risk.
Regarding the laptop…
My nodes were initially hosted on a Debian VM. Since my laptop is a Linux Ubuntu, it should be easy to run them on it. I just had to install Docker on it, following the standard Storj installation documentation.
And you’re done for OS/HW prerequisites!
- Prepare Network configuration
The biggest challenge when moving your nodes and relying on uncontrolled internet connections, is that you have to find a way to forward ports.
Official documentation gives some insight on how to do this even if you can’t manage the router.
I decided to use portmap.io with OpenVPN software for Linux. But it should work also on Windows.
With this, you can create a VPN tunnel from your computer (connected to a hotel Wifi, for example) to a remote server. Then, portmap.io will forward requests sent to a predefined external port (given and decided by portmap.io) to the port of your local computer (that you define).
So basically, all you have to do for 1 given node is to:
1. Create an account on portmap.io
2. Create a configuration with “tcp” as protocol. Do not forget to click on “Generate” and download the configuration file (ovpn file).
3. Create a mapping rule tied to your configuration and type your local storage node port (it should be 28967 by default).
Then, note the remote port (you can find it on the Mapping Rules tab on portmap.io) and the remote IP address (in the configuration file, on the line beginning by “remote”).
In theory, at this step you are done with your network configuration. It is not 100% sure it will work on every network but it did for me (Hotel Wifi).
Once again, you must test it before leaving (by connecting to your smartphone for example in Tethering mode). All you have to do is to run openvpn with the configuration file you downloaded before (.ovpn file) with the following command: openvpn --config <your_configuration>.ovpn on Linux.
To make things a little bit easier each time I will start my nodes, I created a very simple bash script:
#!/bin/bash
sudo killall openvpn ; sudo nohup openvpn --config <your_configuration>.ovpn & sudo docker start <your_node>
This way, the openvpn will run in background and I don’t have to type the whole command.
The first part with killall will kill any old instance of openvpn (I’ll explain later why I used it).
- Migrate all your data to USB disks (USB migration)
Your are just few days ahead before moving. Based on the amount of data, this is probably the good time to start your migration. A good recommendation is to store also the identity folder on the same HDD.
In my case, I just followed the official recommendations (explained somewhere in the official docs or on the forum):
1. Copy files to the new location (time-consuming, don’t stop the node just yet)
rsync --ignore-existing -ravz /media/storj_source/ /media/storj_target/
2. Run it again to copy differences
rsync --ignore-existing -ravz /media/storj_source/ /media/storj_target/
3. Stop the old node and remove its container
docker stop -t 300 <your_node>
docker rm storagenode
4. Copy files again but with the delete command this time. This deletes files in the destination directory if they don’t exist in the source directory.
rsync --delete -ravz /media/storj_source/ /media/storj_target/
Do not run the new nodes at this time. We need to make some adjustments.
- Cap storage capacity for each node
Before starting your new node with existing data, you may need to change those 2 environment variables in thedocker runcommand:
-e ADDRESS=<remote_server_IP> \
-e STORAGE=<capped_storage> \
The address is the one you noted on step 2. The storage capacity depends on what you decided on step 1.
Then, you can run the node and start it.
Do not hesitate to test it in real conditions by connecting to your smartphone and using openvpn. This way you will be able to see if everything is working as expected.
At this time, I just realized that I actually couldn’t connect more than 1 node at a time…
The openvpn configuration allows me to forward only 1 port. At least, I didn’t find how to do it better.
Anyway, I just had to switch regularly between all my nodes to keep them with good enough availability ratio.
D-day!
- Connect your nodes as much as possible!
Now, you just have to keep your node(s) alive as much as possible.
You may have issues with some networks. I had some issues with the Hotel Wifi I still can’t explain since it worked someday and not another day
Anyway.
Hopefully, in my case, when I reached my airbnb, I saw that it was a quite good connection. And I was able to login on the router and to forward ports easily! From there, it was a piece of cake: I just had to leave my laptop always on with the 3 HDDs plugged-in and that’s it!
Now, I reached the final destination, with my own Internet contract and my original hardware.
Just had to migrate the nodes back to the server and that’s all!
Everything works like a charm, even better than before I move :).
It’s definitely worth it!
Learnings & Conclusion
So what did I learn from this experience?
First of all, always try to keep your nodes! This experience may seem very easy and obvious for some, or overkill/overly complicated for others. But it took me several weeks to setup this “moving strategy” and a lot of questions/readings on the forum. So do not give up, your data is definitely valuable!
Always take some time to prepare and test before moving. Even if your plan may seem perfect, try to think about all what could go wrong. I recommend to start preparing all stuffs (from the very step 1) at least 1 week before moving.
Focus your efforts on what really matters: all your nodes may not be at the same age or may not have the same held amount. Even if it’s important to “rotate” your nodes if you can’t connect all of them at the same time, try also to prioritize the most valuable ones.
As much as possible, try to identify network contraints before you begin your travel. You may use public networks (airport, hotel, etc) with different levels of limitations (especially regarding ports opening) or private network, on which you may have more controls on what you want to do (e.g. open and redirect a specific port to your machine).
You really should master the whole Disqualification process (DQ process). Indeed, you may encounter several issues you didn’t expect. By knowing well the DQ process, you will be able to better assess the risk and take better decisions in real time during your travel. For example, If you can connect only 1 of your nodes, which one should you choose because it is most likely to be disqualified?
You will probably use a laptop: do not forget to plug the power! If your laptop is out of power, your nodes will stop abruptly, which is not good and you may be disqualified if data are corrupted. I forgot to do it maybe 3 times during the move…
Your laptop is not a NAS/Server. Because of a bad quality drive (SMR disk) and a very fast Internet connection, my laptop was about to freeze sometimes. This is probably due to I/O issues (a recurring issue that have SMR disks). In order to avoid that, you can reduce the internet connection of your docker node (check wondershaper ; not really recommended in a stable Storj setup but it can help sometimes) or run several nodes spread across multiple HDDs.
Would never really have thought of such a scenario. It’s not exactly the natural environment for a storagenode. But you got through it like a champ! Well done and congratz on the move!