SNO move Return of Experience: A comprehensive story of moving a SNO from Europe to Canada

Just a word

I’ve been using Storj as a Storage Node Operator since about 2 years now.
Few months ago, I had this project of moving to another country. Because of a lot of constraints (including Covid-19 restrictions), this move would have impact the availability of my node and would have probably lead to disqualification of all my nodes. Indeed, for a such move, you have to think about how you will do port forwarding, how you will bring your physical storage, etc.
After all time invested in those nodes and since they are all running well for a quite some time, I didn’t want to gracefully exit, neither leaving my held amount.
That’s why I started to think about how to keep my nodes during this move.

I’d like to share this experience with all Storj community.
I hope some of you will find some good advice. If others think this was not the best approach, at least it may lead to interesting discussions.

Do not consider it as a handbook: some keypoints may be missing, some risks may be underestimated, some tools or operations may not work in your situation and since we are talking about a quite new service, some advice/information may be outdated.
This guide is just a detailed process of what worked for me. I hope it will do it for your as well.

So I hope by sharing my own experience, it will help you to decide or to accomplish a migration in a smooth way.
Have fun!

Being a SNO requires you to invest some time…

Since I heard about Storj, I spent non negligible effort to understand, setup and optimize my node architecture. But most importantly: I learned a lot.
Of course, I also earned a significant amount of money. I mean that, considering the capital I put in it (essentially in buying HDDs) and the operational costs (mostly electricity), I earned some money even if this didn’t make me rich. It is worth it.

Of course, being a SNO is not always a “setup and forget” process.
First you have to make sure you choose the right hardware for a good Return On Investment (ROI). Something reliable and durable, not too expensive though.
But then, you have to monitor, operate, automate, upgrade, replace (hardware), optimize, etc. In other words, among the most important questions, here are some I asked myself: How can I make sure everything is fine with my nodes? How should I know about it? When should I know about a failure (little hint: “as soon as the failure occurs” is generally too late)? What should I do in case of a failure? Do I have a “backup plan”? How can I automate some basic operations such as storing/rolling logs? How can I automate creation of new nodes? How can I secure my hardware and software upgrades (OS upgrades, software upgrades, HDD replacements, etc.)? How can I make my architecture more reliable? more cost-efficient? etc.
And it seems fair because, after all, real customers are paying for the service we are offering as SNO. Not considering some basic questions may lead to Disqualification (DQ).
Among indicators to follow, one the most important is the node availability.

… That is why I didn’t want to throw it all because of moving to another country

That is why, when I had to move from Europe to Canada (a life project we had for a long time), I was a little reluctant about stopping my 5 nodes. At that time of moving, a node could be disqualified if it was offline too long (more than 30 days).
In my case, I couldn’t travel with my server (hosting my nodes), had to send my big stuffs in a container (and expect to get them back in 1 month or 2) and to respect Covid-19 safety measures (hotel + quarantine).

So I had 2 options:

  1. Perform a graceful exit, i.e. saving the money earned and start again from 0 when arrived in Canada.
  2. Find a way to keep them

I finally chose the option 2. It was risky but definitely worth it. Though it is not true in every case.
Indeed, in order to choose wiesely before migrating a node to a new location far away, there are some questions I had to consider and I really recommend you to do so.

  • What is the held amount of my node?
  • What is the age of my node?
  • What are the average earnings (on the current location)?
  • What is the expected average earnings (on the new location)? This one may be difficult to estimate but you have several ways to do it:
    • consider it will be the same
    • extrapolate the current earnings, based on the current and expected bandwidths
    • ask to other SNOs
      No option is perfect but it may help you decide anyway.
      These figures are essential because migrating a node is not a risk-free operation. If your node is offline for too long or if data are corrupted during the process, you may be disqualified and lose your held amount. That is why you have to evaluate whether the migration is worth it.

General approach I followed

So let me give you some figures about the context:

  • 30 days of unavailability: without doing anything, it is fair to estimate that my nodes would be offline for 30 days at least
  • About $30 / month for all my nodes
  • Quite “old” nodes at that time: All of them are all over 10 months old (so no more amount is held on monthly earnings). The oldest one was 20 months.
  • About 80$ in total held amont
  • Expected earnings: I expected to earn at least the same amount each month.

And now let’s recap the constraints:

  • I can’t take the whole server with me (a HP microserver Gen8): it is too heavy and too bulky to put it in my luggage
  • During about 3 weeks, I will have to rely on untrusty Internet connections or at least connections that I may not able to manage myself (airbnb internet connection, Hotel Wifi). It means 2 things:
      1. I will not be able to forward the required ports
      1. I may be disconnected without knowing why and without being able to do anything to restore the connection

I ruled out several options:

  • Migrate nodes on a Public Cloud infrastructure (Google, AWS, Azure or anything else): even with the “free tier”, it was way too expensive. Even for 1 month, mostly because of the “retrieval fees” (network fees you pay when you transfer data from the Cloud to the Internet). Because even if you try to “cap” the bandwidth of your nodes (plus, this may disqualify your node), keep in mind that will have to send to the Internet AT LEAST the same amount of data as your node is storing.
  • Ask a friend: because I don’t want to bother them with it.

Considering those inputs, I decided to keep the nodes with me. In any time.
So the general approach was to:

  1. Copy all the data of my nodes to external HDDs
    • I couldn’t just “take” the internal HDDs of my server because they are formated in a way that my laptop is not able to read it.
  2. Setup nodes on my Linux laptop, that I will have with me all the time (in hotels and airbnbs)
  3. Setup a specific network configuration in order to make sure my nodes will be reachable, even when connected on a Public network
  4. Connect my nodes as much as possible
  5. Migrate the nodes back to the server when I’ll receive it

This may seem quite simple.
But the difficulty was the preparation. I needed a “plug and play” solution. I wouldn’t have time to troubleshoot if there was an issue. That is why I needed to prepare all what could be prepared before and automate all what could be automated.

Step by step migration

Here is the Step-by-step process I followed.

1 week before

  1. OS/HW Requirements

To prepare my “temporary” storage (where nodes will store their data during the move), I used external USB HDDs.
Of course, you can use internal storage of your laptop. Interesting thing is that you don’t have to use the “full target” capacity of your node (specified with -e STORAGE ="" in the docker run command, and displayed in the node dashboard). You can just consider the amount of current space used by the node. When starting this node on the laptop (docker run command), you will just have to change the storage environment variable to cap the storage capacity to the current usage (or a little bit more, it’s safer) and your node will be considered as full and will not use any more space. Plus, this has the advantage of reducing I/O and then reducing risk of disk failure.

In my case, I just had 3 external USB HDDs for my 4 nodes. So I decided that it could be good enough to store 2 nodes data on a same disk.
Of course, it is risky since 1 hardware failure would result in losing 2 nodes. Plus, I/O would be more intensive on the same disk since it is used by 2 nodes. But I accepted the risk.

Regarding the laptop…
My nodes were initially hosted on a Debian VM. Since my laptop is a Linux Ubuntu, it should be easy to run them on it. I just had to install Docker on it, following the standard Storj installation documentation.

And you’re done for OS/HW prerequisites!

  1. Prepare Network configuration
    The biggest challenge when moving your nodes and relying on uncontrolled internet connections, is that you have to find a way to forward ports.

Official documentation gives some insight on how to do this even if you can’t manage the router.
I decided to use portmap.io with OpenVPN software for Linux. But it should work also on Windows.

With this, you can create a VPN tunnel from your computer (connected to a hotel Wifi, for example) to a remote server. Then, portmap.io will forward requests sent to a predefined external port (given and decided by portmap.io) to the port of your local computer (that you define).
So basically, all you have to do for 1 given node is to:
1. Create an account on portmap.io
2. Create a configuration with “tcp” as protocol. Do not forget to click on “Generate” and download the configuration file (ovpn file).
3. Create a mapping rule tied to your configuration and type your local storage node port (it should be 28967 by default).

Then, note the remote port (you can find it on the Mapping Rules tab on portmap.io) and the remote IP address (in the configuration file, on the line beginning by “remote”).

In theory, at this step you are done with your network configuration. It is not 100% sure it will work on every network but it did for me (Hotel Wifi).
Once again, you must test it before leaving (by connecting to your smartphone for example in Tethering mode). All you have to do is to run openvpn with the configuration file you downloaded before (.ovpn file) with the following command: openvpn --config <your_configuration>.ovpn on Linux.
To make things a little bit easier each time I will start my nodes, I created a very simple bash script:

#!/bin/bash
sudo killall openvpn ; sudo nohup openvpn --config <your_configuration>.ovpn & sudo docker start <your_node>

This way, the openvpn will run in background and I don’t have to type the whole command.
The first part with killall will kill any old instance of openvpn (I’ll explain later why I used it).

  1. Migrate all your data to USB disks (USB migration)
    Your are just few days ahead before moving. Based on the amount of data, this is probably the good time to start your migration. A good recommendation is to store also the identity folder on the same HDD.

In my case, I just followed the official recommendations (explained somewhere in the official docs or on the forum):
1. Copy files to the new location (time-consuming, don’t stop the node just yet)
rsync --ignore-existing -ravz /media/storj_source/ /media/storj_target/
2. Run it again to copy differences
rsync --ignore-existing -ravz /media/storj_source/ /media/storj_target/
3. Stop the old node and remove its container
docker stop -t 300 <your_node>
docker rm storagenode
4. Copy files again but with the delete command this time. This deletes files in the destination directory if they don’t exist in the source directory.
rsync --delete -ravz /media/storj_source/ /media/storj_target/

Do not run the new nodes at this time. We need to make some adjustments.

  1. Cap storage capacity for each node
    Before starting your new node with existing data, you may need to change those 2 environment variables in the docker run command:
-e ADDRESS=<remote_server_IP> \
-e STORAGE=<capped_storage> \

The address is the one you noted on step 2. The storage capacity depends on what you decided on step 1.

Then, you can run the node and start it.
Do not hesitate to test it in real conditions by connecting to your smartphone and using openvpn. This way you will be able to see if everything is working as expected.

At this time, I just realized that I actually couldn’t connect more than 1 node at a time…
The openvpn configuration allows me to forward only 1 port. At least, I didn’t find how to do it better.
Anyway, I just had to switch regularly between all my nodes to keep them with good enough availability ratio.

D-day!

  1. Connect your nodes as much as possible!
    Now, you just have to keep your node(s) alive as much as possible.
    You may have issues with some networks. I had some issues with the Hotel Wifi I still can’t explain since it worked someday and not another day :confused:
    Anyway.
    Hopefully, in my case, when I reached my airbnb, I saw that it was a quite good connection. And I was able to login on the router and to forward ports easily! From there, it was a piece of cake: I just had to leave my laptop always on with the 3 HDDs plugged-in and that’s it!

Now, I reached the final destination, with my own Internet contract and my original hardware.
Just had to migrate the nodes back to the server and that’s all!

Everything works like a charm, even better than before I move :).
It’s definitely worth it!

Learnings & Conclusion

So what did I learn from this experience?

First of all, always try to keep your nodes! This experience may seem very easy and obvious for some, or overkill/overly complicated for others. But it took me several weeks to setup this “moving strategy” and a lot of questions/readings on the forum. So do not give up, your data is definitely valuable!

Always take some time to prepare and test before moving. Even if your plan may seem perfect, try to think about all what could go wrong. I recommend to start preparing all stuffs (from the very step 1) at least 1 week before moving.

Focus your efforts on what really matters: all your nodes may not be at the same age or may not have the same held amount. Even if it’s important to “rotate” your nodes if you can’t connect all of them at the same time, try also to prioritize the most valuable ones.

As much as possible, try to identify network contraints before you begin your travel. You may use public networks (airport, hotel, etc) with different levels of limitations (especially regarding ports opening) or private network, on which you may have more controls on what you want to do (e.g. open and redirect a specific port to your machine).

You really should master the whole Disqualification process (DQ process). Indeed, you may encounter several issues you didn’t expect. By knowing well the DQ process, you will be able to better assess the risk and take better decisions in real time during your travel. For example, If you can connect only 1 of your nodes, which one should you choose because it is most likely to be disqualified?

You will probably use a laptop: do not forget to plug the power! If your laptop is out of power, your nodes will stop abruptly, which is not good and you may be disqualified if data are corrupted. I forgot to do it maybe 3 times during the move…

Your laptop is not a NAS/Server. Because of a bad quality drive (SMR disk) and a very fast Internet connection, my laptop was about to freeze sometimes. This is probably due to I/O issues (a recurring issue that have SMR disks). In order to avoid that, you can reduce the internet connection of your docker node (check wondershaper ; not really recommended in a stable Storj setup but it can help sometimes) or run several nodes spread across multiple HDDs.

7 Likes

Bravo! That’s quite some dedication, and sounds like a fun adventure.

1 Like

International moves are never fun. If it had been me I probably would have brought the Microserver with me in one suitcase. Good job to keep your nodes alive.
When I did my move from Canada to Russia I had two suitcases, a rucksack and two cats and carried the lot myself. Inside the suitcases and rucksack were at least 4 computers.
I’ve done trips with bigger machines too… I flew back from NZ to Australia with a Dell Poweredge and an Alphastation in the suitcase. Bringing back some hardware for my then employer that was surplus at one site but still needed at another.

1 Like

This is part of the reason I have used Dell Precision laptops for quite a number of years now.
They are basically mobile workstations so tend to have a lot more power than standard laptops from the same era. My wife uses a Dell M6800 as her main work machine and loves it.

1 Like

I’m glad that you finally successfully finished your move and seems completed the Family Project and didn’t break your own SNO project!
Congratulations!

Some notes though. There is a typo:

It should be:

#!/bin/bash
sudo killall openvpn; sudo nohup openvpn --config <your_configuration>.ovpn && sudo docker start <your_node>

But I think, a much simpler way is just enable OpenVPN service:

sudo systemctl enable openvpn

Put your config to /etc/openvpn/client/ and run the service

sudo systemctl start openvpn

Regarding external disks - you could also use an external cases instead of external disks and put your internal disks to the external case(s), this is usually costs less than a whole external disk with the comparable capacity. However, your laptop may not supply enough power to feed such a cases.

1 Like

I love that you kind of took your nodes on the road with you. :slight_smile: Would never really have thought of such a scenario. It’s not exactly the natural environment for a storagenode. But you got through it like a champ! Well done and congratz on the move!

4 Likes

One note from me here. These services are expensive because they provide way more features than a storage node requires, for example redundancy. It is possible to find providers of services without these features that are much cheaper, with costs comparable to what Storj pays.

You are right.
But my point was to say that it is not cost efficient to store a Storj node on these Cloud platform, especially if you plan to get your node back on-premises.
If you know a cheap cloud platform, do not hesitate to share it.

Well, I know Hetzner’s storage boxes can be rented at 40 EUR for 10 TB, and if you add their cheapest VPS at 2.5 EUR to act as a Docker node, it’s still close to the ~4 EUR/TB that I recently earn from Storj. For a brief time I hosted a node there simply because I rented a box a bit too large for my purposes, and it worked well after making sure the databases are actually kept on the VPS. Hetzner actually has even better deals if you want long-term and >40TB, but I assume your nodes are around 8-10 TB given your stated payouts.

Going more extreme, it’s possible to host a node on shared hosting of seedhost.eu, getting e.g. 15.8 TB for 34 EUR, or 7.9 TB for 19 EUR. This is non-redundant though—should be fine for Storj. Also you can’t use Docker there, but it’s still possible to run a bare node there either by compiling code manually or extracting binaries from Docker images. Again, tested, would be good enough for parking a node for few months. They do have dedicated boxes which would be Docker-capable, but they’re more expensive.

In both cases the monthly traffic allocated to these accounts is way above what Storj requires. Probably not as stable as more expensive cloud services, but they’d still get a decent online score.

There’s likely more providers like that, I just know of these two. For smaller nodes lowendbox.com sometimes has nice deals.

2 Likes

Actually I knew this provider (Hetzner).
I forgot why I couldn’t go with this option but I remember now!

I think you are right, it may be a very good option for some.
But in my case, I didn’t have time to send data to Hetzner before leaving. Before moving, I was in an apartment with a quite old ADSL line. If I remember well, my upload speed was something like 1MB/s at peak times. About 10TB of data would have taken about 4 months to be uploaded.

But this is definitely a good option to assess if you have a fiber-based connection!

4 Likes