Node migration over wan

[xavidpr4]

Yesterday I migrated a node from one siteI have to another site via WAN. To sync the data it took about 2 days (about 900GB transferred).

Everything was ok with the Rsync. The last steps I performed were:

• shutdown the source node by doing “docker stop -t 300 storagenode”
• issue the last Rsync (however I forgot to put the --delete). It took 10 min to sync the last data.

Then I started the node in the new place and i realized that, even the node was starting I was not showing properly. Then I checked the logs and saw that the database was malformed.

I restarted the node three or four times, just to see if this could fix the issue, but nothing.

Then I checked 2 things:

• How to properly migrate a node (and realize about the --delete parameter)
• How to fix “the database is malformed”, which is a nightmare.

I’d tried for issuing the rsync again but with --delete parameter now. After doing that, the node started well and all the information apparently was ok, showing all my data and so on.

My question is, being about 2 minutes or 3 in each try with the database malformed, can damage my node?

And the second question is, if after issuing the rsync with the --delete parameter it worked properly (even considering that the previous tries weren’t correct) this means that everything is ok or do i have to worry for something?

Just to clarify, it happened something similar to that:

Linux Node migration tip (Step 8)

Hello everybody. I would like to give everyone who migrates their nodes the following tip: Manually back up your databases to make sure you are backed up or skip step 8 until your node runs again. Step 8 in the documentation deletes all data from the old hard drive. I only delete when everything is running again so I skipped this step. After running rsync several times, I started the new node and my log was full of: "failed to add bandwidth usage {“error”: “bandwidthdb error: database disk …

Thanks!!

[kevink]

@xavidpr4

And the second question is, if after issuing the rsync with the --delete parameter it worked properly (even considering that the previous tries weren’t correct) this means that everything is ok or do i have to worry for something?

Your database was considered malformed because your last rsync without --delete copied a “good” database but the .wal files with the “temporary changes” were still there, which is likely what caused problems. Your final rsync --delete then got rid of those .wal files and the DBs were fine.

@xavidpr4

My question is, being about 2 minutes or 3 in each try with the database malformed, can damage my node?

Unlikely, the worst that could have happened is some audits failing (but not sure that is still a problem). So if your audit/suspension scores are still fine, I’m sure your node will be fine.

[joesmoe]

I’ve made the same mistake. I just ended up deleting the nodes is crewed up, but I agree with Kevink, may as well let it run and see what happens, maybe not enough is messed up to cause long term problems.

[xavidpr4]

Thanks Kevink for your clear explanation. I will let it run and let’s see. Hope that everything is fine as it took about 3 months to fill this TB of data.