Dear storj,
I am sorry to bring this topic but I need a help on the general topic of copying my data from first backup server to the second one. The hosted the main server from the server hosting company which is California which is located exactly to the opposite of where we are which means the first server is accessing the main server data with a limited bandwidth from national ISP. So if I wanna have a minimum downtime I should first mirror the first server with the second one over LAN. But will the transfer speed be really fast enough over an fiber optic if I wanna mirror the second backup server from the first server? Actually I have been planning with my friend to work on a project of setting up a server that is 400 kilometers away from the office. We are planning to subscribe for a leased line connectivity for internet and intranet. The dedicated leased line will have a bandwidth of 5 Mbps internet uplink and downlink and same goes from intranet. He owes a office for website building and content writing. So I suggested him to use Libre office Online for content creation. But the reason I am contacting is also because I am gonna be using a qnap nas for the second backup server. What should I do?
Is there an app in qnap to help me overcome this problem and mitigating the downtime?
And I have the other question regarding dedicated leased line connectivity. How should I configure my router if I wanna use the same optics fiber for the internet and intranet? Is there any app in qnap for copying data from the other server without closing down the server for a while and copying the data. If only downtime wasn’t the problem I could have easily copied the data by simply inserting the blank HDD and the HDD with data using the same NAS. If there are two servers, one that is connected over internet and the other which is connected over LAN which is connected in the same router, will I get the speed of the LAN while mirroring the data?
Thank you
Dhiwahang Rai
There are a lot of questions there and I’m not sure I get the whole setup ^^
You want to migrate a node from one server to another, remotely (via Internet connection), is that correct?
A 5mbps connection doesn’t seem fast enough for this to go well, but you’re suggesting you may have access to a fiber connection: in which case I assume you would have at least 100mbps on it: That should be enough but depending on the size of your node it may take days, maybe weeks for the first sync.
Here is the official documentation for migrating a node, check it out:
If it doesn’t fit your particular case, I think it would be helpful for us to know:
- What is size of the Node we’re talking about (how much data does it store)?
- What is the setup you want to migrate from (OS, storage device filesystem (ext4, ntfs, …), available connection speed, network setup as in is it behind CGNAT, firewall, …)
- What is the setup you want to migrate to
- If you have the possibility to ship the hard drive containing the node data: if that’s the case, the procedure could be pretty different (and potentially faster… it depends).
I am sorry to bring this topic since it’s not related to storj but is about a personal project. My friend owes and runs a company of content creation and website building in Kathmandu. I suggested him to move the physical location of the backup server which is located in his office in Kathmandu to my village which is 400 kilometers away from the Kathmandu city.
But first I suggested him to set the time of vetting period to 3 months. Since we also have to calculate all the negative factors like power cuts, internet cut downs and all that can jeopardize the project. He is using a PC whose OS is windows server. He has a data which is over 24 TB. Please suggest me which RAID should I use for this amount of data? His concerns security as the top priority for his data.
I am planning to use Qnap 18 bay NAS as a second backup server. He is also planning to encrypt the disks since security is one of the top concern for him.
So our plan is to copy data by mirroring the server from his office over the LAN. Since a day of server downtime could cost him about $1000 of loss. So I want to minimize that downtime to 2 hours where those time would used for the physical travel and installation process.
So here comes the question. The main server is in California, US which is hosted by a server hosting company. The first backup server is in his office which he is planning to move the responsibility to the second server after the end of the vetting period. For copying of data to the second server by mirroring (since we want to minimize the downtime) where we will be using the same LAN as of the first server, is it possible to move data from the servers with sync? I want the process to be fast. That is why I will be using the same LAN connection using fiber optics to the first and the second backup server so that the old data stored in the first server transfers fast enough. And I am mirroring the first and second backup server with he main one so that our downtime occurrence would be less while copying old data and syncing the new data too with the main server. In the middle of this process he will be notifying his clients about the downtime.
Will all this plan workout as the pre-setted target to meet the goal or will I be missing something?
Will the transfer of old files from the first backup server take place or will it be directed to the main server and take a lot if time since the ISP companies limits our bandwidth.
Since I am using a Qnap product, will I be able to view the dashboard over the internet and the intranet?
I also suggested him to use libre office online and subscribe for intranet for content writing purpose.
Since I am new to all this, will you too please suggest me the router settings I should be configuring so that I can use the internet and intranet over the same wire in a dedicated leased line connectivity? And planning to use a fiber optics router.
Maybe you should suggest to him to move his data onto Storj DCS instead?
With that
he should be easily able to pay for it.
Yeah, I did!
But he is not up for it dude!
Sorry but I tried my best to persuade him, but couldn’t!
Tell me a thing, if a SNO looses the data from their side, will the data be lost forever?
Weird. Why is that? He can try out the free tier first and then move to a paid tier.
It is much better than running his backup server in his office, which is honestly dumb.
And he would not have to make his head about moving servers.
Suggest to him to just try it out with lets say 1 TB and see how it works.
A node does not store original data. All data gets encrypted and erasure coded hacked to pieces before getting stored. Only these erasure coded pieces get stored on nodes. The beauty of this approach is that the original data can get reconstructed from the pieces as long as there are enough pieces available. So if a single node looses data it does not matter as there are enough other nodes who will provide enough pieces to reconstruct the underlying data. And if the number of pieces falls below a certain threshold, new pieces will get generated by repair workers to make sure that at all times enough pieces are available to reconstruct any data.
So far Storj has lost many pieces, but not a single bit of customer files because of that design.
Edit: You know, even the Internet Archive is trusting Storj so much to test for backups:
https://twitter.com/internetarchive/status/1467478175374614531
I don’t see how your friend could have tighter requirements than the Internet Archive.
I can think of a few (what looks to me, valid) reasons.
It’s a bit difficult to understand, but let me know if I got this right:
- There is a “main server” somewhere in California, accessing it is slow because of slow internet.
- Your friend has a backup server in his office. I assume this is for events if data is lost in the main server, so you can restore it.
- He wants to have a second backup server 400km away.
- The problem is how to copy the data from the first backup server to the second.
If I got this right, if both backup servers are the same, you can just copy the files to the second backup server and then move it 400km away.
You can use VLANs to use “one cable as multiple cables”, though depending on how everything is set up you may have to do something differently.
Sorry for mixing up two subject in one. The later post of question was a general query that popped into my head.
But the prime concern for me is that will I be able to copy my data from my servers (syncing with the main server so that I can be online and copying the old data from the first backup server over the Lan so that the transfer speed would be very fast. Or will my transfer limit be limited to the ISP speed. Since there are two servers, main server for being online and the first backup server which is in his office. I want to to copy the data from his office server which is the existing server through which he locally access the data, which syncs the data from the main server which is in the California. But the main quest is that will I be limited to the ISP’s bandwidth or will I have the full bandwidth of the router?
Had it not been the necessity to make the office server online with the main server, I would I have take out the HDD from the office server and insert it into the NAS and the copied the data. But the problem is that the office server have to be online with the main server while copying the data from the office server. I wanna copy my data as quick as possible being online with the main server. But while copying the data to the NAS which is connected to the router of the office server, will I get the speed of the LAN or the speed provided by the ISP?
And all this process just to make the downtime less.
It has became a little hard to even make you understand the concept.
In theory, you should be able to copy from the office server at LAN speed, though in practice it would depend on what is in those servers and how they are organized.