Moving from Windows to Ubuntu and back

[Alexey]

Continuing the discussion from Moving the Node from Windows to Raspberry PI3:

I would add my personal experience with NTFS on Linux.
Due to the forced update of Windows 10 to the next release, my system became unstable: it crashed into BSOD, rolled back the update, but since this update is mandatory, Windows is updated to the next release, crashes to BSOD and everything repeats.
After a day of such behavior and inability to prevent an upgrade, I decided to migrate to Linux, since this system is not used as a Game Center anymore.

So, I migrated to Ubuntu.
And after a month I lost two biggest satellite from my oldest node, because it was on NTFS and my system have not had a room to temporary move data.
There are many ways how to convert even with such restrictions.
The best would be to shrink NTFS, create LVM partition (create PV, VG, LV), format to ext4 and move data there. If there is not enough space, you can do it step by step.
However, there is one caveat - your system must be stable.

Here is my story

I went through a long process to convert NTFS to ext4 in place:

1. I had shrunk the NTFS partition in Windows VM (KVM) with connected physical drive (node was offline)
2. Created a new partition, had format it to ext4 and migrated a part of the data from NTFS to ext4 with rsync
   Since the whole node cannot be moved, I moved only the part (node was offline).
3. Then I shrunk NTFS again and tried to use GParted to move and extend the partition. NEVER DO THIS.
   During the process I suddenly was surprised with “General protection fault” related to CPU (kernel bug), and as result the partition was partially corrupted. I have had recovered most of the partition, but not enough. With this I lost the first satellite as turned out later (the node was disqualified on Europe-North-1 at the end).
   I googled some time and come to the conclusion that I need to use LVM, it allow to do what I want and:
   3.1. it can do that online with mounted drives and working programs,
   3.2. it’s safe - even after power loss you can continue move where it was interrupted.
4. I installed the LVM and successfully converted GPT partition to LVM (this is a very precise task, you can easily lost everything, see Convert an existing Linux file system to LVM – Pascal Roeleven). So create your LVM partition right away, converting will not be an easy task later.
5. Then I moved another portion of data and experienced several “General protection fault” events during the process. However, I didn’t lost any data so far because of move. BUT. Every time when this happened - the NTFS partition become corrupted. I forced to use Windows VM to recover it after almost every “General protection” event. So there where I started to lose data even more.
6. I shrunk the NTFS after move again, created PV and successfully used this space to extend without any problem. This is a good experience in comparing with GParted.
7. I repeated the process the same way - move a part of the data, shrink NTFS, create a new PV, join it to VG, extend LV and move data from NTFS, do repeat.
8. When all data has been migrated, I started the node. Everything was fine, but “general protection faults” didn’t leave me. As result I lost the second biggest satellite (Saltlake of course, what else?).

I finished in the configuration, where the half of the disk was empty (luckily at the beginning) and half occupied. I joined all small PVs to the two big ones - 4 and 4 TB each (because my node shrunk in 2/3 during the exercise…).

At the end - 4TB from 6TB is lost, the system is unstable, even reboot after “General protection fault” didn’t help, you need to explicitly press the Reset button on your PC (do you know where is it?).

Probation period with Linux

After a month of almost every day practice to search a “Reset” button on my PC (I never used it before, my PC box have a design where Reset button is not explicitly visible, as a Power button, they are part of the box, you need to know exactly where they are, but I learned where they are . You may not believe me, but in the previous 8 years on Windows, I never had to look for them), so I decided to return back to Windows, even if I successfully use Linux a lot every day on my daily work and many other place.

As turned out - Linux is very sensitive to hardware issues, and they are not handled well even after so many years. My 8 years old system was surprisingly very unstable under Linux, especially if I use VMs (and I need them for my work).

I have a Raspberry Pi which is working without my attention since 2018 and it’s live inside the sofa in a living room (because the sockets were behind the sofa …).
This raspberry Pi uses ext4 from day zero and thus have no problems so far.

Back to the Windows

Since I finished with two PVs, the move back was an easy task.

1. I had moved all data from the first PV (at the beginning of the disk) to the second (online, I love LVM for that).
2. Removed the first PV, created a usual Windows partition instead, formatted it to NTFS (still in Linux), tried to migrate with rsync, but “General protection fault” come every 12 hours, so, I really was disappointed and a little bit angry, so…
3. Installed the latest Windows to system drive (and of course I forgot to move nodes databases back to storage drives…)
4. Enabled Hyper-V, installed mutlipass
5. Created a simple light-weight ubuntu VM, mounted physical drive to it
6. THE Linux VM WAS ROCK STABLE. I’m shocked. The same HW, nothing changed, except host OS. Boomer.
7. I have migrated all remained data to NTFS
8. Stopped the VM
9. I have removed the last PV and extended NTFS.

No one BSOD or general protection fault during all operations

Oh my God. Heaven and Earth. The Windows working a week without any reboot so far. I cannot believe, if it was not my system. Windows is working ROCK SOLID. impossible…

Few gotchas

1. Do not use NTFS on Linux. Especially after Windows 10. This OS uses compression and deduplication features on NTFS by default, without special plugins to ntfs-3g it will be corrupted under Linux.
2. If your hardware works unstable under Linux - do not use Linux. This problem will kill your system.
3. If you do not have enough space to backup your data, you may use LVM to move data step by step, but you will end with several PVs on the same or (dangerous!) several disks. You can move data and merge PVs on the same disk online, if you have enough free space (need to move from the end of the partition at beginning of the disk every time). However, if the source filesystem is NTFS and accessed from Linux - you likely lose data because of gotcha 1
4. Do not use GParted to move partitions, use LVM. It will be slower, but much safer, if PVs on the same disk.
5. Do not shrink NTFS under Linux, you will lose data. Do so on Windows.
6. If you use LVM - do not use simple volumes across several drives. You will lose data with one disk corruption. If you forced to - use volume with parity. However, with todays disks you have a great chance to lose all your volume even if you use volume with parity, see

Hardware configuration and receiving mail with token

8 disks is not few disks. You may think you’re running a small risk, but you’re absolutely not. Please read this: The Latest - Whether RAID 5 is still safe in 2019 - Digistor Australia If you don’t want to read the entire article, here is the most important image. [image] With your setup in RAID5, you’re almost guaranteed to fail a rebuild. Have enterprise level HDD’s with lower URE rates? The image does look better, but still not great at all. [image] It seems the P440 supports online …

[SeWIR]

@Alexey

If your hardware works unstable under Linux

Dont use newest HW under Linux. Or maybe youre HW is corrupt (BGA misscontact, RAM errors, etc.)

@Alexey

Do not use NTFS on Linux…

…NEVER.

[Alexey]

@SeWIR

Don t use newest HW under Linux.

It doesn’t work normally on old HW (2013) too. I don’t know, what the HW is needed
Seems SOC like raspberry Pi are good enough.

[SeWIR]

But linux have some logs mechanism. If your`e HW is not working under Linux, or catch Kernel Panic - in 98% u have some trouble with HW.

Linux doesn`t have utils for correcting errors on NTFS, just check and manage some state flags. ntfsfix
is not an analog of chkdsk under Windows.

[Alexey]

Yes, I know. However, as turned out, Windows is working fine on the same HW with problems under Linux. This is not a first time.
Linux is a great way to check your hardware at edge.

@SeWIR

Linux doesn`t have utils for correcting errors on NTFS, just check and manage some state flags. ntfsfix
is not an analog of chkdsk under Windows.

This is why I used the Windows VM to check every time corrupted NTFS while it used heavily in Linux by storagenode.
You need to stop the storagenode container, unmount the disk and run the Windows VM with the physical disk attached. Then you need to run chkdsk /f D: as an Administrator (the disk letter you can see with Get-PSDrive in the PowerShell).
After the fix, you need to shutdown the Windows VM (do not detach the disk while VM is running!), remove the disk from the VM configuration to do not accidentally run the VM, while the physical disk is used, then you can mount the disk back to your system and run the storagenode again.

However, as I said - using the NTFS under Linux is a best way to lose your node (after running more than one node with the same identity ).