Migrating a node

[humbfig]

The first one is BTRFS…

[Alexey]

I see, that explains a too high memory usage.
You can take a look on:

Information to SNO's regarding ZFS/btrfs vs. ext4 on spinners

I use BTRFS without issues now, but I had to disable future CoW and remove CoW from my existing files. My current BTRFS /etc/fstab settings are: defaults,nofail,noatime,nodiratime,nodatacow,noautodefrag. I ultimately had to disable CoW, shutdown my node, move the data, and copy it back (rewrite without CoW), then restart the node. I also tried a defrag before this (in theory it would allow you to do everything with the node online, but that was essentially locking up my node (watch your RAM u…

and

@Toyoo

btrfs was more than twice slower under all metrics even under the best settings I found (mkfs.btrfs -m single -d single, mount -o nodatacow,noatime,max_inline=0). I suspect that bandwidth.db updates dominate I/O performance there, maybe moving them to flat files—like the orders were moved—would help. But at this point this is only a suspicion and I’m waiting for some other tests to finish to check this as well.

and

-t 300 not enough

Finally finished migration. The difference is clear: I’m barely getting 20 writes per second, when I used to have even 200 on btrfs, when comparing a similar period in terms of a number and size of upload requests. I didn’t expect the difference will be so huge. One thing is that the btrfs partition was accidentally created with metadata in DUP scheme, which probably explains half of the difference. Maybe being log-structured might explain the other half? So while I don’t have hard evidence ag…

[Frenzy]

I had a Synology DS1821+ with 4GB RAM get in trouble with responsiveness. Yes, it’s the BTRFS disk/node.

Fixed it by limiting the memory for the container in the docker-compose.yml:

version: "3.2"

services:
  storagenode:
    ...
    mem_limit: "1536M"

I also ordered some new memory, after which I’ll be removing the memory limit again.

On topic:
This is the BTRFS node that’s currently running the Rsync to an ext4 disk. It’s synced 3.9TB out of 6.3TB so far.

It seems a storagenode on BTRFS will cause a normal drive to sit at 100% utilisation at around 6TB of data, at which the I/O wait become problematic.
These results come from a WD Red 8TB drive and a WD whitelabel (shucked) 12TB drive.

[humbfig]

@Frenzy

It seems a storagenode on BTRFS will cause a normal drive to sit at 100% utilisation at around 6TB of data, at which the I/O wait become problematic.

yep. I second that. The BTRFS node was almost bearable (with the help of some ssd cache of the db’s). But once it got to 6-7TB, the shit really hit the fan…

[EasyRhino]

my nodes are only about 1TB with btrfs, but I noticed that performance was very slow if COW was enabled, but seemed better if COW was disabled in my storj folder?

https://wiki.archlinux.org/title/Btrfs

(chattr +C)

[Alexey]

It would be a little better, but not as good as with ext4.

[Toyoo]

Disable CoW, disable duplicated metadata (use profile single for both data and metadata), disable inlining and atime, disable synchronous writes (needs recompilation of a storage node binary and breaks the ToS on a technicality), only then it becomes equivalent in speed to running on a tuned ext4.

[humbfig]

@Toyoo

Disable CoW, disable duplicated metadata (use profile single for both data and metadata), disable inlining and atime, disable synchronous writes (needs recompilation of a storage node binary and breaks the ToS on a technicality), only then it becomes equivalent in speed to running on a tuned ext4.

Better saying to use a “Better File system”. Meaning, not to use BTRFS…

[Toyoo]

btrfs is a decent file system, just not for Storj purposes. It’s hard to compete with ext4 if your use case doesn’t actually need any of features specific to btrfs. And, actually, if there was a version of ext4 with no support for hardlinks (instead keeping inodes directly in the directory tree), it would be even faster for storage nodes.

[Alexey]

Sometimes I think that it would be interesting to see how each FS is performing in case of storagenode, ext2, ext3, ext4 and xfs. I tested only COW FS like BTRFS and ZFS and was pretty disappointed about performance, so far only ext4 is performing better.

[Vortal]

I just tested a bunch of file systems after having issues with my node. In my experience the best file system to use currently is BTRFS with pinned meta data on a large nvme cache. After that it would be ext4

[Alexey]

You may check other Topics tagged btrfs to see that BTRFS is a worst FS for storagenode at the moment, especially when it collected more than a TB of data.

[snorkel]

I don’t know about BTRFS, but with ext4 I never had a Fatal error timout crash, on 6TB stored or 4TB stored, both drives on a NAS with 1GB RAM, no ssd. The machine makes same earnings like a 18GB RAM node with 10TB stored. Small drives are Iron Wolfs 8TB. So my go to fs is ext4.

[Vortal]

I am running 2 nodes side by side right now

EXT4 nvme cached

And

BTRFS nvme cached with the BTRFS metadata pinned to the nvme cache.

At this point the BTRFS file system is doing like 20% better performance wise than the ext4. However the nodes are small. Will continue to monitor and watch their performance. I may very well be moving the BTRFS to ext4, but right now it dosent seem that way.

[Toyoo]

Is this pinning a Synology-specific feature?

[Vortal]

I don’t believe so. Synology is the only out the box, but I’m pretty sure you can use BTRFS DUP on metadata to the nvme and that is what Synology is calling pinning.

[EasyRhino]

just following up on this thread because it was the most cromulent when I was searching.

I’ve been working on migrating to new disks, and the old disks are btrfs, and it’s been totally the wooorrrssst

The first drive had 2.5tb of storj data on it. The initial copy took two weeks. The btrfs source disk was only showing about 2MB/s of read performance while simply traversiing the file tree looking for files to copy. Then when real files were being transferred the performance fluctuated between 2MB/s and 6MB/s. At the tail end there were some bigger spikes in speed, but it was mega slow. Then the final --delete rsync took about a day.

The second drive has 5.5tb of data. I used the long wait time to “prepare” it as much as possible. Did a btrfs defrag, made sure the noatime attribute was set on every file (as well as drive mount options), switched the btrfs metadata from dupe to single (no idea if this really changed anything). And it was still mega slow, but not quite as bad. The initial copy of 5.5 tb took about a week.

[Alexey]

Yes, BTRFS is known to be slow for the storage node data, and thus not recommended.

[arrogantrabbit]

@EasyRhino

The first drive had 2.5tb of storj data on it. The initial copy took two weeks. The btrfs source disk was only showing about 2MB/s of read performance while simply traversiing the file tree looking for files to copy. Then when real files were being transferred the performance fluctuated between 2MB/s and 6MB/s. At the tail end there were some bigger spikes in speed, but it was mega slow. Then the final --delete rsync took about a day.

If you had BTRFS – you could simply do btrfs send | btrfs receive. The slowness is because you were attempting to copy file by file. Storj creates a lot of small files, so most of the time of course will be spent seeking, even if you have metadata cached. .

Sending filesystem will just stream everything at max sequential speed.

@EasyRhino

Did a btrfs defrag

big mistake.

@EasyRhino

btrfs metadata from dupe to single

Oh god…

@EasyRhino

And it was still mega slow, but not quite as bad. The initial copy of 5.5 tb took about a week.

Read about btrfs send/btrfs receive, and review this: Zfs: moving storagenode to another pool, fast, and with no downtime. It is about ZFS, but with btrfs you can do the same thing almost verbatim. Otherwise why are you using btrfs if you are ignoring all its features and advantages, might as well use ext4 then:)

[snorkel]

Maybe because Syno is pushing hard the adoption of btrfs, with all pros and shiny things. And if you don’t dig dipper, or you are a novice, you go with manufacturer recomandations.