EXT4 And Storagenode Free Space & Optimization

[svet0slav]

Hi, guys! I am wondering whether I should increase node sizes.

How much free space do you leave on EXT4? Any optimizations necessary for EXT4 in general?

[JWvdV]

On tuning ext4 for storage nodes

The ext4 file system is usually the default file system on Linux, as well as many NAS devices. It has several knobs that can be used to adapt the file system for some specific use cases. I’ve been doing some experiments lately on how to tune them for storage nodes, and found that there are some nice gains to get from the right kind of tuning. An inode is a place on a file system that stores information on files or directories, such as the file name, modification date, access permissions and so …

I’m reserving about 7%, maximum 100GB on every node.

But really: these are really minor optimizations. Not making you winning the race.

[daki82]

I go with ~1TB on my 12 and 20TB (10 btw 16TB reseverd for node data) drives, 100GB sounds bad, sometimes there is more than 100gb in trash.

[JWvdV]

But thrash counts in the node size, although you’re not being paid for it.

[snorkel]

I think the “overused” should be one of the factors took into account. Not trash.

[daki82]

Ah, you know about the 10% recommendation. Because just in case…filewalkers n things.

[JWvdV]

Yeah, I know.
But 10% of 20TB is obviously too much, and actually I never had a situation in which overuse was more than sone 100MBs.

[daki82]

So why not 5% instead of 1%.?

We will not come to a new recommendation. So the personal opinion is all we have.

[arrogantrabbit]

@JWvdV

But 10% of 20TB is obviously too much

Why? The recommended amount of free space is specified in percentages and not absolute bytes on purpose. The larger the volume, the more breathing room you need to have for the filesystem to do the job effectively. And those thresholds depend on the usage. For static data, like Storj, 5% will suffice, according to various sources, but I"m not an expert on ext4

Generally, when you are approaching 70% of storage used, it’s time to add more storage. Approaching the 90% volume utilization is not healthy for the filesystem and performance.

[JWvdV]

@arrogantrabbit

The larger the volume, the more breathing room you need to have for the filesystem to do the job effectively.

Why, no data backing up that it should be linear proportional.

There seem to be several reasons why they advice the 10%:

1. To account for the additional space used for the orders, databases and identity files. But these are only partially related to the size of the node (download orders), the remainder aren’t. Moreover is only several 100MBs.
2. To prevent fragmentation on an extent-based file system. But this is also depending on the size of files you save. Since we’re talking about file sizes of about 1MB on average, some GBs will be more than sufficient for this if necessary at all.
3. In order to have some space, for bugs to happen; for example A) if there are files being downloaded although the node has already been been filled up. But these problems will arise in smaller nodes she’s before they happen on bigger nodes, so you’re already been warned far ahead of not already been fixed before they become troublesome. This actually also happens when you delete the databases on the node, and the used space filewalker had to rebuild the usage statistics. Or for example B) if files aren’t being deleted, although already deducted from the usage statistics anyhow. But on bigger nodes, there is older data which tends to be less dynamic than younger data. So also these problems will be sooner troublesome on smaller than on bigger nodes.

So, essentially the space reserved is taken proportional to the disk size the node resides on. Although, I can’t think of any process or bug, that’s likely to behave proportional to the size of your node/disk.

Therefore we’re reserving space for some unforeseeable event (I at least can’t foresee), which leaves the question why it should be proportional (1, 3, 5, 8, 10, 15 or 20%) to disk size (why not square root of disk size, which would be statistical more likely). And why not just fixed 50GB?

I think, that 10% is made out of thin air.

[JWvdV]

@arrogantrabbit

Generally, when you are approaching 70% of storage used, it’s time to add more storage. Approaching the 90% volume utilization is not healthy for the filesystem and performance.

May be, but question is whether it’s relevant for STORJ. I mean, the performance of internet speed is a magnitude lower than disk speed. So it might probably never be troublesome.

Besides, the degrade of performance might be more proportional to usage than fullness. See for example: https://www.usenix.org/system/files/hotstorage19-paper-conway.pdf

Besides, performance loss is also part of the disk geometry: the start of the disk is at the outer side, and the written to the inner side. Therefore track speed at the start of the disk is significantly higher than at the end of the disk. See for example Large Spinning Hard Disk Performance Study •

So, it’s what you mean by healthy.
It’s very normal to have a performance degradation using a HDD when the disk filled up. Part of the design so to say.

[Alexey]

I would add, if the node detect, that it has less than 500MB of free space, it will reports as full to the satellites, independently of the specified allocation.
However, if we introduce a bug here…, well - you are doomed.

P.S. I’m guilty, I specified the allocation to have 100GB of free space on my 9TB of total offering…
However I seen scary messages that I have 100GB overused (I still has several hundreds GBs of the free space on that drive), so…

[snorkel]

We need guys like you that are willing to get their hands dirty and follow the forbidden paths, so we can discover the “what if…” and the limits of storagenodes.
Thak you for your service!

[Toyoo]

What is the best percent for overhead space?

I had it very close, below 5 GB of free space on a 5.5TB volume. That was, uh, kinda an accident. Fragmentation started showing up, but except for that, it was working well. Though given the code is still young, I wouldn’t recommend it either, bugs may happen.

@JWvdV

There seem to be several reasons why they advice the 10%:

One more reason: to account for file system overhead, which is not easy to measure. This includes things like cluster size, file metadata storage.

[JWvdV]

@Alexey

However, if we introduce a bug here…, well - you are doomed.

O yeah, but then we already passed at least one or two other bugs and/or thrown away databases.

@Toyoo

One more reason: to account for file system overhead, which is not easy to measure.

For sure, meta data normally is far below 1% and in most filesystems already (partially) reserved beforehand. And therefore isn’t reported as usage space.

Cluster size thing is actually uprounding of the file size. Most file systems are using a cluster size of 4096 bytes. With an average of 1MB files on STORJ this means you need to take (4KB/1MB/2 =) 0.2% due to cluster size overhead.

So, accounting for this all, you probably only have to reserve about 2% with an absolute minimum of about 1GB to be safe; and then it’s even a quite spacious taken safeguard.

But I still don’t see any substantial underpin off the 10%, which still seems to be made out of thin air.

[snorkel]

You must think about pros and cons. If you reserve 500 GB, you loose 0,75$ per month, but your node will most likely never crash. At least from bad allocation.
If you reserve 10GB, you gain like 0.75$ per month more, which will make you rich in no time , but most likely your node won’t survive too much.
So ask yourself… is it worth the risk?

[arrogantrabbit]

@JWvdV

With an average of 1MB files on STORJ this means you need to take (4KB/1MB/2 =) 0.2% due to cluster size overhead.

This 1MB assumption is two orders of magnitude off:

From 11TB node distribution of sizes looks like so:

  1k: 3312920
  2k: 3890418
  4k: 3472822
  8k: 5493570
 16k: 5553461
 32k: 7377023
 64k: 3034593
128k: 4795218
256k: 2811797
512k: 461677
  1M: 323325
  2M: 2322707

The median file size is therefore 16k, not 1M. So your cluster size overhead will be over 12%

[Mitsos]

IMNSHO: percentages don’t make sense when talking about large drives (I’m talking 20TB drives). I’m not wasting 2TB (=$3, literally two months electricity for the drive) for “what ifs”.

[Alexey]

Do not forget to account databases - they are not accounted in the used space.

[JWvdV]

@Alexey

Do not forget to account databases

See:

@JWvdV

To account for the additional space used for the orders, databases and identity files. But these are only partially related to the size of the node (download orders), the remainder aren’t. Moreover is only several 100MBs

@arrogantrabbit

The median file size is therefore 16k, not 1M. So your cluster size overhead will be over 12%

In this case you need mean -as in average- file size. And not median. Taking your data it’s about 170KB, which means just 1.2% if the filesystem doesn’t support inline data (or only very small like ext4) in case of block size of 4k.

In ext4 you can decrease block size to 1024b or 2048b when formatting, although I would only consider it on devices using 512b sector size.