Disk usage discrepancy?

[Toyoo]

@Dunc4n1d4h0

in the worst possible way

I didn’t say that. However I can say that

@Dunc4n1d4h0

node software from that point of view, just writes/reads files to disk

is ignorant.

@Dunc4n1d4h0

In my opinion terrible design. Whoever you paid those “hundreds of man-hours of a skilled software engineer” it was a waste

Sorry, not sure I understand you. I didn’t pay for development of the current solution, I’m just a node operator.

@Dunc4n1d4h0

Avoiding optimalization to keep compatibility z few or less % of hardware?

Not that I would ever run something that is supposed to run reliably on Windows,
but you write that instead of focusing on optimizing performance on the most popular consumer OS you spent time making the program run on some archaic NAS?

I know you are trying to mock Storj developers, but you shouldn’t. It’s them who made it work as a network, that’s a big achievement already.

@Dunc4n1d4h0

I’ve been here since 2020 and I think I’ve already managed to forget a lot of problems

The way you refer to Storj developers, it sounds like you’re new here. It would help a lot if you actually acted like wanting discussion.

@thepaul

If your problem is only with the lazy filewalker: if the lazy filewalker takes more than, say, 5x the time of the non-lazy one then your system is underprovisioned. It does not have enough I/O overhead to support the number of nodes it is running.

One thing that the lazy file walker broke is that before if a file was to be moved/removed, this was done immediatelly after reading its direntry and inode. As such, these structures were still in cache and did not have to be re-read to actually move/remove the file. Now however, if a low-memory system has to first read all inodes, and only then move/remove files, their file system structures could have already been removed from memory cache.

Another obvious one is that previously if GC was interrupted, at least some files were already moved/removed. Now we need to wait until the whole reading phase is finished.

@thepaul

If that ends up being the case, there may be some deeper magic we can leverage (i.e., maybe if we go lower-level than the Golang standard library and use openat and fstatat directly it would make a big difference? Or maybe reimplement os.(*File).ReadDir() so we can tune buffer sizes to getdents64()?).

I wrote down one suggestion here.

Another suggestion would be to give up on the temp directory, and write uploads directly to the blobs directory. This saves 2 synchrononous writes (plus some reads as well) to directories, plus at least ext4 will try to place the inode close to the blobs subdirectory, which may later save on seeks for both downloads and file walkers.

Though, better returns would come from giving up on using one file per piece.

@thepaul

On Windows, we may be doing all sorts of things wrong. We have a few engineers who are experienced Windows programmers, but I don’t think we’ve had time for them to take a close look at file traversal and see what we could be doing to make it faster.

Out of curiosity, are there any statistics how much of storage is hosted on Windows nodes? @Alexey suggested in the past we can establish a rough proportion of Windows to non-Windows nodes by looking at the number of downloads on Github, but it is probably also useful to look at total storage of Windows vs. non-Windows nodes.