Disk usage discrepancy?

[thepaul]

@Toyoo

One thing that the lazy file walker broke is that before if a file was to be moved/removed, this was done immediatelly after reading its direntry and inode. As such, these structures were still in cache and did not have to be re-read to actually move/remove the file. Now however, if a low-memory system has to first read all inodes, and only then move/remove files, their file system structures could have already been removed from memory cache.

Another obvious one is that previously if GC was interrupted, at least some files were already moved/removed. Now we need to wait until the whole reading phase is finished.

@Toyoo

I wrote down one suggestion here .

Another suggestion would be to give up on the temp directory, and write uploads directly to the blobs directory. This saves 2 synchrononous writes (plus some reads as well) to directories, plus at least ext4 will try to place the inode close to the blobs subdirectory, which may later save on seeks for both downloads and file walkers.

Ah, if we’re talking about the manipulation of the files, then yes, as you rightly point out, we have lots of room for improvement. I was only talking above about the directory traversal piece by itself.

Your suggestions are good ones and some we’ve talked about doing before. Some are harder than others. For example, giving up on the temp dir would mean we need some other way to differentiate partial uploads from full, intact blobs. We could try to use a database to keep track of what files are partial and which are not, but we’ve had lots of problems depending too much on sqlite databases or other single-file dbs. We could use different filenames, but then the rename() step still needs to do multiple writes (it does a link and unlink, unless Linux is able to overwrite a dirent with a new filename now). Also the directories would have a lot more entries in them to go through, which could hurt the performance of GET operations.

@Toyoo

Though, better returns would come from giving up on using one file per piece.

This is probably true, but we haven’t yet found the ideal way to do this that (a) doesn’t involve basically reimplementing the filesystem; (b) doesn’t hurt performance (both TTFB and throughput) for GET operations; and (c) still provides ranged reads and so on. We’ve had some promising results with experiments using LSM tree storage, but there are still big challenges to get over.

Regardless of all that, there’s a lot we can do, and as you note, it’s just a matter of assigning resources to the problem.

@Toyoo

Out of curiosity, are there any statistics how much of storage is hosted on Windows nodes? @Alexey suggested in the past we can establish a rough proportion of Windows to non-Windows nodes by looking at the number of downloads on Github, but it is probably also useful to look at total storage of Windows vs. non-Windows nodes.

I think this information can be put together; nodes send build hashes with their telemetry data. If we collected all the hashes of released builds together with the target platforms, we’d at least have a good idea of the percentage of Windows nodes. But that would take a nontrivial amount of work. I don’t think we have anything better available, at least within the sphere of the project I’m familiar with. Downloads on Github is probably as good as anything. That would be one of the first things to fix once someone can be dedicated to improving node performance.