Updates on Test Data

[Alexey]

I know, and there is a work in the team, but lightly exposed right now
like

Disk usage discrepancy?

Ah, if we’re talking about the manipulation of the files, then yes, as you rightly point out, we have lots of room for improvement. I was only talking above about the directory traversal piece by itself. Your suggestions are good ones and some we’ve talked about doing before. Some are harder than others. For example, giving up on the temp dir would mean we need some other way to differentiate partial uploads from full, intact blobs. We could try to use a database to keep track of what files a…

Has it ever been considered to use in-memory databases?

It would require a separate chore to move data from the operational database to the historical database, it also will require to redo how this information is handled in the storagenode API (it must combine data from two sources (three if we account both operational databases - in-memory and on the disk, also historic DB) for each metric). it’s as complicated as it sounds. As result - it likely slow down the node and anything what would request these metrics. I do have more hope into this resea…

Design draft: a low I/O piece storage

Thanks for this design doc @Toyoo, It’s a well written design, I like it. And finally I had some time to collect my thoughts. TLDR; I like the approach, but I am not sure about the implementation. So far I have heard about multiple attempt to improve the piece store. They are usually save the metadata of pieces in a database like system and save the raw data in custom files. (technically it would be possible to store both metadata and data in database, but it’s quite slow. I tried it, and mea…