Has it ever been considered to use in-memory databases?

[jammerdan]

@Alexey

how long it would take for example for the copy of bandwidth.db and/or piece_expiration.db

There is one more thing with that: The historical data.

Honestly: Historical data should be separated from the actual data.
This would help for 2 reasons:

1. The databases and their (in-)memory footprint would become significantly smaller.
2. In case of database problems the SNO would not lose the historical data but only let’s say the actual month or something.

Historical data don’t need fast access and almost never an update and could easily exist as a flat file on a spinning disk.

[Alexey]

It would require a separate chore to move data from the operational database to the historical database, it also will require to redo how this information is handled in the storagenode API (it must combine data from two sources (three if we account both operational databases - in-memory and on the disk, also historic DB) for each metric).
it’s as complicated as it sounds. As result - it likely slow down the node and anything what would request these metrics.

I do have more hope into this research:

@elek

So far I have heard about multiple attempt to improve the piece store. They are usually save the metadata of pieces in a database like system and save the raw data in custom files.

(technically it would be possible to store both metadata and data in database, but it’s quite slow. I tried it, and measured it )

Storing metadata and data in separated locations can be done in multiple different ways:

1. choose a database which already supports it, and stores the data outside of the main indexed data.

Badger looks like sg. like this. It stores the data outside of the SST files, and most of the LSM tree operation doesn’t need to touch the binary files itself.

Jacob has proof of concept of this approach (https://review.dev.storj.io/c/storj/storj/+/12066 ) and I also had an earlier experiment (GitHub - elek/storj-badger-storage).

[jammerdan]

Yes it would require to find a way to manage that. But the general idea is clear: Basically you have 2 databases where necessary. My idea would be to have one for the current month and one for the historical data. After each month the data is moved into the historical database. The API then would query the current months database for the current months data and the historical database when data before that is requested.

@Alexey

I do have more hope into this research:

At the end I don’t mind what solution will be implemented.
But I strongly hope that Storj will work something out to eliminate these database and IO issues. It is too sad to spend IOs on database operations and not for execution on customer files.

[Alexey]

… and three if you request an overlap interval. So, it should be supported. This is what would make it complicated.
However, let developers to decide how it might be implemented.

[Toyoo]

@Egon

With regards to something like bandwidth there could be two approaches.

There’s a third approach possible: update bandwidth.db while sending orders. Orders already have information on the amount of bandwidth spent, and they are sent on a regular basis.