Thank you very much! This is really interesting, I think to see a similar stats (like stored files, db size, node count, space free and used) on the Town hall or blog will interesting
Even though the point is a little moot, I did have a look at the databases. But unfortunately the storagenode looks to have daily windows in its database for storage used. This is probably a summary of all snapshots the satellite has generated in a given day. And some days are missing, which suggests walking all files took more than 24 hours that time.
I was hoping the exact snapshots the satellite generates are sent back to the nodes, so we could locally calculate the average stored amount reported back per snapshot based on the time passed between snapshots. The current aggregation makes that impossible and leads to nonsensical raw numbers in the node databases. However, you can still average these numbers over a longer time to see whether the total makes sense and for my node it does seem to line up with actual space usage. But it’s a lot harder to be really precise that way.
What I can say is that the data shows it catches up.
You can see the missing day on the 22nd and the higher number the next day.
Have a look at the design blueprint: https://review.dev.storj.io/c/storj/storj/+/1732/3/docs/blueprints/sparse-order-storage.md#111
The most important part is
O(log n). This indicates non-linear scaling of storage requirements and related performance. Instead it will scale logarithmically.
This will get you several orders of magnitude of extra breathing room. So if everything works out as planned, this should be able to eliminate the bottleneck.
This is what I thought while reading your first explanation as well - why do the equivalent of
du every day instead to it less often and keep a running total in between.
Well, that explains why I get weird values when I try to graph how much data from each satelite my node has (using the API to avoid running
If some of this issue resolves around API saying a higher number that is because this value is in cents
This is unrelated to space allocated/used