Saltlake used space wrong calculation on dashboard that can affect payout calculation... again

[anon68609175]

Thank you very much! This is really interesting, I think to see a similar stats (like stored files, db size, node count, space free and used) on the Town hall or blog will interesting

[BrightSilence]

Even though the point is a little moot, I did have a look at the databases. But unfortunately the storagenode looks to have daily windows in its database for storage used. This is probably a summary of all snapshots the satellite has generated in a given day. And some days are missing, which suggests walking all files took more than 24 hours that time.

I was hoping the exact snapshots the satellite generates are sent back to the nodes, so we could locally calculate the average stored amount reported back per snapshot based on the time passed between snapshots. The current aggregation makes that impossible and leads to nonsensical raw numbers in the node databases. However, you can still average these numbers over a longer time to see whether the total makes sense and for my node it does seem to line up with actual space usage. But it’s a lot harder to be really precise that way.

What I can say is that the data shows it catches up.

You can see the missing day on the 22nd and the higher number the next day.

@andrew2.hart

But if the system struggles at a few peta bytes how will it work at 100 petabytes? Does it need to scale out instead?

Have a look at the design blueprint: https://review.dev.storj.io/c/storj/storj/+/1732/3/docs/blueprints/sparse-order-storage.md#111

The most important part is O(log n). This indicates non-linear scaling of storage requirements and related performance. Instead it will scale logarithmically.

This will get you several orders of magnitude of extra breathing room. So if everything works out as planned, this should be able to eliminate the bottleneck.

[Pentium100]

@zeebo

That said, we can do something like a hybrid approach where we keep a running total that we add and subtract from and only write the delta to the database when it’s either large enough to be relevant or stale enough, and use the periodic walk to get a true value. It would be even better if we could do the walk as a batch job in the background operating on database backup/snapshots so that we can more easily scale it and backfill data if there’s unexpected delays.

This is what I thought while reading your first explanation as well - why do the equivalent of du every day instead to it less often and keep a running total in between.

Well, that explains why I get weird values when I try to graph how much data from each satelite my node has (using the API to avoid running du).

[1qazx]

If some of this issue resolves around API saying a higher number that is because this value is in cents

[Alexey]

This is unrelated to space allocated/used