if you are already uses Grafana, you may calculate your own estimation, even if the report from the particular satellite is missing. Also, do not account missing days in your average calculation.
Maybe Iāll have to do something like that.
Currently I am using the already averaged data from the API and not the values per day.
Normally I think if there is no data for a day, it should not be calculated with 0 as value. This sounds wrong. If there is no data, this does not mean zero. It means data is missing, so maybe the last available data should be used instead.
But again, what use does data have, that I calculate how I like it. This is not what I would like to see. I need accurate satellite data I can rely on. If I make my own calculations I can make up anything.
perhaps you need to use the latest available report instead. Seems the average is way off with these gaps (and I do not expect that it would be changed, unfortunately, at least now).
yes, I agree, however, we have what we have now. And itās unlikely will be fixed soonā¦
right now itās not guaranteed, Iām sorry. It will be precise for the past month, but not for the current period, at least now. So, the estimation would beā¦ an estimation?
Using the Prometheus endpoint to calculate the payouts for example is not reliable at all, as I looked and it looks like there is no TBm value nor payout exported there.
Thus you can only use the node or sat reported instant used space and mutliply that by the payout rate, but as I have realized during these load tests, this is not representative of the paid data at all, as the the node might have gained 10TB in the last week of the month, but of course you wonāt get paid for 10TBs if the node only held it for 7 days. And the sat data have gaps in them as well, so this gets even more unreliable if there was no BF for the last two weeks while some data were actually purged from the network.
So after abandoning the Prometheus node_exporter written in Python, which grabbed the data via HTTP API, and after migrating to node Prometheus endpoint, Iām no longer wasting time to look at those metrics, as they are either missing or are simply wrong.
And it helps with the anxiety not to look at that every day and try to solve all the unsolvable problems by throwing more and more hardware at the thing, or by making purchasing decisions based on predicted payouts that are simply not true.
1 Like
You are very correct, unfortunately. Right now, we (SNO) do not have a reliable source of truth. The node paid for the submitted orders, but our dashboard doesnāt use this info! It uses the (probably) correct data from the databases about used space, and some scripts are trying to compare the reported (unreliable) used space from the satellites with the (not reliable too) information from the local databases (which are updated only in a happy case - no one filewalkers are failed, no database errors)ā¦ I would assume, that using a dashboard for anything is waste of time, unless you do not have any errors, related to a databases/filewalker (are you?)ā¦
2 Likes
Exactly.
Maybe the solution would be to make the Storagenode headless (very ideally with an option to dedicate the whole mountpoint, so we will no longer have to babysit the available space), and make the dashboard sort of a cloudy one on the satellite side.
To get what the satellite thinks we have, we would login or call some API, and to get what the actual used space is, we can use the tools each OS has. And then we can compare these values on our own to see if there might be some potential problems.
I also quite donāt get why there is a satellite job running each day to recalculate the used space on all the nodes.
Why not making this realtime in some sort of an in memory database, such as Redis, where after submitting the orders the database would simply be updated with used_space += uploaded_piece_size, egress += downloaded_piece_size for each node - two update statements per each node per order I assume, and some more in case stuff will be deleted from the network.
Then all these recalculations would be only needed as a source of truth for the payout purposes and this way there would be no late data, no gaps etc.
1 Like
Something like a āstorage.use-os-free-space: trueā config.yaml flag would be really nice (even if you maybe also had to specify the mount/drive to go with it).
I donāt care how much space a node uses: just that it tries to leave 500GB free.
5 Likes
It would be a little bit more complicated. The satellite calculates the used space based on a signed orders, sent from the nodes. Thatās mean that statistical information should be added in the same request from the customer and it will slow down the response. We do not want to slow down, we need to speedup!
So, I do not think that collecting the statistic on the fly is a good move. Thus itās a separate process. Summarizing the data is never fast, as far as I know.
Or do you suggest to implement it on the nodeās side? Because right now the node doing exactly the suggested steps. When the upload is finished, it increases the used space in the database. If the piece is trashed, it reduces the used space and increases the trash used space. If the piece is deleted due to expiration (TTL), it reduces the used space. If the piece is deleted from the trash due to expiration (trash-filewalker), it reduces the trash used space.
Databases are cached in memory and flushed to the disk every hour by default. So, some kind of temporary REDIS.
1 Like
Thatās a good idea! And specify it for the nodes which uses the whole disk/dataset/partition or setup quotas.
Could you please add it to the Storage Node feature requests - voting - Storj Community Forum (official)?
I cannot imagine this would bring any noticeable performance hit, especially as satellites communicate with nodes and customers over long distances with significant latencies.
You can also make this call async and delegate it to another process, so it wonāt slow down the usual stuff.
On the satellite side.
On the node side it was fine until the satellites stopped informing the nodes about the deleted pieces in realtime. Now to estimate how much data is unpaid we have to wait for sat report, which lately is very unreliable because it takes long.
How long will it take to count if the network will be 10 or even 100 times as big as it is now?
And to speed it up probably wonāt be as easy as throwing more cores and memory at it, as you would have done that already if that would be the case.
So fundamentally this approach isnāt a good one I think.
3 Likes
You are right, we must find a way to speed up the process, if the network is too big to a point that the calculation canāt finish in few days (at the end of a month) then how long SNOs must wait to get paid?
SNOs have been getting paid at the start of every month, like clockwork, for years. Havenāt they?
Numbers in the node UI have nothing to do with if nodes get paid or not.
1 Like
Yeah the clockwork varies from the 4th to 14th of the month, lol. And I donāt say for today or tomorrow.
This usually depends on when all nodes would submit orders (up to 48h), working days (because if the payout would stuck on weekend we need someone, who would restart the payout or fix the issues if any), some retries (because of fee fluctuations) to send as much payouts as possible and some issues like the last one when the zkSync Lite suddenly stopped to support to pay fee with STORJ tokens. We waited a little bit longer to get a response from the Matter Labs, and then sent the stuck zkSync Lite payout via L1 as a fallback (with its Minimum Payout Threshold unfortunately, but with 3% bonus as well).
1 Like
I do not think that the not sent report from the long running tally affects the payout.
I have more questions. So these numbers can be trust, or not? Because the 2.92TBm is totally wrong due to the wrong chart. Those numbers are for local display only?
Thereās a discrepancy for me between the satelliteās average usage across all nodes, which is less than 50% of the nodesā own used space reports. Unfortunately, neither source appears to be trustworthy.
3 Likes
Do you have a plan for the next month?
Lite support has ended, so itās not a problem next month.
āFuture payouts will simply choose your preference between zkSync Era and Layer 1ā
You should be using a L1 exchange address and immediately selling to fiat anyways 
3 Likes
I endorse this advice unless/until itās possible to move zksync-era to a fiat cashout without triggering nasty ethereum gas fees.
1 Like