Well your shorthand to use last months end size doesn’t correct for the other differences. I’ve been thinking about adding more numbers to better show the differences, but I haven’t found the time to implement those changes recently. Regardless, there will be gaps I can’t show. Your node may have a discrepancy in calculating used space, leading to a mismatch of Total disk use calculated and actual disk use. The filesystem may cause larger usage on disk than the actual file size as well. Files that have been removed by customers but not yet cleaned up by garbage collection can’t be detected by the node. So there will always be gaps. But I’m hoping some additional stats may show where those gaps are unreasonably high.
I also saw some of the code I’m using has been deprecated, and throws a warning in later versions of python. I’m hoping to tackle that soon. Despite the warning, it should still work though. For now at least.
Thanks for the explanation. That all makes sense. If you are able to tweak things to make that easier to see and analyze I won’t complain, but just having a general understanding of what is going on and the different factors that can impact that behind the scenes helps alleviate concern that things are working ok and that i don’t have a bigger issue building up where something in the system thinks i have way more data than i actually have. And of course, its always good to know im getting compensation for the storage im providing!
The first audit can take a while, though you also seem to have little data for a 2 week old node. Is it sharing ingress with other nodes? This will slow down vetting too.
Nope, it’s just one node, no neighbours. I remember that they said they reduced the percent of data for unvetted nodes. Maybe this is the normal traffic now.
I see similar for a node added 2 days after… extremly low traffic, and no audits for vetting at all.
It would indicate the the adjustment made a few week ago was tightening it too much. At this rate it’ll never complete
Edit: I just noticed the calculation in this version is wrong. Fix incoming now. Do not use v13.2.0.
As promised a new release today that adds more detail regarding trash. This new data can be used to more easily compare the satellite reported Disk Average So Far, which is used for payouts, against the Disk Current, calculated by the file walker locally. By separating Trash from Blobs, you can compare Blobs to the satellite reports to get an idea of how much data still hasn’t been cleaned up by garbage collection. Please keep in mind that Disk Average So Far refers to a monthly average, so on growing nodes, it is expected that this number is lower than Disk Current Blobs. You can refer to the ingress amounts to see if the difference is reasonable.
Apologies for the previous bugged version. I should have tested more, but the numbers seemed to line up on the specific node I was testing with. I also made a small change in layout to make it look more clear.
So when i look at my current standings, i see Disk Total is 16.72, with trash being 825G, blobs being 15.89TB, and Disk Avg being 10.89TB. If nothing were to change from now until the end of the month, should the disk avg be fairly close to the Blobs total? Or is being off a few TB possible and not concerning depending on some of the factors you mentioned before? And should i be moving this to a new post at this point to keep this thread clean?
I would say that difference is definitely abnormal, even in the current situation where we know garbage collection isn’t working ideally from the satellite end. I’m not seeing differences that large on my nodes. Though I do see up to about a TB or a little more. 5TB is extreme though. It’s worth checking your logs to see if garbage collection runs into any issues, but I recommend you move those questions to a more appropriate topic. There are already several active topics covering this issue.
It seems like the deprecation of the utcnow() function has lead to a scenario where there is no way to support all python versions anymore. It now requires version 3.2 or higher. Can you check your python version on QNAP? It’s possible that your system has versions for python 2 and 3. You can also try calling the script using python3 instead of python at the start.
That version should definitely work. I’m running it on 3.8 myself. It’s possible it’s running python 2 over ssh. Can you run python --version to see what version it is using?
Looks like python3 isn’t in your PATH. You might be able to fix that by running /etc/profile.d/python3.bash. But you’ll likely have to do that after every reboot.
Traceback (most recent call last):
File "/Users/xxx/Desktop/storj/earnings.py", line 379, in <module>
for data in con.execute(query):
^^^^^^^^^^^^^^^^^^
sqlite3.DatabaseError: database disk image is malformed
Unfortunately it seems one of your databases is corrupt. This may also impact your nodes performance. Make sure you don’t ever access the databases over a network connection like SMB NFS etc.
See this page for info on how to fix a corrupt database.
