Debugging space usage discrepancies

Shouldn’t hex representation of satellite id be same for every node ?

004AE89E970E703DF42BA4AB1416A3B30B7E1D8E14AA0E558F7EE26800000000
004AE89E970E703DF42BA4AB1416A3B30B7E1D8E14AA0E558F7EE26800000000
004AE89E970E703DF42BA4AB1416A3B30B7E1D8E14AA0E558F7EE26800000000
004AE89E970E703DF42BA4AB1416A3B30B7E1D8E14AA0E558F7EE26800000000
004AE89E970E703DF42BA4AB1416A3B30B7E1D8E14AA0E558F7EE26800000000
004AE89E970E703DF42BA4AB1416A3B30B7E1D8E14AA0E558F7EE26800000000

:point_up: This is hex(satellite_id) column from storage_usage.db

Yes, they are the same, but in my special example environment there was only one satellite (which was easier to query), and your database should have 4 (eu, us, ap, saltlake). Just don’t forget to check all of them.

2 Likes

That is effectively almost all production nodes.

@NotPaidForBugReport and not paid for bugs also?
seems fair for me :wink:

Is it too much to ask to make your post using professional language and in a civilized manner ?

2 Likes

I regret to have to answer in the negative. In the old days, when I believed in the project and actively developed as an operator, I wrote very tactfully and culturally. However, as the years passed, I observed a complete disregard for adequate comments and indications of errors. For this reason, I started writing the way I am writing now.

2 Likes

Some of the reports were not rewarded because the program was quickly closed after launch. I can even say that I am partially satisfied with the results of that work.

I apologise for the wrong assumption. :flushed:

I thought you make no bug reports, because you are not paid for.

To my defense: Without that knowledge (Quote), your username is verry missunderstandable.

2 Likes

Hi @NotPaidForBugReport

2 things:

1: I can’t allow this kind of language to remain on the Forum

Would you like to edit it to appropriate language for a company forum, or should I do that for you?

2: Your name is causing controversy again because it is misleading.
Would you consider changing your name to something more neutral?

Storj does in fact pay for bug reports. When I asked you ages ago if you had an outstanding bounty for a security report, based solely on your name, you told me there was no issue for me to resolve.

But now community members are worried because of your name.

If anyone has a Bug Report to submit please send it to security-reports@storj.io
I will help you through the process from there. Storj has a solid history of paying for valid Bug Reports. If anyone has Bug Bounty related issues please email security-reports@storj.io and I will help you resolve them.

4 Likes

Storj absolutely pays out all awarded Bug Bounties.

3 Likes

That is not what you told me when you were approached on the subject because of your name.

The Storj Bug Bounty Program is very much active and available for anyone to submit their findings.

Please do not spread misinformation.

3 Likes

Hello.

I created a forum post in this thread about 18 hours ago, but it was deleted without any notification to me of the reason. I would still like my post to be published, it will be interesting and useful to everyone.

Thanks.

Hi @AiS1972 ,

Can you please DM me with a little more to go on so I can investigate?
Maybe it was autoflagged by the system.

1 Like

Hello bre , elec and other gentlemen!

I am re-writing my post and hope that if it is deleted, it will be done openly and indicating the reason for the deletion.

My node has 45.4 million pieces in the US1 satellite catalog
NodeID: 12epLYjk********
joined 2019-05-31

Currently our largest node has 26M pieces, so bumping the bloom size to 4MiB will probably help…

so… 26 million and 45.4 million.
I already raised the topic about where our money is Lebowski here - Mismatch of proportions, or where is our money, Lebowski? - #16 by elek

As I understand it, it turned out to be an old parable - the path to hell is paved with good intentions. They wanted to save satellite resources - they received unpaid space for operators.

After implementing deletion using the bloom filter, there were discrepancies with the space that these nodes occupy on the disk and the space for which node operators are paid.
Previously, while removing each piece was a separate operation, there were no discrepancies. And the older the node and the more pieces the satellite directory contains, the more pieces that are on the operator’s disk, but there is no payment for this. The most problematic satellite is US1, which has the maximum number of pieces.

The fact that a mistake occurred is normal, it doesn’t happen to anyone. But I propose not to hush it up and delete posts, but to solve it together by holding hands together.

I think it would be nice to compensate operators for all these existing problems in the software, say by 25%

4 Likes

Hi bre!

I published the post again, but the previous one was visible after publication, and in the morning I no longer saw it.

Thanks
Bye

2 Likes

This makes sense, for my one node that I’m investigating my ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa folder has 31M files.

The same thing happened when they decided to implement the 7 day trash retention, they some times don’t bother to send the bloom filter for weeks and when finally arrives those already stale files stays for seven more days on the disks. unpaid!!

This is unacceptable. I will be sending the trash to /dev/null from now on. I don’t care if i get disqualified anymore.

Even though I’m a SNO and I have 100 TB of data, and of course big discrepancies on all my 9 nodes, I don’t want Storj to pay for data that is suppose to be deleted and isn’t payed by customers. At first I thought there was a bug and the occupied space was wrongly accounted by satellites and we are somehow cheated, but it turns out that is just trash data wrongly manipulated. We are all here in uncharted territory and of course unpredictible will happen. I believe nobody has worked with so many files on a single HDD, no RAID, and we just discover how to manage 10-20+ TB of data for the purpose of a storagenode. We even try to reinvent filesystems addapted for this unique case. So, let’s give them a break because nobody tries to cheat anybody, and Storj team is the most interested in making things work better and fair for everyone. Anyone of us can leave tomorrow and don’t realy care to much, but they run a business and a unique project, so they need to make it work better and better.
With so many different setups and so many problems at the SNO’s end because of unreliable or underpowered hardware, of course some bugs are misstaken with operator problems. It’s always best to report what we see strange and try to investigate, and report again. This forum and the admins are top notch. In the end all problems get solved.

10 Likes

Neither do I, but the way I see it, and precisely because this is a business both ways, they should put the same effort to make sure there is no a unpaid file in SNO’s disk as they put in not charging the customer for files they don’t hold.

But time after time it seems that they discover these problems only because we complain. Are they incapable to run some nodes, monitor them, and discover them preemptively?. see for example
this. that is an example of a problem not related to inadequate hardware.

yes of course, but then, the first answer most get is to blame the SNO’s installation.

2 Likes

We do bring issues to the attention of the devs. There is also GitHub where anyone can post an issue and get a response from the engineers directly on a resolution. We appreciate your understanding as the team balances priorities.

3 Likes