Additional Node Self audition system

My Idea, is that we need to port some self audition system inside node itself.

When data loaded, satelite will send to node audit hashes and node will make audit of its data all the time like round robin. Report to satellite. If node data curruption found then by self audition, then Satelite try to audit this area. Like this lost and curapted data, can be found very easy and much faster.

Also if data get curuted and node get DQ, not curupted data can be self audited and make GE for safe audited data.

This is not needed. The storage node has stored the signed hash on upload and could compare the data against that at any time.

This would mean cheaters don’t get punished. What is the point of DQ if the storage node is able to get away with it?

I not realy ment GE, just upload data back to network, and may be 10-20% of held amount back. so cheater will loose lot. But Network can save lot of money on unnesesary data recunstruction.

So this way audition is much cheaper for the system, than satelite audition of data, by combyning this types of audition Satellite will know faster about data curruption or data lost and spend less own resorses, as was mentiond by storj team than Sattelite audition is very expencive.

What’s the incentive for the storagenode in reporting issues earlier though? I might just want to rip that code out of my storagenode so it can survive a lot longer with these issues going unnoticed.

Nodes self reporting can’t be trusted. And if it can’t be trusted it also doesn’t provide reliable information to satellites. So the only feature this would add is allow SNOs to shoot themselves in the foot earlier than their nodes would otherwise get disqualified.

there is no diference get DQ fast or 1-2 day after. But this can save some part of held amount in cace of small errors, and system wont need to rebuild terabytes of data, so all will win in some part.

Who would spend days up to weeks to upload data for maybe 10-20% Held amount? A 10 TB node would need 23 days to upload its data at 5MB/s.
My node has 44$ held amount. I wouldn’t waste that much time for maybe 8$. I’d start over with a new node immediately.

when you will have 10TB full there will be some 100-200 in held ammount, my node uploded GE on 20-30 mbit

Current estimate according to BrightSilence’s estimator would be ~110$ held back amount.

That’s not the situation. Nodes can survive a very long time with just a small portion of data being corrupt or missing. They may never be disqualified simply because only a fraction of data is ever audited. It’s not a matter of now or after 1-2 days. But disqualified now or potentially never.

when i made mistake with my node with 2 TB data, i lost 10-50 mb of now data, DQ was same evning.

Simple solution don’t mess with your nodes data…

1 Like

I highly doubt those numbers. Losing that little data almost certainly would go unnoticed for a really long time.

Look for example at this post where the user hasn’t yet failed a single audit: Storagenode had corrupted data in over 400 blocks on the hard-drive storage

It would be nice to be able to scan and verify the data just so I can know for myself whether I should start a new node because my current one will be disqualified soon or whether I lost data after some crash etc. I do not see much use in reporting the failures to the satellite and getting disqualified earlier though.

1 Like

The system can’t trust a self audit. How should the satellite filter out the storage node that claim to have a file out of the nodes that are holding the file. In a world with self audits they will both report the same.

This wouldn’t work. The signed piece hash is stored inside the piece. You could only verify that the data is not corrupted but you wouldn’t notice missing data.

You could take the garbage collection bloom filter. It contains a piece count. The storage node keeps track of a piece count per satellite as well. You wouldn’t know which pieces you are missing but you would know how many pieces you are missing.