I see. This does make sense, though looking at this problem from a SNO perspective I wonder whether we will have another file walker-like process this way. Just like the retain request, a single API call to a node may trigger a large number of random I/O stat calls.
We know GC runs roughly every 5 days. Are there any plans on how often this call will be used and how many pieces will be checked in a single call?
This is just an intermittent burst though. Itâs over quite fast and most of the time your node is not dealing with it. So far it looks nothing like the heavy load of GC.
Iâm observing these bursts of a single-byte audits for some time now. Right now theyâre probably limited by the latency of making a separate request for each such file, but a single API call may ask for many of them.
To be fair, my SSD cache had just been installed and the file walker was running on all my nodes at the time. Since the cache was still new, this first run didnât benefit much from it yet. Then again normally audits give us minutes to finish, that clearly wasnât the case here.
I have seen lots of audits at the end of the last year, (up to 2.5 audits/s), but no audits right now. I guess the same verify process has been running before and is already done with my node.
Is this going to make vetting faster? I havenât seen any new audits on my 2 unvetted nodes.
This probably wonât, but the changes to audits in general will. Donât think that part is live yet though.
Thanks for letting me know. Hopefully this speeds up the vetting processâŚ
@brandon I still see no audits⌠should it be so? Will they be there again with 1.70.2 or is this version update not related?
My node running 1.70.2 had no audits the last 3 days (15th-17th Jan). Audit attempts have never previously been affected by the node version. It looks like they are still working on the new scaled audit system.
Hey @binary, we have not enabled audits again. We expect to do so soon; we are just wrapping up a few final checks. Once we enable them again, we will post on this thread so that you and the rest of the community are aware.
Hey Everyone; we enabled audits on the US2 satellite around 18:00 UTC. We will monitor to ensure everything is going as expected before enabling it on other satellites. Please let us know if you notice anything âweirdâ so we can investigate.
My node is getting vetted, but havenât had any audits from US2. Is this normal?
lgtm (20 char) . . . . .
It might be, us2 is storing a small amount of data and your node may not be holding any pieces from it.
Yeah, that makes sense. Vetting still takes soo long lol
I see now 3 satellites are with audits enabled. Any news when would it be enabled on the rest?
[root@localhost ~]# tail -1000000 /storj/node.log | grep AUDIT | grep -Po "Satellite ID\": \S+" | sort | uniq -c | sort -k1nr
62 Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo",
58 Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB",
12 Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE",
[root@localhost ~]#
Makes sense that they would test Saltlake and Europe-North for some higher load testing before enabling on customer facing satellites. Iâm guessing it wonât be enabled for other sats on a Friday. Probably not a good idea to go into the weekend with such a change just live.
In 25 hours, my 3 months old node with 1TB data shows:
-eu-north: +3 audits
-saltlake: +0 audits
-us2: +3 audits
The data sats are all vetted.
