Edit: Upon further investigation we’ve noticed this issue is causing deleted pieces to still be audited and scores are dropping on ap1 as a result. We’ve seen scores as low as 80% already (as of 2021-07-23T10:00:00Z (UTC)). Nodes may be at risk if this persists. Two of my nodes suddenly started get…

We haven’t found the root cause yet, but I’ll share what we have: • on AP1 we started deletion of the bucket with massive number of segments • during that time we find out that audit failure rate it going up for AP1 very fast • after a research we discovered that pieces that are causing failures …

duno if this is related… but noticed it and the timing of it seems to correlate pretty closely… ofc the graphs in question isn’t very useful since they have no reference numbers to gauge the drop with… most of the egress and ingress graphs from the satellites seems to drop dead around the last sto…

It’s not related. The bad statistics were mentioned elsewhere and they’ve made some corrections.

One small update on this issue. I’m now seeing errors for repairs on these pieces as well. Here’s an example. 2021-07-22T20:08:41.735Z INFO piecedeleter delete piece sent to trash {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Piece ID": "BPBMRLTWVDS325O6…

Yep, same here. BTW, a question to @Storjlings, would it make sense to temporarily suspend, or significantly reduce the rate of audits on the affected satellite, so that node operators affected by false failed audits can sleep well? I hope this is a tunable satellite parameter…

I will sleep just fine after reading this message. I wouldn’t worry about it. I know from experiences during the beta that Storj Labs will do the right thing in these scenarios. [image] heunland: We already have the node IDs affected by this issue so in the event that any of them should get dis…

this is unlikely to be related to the issue on AP1, if you are seeing similar errors on EU1, please post your log on a new thread, lets keep this one focused on the issue on AP1 thanks.

Thanks for pointing this out - devs are already aware of this, but this will not affect node reputation. They are already working on applying the fix so please stand by.

12 hours without an audit failure on any satellite… woop woop seems like the fix is working, a bit to close for comfort, ofc thats easy to say…

I have this situation by today morning: [ap1 audit] I was looking into the storagenode.log file and found quite many of these kind of lines, 1 or 2 almost in every minute: 2021-07-24T12:22:23.248+0200 ERROR piecestore download failed {Piece ID: ZGW2ZEQM44G2PGUU2LINYHVI5HKUIMPSZD4OJNVL3MRVFSF…

Spoke to soon… the problem persists. during the last 4 hours i got another 2 failures on trying to track them down…

Storj Community Forum (official)

Audit scores dropping on ap1; Storj is working on a fix, SNO's don't need to do anything. ERROR piecedeleter could not send delete piece to trash >> Pieces error: v0pieceinfodb: sql: no rows in result set

Node Operators troubleshooting