Thank you for this more detailed info! This is what I like to see!
1 Like
Alexey, thanks for putting together that synopsis. The issue and resolution above was a fairly complex sequence of events from alert to resolution. It’s unfortunate that node operators were temporarily impacted and reported the incident. We communicated regularly in the forum about the status and resolution.
We do our best to be as transparent as possible with the community, but we can only share what we know. This issue took time to diagnose, triage and ultimately resolve. We definitely appreciate the collaboration and responsiveness of the community on this issue, as well.
The real question/request seems to be that we should try to communicate earlier in the process when we experience a high severity issue. We’ll review our internal escalation process to identify whether there is an opportunity to improve how and when we share alerts and updates with the community.
With this particular incident, I think the team shared the information we had available as quickly as possible under the circumstances. We’re always interested in community feedback on how we can do better in the future, although I hope we don’t have another incident like this one.
13 Likes
Sorry @Alexey I haven’t been here for a few days.
I could still provide logs if needed.
No, it’s not needed, thank you.
Thank you for clarify and quick response too.
If I can suggest, should be good, when you spot some problems, or have some suspicious with impact to SNO’s, please, share it with us. I think, that our reactions will be more accurate :).
When the problem arose, no one expected it to affect anyone. There was nothing to report.
As soon as it started to affect the audit, an update was published.
3 Likes
A post was split to a new topic: Piecedeleter could not send delete piece to trash … “error”: “pieces error: v0pieceinfodb: sql: no rows in result set”