Error during graceful exit?

Node Operators troubleshooting
1

Hello all,

I’m doing a GE on one of my nodes as I cannot maintain it anymore (I have other nodes, but this one in particular I have to GE it, and it was a small 500GB one).

It is also working for me as a testing experiment of how a GE works and how much time does it take, so it is a nice experiment.

So far it is working fine and pieces are transferred to other nodes as expected, but I have seen that after sending a batch of pieces to nodes, the satellite ask my node to delete the pieces from disk, but an error is returned:

2023-06-26T05:54:35.595Z	WARN	gracefulexit:service	failed to get piece size	{"process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "action": "delete all pieces", "Piece ID": "VWCYJTEOC4ZHXJOYKRZJK5JMUDE4L6QFZT2CB5LZMDIKO6I5J6AA", "error": "lstat storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/vw/cyjteoc4zhxjoykrzjk5jmude4l6qfzt2cb5lzmdiko6i5j6aa.sj1: no such file or directory"}

I’m suspecting that my node already deleted it once it is successfully sent to another node, and the second try to remove it is failing. That’s my guess because it is happening like:

  1. a lot of successful sendings to other nodes
  2. a lot of those errors

and they happen in clearly different batches, not in the same lines of logs, not mixed at all.

Is that the case?

2

Have you tried searching that piece in your log ?

3

I tried but couldn’t find it. I recreated the docker-compose of that node a few hours ago and logs got lost :frowning: I will check on the next set of errors when they appear.

4

I checked a bit deeper and that file that was tried to delete is from a satellite I already gracefully exited days ago and there is not even a folder for that satellite in the blobs folder :confused: