Audit failure for "invalid piece file for storage format version 1: too small for header (0 < 512)"

Storage Node Operators troubleshooting
#1

What could have caused this audit failure ?

ERROR   piecestore      download failed {"Piece ID": "IORMOMWSXIWXEEGXW4MPXIEOTPWLKMGMEEYLE5BCS7VTNC3JJNYQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT", "error": "pieces error: invalid piece file for storage format version 1: too small for header (0 < 512)", "errorVerbose": "pieces error: invalid piece file for storage format version 1: too small for header (0 < 512)\n\tstorj.io/storj/storagenode/pieces.NewReader:229\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:272\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:502\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1004\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:29\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
#2

Sounds like file corruption. Did you check the storage for file system errors?

#3

Yes, nothing so far. I strongly suspect power outage causing abrupt shutdown might be one reason. I would like to pinpoint the exact issue though.

#4

That might be tough. Did you check the file mentioned? I’m guessing it’s 0 bytes. Unsafe shutdown could have caused it I guess, but I’m not sure how you can ever be 100% sure.

#5

That’s the first thing I did, yes it is zero. I suspect during copy from temp to final location there was outage. Could anything else cause it though?

#6

In theory it could be hardware failure of the hdd or even memory. But I’d say that’s not very likely. Does the file creation time align with a power outage that you know of?

#7

File was created on 23rd August 2020. I have purged that old data from my memory :nerd_face:

1 Like
#8

Sure, you’ve purged it from your brain, but what about logs? You’d definitely see a restart around that time.

#9

Apparently system logs go as far as Nov 2020 so no luck there.

#10

That’s too bad. But for what it’s worth, if it took half a year for an audit failure to happen, there’s certainly not going to be extensive damage.

1 Like
#11

I am not OCD about it but owing to my name I like scores to be 100 not 99.99999999. Also such errors are educational.

Next on the to-do list is finding all files that are ZERO in size.

I would still appreciate any more valuable input anyone else may have.

3 Likes