Order file already closed

Node Operators troubleshooting
1 
2024-05-22T07:36:38Z ERROR piecestore failed to add order       { "error": "order: close config/orders/unsent/unsent-orders-1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE-1716361200000000000.v1: file already closed", "errorVerbose": "order: close config/orders/unsent/unsent-orders-1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE-1716361200000000000.v1: file already closed\n\tstorj.io/storj/storagenode/orders.(*FileStore).getWritableUnsent:151\n\tstorj.io/storj/storagenode/orders.(*FileStore).BeginEnqueue.func1:128\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:906\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func8.1:784\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func8:822\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:839\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:167\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:109\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:157\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}

File is present. Node is at v1.104.5 The line keeps on repeating for a long time. How to handle this ?

2

Its been 3 hours since the above post and I still see the same error repeating over and over …

2024-05-22T10:27:11Z INFO piecestore uploaded                   { "Piece ID": "X6GI6QDS4FKFYPBSBJZF3732TJ3P75NBV3RYM6UM33SSBXOUCJVQ", "Satellite ID": US1 "Action": "PUT", "Remote Address": "172.17.0.1:43626", "Size": 8704}
2024-05-22T10:27:12Z INFO piecestore upload started             { "Piece ID": "ZAMGCDYWWIOVD6JU7CNRHQYSY3VQCBVHZB7WKBOAX3GZCTRREIFA", "Satellite ID": US1 "Action": "PUT", "Remote Address": "172.17.0.1:43986", "Available Space": 1287.433 GB }
2024-05-22T10:27:12Z ERROR piecestore failed to add order       { "error": "order: close config/orders/unsent/unsent-orders-1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE-1716361200000000000.v1: file already closed", "errorVerbose": "order: close config/orders/unsent/unsent-orders-1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE-1716361200000000000.v1: file already closed\n\tstorj.io/storj/storagenode/orders.(*FileStore).getWritableUnsent:151\n\tstorj.io/storj/storagenode/orders.(*FileStore).BeginEnqueue.func1:128\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:906\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:538\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:167\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:109\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:157\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-05-22T10:27:12Z ERROR piecestore upload failed     { "Piece ID": "S7WLNDWHZLC3L3XMWK54WTGYSCKRL5VT46SRENMRILOROJCRA6PA", "Satellite ID": SALTLAKE "Action": "PUT", "Remote Address": "172.17.0.1:46894", "Size": 917504, "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:229"}
2024-05-22T10:27:12Z ERROR piecestore failed to add order       { "error": "order: close config/orders/unsent/unsent-orders-1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE-1716361200000000000.v1: file already closed", "errorVerbose": "order: close config/orders/unsent/unsent-orders-1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE-1716361200000000000.v1: file already closed\n\tstorj.io/storj/storagenode/orders.(*FileStore).getWritableUnsent:151\n\tstorj.io/storj/storagenode/orders.(*FileStore).BeginEnqueue.func1:128\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:906\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:520\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:167\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:109\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:157\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-05-22T10:27:12Z INFO piecestore uploaded                   { "Piece ID": "ZAMGCDYWWIOVD6JU7CNRHQYSY3VQCBVHZB7WKBOAX3GZCTRREIFA", "Satellite ID": US1 "Action": "PUT", "Remote Address": "172.17.0.1:43986", "Size": 16384}
2024-05-22T10:27:13Z INFO piecestore upload started             { "Piece ID": "GR4ANK3SXDCN4GL5ICAF6YJVANAPLRDIMI6IQWXZXHTUXRIFSSUQ", "Satellite ID": US1 "Action": "PUT", "Remote Address": "172.17.0.1:46276", "Available Space": 1287.433 GB }
2024-05-22T10:27:13Z ERROR piecestore failed to add order       { "error": "order: close config/orders/unsent/unsent-orders-1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE-1716361200000000000.v1: file already closed", "errorVerbose": "order: close config/orders/unsent/unsent-orders-1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE-1716361200000000000.v1: file already closed\n\tstorj.io/storj/storagenode/orders.(*FileStore).getWritableUnsent:151\n\tstorj.io/storj/storagenode/orders.(*FileStore).BeginEnqueue.func1:128\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:906\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:520\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:167\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:109\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:157\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-05-22T10:27:13Z INFO piecestore uploaded                   { "Piece ID": "GR4ANK3SXDCN4GL5ICAF6YJVANAPLRDIMI6IQWXZXHTUXRIFSSUQ", "Satellite ID": US1 "Action": "PUT", "Remote Address": "172.17.0.1:46276", "Size": 17408}
2024-05-22T10:27:13Z INFO piecestore upload started             { "Piece ID": "VNOOL36KYX7GOYDPR233HHVMYUFOU3MRCBJDVMXAY2TS4GQIREEQ", "Satellite ID": US1 "Action": "PUT", "Remote Address": "172.17.0.1:46920", "Available Space": 1287.433 GB }
2024-05-22T10:27:13Z ERROR piecestore failed to add order       { "error": "order: close config/orders/unsent/unsent-orders-1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE-1716361200000000000.v1: file already closed", "errorVerbose": "order: close config/orders/unsent/unsent-orders-1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE-1716361200000000000.v1: file already closed\n\tstorj.io/storj/storagenode/orders.(*FileStore).getWritableUnsent:151\n\tstorj.io/storj/storagenode/orders.(*FileStore).BeginEnqueue.func1:128\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:906\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:520\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:167\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:109\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:157\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}

The file is gone from unsent folder as of this post.

3

Thank you for alerting us.
I have forwarded this internally.

2 Likes
4

I believe that your disk is dying, accordingly your post

5

I think so too but once above erorr is seen then it doesn’t go away till node is restarted or crashed. In my case it crashed twice so far.