Lots of unsent orders

Looking into this months payment I noticed that I didn’t get as much as I expected.
After reading some posts here I found some information that it might be related to unset order files.
Looking into one of my nodes there where over 7000 unsent orders and the filedates go back from sept until today.

I know there is a very similar thread, but the errors in that looked different.

So the question is:

  1. How do I fix this?
  2. Read somewhere that orders are not valid after 48 hours, do that mean that i’ve lost a lot because of this? (This node is quite large and is storing over 13TB of data).

Here is the log:

2020-11-04T03:37:10.111Z WARN ordersfilestore Corrupted order detected in orders file {"error": "ordersfile corrupt entry: ordersfile: unexpected EOF", "errorVerbose": "ordersfile corrupt entry: ordersfile: unexpected EOF\n\tstorj.io/storj/storagenode/orders/ordersfile.(*fileV0).ReadOne:97\n\tstorj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite.func1:237\n\tpath/filepath.walk:360\n\tpath/filepath.walk:384\n\tpath/filepath.Walk:406\n\tstorj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite:191\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrdersFromFileStore:393\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders:187\n\tstorj.io/storj/storagenode/orders.(*Service).Run.func1:134\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"} 2020-11-04T03:37:10.201Z ERROR orders listing orders {"error": "ordersfile: proto: pb.Order: illegal tag 0 (wire type 0)", "errorVerbose": "ordersfile: proto: pb.Order: illegal tag 0 (wire type 0)\n\tstorj.io/storj/storagenode/orders/ordersfile.(*fileV0).ReadOne:113\n\tstorj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite.func1:237\n\tpath/filepath.walk:360\n\tpath/filepath.walk:384\n\tpath/filepath.Walk:406\n\tstorj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite:191\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrdersFromFileStore:393\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders:187\n\tstorj.io/storj/storagenode/orders.(*Service).Run.func1:134\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"} 2020-11-04T04:37:00.818Z WARN ordersfilestore Corrupted order detected in orders file {"error": "ordersfile corrupt entry: ordersfile: unexpected EOF", "errorVerbose": "ordersfile corrupt entry: ordersfile: unexpected EOF\n\tstorj.io/storj/storagenode/orders/ordersfile.(*fileV0).ReadOne:97\n\tstorj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite.func1:237\n\tpath/filepath.walk:360\n\tpath/filepath.walk:384\n\tpath/filepath.Walk:406\n\tstorj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite:191\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrdersFromFileStore:393\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders:187\n\tstorj.io/storj/storagenode/orders.(*Service).Run.func1:134\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"} 2020-11-04T04:37:00.911Z ERROR orders listing orders {"error": "ordersfile: proto: pb.Order: illegal tag 0 (wire type 0)", "errorVerbose": "ordersfile: proto: pb.Order: illegal tag 0 (wire type 0)\n\tstorj.io/storj/storagenode/orders/ordersfile.(*fileV0).ReadOne:113\n\tstorj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite.func1:237\n\tpath/filepath.walk:360\n\tpath/filepath.walk:384\n\tpath/filepath.Walk:406\n\tstorj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite:191\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrdersFromFileStore:393\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders:187\n\tstorj.io/storj/storagenode/orders.(*Service).Run.func1:134\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

That’s quite a lot and will affect your payout.

Yes you will have lost payment for last 2 months since they date back to September.

Have a look at this thread.

Thank you.
I thought that I had my nodes in order, but I guess not.
I actually guessed by the error and due to the fact that the oldest files where a bit smaller that they where corrupted.
After removing the oldest files and restarted my node the node started sending loads of orders.
But I don’t understand the fact that one or two corrupted files can cause this. I think it should be handled in a better way so it tries more files.

Not really happy about this…

3 Likes