i brought this home today and there is a persistent error, node version 1.11.1 windows 10 pro v 2004,2020-09-10T02:06:45.623Z ERROR orders listing orders {“error”: “order: unexpected EOF”, “errorVerbose”: “order: unexpected EOF\n\tstorj.io/storj/storagenode/orders.readOrder:561\n\tstorj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite.func1:245\n\tpath/filepath.walk:360\n\tpath/filepath.walk:384\n\tpath/filepath.Walk:406\n\tstorj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite:196\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrdersFromFileStore:398\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders:192\n\tstorj.io/storj/storagenode/orders.(*Service).Run.func1:139\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
same error here and got suspended in two sattelites
but I don’t see a satellite listed on my error
2020-09-10T11:53:51.382-0300 ERROR orders listing orders {"error": "order: unexpected EOF", "errorVerbose": "order: unexpected EOF\n\tstorj.io/storj/storagenode/orders.readOrder:555\n\tstorj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite.func1:245\n\tpath/filepath.walk:360\n\tpath/filepath.walk:384\n\tpath/filepath.Walk:406\n\tstorj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite:196\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrdersFromFileStore:398\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders:192\n\tstorj.io/storj/storagenode/orders.(*Service).Run.func1:139\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
In the Dashboard I appear with suspension in 2 nodes and this type of error is repeated several times
in my case this error appears in the logs about every minute or so, this node at remote location and I brought it home yesterday and that is when I began to follow the logs and noticed it, I hope one of the storjlings will check this thread if not I will submit a support ticket.
Have the same error on one of my nodes since last night. No new files in the archived folder since then and files are piling up in the unsent folder. But this node is not suspended on any satellite. Your suspension must have another cause.
Edit: Just noticed that it happens also on another node, but on this one for more than 2 days now. Both nodes had problems in the last few days and I had to kill the docker process. The hard disks completely locked up. Couldn’t read nor write to these disks and had to reboot the server. I suspect a controller problem.
If you want to get attention for a new issue it really doesn’t help to post it on a very old unrelated topic.
Unexpected EOF (end of file) for orders suggests to me that one of the unsent order files got corrupted.
I don’t really know of a good way to find out which one. But you could start by looking if one is significantly smaller than the others.
You can likely fix it by simply removing (or probably better moving to a backup location) the corrupt file. If all else fails you can get rid of all the unsent files and it probably starts working again.
However, probably best to wait for confirmation from a Storjling on this approach. I don’t know if messing with these files could break things. At the least you won’t be paid for any orders that weren’t sent.
Edit: Additional note. It seems likely that the file causing the problem is one of the oldest files in the folder. So if you start removing/moving files you probably want to start with the oldest ones first. But again, don’t do anything until someone from Storj has given you a go on this approach.
I just had the same thought of moving the files one by one out that folder starting with the oldest. I suspect the sending order process terminates at the first error and doesn’t process the other files. But I rather wait for some advice from one of the Storjlings. Maybe @Alexey can split this topic and change the title accordingly.
SUCCESS!
I couldn’t wait any longer and tried it out. I stopped one of the nodes, moved the oldest file out of this folder and started the node. Lo and behold after the third file all of the remaining orders were sent. 2 of the 3 files were corrupt. I’ll keep them for a few more days in case one of the Storjlings advises how to fix these files.
As far as I can tell, this is the most likely scenario - an orders file became corrupted somehow and the order cannot be read. As a quick fix it should be easy to temporarily move the corrupted file outside of the unsent orders directory, assuming you can figure out which one is corrupted.Also, if people with this issue could post what OS their node is running on, it could help us in the process of making the system more robust.
I am looking into a fix for the error being discussed here, but you should know that you getting suspended is likely unrelated - this particular error should only affect the sending of orders to the satellite, not audits. Unless you are for some reason returning this error as a response to audits, in which case I am very curious and would love to see your logs.
In my case, Ubuntu 16.04, Docker version 19.03.12, build 48a66213fe
@donald.m.motsinger, in what folder would that file be located, win. 10 pro v.2004 gui
In the "C:\Program Files\Storj\Storage Node\orders"
i have a file called unsent and it has a lot of unsent orders from 3 different satellites do i remove the all
read my post above
@donald.m.motsinger I have dozens of unsent orders from three satellites each was that your case
Start moving them out of that folder one by one until the orders get sent again. I stopped my node each time.
well it appears to have worked I’m watching the logs to see if the EOF error re-appears again, while you used a surgical process to eliminate your error I gave up on the surgical after removing five five files from each of the satellites without having the expected results and chose a sledge hammer
If you still have those files somewhere, you might want to consider putting them back one by one so that any orders in non-corrupted files can still be sent in. At least, if you consider that to be worth your time/energy.
FYI @ everyone here, the fix for this has been merged, and should be in the next release. Thank you for all the helpful information