Corrupted order detected in orders file

[jammerdan]

    Corrupted order detected in orders file#011{"error": "ordersfile corrupt entry: ordersfile: 
    unexpected EOF"

I thought this error whould have been fixed by this version.

Also I am seeing error": "ordersfile: proto: can't skip unknown wire type 7"

How to fix?

[nerdatwork]

What is the version of your node ?

[jammerdan]

It is 1.15.3 and this posting must be at least 20 characters.

[nerdatwork]

Reporting Error - illegal tag 0 (wire type 0)

With the release v1.16.1, we expected that this issue is fixed. The release was 100% rolled out and the corresponding docker images published yesterday. Please if your node isn’t in that release, update it, and if it is and you’re seeing this error, report it again. Thank you so much for your collaboration.

[SGC]

since you have had issues with orders

you might want to check /storj/orders/archive and unsent
and check that you don’t have large amounts of rejected orders or orders that are over 48hours old…
since it can block your traffic.

so might be worth checking… when the other thing is fixed…

A lot of unsent orders

Stop the storagenode Remove all orders older than 48 hours (they will not be accepted because they expired already) Move all remaining orders to the other folder Move orders by batches back to the orders/unsent Restart the storagenode If orders are sent, repeat from p.4 if orders are stuck, move half to the other folder and repeat from p.5 Repeat p.7 until you will have a broken order - please remove it. Repeat from p.4 until all orders are sent or removed. Then please remove orders from the o…

[jammerdan]

I updated to v1.16.1 now and saw new logging about orders being processed. So it seems that this version is attempting to fix it. Will check if it has been resolved.

[nurvmokka]

Hi,
I’m running the v1.16.1 and I still have the errors. And the storagenode is keeping restarting… That start to happen since the update to that version

Here goes the logs:

2020-11-12T17:07:35.336Z WARN ordersfilestore Corrupted order detected in orders file {“error”: “ordersfile corrupt entry: unexpected EOF”, “errorVerbose”: "ordersfile corrupt entry: unexpected EOF\n\tstorj.io/storj/storagenode/orders/ordersfile.

[jammerdan]

Check if this helps: With version 1.16.1 2nd node not working

[snorkel]

I had some improper shutdown/s today, eventhough the NAS is set to shutdown after 5 min of UPS going without power and the batterys are 1,5 years old and have plenty of power. But, Cyberpower, you know…
The nodes are back online, but I see on each an order file corrupted. Here is the log for one:

WARN    ordersfilestore Corrupted order detected in orders file {"Process": "storagenode", "error": "ordersfile corrupt entry: invalid order size: 33004 is over the maximum 32768", "errorVerbose": "ordersfile corrupt entry: invalid order size: 33004 is over the maximum 32768\n\tstorj.io/storj/storagenode/orders/ordersfile.(*fileV1).ReadOne:192\n\tstorj.io/storj/storagenode/orders.(*FileStore).getUnsentInfoFromUnsentFile:303\n\tstorj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite.func1:257\n\tstorj.io/storj/storagenode/orders.walkFilenamesInPath:465\n\tstorj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite:233\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders:182\n\tstorj.io/storj/storagenode/orders.(*Service).Run.func1:131\n\tstorj.io/common/sync2.(*Cycle).Run:163\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93"}
WARN    ordersfilestore Corrupted order detected in orders file {"Process": "storagenode", "error": "ordersfile corrupt entry: ordersfile: checksum does not match", "errorVerbose": "ordersfile corrupt entry: ordersfile: checksum does not match\n\tstorj.io/storj/storagenode/orders/ordersfile.(*fileV1).ReadOne:215\n\tstorj.io/storj/storagenode/orders.(*FileStore).getUnsentInfoFromUnsentFile:303\n\tstorj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite.func1:257\n\tstorj.io/storj/storagenode/orders.walkFilenamesInPath:465\n\tstorj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite:233\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders:182\n\tstorj.io/storj/storagenode/orders.(*Service).Run.func1:131\n\tstorj.io/common/sync2.(*Cycle).Run:163\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93"}

What should I do? Is there a quick way to identify the corrupted file and remove it?

[arrogantrabbit]

@snorkel

What should I do?

You should buy a UPS. What you have is not an ups.

@snorkel

1,5 years old

Time to replace batteries. Cyberpower do not know how to charge them properly (they degrade fast), how to calibrate them properly (batter maybe still OK but they expect it to be perfect → FAIL) or how to source non-shit batteries in the first place.

I doubt their consumer units are actually made by cyberpower. Often server and serious equipment is made by an actual vendor and the consumer crap is white label, spiritually alibaba quality.

@snorkel

Is there a quick way to identify the corrupted file and remove it?

I would ignore it. Or delete obviously wrong sized orders. Or wait a day and delete all order files older than a day. Also check filesystem.

It’s almost impossible to configure proper shutdown on cyberpower. Their firmware is broken. On FreeBSD it’s even worse use to different shutdown behavior: when shutdown utility returns on Linux filesystem is already remounted readonly. Not so on FreeBSD

Screw that despicable company that plays Russian roulette with customer data. Stop giving them money.

[snorkel]

They are true synwave and way cheaper than APC. And they do their job for almost 5 years now. I have 13 of them.
I replace the original batteries after 3 years with Ultracells that last for 12 years.
In the one location, there are many power outages during the week. Something had got to happen, maybe hashstore was too busy from a startup…

[arrogantrabbit]

@snorkel

They are true synwave

Marketing bullshit. What is “false sine wave”? It equipment does not need sine wave. It rectifies it anyway, and then uses DC->DC converter. FCC require proper filtering, yes, but sine wave is nonsense in this context. In fact, if you properly filter… you get base frequency sine wave… soo…

It’s like writing on a can of juice: Does not contain polonium. Technically true, but just as useful.

@snorkel

way cheaper than APC

I wonder why

TBF, not all APC models are worth buying: cheap ones are also shit. SmartUPS are good.

[snorkel]

I have 2 dirs in orders: archive and unsent.
In which one should I delete older than 48h files?
In archive there are files 8 days old.
In unsent there are only files from today.

Edit: the corrupted ones seem to be removed and no errors in logs, after restart and dbs deletion.
Maybe DSM deleted them and repaired fs if there was a problem, or storagenode took care of them.
I don’t see any files of the specified sizes.

[Alexey]

Usually in unsent. As far as I remember, the node can deal with corrupted orders automatically.