Confused by September Payout

[kbch]

I had about 1TB TX Traffic but just received 15STJ/~7$. So I think something is wrong.

If I now check Info & Estimation in the Dashboard it shows me just 154GB Download Traffic for September.

When checking with earnings.py I get different results

September 2020 (Version: 9.4.0)
			TYPE		PRICE			DISK	   BANDWIDTH		PAYOUT
Upload			Ingress		-not paid-			    28.02 GB
Upload Repair		Ingress		-not paid-			    29.00 GB
Download		Egress		20   USD / TB			   863.54 GB	     17.27 USD
Download Repair		Egress		10   USD / TB			   207.44 GB	      2.07 USD
Download Audit		Egress		10   USD / TB			     4.52 MB	      0.00 USD
Disk Average Month	Storage		1.50 USD / TBm	     2.99 TBm			      4.48 USD
Disk Usage		Storage		-not paid-	     2.15 PBh
_______________________________________________________________________________________________________+
Total							     2.99 TBm	     1.13 TB	     23.83 USD

Payout and held amount by satellite:
SATELLITE	JOINED	   MONTH HELD%	  HELD TOTAL	      EARNED	        HELD	        PAID
us-central-1	2019-06-11    16    0%	    0.76 USD	  1.2611 USD	  0.0000 USD	  1.2611 USD
    HELD AMOUNT RETURNED		   -0.38 USD					  0.3813 USD
    AFTER RETURN			    0.38 USD	  1.2611 USD	  0.0000 USD	  1.6423 USD
europe-west-1	2019-06-10    16    0%	    0.76 USD	  2.4103 USD	  0.0000 USD	  2.4103 USD
    HELD AMOUNT RETURNED		   -0.38 USD					  0.3806 USD
    AFTER RETURN			    0.38 USD	  2.4103 USD	  0.0000 USD	  2.7908 USD
europe-north-1	2020-04-18     6   50%	    4.96 USD	  2.5739 USD	  1.2869 USD	  1.2869 USD

asia-east-1	2019-06-10    16    0%	    0.45 USD	  1.2980 USD	  0.0000 USD	  1.2980 USD
    HELD AMOUNT RETURNED		   -0.23 USD					  0.2275 USD
    AFTER RETURN			    0.23 USD	  1.2980 USD	  0.0000 USD	  1.5254 USD
saltlake	2020-02-11     8   25%	   31.06 USD	 16.2822 USD	  4.0706 USD	 12.2117 USD

stefan-benten	2019-06-10    16    0%	  133.72 USD	  0.0036 USD	  0.0000 USD	  0.0036 USD
    HELD AMOUNT RETURNED		 -133.72 USD					133.7153 USD
    AFTER RETURN			    0.00 USD	  0.0036 USD	  0.0000 USD	133.7189 USD
    PAYOUT NOTES: Graceful Exit
_____________________________________________________________________________________________________+
TOTAL					  171.71 USD	 23.8289 USD	  5.3575 USD	 18.4714 USD
    HELD AMOUNT RETURNED		 -134.70 USD					134.7046 USD
    AFTER RETURN			   37.01 USD	 23.8289 USD	  5.3575 USD	153.1761 USD

Node id: 12a5Q1RaPxYqSyVFiFgmaHnswDWU3qVZ3gBJFUGdDuRAuCHhrqf

[Alexey]

This is wallet, not a NodeID from the dashboard. Also, you cut the held amount from both reports.
Make sure that you checking the same node in the Earnings calculator.
The node with specified wallet has submitted orders on $7.80, the $1.77 is held, $0.99 of total held amount is returned back, so the payout is $7.01

[kbch]

Now all Information is in the post.

[Alexey]

Did you have a downtime in the September?
Do you have logs for September?
If so, please, search for errors with orders.

[kbch]

I have of these, what can I do against them? Are they responsible for this? (Downtime of the Node during September was 10 Minutes)

2020-10-07T21:47:17.623Z ERROR orders listing orders {"error": "order: proto: pb.Order: illegal tag 0 (wire type 0)", "errorVerbose": "order: proto: pb.Order: illegal tag 0 (wire type 0)\n\tstorj.io/storj/storagenode/orders.readOrder:593\n\tstorj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite.func1:257\n\tpath/filepath.walk:360\n\tpath/filepath.walk:384\n\tpath/filepath.Walk:406\n\tstorj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite:201\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrdersFromFileStore:398\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders:192\n\tstorj.io/storj/storagenode/orders.(*Service).Run.func1:139\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

[nerdatwork]

@kbch

“order: proto: pb.Order: illegal tag 0 (wire type 0)”

Error: "order: proto: pb.OrderLimit: illegal tag 0 (wire type 0)

ERROR orders listing orders {“error”: “order: proto: pb.OrderLimit: illegal tag 0 (wire type 0)”, “errorVerbose”: “order: proto: pb.OrderLimit: illegal tag 0 (wire type 0)\n\tstorj.io/storj/storagenode/orders.readLimit:526\n\tstorj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite.func1:238\n\tpath/filepath.walk:360\n\tpath/filepath.walk:384\n\tpath/filepath.Walk:406\n\tstorj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite:196\n\tstorj.io/storj/storagenode/orders.(*S…

[Alexey]

Unfortunately any unsent order = lost payment for it.
The satellite accounting only settled orders.

[kbch]

I will now move unsent orders piece by piece in the folder so it can upload them till I get to the broken one (~5k unsent orders ATM).

How can I prevent this in the future?

[BrightSilence]

I think a fix was implemented for the unexpected EOF, but it seems there are other errors that can arise. @Alexey, do you know if there is a ticket yet to expand this fix to any error that might happen so that it skips at most one order file if an error occurs? It seems there have been several other errors reported now related to these files and people only find out during payouts.

Though the downside is obviously that the errors won’t be found at all if the faulty files are just skipped. Kind of a tough one.

[kevink]

@BrightSilence

Though the downside is obviously that the errors won’t be found at all if the faulty files are just skipped. Kind of a tough one.

We’d need a dashboard notification for such errors. but of course for that to work storj would need to get their log levels straight first.

[Alexey]

The fix was to do not fail the node on those errors and just report and skip.

[Alexey]

This is not needed unless your node stop to working.
The prevention of file corruption should be the same as for regular file errors.
I would like to suggest you to check this disk for errors when all services which could use it are stopped.

[BrightSilence]

Then it looks like either people still reporting this issue are still on older version or it only reports and skips the specific EOF error. Because the situation described sounds exactly the same to me, it’s just triggered by a different error.

@kbch what node version are you on?

[Alexey]

The difference is that the node crashed due to a FATAL ERROR and stopped working, but now it should throw an ERROR and continue.

[BrightSilence]

Are we talking about the same thing?
I was referring to this error:

Order: unexpected EOF

i brought this home today and there is a persistent error, node version 1.11.1 windows 10 pro v 2004,2020-09-10T02:06:45.623Z ERROR orders listing orders {“error”: “order: unexpected EOF”, “errorVerbose”: “order: unexpected EOF\n\tstorj.io/storj/storagenode/orders.readOrder:561\n\tstorj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite.func1:245\n\tpath/filepath.walk:360\n\tpath/filepath.walk:384\n\tpath/filepath.Walk:406\n\tstorj.io/storj/storagenode/orders.(*FileStore).…

That was never a FATAL error. The node kept running, it was just the sending of orders that was interrupted. The fix changed that behavior to skip faulty files and send all other orders anyway. However, what’s posted in this topic seems to lead to the node behavior prior to the fix, where no orders are sent, which leads me to believe the implemented fix was narrowly targeted at the specific unexpected EOF error.

[kbch]

I’m running v1.13.3 which seems the latest Version on Dockerhub.

I think the big issue with this fault is, that the Node is still working like normal. It generates traffic like normal. But you don’t get paid as it can’t upload the orders which you will just notice on payout. So if you get an issue with your orders on 2. of a Month you will notice it maybe on 7. of next month (But that should be fixed with the bugfix they did to skip broken orders…)

[BrightSilence]

v1.13.3 should already include the fix mentioned in the topic I linked to. So I think that fix isn’t sufficient for your scenario.

[kbch]

You are right. I think we are mixing up two different things here. I got the error mentioned Confused by September Payout here which is different to the EOF error!

[Alexey]

Seems not
I’ll search for the current bug and will create a new one, if nothing find.

Update: the bug has been submitted.