ERROR piecestore upload failed

[salmanparacha]

Seeing lots of errors like these since 2 days… Can someone please help? My node was running fine for the last 20 days with very few errors.

2021-09-30T12:16:16.525Z ERROR piecestore upload failed {“Piece ID”: “UKW5LXSIIP7NWNZDQ2KAVRL7T6QVEQ4TW7IDGTP3PR2JQZNHVH4A”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “unexpected EOF”, “errorVerbose”: “unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:82\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:342\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:209\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:102\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:95\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52”, “Size”: 0}
2021-09-30T12:35:27.191Z ERROR piecestore upload failed {“Piece ID”: “ADXOVD6TNK4562M7LWIS6JPAMFSOJQW3AONN4WIX6A54GERVLIIQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “unexpected EOF”, “errorVerbose”: “unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:82\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:342\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:209\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:102\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:95\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52”, “Size”: 0}
2021-09-30T13:57:38.104Z ERROR piecestore upload failed {“Piece ID”: “XD5QIQI2LOLKFPO33SXM5DYIUQ4DAJ5VGZZOIDU26S4RRUKHV26A”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “error”: “unexpected EOF”, “errorVerbose”: “unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:82\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:342\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:209\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:102\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:95\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52”, “Size”: 65536}
2021-09-30T14:20:54.671Z ERROR piecestore download failed {“Piece ID”: “BW5XDN6MJPW7W7GX2D5MPUQJM64AFZOFI4WP2YITCPPSCUX4SLIA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”, “error”: “context deadline exceeded”, “errorVerbose”: “context deadline exceeded\n\tstorj.io/common/rpc/rpcstatus.Wrap:73\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func6:656\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:674\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:217\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:102\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:95\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52”}

[Stob]

Hi @salmanparacha

You have both uploads and downloads failing, and timeouts being exceeded. It looks to be a disk issue, so please check the drive for free space and integrity.

The sudden and continuing nature of the recent errors could point to a failing disk.

[Luckymaster]

Hello Storj team!

I also have these errors. Not always. However, the drive is in a Qnap NAS. No RAID. And brand new. 2 of 10.41 TB used. I can’t easily do a database analysis there? If so, how is that supposed to work?
The latest QTS 5.1.0.2466 is installed.
Container Station version 3.0.4.533 (2023/07/19)
Storj Node VERSION v1.84.1

I’ve always made sure…that I use the command :
docker stop -t 300 storage node
use before restarting the NAS.

CPU Annapurna Labs Alpine AL314 Quad-core ARM Cortex-A15 CPU @ 1.70GHz

Total memory 8 GB (8 GB usable)

I ask for support.

023-08-09T11:44:30Z ERROR piecestore upload failed {“process”: “storagenode”, “Piece ID”: “3PJSBZ6HOQCPEQJA76SLDF5FL6BUSLDZO7VSXL3RLQDX76QUNA6Q”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “error”: “context canceled”, “errorVerbose”: “context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:75\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func6:500\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:531\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35”, “Size”: 2319360, “Remote Address”: “216.66.40.83:34852”}

2023-08-09T11:46:45Z ERROR piecestore upload failed {“process”: “storagenode”, “Piece ID”: “RJAW3JACI3TALVOKT246DX7GUVRDKMND5L5NTDHPTYONKO5B6ZIA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “context canceled”, “errorVerbose”: “context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:75\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func6:500\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35”, “Size”: 8704, “Remote Address”: “72.52.83.202:49890”}

2023-08-09T11:46:46Z ERROR piecestore upload failed {“process”: “storagenode”, “Piece ID”: “BM3GVX74ILC5XBLK5QJPDLQ7FVLYNPNUOBEWHKQ5L7AH3ATMIQIQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “error”: “context canceled”, “errorVerbose”: “context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:75\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func6:500\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35”, “Size”: 4864, “Remote Address”: “184.104.224.99:46228”}

2023-08-09T11:46:46Z ERROR piecestore upload failed {“process”: “storagenode”, “Piece ID”: “BM3GVX74ILC5XBLK5QJPDLQ7FVLYNPNUOBEWHKQ5L7AH3ATMIQIQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “error”: “context canceled”, “errorVerbose”: “context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:75\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func6:500\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35”, “Size”: 4864, “Remote Address”: “184.104.224.99:46228”}

2023-08-09T11:46:47Z ERROR piecestore upload failed {“process”: “storagenode”, “Piece ID”: “QGKCFZJ2KJDAGL66Q4WXXAETNHZW5FMCC5YKF7AK5SP3XT3FW45A”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “context canceled”, “errorVerbose”: “context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:75\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func6:500\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35”, “Size”: 36864, “Remote Address”: “5.161.128.79:36222”}

2023-08-09T11:46:49Z ERROR piecestore upload failed {“process”: “storagenode”, “Piece ID”: “72QJ7GVWIM5PBZFBCSDZM4KSTX24N3OTXONYIMOWDXEQHOAKIEZQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “error”: “context canceled”, “errorVerbose”: “context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:75\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func6:500\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35”, “Size”: 3584, “Remote Address”: “184.104.224.98:30228”}

[JWvdV]

It depends…

They’re is some essential information missing here:

1. what type of drive is in the system?
2. Do you (almost) only having this kind of lines in you logs? Or are there also lines with “uploaded”?
3. Does your node increase in size day by day, if so: how much?
4. What country are you living in?
5. What is your internet speed?

If 3 is >10GB/day, you shouldn’t worry at all.
If 3 is >5GB/day, you only should bothering if you’re living in western Europe or North-America with 100mbps internet connection.
Is 3 is below 5GB/day, you should 1) check for neighbors in your network, 2) check your internet speed, 3) find out the type of drive, 4) check the CPU-usage, 5) check on whether there are also “uploaded” lines in your log and 6) come back.

Bottom line: growth of your node is the aim, not a splendid error-less log.

[Alexey]

@Luckymaster

“Action”: “PUT”, “error”: “context canceled”

This is a usual error when the uplink “cuts a long tail” (your node is loosed the race), unless there is a FATAL error somewhere later in the log.

@Alexey

When the customer want to upload a file, their uplink requests 110 nodes for each segment of the file and start uploads in parallel, when the first 80 for each segment of the file are completed, all remained got cancelled. The same for downloads, but uplink will request 39 nodes contained pieces for each segment of the file and starts downloads in parallel, as soon as the first 29 for each segment of the file are completed, all remained got canceled. So the customer always uploads and downloads to/from the fastest nodes to their location.

[Vortal]

Looks like next release you will no longer have to remind users this is a “normal” error. @Mircoxi fixed up the logs and added a note about context canceled to the log. She is a real champion!

[Luckymaster]

Very interesting. However, I didn’t notice these many errors in the past. I will keep watching. Version 1.84.1 is still current for me. Sorry that I’m just now replying. The timing didn’t always fit Thanks for the support Ingress and Egrss will probably rise again.

[Luckymaster]

Hello! I was now able to find out that the error message: upload canceled (race lost or node shutdown) now appears much more often after I now show it much more often on 2 other nodes that previously did not or rarely displayed it.
I changed the size of the nodes. Now I have the salad.

In my configuration a Qnap (ARM 32bit) runs with Docker
Each of the 3 nodes is on its own hard drive (no RAID) and no more SMR technology (WD). Only CMR hard drives! 2 WD and an Ironwolf.

Node config:

docker run -d --restart=always --stop-timeout 300
-p 28969:28967/tcp
-p 28969:28967/udp
-p 14004:14002
-e WALLET=“xxx”
-e EMAIL=“xxx”
-e ADDRESS=“xxx”
-e STORAGE=“2.5TB”
–mount type=bind,source=“/share/Storj3/Identity/storagenode”,destination=/app/identity
–mount type=bind,source=“/share/Storj3/speicher”,destination=/app/config
–name storagenode3 storjlabs/storagenode:latest

Are there any errors in the configuration here? Is it the Qnap with a slow ARM processor that has 4 cores but is still 32bit?!?

[Alexey]

It’s usual “context canceled” error, your node was slower than others. You cannot be close to the everyone customer in the world, so your node cannot win all races, this is normal.