Uploads via S3 gateway; successful puts less than success threshold

Developer Discussions DCS
1

Just received my Tardigrade invite. Signed up on us-central. Got API key. Configured gateway_linux_amd64 according to docs. Attempting to upload using aws s3 sync a folder with 1758 files, total size 35MB.

Seeing these on the client:

2020/01/02 18:02:04 ERROR : myfile.txt: Failed to copy: s3 upload: 500 Internal Server Error: <?xml version="1.0" encoding="UTF-8"?>
    <Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message><Key></Key><BucketName></BucketName><Resource>/cherking/myfile.txt</Resource><RequestId>3L137</RequestId><HostId>3L137</HostId></Error>

And seeing this on the output of gateway:

    2020-01-02T17:58:20.906Z        ERROR   gateway error:  {"error": "segment error: ecclient error:
 successful puts (61) less than success threshold (80)", "errorVerbose": "segment error: ecclient error:
 successful puts (61) less than success threshold (80)\n\tstorj.io/storj/uplink/ecclient.
(*ecClient).Put:176\n\tstorj.io/storj/uplink/storage/segments.
(*segmentStore).Put:74\n\tstorj.io/storj/uplink/storage/streams.
(*streamStore).upload:257\n\tstorj.io/storj/uplink/storage/streams.
(*streamStore).Put:113\n\tstorj.io/storj/uplink/storage/streams.
(*shimStore).Put:50\n\tstorj.io/storj/uplink/stream.NewUpload.func1:52\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2

The error indicates that at the time you attempted this folder sync, there were not enough nodes available to assure that the file would have enough redundancy of pieces (upload threshold is currently 80 nodes). Have you tried this several times and you are still seeing this same issue? We already have a dev looking into the problem now. Perhaps trying to synch all 1758 files in the folder at the same time is a bit much, could you try with a folder that has a smaller number of files in it?

3

Please check the version of your gateway. I bet you are running an old version that is using grpc. The storage nodes will accept only 7 concurrent grpc uploads. drpc is unlimited and blocking way more than 7 connections.

Updating to the latest gateway version should fix it.

2 Likes
4

I downloaded gateway from here, about 30m before posting this:
https://github.com/storj/storj/releases/latest/download/gateway_linux_amd64.zip

5

I’m actually using rclone to sync this directory. Rclone uses S3 protocol and is limiting me to only 4 simultaneous uploads at a time.
I’m not sure if it is rclone or the gateway or tardigrade in general, but in the past 30m, I’ve only managed to upload 4.4MB of data. It’s giving me an estimate of 2.3hrs to upload 35MB. That’s not very good.

6

I have seen that our own developers download it but are still running a different version. Please check the output of the gateway binary you are executing. gateway version should print it out.

7 
# ./gateway_linux_amd64 version
2020-01-02T18:42:47.871Z	INFO	Configuration loaded from: /root/.local/share/storj/gateway/config.yaml
Release build
Version: v0.28.4
Build timestamp: 01 Jan 20 11:54 UTC
Git commit: 5c25d3d6d31dde11405aa28281d37c1e4338e1de
8

Great. Thank you for confirming it.

Now please run your gateway with log level debug. You can do that via config file or by adding --log.level debug
This should hopefully print out the error message the storage nodes are returning. I place my second bet on clock out of sync.

9

I’ve switched to the official aws cli, just to rule out anything rclone may . be doing. Getting even slower results. In debug logging of the . gateway, seeing these

2020-01-02T18:48:26.710Z	DEBUG	ecclient	Upload to storage node failed	{"Node ID": "143BFywMhDey8ShCU1pAhpVF4FQKe6fbamnSjPQhfjCMYHTd6o", "error": "protocol: expected piece hash; storage node overloaded", "errorVerbose": "group:\n--- protocol: expected piece hash\n\tstorj.io/storj/uplink/piecestore.(*Upload).Commit:229\n\tstorj.io/storj/uplink/piecestore.(*BufferedUpload).Commit:45\n\tstorj.io/storj/uplink/piecestore.(*LockingUpload).Commit:105\n\tstorj.io/storj/uplink/ecclient.(*ecClient).PutPiece.func3:236\n\tstorj.io/storj/uplink/ecclient.(*ecClient).PutPiece:266\n\tstorj.io/storj/uplink/ecclient.(*ecClient).Put.func1:112\n--- storage node overloaded"}


2020-01-02T18:48:29.778Z	DEBUG	ecclient	Failed dialing for putting piece to node	{"Piece ID": "ZMPYXKZQN4EPSIMKCPO4IJEILYVOOSAHJ5VDQW2YKVDOK2N5OSIQ", "Node ID": "12MzeTyyZKxFzdUAy5FaoBJGeXW5ucUmY3DugAncsZS1F1PLL64", "error": "piecestore: rpccompat: dial tcp: lookup stadsbygd.ddns.net: Try again", "errorVerbose": "piecestore: rpccompat: dial tcp: lookup stadsbygd.ddns.net: Try again\n\tstorj.io/storj/pkg/rpc.Dialer.dialTransport:54\n\tstorj.io/storj/pkg/rpc.Dialer.dial:31\n\tstorj.io/storj/pkg/rpc.Dialer.DialNode:124\n\tstorj.io/storj/uplink/piecestore.Dial:51\n\tstorj.io/storj/uplink/ecclient.(*ecClient).dialPiecestore:67\n\tstorj.io/storj/uplink/ecclient.(*ecClient).PutPiece:197\n\tstorj.io/storj/uplink/ecclient.(*ecClient).Put.func1:112"}

2020-01-02T18:49:29.810Z	DEBUG	ecclient	Failed dialing for putting piece to node	{"Piece ID": "UUIPODGMY6FFYPSK2C373TYLSW7XCOG6LCA2SQJNJPKCLOWXICGA", "Node ID": "12QFiGDkFVJ2eqFNpyxdyLatq2jgACezCXZtMQLd7QXTVUv5yhK", "error": "piecestore: rpccompat: dial tcp: lookup hurststorj.ddns.net: Try again", "errorVerbose": "piecestore: rpccompat: dial tcp: lookup hurststorj.ddns.net: Try again\n\tstorj.io/storj/pkg/rpc.Dialer.dialTransport:54\n\tstorj.io/storj/pkg/rpc.Dialer.dial:31\n\tstorj.io/storj/pkg/rpc.Dialer.DialNode:124\n\tstorj.io/storj/uplink/piecestore.Dial:51\n\tstorj.io/storj/uplink/ecclient.(*ecClient).dialPiecestore:67\n\tstorj.io/storj/uplink/ecclient.(*ecClient).PutPiece:197\n\tstorj.io/storj/uplink/ecclient.(*ecClient).Put.func1:112"}
10

2020-01-02T18:50:54.855Z DEBUG ecclient Upload to storage node failed {“Node ID”: “12mfQqhAPY5PjBMLf5JC8dBPzfVVyD7xEZ8vED49GXGsGarwu1U”, “error”: “protocol: expected piece hash; serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:77\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:319\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:268\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:1064\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”, “errorVerbose”: “group:\n— protocol: expected piece hash\n\tstorj.io/storj/uplink/piecestore.(*Upload).Commit:229\n\tstorj.io/storj/uplink/piecestore.(*BufferedUpload).Commit:45\n\tstorj.io/storj/uplink/piecestore.(*LockingUpload).Commit:105\n\tstorj.io/storj/uplink/ecclient.(*ecClient).PutPiece.func3:236\n\tstorj.io/storj/uplink/ecclient.(*ecClient).PutPiece:266\n\tstorj.io/storj/uplink/ecclient.(*ecClient).Put.func1:112\n— serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:77\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:319\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:268\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:1064\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}

11

Tried uploading a single large file 300MB. Failed. Various errors all throughout the gateway debug logs. Here are some:

2020-01-02T20:37:42.303Z ERROR gateway error: {“error”: “Storj Gateway error: pending upload “Upload3” missing”, “errorVerbose”: “Storj Gateway error: pending upload “Upload3” missing\n\tstorj.io/storj/pkg/miniogw.(*MultipartUploads).Remove:230\n\tstorj.io/storj/pkg/miniogw.(*gatewayLayer).AbortMultipartUpload:96\n\tstorj.io/storj/pkg/miniogw.(*layerLogging).AbortMultipartUpload:143\n\tgithub.com/minio/minio/cmd.objectAPIHandlers.AbortMultipartUploadHandler:1297\n\tnet/http.HandlerFunc.ServeHTTP:2007\n\tgithub.com/gorilla/mux.(*Router).ServeHTTP:212\n\tgithub.com/minio/minio/cmd.securityHeaderHandler.ServeHTTP:657\n\tgithub.com/minio/minio/cmd.rateLimit.ServeHTTP:642\n\tgithub.com/minio/minio/cmd.pathValidityHandler.ServeHTTP:602\n\tgithub.com/minio/minio/cmd.httpStatsHandler.ServeHTTP:541\n\tgithub.com/minio/minio/cmd.requestSizeLimitHandler.ServeHTTP:66\n\tgithub.com/minio/minio/cmd.requestHeaderSizeLimitHandler.ServeHTTP:91\n\tgithub.com/minio/minio/cmd.crossDomainPolicy.ServeHTTP:51\n\tgithub.com/minio/minio/cmd.redirectHandler.ServeHTTP:243\n\tgithub.com/minio/minio/cmd.minioReservedBucketHandler.ServeHTTP:301\n\tgithub.com/minio/minio/cmd.cacheControlHandler.ServeHTTP:270\n\tgithub.com/minio/minio/cmd.timeValidityHandler.ServeHTTP:371\n\tgithub.com/rs/cors.(*Cors).Handler.func1:200\n\tnet/http.HandlerFunc.ServeHTTP:2007\n\tgithub.com/minio/minio/cmd.resourceHandler.ServeHTTP:492\n\tgithub.com/minio/minio/cmd.authHandler.ServeHTTP:300\n\tgithub.com/minio/minio/cmd.reservedMetadataHandler.ServeHTTP:134\n\tgithub.com/minio/minio/cmd/http.(*Server).Start.func1:108\n\tnet/http.HandlerFunc.ServeHTTP:2007\n\tnet/http.serverHandler.ServeHTTP:2802\n\tnet/http.(*conn).serve:1890”}

2020-01-02T19:13:49.853Z DEBUG ecclient Upload to storage node failed {“Node ID”: “1T9dRgsaydcSXCg3R4FD77cPovnMXqkRVYEmxtnYKYy1VzDbGR”, “error”: “protocol: expected piece hash; context canceled”, “errorVerbose”: “group:\n— protocol: expected piece hash\n\tstorj.io/storj/uplink/piecestore.(*Upload).Commit:229\n\tstorj.io/storj/uplink/piecestore.(*BufferedUpload).Commit:45\n\tstorj.io/storj/uplink/piecestore.(*LockingUpload).Commit:105\n\tstorj.io/storj/uplink/ecclient.(*ecClient).PutPiece.func3:236\n\tstorj.io/storj/uplink/ecclient.(*ecClient).PutPiece:266\n\tstorj.io/storj/uplink/ecclient.(*ecClient).Put.func1:112\n— context canceled”}

2020-01-02T19:07:21.786Z DEBUG ecclient Upload to storage node failed {“Node ID”: “12mcDKykUvpULJf7ApAx8UNM9mKE4Uz3z59jVoe4bRymxEDkfYn”, “error”: “piecestore: rpccompat: context canceled”, “errorVerbose”: “piecestore: rpccompat: context canceled\n\tstorj.io/storj/pkg/rpc.Dialer.dialTransport:74\n\tstorj.io/storj/pkg/rpc.Dialer.dial:31\n\tstorj.io/storj/pkg/rpc.Dialer.DialNode:124\n\tstorj.io/storj/uplink/piecestore.Dial:51\n\tstorj.io/storj/uplink/ecclient.(*ecClient).dialPiecestore:67\n\tstorj.io/storj/uplink/ecclient.(*ecClient).PutPiece:197\n\tstorj.io/storj/uplink/ecclient.(*ecClient).Put.func1:112”}

12

@utdrmac are you still having this issue?

13

How ist the sync command looks like?

14

Ok got it

aws s3 --endpoint=http://localhost:7777/ sync directory s3://bucket-name

15

Have not tried since originally posting. Was hoping to get some replies/answers from dev team before trying again.

16

Ok let me boost the signal on it ; it is the weekend so it may take a little while to hear anything, but hang in there :slight_smile:

17

Failed DNS resolutions like this make me think there’s some larger issue. I’ll put a ticket into the system to try to categorize what different errors we can get from these Dynamic DNS providers under various circumstance.

dial tcp: lookup hurststorj.ddns.net: Try again

These context cancelled issues also feel like simple “bad/slow connection” type of noise. They’re essentially connection timeouts.

“piecestore: rpccompat: context canceled”, “errorVerbose”: "piecestore: rpccompat: context canceled

The following may indicate a programming issue on our part, but it seems to result from an already aborted upload:

“Storj Gateway error: pending upload “Upload3” missing

I haven’t talked to the rest of the team yet, but I will take a close look at this issue:

"protocol: expected piece hash; serial number is already used: usedserialsdb error: database disk image is malformed

Overall, yeah, there’s something definitely wrong with your setup. Would it be possible to try it on another computer? Or perhaps if you’re on wifi, to try a hard-wired connection? I know these are thin “did you try turning it off and turning it back on” words of advice, but right now the arrows are mostly pointing at “bad internet.” :confused:

2 Likes
18

Hey @wthorp Everything is hardwired already and the DNS resolver is the same in use by everyone else on this network and I have not received any complaints/tickets. My laptop (also hardwired) also uses same and I’ve got no issues. I’ll try running the gateway and try some uploads on my laptop. Can’t imagine what is wrong with “my setup” since the only thing running is the gateway.

19

Im experiencing the same issue, did you manage to resolve this yourself?

here are mine logs:
2020-01-27T23:57:05.378+0100 DEBUG ecclient Uploading to storage nodes {“Erasure Share Size”: 256, “Stripe Size”: 7424, “Repair Threshold”: 35, “Optimal Threshold”: 80}
2020-01-27T23:57:05.556+0100 DEBUG ecclient Upload to storage node failed {“Node ID”: “1266KKxCkjttyvGTbVmxxwRyDwSQrGjWtq5hHfvfRNwywGsNRfM”, “error”: “protocol: expected piece hash; storage node overloaded”, “errorVerbose”: “group:\n— protocol: expected piece hash\n\tstorj.io/uplink/piecestore.(*Upload).Commit:229\n\tstorj.io/uplink/piecestore.(*BufferedUpload).Commit:45\n\tstorj.io/uplink/piecestore.(*LockingUpload).Commit:105\n\tstorj.io/uplink/ecclient.(*ecClient).PutPiece.func3:237\n\tstorj.io/uplink/ecclient.(*ecClient).PutPiece:267\n\tstorj.io/uplink/ecclient.(*ecClient).Put.func1:113\n— storage node overloaded”}
2020-01-27T23:57:06.298+0100 DEBUG ecclient Success threshold reached. Cancelling remaining uploads. {“Optimal Threshold”: 80}
2020-01-27T23:57:06.298+0100 DEBUG ecclient Failed dialing for putting piece to node{“Piece ID”: “NIC4KIUYXTIIJFMLBGQCDATSE3JDJZW2YHEY4AGHDJ4WDNZGUY5Q”, “Node ID”: “121pBmbqeC8pvUEdZCfv2NnqnQ9L5kN6wfxDN1Y5gXV4J2mxHEr”, “error”: “piecestore: rpccompat: context canceled”, “errorVerbose”: “piecestore: rpccompat: context canceled\n\tstorj.io/common/rpc.Dialer.dialTransport:242\n\tstorj.io/common/rpc.Dialer.dial:219\n\tstorj.io/common/rpc.Dialer.DialNode:127\n\tstorj.io/uplink/piecestore.Dial:51\n\tstorj.io/uplink/ecclient.(*ecClient).dialPiecestore:68\n\tstorj.io/uplink/ecclient.(*ecClient).PutPiece:198\n\tstorj.io/uplink/ecclient.(*ecClient).Put.func1:113”}
2020-01-27T23:57:06.299+0100 DEBUG ecclient Failed dialing for putting piece to node{“Piece ID”: “JJ3F6CEDTA7CBWXD6CAV4UPQPC2UZHNXRC46JNBO6LHLWHPYJWFQ”, “Node ID”: “1MmANcTYTiwL4xWCutX9Vm5FX7Lp3tgUWWtnVxLnKS1vLivPj1”, “error”: “piecestore: rpccompat: context canceled”, “errorVerbose”: “piecestore: rpccompat: context canceled\n\tstorj.io/common/rpc.Dialer.dialTransport:242\n\tstorj.io/common/rpc.Dialer.dial:219\n\tstorj.io/common/rpc.Dialer.DialNode:127\n\tstorj.io/uplink/piecestore.Dial:51\n\tstorj.io/uplink/ecclient.(*ecClient).dialPiecestore:68\n\tstorj.io/uplink/ecclient.(*ecClient).PutPiece:198\n\tstorj.io/uplink/ecclient.(*ecClient).Put.func1:113”}
2020-01-27T23:57:06.299+0100 DEBUG ecclient Failed dialing for putting piece to node{“Piece ID”: “2KXZKOG2NIFAAI3XF7UXCZQMAP5USOQ4USZBDQT4X4FSNP5QQEDQ”, “Node ID”: “1X1usWVChWYGpwekPLqqkrs7xCAWCTnjZU8VdJ67nDnacn85ya”, “error”: “piecestore: rpccompat: context canceled”, “errorVerbose”: “piecestore: rpccompat: context canceled\n\tstorj.io/common/rpc.Dialer.dialTransport:242\n\tstorj.io/common/rpc.Dialer.dial:219\n\tstorj.io/common/rpc.Dialer.DialNode:127\n\tstorj.io/uplink/piecestore.Dial:51\n\tstorj.io/uplink/ecclient.(*ecClient).dialPiecestore:68\n\tstorj.io/uplink/ecclient.(*ecClient).PutPiece:198\n\tstorj.io/uplink/ecclient.(*ecClient).Put.func1:113”}
2020-01-27T23:57:06.299+0100 DEBUG ecclient Upload to storage node failed {“Node ID”: “121pBmbqeC8pvUEdZCfv2NnqnQ9L5kN6wfxDN1Y5gXV4J2mxHEr”, “error”: “piecestore: rpccompat: context canceled”, “errorVerbose”: “piecestore: rpccompat: context canceled\n\tstorj.io/common/rpc.Dialer.dialTransport:242\n\tstorj.io/common/rpc.Dialer.dial:219\n\tstorj.io/common/rpc.Dialer.DialNode:127\n\tstorj.io/uplink/piecestore.Dial:51\n\tstorj.io/uplink/ecclient.(*ecClient).dialPiecestore:68\n\tstorj.io/uplink/ecclient.(*ecClient).PutPiece:198\n\tstorj.io/uplink/ecclient.(*ecClient).Put.func1:113”}
2020-01-27T23:57:06.299+0100 DEBUG ecclient Upload to storage node failed {“Node ID”: “1MmANcTYTiwL4xWCutX9Vm5FX7Lp3tgUWWtnVxLnKS1vLivPj1”, “error”: “piecestore: rpccompat: context canceled”, “errorVerbose”: “piecestore: rpccompat: context canceled\n\tstorj.io/common/rpc.Dialer.dialTransport:242\n\tstorj.io/common/rpc.Dialer.dial:219\n\tstorj.io/common/rpc.Dialer.DialNode:127\n\tstorj.io/uplink/piecestore.Dial:51\n\tstorj.io/uplink/ecclient.(*ecClient).dialPiecestore:68\n\tstorj.io/uplink/ecclient.(*ecClient).PutPiece:198\n\tstorj.io/uplink/ecclient.(*ecClient).Put.func1:113”}
2020-01-27T23:57:06.299+0100 DEBUG ecclient Upload to storage node failed {“Node ID”: “1x7MLrqKiR8EVCwkQTMJRS9iA9tu2G6DfvcL1TrizE4x7Nb1i3”, “error”: “protocol: expected piece hash; context canceled”, “errorVerbose”: “group:\n— protocol: expected piece hash\n\tstorj.io/uplink/piecestore.(*Upload).Commit:229\n\tstorj.io/uplink/piecestore.(*BufferedUpload).Commit:45\n\tstorj.io/uplink/piecestore.(*LockingUpload).Commit:105\n\tstorj.io/uplink/ecclient.(*ecClient).PutPiece.func3:237\n\tstorj.io/uplink/ecclient.(*ecClient).PutPiece:267\n\tstorj.io/uplink/ecclient.(*ecClient).Put.func1:113\n— context canceled”}
2020-01-27T23:57:06.300+0100 DEBUG ecclient Upload to storage node failed {“Node ID”: “1X1usWVChWYGpwekPLqqkrs7xCAWCTnjZU8VdJ67nDnacn85ya”, “error”: “piecestore: rpccompat: context canceled”, “errorVerbose”: “piecestore: rpccompat: context canceled\n\tstorj.io/common/rpc.Dialer.dialTransport:242\n\tstorj.io/common/rpc.Dialer.dial:219\n\tstorj.io/common/rpc.Dialer.DialNode:127\n\tstorj.io/uplink/piecestore.Dial:51\n\tstorj.io/uplink/ecclient.(*ecClient).dialPiecestore:68\n\tstorj.io/uplink/ecclient.(*ecClient).PutPiece:198\n\tstorj.io/uplink/ecclient.(*ecClient).Put.func1:113”}
2020-01-27T23:57:06.300+0100 DEBUG ecclient Failed dialing for putting piece to node{“Piece ID”: “JGIQYVEXHSBQFC66YGT2L24A2URYTSLSEN7CEZRBNYC6QPX7IQFQ”, “Node ID”: “125PWvjBm9ahyQNBCCWrMVqfi9xiJ3Z7eTChELaoErssknxqxrG”, “error”: “piecestore: rpccompat: context canceled”, “errorVerbose”: “piecestore: rpccompat: context canceled\n\tstorj.io/common/rpc.Dialer.dialTransport:242\n\tstorj.io/common/rpc.Dialer.dial:219\n\tstorj.io/common/rpc.Dialer.DialNode:127\n\tstorj.io/uplink/piecestore.Dial:51\n\tstorj.io/uplink/ecclient.(*ecClient).dialPiecestore:68\n\tstorj.io/uplink/ecclient.(*ecClient).PutPiece:198\n\tstorj.io/uplink/ecclient.(*ecClient).Put.func1:113”}
2020-01-27T23:57:06.300+0100 DEBUG ecclient Upload to storage node failed {“Node ID”: “1uPGUfhX4N8GANJc2Nwyc42HH3nPwMZyQPtZnB8ZVQze3gaPqK”, “error”: “protocol: expected piece hash; context canceled”, “errorVerbose”: “group:\n— protocol: expected piece hash\n\tstorj.io/uplink/piecestore.(*Upload).Commit:229\n\tstorj.io/uplink/piecestore.(*BufferedUpload).Commit:45\n\tstorj.io/uplink/piecestore.(*LockingUpload).Commit:105\n\tstorj.io/uplink/ecclient.(*ecClient).PutPiece.func3:237\n\tstorj.io/uplink/ecclient.(*ecClient).PutPiece:267\n\tstorj.io/uplink/ecclient.(*ecClient).Put.func1:113\n— context canceled”}
2020-01-27T23:57:06.300+0100 DEBUG ecclient Upload to storage node failed {“Node ID”: “1AByenV1LLduZymLPkgL38vhkyZmmkrtEUAVRbDjaogiPDX43s”, “error”: “protocol: expected piece hash; context canceled”, “errorVerbose”: “group:\n— protocol: expected piece hash\n\tstorj.io/uplink/piecestore.(*Upload).Commit:229\n\tstorj.io/uplink/piecestore.(*BufferedUpload).Commit:45\n\tstorj.io/uplink/piecestore.(*LockingUpload).Commit:105\n\tstorj.io/uplink/ecclient.(*ecClient).PutPiece.func3:237\n\tstorj.io/uplink/ecclient.(*ecClient).PutPiece:267\n\tstorj.io/uplink/ecclient.(*ecClient).Put.func1:113\n— context canceled”}
2020-01-27T23:57:06.300+0100 DEBUG ecclient Upload to storage node failed {“Node ID”: “12GE8skw2vSubpQyJc5MG981jSatx7KPeeGnuY1ijY5Xytf9UZt”, “error”: “protocol: expected piece hash; context canceled”, “errorVerbose”: “group:\n— protocol: expected piece hash\n\tstorj.io/uplink/piecestore.(*Upload).Commit:229\n\tstorj.io/uplink/piecestore.(*BufferedUpload).Commit:45\n\tstorj.io/uplink/piecestore.(*LockingUpload).Commit:105\n\tstorj.io/uplink/ecclient.(*ecClient).PutPiece.func3:237\n\tstorj.io/uplink/ecclient.(*ecClient).PutPiece:267\n\tstorj.io/uplink/ecclient.(*ecClient).Put.func1:113\n— context canceled”}
2020-01-27T23:57:06.300+0100 DEBUG ecclient Upload to storage node failed {“Node ID”: “1gyh6Ga6zzM8umX6JMZJRKPmNoJHE7MDE3SAJ9cU4Hah9eLzCL”, “error”: “protocol: expected piece hash; context canceled”, “errorVerbose”: “group:\n— protocol: expected piece hash\n\tstorj.io/uplink/piecestore.(*Upload).Commit:229\n\tstorj.io/uplink/piecestore.(*BufferedUpload).Commit:45\n\tstorj.io/uplink/piecestore.(*LockingUpload).Commit:105\n\tstorj.io/uplink/ecclient.(*ecClient).PutPiece.func3:237\n\tstorj.io/uplink/ecclient.(*ecClient).PutPiece:267\n\tstorj.io/uplink/ecclient.(*ecClient).Put.func1:113\n— context canceled”}
2020-01-27T23:57:06.300+0100 DEBUG ecclient Failed dialing for putting piece to node{“Piece ID”: “5UQESXB6PV3NO7EQNNNDCYPNEISL2PK754FJE44PGW7XASB52XLQ”, “Node ID”: “16WC3pbGTpr4eWGNuEYxAWMqtDdcT6F6gNonKJcxuaze8PdKDt”, “error”: “piecestore: rpccompat: context canceled”, “errorVerbose”: “piecestore: rpccompat: context canceled\n\tstorj.io/common/rpc.Dialer.dialTransport:262\n\tstorj.io/common/rpc.Dialer.dial:219\n\tstorj.io/common/rpc.Dialer.DialNode:127\n\tstorj.io/uplink/piecestore.Dial:51\n\tstorj.io/uplink/ecclient.(*ecClient).dialPiecestore:68\n\tstorj.io/uplink/ecclient.(*ecClient).PutPiece:198\n\tstorj.io/uplink/ecclient.(*ecClient).Put.func1:113”}
2020-01-27T23:57:06.300+0100 DEBUG ecclient Upload to storage node failed {“Node ID”: “12pwgD9t7UWEh23f21z44bwGGCT76RLpv2c8zwmmkWKSw9vryPr”, “error”: “protocol: expected piece hash; context canceled”, “errorVerbose”: “group:\n— protocol: expected piece hash\n\tstorj.io/uplink/piecestore.(*Upload).Commit:229\n\tstorj.io/uplink/piecestore.(*BufferedUpload).Commit:45\n\tstorj.io/uplink/piecestore.(*LockingUpload).Commit:105\n\tstorj.io/uplink/ecclient.(*ecClient).PutPiece.func3:237\n\tstorj.io/uplink/ecclient.(*ecClient).PutPiece:267\n\tstorj.io/uplink/ecclient.(*ecClient).Put.func1:113\n— context canceled”}
2020-01-27T23:57:06.301+0100 DEBUG ecclient Upload to storage node failed {“Node ID”: “12eJYVoggwzhU2GtFG2vZJRWwdFwWTff3N9QmdP9ijWVgJw3zFz”, “error”: “protocol: expected piece hash; context canceled”, “errorVerbose”: “group:\n— protocol: expected piece hash\n\tstorj.io/uplink/piecestore.(*Upload).Commit:229\n\tstorj.io/uplink/piecestore.(*BufferedUpload).Commit:45\n\tstorj.io/uplink/piecestore.(*LockingUpload).Commit:105\n\tstorj.io/uplink/ecclient.(*ecClient).PutPiece.func3:237\n\tstorj.io/uplink/ecclient.(*ecClient).PutPiece:267\n\tstorj.io/uplink/ecclient.(*ecClient).Put.func1:113\n— context canceled”}
2020-01-27T23:57:06.301+0100 DEBUG ecclient Upload to storage node failed {“Node ID”: “12WXCE1QF9SiGBW99Hp4RjUttXGRJPRLZxqLVNNBkSA4JbmxcN”, “error”: “protocol: expected piece hash; context canceled”, “errorVerbose”: “group:\n— protocol: expected piece hash\n\tstorj.io/uplink/piecestore.(*Upload).Commit:229\n\tstorj.io/uplink/piecestore.(*BufferedUpload).Commit:45\n\tstorj.io/uplink/piecestore.(*LockingUpload).Commit:105\n\tstorj.io/uplink/ecclient.(*ecClient).PutPiece.func3:237\n\tstorj.io/uplink/ecclient.(*ecClient).PutPiece:267\n\tstorj.io/uplink/ecclient.(*ecClient).Put.func1:113\n— context canceled”}
2020-01-27T23:57:06.301+0100 DEBUG ecclient Upload to storage node failed {“Node ID”: “125PWvjBm9ahyQNBCCWrMVqfi9xiJ3Z7eTChELaoErssknxqxrG”, “error”: “piecestore: rpccompat: context canceled”, “errorVerbose”: “piecestore: rpccompat: context canceled\n\tstorj.io/common/rpc.Dialer.dialTransport:242\n\tstorj.io/common/rpc.Dialer.dial:219\n\tstorj.io/common/rpc.Dialer.DialNode:127\n\tstorj.io/uplink/piecestore.Dial:51\n\tstorj.io/uplink/ecclient.(*ecClient).dialPiecestore:68\n\tstorj.io/uplink/ecclient.(*ecClient).PutPiece:198\n\tstorj.io/uplink/ecclient.(*ecClient).Put.func1:113”}
2020-01-27T23:57:06.301+0100 DEBUG ecclient Upload to storage node failed {“Node ID”: “127e94SNZi4EcHco61XX9tHoU18qH5ww76PwBBA69ffjii6xra9”, “error”: “protocol: expected piece hash; context canceled”, “errorVerbose”: “group:\n— protocol: expected piece hash\n\tstorj.io/uplink/piecestore.(*Upload).Commit:229\n\tstorj.io/uplink/piecestore.(*BufferedUpload).Commit:45\n\tstorj.io/uplink/piecestore.(*LockingUpload).Commit:105\n\tstorj.io/uplink/ecclient.(*ecClient).PutPiece.func3:237\n\tstorj.io/uplink/ecclient.(*ecClient).PutPiece:267\n\tstorj.io/uplink/ecclient.(*ecClient).Put.func1:113\n— context canceled”}
2020-01-27T23:57:06.301+0100 DEBUG ecclient Upload to storage node failed {“Node ID”: “16WC3pbGTpr4eWGNuEYxAWMqtDdcT6F6gNonKJcxuaze8PdKDt”, “error”: “piecestore: rpccompat: context canceled”, “errorVerbose”: “piecestore: rpccompat: context canceled\n\tstorj.io/common/rpc.Dialer.dialTransport:262\n\tstorj.io/common/rpc.Dialer.dial:219\n\tstorj.io/common/rpc.Dialer.DialNode:127\n\tstorj.io/uplink/piecestore.Dial:51\n\tstorj.io/uplink/ecclient.(*ecClient).dialPiecestore:68\n\tstorj.io/uplink/ecclient.(*ecClient).PutPiece:198\n\tstorj.io/uplink/ecclient.(*ecClient).Put.func1:113”}

im on the latest linux x64 version from github:
INFO running on version v0.30.5

Thank you

20

@pavel, can you check to see if your file was uploaded?

uplink ls sj://yourbucketname

The reason I ask is because of this:

2020-01-27T23:57:06.298+0100 DEBUG ecclient Success threshold reached. Cancelling remaining uploads. {“Optimal Threshold”: 80}

and the rest of the logs appear to be context cancelled