Changelog v0.23.5

littleskunk · October 21, 2019, 12:56pm

dRPC bugfix

Odmin · October 21, 2019, 1:00pm

So, on this release, we also should keep eyes on dRPC events?

littleskunk · October 21, 2019, 2:09pm

yes please

Odmin · October 21, 2019, 2:18pm

will be done

(add more dummy character for compleate this post)

littleskunk · October 21, 2019, 3:23pm

That is not a dRPC error.

Vadim · October 21, 2019, 5:44pm

After update have same error

2019-10-21T20:42:29.390+0300	INFO	piecestore	download failed	{“Piece ID”: “467BJE3HKM5ER56VSZZ2PHRQ3XURXMASGXU7BVP54XT7OWWW5V5A”, “SatelliteID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “GET”, “error”: “piecestore: piecestore protocol: rpc error: code = Canceled desc = context canceled”, “errorVerbose”: “piecestore: piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(Endpoint).doDownload.func3:570\n\tgolang.org/x/sync/errgroup.(Group).Go.func1:57”}
2019-10-21T20:42:29.390+0300	ERROR	server	gRPC stream error response	{“error”: “piecestore: piecestore protocol: rpc error: code = Canceled desc = context canceled”, “errorVerbose”: “piecestore: piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(Endpoint).doDownload.func3:570\n\tgolang.org/x/sync/errgroup.(Group).Go.func1:57”}

littleskunk · October 21, 2019, 6:09pm

That is not a dRPC error

Vadim · October 21, 2019, 6:12pm

on the github there was dRPC grpc complient writen

Alexey · October 22, 2019, 8:56am

A post was split to a new topic: Since the new version I have been getting very low activity. Is this normal?

Odmin · October 22, 2019, 5:11am

@littleskunk

I caught it again:

docker logs storagenode 2>&1 | grep drpc

2019-10-21T23:22:50.936Z        INFO    piecestore      download failed {"Piece ID": "3YUPIMXWIA6H2UFMYPS3THD7RZV44UUJXPLWHEWBJQHYDMMUIWUA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET", "error": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:47514: write: broken pipe", "errorVerbose": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:47514: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:161\n\tstorj.io/drpc/drpcwire.SplitN:25\n\tstorj.io/drpc/drpcstream.(*Stream).RawWrite:198\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:230\n\tstorj.io/storj/pkg/pb.(*drpcPiecestoreDownloadStream).Send:905\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func3:561\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-10-21T23:42:50.800Z        INFO    piecestore      download failed {"Piece ID": "3YUPIMXWIA6H2UFMYPS3THD7RZV44UUJXPLWHEWBJQHYDMMUIWUA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET", "error": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:40132: write: broken pipe", "errorVerbose": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:40132: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:161\n\tstorj.io/drpc/drpcwire.SplitN:25\n\tstorj.io/drpc/drpcstream.(*Stream).RawWrite:198\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:230\n\tstorj.io/storj/pkg/pb.(*drpcPiecestoreDownloadStream).Send:905\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func3:561\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-10-22T00:07:51.152Z        INFO    piecestore      download failed {"Piece ID": "3YUPIMXWIA6H2UFMYPS3THD7RZV44UUJXPLWHEWBJQHYDMMUIWUA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET", "error": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:37864: write: broken pipe", "errorVerbose": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:37864: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:161\n\tstorj.io/drpc/drpcwire.SplitN:25\n\tstorj.io/drpc/drpcstream.(*Stream).RawWrite:198\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:230\n\tstorj.io/storj/pkg/pb.(*drpcPiecestoreDownloadStream).Send:905\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func3:561\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-10-22T01:32:50.689Z        INFO    piecestore      download failed {"Piece ID": "3YUPIMXWIA6H2UFMYPS3THD7RZV44UUJXPLWHEWBJQHYDMMUIWUA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET", "error": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:41448: write: broken pipe", "errorVerbose": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:41448: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:161\n\tstorj.io/drpc/drpcwire.SplitN:25\n\tstorj.io/drpc/drpcstream.(*Stream).RawWrite:198\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:230\n\tstorj.io/storj/pkg/pb.(*drpcPiecestoreDownloadStream).Send:905\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func3:561\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-10-22T03:47:50.731Z        INFO    piecestore      download failed {"Piece ID": "3YUPIMXWIA6H2UFMYPS3THD7RZV44UUJXPLWHEWBJQHYDMMUIWUA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET", "error": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:40306: write: broken pipe", "errorVerbose": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:40306: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:161\n\tstorj.io/drpc/drpcwire.SplitN:25\n\tstorj.io/drpc/drpcstream.(*Stream).RawWrite:198\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:230\n\tstorj.io/storj/pkg/pb.(*drpcPiecestoreDownloadStream).Send:905\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func3:561\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-10-22T04:37:50.795Z        INFO    piecestore      download failed {"Piece ID": "3YUPIMXWIA6H2UFMYPS3THD7RZV44UUJXPLWHEWBJQHYDMMUIWUA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET", "error": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:35788: write: broken pipe", "errorVerbose": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:35788: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:161\n\tstorj.io/drpc/drpcwire.SplitN:25\n\tstorj.io/drpc/drpcstream.(*Stream).RawWrite:198\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:230\n\tstorj.io/storj/pkg/pb.(*drpcPiecestoreDownloadStream).Send:905\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func3:561\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

Full details about that piece:

docker logs storagenode 2>&1 | grep "3YUPIMXWIA6H2UFMYPS3THD7RZV44UUJXPLWHEWBJQHYDMMUIWUA"

Link to Pastebin (too many symbols for one post on forum)

The same information (but another error) from another node:

docker logs storagenode 2>&1 | grep drpc

2019-10-22T03:53:24.431Z INFO piecestore download failed {"Piece ID": "TARIPOECDXTLZBZINPAKZOJLZOQKDLQABKPYYD3U24UYHXP7OWHA", "SatelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "error": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:47844: use of closed network connection", "errorVerbose": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:47844: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:161\n\tstorj.io/drpc/drpcwire.SplitN:25\n\tstorj.io/drpc/drpcstream.(*Stream).RawWrite:198\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:230\n\tstorj.io/storj/pkg/pb.(*drpcPiecestoreDownloadStream).Send:905\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func3:561\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

Full details about that piece:

docker logs storagenode 2>&1 | grep "TARIPOECDXTLZBZINPAKZOJLZOQKDLQABKPYYD3U24UYHXP7OWHA"

Link to Pastebin (too many symbols for one post on forum)

naxbc · October 22, 2019, 9:12am

Same here:

2019-10-22T07:40:53.448Z	INFO	piecestore	download failed	{“Piece ID”: “MZRQJ4OJV2PODEGAMNU6QGEIMFEMPFLZEMHBMKJY4RYMAU2ZO4JQ”, “SatelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”, “error”: “piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:47290: write: connection reset by peer”, “errorVerbose”: “piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:47290: write: connection reset by peer\n\tstorj.io/drpc/drpcstream.(Stream).pollWrite:161\n\tstorj.io/drpc/drpcwire.SplitN:25\n\tstorj.io/drpc/drpcstream.(Stream).RawWrite:198\n\tstorj.io/drpc/drpcstream.(Stream).MsgSend:230\n\tstorj.io/storj/pkg/pb.(drpcPiecestoreDownloadStream).Send:905\n\tstorj.io/storj/storagenode/piecestore.(Endpoint).doDownload.func3:561\n\tgolang.org/x/sync/errgroup.(Group).Go.func1:57”}
2019-10-22T07:41:53.397Z	INFO	piecestore	download failed	{“Piece ID”: “MZRQJ4OJV2PODEGAMNU6QGEIMFEMPFLZEMHBMKJY4RYMAU2ZO4JQ”, “SatelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”, “error”: “piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:48034: write: connection reset by peer”, “errorVerbose”: “piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:48034: write: connection reset by peer\n\tstorj.io/drpc/drpcstream.(Stream).pollWrite:161\n\tstorj.io/drpc/drpcwire.SplitN:25\n\tstorj.io/drpc/drpcstream.(Stream).RawWrite:198\n\tstorj.io/drpc/drpcstream.(Stream).MsgSend:230\n\tstorj.io/storj/pkg/pb.(drpcPiecestoreDownloadStream).Send:905\n\tstorj.io/storj/storagenode/piecestore.(Endpoint).doDownload.func3:561\n\tgolang.org/x/sync/errgroup.(Group).Go.func1:57”}
2019-10-22T07:42:53.892Z	INFO	piecestore	download failed	{“Piece ID”: “MZRQJ4OJV2PODEGAMNU6QGEIMFEMPFLZEMHBMKJY4RYMAU2ZO4JQ”, “SatelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”, “error”: “piecestore: piecestore protocol: context canceled”, “errorVerbose”: “piecestore: piecestore protocol: context canceled\n\tstorj.io/drpc/drpcstream.(Stream).RawWrite:198\n\tstorj.io/drpc/drpcstream.(Stream).MsgSend:230\n\tstorj.io/storj/pkg/pb.(drpcPiecestoreDownloadStream).Send:905\n\tstorj.io/storj/storagenode/piecestore.(Endpoint).doDownload.func3:561\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2019-10-22T07:49:53.377Z	INFO	piecestore	download failed	{“Piece ID”: “MZRQJ4OJV2PODEGAMNU6QGEIMFEMPFLZEMHBMKJY4RYMAU2ZO4JQ”, “SatelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”, “error”: “piecestore: piecestore protocol: context canceled”, “errorVerbose”: “piecestore: piecestore protocol: context canceled\n\tstorj.io/drpc/drpcstream.(Stream).RawWrite:198\n\tstorj.io/drpc/drpcstream.(Stream).MsgSend:230\n\tstorj.io/storj/pkg/pb.(drpcPiecestoreDownloadStream).Send:905\n\tstorj.io/storj/storagenode/piecestore.(Endpoint).doDownload.func3:561\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2019-10-22T07:55:53.350Z	INFO	piecestore	download failed	{“Piece ID”: “MZRQJ4OJV2PODEGAMNU6QGEIMFEMPFLZEMHBMKJY4RYMAU2ZO4JQ”, “SatelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”, “error”: “piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:34670: use of closed network connection”, “errorVerbose”: “piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:34670: use of closed network connection\n\tstorj.io/drpc/drpcstream.(Stream).pollWrite:161\n\tstorj.io/drpc/drpcwire.SplitN:25\n\tstorj.io/drpc/drpcstream.(Stream).RawWrite:198\n\tstorj.io/drpc/drpcstream.(Stream).MsgSend:230\n\tstorj.io/storj/pkg/pb.(drpcPiecestoreDownloadStream).Send:905\n\tstorj.io/storj/storagenode/piecestore.(Endpoint).doDownload.func3:561\n\tgolang.org/x/sync/errgroup.(Group).Go.func1:57”}
2019-10-22T08:24:53.310Z	INFO	piecestore	download failed	{“Piece ID”: “MZRQJ4OJV2PODEGAMNU6QGEIMFEMPFLZEMHBMKJY4RYMAU2ZO4JQ”, “SatelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”, “error”: “piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:36964: use of closed network connection”, “errorVerbose”: “piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:36964: use of closed network connection\n\tstorj.io/drpc/drpcstream.(Stream).pollWrite:161\n\tstorj.io/drpc/drpcwire.SplitN:25\n\tstorj.io/drpc/drpcstream.(Stream).RawWrite:198\n\tstorj.io/drpc/drpcstream.(Stream).MsgSend:230\n\tstorj.io/storj/pkg/pb.(drpcPiecestoreDownloadStream).Send:905\n\tstorj.io/storj/storagenode/piecestore.(Endpoint).doDownload.func3:561\n\tgolang.org/x/sync/errgroup.(Group).Go.func1:57”}
2019-10-22T08:44:53.454Z	INFO	piecestore	download failed	{“Piece ID”: “MZRQJ4OJV2PODEGAMNU6QGEIMFEMPFLZEMHBMKJY4RYMAU2ZO4JQ”, “SatelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”, “error”: “piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:54114: write: connection reset by peer”, “errorVerbose”: “piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:54114: write: connection reset by peer\n\tstorj.io/drpc/drpcstream.(Stream).pollWrite:161\n\tstorj.io/drpc/drpcwire.SplitN:25\n\tstorj.io/drpc/drpcstream.(Stream).RawWrite:198\n\tstorj.io/drpc/drpcstream.(Stream).MsgSend:230\n\tstorj.io/storj/pkg/pb.(drpcPiecestoreDownloadStream).Send:905\n\tstorj.io/storj/storagenode/piecestore.(Endpoint).doDownload.func3:561\n\tgolang.org/x/sync/errgroup.(Group).Go.func1:57”}
2019-10-22T08:46:53.426Z	INFO	piecestore	download failed	{“Piece ID”: “MZRQJ4OJV2PODEGAMNU6QGEIMFEMPFLZEMHBMKJY4RYMAU2ZO4JQ”, “SatelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”, “error”: “piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:55236: use of closed network connection”, “errorVerbose”: “piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:55236: use of closed network connection\n\tstorj.io/drpc/drpcstream.(Stream).pollWrite:161\n\tstorj.io/drpc/drpcwire.SplitN:25\n\tstorj.io/drpc/drpcstream.(Stream).RawWrite:198\n\tstorj.io/drpc/drpcstream.(Stream).MsgSend:230\n\tstorj.io/storj/pkg/pb.(drpcPiecestoreDownloadStream).Send:905\n\tstorj.io/storj/storagenode/piecestore.(Endpoint).doDownload.func3:561\n\tgolang.org/x/sync/errgroup.(Group).Go.func1:57”}
2019-10-22T08:49:53.346Z	INFO	piecestore	download failed	{“Piece ID”: “MZRQJ4OJV2PODEGAMNU6QGEIMFEMPFLZEMHBMKJY4RYMAU2ZO4JQ”, “SatelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”, “error”: “piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:59142: write: connection reset by peer”, “errorVerbose”: “piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:59142: write: connection reset by peer\n\tstorj.io/drpc/drpcstream.(Stream).pollWrite:161\n\tstorj.io/drpc/drpcwire.SplitN:25\n\tstorj.io/drpc/drpcstream.(Stream).RawWrite:198\n\tstorj.io/drpc/drpcstream.(Stream).MsgSend:230\n\tstorj.io/storj/pkg/pb.(drpcPiecestoreDownloadStream).Send:905\n\tstorj.io/storj/storagenode/piecestore.(Endpoint).doDownload.func3:561\n\tgolang.org/x/sync/errgroup.(Group).Go.func1:57”}
2019-10-22T09:03:53.305Z	INFO	piecestore	download failed	{“Piece ID”: “MZRQJ4OJV2PODEGAMNU6QGEIMFEMPFLZEMHBMKJY4RYMAU2ZO4JQ”, “SatelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”, “error”: “piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:45860: use of closed network connection”, “errorVerbose”: “piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:45860: use of closed network connection\n\tstorj.io/drpc/drpcstream.(Stream).pollWrite:161\n\tstorj.io/drpc/drpcwire.SplitN:25\n\tstorj.io/drpc/drpcstream.(Stream).RawWrite:198\n\tstorj.io/drpc/drpcstream.(Stream).MsgSend:230\n\tstorj.io/storj/pkg/pb.(drpcPiecestoreDownloadStream).Send:905\n\tstorj.io/storj/storagenode/piecestore.(Endpoint).doDownload.func3:561\n\tgolang.org/x/sync/errgroup.(Group).Go.func1:57”}

peem · October 23, 2019, 6:33pm

i have only one drpc entry

2019-10-22T19:43:15.466Z INFO piecestore download failed {“Piece ID”: “LXXZGNQC7RWOOME4POKN62MURJR6JIH3YVY2J24UFK35M3ZT7OAA”, “SatelliteID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET”, “error”: “piecestore: piecestore protocol: write tcp 172.17.0.5:28967->34.67.235.103:36480: use of closed network connection”, “errorVerbose”: “piecestore: piecestore protocol: write tcp 172.17.0.5:28967->34.67.235.103:36480: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:161\n\tstorj.io/drpc/drpcwire.SplitN:25\n\tstorj.io/drpc/drpcstream.(*Stream).RawWrite:198\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:230\n\tstorj.io/storj/pkg/pb.(*drpcPiecestoreDownloadStream).Send:905\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func3:561\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}