Changelog v0.24.5

littleskunk · October 28, 2019, 2:49pm

Gracefull Exit
We are close to the finish line but it is not completed. Don’t execute it. Don’t test it. Don’t even try it. If you are waiting for gracefull exit we can use the time to talk about it first.

Remove default concurrency limit
By default a storage node had a limit of 6 concurrent transfers and will reject additional uploads until one of the ongoing transfers is finished. We believe with dRPC the limit is not needed anymore. By default the storage nodes will now take as many uploads as they can. You can still set the limit in your config file like before. Removing it from the config will reset it to the new default value.

Storj-sim requires postgres
Storj-sim is our local test network. It now requires postgres. We have updated the wiki: https://github.com/storj/storj/wiki/Test-network

naxbc · October 28, 2019, 3:23pm

How about the Raspberry´s nodes? Won´t they be overwhelmed now with this change?
What is the “new default value”?

sorry2xs · October 28, 2019, 4:08pm

why are Kademlia entries still present in the config.yaml if Kademlia is no longer used, also web dashboard is not displaying correctly and undefined.undefinned.undefined is displayed

Your Storage Node Stats

Current period: October

Online

Node Version

vundefined.undefined.undefined

Choose your satellite: All satellites

Utilization & Remaining

Bandwidth Used This Month

315.53GB

Disk Space Used This Month

1.72PB*h

Remaining on the Node

Bandwidth Remaining

99.68TB

Disk Space Remaining

1.32TB

heunland · October 28, 2019, 4:19pm

As noted in the changelog, you can still continue setting your own concurrency limit on your Pi´s config file as before?

littleskunk · October 28, 2019, 4:33pm

Because we will never touch your config file. Take a look at your logfile and you should notice some warnings about deprecated confilg values.

naxbc · October 28, 2019, 4:33pm

Apparently yes, I´ll keep monitoring it, thanks.

sorry2xs · October 28, 2019, 5:05pm

I do see this one (gracefulexit:chore running graceful exit chore.) also can I stop rm the container and then delete the config.yaml and pull, docker run so that I could get a new clean config.yaml with out doing any harm to the node

bovcan · October 28, 2019, 5:52pm

is new version out already? my node is still 0.23.5

heunland · October 28, 2019, 6:18pm

Your node may take up to 72 hours to auto-update if you are using Watchtower. There is no need to worry if your node has not updated itself yet.

Vadim · October 28, 2019, 6:34pm

Dashboard is gone wired on chrome
chrome
on edge looks OK

Vadim · October 28, 2019, 6:36pm

Whay today my node made update if it was up to date?
my node was update some days ago 0.24.5
today updated once more to 0.24.5 but file size is different from previus.

Vadim · October 28, 2019, 6:39pm

My bad it was 23.5 now it 24.5 but dashboard on chrome is wired

peem · October 28, 2019, 6:42pm

Ctrl+F5 refresh page

Vadim · October 28, 2019, 6:43pm

Thanks gone mutch better.

Odmin · October 29, 2019, 5:42am

@littleskunk I still keep eyes on drpc errors, here some after node upgrade.

docker logs storagenode 2>&1 | grep drpc

2019-10-28T22:02:51.098Z        INFO    piecestore      download failed {"Piece ID": "3YUPIMXWIA6H2UFMYPS3THD7RZV44UUJXPLWHEWBJQHYDMMUIWUA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET", "error": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:36344: write: connection reset by peer", "errorVerbose": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:36344: write: connection reset by peer\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:162\n\tstorj.io/drpc/drpcwire.SplitN:25\n\tstorj.io/drpc/drpcstream.(*Stream).RawWrite:199\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:231\n\tstorj.io/storj/pkg/pb.(*drpcPiecestoreDownloadStream).Send:905\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func3:565\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-10-28T22:09:08.810Z        INFO    piecestore      download failed {"Piece ID": "3YUPIMXWIA6H2UFMYPS3THD7RZV44UUJXPLWHEWBJQHYDMMUIWUA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET", "error": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:43636: use of closed network connection", "errorVerbose": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:43636: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:162\n\tstorj.io/drpc/drpcwire.SplitN:25\n\tstorj.io/drpc/drpcstream.(*Stream).RawWrite:199\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:231\n\tstorj.io/storj/pkg/pb.(*drpcPiecestoreDownloadStream).Send:905\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func3:565\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-10-28T23:12:51.721Z        INFO    piecestore      download failed {"Piece ID": "3YUPIMXWIA6H2UFMYPS3THD7RZV44UUJXPLWHEWBJQHYDMMUIWUA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET", "error": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:53050: write: broken pipe", "errorVerbose": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:53050: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:162\n\tstorj.io/drpc/drpcwire.SplitN:25\n\tstorj.io/drpc/drpcstream.(*Stream).RawWrite:199\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:231\n\tstorj.io/storj/pkg/pb.(*drpcPiecestoreDownloadStream).Send:905\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func3:565\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-10-29T00:42:51.155Z        INFO    piecestore      download failed {"Piece ID": "3YUPIMXWIA6H2UFMYPS3THD7RZV44UUJXPLWHEWBJQHYDMMUIWUA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET", "error": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:34192: write: broken pipe", "errorVerbose": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:34192: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:162\n\tstorj.io/drpc/drpcwire.SplitN:25\n\tstorj.io/drpc/drpcstream.(*Stream).RawWrite:199\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:231\n\tstorj.io/storj/pkg/pb.(*drpcPiecestoreDownloadStream).Send:905\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func3:565\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-10-29T00:52:51.294Z        INFO    piecestore      download failed {"Piece ID": "3YUPIMXWIA6H2UFMYPS3THD7RZV44UUJXPLWHEWBJQHYDMMUIWUA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET", "error": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:44670: write: broken pipe", "errorVerbose": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:44670: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:162\n\tstorj.io/drpc/drpcwire.SplitN:25\n\tstorj.io/drpc/drpcstream.(*Stream).RawWrite:199\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:231\n\tstorj.io/storj/pkg/pb.(*drpcPiecestoreDownloadStream).Send:905\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func3:565\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-10-29T01:17:51.173Z        INFO    piecestore      download failed {"Piece ID": "3YUPIMXWIA6H2UFMYPS3THD7RZV44UUJXPLWHEWBJQHYDMMUIWUA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET", "error": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:42654: write: broken pipe", "errorVerbose": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:42654: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:162\n\tstorj.io/drpc/drpcwire.SplitN:25\n\tstorj.io/drpc/drpcstream.(*Stream).RawWrite:199\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:231\n\tstorj.io/storj/pkg/pb.(*drpcPiecestoreDownloadStream).Send:905\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func3:565\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-10-29T01:32:50.990Z        INFO    piecestore      download failed {"Piece ID": "3YUPIMXWIA6H2UFMYPS3THD7RZV44UUJXPLWHEWBJQHYDMMUIWUA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET", "error": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:58346: write: broken pipe", "errorVerbose": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:58346: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:162\n\tstorj.io/drpc/drpcwire.SplitN:25\n\tstorj.io/drpc/drpcstream.(*Stream).RawWrite:199\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:231\n\tstorj.io/storj/pkg/pb.(*drpcPiecestoreDownloadStream).Send:905\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func3:565\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-10-29T04:02:51.307Z        INFO    piecestore      download failed {"Piece ID": "3YUPIMXWIA6H2UFMYPS3THD7RZV44UUJXPLWHEWBJQHYDMMUIWUA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET", "error": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:45890: write: broken pipe", "errorVerbose": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:45890: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:162\n\tstorj.io/drpc/drpcwire.SplitN:25\n\tstorj.io/drpc/drpcstream.(*Stream).RawWrite:199\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:231\n\tstorj.io/storj/pkg/pb.(*drpcPiecestoreDownloadStream).Send:905\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func3:565\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

Details about that piece:
docker logs storagenode 2>&1 | grep "3YUPIMXWIA6H2UFMYPS3THD7RZV44UUJXPLWHEWBJQHYDMMUIWUA"

Link to Pastebin, a lot of information for this post.

jerrydp · October 29, 2019, 11:26am

Is this normal that I see this on my node?

gracefulexit:chore running graceful exit chore.

I’ve never requested graceful exit
Node ID 1pXCvt4E3nczEXL1xpyyMjsNoSxf3yjfgg2AggiBo5KBJedDGK

Odmin · October 29, 2019, 11:31am

This information message persists on all storagenodes.

docker logs storagenode 2>&1 | grep gracefulexit

Summary

2019-10-28T19:11:53.716Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T19:26:40.084Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T19:41:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T19:56:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T20:11:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T20:26:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T20:41:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T20:56:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T21:11:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T21:26:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T21:41:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T21:56:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T22:11:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T22:26:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T22:41:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T22:56:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T23:11:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T23:26:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T23:41:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T23:56:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T00:11:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T00:26:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T00:41:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T00:56:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T01:11:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T01:26:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T01:41:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T01:56:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T02:11:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T02:26:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T02:41:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T02:56:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T03:11:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T03:26:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T03:41:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T03:56:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T04:11:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T04:26:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T04:41:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T04:56:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T05:11:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T05:26:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T05:41:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T05:56:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T06:11:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T06:26:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T06:41:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T06:56:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T07:11:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T07:26:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T07:41:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T07:56:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T08:11:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T08:26:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T08:41:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T08:56:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T09:11:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T09:26:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T09:41:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T09:56:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T10:11:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T10:26:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T10:41:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T10:56:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T11:11:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T11:26:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.

jerrydp · October 29, 2019, 11:34am

Thank you for the reply
But what does this message mean, should I worry or treat it as normal operation?

littleskunk · October 29, 2019, 2:30pm

That is just an idle message from the graceful exit job. As long as you don’t call gracefull exit the job will skip all the time.

jerrydp · October 29, 2019, 2:43pm

Thank you for the answer