Changelog v0.24.5

Gracefull Exit
We are close to the finish line but it is not completed. Don’t execute it. Don’t test it. Don’t even try it. If you are waiting for gracefull exit we can use the time to talk about it first.

Remove default concurrency limit
By default a storage node had a limit of 6 concurrent transfers and will reject additional uploads until one of the ongoing transfers is finished. We believe with dRPC the limit is not needed anymore. By default the storage nodes will now take as many uploads as they can. You can still set the limit in your config file like before. Removing it from the config will reset it to the new default value.

Storj-sim requires postgres
Storj-sim is our local test network. It now requires postgres. We have updated the wiki: https://github.com/storj/storj/wiki/Test-network

3 Likes

How about the Raspberry´s nodes? Won´t they be overwhelmed now with this change?
What is the “new default value”?

why are Kademlia entries still present in the config.yaml if Kademlia is no longer used, also web dashboard is not displaying correctly and undefined.undefinned.undefined is displayed

Your Storage Node Stats

Current period: October

Online

Node Version

vundefined.undefined.undefined

Choose your satellite: All satellites

Utilization & Remaining

Bandwidth Used This Month

315.53GB

Disk Space Used This Month

1.72PB*h

Remaining on the Node

Bandwidth Remaining

99.68TB

Disk Space Remaining

1.32TB

As noted in the changelog, you can still continue setting your own concurrency limit on your Pi´s config file as before?

1 Like

Because we will never touch your config file. Take a look at your logfile and you should notice some warnings about deprecated confilg values.

Apparently yes, I´ll keep monitoring it, thanks.

I do see this one (gracefulexit:chore running graceful exit chore.) also can I stop rm the container and then delete the config.yaml and pull, docker run so that I could get a new clean config.yaml with out doing any harm to the node

is new version out already? my node is still 0.23.5

Your node may take up to 72 hours to auto-update if you are using Watchtower. There is no need to worry if your node has not updated itself yet.

Dashboard is gone wired on chrome


on edge looks OK

Whay today my node made update if it was up to date?
my node was update some days ago 0.24.5
today updated once more to 0.24.5 but file size is different from previus.

My bad it was 23.5 now it 24.5 but dashboard on chrome is wired

Ctrl+F5 refresh page

1 Like

Thanks gone mutch better.

@littleskunk I still keep eyes on drpc errors, here some after node upgrade.

docker logs storagenode 2>&1 | grep drpc

2019-10-28T22:02:51.098Z        INFO    piecestore      download failed {"Piece ID": "3YUPIMXWIA6H2UFMYPS3THD7RZV44UUJXPLWHEWBJQHYDMMUIWUA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET", "error": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:36344: write: connection reset by peer", "errorVerbose": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:36344: write: connection reset by peer\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:162\n\tstorj.io/drpc/drpcwire.SplitN:25\n\tstorj.io/drpc/drpcstream.(*Stream).RawWrite:199\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:231\n\tstorj.io/storj/pkg/pb.(*drpcPiecestoreDownloadStream).Send:905\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func3:565\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-10-28T22:09:08.810Z        INFO    piecestore      download failed {"Piece ID": "3YUPIMXWIA6H2UFMYPS3THD7RZV44UUJXPLWHEWBJQHYDMMUIWUA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET", "error": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:43636: use of closed network connection", "errorVerbose": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:43636: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:162\n\tstorj.io/drpc/drpcwire.SplitN:25\n\tstorj.io/drpc/drpcstream.(*Stream).RawWrite:199\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:231\n\tstorj.io/storj/pkg/pb.(*drpcPiecestoreDownloadStream).Send:905\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func3:565\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-10-28T23:12:51.721Z        INFO    piecestore      download failed {"Piece ID": "3YUPIMXWIA6H2UFMYPS3THD7RZV44UUJXPLWHEWBJQHYDMMUIWUA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET", "error": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:53050: write: broken pipe", "errorVerbose": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:53050: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:162\n\tstorj.io/drpc/drpcwire.SplitN:25\n\tstorj.io/drpc/drpcstream.(*Stream).RawWrite:199\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:231\n\tstorj.io/storj/pkg/pb.(*drpcPiecestoreDownloadStream).Send:905\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func3:565\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-10-29T00:42:51.155Z        INFO    piecestore      download failed {"Piece ID": "3YUPIMXWIA6H2UFMYPS3THD7RZV44UUJXPLWHEWBJQHYDMMUIWUA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET", "error": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:34192: write: broken pipe", "errorVerbose": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:34192: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:162\n\tstorj.io/drpc/drpcwire.SplitN:25\n\tstorj.io/drpc/drpcstream.(*Stream).RawWrite:199\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:231\n\tstorj.io/storj/pkg/pb.(*drpcPiecestoreDownloadStream).Send:905\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func3:565\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-10-29T00:52:51.294Z        INFO    piecestore      download failed {"Piece ID": "3YUPIMXWIA6H2UFMYPS3THD7RZV44UUJXPLWHEWBJQHYDMMUIWUA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET", "error": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:44670: write: broken pipe", "errorVerbose": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:44670: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:162\n\tstorj.io/drpc/drpcwire.SplitN:25\n\tstorj.io/drpc/drpcstream.(*Stream).RawWrite:199\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:231\n\tstorj.io/storj/pkg/pb.(*drpcPiecestoreDownloadStream).Send:905\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func3:565\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-10-29T01:17:51.173Z        INFO    piecestore      download failed {"Piece ID": "3YUPIMXWIA6H2UFMYPS3THD7RZV44UUJXPLWHEWBJQHYDMMUIWUA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET", "error": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:42654: write: broken pipe", "errorVerbose": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:42654: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:162\n\tstorj.io/drpc/drpcwire.SplitN:25\n\tstorj.io/drpc/drpcstream.(*Stream).RawWrite:199\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:231\n\tstorj.io/storj/pkg/pb.(*drpcPiecestoreDownloadStream).Send:905\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func3:565\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-10-29T01:32:50.990Z        INFO    piecestore      download failed {"Piece ID": "3YUPIMXWIA6H2UFMYPS3THD7RZV44UUJXPLWHEWBJQHYDMMUIWUA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET", "error": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:58346: write: broken pipe", "errorVerbose": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:58346: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:162\n\tstorj.io/drpc/drpcwire.SplitN:25\n\tstorj.io/drpc/drpcstream.(*Stream).RawWrite:199\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:231\n\tstorj.io/storj/pkg/pb.(*drpcPiecestoreDownloadStream).Send:905\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func3:565\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-10-29T04:02:51.307Z        INFO    piecestore      download failed {"Piece ID": "3YUPIMXWIA6H2UFMYPS3THD7RZV44UUJXPLWHEWBJQHYDMMUIWUA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET", "error": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:45890: write: broken pipe", "errorVerbose": "piecestore: piecestore protocol: write tcp 172.17.0.3:28967->34.67.235.103:45890: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:162\n\tstorj.io/drpc/drpcwire.SplitN:25\n\tstorj.io/drpc/drpcstream.(*Stream).RawWrite:199\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:231\n\tstorj.io/storj/pkg/pb.(*drpcPiecestoreDownloadStream).Send:905\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func3:565\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

Details about that piece:
docker logs storagenode 2>&1 | grep "3YUPIMXWIA6H2UFMYPS3THD7RZV44UUJXPLWHEWBJQHYDMMUIWUA"

Link to Pastebin, a lot of information for this post.

Is this normal that I see this on my node?

gracefulexit:chore running graceful exit chore.

I’ve never requested graceful exit
Node ID 1pXCvt4E3nczEXL1xpyyMjsNoSxf3yjfgg2AggiBo5KBJedDGK

This information message persists on all storagenodes.

docker logs storagenode 2>&1 | grep gracefulexit

Summary
2019-10-28T19:11:53.716Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T19:26:40.084Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T19:41:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T19:56:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T20:11:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T20:26:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T20:41:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T20:56:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T21:11:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T21:26:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T21:41:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T21:56:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T22:11:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T22:26:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T22:41:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T22:56:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T23:11:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T23:26:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T23:41:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-28T23:56:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T00:11:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T00:26:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T00:41:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T00:56:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T01:11:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T01:26:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T01:41:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T01:56:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T02:11:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T02:26:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T02:41:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T02:56:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T03:11:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T03:26:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T03:41:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T03:56:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T04:11:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T04:26:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T04:41:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T04:56:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T05:11:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T05:26:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T05:41:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T05:56:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T06:11:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T06:26:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T06:41:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T06:56:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T07:11:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T07:26:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T07:41:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T07:56:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T08:11:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T08:26:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T08:41:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T08:56:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T09:11:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T09:26:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T09:41:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T09:56:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T10:11:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T10:26:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T10:41:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T10:56:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T11:11:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.
2019-10-29T11:26:38.409Z        INFO    gracefulexit:chore      running graceful exit chore.

Thank you for the reply
But what does this message mean, should I worry or treat it as normal operation?

That is just an idle message from the graceful exit job. As long as you don’t call gracefull exit the job will skip all the time.

1 Like

Thank you for the answer