Changelog v0.22.1

Database Migration
We split the sqlite3 db into multiple databases. The migration will needs several minutes to hours. Doing that time you can’t open the dashboard. Please be patient and wait for it to finish. You can watch the progress in the logfile.

Windows Installer
We have an early version ready. It is not finished and not fully tested yet. So don’t use it for your storage nodes but if you feel like you want to be an early tester please take a look at Windows Installer early test version

Online Status
Storage nodes are sending a checkin message to the satellite once per hour. No other kademlia message that would reset the last contact on the dashboard. We have replaced it with an OFFLINE/ONLINE idicator. Online is everything that has received a message in the last 2 hours.
Vetted nodes should get messages frequently and the idicator should work fine. Unvetted nodes might see OFFLINE after a restart for a few minutes because we designed the checkin methode in a way that storage nodes will not send a message on every restart (don’t DDOS the satellite)

Repair Tests
The durability is working better than expected. Not a single repair was needed for a few month now. To make sure our calculations are correct we increase the repair threshold to force repair even if they are not needed. We expect the first repair traffic next week. Let us know if you see any strange behavior.

GRPC Replacement
By default all storage nodes will use GRPC. We found out that GRPC is to heavy for our needs. It has some interesting side effects that we didn’t expect. We replace it with our own solution that we call D(Dump)RPC. Our plan is to run DRPC tests and activate DRPC with a future release. Again let us know if you see any strange behavior especially around uploads and downloads. I hope that we can run the first DRPC tests with the current release.

7 Likes

It seems you storjlings are intent on keeping me busy. This will definitely require a good bit of rework for the earnings calculator. :slight_smile:

2 Likes

A post was split to a new topic: FATAL Unrecoverable error {“error”: “Error creating tables for master database on storagenode: migrate: storage node database error: migrate tables:: attempt to write a readonly database

Yes! That was the main reason for the change :smiley:

3 Likes

Only took me half an hour :stuck_out_tongue_winking_eye: you can do better!

Looks like there are quite a few great updates in v0.22.1 though. No more Kademlia debug mess in the logs!

My regular question, could you please update “Public Road Map”? :slight_smile:

1 Like

is V 0.22.1 update only for windows nodes

What made you come to that conclusion ?

the Mac OS node is still on V 0.21.4 15 hours after the windows node updated to V0.22.1

Watchtower command checks for updates every 6 hours if you notice the --interval flag

You should remove and add the watchtower as the older version 0.3.8 has been pushed.

docker stop -t 300 watchtower
docker rm watchtower

Then use the watchtower command given in support docs

To check the status of containers use

docker ps -a

well I did I manual update but cli dashboard will not load and the same goes for web dashboard2019-10-04T12:46:44.180Z INFO Configuration loaded from: /app/config/config.yaml
2019-10-04T12:46:44.223Z INFO Node ID: 1GYPBayDtCCPbrsARdCwPBLTz2S2MGNkdxQigoSnG7YLnEHPjp

I think you missed that part.

thanks patience is what it took

1 Like

why if last contact has been removed , it is still called “last contact” and not something like “status”?

1 Like

Sounds like an idea which you can post on ideas.storj.io :slight_smile:

1 Like

@littleskunk I and my router’s just say you Thank you very much! :slight_smile:
When Kademlia was removed, no ton of dns requests and tons of small packets :+1:

@littleskunk, what strange behavior would qualify? Can you give an example?

Would the upload started and upload failed instantly, qualify as strange behavior?

By default all storage nodes are still using grpc. Maybe 1% uploads and download will be drpc. So I would say if you notice a pattern that happens on most of the uploads and downloads that is probably not drpc. If you get a new kind of error message once in a while or frequently but in a limitted time window that could be drpc. In general I am interested in finding bugs early even if that means I have to deal with a few unrelated bugs. So don’t worry if you error might be unrelated. I can deal with that.

Looks like all my upload is grpc in the logs, no drpc at present.

@littleskunk
I have something with drpc:

docker logs storagenode 2>&1 | grep drpc

2019-10-05T18:06:43.441Z        INFO    piecestore      download failed {"Piece ID": "2BSARRXA6IKEJMBWW4UBA6LAPQVV4X2RA3532AI25JDUWO3EMDUA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET", "error": "piecestore: piecestore protocol: expected order as the message", "errorVerbose": "piecestore: piecestore protocol: expected order as the message\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func4:587\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:606\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Download:370\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func2:838\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:172\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:150"}
2019-10-06T12:14:48.924Z        INFO    piecestore      download failed {"Piece ID": "2BSARRXA6IKEJMBWW4UBA6LAPQVV4X2RA3532AI25JDUWO3EMDUA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET", "error": "piecestore: piecestore protocol: expected order as the message", "errorVerbose": "piecestore: piecestore protocol: expected order as the message\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func4:587\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:606\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Download:370\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func2:838\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:172\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:150"}
2019-10-06T12:17:31.226Z        INFO    piecestore      download failed {"Piece ID": "2BSARRXA6IKEJMBWW4UBA6LAPQVV4X2RA3532AI25JDUWO3EMDUA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET", "error": "piecestore: piecestore protocol: expected order as the message", "errorVerbose": "piecestore: piecestore protocol: expected order as the message\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func4:587\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:606\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Download:370\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func2:838\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:172\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:150"}
2019-10-07T11:27:42.291Z        INFO    piecestore      download failed {"Piece ID": "2BSARRXA6IKEJMBWW4UBA6LAPQVV4X2RA3532AI25JDUWO3EMDUA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET", "error": "piecestore: piecestore protocol: expected order as the message", "errorVerbose": "piecestore: piecestore protocol: expected order as the message\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func4:587\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:606\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Download:370\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func2:838\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:172\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:150"}
1 Like