Database Migration
We split the sqlite3 db into multiple databases. The migration will needs several minutes to hours. Doing that time you can’t open the dashboard. Please be patient and wait for it to finish. You can watch the progress in the logfile.
Windows Installer
We have an early version ready. It is not finished and not fully tested yet. So don’t use it for your storage nodes but if you feel like you want to be an early tester please take a look at Windows Installer early test version
Online Status
Storage nodes are sending a checkin message to the satellite once per hour. No other kademlia message that would reset the last contact on the dashboard. We have replaced it with an OFFLINE/ONLINE idicator. Online is everything that has received a message in the last 2 hours.
Vetted nodes should get messages frequently and the idicator should work fine. Unvetted nodes might see OFFLINE after a restart for a few minutes because we designed the checkin methode in a way that storage nodes will not send a message on every restart (don’t DDOS the satellite)
Repair Tests
The durability is working better than expected. Not a single repair was needed for a few month now. To make sure our calculations are correct we increase the repair threshold to force repair even if they are not needed. We expect the first repair traffic next week. Let us know if you see any strange behavior.
GRPC Replacement
By default all storage nodes will use GRPC. We found out that GRPC is to heavy for our needs. It has some interesting side effects that we didn’t expect. We replace it with our own solution that we call D(Dump)RPC. Our plan is to run DRPC tests and activate DRPC with a future release. Again let us know if you see any strange behavior especially around uploads and downloads. I hope that we can run the first DRPC tests with the current release.
well I did I manual update but cli dashboard will not load and the same goes for web dashboard2019-10-04T12:46:44.180Z INFO Configuration loaded from: /app/config/config.yaml
2019-10-04T12:46:44.223Z INFO Node ID: 1GYPBayDtCCPbrsARdCwPBLTz2S2MGNkdxQigoSnG7YLnEHPjp
By default all storage nodes are still using grpc. Maybe 1% uploads and download will be drpc. So I would say if you notice a pattern that happens on most of the uploads and downloads that is probably not drpc. If you get a new kind of error message once in a while or frequently but in a limitted time window that could be drpc. In general I am interested in finding bugs early even if that means I have to deal with a few unrelated bugs. So don’t worry if you error might be unrelated. I can deal with that.
2019-10-05T18:06:43.441Z INFO piecestore download failed {"Piece ID": "2BSARRXA6IKEJMBWW4UBA6LAPQVV4X2RA3532AI25JDUWO3EMDUA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET", "error": "piecestore: piecestore protocol: expected order as the message", "errorVerbose": "piecestore: piecestore protocol: expected order as the message\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func4:587\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:606\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Download:370\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func2:838\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:172\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:150"}
2019-10-06T12:14:48.924Z INFO piecestore download failed {"Piece ID": "2BSARRXA6IKEJMBWW4UBA6LAPQVV4X2RA3532AI25JDUWO3EMDUA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET", "error": "piecestore: piecestore protocol: expected order as the message", "errorVerbose": "piecestore: piecestore protocol: expected order as the message\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func4:587\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:606\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Download:370\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func2:838\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:172\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:150"}
2019-10-06T12:17:31.226Z INFO piecestore download failed {"Piece ID": "2BSARRXA6IKEJMBWW4UBA6LAPQVV4X2RA3532AI25JDUWO3EMDUA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET", "error": "piecestore: piecestore protocol: expected order as the message", "errorVerbose": "piecestore: piecestore protocol: expected order as the message\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func4:587\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:606\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Download:370\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func2:838\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:172\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:150"}
2019-10-07T11:27:42.291Z INFO piecestore download failed {"Piece ID": "2BSARRXA6IKEJMBWW4UBA6LAPQVV4X2RA3532AI25JDUWO3EMDUA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET", "error": "piecestore: piecestore protocol: expected order as the message", "errorVerbose": "piecestore: piecestore protocol: expected order as the message\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func4:587\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:606\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Download:370\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func2:838\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:172\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:150"}