Sudden continuous drop in traffic

[dimisjim]

Hey there,

I am a new operator, started 3 weeks ago.
I am already vetted in one of the satellites (saltlake), others are still in the process (around 20%).

Starting from April 4, I am seeing extremely low levels of traffic processed. From around 80GB per day it went down to less than 1GB

I still have around 200GB left in my disk.

Is this normal?

[nerdatwork]

This isn’t mining so you can’t expect a constant flow of data. Also you are new so your node takes time to be fully vetted (around a month) on each satellite.

[twl]

@dimisjim

Is this normal?

I’d say it’s not specifically uncommon and therefore no reason to be worried.

[dimisjim]

@twl this is still the case:

could it be that my drive has failed?
Node UI still says that I am online though

[deathlessdd]

Im seeing more egress right now then ingress so since your node is new no one is downloading anything off your node. So for you its expected and your still vetting so I wouldn’t expect it to be 100% flow every day.

[twl]

@dimisjim

could it be that my drive has failed?

Check your drive with the freeware CDI if you are concerned about it failing:

CrystalDiskInfo

Other than that, pls wait for vetting process to finish on all satellites and stay confident

You can also post the last 20 lines of your log file here, someone sure will check that out and might be able to give further advice.

[dimisjim]

@twl I am on linux

2020-04-08T19:12:34.166Z ERROR version running on not allowed/outdated version v0.34.6

^ this explains a lot

I upgraded to 1.1.1. Let’s see what happens now

[twl]

@dimisjim

^ this explains a lot

The power of logs, eh?

Good luck mate

[dimisjim]

Not really, I am keep getting errors like this now:

|2020-04-08T19:45:51.154Z|INFO|piecestore|upload started|{"Piece ID": "SLLN3KH2ARTVF643WE5ADJKB52VPITZI2RPQZKLRWIJC7XKWVCMA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 248277232000}|
|---|---|---|---|---|
|2020-04-08T19:45:51.997Z|INFO|piecestore|upload canceled|{"Piece ID": "SLLN3KH2ARTVF643WE5ADJKB52VPITZI2RPQZKLRWIJC7XKWVCMA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/pb/pbgrpc.init.0.func3:70\n\tstorj.io/common/rpc/rpcstatus.Wrap:77\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:452\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:987\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}|
|2020-04-08T19:46:18.849Z|INFO|piecestore|upload started|{"Piece ID": "7S5AWXFWW7OCREXNJWNOX4Y3NVRPMTZBMILDJREVVPPJRALVXRNA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT", "Available Space": 248276487808}|
|2020-04-08T19:46:19.319Z|INFO|piecestore|upload canceled|{"Piece ID": "7S5AWXFWW7OCREXNJWNOX4Y3NVRPMTZBMILDJREVVPPJRALVXRNA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/pb/pbgrpc.init.0.func3:70\n\tstorj.io/common/rpc/rpcstatus.Wrap:77\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:452\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:987\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}|
|2020-04-08T19:46:32.269Z|INFO|piecestore|upload started|{"Piece ID": "GL4HCRDAVHZO4CA6TQ5G55RVLCKKU74BTGHUMEVJSLX3FS3NXEXA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT", "Available Space": 248276468096}|
|2020-04-08T19:46:32.374Z|INFO|piecestore|upload canceled|{"Piece ID": "GL4HCRDAVHZO4CA6TQ5G55RVLCKKU74BTGHUMEVJSLX3FS3NXEXA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/pb/pbgrpc.init.0.func3:70\n\tstorj.io/common/rpc/rpcstatus.Wrap:77\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:452\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:987\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}|
|2020-04-08T19:47:14.063Z|INFO|piecestore|upload started|{"Piece ID": "4YHBMXNMP6BSDZSKKVK224ET4M6D7PF23WPGI63MLJ2KWLDT34OA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 248276280448}|
|2020-04-08T19:47:15.800Z|INFO|piecestore|upload canceled|{"Piece ID": "4YHBMXNMP6BSDZSKKVK224ET4M6D7PF23WPGI63MLJ2KWLDT34OA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/pb/pbgrpc.init.0.func3:70\n\tstorj.io/common/rpc/rpcstatus.Wrap:77\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:452\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:987\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}|
|2020-04-08T19:47:24.726Z|INFO|piecestore|upload started|{"Piece ID": "S5WB4F4NRGDJTHATV5BSJGJGICYHJJVT46VV37BV2YNDBLKPILBQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 248274143104}|
|2020-04-08T19:47:24.806Z|INFO|piecestore|upload canceled|{"Piece ID": "S5WB4F4NRGDJTHATV5BSJGJGICYHJJVT46VV37BV2YNDBLKPILBQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/pb/pbgrpc.init.0.func3:70\n\tstorj.io/common/rpc/rpcstatus.Wrap:77\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:452\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:987\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}|

[twl]

Use the forum search feature please. 3 secs of copypasting netted me this:

Context cancelled on all uploads/downloads

Hello, Since a few months i was running a node that was working perfectly, now few days ago its throwing “Context cancelled” errors massive, for uploads and for downloads, i tried to reinstall and rebuild the databases, and even no work, still the same problem, Its any Known Bugs or problems about this? I dont find any paste-service for paste the code, so i use Google Drive https://drive.google.com/open?id=1ZHmVlHBEGF2MU1ZJqkgstNX6bPjta78Z

[dimisjim]

Sure, but there is no definitive answer there.

Thanks for trying though

[BrightSilence]

It’s normal, some only see less than 10% of them succeed. It depends mostly on latency between your node and the customer. This includes your own disk IO. How the disks are connected to the node makes a big difference.