Could someone help to view my node status.
my node is almost 1 month old.
Is this normal?
NO! I’d stop your node immediately
1 Like
if i was to hazard a guess you are most likely exporting your logs and adding something to them so that successrate script cannot read it right… and then your node has just finished its vetting period and is entering the network…
but like nerdatwork says… we will need some log data to really get any clear idea about whats going on with it… but if the audits are 100% on all satellites then i would think it’s your logs that are facked, because the successrate output doesn’t make sense… but you should surely investigate it.
maybe @BrightSilence should look at this, if this is the log being wrong, there might be some minor improvement that could make it so the script would simply state that it cannot read the logs correctly, or maybe just shutdown the node on a horrible result like this lol
i find it telling tho that the dashboard shows data coming in… but the audits are just 0 across the board which doesn’t make sense…
It looks like the storage mount point has been unmounted… and the node is receiving data, but may soon become DQ-ed as a result of not being able to read past uploads for Audits.
Stop the node… and make sure the mount point for the storage drive is properly connected and accessible.
if it had lost the mount wouldn’t it show failed audits…?. and didn’t they actually implement a software security feature against losing access to the storagenode data in the last update, which should shutdown the node…
In that case I would expect to see no upload errors. For that reason I would say the script is working very good here.
Well with version v1.11 that shouldn’t be possible. The filesystem has to be mounted at least read only (write check was introduced with v1.12). If the filesystem gets unmounted the storage node should go into a crash loop. I wouldn’t expect to see the storage node dashboard in that situation.
Now I wonder if a storage node with corrupted sqlite db would still accept data. I believe yes and would place my bet on a read only filesystem / wrong access grants.
2 Likes
i have seen the successrate script go pretty crazy just like this previously when somebody had added like a ( or something like that to the front of each log line during export… it made it miss some stuff and still keep tract over others… but yeah i duno… just my guess
or system clock out of sync or satellite untrusted. These 2 errors would also fit here.
1 Like
Could be…
Yeah… I think you’re probably correct.
But the end result might be the same…
With a read only filesystem the storage node should still be successful on audits but not getting paid for any download traffic inclusive audits and repair.
With a system clock out of sync I would expect to see 100% upload failures and no cancelation. So my bet is still on read only filesystem / wrong permissions.
I am not willing to bet your held back amount on it. So better stop the node first and then look into the logfiles. 
1 Like
This is definitely not the case here. It would show 0’s everywhere if the file was empty, throw an error if the log couldn’t be read or didn’t exist. There are logs, they are just full of errors. My guess is that the path is write protected either because the user running the node doesn’t have the correct permissions or the file system has been mounted read only because of errors found. (just noticed this was already mentioned)
well i checked the old post with a log that looked weird… it does look like added data in the log lines only explain why some of the stats are are blank… 0
@BrightSilence
if it was write tho shouldn’t it show successful audits… ofc that may be if he added something to the log … maybe it’s multiple problems in this one…
the guy with blank log stats for some of the data did have fine and correct numbers in those that did show…, so yeah his logs must be a mess
Not if there is no data on the node.
would that be because it doesn’t have access to anything aside from the identity?
Who knows, lets not speculate further until @bossbambkk shows the actual logs. Pretty hard to figure it out with just counts. But only uploads and only failures suggests to me an empty node (even according to the satellites, otherwise you’d see failed downloads) and a unwriteable storage location.
2020-09-22T12:32:51.601+0700 INFO piecestore upload started {“Piece ID”: “MONUWFQRYC5KA4UJF7EZMZKXK2P2YWZ7FEVRL2HHZ4TTDINIINXA”, “Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “Action”: “PUT”, “Available Space”: 1455275813120}
2020-09-22T12:32:51.643+0700 ERROR piecestore upload failed {“Piece ID”: “MONUWFQRYC5KA4UJF7EZMZKXK2P2YWZ7FEVRL2HHZ4TTDINIINXA”, “Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “Action”: “PUT”, “error”: “pieces error: filestore error: chmod E:\temp\blob-471734754.partial: Access is denied.”, “errorVerbose”: “pieces error: filestore error: chmod E:\temp\blob-471734754.partial: Access is denied.\n\tstorj.io/storj/storage/filestore.(*blobWriter).Commit:120\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit.func1:130\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit:198\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:383\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:996\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-09-22T12:33:15.776+0700 INFO piecestore upload started {“Piece ID”: “M2SJTWN6SU6K3CHWJLFIEH5ZGIJCAAZA6C2UUW6HHYM6EIBHJTHA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT_REPAIR”, “Available Space”: 1455275811840}
2020-09-22T12:33:21.102+0700 INFO piecestore upload started {“Piece ID”: “D7VEQI462Z6BH4ALMLCIGTMNSYOWTKDIZZBDBEVIMTBT6AGADUXA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “PUT_REPAIR”, “Available Space”: 1455275811840}
2020-09-22T12:33:21.127+0700 ERROR piecestore upload failed {“Piece ID”: “D7VEQI462Z6BH4ALMLCIGTMNSYOWTKDIZZBDBEVIMTBT6AGADUXA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “PUT_REPAIR”, “error”: “pieces error: filestore error: chmod E:\temp\blob-490898788.partial: Access is denied.”, “errorVerbose”: “pieces error: filestore error: chmod E:\temp\blob-490898788.partial: Access is denied.\n\tstorj.io/storj/storage/filestore.(*blobWriter).Commit:120\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit.func1:130\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit:198\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:383\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:996\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-09-22T12:33:25.308+0700 INFO piecestore upload started {“Piece ID”: “Q7VQZFD4AVOSFR7SKC37TQI2HRUDIJDYRIYLFZZ7NUM5OC2XLXSA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Available Space”: 1455275810048}
2020-09-22T12:33:25.558+0700 ERROR piecestore upload failed {“Piece ID”: “Q7VQZFD4AVOSFR7SKC37TQI2HRUDIJDYRIYLFZZ7NUM5OC2XLXSA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “error”: “pieces error: filestore error: chmod E:\temp\blob-085903219.partial: Access is denied.”, “errorVerbose”: “pieces error: filestore error: chmod E:\temp\blob-085903219.partial: Access is denied.\n\tstorj.io/storj/storage/filestore.(*blobWriter).Commit:120\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit.func1:130\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit:198\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:383\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:996\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-09-22T12:33:30.002+0700 INFO piecestore upload started {“Piece ID”: “6L2RFXGQNHAKSPR567FNLSWH6ZMET6NFGZEEGOLQA2FLLWIPQAYA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “PUT_REPAIR”, “Available Space”: 1455275797248}
2020-09-22T12:33:30.168+0700 INFO piecestore upload started {“Piece ID”: “H3KCU6S7NF72473ZEOTY5KRURD4AOJBDCJ7CTJGXDKGDCQWPESQA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT_REPAIR”, “Available Space”: 1455275797248}
2020-09-22T12:33:30.195+0700 ERROR piecestore upload failed {“Piece ID”: “H3KCU6S7NF72473ZEOTY5KRURD4AOJBDCJ7CTJGXDKGDCQWPESQA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT_REPAIR”, “error”: “pieces error: filestore error: chmod E:\temp\blob-627975197.partial: Access is denied.”, “errorVerbose”: “pieces error: filestore error: chmod E:\temp\blob-627975197.partial: Access is denied.\n\tstorj.io/storj/storage/filestore.(*blobWriter).Commit:120\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit.func1:130\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit:198\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:383\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:996\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-09-22T12:33:39.091+0700 ERROR piecestore upload failed {“Piece ID”: “M2SJTWN6SU6K3CHWJLFIEH5ZGIJCAAZA6C2UUW6HHYM6EIBHJTHA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT_REPAIR”, “error”: “pieces error: filestore error: chmod E:\temp\blob-002851289.partial: Access is denied.”, “errorVerbose”: “pieces error: filestore error: chmod E:\temp\blob-002851289.partial: Access is denied.\n\tstorj.io/storj/storage/filestore.(*blobWriter).Commit:120\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit.func1:130\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit:198\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:383\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:996\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-09-22T12:33:45.423+0700 ERROR nodestats:cache Get stats query failed {“error”: “node stats service error: unable to connect to the satellite 118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW: rpccompat: context deadline exceeded”, “errorVerbose”: “node stats service error: unable to connect to the satellite 118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW: rpccompat: context deadline exceeded\n\tstorj.io/storj/storagenode/nodestats.(*Service).dial:156\n\tstorj.io/storj/storagenode/nodestats.(*Service).GetReputationStats:66\n\tstorj.io/storj/storagenode/nodestats.(*Cache).CacheReputationStats.func1:153\n\tstorj.io/storj/storagenode/nodestats.(*Cache).satelliteLoop:259\n\tstorj.io/storj/storagenode/nodestats.(*Cache).CacheReputationStats:152\n\tstorj.io/storj/storagenode/nodestats.(*Cache).Run.func2:119\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2020-09-22T12:33:46.023+0700 ERROR piecestore upload failed {“Piece ID”: “6L2RFXGQNHAKSPR567FNLSWH6ZMET6NFGZEEGOLQA2FLLWIPQAYA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “PUT_REPAIR”, “error”: “pieces error: filestore error: chmod E:\temp\blob-852792374.partial: Access is denied.”, “errorVerbose”: “pieces error: filestore error: chmod E:\temp\blob-852792374.partial: Access is denied.\n\tstorj.io/storj/storage/filestore.(*blobWriter).Commit:120\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit.func1:130\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit:198\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:383\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:996\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-09-22T12:33:50.942+0700 INFO piecestore upload started {“Piece ID”: “ZYVIFKIOYDVLMQQXFPSY2VFMNCBU6XCKZZ4BAEQA4VINVWC6AUNQ”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT_REPAIR”, “Available Space”: 1455271159296}
2020-09-22T12:34:12.551+0700 INFO piecestore upload started {“Piece ID”: “F23JJPV5DEYWEHPGZOOVGDKABX7TJED64UZCARVI6TTSY2KHTXZQ”, “Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “Action”: “PUT”, “Available Space”: 1455271159296}
2020-09-22T12:34:12.595+0700 ERROR piecestore upload failed {“Piece ID”: “F23JJPV5DEYWEHPGZOOVGDKABX7TJED64UZCARVI6TTSY2KHTXZQ”, “Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “Action”: “PUT”, “error”: “pieces error: filestore error: chmod E:\temp\blob-427353687.partial: Access is denied.”, “errorVerbose”: “pieces error: filestore error: chmod E:\temp\blob-427353687.partial: Access is denied.\n\tstorj.io/storj/storage/filestore.(*blobWriter).Commit:120\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit.func1:130\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit:198\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:383\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:996\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-09-22T12:34:13.390+0700 ERROR piecestore upload failed {“Piece ID”: “ZYVIFKIOYDVLMQQXFPSY2VFMNCBU6XCKZZ4BAEQA4VINVWC6AUNQ”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT_REPAIR”, “error”: “pieces error: filestore error: chmod E:\temp\blob-561495768.partial: Access is denied.”, “errorVerbose”: “pieces error: filestore error: chmod E:\temp\blob-561495768.partial: Access is denied.\n\tstorj.io/storj/storage/filestore.(*blobWriter).Commit:120\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit.func1:130\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit:198\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:383\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:996\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-09-22T12:34:17.989+0700 INFO piecestore upload started {“Piece ID”: “25Y57THXXM3SIDJPBOW6TQCP4OCWBHEHC63ZBR5TQNYKTQHQW4DQ”, “Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “Action”: “PUT”, “Available Space”: 1455268838144}
2020-09-22T12:34:18.013+0700 ERROR piecestore upload failed {“Piece ID”: “25Y57THXXM3SIDJPBOW6TQCP4OCWBHEHC63ZBR5TQNYKTQHQW4DQ”, “Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “Action”: “PUT”, “error”: “pieces error: filestore error: chmod E:\temp\blob-210293706.partial: Access is denied.”, “errorVerbose”: “pieces error: filestore error: chmod E:\temp\blob-210293706.partial: Access is denied.\n\tstorj.io/storj/storage/filestore.(*blobWriter).Commit:120\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit.func1:130\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit:198\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:383\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:996\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-09-22T12:34:29.053+0700 INFO piecestore upload started {“Piece ID”: “5CJEME3LTQ533MZP7V363D43P55L6Q2742K54BNZO4NE2LM3A2AQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT_REPAIR”, “Available Space”: 1455268836864}
2020-09-22T12:35:05.749+0700 INFO piecestore upload started {“Piece ID”: “VQ6LJRXKIQC3GOWHERNMAP6WUPB7PVIPHKSCR5AHLGPYP5IBO7RA”, “Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “Action”: “PUT”, “Available Space”: 1455268836864}
2020-09-22T12:35:05.791+0700 ERROR piecestore upload failed {“Piece ID”: “VQ6LJRXKIQC3GOWHERNMAP6WUPB7PVIPHKSCR5AHLGPYP5IBO7RA”, “Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “Action”: “PUT”, “error”: “pieces error: filestore error: chmod E:\temp\blob-871403916.partial: Access is denied.”, “errorVerbose”: “pieces error: filestore error: chmod E:\temp\blob-871403916.partial: Access is denied.\n\tstorj.io/storj/storage/filestore.(*blobWriter).Commit:120\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit.func1:130\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit:198\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:383\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:996\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
here is my log.
You should rather use this command to show last 20 lines of the log instead of downloading 360MB file.
1 Like