I never got an email this node was having any issues and it’s been online since v3 came out with 30+ TB of data on it.
I may be wrong, but judging by your online and audit score everything seems fine.
Lately I have seen that once in a while my suspesion score drops for no reason (online and audit score not changing at all and no downtime of any kind), and then usually recover to 100% within the following hours/day.
For some reason it appears as if suspension scores are being a bit erratic.
Are you still receiving ingress in this node? Are there any lines of interest in the logs? Has anything changed recently with your node?
The only thing I did was stop the node to up the storage allocated and then restart it. I noticed about 5 minutes of downtime for an update a few days ago.
Did you check the logs?
1 Like
Well, I am seeing errors now:
2024-08-05T22:16:21Z ERROR piecestore download failed {“Process”: “storagenode”, “Piece ID”: “XBGTGNJDEBLQBGECX52HI5EU6B3OEK6XKGVNBIK2FMPLAG5A4RCQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 0, “Size”: 13824, “Remote Address”: “172.17.0.1:43030”, “error”: “pieces error: filestore error: unable to open "config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/xb/gtgnjdeblqbgecx52hi5eu6b3oek6xkgvnbik2fmplag5a4rcq.sj1": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/xb/gtgnjdeblqbgecx52hi5eu6b3oek6xkgvnbik2fmplag5a4rcq.sj1: permission denied”, “errorVerbose”: “pieces error: filestore error: unable to open "config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/xb/gtgnjdeblqbgecx52hi5eu6b3oek6xkgvnbik2fmplag5a4rcq.sj1": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/xb/gtgnjdeblqbgecx52hi5eu6b3oek6xkgvnbik2fmplag5a4rcq.sj1: permission denied\n\tstorj.io/storj/storagenode/blobstore/filestore.(*Dir).Open:356\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).Open:94\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:302\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:719\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35”}
You should shutdown the node and fix the permission problem. Did you add the uid and gid part to your docker command? If so, then remove it or change the owner to all files accordingly.
1 Like
Yeah I’m changing permissions now, and apparently this NAS is also stuck trying to delete something for 10+ days now and is still at 0% so something on the backend is stuck. Fun times.
I think a fsck may be in order.
Deletions that stall, permissions that change by themselves…. sounds a bit suspicious for filesystem corruption.
1 Like
Yeah this thing is running like a dog right now. It’s a Synology DS 2422+ so it’s no slouch normally but I deleted a large backup task from it 2 weeks ago and it’s still processing stuck at 0%. Also disk scrubbing hasn’t run in 7 months so it’s probably time. I’m just waiting for these permissions to pass through and will restart this