Node suspended after increasing capacity

pascal · May 5, 2023, 5:17pm

Hi,

Last night I increase the capacity from 4 to 6to. This morning I found that my nodes are suspended.

I checked the errors audit and repair and that’s what I found.

2023-05-05T08:12:26.148902094Z	stdout	2023-05-05T08:12:18.666Z ERROR piecestore download failed {"Process": "storagenode", "Piece ID": "LI5P3HJI57WDUDSWWWMG5JTMUXTYYPGAINESQ4XEFW4RPOSQ6DHA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 0, "Remote Address": "172.17.0.1:58522", "error": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/li/5p3hji57wdudswwwmg5jtmuxtyypgainesq4xefw4rposq6dha.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/li/5p3hji57wdudswwwmg5jtmuxtyypgainesq4xefw4rposq6dha.sj1: permission denied", "errorVerbose": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/li/5p3hji57wdudswwwmg5jtmuxtyypgainesq4xefw4rposq6dha.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/li/5p3hji57wdudswwwmg5jtmuxtyypgainesq4xefw4rposq6dha.sj1: permission denied\n\tstorj.io/storj/storage/filestore.(*Dir).Open:285\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:77\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:282\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:646\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:251\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2023-05-05T08:12:15.624656166Z	stdout	2023-05-05T08:12:15.624Z ERROR piecestore download failed {"Process": "storagenode", "Piece ID": "TDYYA3434WOJPYFLMH7BXF3WDAXQWERV6V5AJ72UMBKAUV4DMBQA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 444928, "Size": 0, "Remote Address": "172.17.0.1:56664", "error": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/td/yya3434wojpyflmh7bxf3wdaxqwerv6v5aj72umbkauv4dmbqa.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/td/yya3434wojpyflmh7bxf3wdaxqwerv6v5aj72umbkauv4dmbqa.sj1: permission denied", "errorVerbose": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/td/yya3434wojpyflmh7bxf3wdaxqwerv6v5aj72umbkauv4dmbqa.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/td/yya3434wojpyflmh7bxf3wdaxqwerv6v5aj72umbkauv4dmbqa.sj1: permission denied\n\tstorj.io/storj/storage/filestore.(*Dir).Open:285\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:77\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:282\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:646\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:251\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2023-05-05T08:12:13.837308363Z	stdout	2023-05-05T08:12:13.836Z ERROR piecestore upload failed {"Process": "storagenode", "Piece ID": "2OTL5CMMVWGVITWLEUNJIJR5WCTRIF5IK2QKC3EC3RS5JZWFNDQQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT_REPAIR", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:75\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func5:498\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:529\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35", "Size": 2174464, "Remote Address": "172.17.0.1:50636"}
2023-05-05T08:11:57.988560972Z	stdout	2023-05-05T08:11:57.988Z ERROR piecestore download failed {"Process": "storagenode", "Piece ID": "WC33QXV33RDKPJ5M7DC4DONMUBG74KFP5VLKT6GGRDYBVXGBBHEA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 1100032, "Size": 0, "Remote Address": "172.17.0.1:35122", "error": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/wc/33qxv33rdkpj5m7dc4donmubg74kfp5vlkt6ggrdybvxgbbhea.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/wc/33qxv33rdkpj5m7dc4donmubg74kfp5vlkt6ggrdybvxgbbhea.sj1: permission denied", "errorVerbose": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/wc/33qxv33rdkpj5m7dc4donmubg74kfp5vlkt6ggrdybvxgbbhea.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/wc/33qxv33rdkpj5m7dc4donmubg74kfp5vlkt6ggrdybvxgbbhea.sj1: permission denied\n\tstorj.io/storj/storage/filestore.(*Dir).Open:285\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:77\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:282\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:646\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:251\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2023-05-05T08:11:56.624924049Z	stdout	2023-05-05T08:11:56.624Z ERROR piecestore download failed {"Process": "storagenode", "Piece ID": "UH4NREC3SYL4CU5Q7PCCXV6KRUUVCY2CTCXLTGVQUADYDK7RG45A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 716288, "Size": 0, "Remote Address": "172.17.0.1:58660", "error": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/uh/4nrec3syl4cu5q7pccxv6kruuvcy2ctcxltgvquadydk7rg45a.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/uh/4nrec3syl4cu5q7pccxv6kruuvcy2ctcxltgvquadydk7rg45a.sj1: permission denied", "errorVerbose": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/uh/4nrec3syl4cu5q7pccxv6kruuvcy2ctcxltgvquadydk7rg45a.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/uh/4nrec3syl4cu5q7pccxv6kruuvcy2ctcxltgvquadydk7rg45a.sj1: permission denied\n\tstorj.io/storj/storage/filestore.(*Dir).Open:285\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:77\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:282\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:646\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:251\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2023-05-05T08:11:53.405499873Z	stdout	2023-05-05T08:11:53.405Z ERROR piecestore upload failed {"Process": "storagenode", "Piece ID": "RKWQ5Q4TFUQFLRMG4DVBXQSM76KK3WJZFSJNDAAW3WRHLWQ6RXVA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:75\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func5:498\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:529\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35", "Size": 181504, "Remote Address": "172.17.0.1:35122"}
2023-05-05T08:11:51.726640647Z	stdout	2023-05-05T08:11:51.726Z ERROR piecestore download failed {"Process": "storagenode", "Piece ID": "KASUBFGMKDZLZCXTOZPPUEZTY7Z73D7Q4W3B2C54WV7F3XCTYIGQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 2063104, "Size": 0, "Remote Address": "172.17.0.1:37120", "error": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/ka/subfgmkdzlzcxtozppuezty7z73d7q4w3b2c54wv7f3xctyigq.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/ka/subfgmkdzlzcxtozppuezty7z73d7q4w3b2c54wv7f3xctyigq.sj1: permission denied", "errorVerbose": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/ka/subfgmkdzlzcxtozppuezty7z73d7q4w3b2c54wv7f3xctyigq.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/ka/subfgmkdzlzcxtozppuezty7z73d7q4w3b2c54wv7f3xctyigq.sj1: permission denied\n\tstorj.io/storj/storage/filestore.(*Dir).Open:285\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:77\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:282\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:646\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:251\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2023-05-05T08:11:51.590732628Z	stdout	2023-05-05T08:11:51.532Z ERROR piecestore upload failed {"Process": "storagenode", "Piece ID": "C3ZH4YWHQEEOONP565BG7NWZIHDSI7O45FAS2RFRZCDNHTWYOMSQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT_REPAIR", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:75\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func5:498\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:529\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35", "Size": 1048576, "Remote Address": "172.17.0.1:50648"}
2023-05-05T08:11:51.590545364Z	stdout	2023-05-05T08:11:51.377Z ERROR piecestore download failed {"Process": "storagenode", "Piece ID": "KASUBFGMKDZLZCXTOZPPUEZTY7Z73D7Q4W3B2C54WV7F3XCTYIGQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 2061056, "Size": 0, "Remote Address": "172.17.0.1:37120", "error": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/ka/subfgmkdzlzcxtozppuezty7z73d7q4w3b2c54wv7f3xctyigq.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/ka/subfgmkdzlzcxtozppuezty7z73d7q4w3b2c54wv7f3xctyigq.sj1: permission denied", "errorVerbose": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/ka/subfgmkdzlzcxtozppuezty7z73d7q4w3b2c54wv7f3xctyigq.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/ka/subfgmkdzlzcxtozppuezty7z73d7q4w3b2c54wv7f3xctyigq.sj1: permission denied\n\tstorj.io/storj/storage/filestore.(*Dir).Open:285\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:77\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:282\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:646\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:251\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2023-05-05T08:11:49.748788616Z	stdout	2023-05-05T08:11:49.501Z ERROR piecestore upload failed {"Process": "storagenode", "Piece ID": "U5ZMDDNVBITBCZ776B53HA5PLL3FHYAEOPHXMJMREGSGKAMOHYLA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:75\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func5:498\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:504\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35", "Size": 512, "Remote Address": "172.17.0.1:35122"}
2023-05-05T08:11:48.115752452Z	stdout	2023-05-05T08:11:48.115Z ERROR piecestore download failed {"Process": "storagenode", "Piece ID": "C5UQ6W46X2DIOHMVGOO4UB4GKIRJLZSK5RIV7UYJMNDYZBIU6MEA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Offset": 1170944, "Size": 0, "Remote Address": "172.17.0.1:46722", "error": "pieces error: filestore error: unable to open \"config/storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/c5/uq6w46x2diohmvgoo4ub4gkirjlzsk5riv7uyjmndyzbiu6mea.sj1\": open config/storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/c5/uq6w46x2diohmvgoo4ub4gkirjlzsk5riv7uyjmndyzbiu6mea.sj1: permission denied", "errorVerbose": "pieces error: filestore error: unable to open \"config/storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/c5/uq6w46x2diohmvgoo4ub4gkirjlzsk5riv7uyjmndyzbiu6mea.sj1\": open config/storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/c5/uq6w46x2diohmvgoo4ub4gkirjlzsk5riv7uyjmndyzbiu6mea.sj1: permission denied\n\tstorj.io/storj/storage/filestore.(*Dir).Open:285\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:77\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:282\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:646\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:251\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2023-05-05T08:11:45.156065001Z	stdout	2023-05-05T08:11:42.805Z ERROR piecestore upload failed {"Process": "storagenode", "Piece ID": "K5QRMHLQQ573MSPBQAZHK3LIPUXLDTISL67PTDGZ2Q43QYKO6QNA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:75\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func5:498\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:529\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35", "Size": 183808, "Remote Address": "172.17.0.1:50632"}
2023-05-05T08:11:42.766422987Z	stdout	2023-05-05T08:11:42.765Z ERROR piecestore upload failed {"Process": "storagenode", "Piece ID": "XLYRF7YGQW5DFAC7CT4C5GDUOVHR6GYDEU34CAQY3WDUZGHMWKLQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:75\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func5:498\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:529\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35", "Size": 181504, "Remote Address": "172.17.0.1:35122"}
2023-05-05T08:11:36.979461175Z	stdout	2023-05-05T08:11:36.979Z ERROR piecestore download failed {"Process": "storagenode", "Piece ID": "7EDUFWMF53TL32ZWA7WV5LY36MH2FLSAI743V3DAYRHCGM3VXMEQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 0, "Remote Address": "172.17.0.1:50630", "error": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/7e/dufwmf53tl32zwa7wv5ly36mh2flsai743v3dayrhcgm3vxmeq.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/7e/dufwmf53tl32zwa7wv5ly36mh2flsai743v3dayrhcgm3vxmeq.sj1: permission denied", "errorVerbose": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/7e/dufwmf53tl32zwa7wv5ly36mh2flsai743v3dayrhcgm3vxmeq.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/7e/dufwmf53tl32zwa7wv5ly36mh2flsai743v3dayrhcgm3vxmeq.sj1: permission denied\n\tstorj.io/storj/storage/filestore.(*Dir).Open:285\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:77\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:282\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:646\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:251\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2023-05-05T08:11:33.410315259Z	stdout	2023-05-05T08:11:33.409Z ERROR piecestore upload failed {"Process": "storagenode", "Piece ID": "6ORIEUMPIJUXLNALWRWYKW5JSRKTD6V774XF4DTWOOWC5Y57PW5Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:75\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func5:498\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:529\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35", "Size": 584448, "Remote Address": "172.17.0.1:56664"}
2023-05-05T08:11:32.061643257Z	stdout	2023-05-05T08:11:32.061Z ERROR piecestore upload failed {"Process": "storagenode", "Piece ID": "HAWRKHUNXMRLPZHB5YSQZIGVWCHUEJKVNNO52BE7PP57MEKPLGDA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT_REPAIR", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:75\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func5:498\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:529\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35", "Size": 399104, "Remote Address": "172.17.0.1:50608"}
2023-05-05T08:11:26.356337219Z	stdout	2023-05-05T08:11:25.392Z ERROR piecestore upload failed {"Process": "storagenode", "Piece ID": "YD4J2CYGBLQGLAPMAO2FXRD2GYV6GKBHNTN63YM53TO5

2023-05-05T08:10:54.647365663Z	stdout	2023-05-05T08:10:53.858Z ERROR piecedeleter could not send delete piece to trash {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "2INKSCZEKTQ5VZCXWUS2IIE7F3PMGAFH4CPROPD2UPP3OVP3APYA", "error": "pieces error: filestore error: chtimes config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/2i/nksczektq5vzcxwus2iie7f3pmgafh4cpropd2upp3ovp3apya.sj1: operation not permitted", "errorVerbose": "pieces error: filestore error: chtimes config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/2i/nksczektq5vzcxwus2iie7f3pmgafh4cpropd2upp3ovp3apya.sj1: operation not permitted\n\tstorj.io/storj/storage/filestore.(*blobStore).Trash:142\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Trash:295\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:388\n\tstorj.io/storj/storagenode/pieces.(*Deleter).deleteOrTrash:185\n\tstorj.io/storj/storagenode/pieces.(*Deleter).work:135\n\tstorj.io/storj/storagenode/pieces.(*Deleter).Run.func1:72\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-05-05T08:10:54.647232587Z	stdout	2023-05-05T08:10:53.768Z ERROR piecestore upload failed {"Process": "storagenode", "Piece ID": "JZ7TWP2I4SVNDZVAHVV2XWSIRWLRNJJE3Z7CLZIDITSC3IF27N2Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:75\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func5:498\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:529\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35", "Size": 181504, "Remote Address": "172.17.0.1:46722"}
2023-05-05T08:10:53.677319031Z	stdout	2023-05-05T08:10:53.676Z ERROR piecestore upload failed {"Process": "storagenode", "Piece ID": "4HLV4Q2OM77T4KIL26ZLZ6EYH2CQQ2C4GGRHTONLQEBIHLS2RL5Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:75\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func5:498\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:529\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35", "Size": 181504, "Remote Address": "172.17.0.1:56868"}
2023-05-05T08:10:48.071887979Z	stdout	2023-05-05T08:10:48.071Z ERROR piecedeleter could not send delete piece to trash {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "KIK6AC3VOPLC6RFZMBF4AKLQLLI4E5PATR3GFLQG4YBV4HAKI5UQ", "error": "pieces error: filestore error: chtimes config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/ki/k6ac3voplc6rfzmbf4aklqlli4e5patr3gflqg4ybv4haki5uq.sj1: operation not permitted", "errorVerbose": "pieces error: filestore error: chtimes config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/ki/k6ac3voplc6rfzmbf4aklqlli4e5patr3gflqg4ybv4haki5uq.sj1: operation not permitted\n\tstorj.io/storj/storage/filestore.(*blobStore).Trash:142\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Trash:295\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:388\n\tstorj.io/storj/storagenode/pieces.(*Deleter).deleteOrTrash:185\n\tstorj.io/storj/storagenode/pieces.(*Deleter).work:135\n\tstorj.io/storj/storagenode/pieces.(*Deleter).Run.func1:72\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-05-05T08:10:45.572440958Z	stdout	2023-05-05T08:10:45.551Z ERROR piecestore upload failed {"Process": "storagenode", "Piece ID": "C7HQ6WFI3F7DM7S3QHV7K4ATTJ2ITA2TM4OHWI5XHM2RHAOUSP3Q", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT_REPAIR", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:75\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func5:498\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:529\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35", "Size": 9472, "Remote Address": "172.17.0.1:50524"}
2023-05-05T08:10:45.410530674Z	stdout	2023-05-05T08:10:45.410Z ERROR piecestore upload failed {"Process": "storagenode", "Piece ID": "7F5L5RGZU5DD67RAGGHTUDGY4BSDJGKLWXCWSANCIH7RCQMZR6IA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:75\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func5:498\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:504\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35", "Size": 1280, "Remote Address": "172.17.0.1:58026"}
2023-05-05T08:10:42.684903458Z	stdout	2023-05-05T08:10:42.684Z ERROR piecestore download failed {"Process": "storagenode", "Piece ID": "U3IFBJHUWZ7PU4SRBCICI7JDK5PHRQ7UFH4QDWKIKOKXF33JDBZQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET_AUDIT", "Offset": 1588224, "Size": 0, "Remote Address": "172.17.0.1:50518", "error": "pieces error: filestore error: unable to open \"config/storage/blobs/qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa/u3/ifbjhuwz7pu4srbcici7jdk5phrq7ufh4qdwkikokxf33jdbzq.sj1\": open config/storage/blobs/qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa/u3/ifbjhuwz7pu4srbcici7jdk5phrq7ufh4qdwkikokxf33jdbzq.sj1: permission denied", "errorVerbose": "pieces error: filestore error: unable to open \"config/storage/blobs/qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa/u3/ifbjhuwz7pu4srbcici7jdk5phrq7ufh4qdwkikokxf33jdbzq.sj1\": open config/storage/blobs/qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa/u3/ifbjhuwz7pu4srbcici7jdk5phrq7ufh4qdwkikokxf33jdbzq.sj1: permission denied\n\tstorj.io/storj/storage/filestore.(*Dir).Open:285\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:77\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:282\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:646\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:251\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2023-05-05T08:10:40.044355451Z	stdout	2023-05-05T08:10:40.043Z ERROR piecestore upload failed {"Process": "storagenode", "Piece ID": "IA4PZ4YQYLPK5SW4OZRJ4SF5OQYS55YMNONGVKFVGGBXFGSNY7WA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT_REPAIR", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:75\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func5:498\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:529\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35", "Size": 768, "Remote Address": "172.17.0.1:50508"}

So the message that comes back a lot is “filestore error: unable to open "config/storage/blobs”. But not sure if it’s relevant…

Is is related to last night when I restarted my docker container ?

Thank you,
Pasc

labooll · May 5, 2023, 5:49pm

Hello @pascal

When you say that you increased the capacity of the node, what did you do exactly ? (just increase the value in the docker command ? extend a disk partition ? …)

Are you able to access to the files via the terminal ?
Is you path correctly mounted ?

Regards,

pascal · May 5, 2023, 6:04pm

I just did stop, then remove the container then

docker run -d --restart unless-stopped --stop-timeout 300
-p 28967:28967/tcp
-p 28967:28967/udp
-p 14002:14002
-e WALLET=“0x************”
-e EMAIL=“@gmail.com"
-e ADDRESS="*:28967”
-e STORAGE=“6TB”
–user $(id -u):$(id -g)
–mount type=bind,source=“/volume1/docker/storj/identity”,destination=/app/identity
–mount type=bind,source=“/volume1/docker/storj/data”,destination=/app/config
–name storagenode storjlabs/storagenode:latest

From 4 to 6…
Not sure again it’s related.

labooll · May 5, 2023, 6:08pm

did you used previously the -user argument to run the node ?
As the error message relate to a permission issue it could be related.

could you a ls -l on directory /volume1/docker/storj/data and check the user and group permissions ?

pascal · May 5, 2023, 7:04pm

Very good question, maybe no.

should a remove the container and start it again without the -user argument ?

labooll · May 5, 2023, 8:21pm

yes you can try like this it and monitor the logs to check if it solved the issue

donald.m.motsinger · May 5, 2023, 9:37pm

I would suggest to put your docker command in a bash script and run this every time you start the node. This way you use the same command. Have a look through the bash history and you might find how you started the node in the past.

pascal · May 5, 2023, 9:47pm

You are so right. That was the problem lol. I had 2 saves of the command. One with user, the other one without. And I coudn’t figure out which one I did last time… You know the story, I just used the wrong one.

So far it seems ok now. How long before the suspension to go away ?

Pasc

mars_9t · May 5, 2023, 10:20pm

Make sure to remove the wrong command from saved ones or at least mark good one with something like “working as of 2023.05.05”. “Future you” will thank “current you” a lot for doing that

pascal · May 6, 2023, 2:55am

Ahah, first thing I did actually. So far half of my nodes are back and the score are going higher.
Thank you for your help, it worked.