that’s what i’m asking in this post XD. I can see that the container is up, the port 28967 is working and listening, but the port 14002 is not working. And this have happened to me twice since last 2 weeks.
The first one solved by restarting the node, but the restart took about more than one hour, and I don’t know why.
So i’m asking if is something wrong with this version which match with these issues.
Alexei posted one solution but is related to loading screen, which is not the same issue.
Its impossible to tell exactly what the issue could be, I can’t see your setup or how you configured it, If you have a firewall setup on the system. Have you checked to see if you have anything in your logs?
Have you updated recently? the OS not the node. Im guessing your running linux?
No.
All available options at the moment:
I have checked the logs:
2020-10-23T03:39:33.554Z ERROR piecestore upload failed {"Piece ID": "34TITP5S65DJN6SNEQWIISSDHYGCUBIMK3S5TIVY55AXQMKEQMMA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT_REPAIR", "error": "pieces error: filestore error: sync config/storage/temp/blob-447106488.partial: read-only file system; chmod config/storage/temp/blob-447106488.partial: read-only file system; remove config/storage/temp/blob-447106488.partial: read-only file system", "errorVerbose": "pieces error: filestore error: sync config/storage/temp/blob-447106488.partial: read-only file system; chmod config/storage/temp/blob-447106488.partial: read-only file system; remove config/storage/temp/blob-447106488.partial: read-only file system\n\tstorj.io/storj/storage/filestore.(*blobWriter).Commit:120\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit.func1:130\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit:198\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:383\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:996\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-10-23T03:39:33.560Z ERROR piecestore failed to add order {"error": "order: open config/orders/unsent/unsent-orders-1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE-1603411200000000000: read-only file system", "errorVerbose": "order: open config/orders/unsent/unsent-orders-1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE-1603411200000000000: read-only file system\n\tstorj.io/storj/storagenode/orders.(*FileStore).getUnsentFile:418\n\tstorj.io/storj/storagenode/orders.(*FileStore).BeginEnqueue.func1:120\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:677\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:384\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:996\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-10-23T03:39:33.565Z ERROR piecestore upload failed {"Piece ID": "ULYB5QILRF3KKAYTPZALM6SHUMMDR5OAXP7HEDCLCPSRCDXP67ZA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT_REPAIR", "error": "pieces error: filestore error: sync config/storage/temp/blob-298387469.partial: read-only file system; chmod config/storage/temp/blob-298387469.partial: read-only file system; remove config/storage/temp/blob-298387469.partial: read-only file system", "errorVerbose": "pieces error: filestore error: sync config/storage/temp/blob-298387469.partial: read-only file system; chmod config/storage/temp/blob-298387469.partial: read-only file system; remove config/storage/temp/blob-298387469.partial: read-only file system\n\tstorj.io/storj/storage/filestore.(*blobWriter).Commit:120\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit.func1:130\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit:198\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:383\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:996\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-10-23T03:39:33.572Z ERROR piecestore failed to add order {"error": "order: open config/orders/unsent/unsent-orders-1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE-1603407600000000000: read-only file system", "errorVerbose": "order: open config/orders/unsent/unsent-orders-1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE-1603407600000000000: read-only file system\n\tstorj.io/storj/storagenode/orders.(*FileStore).getUnsentFile:418\n\tstorj.io/storj/storagenode/orders.(*FileStore).BeginEnqueue.func1:120\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:677\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:384\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:996\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-10-23T03:39:33.574Z ERROR piecestore upload failed {"Piece ID": "MPBENDNXKMGD45367XBVRBQZV4EYHENWEO5WH3WT6PPG5QF6UOWQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT_REPAIR", "error": "pieces error: filestore error: sync config/storage/temp/blob-866077759.partial: read-only file system; chmod config/storage/temp/blob-866077759.partial: read-only file system; remove config/storage/temp/blob-866077759.partial: read-only file system", "errorVerbose": "pieces error: filestore error: sync config/storage/temp/blob-866077759.partial: read-only file system; chmod config/storage/temp/blob-866077759.partial: read-only file system; remove config/storage/temp/blob-866077759.partial: read-only file system\n\tstorj.io/storj/storage/filestore.(*blobWriter).Commit:120\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit.func1:130\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit:198\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:383\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:996\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-10-23T03:39:33.610Z INFO orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs sending {"count": 107}
2020-10-23T03:39:33.612Z INFO orders.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE sending {"count": 67}
2020-10-23T03:39:33.610Z INFO orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 sending {"count": 85}
2020-10-23T03:39:33.613Z INFO orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs finished
2020-10-23T03:39:33.614Z ERROR orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs failed to settle orders for satellite {"satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "error": "order: unable to connect to the satellite: rpc: context canceled", "errorVerbose": "order: unable to connect to the satellite: rpc: context canceled\n\tstorj.io/storj/storagenode/orders.(*Service).settleWindow:464\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrdersFromFileStore.func1:422\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-10-23T03:39:33.611Z INFO orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S sending {"count": 98}
2020-10-23T03:39:33.612Z INFO orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB sending {"count": 218}
2020-10-23T03:39:33.616Z INFO orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S finished
2020-10-23T03:39:33.618Z INFO orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB finished
2020-10-23T03:39:33.619Z ERROR orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB failed to settle orders for satellite {"satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "error": "order: unable to connect to the satellite: rpc: context canceled", "errorVerbose": "order: unable to connect to the satellite: rpc: context canceled\n\tstorj.io/storj/storagenode/orders.(*Service).settleWindow:464\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrdersFromFileStore.func1:422\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-10-23T03:39:33.619Z ERROR orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S failed to settle orders for satellite {"satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "error": "order: unable to connect to the satellite: rpc: context canceled", "errorVerbose": "order: unable to connect to the satellite: rpc: context canceled\n\tstorj.io/storj/storagenode/orders.(*Service).settleWindow:464\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrdersFromFileStore.func1:422\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-10-23T03:39:33.621Z INFO orders.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE finished
2020-10-23T03:39:33.622Z ERROR orders.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE failed to settle orders for satellite {"satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "error": "order: unable to connect to the satellite: rpc: context canceled", "errorVerbose": "order: unable to connect to the satellite: rpc: context canceled\n\tstorj.io/storj/storagenode/orders.(*Service).settleWindow:464\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrdersFromFileStore.func1:422\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-10-23T03:39:33.623Z INFO orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 finished
2020-10-23T03:39:33.624Z ERROR orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 failed to settle orders for satellite {"satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "error": "order: unable to connect to the satellite: rpc: context canceled", "errorVerbose": "order: unable to connect to the satellite: rpc: context canceled\n\tstorj.io/storj/storagenode/orders.(*Service).settleWindow:464\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrdersFromFileStore.func1:422\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-10-23T03:39:33.623Z ERROR orders cleaning filestore archive {"error": "order: remove config/orders/archive/archived-orders-121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6-1602799200000000000-1602806578347127211-ACCEPTED: read-only file system", "errorVerbose": "order: remove config/orders/archive/archived-orders-121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6-1602799200000000000-1602806578347127211-ACCEPTED: read-only file system\n\tstorj.io/storj/storagenode/orders.(*FileStore).CleanArchive.func1:385\n\tpath/filepath.walk:360\n\tpath/filepath.walk:384\n\tpath/filepath.Walk:406\n\tstorj.io/storj/storagenode/orders.(*FileStore).CleanArchive:371\n\tstorj.io/storj/storagenode/orders.(*Service).CleanArchive:170\n\tstorj.io/storj/storagenode/orders.(*Service).Run.func2:148\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
Seems that the storage is read-only? Possible causes?
The same way, as usual:
Possible causes - abruptly shutdown or reset, using a “foreign” filesystems on Linux and a lot of other ways to corrupt the filesystem on the disk, including bad sectors.