Hi I found that my container went in an error when it receibed an error file which could not be processed. This is the first error from the docker container logs
> {
> "Piece ID": "TCXBPBSQDKIHHOTSG6EL56BYENOVRHXGCTJ3AAB5HOZBS324VXDQ",
> "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs",
> "Action": "PUT",
> "error": "pieces error: filestore error: open config/storage/temp/blob-179643383.partial: read-only file system",
> "errorVerbose": "pieces error: filestore error: open config/storage/temp/blob-179643383.partial: read-only file system\n\tstorj.io/storj/storage/filestore.(*blobStore).Create:166\n\tstorj.io/storj/storagenode/pieces.(*Store).Writer:210\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:291\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:996\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"
> }
Then the container degraded into this errors
2020-10-31T18:34:37.520Z ERROR services unexpected shutdown of a runner {"name": "piecestore:monitor", "error": "piecestore monitor: error verifying writability of storage directory: open config/storage/write-test363746309: read-only file system", "errorVerbose": "piecestore monitor: error verifying writability of storage directory: open config/storage/write-test363746309: read-only file system\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func2.1:135\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func2:132\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-10-31T18:34:38.022Z ERROR servers unexpected shutdown of a runner {"name": "debug", "error": "debug: http: Server closed", "errorVerbose": "debug: http: Server closed\n\tstorj.io/private/debug.(*Server).Run.func2:108\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
fatal error: unexpected signal during runtime execution
[signal SIGBUS: bus error code=0x2 addr=0xa6866000 pc=0x800f6c]
runtime stack:
runtime.throw(0x9897de, 0x2a)
/usr/local/go/src/runtime/panic.go:1116 +0x5c
runtime.sigpanic()
/usr/local/go/src/runtime/signal_unix.go:704 +0x398
Then this
goroutine 40 [select, 3920 minutes]:
database/sql.(*DB).connectionOpener(0x2b78d80, 0xa79958, 0x2cb2b20)
/usr/local/go/src/database/sql/sql.go:1126 +0xa4
created by database/sql.OpenDB
/usr/local/go/src/database/sql/sql.go:740 +0xf4
goroutine 37 [chan receive, 3920 minutes]:
storj.io/private/process.Ctx.func1(0x2a8f6c0, 0x2c9f6b8)
/go/pkg/mod/storj.io/private@v0.0.0-20200925142346-4c879709882f/process/exec_conf.go:115 +0x34
created by storj.io/private/process.Ctx
/go/pkg/mod/storj.io/private@v0.0.0-20200925142346-4c879709882f/process/exec_conf.go:114 +0x230
kevink
October 31, 2020, 8:43pm
4
check your filesystem. This error comes up when the filesystem got corrupted and read-only
Well I stopped the node and start again but that didn’t work and then i stopped the node reboot the system and restart the docker container. I can see is running again
kevink
October 31, 2020, 8:45pm
6
you should still check your filesystems. those errors don’t show up randomly for no reason.
Kevin, I know a bit how to but sometimes I might not see thing which are in froint of me. I have a RAID 5. When you say to check the file system what exactly `I should check ?
I mean I want to avoid a future total crash system (node)
kevink
October 31, 2020, 9:29pm
9
https://forum.storj.io/search?q=read-only%20filesystem
There should be a few threads that can help you on how to check your filesystem.
I am monitoring with netdata for the moment and I found tha was an error
I assume you’ve found this in other threads by now, but by checking the file system, we mean run fsck or equivalent on it for the file system you are using.
