Not sure what went wrong or where, but I expanded my array capacity and now Storj is throwing all sorts of fits and shutting down randomly.
My Storj is connected to a dedicated iSCSI slice and it has open permissions
I had to truncate the log due to the character limit, but everything that looks important is in there
2022-02-13T18:57:42.948-0500 INFO piecestore downloaded {"Piece ID": "RV5A7DXKOOYEMCIWOBHRPG6P6AZADAYAEFP7LTIVN66Q5DIFC4UQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET"}
2022-02-13T18:58:15.457-0500 ERROR piecestore download failed {"Piece ID": "TBL7BML6TSGJ35ZXCWE62YQQB7T4KSECFBYB2XES2CASA256EPLQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:73\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:545\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-02-13T18:58:15.465-0500 INFO piecedeleter delete piece sent to trash {"Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "LMGWFUISW7M5MCTS7T4MV3YKXKGPKYG2VZGUJPOZKPMU3VTEVVGQ"}
2022-02-13T18:58:15.467-0500 ERROR pieces:trash emptying trash failed {"error": "pieces error: filestore error: open E:\\Storj\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\kr: Access is denied.", "errorVerbose": "pieces error: filestore error: open E:\\Storj\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\kr: Access is denied.\n\tstorj.io/storj/storage/filestore.(*blobStore).EmptyTrash:154\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).EmptyTrash:310\n\tstorj.io/storj/storagenode/pieces.(*Store).EmptyTrash:367\n\tstorj.io/storj/storagenode/pieces.(*TrashChore).Run.func1:51\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2022-02-13T18:58:15.467-0500 ERROR piecedeleter could not send delete piece to trash {"Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "IKIOJ7H7SGPF2L4DZ4B74UDDFDMCXHF6WVBPXTXMMDJLURG4CHVQ", "error": "pieces error: v0pieceinfodb: sql: no rows in result set", "errorVerbose": "pieces error: v0pieceinfodb: sql: no rows in result set\n\tstorj.io/storj/storagenode/storagenodedb.(*v0PieceInfoDB).Get:131\n\tstorj.io/storj/storagenode/pieces.(*Store).MigrateV0ToV1:404\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:348\n\tstorj.io/storj/storagenode/pieces.(*Deleter).deleteOrTrash:185\n\tstorj.io/storj/storagenode/pieces.(*Deleter).work:135\n\tstorj.io/storj/storagenode/pieces.(*Deleter).Run.func1:72\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2022-02-13T18:58:15.486-0500 ERROR piecestore upload failed {"Piece ID": "PO7MXDN6XCDG5G5BRXWTJN2CY5JG6C5LRCMH4KST4KIDBIHRGNLA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT_REPAIR", "error": "pieces error: filestore error: sync E:\\Storj\\temp\\blob-4283521184.partial: The system cannot find the file specified.", "errorVerbose": "pieces error: filestore error: sync E:\\Storj\\temp\\blob-4283521184.partial: The system cannot find the file specified.\n\tstorj.io/storj/storage/filestore.(*blobWriter).Commit:120\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit.func1:130\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit:198\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:405\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 2319360}
2022-02-13T18:58:15.486-0500 ERROR piecestore failed to add bandwidth usage {"error": "bandwidthdb: disk I/O error: The device is not ready.", "errorVerbose": "bandwidthdb: disk I/O error: The device is not ready.\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:722\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:406\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-02-13T18:58:15.486-0500 ERROR piecestore upload failed {"Piece ID": "UO3BBMCISTLEA5REQEUQIUSGPQVCTG7WEEQQWKZOYAFQ6HQVCIYQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "pieces error: filestore error: sync E:\\Storj\\temp\\blob-987201253.partial: The system cannot find the file specified.", "errorVerbose": "pieces error: filestore error: sync E:\\Storj\\temp\\blob-987201253.partial: The system cannot find the file specified.\n\tstorj.io/storj/storage/filestore.(*blobWriter).Commit:120\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit.func1:130\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit:198\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:405\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 181504}
2022-02-13T18:58:15.513-0500 INFO piecestore download started {"Piece ID": "KP5EZHO6NBFPZMDYEKANO37SREUNRYX3JLVGIVFO5IXODVAZLR4A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2022-02-13T18:58:15.514-0500 INFO piecestore download started {"Piece ID": "U3UGSL7FP2VHXBEWBTWN3UNCNULP3WMTVBMEL4RRXA4YZ6VDJP7Q", "Satellite ID": \6r2fgwqz3manwt4aogq343bfkh2n5vvg4ohqqgggrrunaaaaaaaa\\\\h7\\\\uhdxndex67ofcqeerek2ia4t7oxgrokxcykgbj2gb536uog7jq.sj1\": Access is denied.", "errorVerbose": "pieces error: filestore error: unable to open \"E:\\\\Storj\\\\blobs\\\\6r2fgwqz3manwt4aogq343bfkh2n5vvg4ohqqgggrrunaaaaaaaa\\\\h7\\\\uhdxndex67ofcqeerek2ia4t7oxgrokxcykgbj2gb536uog7jq.sj1\": Access is denied.\n\tstorj.io/storj/storage/filestore.(*Dir).Open:279\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:75\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:262\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:541\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-02-13T18:58:15.882-0500 ERROR piecestore download failed {"Piece ID": "KP5EZHO6NBFPZMDYEKANO37SREUNRYX3JLVGIVFO5IXODVAZLR4A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "error": "pieces error: filestore error: unable to open \"E:\\\\Storj\\\\blobs\\\\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\\\\kp\\\\5ezho6nbfpzmdyekano37sreunryx3jlvgivfo5ixodvazlr4a.sj1\": Access is denied.", "errorVerbose": "pieces error: filestore error: unable to open \"E:\\\\Storj\\\\blobs\\\\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\\\\kp\\\\5ezho6nbfpzmdyekano37sreunryx3jlvgivfo5ixodvazlr4a.sj1\": Access is denied.\n\tstorj.io/storj/storage/filestore.(*Dir).Open:279\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:75\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:262\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:541\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-02-13T18:58:15.893-0500 ERROR piecestore upload failed {"Piece ID": "QHISNGLWR3EJEDH7I6KB623JVL53ZRQCKJTA7LEZIM7MXZVZ24TA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "pieces error: filestore error: truncate E:\\Storj\\temp\\blob-3815658995.partial: The system cannot find the file specified.", "errorVerbose": "pieces error: filestore error: truncate E:\\Storj\\temp\\blob-3815658995.partial: The system cannot find the file specified.\n\tstorj.io/storj/storage/filestore.(*blobStore).Create:170\n\tstorj.io/storj/storagenode/pieces.(*Store).Writer:213\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:301\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 0}
2022-02-13T18:58:15.887-0500 ERROR piecestore download failed {"Piece ID": "MVTEGEODT3GJICGJYB2REPB3Z5QB7FVVUIONAKHGEL2BZ6LVIXMA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "GET_REPAIR", "error": "pieces error: filestore error: unable to open \"E:\\\\Storj\\\\blobs\\\\6r2fgwqz3manwt4aogq343bfkh2n5vvg4ohqqgggrrunaaaaaaaa\\\\mv\\\\tegeodt3gjicgjyb2repb3z5qb7fvvuionakhgel2bz6lvixma.sj1\": Access is denied.", "errorVerbose": "pieces error: filestore error: unable to open \"E:\\\\Storj\\\\blobs\\\\6r2fgwqz3manwt4aogq343bfkh2n5vvg4ohqqgggrrunaaaaaaaa\\\\mv\\\\tegeodt3gjicgjyb2repb3z5qb7fvvuionakhgel2bz6lvixma.sj1\": Access is denied.\n\tstorj.io/storj/storage/filestore.(*Dir).Open:279\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:75\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:262\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:541\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-02-13T18:58:15.894-0500 ERROR piecestore:cache error getting current used space: {"error": "CreateFile E:\\Storj\\blobs\\6r2fgwqz3manwt4aogq343bfkh2n5vvg4ohqqgggrrunaaaaaaaa\\cd/wcn4bvj3svdwwkzawvafq4epb4r3yno6xknqlhgeqvcujfm5bq.sj1: The specified network resource or device is no longer available.; open E:\\Storj\\blobs\\abforhuxbzyd35blusvrifvdwmfx4hmocsva4vmpp3rgqaaaaaaa: Access is denied.; open E:\\Storj\\blobs\\pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa\\22: Access is denied.; open E:\\Storj\\blobs\\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\\23: Access is denied.; open E:\\Storj\\blobs\\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\\22: Access is denied.; disk I/O error: The device is not ready.", "errorVerbose": "group:\n--- CreateFile E:\\Storj\\blobs\\6r2fgwqz3manwt4aogq343bfkh2n5vvg4ohqqgggrrunaaaaaaaa\\cd/wcn4bvj3svdwwkzawvafq4epb4r3yno6xknqlhgeqvcujfm5bq.sj1: The specified network resource or device is no longer available.\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:284\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n--- open E:\\Storj\\blobs\\abforhuxbzyd35blusvrifvdwmfx4hmocsva4vmpp3rgqaaaaaaa: Access is denied.\n--- open E:\\Storj\\blobs\\pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa\\22: Access is denied.\n--- open E:\\Storj\\blobs\\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\\23: Access is denied.\n--- open E:\\Storj\\blobs\\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\\22: Access is denied.\n--- disk I/O error: The device is not ready."}
2022-02-13T18:58:15.894-0500 ERROR services unexpected shutdown of a runner {"name": "piecestore:cache", "error": "CreateFile E:\\Storj\\blobs\\6r2fgwqz3manwt4aogq343bfkh2n5vvg4ohqqgggrrunaaaaaaaa\\cd/wcn4bvj3svdwwkzawvafq4epb4r3yno6xknqlhgeqvcujfm5bq.sj1: The specified network resource or device is no longer available.; open E:\\Storj\\blobs\\abforhuxbzyd35blusvrifvdwmfx4hmocsva4vmpp3rgqaaaaaaa: Access is denied.; open E:\\Storj\\blobs\\pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa\\22: Access is denied.; open E:\\Storj\\blobs\\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\\23: Access is denied.; open E:\\Storj\\blobs\\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\\22: Access is denied.; disk I/O error: The device is not ready.", "errorVerbose": "group:\n--- CreateFile E:\\Storj\\blobs\\6r2fgwqz3manwt4aogq343bfkh2n5vvg4ohqqgggrrunaaaaaaaa\\cd/wcn4bvj3svdwwkzawvafq4epb4r3yno6xknqlhgeqvcujfm5bq.sj1: The specified network resource or device is no longer available.\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:284\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n--- open E:\\Storj\\blobs\\abforhuxbzyd35blusvrifvdwmfx4hmocsva4vmpp3rgqaaaaaaa: Access is denied.\n--- open E:\\Storj\\blobs\\pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa\\22: Access is denied.\n--- open E:\\Storj\\blobs\\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\\23: Access is denied.\n--- open E:\\Storj\\blobs\\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\\22: Access is denied.\n--- disk I/O error: The device is not ready."}
2022-02-13T18:58:15.895-0500 ERROR piecestore download failed {"Piece ID": "U3UGSL7FP2VHXBEWBTWN3UNCNULP3WMTVBMEL4RRXA4YZ6VDJP7Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "error": "pieces error: filestore error: unable to open \"E:\\\\Storj\\\\blobs\\\\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\\\\u3\\\\ugsl7fp2vhxbewbtwn3uncnulp3wmtvbmel4rrxa4yz6vdjp7q.sj1\": Access is denied.", "errorVerbose": "pieces error: filestore error: unable to open \"E:\\\\Storj\\\\blobs\\\\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\\\\u3\\\\ugsl7fp2vhxbewbtwn3uncnulp3wmtvbmel4rrxa4yz6vdjp7q.sj1\": Access is denied.\n\tstorj.io/storj/storage/filestore.(*Dir).Open:279\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:75\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:262\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:541\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-02-13T18:58:15.896-0500 ERROR piecestore upload failed {"Piece ID": "UKJRIQHLNRYHFEOWFW5KAAA2EPIVTY4S765A52A7ITWD3BBYNIEQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "pieces error: filestore error: truncate E:\\Storj\\temp\\blob-4086566496.partial: The system cannot find the file specified.", "errorVerbose": "pieces error: filestore error: truncate E:\\Storj\\temp\\blob-4086566496.partial: The system cannot find the file specified.\n\tstorj.io/storj/storage/filestore.(*blobStore).Create:170\n\tstorj.io/storj/storagenode/pieces.(*Store).Writer:213\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:301\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 0}
2022-02-13T18:58:15.896-0500 INFO piecestore downloaded {"Piece ID": "LTPSUIZFSBK57YKAENQK2ZRGFWCACQPUC5YLCD7OTNUACWCKM3EQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2022-02-13T18:58:15.914-0500 FATAL Unrecoverable error {"error": "CreateFile E:\\Storj\\blobs\\6r2fgwqz3manwt4aogq343bfkh2n5vvg4ohqqgggrrunaaaaaaaa\\cd/wcn4bvj3svdwwkzawvafq4epb4r3yno6xknqlhgeqvcujfm5bq.sj1: The specified network resource or device is no longer available.; open E:\\Storj\\blobs\\abforhuxbzyd35blusvrifvdwmfx4hmocsva4vmpp3rgqaaaaaaa: Access is denied.; open E:\\Storj\\blobs\\pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa\\22: Access is denied.; open E:\\Storj\\blobs\\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\\23: Access is denied.; open E:\\Storj\\blobs\\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\\22: Access is denied.; disk I/O error: The device is not ready.", "errorVerbose": "group:\n--- CreateFile E:\\Storj\\blobs\\6r2fgwqz3manwt4aogq343bfkh2n5vvg4ohqqgggrrunaaaaaaaa\\cd/wcn4bvj3svdwwkzawvafq4epb4r3yno6xknqlhgeqvcujfm5bq.sj1: The specified network resource or device is no longer available.\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:284\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n--- open E:\\Storj\\blobs\\abforhuxbzyd35blusvrifvdwmfx4hmocsva4vmpp3rgqaaaaaaa: Access is denied.\n--- open E:\\Storj\\blobs\\pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa\\22: Access is denied.\n--- open E:\\Storj\\blobs\\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\\23: Access is denied.\n--- open E:\\Storj\\blobs\\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\\22: Access is denied.\n--- disk I/O error: The device is not ready."}
2022-02-14T06:47:20.347-0500 INFO Configuration loaded {"Location": "C:\\Program Files\\Storj\\Storage Node\\config.yaml"}
2022-02-14T06:47:26.372-0500 INFO db.migration Database Version {"version": 53}
2022-02-14T06:47:26.477-0500 INFO preflight:localtime start checking local system clock with trusted satellites' system clock.
2022-02-14T06:47:27.102-0500 INFO preflight:localtime local system clock is in sync with trusted satellites' system clock.
2022-02-14T06:47:27.102-0500 INFO Node started
2022-02-14T06:47:27.102-0500 INFO Public server started on [::]:28967
2022-02-14T06:47:27.102-0500 INFO Private server started on 127.0.0.1:7778
2022-02-14T06:47:32.285-0500 WARN contact:service failed PingMe request to satellite {"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "error": "ping satellite: check-in network: failed to ping node (ID: 161sXNjyPE65jjC5uNXTy1jvwqprwTDzXKHc4UVdBExeUUyGiW) at address: 100.100.100.100:28967, err: contact: failed to dial storage node (ID: 161sXNjyPE65jjC5uNXTy1jvwqprwTDzXKHc4UVdBExeUUyGiW) at address 100.100.100.100:28967 using QUIC: rpc: quic: timeout: no recent network activity", "errorVerbose": "ping satellite: check-in network: failed to ping node (ID: 161sXNjyPE65jjC5uNXTy1jvwqprwTDzXKHc4UVdBExeUUyGiW) at address: 100.100.100.10028967, err: contact: failed to dial storage node (ID: 161sXNjyPE65jjC5uNXTy1jvwqprwTDzXKHc4UVdBExeUUyGiW) at address 100.100.100.100:28967 using QUIC: rpc: quic: timeout: no recent network activity\n\tstorj.io/storj/storagenode/contact.(*Service).requestPingMeOnce:194\n\tstorj.io/storj/storagenode/contact.(*Service).RequestPingMeQUIC:167\n\tstorj.io/storj/storagenode.(*Peer).addConsoleService:841\n\tstorj.io/storj/storagenode.(*Peer).Run:880\n\tmain.cmdRun:251\n\tstorj.io/private/process.cleanup.func1.4:363\n\tstorj.io/private/process.cleanup.func1:381\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfig:88\n\tstorj.io/private/process.Exec:65\n\tmain.(*service).Execute.func1:61\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2022-02-14T06:47:58.679-0500 INFO bandwidth Performing bandwidth usage rollups
2022-02-14T06:47:58.683-0500 WARN console:service unable to get Satellite URL {"Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "error": "console: trust: satellite \"118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW\" is untrusted", "errorVerbose": "console: trust: satellite \"118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW\" is untrusted\n\tstorj.io/storj/storagenode/trust.(*Pool).getInfo:238\n\tstorj.io/storj/storagenode/trust.(*Pool).GetNodeURL:177\n\tstorj.io/storj/storagenode/console.(*Service).GetDashboardData:186\n\tstorj.io/storj/storagenode/console/consoleapi.(*StorageNode).StorageNode:45\n\tnet/http.HandlerFunc.ServeHTTP:2047\n\tgithub.com/gorilla/mux.(*Router).ServeHTTP:210\n\tnet/http.serverHandler.ServeHTTP:2879\n\tnet/http.(*conn).serve:1930"}
2022-02-14T06:48:00.531-0500 INFO orders.12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo sending {"count": 21}
2022-02-14T06:48:00.531-0500 INFO orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 sending {"count": 103}
2022-02-14T06:48:00.532-0500 INFO orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S sending {"count": 1662}
2022-02-14T06:48:00.532-0500 INFO orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB sending {"count": 76}
2022-02-14T06:48:00.532-0500 INFO orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs sending {"count": 537}
2022-02-14T06:48:00.531-0500 INFO orders.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE sending {"count": 640}
2022-02-14T06:48:00.636-0500 INFO orders.12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo finished
2022-02-14T06:48:01.090-0500 INFO orders.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE finished
2022-02-14T06:48:01.106-0500 INFO orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB finished
2022-02-14T06:48:01.349-0500 INFO orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S finished
2022-02-14T06:48:01.468-0500 INFO orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs finished
2022-02-14T06:48:01.552-0500 INFO orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 finished
2022-02-14T06:48:01.570-0500 INFO orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 sending {"count": 114}
2022-02-14T06:48:01.570-0500 INFO orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB sending {"count": 56}
2022-02-14T06:48:01.570-0500 INFO orders.12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo sending {"count": 15}
2022-02-14T06:48:01.571-0500 INFO orders.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE sending {"count": 591}
2022-02-14T06:48:01.571-0500 INFO orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S sending {"count": 1686}
2022-02-14T06:48:01.571-0500 INFO orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs sending {"count": 433}
2022-02-14T06:48:01.667-0500 INFO orders.12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo finished
2022-02-14T06:48:02.091-0500 INFO orders.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE finished
2022-02-14T06:48:02.131-0500 INFO orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB finished
2022-02-14T06:48:02.417-0500 INFO orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S finished
2022-02-14T06:48:02.598-0500 INFO orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 finished
2022-02-14T06:48:03.216-0500 INFO orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs finished
2022-02-14T06:48:04.067-0500 WARN contact:service Your node is still considered to be online but encountered an error. {"Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "Error": "contact: failed to dial storage node (ID: 161sXNjyPE65jjC5uNXTy1jvwqprwTDzXKHc4UVdBExeUUyGiW) at address 100.100.100.100:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2022-02-14T06:48:13.165-0500 ERROR collector unable to delete piece {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "G2IOC3TEFGJVWXPOLMNP3ENJSNHVCGUS2XBB72PUFT5IWLKR7Y7A", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2022-02-14T06:48:13.945-0500 ERROR collector unable to delete piece {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "YHURXVS34GKO65CGGJ6E4ROUMXSCVKQSMJK3E2XLRJ4WU4S3KIFQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2022-02-14T06:48:17.388-0500 INFO collector delete expired {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "NTLSFKLYENIQX42VH2MA2Z2O5GHLZP2BSIBE3WN6TQXMGT6SK4CQ"}
2022-02-14T06:48:41.141-0500 INFO piecestore download started {"Piece ID": "LTQ6AGZDCGMDIKBYOKGSX3HH6QXNIDC6M5GZKRJ53YWA4JXRQNDQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET"}
2022-02-14T06:48:41.231-0500 INFO collector delete expired {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "5LJL4S4QK3WFTMBRZEMARSDFJIZAU4Y3EHK7HETY7J2RXWHSYOUA"}
"PXGXSXWQOXWMP7J65MDT5NU4MQI7ER27ZTLZRKZRFATMLMSUSS6A"}
2022-02-14T06:48:50.149-0500 INFO collector collect {"count": 65}
2022-02-14T06:48:50.534-0500 INFO piecestore download canceled {"Piece ID": "PFIGLAK7PIXGYV4HUTXV5RRGGZXKMQN5AXONSD2RGT3P5SU7SGQQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2022-02-14T06:48:52.916-0500 INFO piecestore download started {"Piece ID": "LEIUJRPJAXGL6WAEZJXWDGE4BYZ5ITNJFVWWHFOIM2WSEQZSMZ2A", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET"}
2022-02-14T06:48:53.107-0500 WARN console:service unable to get Satellite URL {"Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "error": "console: trust: satellite \"118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW\" is untrusted", "errorVerbose": "console: trust: satellite \"118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW\" is untrusted\n\tstorj.io/storj/storagenode/trust.(*Pool).getInfo:238\n\tstorj.io/storj/storagenode/trust.(*Pool).GetNodeURL:177\n\tstorj.io/storj/storagenode/console.(*Service).GetDashboardData:186\n\tstorj.io/storj/storagenode/console/consoleapi.(*StorageNode).StorageNode:45\n\tnet/http.HandlerFunc.ServeHTTP:2047\n\tgithub.com/gorilla/mux.(*Router).ServeHTTP:210\n\tnet/http.serverHandler.ServeHTTP:2879\n\tnet/http.(*conn).serve:1930"}