"INFO piecestore upload failed There is not enough space on the disk" despite over 200GB free on partition

thev · March 6, 2020, 9:47pm

My dashboard displays over 120GB free

but I get following errors:

2020-03-06T22:42:16.330+0100	INFO	piecestore	upload started	{“Piece ID”: “ZMCFXJHUQDLRCTTF2I4QR537QWPHQ5KRJNT36Z2RYUEDDEZEQ65Q”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Bandwidth”: 29860716743424, “Available Space”: 125350051072}
2020-03-06T22:42:16.333+0100	INFO	piecestore	upload failed	{“Piece ID”: “ZMCFXJHUQDLRCTTF2I4QR537QWPHQ5KRJNT36Z2RYUEDDEZEQ65Q”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “pieces error: filestore error: open r:\storj\temp\blob-721553304.partial: There is not enough space on the disk.”, “errorVerbose”: “pieces error: filestore error: open r:\storj\temp\blob-721553304.partial: There is not enough space on the disk.\n\tstorj.io/storj/storage/filestore.(blobStore).Create:143\n\tstorj.io/storj/storagenode/pieces.(Store).Writer:190\n\tstorj.io/storj/storagenode/piecestore.(Endpoint).doUpload:360\n\tstorj.io/storj/storagenode/piecestore.(drpcEndpoint).Upload:257\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:1066\n\tstorj.io/drpc/drpcserver.(Server).doHandle:199\n\tstorj.io/drpc/drpcserver.(Server).HandleRPC:173\n\tstorj.io/drpc/drpcserver.(Server).ServeOne:124\n\tstorj.io/drpc/drpcserver.(Server).Serve.func2:161\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-03-06T22:42:19.829+0100	INFO	piecestore	upload started	{“Piece ID”: “TI6FYFT55QCY4CYLMDVHHKMUOBCM75VAOSNCVS4IYEE2AZLSKMUQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Available Bandwidth”: 29860716743424, “Available Space”: 125350051072}
2020-03-06T22:42:19.832+0100	INFO	piecestore	upload failed	{“Piece ID”: “TI6FYFT55QCY4CYLMDVHHKMUOBCM75VAOSNCVS4IYEE2AZLSKMUQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “error”: “pieces error: filestore error: open r:\storj\temp\blob-167558679.partial: There is not enough space on the disk.”, “errorVerbose”: “pieces error: filestore error: open r:\storj\temp\blob-167558679.partial: There is not enough space on the disk.\n\tstorj.io/storj/storage/filestore.(blobStore).Create:143\n\tstorj.io/storj/storagenode/pieces.(Store).Writer:190\n\tstorj.io/storj/storagenode/piecestore.(Endpoint).doUpload:360\n\tstorj.io/storj/storagenode/piecestore.(drpcEndpoint).Upload:257\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:1066\n\tstorj.io/drpc/drpcserver.(Server).doHandle:199\n\tstorj.io/drpc/drpcserver.(Server).HandleRPC:173\n\tstorj.io/drpc/drpcserver.(Server).ServeOne:124\n\tstorj.io/drpc/drpcserver.(Server).Serve.func2:161\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-03-06T22:42:21.110+0100	INFO	piecestore	upload started	{“Piece ID”: “XOMZHJAHA3JUWKLFU6YLDCABDNKOTRDRUMK5WEAIIZA5QVSHUDOQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Available Bandwidth”: 29860716743424, “Available Space”: 125350051072}
2020-03-06T22:42:21.114+0100	INFO	piecestore	upload failed	{“Piece ID”: “XOMZHJAHA3JUWKLFU6YLDCABDNKOTRDRUMK5WEAIIZA5QVSHUDOQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “error”: “pieces error: filestore error: open r:\storj\temp\blob-080834186.partial: There is not enough space on the disk.”, “errorVerbose”: “pieces error: filestore error: open r:\storj\temp\blob-080834186.partial: There is not enough space on the disk.\n\tstorj.io/storj/storage/filestore.(blobStore).Create:143\n\tstorj.io/storj/storagenode/pieces.(Store).Writer:190\n\tstorj.io/storj/storagenode/piecestore.(Endpoint).doUpload:360\n\tstorj.io/storj/storagenode/piecestore.(drpcEndpoint).Upload:257\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:1066\n\tstorj.io/drpc/drpcserver.(Server).doHandle:199\n\tstorj.io/drpc/drpcserver.(Server).HandleRPC:173\n\tstorj.io/drpc/drpcserver.(Server).ServeOne:124\n\tstorj.io/drpc/drpcserver.(Server).Serve.func2:161\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-03-06T22:42:21.453+0100	INFO	piecestore	upload started	{“Piece ID”: “2RZC5IJLLLR3ANLDAG36FSVMJWWWFXBVFBCSJYTANYJFMWN6DIOA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Bandwidth”: 29860716743424, “Available Space”: 125350051072}
2020-03-06T22:42:21.457+0100	INFO	piecestore	upload failed	{“Piece ID”: “2RZC5IJLLLR3ANLDAG36FSVMJWWWFXBVFBCSJYTANYJFMWN6DIOA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “pieces error: filestore error: open r:\storj\temp\blob-010441313.partial: There is not enough space on the disk.”, “errorVerbose”: “pieces error: filestore error: open r:\storj\temp\blob-010441313.partial: There is not enough space on the disk.\n\tstorj.io/storj/storage/filestore.(blobStore).Create:143\n\tstorj.io/storj/storagenode/pieces.(Store).Writer:190\n\tstorj.io/storj/storagenode/piecestore.(Endpoint).doUpload:360\n\tstorj.io/storj/storagenode/piecestore.(drpcEndpoint).Upload:257\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:1066\n\tstorj.io/drpc/drpcserver.(Server).doHandle:199\n\tstorj.io/drpc/drpcserver.(Server).HandleRPC:173\n\tstorj.io/drpc/drpcserver.(Server).ServeOne:124\n\tstorj.io/drpc/drpcserver.(Server).Serve.func2:161\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-03-06T22:42:22.099+0100	INFO	piecestore	upload started	{“Piece ID”: “GBGLMNMQZZN262Q5DI7TJC2C3DLBC2GORE4OIAL6ZDATJ4GLWRZA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Available Bandwidth”: 29860716743424, “Available Space”: 125350051072}
2020-03-06T22:42:22.103+0100	INFO	piecestore	upload failed	{“Piece ID”: “GBGLMNMQZZN262Q5DI7TJC2C3DLBC2GORE4OIAL6ZDATJ4GLWRZA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “error”: “pieces error: filestore error: open r:\storj\temp\blob-402745932.partial: There is not enough space on the disk.”, “errorVerbose”: “pieces error: filestore error: open r:\storj\temp\blob-402745932.partial: There is not enough space on the disk.\n\tstorj.io/storj/storage/filestore.(blobStore).Create:143\n\tstorj.io/storj/storagenode/pieces.(Store).Writer:190\n\tstorj.io/storj/storagenode/piecestore.(Endpoint).doUpload:360\n\tstorj.io/storj/storagenode/piecestore.(drpcEndpoint).Upload:257\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:1066\n\tstorj.io/drpc/drpcserver.(Server).doHandle:199\n\tstorj.io/drpc/drpcserver.(Server).HandleRPC:173\n\tstorj.io/drpc/drpcserver.(Server).ServeOne:124\n\tstorj.io/drpc/drpcserver.(Server).Serve.func2:161\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-03-06T22:42:22.625+0100	INFO	piecestore	upload started	{“Piece ID”: “VJNBGHR7TKNLCQ5RH5NSTZILRPYNURLXNL72JVGP4DDWE5QJDFLA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”, “Available Bandwidth”: 29860716743424, “Available Space”: 125350051072}
2020-03-06T22:42:22.629+0100	INFO	piecestore	upload failed	{“Piece ID”: “VJNBGHR7TKNLCQ5RH5NSTZILRPYNURLXNL72JVGP4DDWE5QJDFLA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”, “error”: “pieces error: filestore error: open r:\storj\temp\blob-924504891.partial: There is not enough space on the disk.”, “errorVerbose”: “pieces error: filestore error: open r:\storj\temp\blob-924504891.partial: There is not enough space on the disk.\n\tstorj.io/storj/storage/filestore.(blobStore).Create:143\n\tstorj.io/storj/storagenode/pieces.(Store).Writer:190\n\tstorj.io/storj/storagenode/piecestore.(Endpoint).doUpload:360\n\tstorj.io/storj/storagenode/piecestore.(drpcEndpoint).Upload:257\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:1066\n\tstorj.io/drpc/drpcserver.(Server).doHandle:199\n\tstorj.io/drpc/drpcserver.(Server).HandleRPC:173\n\tstorj.io/drpc/drpcserver.(Server).ServeOne:124\n\tstorj.io/drpc/drpcserver.(Server).Serve.func2:161\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-03-06T22:42:22.810+0100	INFO	piecestore	upload started	{“Piece ID”: “XJ42ULDWULSSXRJ3H2OW63AJEXDVOMCJGKZ3ZL574WGBOB7ZXZ2Q”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Bandwidth”: 29860716743424, “Available Space”: 125350051072}
2020-03-06T22:42:22.814+0100	INFO	piecestore	upload failed	{“Piece ID”: “XJ42ULDWULSSXRJ3H2OW63AJEXDVOMCJGKZ3ZL574WGBOB7ZXZ2Q”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “pieces error: filestore error: open r:\storj\temp\blob-879276894.partial: There is not enough space on the disk.”, “errorVerbose”: “pieces error: filestore error: open r:\storj\temp\blob-879276894.partial: There is not enough space on the disk.\n\tstorj.io/storj/storage/filestore.(blobStore).Create:143\n\tstorj.io/storj/storagenode/pieces.(Store).Writer:190\n\tstorj.io/storj/storagenode/piecestore.(Endpoint).doUpload:360\n\tstorj.io/storj/storagenode/piecestore.(drpcEndpoint).Upload:257\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:1066\n\tstorj.io/drpc/drpcserver.(Server).doHandle:199\n\tstorj.io/drpc/drpcserver.(Server).HandleRPC:173\n\tstorj.io/drpc/drpcserver.(Server).ServeOne:124\n\tstorj.io/drpc/drpcserver.(Server).Serve.func2:161\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-03-06T22:42:26.211+0100	INFO	piecestore	upload started	{“Piece ID”: “B5SLUZF7SFFR6NPXUZH4H6ZAQ5POXKXH7V5SQFQTYAYLEJCSGIXA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”, “Available Bandwidth”: 29860716743424, “Available Space”: 125350051072}
2020-03-06T22:42:26.215+0100	INFO	piecestore	upload failed	{“Piece ID”: “B5SLUZF7SFFR6NPXUZH4H6ZAQ5POXKXH7V5SQFQTYAYLEJCSGIXA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”, “error”: “pieces error: filestore error: open r:\storj\temp\blob-560300837.partial: There is not enough space on the disk.”, “errorVerbose”: “pieces error: filestore error: open r:\storj\temp\blob-560300837.partial: There is not enough space on the disk.\n\tstorj.io/storj/storage/filestore.(blobStore).Create:143\n\tstorj.io/storj/storagenode/pieces.(Store).Writer:190\n\tstorj.io/storj/storagenode/piecestore.(Endpoint).doUpload:360\n\tstorj.io/storj/storagenode/piecestore.(drpcEndpoint).Upload:257\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:1066\n\tstorj.io/drpc/drpcserver.(Server).doHandle:199\n\tstorj.io/drpc/drpcserver.(Server).HandleRPC:173\n\tstorj.io/drpc/drpcserver.(Server).ServeOne:124\n\tstorj.io/drpc/drpcserver.(Server).Serve.func2:161\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-03-06T22:42:26.592+0100	INFO	piecestore	upload started	{“Piece ID”: “ON56HDHMVWZKMF3MJUIC5FLLUZCPVPIUFCT2ZKUAZOXBCJ7KZT3Q”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”, “Available Bandwidth”: 29860716743424, “Available Space”: 125350051072}
2020-03-06T22:42:26.595+0100	INFO	piecestore	upload failed	{“Piece ID”: “ON56HDHMVWZKMF3MJUIC5FLLUZCPVPIUFCT2ZKUAZOXBCJ7KZT3Q”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”, “error”: “pieces error: filestore error: open r:\storj\temp\blob-796154432.partial: There is not enough space on the disk.”, “errorVerbose”: “pieces error: filestore error: open r:\storj\temp\blob-796154432.partial: There is not enough space on the disk.\n\tstorj.io/storj/storage/filestore.(blobStore).Create:143\n\tstorj.io/storj/storagenode/pieces.(Store).Writer:190\n\tstorj.io/storj/storagenode/piecestore.(Endpoint).doUpload:360\n\tstorj.io/storj/storagenode/piecestore.(drpcEndpoint).Upload:257\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:1066\n\tstorj.io/drpc/drpcserver.(Server).doHandle:199\n\tstorj.io/drpc/drpcserver.(Server).HandleRPC:173\n\tstorj.io/drpc/drpcserver.(Server).ServeOne:124\n\tstorj.io/drpc/drpcserver.(Server).Serve.func2:161\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-03-06T22:42:30.406+0100	INFO	piecestore	upload started	{“Piece ID”: “HO2MGHZ4AUUOYDUVMN4J33TSTD5MGEARYHFYRYD2HI3DKLRF3TBA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Available Bandwidth”: 29860716743424, “Available Space”: 125350051072}
2020-03-06T22:42:30.409+0100	INFO	piecestore	upload failed	{“Piece ID”: “HO2MGHZ4AUUOYDUVMN4J33TSTD5MGEARYHFYRYD2HI3DKLRF3TBA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “error”: “pieces error: filestore error: open r:\storj\temp\blob-515680927.partial: There is not enough space on the disk.”, “errorVerbose”: “pieces error: filestore error: open r:\storj\temp\blob-515680927.partial: There is not enough space on the disk.\n\tstorj.io/storj/storage/filestore.(blobStore).Create:143\n\tstorj.io/storj/storagenode/pieces.(Store).Writer:190\n\tstorj.io/storj/storagenode/piecestore.(Endpoint).doUpload:360\n\tstorj.io/storj/storagenode/piecestore.(drpcEndpoint).Upload:257\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:1066\n\tstorj.io/drpc/drpcserver.(Server).doHandle:199\n\tstorj.io/drpc/drpcserver.(Server).HandleRPC:173\n\tstorj.io/drpc/drpcserver.(Server).ServeOne:124\n\tstorj.io/drpc/drpcserver.(Server).Serve.func2:161\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-03-06T22:42:33.238+0100	INFO	piecestore	upload started	{“Piece ID”: “KGX5HA4SIWOUBUYJXX6EFTK5J5B2UVPLEWTQBITHPFZKHYNAWSEA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Available Bandwidth”: 29860716743424, “Available Space”: 125350051072}
2020-03-06T22:42:33.241+0100	INFO	piecestore	upload failed	{“Piece ID”: “KGX5HA4SIWOUBUYJXX6EFTK5J5B2UVPLEWTQBITHPFZKHYNAWSEA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “error”: “pieces error: filestore error: open r:\storj\temp\blob-918445938.partial: There is not enough space on the disk.”, “errorVerbose”: “pieces error: filestore error: open r:\storj\temp\blob-918445938.partial: There is not enough space on the disk.\n\tstorj.io/storj/storage/filestore.(blobStore).Create:143\n\tstorj.io/storj/storagenode/pieces.(Store).Writer:190\n\tstorj.io/storj/storagenode/piecestore.(Endpoint).doUpload:360\n\tstorj.io/storj/storagenode/piecestore.(drpcEndpoint).Upload:257\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:1066\n\tstorj.io/drpc/drpcserver.(Server).doHandle:199\n\tstorj.io/drpc/drpcserver.(Server).HandleRPC:173\n\tstorj.io/drpc/drpcserver.(Server).ServeOne:124\n\tstorj.io/drpc/drpcserver.(Server).Serve.func2:161\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-03-06T22:42:33.468+0100	INFO	piecestore	upload started	{“Piece ID”: “UFQ34TT752JWYILOIPRQEHYTCSS46Q3SQ7LJARZHR5HX2X7U6AVA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Bandwidth”: 29860716743424, “Available Space”: 125350051072}
2020-03-06T22:42:33.471+0100	INFO	piecestore	upload failed	{“Piece ID”: “UFQ34TT752JWYILOIPRQEHYTCSS46Q3SQ7LJARZHR5HX2X7U6AVA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “pieces error: filestore error: open r:\storj\temp\blob-018246185.partial: There is not enough space on the disk.”, “errorVerbose”: “pieces error: filestore error: open r:\storj\temp\blob-018246185.partial: There is not enough space on the disk.\n\tstorj.io/storj/storage/filestore.(blobStore).Create:143\n\tstorj.io/storj/storagenode/pieces.(Store).Writer:190\n\tstorj.io/storj/storagenode/piecestore.(Endpoint).doUpload:360\n\tstorj.io/storj/storagenode/piecestore.(drpcEndpoint).Upload:257\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:1066\n\tstorj.io/drpc/drpcserver.(Server).doHandle:199\n\tstorj.io/drpc/drpcserver.(Server).HandleRPC:173\n\tstorj.io/drpc/drpcserver.(Server).ServeOne:124\n\tstorj.io/drpc/drpcserver.(Server).Serve.func2:161\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-03-06T22:42:37.095+0100	INFO	piecestore	upload started	{“Piece ID”: “7EKCIB5JV3WUU5GNDQRQAJHF7LPRT7ZOC4YGBTCJGAN3JP6X5ZYA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Available Bandwidth”: 29860716743424, “Available Space”: 125350051072}
2020-03-06T22:42:37.096+0100	INFO	piecestore	upload started	{“Piece ID”: “VT3CZE62M4GVC5U27QDUV545NY3FSWIAUXGJVJ3V7MTNNAFMGEOA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”, “Available Bandwidth”: 29860716743424, “Available Space”: 125350051072}
2020-03-06T22:42:37.102+0100	INFO	piecestore	upload failed	{“Piece ID”: “VT3CZE62M4GVC5U27QDUV545NY3FSWIAUXGJVJ3V7MTNNAFMGEOA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”, “error”: “pieces error: filestore error: open r:\storj\temp\blob-221176899.partial: There is not enough space on the disk.”, “errorVerbose”: “pieces error: filestore error: open r:\storj\temp\blob-221176899.partial: There is not enough space on the disk.\n\tstorj.io/storj/storage/filestore.(blobStore).Create:143\n\tstorj.io/storj/storagenode/pieces.(Store).Writer:190\n\tstorj.io/storj/storagenode/piecestore.(Endpoint).doUpload:360\n\tstorj.io/storj/storagenode/piecestore.(drpcEndpoint).Upload:257\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:1066\n\tstorj.io/drpc/drpcserver.(Server).doHandle:199\n\tstorj.io/drpc/drpcserver.(Server).HandleRPC:173\n\tstorj.io/drpc/drpcserver.(Server).ServeOne:124\n\tstorj.io/drpc/drpcserver.(Server).Serve.func2:161\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-03-06T22:42:37.102+0100	INFO	piecestore	upload failed	{“Piece ID”: “7EKCIB5JV3WUU5GNDQRQAJHF7LPRT7ZOC4YGBTCJGAN3JP6X5ZYA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “error”: “pieces error: filestore error: open r:\storj\temp\blob-905275252.partial: There is not enough space on the disk.”, “errorVerbose”: “pieces error: filestore error: open r:\storj\temp\blob-905275252.partial: There is not enough space on the disk.\n\tstorj.io/storj/storage/filestore.(blobStore).Create:143\n\tstorj.io/storj/storagenode/pieces.(Store).Writer:190\n\tstorj.io/storj/storagenode/piecestore.(Endpoint).doUpload:360\n\tstorj.io/storj/storagenode/piecestore.(drpcEndpoint).Upload:257\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:1066\n\tstorj.io/drpc/drpcserver.(Server).doHandle:199\n\tstorj.io/drpc/drpcserver.(Server).HandleRPC:173\n\tstorj.io/drpc/drpcserver.(Server).ServeOne:124\n\tstorj.io/drpc/drpcserver.(Server).Serve.func2:161\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}

I am on windows GUI. Could you please help me investigate this issue?
Thanks in advance

Alexey · March 6, 2020, 11:30pm

Please, show how many free space on the disk (Powershell).

sls "storage.path" "C:\Program Files\Storj\Storage Node\config.yaml"

Get-WmiObject Win32_LogicalDisk | Select-Object DeviceID,Size,FreeSpace | %{"{0} {1:N} GiB {2:N} GiB" -f $_.DeviceID, ($_.Size / 1GB), ($_.FreeSpace / 1GB) }

thev · March 7, 2020, 9:41am

PS R:\> sls "r:" "C:\Program Files\Storj\Storage Node\config.yaml"

C:\Program Files\Storj\Storage Node\config.yaml:8:# color: false
C:\Program Files\Storj\Storage Node\config.yaml:14:# console.static-dir: ""
C:\Program Files\Storj\Storage Node\config.yaml:32:# debug.addr: 127.0.0.1:0
C:\Program Files\Storj\Storage Node\config.yaml:62:# log.caller: false
C:\Program Files\Storj\Storage Node\config.yaml:80:# metrics.addr: collectora.storj.io:9000
C:\Program Files\Storj\Storage Node\config.yaml:158:storage.path: r:\storj\
C:\Program Files\Storj\Storage Node\config.yaml:203:# storage2.retain-time-buffer: 48h0m0s


PS R:\> Get-WmiObject Win32_LogicalDisk | Select-Object DeviceID,Size,FreeSpace | %{"{0} {1:N} GiB {2:N} GiB" -f $_.DeviceID, ($_.Size / 1GB), ($_.FreeSpace / 1GB) }
C: 299.96 GiB 206.25 GiB
D: 350.55 GiB 297.58 GiB
E: 302.73 GiB 41.33 GiB
R: 686.87 GiB 241.50 GiB
Y: 10,240.00 GiB 9,940.21 GiB

My storage space in config.yaml looks like this:

# total allocated disk space in bytes
storage.allocated-disk-space: 0.6 TB

# how much disk space a node at minimum has to advertise
storage2.monitor.minimum-disk-space: 600.0 GB

# path to store data in
storage.path: r:\storj\

Alexey · March 7, 2020, 10:14am

How is your HDD connected to the PC?

thev · March 7, 2020, 3:03pm

These are 3xSSD in RAID 5, all connected directly via SATA

EDIT: is it possible, that Windows Storage Space causes this issue?

phkaham · March 7, 2020, 4:08pm

I have same issue, also have Raid5, started February.

Alexey · March 7, 2020, 6:53pm

Please, reduce it to the default 500.0 GB, save the config and restart the storagenode service. The allocated space is specified in a different option, storage.allocated-disk-space.

thev · March 8, 2020, 12:45am

Looks better now, thanks

|2020-03-08T01:44:39.591+0100|INFO|piecestore|uploaded|{"Piece ID": "L4XXE2HPPR3WIGIFZWDDHTY2IMFAMI67YM4VMMSXJEVASN7MFTQA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}|
|2020-03-08T01:44:42.993+0100|INFO|piecestore|upload started|{"Piece ID": "QXDWRHVBWW6TKGEXHNCCQZDVGNKGN4UC5E2KZYK4SA4G37POO3KQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Bandwidth": 29845132043776, "Available Space": 26680221824}|
|2020-03-08T01:44:43.279+0100|INFO|piecestore|upload started|{"Piece ID": "JCWGUP2FBGWPFRDPH6B777JKC5SA2A4T66RCV5MK43AQM3CPQEYA", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT_REPAIR", "Available Bandwidth": 29845132043776, "Available Space": 26680221824}|
|2020-03-08T01:44:44.796+0100|INFO|piecestore|uploaded|{"Piece ID": "JCWGUP2FBGWPFRDPH6B777JKC5SA2A4T66RCV5MK43AQM3CPQEYA", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT_REPAIR"}|
|2020-03-08T01:44:45.004+0100|INFO|piecestore|upload started|{"Piece ID": "QMCVOU3I7NDIEQI2KFK7DU4PYETONM5UE2VUD5ISWDMAIDRLOYWQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Bandwidth": 29845129726720, "Available Space": 26677904256}|
|2020-03-08T01:44:50.485+0100|INFO|piecestore|upload started|{"Piece ID": "FCO4RZXFEAAOPFR5VIFUYO2S7FP6VSNLXDVZIIOIQAYOOBCJGZLQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Bandwidth": 29845129726720, "Available Space": 26677904256}|
|2020-03-08T01:44:52.991+0100|INFO|piecestore|uploaded|{"Piece ID": "FCO4RZXFEAAOPFR5VIFUYO2S7FP6VSNLXDVZIIOIQAYOOBCJGZLQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}|
|2020-03-08T01:44:53.170+0100|INFO|piecestore|upload started|{"Piece ID": "CSEON6F7CEVNHX3AIEOP3UN5XCYPZRLMXN3Y5X4FOVZ4DO75N76Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Bandwidth": 29845127409664, "Available Space": 26675586688}|
|2020-03-08T01:44:54.382+0100|INFO|piecestore|uploaded|{"Piece ID": "QXDWRHVBWW6TKGEXHNCCQZDVGNKGN4UC5E2KZYK4SA4G37POO3KQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT"}|
|2020-03-08T01:44:54.515+0100|INFO|piecestore|uploaded|{"Piece ID": "QMCVOU3I7NDIEQI2KFK7DU4PYETONM5UE2VUD5ISWDMAIDRLOYWQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT"}|
|2020-03-08T01:44:54.620+0100|INFO|piecestore|uploaded|{"Piece ID": "CSEON6F7CEVNHX3AIEOP3UN5XCYPZRLMXN3Y5X4FOVZ4DO75N76Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}|
|2020-03-08T01:44:57.274+0100|INFO|piecestore|upload started|{"Piece ID": "QRZN2LK6GCNEZ4YDJ3Y6NTJEYETNKUZWZKXNLNMN36YSQJT7HVSQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Bandwidth": 29845120458496, "Available Space": 26668633984}|
|2020-03-08T01:44:57.792+0100|INFO|piecestore|upload started|{"Piece ID": "ZFOI2Z6L2EQYN2NBKQPRBMTMKICKZBSN7PT433KCETWVK4YIQG7Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Bandwidth": 29845120458496, "Available Space": 26668633984}|
|2020-03-08T01:44:58.997+0100|INFO|piecestore|upload started|{"Piece ID": "ROPOLQL4WVC34XCF2KCC2HZNUBGYVBO3BOUJGXI2EMIEUV4PSILQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Bandwidth": 29845120458496, "Available Space": 26668633984}|
|2020-03-08T01:44:59.467+0100|INFO|piecestore|upload started|{"Piece ID": "E22C467Y7PXFDFNTTHWCBHCHHF3NA2TZD6N5WD6RHSHYQON4ICUQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Bandwidth": 29845120458496, "Available Space": 26668633984}|
|2020-03-08T01:44:59.937+0100|INFO|piecestore|upload started|{"Piece ID": "PCWHV4BEOCXLVV3BRPNCCM5YQWPHEHHOLLD6O5TH7WRG36X535JQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Bandwidth": 29845120458496, "Available Space": 26668633984}|
|2020-03-08T01:45:00.787+0100|INFO|piecestore|uploaded|{"Piece ID": "QRZN2LK6GCNEZ4YDJ3Y6NTJEYETNKUZWZKXNLNMN36YSQJT7HVSQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}|
|2020-03-08T01:45:00.888+0100|INFO|piecestore|uploaded|{"Piece ID": "ZFOI2Z6L2EQYN2NBKQPRBMTMKICKZBSN7PT433KCETWVK4YIQG7Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}|
|2020-03-08T01:45:00.963+0100|INFO|piecestore|upload failed|{"Piece ID": "ROPOLQL4WVC34XCF2KCC2HZNUBGYVBO3BOUJGXI2EMIEUV4PSILQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:79\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:489\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:257\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:1066\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:199\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:173\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:124\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:161\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}|
|2020-03-08T01:45:01.027+0100|INFO|piecestore|uploaded|{"Piece ID": "E22C467Y7PXFDFNTTHWCBHCHHF3NA2TZD6N5WD6RHSHYQON4ICUQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}|
|2020-03-08T01:45:01.643+0100|INFO|piecestore|upload started|{"Piece ID": "VRWPQA4UQIUTHMGJK4NCKYQKFBYDGIBDVYLDBDVGXERNMW4UV5VA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Bandwidth": 29845113341952, "Available Space": 26661515392}|

thev · March 8, 2020, 3:41pm

Unfortunately, same issue started to occur this morning

2020-03-08T06:22:54.460+0100	INFO	piecestore	deleted	{"Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "TUN2YPKNOYOWRF6I6NOJFUFRLBFIPZWHQL3ZMXDX6FWFKIPBG3VA"}
2020-03-08T06:22:54.676+0100	INFO	piecestore	upload started	{"Piece ID": "HIGSJSZ2BPBIVRECLHQJDAXAONLROA3JII3VWEJ2AVEPC4ZEYSYA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Bandwidth": 29832816721408, "Available Space": 25479907456}
2020-03-08T06:22:54.774+0100	INFO	piecestore	deleted	{"Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "KOMR3D2J3MCDLOG6AX5MBTEL5KSDAUEDIAEPMYCAHOB452GL7PGQ"}
2020-03-08T06:22:54.774+0100	INFO	piecestore	upload failed	{"Piece ID": "HIGSJSZ2BPBIVRECLHQJDAXAONLROA3JII3VWEJ2AVEPC4ZEYSYA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "pieces error: filestore error: truncate r:\\storj\\temp\\blob-774576829.partial: There is not enough space on the disk.", "errorVerbose": "pieces error: filestore error: truncate r:\\storj\\temp\\blob-774576829.partial: There is not enough space on the disk.\n\tstorj.io/storj/storage/filestore.(*blobStore).Create:143\n\tstorj.io/storj/storagenode/pieces.(*Store).Writer:190\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:360\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:257\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:1066\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:199\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:173\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:124\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:161\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-03-08T06:22:54.989+0100	INFO	piecestore	upload started	{"Piece ID": "TMPCWX6C2PDMWO4JJKXIRG2A6NFFJD36OESSUYOQCDNI73NR2HZQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Bandwidth": 29832816721408, "Available Space": 25482225024}
2020-03-08T06:22:55.087+0100	INFO	piecestore	upload failed	{"Piece ID": "TMPCWX6C2PDMWO4JJKXIRG2A6NFFJD36OESSUYOQCDNI73NR2HZQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "pieces error: filestore error: truncate r:\\storj\\temp\\blob-407077112.partial: There is not enough space on the disk.", "errorVerbose": "pieces error: filestore error: truncate r:\\storj\\temp\\blob-407077112.partial: There is not enough space on the disk.\n\tstorj.io/storj/storage/filestore.(*blobStore).Create:143\n\tstorj.io/storj/storagenode/pieces.(*Store).Writer:190\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:360\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:257\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:1066\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:199\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:173\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:124\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:161\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-03-08T06:22:55.175+0100	INFO	piecestore	uploaded	{"Piece ID": "CCKEND7J3GRHVEDV3F4ZYHEHRTAFLI3VGEQD36PWWR34RHTGHZLQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2020-03-08T06:22:55.912+0100	INFO	piecestore	uploaded	{"Piece ID": "XUVMQSDUMGRWGVWPBGHWTVFCAUNRLV2VPAMC4WLCWYSVDQGZGUNA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2020-03-08T06:22:56.492+0100	INFO	piecestore	upload started	{"Piece ID": "T5KZSPYSN7OJFGWFED3T3UJPS7TATDJPWGJQ3TUSRKE2WZZUAY7A", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT_REPAIR", "Available Bandwidth": 29832812087296, "Available Space": 25477589888}
2020-03-08T06:22:56.589+0100	INFO	piecestore	upload failed	{"Piece ID": "T5KZSPYSN7OJFGWFED3T3UJPS7TATDJPWGJQ3TUSRKE2WZZUAY7A", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT_REPAIR", "error": "pieces error: filestore error: truncate r:\\storj\\temp\\blob-031804407.partial: There is not enough space on the disk.", "errorVerbose": "pieces error: filestore error: truncate r:\\storj\\temp\\blob-031804407.partial: There is not enough space on the disk.\n\tstorj.io/storj/storage/filestore.(*blobStore).Create:143\n\tstorj.io/storj/storagenode/pieces.(*Store).Writer:190\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:360\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:257\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:1066\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:199\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:173\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:124\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:161\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-03-08T06:23:01.358+0100	INFO	piecestore	deleted	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "DKROCSFLCKVVDIZAVOHW7FPBSVNU6JTDQPCHVE7LX46BD7OAVTVQ"}
2020-03-08T06:23:01.406+0100	INFO	piecestore	upload started	{"Piece ID": "LNIWVYWTMVU74GHWGW5WBBDVYU7INSCS67KCAEOADONHRDQ454OA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Bandwidth": 29832812087296, "Available Space": 25479907456}
2020-03-08T06:23:01.972+0100	INFO	piecestore	deleted	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "2XHT4UDVE3KBU3UEDD3ZW3RVJFTZT3EKJRAH37TJSGJ6EIDNQZIA"}

Now it looks like this

2020-03-08T16:41:28.796+0100	INFO	piecestore	upload started	{"Piece ID": "TVNBN4ZMMSGXYXE7WX64QUHHB7AW56Y6HOLAJQOSOI5JLSH726OQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Bandwidth": 29830954511360, "Available Space": 24839016704}
2020-03-08T16:41:28.799+0100	ERROR	piecestore	upload failed	{"Piece ID": "TVNBN4ZMMSGXYXE7WX64QUHHB7AW56Y6HOLAJQOSOI5JLSH726OQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "pieces error: filestore error: open r:\\storj\\temp\\blob-146472957.partial: There is not enough space on the disk.", "errorVerbose": "pieces error: filestore error: open r:\\storj\\temp\\blob-146472957.partial: There is not enough space on the disk.\n\tstorj.io/storj/storage/filestore.(*blobStore).Create:143\n\tstorj.io/storj/storagenode/pieces.(*Store).Writer:190\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:330\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:214\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:986\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:199\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:173\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:124\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:161\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-03-08T16:41:28.868+0100	INFO	piecestore	upload started	{"Piece ID": "3BE6CZULB4H6X77RICT3BNJW6KAJXISW7RWVRPKKYCCJSUZIHJHQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Bandwidth": 29830954511360, "Available Space": 24839016704}
2020-03-08T16:41:28.871+0100	ERROR	piecestore	upload failed	{"Piece ID": "3BE6CZULB4H6X77RICT3BNJW6KAJXISW7RWVRPKKYCCJSUZIHJHQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "pieces error: filestore error: open r:\\storj\\temp\\blob-173163832.partial: There is not enough space on the disk.", "errorVerbose": "pieces error: filestore error: open r:\\storj\\temp\\blob-173163832.partial: There is not enough space on the disk.\n\tstorj.io/storj/storage/filestore.(*blobStore).Create:143\n\tstorj.io/storj/storagenode/pieces.(*Store).Writer:190\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:330\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:214\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:986\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:199\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:173\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:124\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:161\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-03-08T16:41:29.488+0100	INFO	piecestore	upload started	{"Piece ID": "AFG2SSY44BIQ3LD24Q2PSOCQPOS2JFXOB7DC7RFJCTWEXB5OJR6A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Bandwidth": 29830954511360, "Available Space": 24839016704}
2020-03-08T16:41:29.491+0100	ERROR	piecestore	upload failed	{"Piece ID": "AFG2SSY44BIQ3LD24Q2PSOCQPOS2JFXOB7DC7RFJCTWEXB5OJR6A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "pieces error: filestore error: open r:\\storj\\temp\\blob-286129463.partial: There is not enough space on the disk.", "errorVerbose": "pieces error: filestore error: open r:\\storj\\temp\\blob-286129463.partial: There is not enough space on the disk.\n\tstorj.io/storj/storage/filestore.(*blobStore).Create:143\n\tstorj.io/storj/storagenode/pieces.(*Store).Writer:190\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:330\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:214\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:986\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:199\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:173\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:124\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:161\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-03-08T16:41:29.603+0100	INFO	piecestore	upload started	{"Piece ID": "PFIESOPBSLJJAK6PUN534ID2N3VOEMLQEFHIZKONMYOCJMVN25HA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Bandwidth": 29830954511360, "Available Space": 24839016704}
2020-03-08T16:41:29.606+0100	ERROR	piecestore	upload failed	{"Piece ID": "PFIESOPBSLJJAK6PUN534ID2N3VOEMLQEFHIZKONMYOCJMVN25HA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "pieces error: filestore error: open r:\\storj\\temp\\blob-569351466.partial: There is not enough space on the disk.", "errorVerbose": "pieces error: filestore error: open r:\\storj\\temp\\blob-569351466.partial: There is not enough space on the disk.\n\tstorj.io/storj/storage/filestore.(*blobStore).Create:143\n\tstorj.io/storj/storagenode/pieces.(*Store).Writer:190\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:330\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:214\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:986\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:199\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:173\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:124\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:161\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-03-08T16:41:30.043+0100	INFO	piecestore	upload started	{"Piece ID": "OLTS7P6BS4RDEFEFPFPJJ6A4JW6UYTDSCCSFKP6O63PRXJGR5QTA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Bandwidth": 29830954511360, "Available Space": 24839016704}
2020-03-08T16:41:30.046+0100	ERROR	piecestore	upload failed	{"Piece ID": "OLTS7P6BS4RDEFEFPFPJJ6A4JW6UYTDSCCSFKP6O63PRXJGR5QTA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "pieces error: filestore error: open r:\\storj\\temp\\blob-257650817.partial: There is not enough space on the disk.", "errorVerbose": "pieces error: filestore error: open r:\\storj\\temp\\blob-257650817.partial: There is not enough space on the disk.\n\tstorj.io/storj/storage/filestore.(*blobStore).Create:143\n\tstorj.io/storj/storagenode/pieces.(*Store).Writer:190\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:330\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:214\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:986\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:199\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:173\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:124\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:161\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-03-08T16:41:30.552+0100	INFO	piecestore	upload started	{"Piece ID": "DTUU5FMXSUG4XEDHTJQC6SJOYVDLZSFU2MMVK2PU32BGHIB5GMPQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Bandwidth": 29830954511360, "Available Space": 24839016704}
2020-03-08T16:41:30.555+0100	ERROR	piecestore	upload failed	{"Piece ID": "DTUU5FMXSUG4XEDHTJQC6SJOYVDLZSFU2MMVK2PU32BGHIB5GMPQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "pieces error: filestore error: open r:\\storj\\temp\\blob-870663660.partial: There is not enough space on the disk.", "errorVerbose": "pieces error: filestore error: open r:\\storj\\temp\\blob-870663660.partial: There is not enough space on the disk.\n\tstorj.io/storj/storage/filestore.(*blobStore).Create:143\n\tstorj.io/storj/storagenode/pieces.(*Store).Writer:190\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:330\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:214\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:986\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:199\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:173\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:124\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:161\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-03-08T16:41:30.756+0100	INFO	piecestore	upload started	{"Piece ID": "NLXDEM6JNKJNY2KD2FJD7M6HJ54VUW255E5OXIW7QKL4F2STZ36A", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Bandwidth": 29830954511360, "Available Space": 24839016704}
2020-03-08T16:41:30.758+0100	ERROR	piecestore	upload failed	{"Piece ID": "NLXDEM6JNKJNY2KD2FJD7M6HJ54VUW255E5OXIW7QKL4F2STZ36A", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "pieces error: filestore error: open r:\\storj\\temp\\blob-052311643.partial: There is not enough space on the disk.", "errorVerbose": "pieces error: filestore error: open r:\\storj\\temp\\blob-052311643.partial: There is not enough space on the disk.\n\tstorj.io/storj/storage/filestore.(*blobStore).Create:143\n\tstorj.io/storj/storagenode/pieces.(*Store).Writer:190\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:330\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:214\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:986\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:199\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:173\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:124\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:161\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-03-08T16:41:34.245+0100	INFO	piecestore	upload started	{"Piece ID": "GB65PLIWCGWTHOEWLYXA7KO7IQUHJYAWVLXTEXCQ6COUTSKIECUQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Bandwidth": 29830954511360, "Available Space": 24839016704}
2020-03-08T16:41:34.249+0100	ERROR	piecestore	upload failed	{"Piece ID": "GB65PLIWCGWTHOEWLYXA7KO7IQUHJYAWVLXTEXCQ6COUTSKIECUQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "pieces error: filestore error: open r:\\storj\\temp\\blob-553510910.partial: There is not enough space on the disk.", "errorVerbose": "pieces error: filestore error: open r:\\storj\\temp\\blob-553510910.partial: There is not enough space on the disk.\n\tstorj.io/storj/storage/filestore.(*blobStore).Create:143\n\tstorj.io/storj/storagenode/pieces.(*Store).Writer:190\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:330\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:214\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:986\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:199\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:173\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:124\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:161\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-03-08T16:41:43.327+0100	INFO	piecestore	upload started	{"Piece ID": "2G3XVWBK4RVEORL54VVQAN6AEGWG2XUMJYDYPJOQ4UOH3WQ7VGPA", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT_REPAIR", "Available Bandwidth": 29830954511360, "Available Space": 24839016704}
2020-03-08T16:41:43.330+0100	ERROR	piecestore	upload failed	{"Piece ID": "2G3XVWBK4RVEORL54VVQAN6AEGWG2XUMJYDYPJOQ4UOH3WQ7VGPA", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT_REPAIR", "error": "pieces error: filestore error: open r:\\storj\\temp\\blob-520871749.partial: There is not enough space on the disk.", "errorVerbose": "pieces error: filestore error: open r:\\storj\\temp\\blob-520871749.partial: There is not enough space on the disk.\n\tstorj.io/storj/storage/filestore.(*blobStore).Create:143\n\tstorj.io/storj/storagenode/pieces.(*Store).Writer:190\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:330\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:214\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:986\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:199\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:173\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:124\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:161\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-03-08T16:41:43.572+0100	INFO	piecestore	upload started	{"Piece ID": "IPD6IHVCWZEEOAM33XMBHUZA3GIQN64QZ2NOE33M5N6CJSJSI6RQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Bandwidth": 29830954511360, "Available Space": 24839016704}
2020-03-08T16:41:43.575+0100	ERROR	piecestore	upload failed	{"Piece ID": "IPD6IHVCWZEEOAM33XMBHUZA3GIQN64QZ2NOE33M5N6CJSJSI6RQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "pieces error: filestore error: open r:\\storj\\temp\\blob-862735328.partial: There is not enough space on the disk.", "errorVerbose": "pieces error: filestore error: open r:\\storj\\temp\\blob-862735328.partial: There is not enough space on the disk.\n\tstorj.io/storj/storage/filestore.(*blobStore).Create:143\n\tstorj.io/storj/storagenode/pieces.(*Store).Writer:190\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:330\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:214\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:986\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:199\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:173\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:124\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:161\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-03-08T16:41:44.921+0100	INFO	piecestore	upload started	{"Piece ID": "CNROE3OSY54IX2MH6QD2ZEYPTEQGZYARIXYPSR5J662Y7GJVZWXA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Bandwidth": 29830954511360, "Available Space": 24839016704}
2020-03-08T16:41:44.924+0100	ERROR	piecestore	upload failed	{"Piece ID": "CNROE3OSY54IX2MH6QD2ZEYPTEQGZYARIXYPSR5J662Y7GJVZWXA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "pieces error: filestore error: open r:\\storj\\temp\\blob-745489343.partial: There is not enough space on the disk.", "errorVerbose": "pieces error: filestore error: open r:\\storj\\temp\\blob-745489343.partial: There is not enough space on the disk.\n\tstorj.io/storj/storage/filestore.(*blobStore).Create:143\n\tstorj.io/storj/storagenode/pieces.(*Store).Writer:190\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:330\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:214\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:986\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:199\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:173\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:124\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:161\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-03-08T16:41:53.154+0100	INFO	version	running on version v0.34.6
2020-03-08T16:41:56.329+0100	INFO	piecestore	upload started	{"Piece ID": "7GTR5PA3HUEJ3ZP5HWJRB3WP5EI6CW2VJJII6TDOOJX2XO35HC7A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Bandwidth": 29830954511360, "Available Space": 24839016704}
2020-03-08T16:41:56.332+0100	ERROR	piecestore	upload failed	{"Piece ID": "7GTR5PA3HUEJ3ZP5HWJRB3WP5EI6CW2VJJII6TDOOJX2XO35HC7A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "pieces error: filestore error: open r:\\storj\\temp\\blob-725770770.partial: There is not enough space on the disk.", "errorVerbose": "pieces error: filestore error: open r:\\storj\\temp\\blob-725770770.partial: There is not enough space on the disk.\n\tstorj.io/storj/storage/filestore.(*blobStore).Create:143\n\tstorj.io/storj/storagenode/pieces.(*Store).Writer:190\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:330\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:214\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:986\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:199\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:173\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:124\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:161\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-03-08T16:41:59.317+0100	INFO	piecestore	upload started	{"Piece ID": "ZDXYTI3SGK7UX2I6VVSTVSV4F7AZGLO3Y763XJLZXFJJIFTLMSXQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Bandwidth": 29830954511360, "Available Space": 24839016704}
2020-03-08T16:41:59.321+0100	ERROR	piecestore	upload failed	{"Piece ID": "ZDXYTI3SGK7UX2I6VVSTVSV4F7AZGLO3Y763XJLZXFJJIFTLMSXQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "pieces error: filestore error: open r:\\storj\\temp\\blob-182157897.partial: There is not enough space on the disk.", "errorVerbose": "pieces error: filestore error: open r:\\storj\\temp\\blob-182157897.partial: There is not enough space on the disk.\n\tstorj.io/storj/storage/filestore.(*blobStore).Create:143\n\tstorj.io/storj/storagenode/pieces.(*Store).Writer:190\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:330\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:214\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:986\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:199\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:173\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:124\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:161\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-03-08T16:42:11.552+0100	INFO	piecestore	upload started	{"Piece ID": "CCSFFDPAV57D6X3K4IFCI2MGKBHW5JUMKYA4TFKKRWKO7H74ZRLQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Bandwidth": 29830954511360, "Available Space": 24839016704}
2020-03-08T16:42:11.555+0100	ERROR	piecestore	upload failed	{"Piece ID": "CCSFFDPAV57D6X3K4IFCI2MGKBHW5JUMKYA4TFKKRWKO7H74ZRLQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "pieces error: filestore error: open r:\\storj\\temp\\blob-084714260.partial: There is not enough space on the disk.", "errorVerbose": "pieces error: filestore error: open r:\\storj\\temp\\blob-084714260.partial: There is not enough space on the disk.\n\tstorj.io/storj/storage/filestore.(*blobStore).Create:143\n\tstorj.io/storj/storagenode/pieces.(*Store).Writer:190\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:330\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:214\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:986\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:199\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:173\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:124\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:161\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-03-08T16:42:16.174+0100	INFO	piecestore	upload started	{"Piece ID": "6MSVYC376JIB5GC7TEP6OUDBTBI4M7XKG7YAF23BVL24L7JXR2DA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Bandwidth": 29830954511360, "Available Space": 24839016704}
2020-03-08T16:42:16.177+0100	ERROR	piecestore	upload failed	{"Piece ID": "6MSVYC376JIB5GC7TEP6OUDBTBI4M7XKG7YAF23BVL24L7JXR2DA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "pieces error: filestore error: open r:\\storj\\temp\\blob-780736355.partial: There is not enough space on the disk.", "errorVerbose": "pieces error: filestore error: open r:\\storj\\temp\\blob-780736355.partial: There is not enough space on the disk.\n\tstorj.io/storj/storage/filestore.(*blobStore).Create:143\n\tstorj.io/storj/storagenode/pieces.(*Store).Writer:190\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:330\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:214\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:986\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:199\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:173\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:124\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:161\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-03-08T16:42:16.429+0100	INFO	piecestore	upload started	{"Piece ID": "YE6TDL4XPN4VZO5ZDHGFCQTRRV3ZEO5HH5XVRFNLBAQWXHO7S6BQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Bandwidth": 29830954511360, "Available Space": 24839016704}
2020-03-08T16:42:16.432+0100	ERROR	piecestore	upload failed	{"Piece ID": "YE6TDL4XPN4VZO5ZDHGFCQTRRV3ZEO5HH5XVRFNLBAQWXHO7S6BQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "pieces error: filestore error: open r:\\storj\\temp\\blob-379190630.partial: There is not enough space on the disk.", "errorVerbose": "pieces error: filestore error: open r:\\storj\\temp\\blob-379190630.partial: There is not enough space on the disk.\n\tstorj.io/storj/storage/filestore.(*blobStore).Create:143\n\tstorj.io/storj/storagenode/pieces.(*Store).Writer:190\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:330\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:214\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:986\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:199\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:173\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:124\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:161\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}

Alexey · March 8, 2020, 5:32pm

How many free space now?

thev · March 8, 2020, 8:00pm

R: 686.87 GiB 241.49 GiB

Alexey · March 8, 2020, 8:31pm

Could you check all databases on your disk: https://support.storj.io/hc/en-us/articles/360029309111-How-to-fix-a-database-disk-image-is-malformed-

thev · March 8, 2020, 8:53pm

PS R:\storj> sqlite3.exe .\bandwidth.db "PRAGMA integrity_check;"
ok
PS R:\storj> sqlite3.exe .\storage_usage.db "PRAGMA integrity_check;"
ok
PS R:\storj> sqlite3.exe .\info.db "PRAGMA integrity_check;"
ok
PS R:\storj> sqlite3.exe .\notifications.db "PRAGMA integrity_check;"
ok
PS R:\storj> sqlite3.exe .\orders.db "PRAGMA integrity_check;"
ok
PS R:\storj> sqlite3.exe .\piece_expiration.db "PRAGMA integrity_check;"
ok
PS R:\storj> sqlite3.exe .\piece_space_used.db "PRAGMA integrity_check;"
ok
PS R:\storj> sqlite3.exe .\pieceinfo.db "PRAGMA integrity_check;"
ok
PS R:\storj> sqlite3.exe .\reputation.db "PRAGMA integrity_check;"
ok
PS R:\storj> sqlite3.exe .\satellites.db "PRAGMA integrity_check;"
ok
PS R:\storj> sqlite3.exe .\used_serial.db "PRAGMA integrity_check;"
ok

Alexey · March 8, 2020, 9:39pm

Thank you!
Please, run this command in the PowerShell:

Get-ChildItem r:\storj -Directory | %{ "$_ {0:N2} GiB" -f ((Get-ChildItem $_.FullName -Recurse | Measure-Object -Property Length -Sum -ErrorAction Stop).Sum / 1GB)}

thev · March 8, 2020, 10:29pm

PS R:\> Get-ChildItem r:\storj -Directory | %{ "$_ {0:N2} GiB" -f ((Get-ChildItem $_.FullName -Recurse | Measure-Object -Property Length -Sum -ErrorAction Stop).Sum / 1GB)}
blobs 441.11 GiB
garbage 0.01 GiB
temp 0.02 GiB
trash 1.41 GiB

Alexey · March 9, 2020, 7:42am

Have you had any problems with the databases of this node?

Please run these commands against copy of the piece_spaced_used.db:

sqlite3 piece_spaced_used.db
select sum(total)/1024/1024/1024 from piece_space_used where satellite_id is not null;
.exit

thev · March 9, 2020, 9:11am

I haven’t had any troubles with this node so far - running from December with no issues

PS R:\storj> sqlite3 piece_spaced_used.db
SQLite version 3.28.0 2019-04-16 19:49:53
Enter ".help" for usage hints.
sqlite> select sum(total)/1024/1024/1024 from piece_space_used where satellite_id is not null;
442
sqlite> .exit
PS R:\storj>

Alexey · March 9, 2020, 8:48pm

Looks right, the database is consistent with disk usage.
Seems the calculation of free space is broken.

thev · March 13, 2020, 11:17am

Is there anything I can do or the only option I have is to run new node from scratch and start gracefull exit on failing node?

UPDATE: I’ve just realiezed that one of the requirements is to have a node that joined joined the network more than 15 month ago so I guess graceful exit is not an option in my case - I will loose all reserved credits for this node. @Alexey, could you confirm?

baker · March 13, 2020, 1:10pm

Currently graceful exit can be done after 6 months. This has been temporarily reduced, and will increase again some time in the future. Unfortunately if you started in Dec, this won’t help you.

I don’t think you’re at the point where you should kill your existing node. Although I don’t have an answer for you, it doesn’t look like your node is beyond repair. Plus you likely have a lot of January payout tied up in escrow, so I think it would be worth holding off on starting over.