Storagenode have free space, during log show node full, and no ingres for 3 days

Logsat some point shown that node full

020-07-11T16:34:09.035+0300 INFO piecestore upload started {“Piece ID”: “AQT22IGTC73TUKOGHDTHNH53J22EUWOMDBPALMFBZFXT5MIZTCBA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “PUT”, “Available Space”: -1907390620928}
2020-07-11T16:34:09.041+0300 ERROR piecestore upload failed {“Piece ID”: “AQT22IGTC73TUKOGHDTHNH53J22EUWOMDBPALMFBZFXT5MIZTCBA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “PUT”, “error”: “out of space”, “errorVerbose”: “out of space\n\tstorj.io/common/rpc/rpcstatus.Error:82\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:360\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:996\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}

at same time on dashboard it shot 900+ GB free space, and no ingress for 3 days since this log input.

Any coment from storj team?

this free space not got free during deletion of data, some got free from GE on stefen benten, but it was before i got some data.

Now after restart it looks like made some recalculations and started geting data. So i think during some Node not calculated durig GE deleted data i think.

There seem to be a few scenarios in which used space isn’t calculated correctly. This sounds very similar to what happens during GC, where it’s counted twice. A restart usually helps, but shouldn’t be needed.

Yes, it’s just need sometimes a few hours before update the information.

the best informatic solution.

Turn off,
turn on
:rofl:

Works most of the time :sweat_smile:

1 Like

:man_shrugging:
The binary logic, so binary solutions

1 Like