Negative value of remaining space (-1.7 Mb available)

Node Operators FAQ
1

Hi all.

I am encountered an error as storage node operator.

Here are my steps to reproduce this error

  1. I am allocated 1.5 Tb of space for storage node. All space was allocated, 2.0 Mb was left, no incoming PUT requests in logs.
  2. Then I am increased available space to 1.7 Tb. Node continues to download data (200 GB).
  3. 1.7 Tb was filled. But when I call the dashboard I see a NEGATIVE value of remaining space (-1.7 Mb). Satellites continue trying to make PUT requests to my node, but all those PUT requests comes up with an error.

My logs:

2019-08-06T20:21:02.920Z	INFO	piecestore	upload failed	{"Piece ID": "7SOAS57PXTORZBWSAMPPMO5IBF44M2KV2VFRHYEHKZEMQ2IGX67A", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT", "error": "piecestore protocol: out of space", "errorVerbose": "piecestore protocol: out of space\n\tstorj . io/storj/storagenode/piecestore.(*Endpoint).Upload:316\n\tstorj . io/storj/pkg/pb._Piecestore_Upload_Handler:701\n\tstorj . io/storj/pkg/server.logOnErrorStreamInterceptor:23\n\tgoogle . golang . org/grpc.(*Server).processStreamingRPC:1209\n\tgoogle . golang . org/grpc.(*Server).handleStream:1282\n\tgoogle . golang . org/grpc.(*Server).serveStreams.func1.1:717"}
2019-08-06T20:21:02.920Z	ERROR	piecestore protocol: out of space
	storj . io/storj/storagenode/piecestore.(*Endpoint).Upload:316
	storj . io/storj/pkg/pb._Piecestore_Upload_Handler:701
	storj.io/storj/pkg/server.logOnErrorStreamInterceptor:23
	google . golang . org/grpc.(*Server).processStreamingRPC:1209
	google . golang . org/grpc.(*Server).handleStream:1282
	google . golang . org/grpc.(*Server).serveStreams.func1.1:717

GET requests finishes fine, GET_AUDIT requests is also ok.

Here are my questions:

  1. Will my node reputation suffer from continuously failing PUT requests?
  2. How can I tell satellites to stop making PUT requests to my node?
  3. I tried to allocate some additional space in my node (set available space to 1.71 GB) but it didnt’t help (docker stop, docker rm, docker run with new parameters). Available space still -1.7 Mb.

If I stop my storage node my reputation will suffer (lack of uptime). If I continue to run my node my reputation will suffer (because of errors).

What can I do in this situation?

2

UPD: restarted my node three times. Available space still -1.7 Mb but PUT requests from satellites are stopped.

3

Error when posting log messages to forum: I see a warning “new members can post only 2 links per message”, but I don’t post links, I just post an error stack containing paths to Go components (google[DOT]golang[DOT]org) - can you make an exception for “links” like this (now I have to manually place spaces in log messages or didn’t attach stack traces at all witch is not a good idea if I want to report an error)?

4

The negative numbers means that storagenode used more space, than allocated. It still could be possible.
We don’t recommend put to allocation more than 90% of available space, because it could be overused, since it’s alpha.
Please, keep your node online. Sometimes data downloaded back by customers and could be removed if they removes it.

1 Like