Disk space is less than requested

Node Operators troubleshooting
1

After storage node update to v0.31.9 I see this warning in log:

2020-01-30T09:11:33.179673871Z 2020-01-30T09:11:33.179Z INFO    piecestore:monitor      Remaining Bandwidth     {"bytes": 319215795429120}
2020-01-30T09:11:33.179799004Z 2020-01-30T09:11:33.179Z WARN    piecestore:monitor      Disk space is less than requested. Allocating space     {"bytes": 6704352774912}
2020-01-30T09:11:33.257962522Z 2020-01-30T09:11:33.257Z INFO    version running on version v0.31.9

df -h

image
(3.9+6.1= 10T)
df

image
image728Ă—181 6.19 KB

Docker run have -e STORAGE="10TB"

So, calculation used space+freespace is wrong?

I changed docker run to -e STORAGE="7TB" and recreate storage node, but warning is the same.

So, it definitely bug in space calculations (free + already used by storage node)

7 Likes
2

It seems like I have the same problem:
2020-01-30T07:51:16.320Z WARN piecestore:monitor Disk space is less than requested. Allocating space {“bytes”: 194734370816}

Didn’t check initially because it started correctly and dashboard looks ok…

Only my 3rd node doesn’t have the problem but it is running on a 4TB drive and only has 500GB allocated.

1 Like
3

Hi,

after the update to the newest version, my nodes don’t start anymore. Here are the logs:

2020-01-30T15:59:40.667Z INFO version running on version v0.31.9
2020-01-30T15:59:40.674Z INFO db.migration Database Version {“version”: 29}
2020-01-30T15:59:46.259Z INFO preflight:localtime start checking local system clock with trusted satellites’ system clock.
2020-01-30T15:59:47.118Z INFO preflight:localtime local system clock is in sync with trusted satellites’ system clock.
2020-01-30T15:59:47.119Z INFO bandwidth Performing bandwidth usage rollups
2020-01-30T15:59:47.119Z INFO trust Scheduling next refresh {“after”: “6h28m45.657780227s”}
2020-01-30T15:59:47.119Z INFO Node 1fF2Mcm23Yp… started
2020-01-30T15:59:47.119Z INFO Public server started on [::]:28967
2020-01-30T15:59:47.119Z INFO Private server started on 127.0.0.1:7778
2020-01-30T15:59:47.127Z INFO piecestore:monitor Remaining Bandwidth {“bytes”: 1999959247955456}
2020-01-30T15:59:47.127Z WARN piecestore:monitor Disk space is less than requested. Allocating space {“bytes”: 86292287488}
2020-01-30T15:59:47.127Z ERROR piecestore:monitor Total disk space less than required minimum {“bytes”: 300000000000}
2020-01-30T15:59:47.128Z ERROR version Failed to do periodic version check: version control client error: Get https://version.storj.io: context c anceled
2020-01-30T15:59:47.128Z ERROR pieces:trash emptying trash failed {“error”: “pieces error: filestore error: context canceled”, "errorVerbose ": “pieces error: filestore error: context canceled\n\tstorj.io/storj/storage/filestore.(*blobStore).EmptyTrash:127\n\tstorj.io/storj/storagenode/pieces.( *BlobsUsageCache).EmptyTrash:278\n\tstorj.io/storj/storagenode/pieces.(*Store).EmptyTrash:329\n\tstorj.io/storj/storagenode/pieces.(*TrashChore).Run.func1 :51\n\tstorj.io/common/sync2.(*Cycle).Run:87\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2020-01-30T15:59:47.128Z ERROR pieces:trash emptying trash failed {“error”: “pieces error: filestore error: context canceled”, "errorVerbose ": “pieces error: filestore error: context canceled\n\tstorj.io/storj/storage/filestore.(*blobStore).EmptyTrash:127\n\tstorj.io/storj/storagenode/pieces.( *BlobsUsageCache).EmptyTrash:278\n\tstorj.io/storj/storagenode/pieces.(*Store).EmptyTrash:329\n\tstorj.io/storj/storagenode/pieces.(*TrashChore).Run.func1 :51\n\tstorj.io/common/sync2.(*Cycle).Run:87\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2020-01-30T15:59:47.128Z ERROR pieces:trash emptying trash failed {“error”: “pieces error: filestore error: context canceled”, "errorVerbose ": “pieces error: filestore error: context canceled\n\tstorj.io/storj/storage/filestore.(*blobStore).EmptyTrash:127\n\tstorj.io/storj/storagenode/pieces.( *BlobsUsageCache).EmptyTrash:278\n\tstorj.io/storj/storagenode/pieces.(*Store).EmptyTrash:329\n\tstorj.io/storj/storagenode/pieces.(*TrashChore).Run.func1 :51\n\tstorj.io/common/sync2.(*Cycle).Run:87\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2020-01-30T15:59:47.128Z ERROR pieces:trash emptying trash failed {“error”: “pieces error: filestore error: context canceled”, "errorVerbose ": “pieces error: filestore error: context canceled\n\tstorj.io/storj/storage/filestore.(*blobStore).EmptyTrash:127\n\tstorj.io/storj/storagenode/pieces.( *BlobsUsageCache).EmptyTrash:278\n\tstorj.io/storj/storagenode/pieces.(*Store).EmptyTrash:329\n\tstorj.io/storj/storagenode/pieces.(*TrashChore).Run.func1 :51\n\tstorj.io/common/sync2.(*Cycle).Run:87\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
Error: piecestore monitor: disk space requirement not met

I got nearly the same failure on 3 different Nodes. Before the update all Nodes running fine.

4

After updating to v0.31.9 I noticed this warning in the log (it probably was there before, but I did not notice it)

WARN    piecestore:monitor      Disk space is less than requested. Allocating space     {"bytes": 3278521470976}

It is strange, because I have enough space on the drive:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       7.5T  4.2T  3.0T  59% /storj

There is only the node data on the disk and I have set 7200GB as the disk space in the node start command.

Does the node check for enough free space, while ignoring the space that it has used?
The node seems to work OK, just that this warning is weird.

2 Likes
5

I got this error on one of my nodes with 0.30.5 when it upgraded but it didn’t happen with 0.31.9 so I’m not sure why it threw that. The dashboard showed the correct amount of advertised capacity. Definitely be curious why this shows up in case it happens in the future.

6

A new update has just been released that should fix the used space calculation. Check it out here:

2 Likes
7

Thanks a lot @littleskunk !