Size discrepancy only on 1 node

Hi,

I started getting warnings from my Linux OS regarding the amount of free space left available on the disk I use for one of my nodes. According to the dashboard I have this:

However, according to the OS, I’m using 2TB more:

rocky@ns8:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
...
/dev/sdc1        15T   14T  1.5T  91% /mnt/storagenode

Another couple of commands:

storj@ns8:~$ du -s --si --apparent-size /mnt/storagenode
12T     /mnt/storagenode
storj@ns8:~$ du -s --si  /mnt/storagenode
15T     /mnt/storagenode
storj@ns8:~$

The disk in question is a zvol on my NAS accessed via iSCSI.

Any ideas on what could be eating the extra 2TB or how I investigate this.

What the filesystem on that zvol? Also, do you use symlinks/hardlinks?

It’s xfs. No hard/soft links.

Under /mnt I have a local directory for storagenode.db via: storage2.database-dir: /mnt/storagenode.db/

And everything else is under the iSCSI mount of /mnt/storagenode

See if this applies: Note about hashstore preallocation on XFS

What a reclaimable usage? Is it included to “Used”?

Bingo, that really looks like it. Comparing the sizes at various directory levels I get:

Running du with --apparent-size:

9.4T    /mnt/storagenode/data/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S
249G    /mnt/storagenode/data/storage/hashstore/121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6
757G    /mnt/storagenode/data/storage/hashstore/12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs
516M    /mnt/storagenode/data/storage/hashstore/1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE

11T     /mnt/storagenode/data/storage/hashstore
11T     /mnt/storagenode/data/storage

And without:

13T     /mnt/storagenode/data/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S
333G    /mnt/storagenode/data/storage/hashstore/121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6
1006G   /mnt/storagenode/data/storage/hashstore/12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs
1.6G    /mnt/storagenode/data/storage/hashstore/1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE

14T     /mnt/storagenode/data/storage/hashstore
14T     /mnt/storagenode/data/storage

It looks like the “Used” value is the actual size of the data, not the amount of disk spce it’s using.