# Disk usage discrepancy?

**URL:** <https://forum.storj.io/t/disk-usage-discrepancy/24715>\
**Category:** troubleshooting\
**Tags:** storage, xfs, btrfs, ntfs, zfs\
**Created:** [December 19, 2023, 6:18pm UTC](https://forum.storj.io/t/disk-usage-discrepancy/24715 "2023-12-19T18:18:00Z")\
**Posts on this page:** 20\
**Page:** 23

<div class="post-metadata">

**Author:** ![digitalfrank](https://storj.bcdn.literatehosting.com/letter_avatar/digitalfrank/32/6_449c533f586939b6e5f246632a46421f.png) [@digitalfrank](https://forum.storj.io/u/digitalfrank)\
**Post date:** [March 25, 2024, 2:00pm UTC](https://forum.storj.io/t/disk-usage-discrepancy/24715/480 "2024-03-25T14:00:04Z")

</div>

Hi, i have this flag activated for all my nodes. I have 30 nodes but i don’t have nothing error or slowly filewalker. I have activate the filewalker with high priority. I have nodes windows on VM with 12 core e 48gb ram. I have 12 nodes on this machine.

@daki82 you have experience that if this flag is off, the filewalker run more speedly?

---

<div class="post-metadata">

**Author:** ![Vicente](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/vicente/32/1123_2.png) [@Vicente](https://forum.storj.io/u/Vicente)\
**Post date:** [March 25, 2024, 2:52pm UTC](https://forum.storj.io/t/disk-usage-discrepancy/24715/481 "2024-03-25T14:52:30Z")

</div>

I have unchecked, it is processing files.

---

<div class="post-metadata">

**Author:** ![daki82](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/daki82/32/1128_2.png) [@daki82](https://forum.storj.io/u/daki82)\
**Post date:** [March 25, 2024, 8:05pm UTC](https://forum.storj.io/t/disk-usage-discrepancy/24715/482 "2024-03-25T20:05:37Z")

</div>

Well, i guess it does not have to update the index file all the time…who knows.

> **[Search indexing in Windows 10: FAQ - Microsoft Support](https://support.microsoft.com/en-us/windows/search-indexing-in-windows-10-faq-da061c83-af6b-095c-0f7a-4dfecda4d15a)**
>
> Get answers to questions about how indexing affects searches in Windows 10.

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [March 26, 2024, 3:18am UTC](https://forum.storj.io/t/disk-usage-discrepancy/24715/483 "2024-03-26T03:18:50Z")

</div>

Unlikely. This indexing thing took less iops than all believes.  
As far as I can see, almost any Windows VM is affected, especially if the Windows VM is running not on Hyper-V but on some Linux distro. The bare metal or a Linux VM with ext4 likely would not have an issue.

How did you connect the disk to this VM?

---

<div class="post-metadata">

**Author:** ![digitalfrank](https://storj.bcdn.literatehosting.com/letter_avatar/digitalfrank/32/6_449c533f586939b6e5f246632a46421f.png) [@digitalfrank](https://forum.storj.io/u/digitalfrank)\
**Post date:** [March 26, 2024, 5:01am UTC](https://forum.storj.io/t/disk-usage-discrepancy/24715/484 "2024-03-26T05:01:23Z")

</div>

the vm is on vmware esxi. the disk to the VM was connected by creating a raid 0 in the raid card and then created a datastore of the entire disk on ESXI and then passed completely to the VM.

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [March 26, 2024, 8:14am UTC](https://forum.storj.io/t/disk-usage-discrepancy/24715/485 "2024-03-26T08:14:07Z")

</div>

> [@digitalfrank](#):
>
> raid 0

1 disk failure and data is gone. But I think you know this.  
As expected it’s not a Hyper-V VM, where they have a relatively good integration.  
You may try this method to improve speed of the disk subsystem:

> [@NTFS Disable 8dot3name](https://forum.storj.io/t/ntfs-disable-8dot3name/25624):
>
> Fresh off an ReFS volume that went RAW, I’ve decided to be equally reckless and disable the 8 dot 3 short names for my replacement NTFS volume. I was shocked to not find a single mention of it in this forum, so I’ll report back once this bad decision ends up in another catastrophic fail, but the numbers looks promising for the Storj trillion tiny file configurations. Disable across system: fsutil behavior set disable8dot3 Strip existing: fsutil 8dot3name strip /f…

The final goal - is to have all filewalkers successfully finish their work:

> [@Storagenode.exe stopping for too long, for 10h+](https://forum.storj.io/t/storagenode-exe-stopping-for-too-long-for-10h/25316/12):
>
> Windows/PowerShell Logs should be at least info level Used space filewalker Regulated by the parameter in the config.yaml file or as a command line argument (add -- before the parameter, remove :, place it after the image name in your docker run command): # if set to true, all pieces disk usage is recalculated on startup (default true) storage2.piece-scan-on-startup: true to watch for a progress: cat "C:\Program Files\Storj\Storage Node\*.log" -Wait | sls "used-space-filewalker" | sls "sta…

---

<div class="post-metadata">

**Author:** ![digitalfrank](https://storj.bcdn.literatehosting.com/letter_avatar/digitalfrank/32/6_449c533f586939b6e5f246632a46421f.png) [@digitalfrank](https://forum.storj.io/u/digitalfrank)\
**Post date:** [March 27, 2024, 5:58am UTC](https://forum.storj.io/t/disk-usage-discrepancy/24715/486 "2024-03-27T05:58:22Z")

</div>

Thanks @Alexey ,I certainly know the problems of raid 0. But to use the storj policy or more importantly the advice to use one disk per node and not use raid 1 or raid 5 or 6, I did it this way. However, I have not had a problem for over 3.5 years and the disks complete the filewalkers completely and I have never had differences in used space or strange blocks apart from once on a node but there it was my mistake. I currently have the multinode dashboard which shows me that I have 120tb occupied and the monthly average shows me 116.5 but this month I had problems with scheduled electricity downtime due to maintenance of the supplier’s electrical substations and therefore I have an online audit at 98% which negatively affects the average. The rest is all ok. Anyway thanks for the advice and the notification, I learned something else I didn’t know.

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [March 27, 2024, 7:25am UTC](https://forum.storj.io/t/disk-usage-discrepancy/24715/487 "2024-03-27T07:25:00Z")

</div>

> [@digitalfrank](#):
>
> But to use the storj policy or more importantly the advice to use one disk per node and not use raid 1 or raid 5 or 6, I did it this way.

This is not a policy, we recommended to do not use any RAID at all (any level), unless it’s already exist.  
But yes, it should be one node per disk, see [How to add an additional drive? - Storj Docs](https://docs.storj.io/node/faq/how-to-add-an-additional-drive).

> [@digitalfrank](#):
>
> However, I have not had a problem for over 3.5 years and the disks complete the filewalkers completely and I have never had differences in used space or strange blocks apart from once on a node but there it was my mistake.

Yes, the problem become noticeable when your disk is close to a full capacity, including the how NTFS works and that it requires a periodic defragmentation, especially after such usage produced by our customers.  
The virtualization makes situation only worse, especially when you use Windows VM on the Linux host. It’s better to use docker in this case - much less wasted resources and IOPSes.

---

<div class="post-metadata">

**Author:** ![GiantJack](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/giantjack/32/1122_2.png) [@GiantJack](https://forum.storj.io/u/GiantJack)\
**Post date:** [March 28, 2024, 12:16am UTC](https://forum.storj.io/t/disk-usage-discrepancy/24715/488 "2024-03-28T00:16:55Z")

</div>

just noticed 1 node is not getting new data, checked system:

```auto
root@raspberrypi:/store# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 29G 9.8G 18G 36% /
/dev/sda1 9.1T 9.0T 39G 100% /store

```

 ![image](https://storj-s3.bcdn.literatehosting.net/original/2X/a/a471972f804b3e9db90bf69673dd83ebfd2d809a.png)

what is the best way to get this fixed?

---

<div class="post-metadata">

**Author:** ![nerdatwork](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/nerdatwork/32/1135_2.png) [@nerdatwork](https://forum.storj.io/u/nerdatwork)\
**Post date:** [March 28, 2024, 12:42am UTC](https://forum.storj.io/t/disk-usage-discrepancy/24715/489 "2024-03-28T00:42:26Z")

</div>

> [@GiantJack](#):
>
> not getting new data,

What is the version of this node ?

Have you read this thread ? 👇

> [@Disk usage discrepancy?](https://forum.storj.io/t/disk-usage-discrepancy/24715):
>
> something wrong with my node???..online score just drop down…how to get 100% again?

---

<div class="post-metadata">

**Author:** ![GiantJack](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/giantjack/32/1122_2.png) [@GiantJack](https://forum.storj.io/u/GiantJack)\
**Post date:** [March 28, 2024, 2:10am UTC](https://forum.storj.io/t/disk-usage-discrepancy/24715/490 "2024-03-28T02:10:33Z")

</div>

version 1.96.6  
i did take a fast look but was thinking its another problem because i miss 2.3TB not some GB only (i would not care) but this much feels bad

---

<div class="post-metadata">

**Author:** ![GiantJack](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/giantjack/32/1122_2.png) [@GiantJack](https://forum.storj.io/u/GiantJack)\
**Post date:** [March 28, 2024, 2:17am UTC](https://forum.storj.io/t/disk-usage-discrepancy/24715/491 "2024-03-28T02:17:13Z")

</div>

maybe i should not open a new thread, looks like same problem on 1 of my nodes:  
just noticed 1 node is not getting new data, checked system:

```auto
root@raspberrypi:/store# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 29G 9.8G 18G 36% /
/dev/sda1 9.1T 9.0T 39G 100% /store

```

 ![image](https://storj-s3.bcdn.literatehosting.net/original/2X/a/a471972f804b3e9db90bf69673dd83ebfd2d809a.png)

RPI 2  
9TB external HDD  
ext4 (like all my nodes but only this one scam my free space)

makes me a bit angry this bug, cant be that hard for this piece of software to check what files it think are used (the 6.66TB ) and compre to what is really on the filesystem… in other words please fix this annoying bug asap

---

<div class="post-metadata">

**Author:** ![nerdatwork](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/nerdatwork/32/1135_2.png) [@nerdatwork](https://forum.storj.io/u/nerdatwork)\
**Post date:** [March 28, 2024, 2:20am UTC](https://forum.storj.io/t/disk-usage-discrepancy/24715/492 "2024-03-28T02:20:00Z")

</div>

Do you see errors in your log about filewalkers ?

---

<div class="post-metadata">

**Author:** ![GiantJack](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/giantjack/32/1122_2.png) [@GiantJack](https://forum.storj.io/u/GiantJack)\
**Post date:** [March 28, 2024, 2:20am UTC](https://forum.storj.io/t/disk-usage-discrepancy/24715/493 "2024-03-28T02:20:54Z")

</div>

too bad i have mostly disable logs but will try to active them again

---

<div class="post-metadata">

**Author:** ![GiantJack](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/giantjack/32/1122_2.png) [@GiantJack](https://forum.storj.io/u/GiantJack)\
**Post date:** [March 28, 2024, 2:30am UTC](https://forum.storj.io/t/disk-usage-discrepancy/24715/494 "2024-03-28T02:30:17Z")

</div>

found something but looks like no error:

```auto

2024-03-28T02:28:33Z INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {"process": "storagenode", "satelliteID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB"}
2024-03-28T02:28:33Z INFO lazyfilewalker.used-space-filewalker starting subprocess {"process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-03-28T02:28:33Z INFO lazyfilewalker.used-space-filewalker subprocess started {"process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-03-28T02:28:35Z INFO lazyfilewalker.used-space-filewalker.subprocess Database started {"process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "process": "storagenode"}
2024-03-28T02:28:35Z INFO lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started {"process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "process": "storagenode"}

```

i think maybe i enable lazy filewalker long time ago because RPI 2 is old and very slow CPU

---

<div class="post-metadata">

**Author:** ![GiantJack](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/giantjack/32/1122_2.png) [@GiantJack](https://forum.storj.io/u/GiantJack)\
**Post date:** [March 28, 2024, 2:48am UTC](https://forum.storj.io/t/disk-usage-discrepancy/24715/495 "2024-03-28T02:48:34Z")

</div>

removed container and made a new but still lazy filewalker in logs:

```auto
root@raspberrypi:/store/storagenode# docker logs -f node2 --tail 100 | grep file
2024-03-28T02:44:22Z INFO lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker completed {"process": "storagenode", "satelliteID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "process": "storagenode", "piecesTotal": 12579867904, "piecesContentSize": 12573946112}
2024-03-28T02:44:22Z INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {"process": "storagenode", "satelliteID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB"}
2024-03-28T02:44:22Z INFO lazyfilewalker.used-space-filewalker starting subprocess {"process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-03-28T02:44:22Z INFO lazyfilewalker.used-space-filewalker subprocess started {"process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-03-28T02:44:24Z INFO lazyfilewalker.used-space-filewalker.subprocess Database started {"process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "process": "storagenode"}
2024-03-28T02:44:24Z INFO lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started {"process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "process": "storagenode"}

```

just checked grafana and this looks like about the start of the problem, like 6 days ago on 22.3

 ![image](https://storj-s3.bcdn.literatehosting.net/original/2X/9/9da34ad4146fa94a5902afd31fbdf6278b9e8435.png)

edit: looks like fileworker is doing some IO hope it fixes itself:

 ![image](https://storj-s3.bcdn.literatehosting.net/original/2X/2/2c1709f93eee80770a324c6ab5a72fd9f990b0c8.png)

edit² just saw version changed to 1.95.1  
strange it downgraded itself ?

---

<div class="post-metadata">

**Author:** ![nerdatwork](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/nerdatwork/32/1135_2.png) [@nerdatwork](https://forum.storj.io/u/nerdatwork)\
**Post date:** [March 28, 2024, 3:10am UTC](https://forum.storj.io/t/disk-usage-discrepancy/24715/496 "2024-03-28T03:10:51Z")

</div>

> [@GiantJack](#):
>
> strange it downgraded itself ?

There was an issue with 1.97 so the minimum allowed version was upgraded to 1.95.1.

Keep an eye on filewalkers and see if they complete properly without an error.

---

<div class="post-metadata">

**Author:** ![GiantJack](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/giantjack/32/1122_2.png) [@GiantJack](https://forum.storj.io/u/GiantJack)\
**Post date:** [March 28, 2024, 3:13am UTC](https://forum.storj.io/t/disk-usage-discrepancy/24715/497 "2024-03-28T03:13:42Z")

</div>

will do, there is for sure something going on  
/dev/sda1 9.1T 9.0T 32G 100% /store  
it decrease from 33G to 32G free space 😕  
have to go sleep soon maybe ill restart the node with storage set to 6TB that it will not try to fill the HDD more than can fit

---

<div class="post-metadata">

**Author:** ![nerdatwork](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/nerdatwork/32/1135_2.png) [@nerdatwork](https://forum.storj.io/u/nerdatwork)\
**Post date:** [March 28, 2024, 3:30am UTC](https://forum.storj.io/t/disk-usage-discrepancy/24715/498 "2024-03-28T03:30:10Z")

</div>

> [@GiantJack](#):
>
> storage set to 6TB

You should definitely do that.

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [March 28, 2024, 4:21am UTC](https://forum.storj.io/t/disk-usage-discrepancy/24715/499 "2024-03-28T04:21:23Z")

</div>

You likely have a “context canceled” errors with filewalkers or “FATAL” errors and restarts. These read timeouts from the disk may also affect your audit score, so it’s better to figure out why your disk is so slow to respond.  
It could be a bad USB connection/cable, not enough power, or you do not have an external power supply for your disk or it’s not enough, or the USB controller in the HDD enclosure is simple bad or malfunctioning or overheated.

[Previous page](https://forum.storj.io/t/disk-usage-discrepancy/24715.md?page=22)

[Next page](https://forum.storj.io/t/disk-usage-discrepancy/24715.md?page=24)
