# Updates on Test Data **URL:** \ **Category:** Announcements\ **Created:** [April 30, 2024, 3:36pm UTC](https://forum.storj.io/t/updates-on-test-data/26034 "2024-04-30T15:36:45Z")\ **Posts on this page:** 20\ **Page:** 76
**Author:** ![jammerdan](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/jammerdan/32/1121_2.png) [@jammerdan](https://forum.storj.io/u/jammerdan)\ **Post date:** [July 1, 2024, 3:37pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/1689 "2024-07-01T15:37:53Z")
> [@littleskunk](#): > > As a workaround I will just tell my nodes that they have a 50TB drive each. > [@littleskunk](#): > > I think we filled like 3000 nodes in the past week. I doubt that all operators of those 3000 nodes will read this workaround. ---
**Author:** ![littleskunk](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/littleskunk/32/63_2.png) [@littleskunk](https://forum.storj.io/u/littleskunk)\ **Post date:** [July 1, 2024, 4:44pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/1690 "2024-07-01T16:44:35Z")
> [@striker43](#): > > So if we all do this, this will mess up your capacity planning a lot I guess 😃 No it doesn’t. The node will report the amount of free space on disk. If you set allocated space to some extreme high value you would see this in the logs: ```auto 2024-07-01T18:43:18+02:00 WARN piecestore:monitor Disk space is less than requested. Allocated space is {"Process": "storagenode", "bytes": 16782805938716} ``` ---
**Author:** ![snorkel](https://storj.bcdn.literatehosting.com/letter_avatar/snorkel/32/5_5575768a8748004e209b776fc1b2916d.png) [@snorkel](https://forum.storj.io/u/snorkel)\ **Post date:** [July 1, 2024, 4:54pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/1691 "2024-07-01T16:54:22Z")
So how sure are you we won’t destroy our nodes? I thought about this workaround too, but I didn’t risk it. What if there is again some bug that will ignore the 5GB limit? ---
**Author:** ![zip](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/zip/32/7813_2.png) [@zip](https://forum.storj.io/u/zip)\ **Post date:** [July 1, 2024, 4:56pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/1692 "2024-07-01T16:56:13Z")
How reliable is this? Because I had nodes crashing just recently with drive out of space and I had to delete the trash manually to free up the space be able to restart the node. Databases were on SSD, space was set correctly, it however was not accounted correctly I assume because of some bug. These are running on LVM and in some cases not all of the drive is allocated for the node - might this be the reason? ---
**Author:** ![littleskunk](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/littleskunk/32/63_2.png) [@littleskunk](https://forum.storj.io/u/littleskunk)\ **Post date:** [July 1, 2024, 5:04pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/1693 "2024-07-01T17:04:47Z")
> [@snorkel](#): > > So how sure are you we won’t destroy our nodes? I was talking about my nodes. They have plenty of free space available and are not getting full in the next few weeks. If you want some extra protection you can use a code snipet like this: ```auto diskspace=`curl -s 127.0.0.1:1500$i/api/sno/ | jq .diskSpace` used=`echo $diskspace | jq .used` free=[run some df command to find out free space here] trash=`echo $diskspace | jq .trash` sed -i "s/storage\.allocated-disk-space: .*/storage.allocated-disk-space: $(($used+$trash+free)) B/g" [path to your storagenode config] ``` Maybe substract like 100GB from the free space and than you have your safety margin. ---
**Author:** ![littleskunk](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/littleskunk/32/63_2.png) [@littleskunk](https://forum.storj.io/u/littleskunk)\ **Post date:** [July 1, 2024, 5:08pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/1694 "2024-07-01T17:08:31Z")
> [@zip](#): > > Because I had nodes crashing just recently with drive out of space I don’t understand. The workaround I was talking about was for nodes that believe they are full when they still have a few TB free space on disk. If your node is crashing because it is full than you might want to reduce your allocation. Why would you want to do the oposit? That would make your situation worse right? ---
**Author:** ![zip](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/zip/32/7813_2.png) [@zip](https://forum.storj.io/u/zip)\ **Post date:** [July 1, 2024, 5:18pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/1695 "2024-07-01T17:18:58Z")
> [@littleskunk](#): > > There should be a free space check that should still limit the size to the actual HDD size but it would ignore the size the node believes it has and uses that free space on disk instead. If there is a free space check to determine the actual free space on the drive then the nodes shouldn’t be crashing because of drive being full. My understanding is this was changed to 5GB recently, so the node should stop accepting uploads as the actual drive free space approaches 5GB of space left. And I’m saying I had nodes that had less space allocated than what was the actual drive size and yet they crashed because the drive ran out of space. ---
**Author:** ![littleskunk](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/littleskunk/32/63_2.png) [@littleskunk](https://forum.storj.io/u/littleskunk)\ **Post date:** [July 1, 2024, 5:24pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/1696 "2024-07-01T17:24:38Z")
> [@zip](#): > > If there is a free space check to determine the actual free space on the drive then the nodes shouldn’t be crashing because of drive being full. Your filesystem might return incorrect data to the storagenode. In that case it would continue accepting more pieces than it should have. > [@zip](#): > > And I’m saying I had nodes that had less space allocated than what was the actual drive size and yet they crashed because the drive ran out of space. Don’t write other data on the same drive. This sounds more like a topic that was discussed in a few other threads already. Just follow the advice you can read up there. @Alexey might even have some links for you. ---
**Author:** ![Vadim](https://storj.bcdn.literatehosting.com/letter_avatar/vadim/32/5_5575768a8748004e209b776fc1b2916d.png) [@Vadim](https://forum.storj.io/u/Vadim)\ **Post date:** [July 1, 2024, 5:24pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/1697 "2024-07-01T17:24:44Z")
Did you had database on same HDD or only data? ---
**Author:** ![zip](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/zip/32/7813_2.png) [@zip](https://forum.storj.io/u/zip)\ **Post date:** [July 1, 2024, 5:32pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/1698 "2024-07-01T17:32:28Z")
> [@littleskunk](#): > > Don’t write other data on the same drive. This sounds more like a topic that was discussed in a few other threads already. Just follow the advice you can read up there. @Alexey might even have some links for you. > [@Vadim](#): > > Did you had database on same HDD or only data? Databases are on separate SSDs, LVM caches are on separate SSDs and the drive was being used only for single storagenode and for nothing else, using ext4 with default settings except of -m 0. As a rule of thumb I leave around 5% of the drive (more specifically LV) as unused. So the conclusion would be this free space check isn’t reliable, as hasn’t been the trash accounting, the BFs and so on. Or maybe it is just me and in that case please accept my apology. ---
**Author:** ![snorkel](https://storj.bcdn.literatehosting.com/letter_avatar/snorkel/32/5_5575768a8748004e209b776fc1b2916d.png) [@snorkel](https://forum.storj.io/u/snorkel)\ **Post date:** [July 1, 2024, 5:58pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/1699 "2024-07-01T17:58:46Z")
The autostop for 5GB works with the space tracked by storagenode. All my full nodes show free space 4.46GB. Regarding the space reported by OS, I didn’t got there yet, to test it. I have also ext4 drives, but they have Synology DSM on them, so the drives are used by other software too. Because of this, I could never allocate full drive. Maybe on my Ubuntu machine I could test this, but to risk 22TB of data? 🤔 ---
**Author:** ![arrogantrabbit](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/arrogantrabbit/32/11200_2.png) [@arrogantrabbit](https://forum.storj.io/u/arrogantrabbit)\ **Post date:** [July 1, 2024, 6:24pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/1700 "2024-07-01T18:24:23Z")
> [@snorkel](#): > > but to risk 22TB of data? 🤔 What’s the risk? Data can either be written to the volume or not. ---
**Author:** ![flo82](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/flo82/32/10747_2.png) [@flo82](https://forum.storj.io/u/flo82)\ **Post date:** [July 1, 2024, 6:32pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/1701 "2024-07-01T18:32:22Z")
wait for this [storagenode/blobstore: blobstore with caching file stat information (… · storj/storj@2fceb6c · GitHub](https://github.com/storj/storj/commit/2fceb6c0fd8f19a23abc55808e5e49ed63750d4c) ---
**Author:** ![Mitsos](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/mitsos/32/1125_2.png) [@Mitsos](https://forum.storj.io/u/Mitsos)\ **Post date:** [July 1, 2024, 6:34pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/1702 "2024-07-01T18:34:48Z")
Thanks, but doesn’t help me: ```auto filestat cache is incompatible with lazy file walker. Please use --pieces.enable-lazy-filewalker=false ``` ---
**Author:** ![flo82](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/flo82/32/10747_2.png) [@flo82](https://forum.storj.io/u/flo82)\ **Post date:** [July 1, 2024, 6:37pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/1703 "2024-07-01T18:37:32Z")
If the performance tests are real - then everyone will switch to “non-lazy” filewalker. If the (cache) database sits on a ssd, you’ll save an massive amount of I/Os. ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [July 2, 2024, 4:41am UTC](https://forum.storj.io/t/updates-on-test-data/26034/1705 "2024-07-02T04:41:18Z")
> [@Vadim](#): > > Alexey told that this problem was on 104.X was fixed on 105.x but need to filewalker go over on start to update databases. That was for a trash filewalker. This one is something new - the TTL data removing doesn’t update the usage as was with the trash filewalker. ---
**Author:** ![MarviBiene](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/marvibiene/32/1124_2.png) [@MarviBiene](https://forum.storj.io/u/MarviBiene)\ **Post date:** [July 2, 2024, 4:59am UTC](https://forum.storj.io/t/updates-on-test-data/26034/1706 "2024-07-02T04:59:07Z")
Wait, so there is free space from the deleted ttl data, but it’s not updated to the node itself? So it’s still not accepting data? Will it fix itself then or do we have to do something? Is there a workaround available, if it doesn’t fix itself? ---
**Author:** ![snorkel](https://storj.bcdn.literatehosting.com/letter_avatar/snorkel/32/5_5575768a8748004e209b776fc1b2916d.png) [@snorkel](https://forum.storj.io/u/snorkel)\ **Post date:** [July 2, 2024, 5:46am UTC](https://forum.storj.io/t/updates-on-test-data/26034/1707 "2024-07-02T05:46:14Z")
So even the ver 107 won’t fix this bug? 🤦🏻‍♂️ ---
**Author:** ![nerdatwork](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/nerdatwork/32/1135_2.png) [@nerdatwork](https://forum.storj.io/u/nerdatwork)\ **Post date:** [July 2, 2024, 5:54am UTC](https://forum.storj.io/t/updates-on-test-data/26034/1708 "2024-07-02T05:54:37Z")
> [@MarviBiene](#): > > but it’s not updated to the node itself? In my observation, even operating system ( Windows ) won’t show any new free space after TTL data is being deleted. This makes me wonder if TTL data is really being deleted or not as we only see “count” line in the log not the original `deleted expired piece`. ---
**Author:** ![Vadim](https://storj.bcdn.literatehosting.com/letter_avatar/vadim/32/5_5575768a8748004e209b776fc1b2916d.png) [@Vadim](https://forum.storj.io/u/Vadim)\ **Post date:** [July 2, 2024, 5:59am UTC](https://forum.storj.io/t/updates-on-test-data/26034/1709 "2024-07-02T05:59:55Z")
it not deleting all at once, step by step as was uploaded. [Previous page](https://forum.storj.io/t/updates-on-test-data/26034.md?page=75) [Next page](https://forum.storj.io/t/updates-on-test-data/26034.md?page=77)