# Is there a safe mechanism regardless of what local dashboard says?

**URL:** <https://forum.storj.io/t/is-there-a-safe-mechanism-regardless-of-what-local-dashboard-says/26627>\
**Category:** FAQ\
**Tags:** storage, web-dashboard, used-space\
**Created:** [June 13, 2024, 9:35pm UTC](https://forum.storj.io/t/is-there-a-safe-mechanism-regardless-of-what-local-dashboard-says/26627 "2024-06-13T21:35:21Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ruskiem](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/ruskiem/32/64_2.png) [@Ruskiem](https://forum.storj.io/u/Ruskiem)\
**Post date:** [June 13, 2024, 9:35pm UTC](https://forum.storj.io/t/is-there-a-safe-mechanism-regardless-of-what-local-dashboard-says/26627/1 "2024-06-13T21:35:21Z")

</div>

If piece-scan-on-startup is at false, a node operates and even have a good success score,  
but dashboard thinks it has 5TB free, but in fact it has 900GB,

is there a safe mechanism regardless of what local dashboard says?  
Will the node stop filling before the real space ends?  
Or data will “fall off the cliff” and node gets disqualificated?

(if You want, You can make it public, in right thread, i don’t know where to post it.)

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [June 14, 2024, 3:48am UTC](https://forum.storj.io/t/is-there-a-safe-mechanism-regardless-of-what-local-dashboard-says/26627/2 "2024-06-14T03:48:01Z")

</div>

Made this topic public on June 14, 2024, 3:48am UTC



---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [June 14, 2024, 3:51am UTC](https://forum.storj.io/t/is-there-a-safe-mechanism-regardless-of-what-local-dashboard-says/26627/3 "2024-06-14T03:51:46Z")

</div>

The node should report as full as soon as there not more than 5GB left (earlier this threshold was 500MB).

> [@Upcoming storage node improvements including benchmark tool](https://forum.storj.io/t/upcoming-storage-node-improvements-including-benchmark-tool/26096/22):
>
> We also need to increase the free space buffer from 500MB to 5GB. Because of the TTL nodes will switch between full and free space more frequently. Combined with an higher upload rate a 500MB buffer would give the customer a bad experience.

---

<div class="post-metadata">

**Author:** ![Mitsos](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/mitsos/32/1125_2.png) [@Mitsos](https://forum.storj.io/u/Mitsos)\
**Post date:** [June 14, 2024, 3:59am UTC](https://forum.storj.io/t/is-there-a-safe-mechanism-regardless-of-what-local-dashboard-says/26627/4 "2024-06-14T03:59:52Z")

</div>

I haven’t seen that exact discrepancy on my nodes, when things fall out of sync (data on dashboard != actual data on disk) it’s always the opposite: the node thinks it has less free space than what is actually on the disk. In this case the node behaves as @Alexey said, traffic stops when the node approaches ~5GB (seen it as low as 3.5GB) and the disk is left with free space that can’t get filled.

A run of used-space fixes it ofc.

---

<div class="post-metadata">

**Author:** ![jammerdan](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/jammerdan/32/1121_2.png) [@jammerdan](https://forum.storj.io/u/jammerdan)\
**Post date:** [June 14, 2024, 4:19am UTC](https://forum.storj.io/t/is-there-a-safe-mechanism-regardless-of-what-local-dashboard-says/26627/5 "2024-06-14T04:19:38Z")

</div>

> [@Mitsos](#):
>
> A run of used-space fixes it ofc.

If it runs and finishes successfully…

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [June 14, 2024, 7:37am UTC](https://forum.storj.io/t/is-there-a-safe-mechanism-regardless-of-what-local-dashboard-says/26627/6 "2024-06-14T07:37:06Z")

</div>

> [@Mitsos](#):
>
> when the node approaches ~5TB (seen it as low as 3.5TB)

hm, I meant something different: it will reporting to the satellites that’s full, when it has less or equal than 5GB in the allocation/disk.

---

<div class="post-metadata">

**Author:** ![Mitsos](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/mitsos/32/1125_2.png) [@Mitsos](https://forum.storj.io/u/Mitsos)\
**Post date:** [June 14, 2024, 7:40am UTC](https://forum.storj.io/t/is-there-a-safe-mechanism-regardless-of-what-local-dashboard-says/26627/7 "2024-06-14T07:40:33Z")

</div>

My bad on the wording. Yes that’s what I meant, when it’s at 5GB free it reports it to the satellite. Sometimes this overshoots (understandably) and it goes as low as 3.5GB.

Will edit the values 😅

---

<div class="post-metadata">

**Author:** ![Ruskiem](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/ruskiem/32/64_2.png) [@Ruskiem](https://forum.storj.io/u/Ruskiem)\
**Post date:** [June 14, 2024, 12:32pm UTC](https://forum.storj.io/t/is-there-a-safe-mechanism-regardless-of-what-local-dashboard-says/26627/8 "2024-06-14T12:32:34Z")

</div>

oh? i see. i hope it really keep track of real disk space left, and do reports, im going to see for real if thats true i guess

---

<div class="post-metadata">

**Author:** ![redgyuf](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/redgyuf/32/1137_2.png) [@redgyuf](https://forum.storj.io/u/redgyuf)\
**Post date:** [June 16, 2024, 10:11pm UTC](https://forum.storj.io/t/is-there-a-safe-mechanism-regardless-of-what-local-dashboard-says/26627/9 "2024-06-16T22:11:02Z")

</div>

Is this 5GB configurable? As it seems much more reliable than the 5-6days long filewalker in somecases

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [June 17, 2024, 4:22am UTC](https://forum.storj.io/t/is-there-a-safe-mechanism-regardless-of-what-local-dashboard-says/26627/10 "2024-06-17T04:22:04Z")

</div>

No, it’s hardcoded.  
And I didn’t get how is it related to the filewalker?

---

<div class="post-metadata">

**Author:** ![redgyuf](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/redgyuf/32/1137_2.png) [@redgyuf](https://forum.storj.io/u/redgyuf)\
**Post date:** [June 17, 2024, 6:57am UTC](https://forum.storj.io/t/is-there-a-safe-mechanism-regardless-of-what-local-dashboard-says/26627/11 "2024-06-17T06:57:13Z")

</div>

for me it took now about 6 days for the lazy walker to finish, and before this i restarted the node for config change, with the restart I lost more than a TB of used space on the dashboard, so even if i set the max space usable in the config to 7.6TB, my usage from 6.8TB dropped to like 5.7TB.

And with this much difference in the real usage I hit the real freespace limit like 2-3 days before the node finished its filewalking.

So the 5GB hardcoded limit worked flawlessly, while the filewalker is unreliable when we get like TBs of ingress/day, with the usual 30-60GB, basicly nobody cares and because there is also a low load caused by the ingress its finishes faster.

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [June 17, 2024, 7:03am UTC](https://forum.storj.io/t/is-there-a-safe-mechanism-regardless-of-what-local-dashboard-says/26627/12 "2024-06-17T07:03:32Z")

</div>

The filewalker may not succeed if you also have a databases related errors in your logs.

---

<div class="post-metadata">

**Author:** ![redgyuf](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/redgyuf/32/1137_2.png) [@redgyuf](https://forum.storj.io/u/redgyuf)\
**Post date:** [June 17, 2024, 7:50am UTC](https://forum.storj.io/t/is-there-a-safe-mechanism-regardless-of-what-local-dashboard-says/26627/13 "2024-06-17T07:50:08Z")

</div>

maybe, but still it is unreliable/slow under heavy load, and the phisical free space checks seems to be better as a failsafe, especially if the disk is dedicated to storj and the DB is moved to like an SSD in my opinion.

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [June 17, 2024, 7:54am UTC](https://forum.storj.io/t/is-there-a-safe-mechanism-regardless-of-what-local-dashboard-says/26627/14 "2024-06-17T07:54:21Z")

</div>

You may disable the lazy mode, then it will finish faster (however, it would likely affect the node’s success rate).

---

<div class="post-metadata">

**Author:** ![jammerdan](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/jammerdan/32/1121_2.png) [@jammerdan](https://forum.storj.io/u/jammerdan)\
**Post date:** [June 17, 2024, 8:39am UTC](https://forum.storj.io/t/is-there-a-safe-mechanism-regardless-of-what-local-dashboard-says/26627/15 "2024-06-17T08:39:42Z")

</div>

> [@redgyuf](#):
>
> maybe, but still it is unreliable/slow under heavy load

Storj has become aware of this too luckily: [Storage node performance for filewalker is still extremely slow on large nodes · Issue #6998 · storj/storj · GitHub](https://github.com/storj/storj/issues/6998)  
So there is hope they are working something out.

In my opinion we need to get rid of filewalkers and databases as much as possible.

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [June 18, 2024, 7:17am UTC](https://forum.storj.io/t/is-there-a-safe-mechanism-regardless-of-what-local-dashboard-says/26627/16 "2024-06-18T07:17:22Z")

</div>

Yes, but nobody suggested a better solution for that so far.  
Databases are used as a cache (to do not scan constantly), the filewalkers should collect pieces, the garbage, empty the trash and remove TTL based pieces.  
There is also `used-space-filewalker` to keep databases updated if there was issues with the databases. But it also requires to be able to update databases. The loop is closed…

We are investigating a usage of a badger in hope, that it can solve performance issues with the plain storage… But nobody wants to avoid databases (because they are speedup things a lot)…

---

<div class="post-metadata">

**Author:** ![jammerdan](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/jammerdan/32/1121_2.png) [@jammerdan](https://forum.storj.io/u/jammerdan)\
**Post date:** [June 20, 2024, 5:07am UTC](https://forum.storj.io/t/is-there-a-safe-mechanism-regardless-of-what-local-dashboard-says/26627/17 "2024-06-20T05:07:02Z")

</div>

> [@Alexey](#):
>
> Yes, but nobody suggested a better solution for that so far.

I am not a developer so I cannot make suggestions how to this can be achieved. But the flaws and conceptual problems of the current implementation are getting obvious more and more every day.

We have seen at least ideas that try to reduce or optimize the pressure on the storage.

One of the best solutions seems to not use the databases at all. It was suggested many times to make them optional or at least as many of them as possible as an unknown number of SNOs have no use for the data they hold.

I believe that is a suggestion.

Other suggestion is to move as much of it into RAM. Running the databases not on the slowest storage but on the fastest available, seems logical to me.

Other suggestion is to separate actual data from history to keep the working databases and data loss in case of corruption small.

And if Storj is running out of ideas, they should talk to the developers of Sqlite:

> [@Suggestion to have Sqlite optimized or tailored](https://forum.storj.io/t/suggestion-to-have-sqlite-optimized-or-tailored/26730):
>
> With all those issues with the databases and Sqlite and [enhancement suggestions made](https://github.com/storj/storj/issues/7010), has Storj ever contacted the developers of this for support or modifications? [SQLite Support Options](https://www.sqlite.org/support.html) Paid Professional Support If you would like [professional support](https://www.sqlite.org/prosupport.html) for SQLite or if you want custom modifications performed by the original author of SQLite, these services are available for a modest fee. For additional information contact: D. Richard Hipp Hwaci - Applied Software Research 704.948.4565 [drh…](mailto:drh@hwaci.com)

Maybe they have additional ideas or can implement something that helps the Storj use case.

Better solutions for the filewalkers have been made to not to run them on every restart and make it configurable when to run them. Also saving state has been suggested and of course helps when it does not restart from the scratch every time.  
Some of this is already getting implemented but for some reason not rolled out as fast as it should be.

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [June 20, 2024, 7:11am UTC](https://forum.storj.io/t/is-there-a-safe-mechanism-regardless-of-what-local-dashboard-says/26627/18 "2024-06-20T07:11:31Z")

</div>

> [@jammerdan](#):
>
> Other suggestion is to move as much of it into RAM.

But I believe that you run your node on a RAM constrained hardware?

> [@jammerdan](#):
>
> Other suggestion is to separate actual data from history to keep the working databases and data loss in case of corruption small.

this is actually a great one. Do you have a post for that to share with the team?

> [@jammerdan](#):
>
> Better solutions for the filewalkers have been made to not to run them on every restart and make it configurable when to run them.

I think it’s in work already. But not like as described… more like a resume of the interrupted process. It’s not merged so far.

---

<div class="post-metadata">

**Author:** ![jammerdan](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/jammerdan/32/1121_2.png) [@jammerdan](https://forum.storj.io/u/jammerdan)\
**Post date:** [June 20, 2024, 7:51am UTC](https://forum.storj.io/t/is-there-a-safe-mechanism-regardless-of-what-local-dashboard-says/26627/19 "2024-06-20T07:51:47Z")

</div>

> [@Alexey](#):
>
> But I believe that you run your node on a RAM constrained hardware?

I am running more than one node. Some are on constrained hardware like the Odroid HC2 others are not.  
It would be a blessing to run the databases fully in RAM with only occasionally backups to disk like every 10 minutes.  
Together with the separation of history data, I would lose only the last minutes in case of power loss or corruption.

> [@Alexey](#):
>
> this is actually a great one. Do you have a post for that to share with the team?

No sorry, this is an idea I came up with and I had voiced it in the past her one the forum already. How such could implemented, I don’t know. But of course it would make sense to move old data out of the way to some kind of “archive” database because access to it is not frequently needed and data will not be changed.

> [@Alexey](#):
>
> It’s not merged so far.

Unfortunately not, right…

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [June 20, 2024, 8:43am UTC](https://forum.storj.io/t/is-there-a-safe-mechanism-regardless-of-what-local-dashboard-says/26627/20 "2024-06-20T08:43:38Z")

</div>

> [@jammerdan](#):
>
> I would lose only the last minutes in case of power loss or corruption.

perhaps more. The interval is 1h by default.

[Next page](https://forum.storj.io/t/is-there-a-safe-mechanism-regardless-of-what-local-dashboard-says/26627.md?page=2)
