# Script to run used space filewalker limit to one at a time

**URL:** <https://forum.storj.io/t/script-to-run-used-space-filewalker-limit-to-one-at-a-time/26922>\
**Category:** STORJLINGS\
**Tags:** storagenode, filewalker, multiple-nodes\
**Created:** [July 3, 2024, 1:38pm UTC](https://forum.storj.io/t/script-to-run-used-space-filewalker-limit-to-one-at-a-time/26922 "2024-07-03T13:38:01Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![littleskunk](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/littleskunk/32/63_2.png) [@littleskunk](https://forum.storj.io/u/littleskunk)\
**Post date:** [July 3, 2024, 1:38pm UTC](https://forum.storj.io/t/script-to-run-used-space-filewalker-limit-to-one-at-a-time/26922/1 "2024-07-03T13:38:01Z")

</div>

I am running 8 nodes on 8 HDDs connected to a Pi5. As you can imagin this is kind of borderline but it works surprisingly good. (still testing and I am going to share it in a different thread in a few days). One of the challenge is to stretch out maintenance jobs to make sure ongoing uploads are not impacted. CPU time is rare so lets use it most efficient.

Here is my script to run filewalker on all 8 nodes but only 1 at a time. You might want to copy and adopt that for your needs. It does require access to the debug endpoint of the storage node.

```auto
for i in {8..1}
do
        echo $i
        sed -i 's\storage2.piece-scan-on-startup: false\storage2.piece-scan-on-startup: true\g' /mnt/sn$i/storagenode$i/storagenode/config.yaml
        sudo systemctl restart storagenode$i
        sleep 30
        sed -i 's\storage2.piece-scan-on-startup: true\storage2.piece-scan-on-startup: false\g' /mnt/sn$i/storagenode$i/storagenode/config.yaml
		
        while :
        do
                if (( `curl -s 127.0.0.1:$((16000+$i))/mon/ps | grep SpaceUsedTotalAndBySatellite | wc -c` <= 0 ))
                then
                        break
                else
                        sleep 30
                fi
        done
done

```

Now there is a chance that one of the storage nodes failed to run it for what ever reason. Maybe a restart or so. The following 2 code snipets will print out more details.

```auto
for i in {8..1}
do
	echo $i
	curl -s 127.0.0.1:$((16000+$i))/mon/funcs | grep -A 2 SpaceUsedTotalAndBySatellite
done

Example Output:
8
[5437966273498512387] storj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalA ndBySatellite
  parents: 631927965255748052
  current: 0, highwater: 1, success: 1, errors: 0, panics: 0
7
[488868047661742421] storj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAn dBySatellite
  parents: 2052289714535893614
  current: 0, highwater: 1, success: 1, errors: 0, panics: 0

```

```auto
for i in {8..1}
do
	echo $i
	curl -s 127.0.0.1:$((16000+$i))/mon/funcs | grep -A 29 SpaceUsedTotalAndBySatellite
done

Example Output:
8
[5437966273498512387] storj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite
  parents: 631927965255748052
  current: 0, highwater: 1, success: 1, errors: 0, panics: 0
  success times:
    0.00: 22m49.347719168s
    0.10: 22m49.347719168s
    0.25: 22m49.347719168s
    0.50: 22m49.347719168s
    0.75: 22m49.347719168s
    0.90: 22m49.347719168s
    0.95: 22m49.347719168s
    1.00: 22m49.347719168s
    avg: 22m49.347710786s
    ravg: 22m49.347719168s
    recent: 22m49.347710786s
    sum: 22m49.347710786s
  failure times:
    0.00: 0s
    0.10: 0s
    0.25: 0s
    0.50: 0s
    0.75: 0s
    0.90: 0s
    0.95: 0s
    1.00: 0s
    avg: 0s
    ravg: 0s
    recent: 0s
    sum: 0s

```

---

<div class="post-metadata">

**Author:** ![Mitsos](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/mitsos/32/1125_2.png) [@Mitsos](https://forum.storj.io/u/Mitsos)\
**Post date:** [July 3, 2024, 3:29pm UTC](https://forum.storj.io/t/script-to-run-used-space-filewalker-limit-to-one-at-a-time/26922/2 "2024-07-03T15:29:27Z")

</div>

So the 1 node per core doesn’t apply anymore?

---

<div class="post-metadata">

**Author:** ![littleskunk](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/littleskunk/32/63_2.png) [@littleskunk](https://forum.storj.io/u/littleskunk)\
**Post date:** [July 3, 2024, 3:47pm UTC](https://forum.storj.io/t/script-to-run-used-space-filewalker-limit-to-one-at-a-time/26922/3 "2024-07-03T15:47:16Z")

</div>

That is unrelated to this topic. The Pi5 has 4 cores. To be safe I would still recommend running just 4 nodes on it and not 8 as I am doing it right now. I doesn’t change much on the script above. I would still run just a single used space filewalker at a time.

---

<div class="post-metadata">

**Author:** ![Mitsos](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/mitsos/32/1125_2.png) [@Mitsos](https://forum.storj.io/u/Mitsos)\
**Post date:** [July 3, 2024, 3:48pm UTC](https://forum.storj.io/t/script-to-run-used-space-filewalker-limit-to-one-at-a-time/26922/4 "2024-07-03T15:48:30Z")

</div>

Should we start a new topic on who can “bend” the supplier terms then? I would love to know more about this topic.

---

<div class="post-metadata">

**Author:** ![littleskunk](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/littleskunk/32/63_2.png) [@littleskunk](https://forum.storj.io/u/littleskunk)\
**Post date:** [July 3, 2024, 4:02pm UTC](https://forum.storj.io/t/script-to-run-used-space-filewalker-limit-to-one-at-a-time/26922/5 "2024-07-03T16:02:22Z")

</div>

You might have a wrong impression of why I am running this Pi5 experiment. I am not recommending 8 nodes on a Pi5. I am sharing my script with you and it will work even with just 2 nodes.

---

<div class="post-metadata">

**Author:** ![Craig](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/craig/32/1122_2.png) [@Craig](https://forum.storj.io/u/Craig)\
**Post date:** [July 3, 2024, 4:15pm UTC](https://forum.storj.io/t/script-to-run-used-space-filewalker-limit-to-one-at-a-time/26922/6 "2024-07-03T16:15:07Z")

</div>

It looks like this script assumes use of the native Linux client and has been configured for management with systemctl. Would similar be able to be done for docker installs replacing the `systemctl` entry with `docker restart` of the container? Unsure if the docker container will re-read the yaml on restart of if it would take a full stop/run to do that.

I guess the `for i in` would also need to be swapped with some logic that would get the container list from `docker ps -a -q` as well…

Thanks for sharing your script! I’ll be watching for your RPi thread as I’ll want to jump in with some of my recent observations of my RPi node.

EDIT: hmm… the `curl` line throws a twist in there for docker as well. I am interested in figuring this out for docker nodes and you’ve given me a great start to investigate!

MORE EDIT: I also realized I am not yet making the debug available on my nodes. For those like me that need this info, here’s littleskunk’s previous post on enabling debug, at least for docker nodes:

> [@Guide to debug my storage node, uplink, s3 gateway, satellite](https://forum.storj.io/t/guide-to-debug-my-storage-node-uplink-s3-gateway-satellite/1372):
>
> We have integrated a package called monkit in all storage nodes. It is very usefull for debuging. By default it will open a random port. In order to connect to the debug port you will need to: Add debug.addr: ":5999" to the config file. Now the port is fixed and will not change on every restart. Add -p 127.0.0.1:5999:5999 to the docker run command. Ready for action! You can request the following informations: curl localhost:5999/mon/funcs Find out how long a function like GetExpired need…

---

<div class="post-metadata">

**Author:** ![littleskunk](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/littleskunk/32/63_2.png) [@littleskunk](https://forum.storj.io/u/littleskunk)\
**Post date:** [July 3, 2024, 5:04pm UTC](https://forum.storj.io/t/script-to-run-used-space-filewalker-limit-to-one-at-a-time/26922/7 "2024-07-03T17:04:23Z")

</div>

You already nailed it. Yes this should work with docker as well. Instead of obtaining the list of storagenodes I just named them storagenode1 - storagnode8. It should work the same way with docker. Just replace the number in the name with $i

---

<div class="post-metadata">

**Author:** ![Craig](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/craig/32/1122_2.png) [@Craig](https://forum.storj.io/u/Craig)\
**Post date:** [July 3, 2024, 5:51pm UTC](https://forum.storj.io/t/script-to-run-used-space-filewalker-limit-to-one-at-a-time/26922/8 "2024-07-03T17:51:36Z")

</div>

Sigh… and today I learned that you can manage docker containers by the name. This entire time I’ve been managing them by the container ID value. I swear at one point I tried to do something via the docker container name and it wasn’t allowed, forcing me to instead use the container ID. Well this is going to make my life a lot easier including tweaking this script for docker systems.

---

<div class="post-metadata">

**Author:** ![Solu](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/solu/32/62_2.png) [@Solu](https://forum.storj.io/u/Solu)\
**Post date:** [July 3, 2024, 6:02pm UTC](https://forum.storj.io/t/script-to-run-used-space-filewalker-limit-to-one-at-a-time/26922/9 "2024-07-03T18:02:44Z")

</div>

Awesome thank you, now i will think again about my 48 node cluster 🤔

---

<div class="post-metadata">

**Author:** ![EasyRhino](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/easyrhino/32/1133_2.png) [@EasyRhino](https://forum.storj.io/u/EasyRhino)\
**Post date:** [July 3, 2024, 8:24pm UTC](https://forum.storj.io/t/script-to-run-used-space-filewalker-limit-to-one-at-a-time/26922/10 "2024-07-03T20:24:56Z")

</div>

seems like this might also be a good idea if more than one node is sharing a disk or array.

(of course you’re not supposed to be running multiple nodes on one disk)

---

<div class="post-metadata">

**Author:** ![Toyoo](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/toyoo/32/1126_2.png) [@Toyoo](https://forum.storj.io/u/Toyoo)\
**Post date:** [July 3, 2024, 8:38pm UTC](https://forum.storj.io/t/script-to-run-used-space-filewalker-limit-to-one-at-a-time/26922/11 "2024-07-03T20:38:46Z")

</div>

> [@littleskunk](#):
>
> Here is my script to run filewalker on all 8 nodes but only 1 at a time. You might want to copy and adopt that for your needs. It does require access to the debug endpoint of the storage node.  
> […]

Seems like a lot of work. I’ve just wrapped the file walker with a shared [`flock(2)`](https://www.man7.org/linux/man-pages/man2/flock.2.html).

---

<div class="post-metadata">

**Author:** ![striker43](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/striker43/32/1135_2.png) [@striker43](https://forum.storj.io/u/striker43)\
**Post date:** [July 4, 2024, 3:59am UTC](https://forum.storj.io/t/script-to-run-used-space-filewalker-limit-to-one-at-a-time/26922/12 "2024-07-04T03:59:35Z")

</div>

> [@Solu](#):
>
> Awesome thank you, now i will think again about my 48 node cluster 🤔

On a raspberry pi? 😛

---

<div class="post-metadata">

**Author:** ![Solu](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/solu/32/62_2.png) [@Solu](https://forum.storj.io/u/Solu)\
**Post date:** [July 4, 2024, 5:12am UTC](https://forum.storj.io/t/script-to-run-used-space-filewalker-limit-to-one-at-a-time/26922/13 "2024-07-04T05:12:12Z")

</div>

Well, i trust the pi to handle a lot but 48… could become quite a challenge 😝

So for this time i’ll go with a HP server 😉

---

<div class="post-metadata">

**Author:** ![Craig](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/craig/32/1122_2.png) [@Craig](https://forum.storj.io/u/Craig)\
**Post date:** [July 5, 2024, 2:39pm UTC](https://forum.storj.io/t/script-to-run-used-space-filewalker-limit-to-one-at-a-time/26922/14 "2024-07-05T14:39:32Z")

</div>

> [@Toyoo](#):
>
> Seems like a lot of work. I’ve just wrapped the file walker with a shared [`flock(2)`](https://www.man7.org/linux/man-pages/man2/flock.2.html).

Would you mind expanding on this? I looked at the man page for flock and it seems this is for a higher level programming than a shell script. Maybe you meant to link to [flock(1)](https://www.man7.org/linux/man-pages/man1/flock.1.html)? I’ve never used this in Linux but from knowledge of object locks on other platforms (I’m an IBM i (AS/400) admin/engineer by trade) I can definitely see how the logic could work. Thanks!

---

<div class="post-metadata">

**Author:** ![Toyoo](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/toyoo/32/1126_2.png) [@Toyoo](https://forum.storj.io/u/Toyoo)\
**Post date:** [July 5, 2024, 9:15pm UTC](https://forum.storj.io/t/script-to-run-used-space-filewalker-limit-to-one-at-a-time/26922/15 "2024-07-05T21:15:18Z")

</div>

> [@Craig](#):
>
> Maybe you meant to link to [flock(1)](https://www.man7.org/linux/man-pages/man1/flock.1.html)?

No, I’ve literally wrapped the filewalker itself with flock(2) by patching the source code.

---

<div class="post-metadata">

**Author:** ![flo82](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/flo82/32/10747_2.png) [@flo82](https://forum.storj.io/u/flo82)\
**Post date:** [July 6, 2024, 7:20am UTC](https://forum.storj.io/t/script-to-run-used-space-filewalker-limit-to-one-at-a-time/26922/16 "2024-07-06T07:20:44Z")

</div>

That is against the terms:

You will not modify or attempt to modify the Storage Node Software for any purpose

[https://www.storj.io/legal/supplier-terms-conditions](https://www.storj.io/legal/supplier-terms-conditions)

---

<div class="post-metadata">

**Author:** ![pasatmalo](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/pasatmalo/32/1127_2.png) [@pasatmalo](https://forum.storj.io/u/pasatmalo)\
**Post date:** [July 6, 2024, 7:56am UTC](https://forum.storj.io/t/script-to-run-used-space-filewalker-limit-to-one-at-a-time/26922/17 "2024-07-06T07:56:57Z")

</div>

> [@flo82](#):
>
> That is against the terms:
> 
> You will not modify or attempt to modify the Storage Node Software for any purpose

This script is not modifying the software in any way, all that its doing is coordinating the enabling/disabling of the piece scan on startup flag in the configuration file. Changing values of a configuration file is not modification of the software. Modifying the software would be changing the actual code to alter specific behaviours.

---

<div class="post-metadata">

**Author:** ![flo82](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/flo82/32/10747_2.png) [@flo82](https://forum.storj.io/u/flo82)\
**Post date:** [July 6, 2024, 8:44am UTC](https://forum.storj.io/t/script-to-run-used-space-filewalker-limit-to-one-at-a-time/26922/18 "2024-07-06T08:44:13Z")

</div>

The script is fine.

With “sourcecode” I thought he was referring to the storj sourcecode. If it is not the storj sourcecode it will be fine and i’m sorry.

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [July 6, 2024, 11:04am UTC](https://forum.storj.io/t/script-to-run-used-space-filewalker-limit-to-one-at-a-time/26922/19 "2024-07-06T11:04:59Z")

</div>

However, I would prefer to have a PR for the team’s review. It sounds like a useful addition…  
@Toyoo FYI

---

<div class="post-metadata">

**Author:** ![Toyoo](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/toyoo/32/1126_2.png) [@Toyoo](https://forum.storj.io/u/Toyoo)\
**Post date:** [July 6, 2024, 2:02pm UTC](https://forum.storj.io/t/script-to-run-used-space-filewalker-limit-to-one-at-a-time/26922/20 "2024-07-06T14:02:43Z")

</div>

> [@Alexey](#):
>
> However, I would prefer to have a PR for the team’s review. It sounds like a useful addition…

Sorry, I don’t have time to create a PR out of it. Besides, I have only implemented a Linux versionof it, I don’t know how to do this on Windows. However, it’s simple enough that I’m pretty sure any Storj developer could reimplement it pretty quickly.

> [@flo82](#):
>
> You will not modify or attempt to modify the Storage Node Software for any purpose

❤