# Not calling os.MkdirAll all the time

**URL:** <https://forum.storj.io/t/not-calling-os-mkdirall-all-the-time/27851>\
**Category:** Node Operators\
**Created:** [September 12, 2024, 12:14am UTC](https://forum.storj.io/t/not-calling-os-mkdirall-all-the-time/27851 "2024-09-12T00:14:20Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![Toyoo](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/toyoo/32/1126_2.png) [@Toyoo](https://forum.storj.io/u/Toyoo)\
**Post date:** [September 12, 2024, 12:14am UTC](https://forum.storj.io/t/not-calling-os-mkdirall-all-the-time/27851/1 "2024-09-12T00:14:21Z")

</div>

Inspired by this post:

> [@When will "Uncollected Garbage" be deleted?](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/256):
>
> The OS does have to read the inode on `unlink`, but of course it unfortunately doesn’t return any of the info from the deleted inode to the `unlink` caller. We have to call `stat` separately. I’m not sure why a subsequent `unlink` doesn’t read the inode from cache at that point, but some profiling (on ext4+linux) suggests that it does not, in at least some cases. Maybe for something like a delete, a cached inode is not good enough? I haven’t gone into depth here.

I wonder whether maybe it would also make sense to cache information about existing directories, so that a node wouldn’t do the equivalent of `mkdir -p` for each single upload, trash piece deletion or GC file move operation. With Python-like pseudocode:

```
directories_we_know_exist = set()

# replace each existing call to os.MkdirAll with:
if directory not in directories_we_know_exist:
    os.MkdirAll(directory, …)
    directories_we_know_exist.add(directory)

```

As storage nodes never removes directories which would ever be used again, we wouldn’t have to worry about cache invalidation. And given the number of directories used by nodes is very small, it could be a simple RAM-based cache, no need to persist it.

If not calling stat() in the linked thread helped despite no clear explanation why system cache doesn’t help, not calling mkdir() might help as well… I can’t test this hypothesis, as on my machine there’s no difference in either stat() or mkdir(). But it seems thepaul has access to an environment that might have the necessary qualities to test it.

---

<div class="post-metadata">

**Author:** ![thepaul](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/thepaul/32/65_2.png) [@thepaul](https://forum.storj.io/u/thepaul)\
**Post date:** [September 12, 2024, 3:29am UTC](https://forum.storj.io/t/not-calling-os-mkdirall-all-the-time/27851/2 "2024-09-12T03:29:24Z")

</div>

I’ve been thinking, instead, that we should just omit the `mkdir -p` call entirely. Only when the `open`/`rename` fails with `ENOENT` would we do the `mkdir` and then try again. The directory is already present the vast majority of the time, I would think.

Something like this:

```go
f, err := os.OpenFile(destPath, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o644)
if err != nil {
    if errors.Is(err, fs.ErrNotExist) {
        err = os.MkdirAll(filepath.Dir(destPath), 0o700)
        if err != nil {
            return err
        }
        f, err = os.OpenFile(destPath, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o644)
        if err != nil {
            return err
        }
    } else {
        return err
    }
}

```

Does that sound like a good alternative?

Although, if the `mkdir` doesn’t cost anything on your machine, and your machine is more representative of nodes in the community, maybe it’s not worth doing.

---

<div class="post-metadata">

**Author:** ![edo](https://storj.bcdn.literatehosting.com/letter_avatar/edo/32/5_5575768a8748004e209b776fc1b2916d.png) [@edo](https://forum.storj.io/u/edo)\
**Post date:** [September 12, 2024, 7:08am UTC](https://forum.storj.io/t/not-calling-os-mkdirall-all-the-time/27851/3 "2024-09-12T07:08:33Z")

</div>

@thepaul, I see what you’re aiming for, and it makes sense since the directory is usually there. However, using exceptions for regular control flow—like relying on ENOENT to create directories—is known as the _Exception as Control Flow anti-pattern_. While it can work, it can also make the code harder to maintain over time.

That said, if directory creation is rare, this approach might save some time! Just something to keep in mind. 😊

---

<div class="post-metadata">

**Author:** ![Toyoo](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/toyoo/32/1126_2.png) [@Toyoo](https://forum.storj.io/u/Toyoo)\
**Post date:** [September 12, 2024, 8:43am UTC](https://forum.storj.io/t/not-calling-os-mkdirall-all-the-time/27851/4 "2024-09-12T08:43:35Z")

</div>

> [@thepaul](#):
>
> Does that sound like a good alternative?

Sure!

> [@edo](#):
>
> Exception as Control Flow

> Yet we should not pass up our opportunities in that critical 3%

(Knuth)

---

<div class="post-metadata">

**Author:** ![Mitsos](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/mitsos/32/1125_2.png) [@Mitsos](https://forum.storj.io/u/Mitsos)\
**Post date:** [September 12, 2024, 1:34pm UTC](https://forum.storj.io/t/not-calling-os-mkdirall-all-the-time/27851/5 "2024-09-12T13:34:26Z")

</div>

```auto
$ storagenode setup
Getting things ready...
Creating satellite directories...
US1 satellite: Creating gf/77 directory...

```

---

<div class="post-metadata">

**Author:** ![arrogantrabbit](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/arrogantrabbit/32/11200_2.png) [@arrogantrabbit](https://forum.storj.io/u/arrogantrabbit)\
**Post date:** [September 12, 2024, 2:41pm UTC](https://forum.storj.io/t/not-calling-os-mkdirall-all-the-time/27851/6 "2024-09-12T14:41:26Z")

</div>

> [@edo](#):
>
> However, using exceptions for regular control flow—like relying on ENOENT to create directories

Folder not existing _is_ an exception. On the mature node all these folders are expected to exist.

Another approach is to pre-create all those folders.

On the third hand I don’t believe mkdir is a bottleneck worth focusing on. Did anyone actually measured time mkdir takes? I would expect it to check existence of the directly path and bail out.

---

<div class="post-metadata">

**Author:** ![edo](https://storj.bcdn.literatehosting.com/letter_avatar/edo/32/5_5575768a8748004e209b776fc1b2916d.png) [@edo](https://forum.storj.io/u/edo)\
**Post date:** [September 12, 2024, 3:02pm UTC](https://forum.storj.io/t/not-calling-os-mkdirall-all-the-time/27851/7 "2024-09-12T15:02:31Z")

</div>

> [@arrogantrabbit](#):
>
> Folder not existing _is_ an exception.

Thanks for your input! While it’s true that it’s an exception when the folder does not exist, the point I’m making is more about good coding practices. It’s generally better to avoid using exception handling to control regular logic flow, such as creating directories in this case. Exceptions should ideally handle logging for example, rather than being part of the normal flow of logic.

---

<div class="post-metadata">

**Author:** ![Mark](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/mark/32/1128_2.png) [@Mark](https://forum.storj.io/u/Mark)\
**Post date:** [September 12, 2024, 8:50pm UTC](https://forum.storj.io/t/not-calling-os-mkdirall-all-the-time/27851/8 "2024-09-12T20:50:57Z")

</div>

> [@Mitsos](#):
>
> ```auto
> storagenode setup
> Getting things ready...
> Creating satellite directories...
> US1 satellite: Creating gf/77 directory...
> 
> ```

At the time of setup, I don’t think the setup command knows for sure which satellites the node operator plans on using now or later on. There could be new satellites in the future. Might want to create all directories for a satellite the first time a user connects to that satellite.

---

<div class="post-metadata">

**Author:** ![arrogantrabbit](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/arrogantrabbit/32/11200_2.png) [@arrogantrabbit](https://forum.storj.io/u/arrogantrabbit)\
**Post date:** [September 12, 2024, 9:03pm UTC](https://forum.storj.io/t/not-calling-os-mkdirall-all-the-time/27851/9 "2024-09-12T21:03:52Z")

</div>

> [@edo](#):
>
> Exceptions should ideally handle logging for example

I’m not sure I understand what are you describing here.

> [@edo](#):
>
> Exceptions should ideally handle logging for example, rather than being part of the normal flow of logic.

Why?

If an expected, uncexceptional, part-of-normal-operation event A happens 0.001% of the time, but the check for it will be executed 100% of the time, it’s perfectly fine to use exception mechanics to handle case A. Slowing down the 100% of cases just so that the GOF is happy is highly counterproductive. These patterns are nice to ponder at in the academic setting, but the reality is very different.

> [@edo](#):
>
> While it can work, it can also make the code harder to maintain over time.

It can also make the code easier to maintain: the code will reflect only business logic, and the setup (e.g. folder creation) will be handled elsewhere.

In fact, it’s a very useful approach – write simple code assuming all prerequisites are satisfied, and handle missing dependencies in the exception handler. Makes the code very readable and simple. Simple → good.

Again, the fact that some books don’t like it is a problem entirely in the heads of the authors and does not require solutions, let alone anyone changing their behavior, outside of it.

(disclaimer, I’ve read that book at the early stage of my career, and 25 years later I strongly consider it 100% waste of time. Real world problems defined by real world constraints ~~rarely~~ never fit into pure academic constructs.)

---

<div class="post-metadata">

**Author:** ![Toyoo](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/toyoo/32/1126_2.png) [@Toyoo](https://forum.storj.io/u/Toyoo)\
**Post date:** [September 13, 2024, 12:13am UTC](https://forum.storj.io/t/not-calling-os-mkdirall-all-the-time/27851/10 "2024-09-13T00:13:37Z")

</div>

> [@Mitsos](#):
>
> ```auto
> US1 satellite: Creating gf/77 directory...
> 
> ```

Satellites are supposed to come and go in future.

> [@arrogantrabbit](#):
>
> On the third hand I don’t believe mkdir is a bottleneck worth focusing on. Did anyone actually measured time mkdir takes? I would expect it to check existence of the directly path and bail out.

That was exactly my thought about the stat call in the original thread, yet it seems for some environments it does matter.

---

<div class="post-metadata">

**Author:** ![Ambifacient](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/ambifacient/32/7815_2.png) [@Ambifacient](https://forum.storj.io/u/Ambifacient)\
**Post date:** [September 13, 2024, 3:22am UTC](https://forum.storj.io/t/not-calling-os-mkdirall-all-the-time/27851/11 "2024-09-13T03:22:01Z")

</div>

> [@Toyoo](#):
>
> Satellites are supposed to come and go in future.

Maybe then folder creation can be tied to trust/untrust.

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [September 13, 2024, 7:46am UTC](https://forum.storj.io/t/not-calling-os-mkdirall-all-the-time/27851/12 "2024-09-13T07:46:18Z")

</div>

I have measured, not for the Go code and not for the node, but well… I use `mkdir -p` for years to do not break my scripts just because someone (for example - me) deleted a required folder.  
From my tests those days `if [! -d "dir"]; then mkdir -p "dir"; fi` was a much slower on amounts after 1,000. And I needed a very similar directory structure.

---

<div class="post-metadata">

**Author:** ![Mitsos](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/mitsos/32/1125_2.png) [@Mitsos](https://forum.storj.io/u/Mitsos)\
**Post date:** [September 13, 2024, 2:20pm UTC](https://forum.storj.io/t/not-calling-os-mkdirall-all-the-time/27851/13 "2024-09-13T14:20:54Z")

</div>

> [@Toyoo](#):
>
> Satellites are supposed to come and go in future.

I’ve also heard rumors of the heat death of the universe at some point.

I’m 100% sure a simple pull of the trusted satellite list on node restart and a forked process to pre-create the directories is easy to implement.