Lsof: (stat: No such file or directory)

Node Operators troubleshooting
, ,
1

I’m full on all my nodes. should I be worried?

storageno 3218927 3219213 storageno root mem REG 252,0 4594899 /app/storagenode (stat: No such file or directory)
storageno 3218927 3219871 storageno root mem-W REG 0,37 26 /app/config/revocations.db (stat: No such file or directory)
storageno 3218927 3219871 storageno root mem REG 0,37 9455810 /app/config/storage/filestatcache/003883.sst (stat: No such file or directory)
storageno 3218927 3219871 storageno root mem REG 0,37 9463320 /app/config/storage/filestatcache/003953.sst (stat: No such file or directory)
storageno 3218927 3219871 storageno root mem REG 0,37 9463215 /app/config/storage/filestatcache/003952.sst (stat: No such file or directory)
storageno 3218927 3219871 storageno root mem REG 0,37 9461957 /app/config/storage/filestatcache/003987.sst (stat: No such file or directory)
storageno 3218927 3219871 storageno root mem REG 0,37 9456456 /app/config/storage/filestatcache/00001.mem (stat: No such file or directory)
storageno 3218927 3219871 storageno root mem REG 0,37 39039703 /app/config/storage/filestatcache/000063.vlog (stat: No such file or directory)
storageno 3218928 root mem REG 252,0 4594898 /app/storagenode-updater (stat: No such file or directory)
storageno 3218928 3218934 storageno root mem REG 252,0 4594898 /app/storagenode-updater (stat: No such file or directory)
storageno 3218928 3218935 storageno root mem REG 252,0 4594898 /app/storagenode-updater (stat: No such file or directory)

ps -p 3218928 -o pid,cmd
PID CMD
3218928 /app/storagenode-updater run --binary-location /app/storagenode --config-dir config --identity-dir identity --version.server-address=https://version.storj.io
root@zfstest:~# ps -p 3218927 -o pid,cmd
PID CMD
3218927 /app/storagenode run --config-dir config --identity-dir identity --version.server-address=https://version.storj.io --storage.allocated-disk-space=16TB

2

This is usually happen if the filesystem is corrupted or something is really missing.
Is it a binary setup or docker?
I would suggest to stop the node and check and fix the filesystem, then check the mount for /app/config.

3

Docker install. All nodes are up and running. No audit fails.
Fscked

4

Seems these errors from the system journals, not from the node, right?

Could it be a permissions issue? Do you run containers with sudo? Do you use --user option?
Please post the output inside the container:

docker exec -it storagenode ls -l /app/config

You need to figure out, which one node though.

5

I assume @agente didn’t post the whole output of lsof (Please put the loglines between 3 backticks otherwise they’re hard to read.)

I always have these “stat: No such file or directory” “errors”. They only appear with the file descriptor “mem”. Further down there are the same files with the real file descriptor without errors. I just ignore these “errors”

ChatGPT says this

The error message stat: No such file or directory related to the mem file descriptor in lsof typically occurs when the process has a memory-mapped file (such as a shared library, a dynamically loaded module, or a memory-mapped file from disk) that has since been removed from the file system.

6

LLM is not an authoritative source of any, let alone technical, information. Quoting LLM is like quoting a random word generator. You have to verify the output. And if you do — quote that instead.

7

Interesting. My docker nodes doesn’t seems have them.
Could it be a ZFS-specific somehow?

8

I’m using ext4 for my nodes.

9

I have this error on both. zfs and ext4. btw nodes are working without issue.
I will post more “well formatted” logs

10

We asked this question, maybe it is not an issue, but I do not trust LLM too :slight_smile:

11

In this case at least, the LLM is partially correct. That can happen when a file has been opened and has been unlinked from the filesystem since. The more likely situation here, though, is that the file is in a different mount namespace (e.g., a docker container) from the lsof instance and can’t be looked up by its path alone. There are probably other situations where this can happen as well.

Whether it shows the full “No such file or directory” error in each situation appears to depend on what version of lsof you have. The authors would prefer it not to happen, so they play whack-a-mole with these situations and try to eliminate the error. For example, see https://github.com/lsof-org/lsof/pull/239 .

In short, you don’t need to worry about this.

4 Likes