Node stopped working - trust-cache.json: permission denied

Node Operators troubleshooting
1

Approx. two days ago the node stopped working out of a sudden!? I am not aware of the reason nor did I do changes, updates or any of the sort. I am stuck with the log message not knowing what to do:

2024-01-22T16:09:38Z FATAL Unrecoverable error {“process”: “storagenode”, “error”: “trust: open config/trust-cache.json: permission denied”, “errorVerbose”: “trust: open config/trust-cache.json: permission denied\n\tstorj.io/storj/storagenode/trust.LoadCacheData:105\n\tstorj.io/storj/storagenode/trust.LoadCache:36\n\tstorj.io/storj/storagenode/trust.NewPool:91\n\tstorj.io/storj/storagenode.New:405\n\tmain.cmdRun:79\n\tmain.newRunCmd.func1:32\n\tstorj.io/private/process.cleanup.func1.4:393\n\tstorj.io/private/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomOptions:112\n\tmain.main:30\n\truntime.main:267”}

Version is as of today latest.
There are no other log errors.
Renaming the json didn´t work, it wasn´t recreated.

Any help is appreciated.
Additional infos needed? Don´t hesitate to request.

Existing topics seem to be very old, couldn´t get to know what to do.

2

You have a file permission issue here. Node is unable to open the .json file.

3 Likes
3

Hello @Barn,
Welcome back!

Seems you added --user $(id -u):$(id -g) to your docker run command recently, but used sudo before.
You have two options:

  • either remove --user $(id -u):$(id -g) and continue run with sudo, but need to change the owner to root for all storagenode data recursively;
  • or do not use sudo (perhaps you also need to fix your docker setup: Linux post-installation steps for Docker Engine | Docker Docs) and change the owner to $(id -u):$(id -g) for storagenode data recursively.
2 Likes
4

Thank you very much for the helpful support.

@Alexey I opt in for bullet point number 1 and I am currently changing the owner back to root again.
You were right with the newly added --user, I had a offline problem with the node recently (~start of january or alike) and updated the socker command with the wiki also because of QUIC.
The node is switching between suspended states now. I hope the best.

Can you asses how long the owner change will take for a 18TB node?

(Ah it would be such a pitty to loose that node… it is finally full after ~4 years (?)… grrr…)

5

Depends on various conditions:

  • How many pieces
  • How the disk is connected
  • What’s filesystem
  • How much RAM
  • What’s CPU
  • What’s disk controller
  • etc.

In short - I have no idea, how long it would take on your system.
You wouldn’t lose your node if you did not lose more than 4% of data. With online score below 60% your node will be suspended, but you can recover it being 30 days online.

6

Understood. Thanks again for the help and the quick support.

Issue solved for now.
Node is stable and unsuspended since ~ 18 hours.

1 Like