Node won't start after adding zksync

Node Operators troubleshooting
,
1

My node was running for several months without issue when I tried to “upgrade” to zksync for faster payouts. Now it won’t start at all. I get this error in the logs followed by a very long list of options:

Error: open config/config.yaml: permission denied
Usage:
  storagenode run [flags]

Flags:
      --bandwidth.interval duration                              how frequently bandwidth usage

I copied config.yaml into a new config directory and did a chmod 777 on each. Now I get:

Error: Failed to load identity: file or directory not found: open identity/identity.cert: permission denied
	storj.io/common/identity.Config.Load:326
	main.cmdRun:56
	main.newRunCmd.func1:33

The identity files are all in the proper location. If I remove the changes to config and config.yaml, it goes back to the first error.

I am using Ubuntu 22.04 with the docker image. The only thing I changed which triggered this mess was to add the --operator.wallet-features=zksync-era
Then all hell broke loose.

2

You need to modify the existing line, not adding a second operator.wallet-features this will break newer versions

just a guess

Welcome.

3

Whether I start docker with the zksync line or without it makes no difference.

4

You need to give the user that runs the node permission to read this file. You saved it and reset permissions to yourself, or root, but storagenode runs as another user and no longer has access.

Check owner with ls -al and fix them with chown.

If you have AppArmor or SELinux active — it’s more complex, do that from the container.

2 Likes
5

I chown’d all the directories down to the ca.cert and finally made some progress. Now I have this:

Error: Error opening database on storagenode: group:
--- stat config/storage/blobs: permission denied
--- stat config/storage/temp: permission denied
--- stat config/storage/trash: permission denied

These directories are all under /mnt/storj/storj-data/database
They are all owner=root

It seems like something changed with that Ubuntu update I did last night. All this stuff was just fine until then.

6

Turn off AppArmor temporarily; if this resolves the issue -you’ll know the culprit.

But does storagenode run as root? Root in the container is not the same thing as root on the host. Well, technically it is, but docker tries to prevent that, and AppArmor does too. I’ve tried to explained it somewhat here

7

I have AppArmor installed but didn’t realize (realise?) it. ufw has port 28967 open.
I just pulled the latest docker from storj - no difference.
I always run docker with sudo. I thought that would take care of any permission problems but I guess not. There is no root user account on Ubuntu so I can’t do a “su root”. Is running docker with sudo not typical?
I’m going to try chown’ing -R on the storj-data drive and see what happens.

8

Try suggestions in my comment above first. Don’t mess with permissions without understanding what happened first.

Root may be in the different namespace. Storagenode may be running as a different user altogether ( I don’t remember what storj does in their container)

9

Thanks for your help. I’m done.

10

what was the culprit?

11

Hello @a75019b4749adb88f980,
Welcome to the forum!

these ones now related to a storage location, not a databases location.
I guess you also added --user $(id -u):$(id -g) to your docker run command, so you need to change an owner to that user at least.

12

All my problems were “solved” with a hard drive crash. :wink:
I ordered a new one and am trying to restart but when the node starts, it always starts with the old address. I re-did the identity stuff but somehow the nodeID is the old one. Is there something else I have to remove?

13

Guess you forgot to remove the old identity in the standard folder?

14

I did the identity calculation on another machine and tried to copy stuff over but I guess it didn’t work somehow. Just did it again and I up!!!

2 Likes
15

The auth token doesn’t matter. It will just sign any provided unsigned identity.
So, if you provide an old one - you will get the old one but signed identity (without its data of course) and it will be disqualified as soon as you bring it online.
The Auth Token doesn’t make your identity unique, only the identity itself makes it.

1 Like