Problem after make an apt update &&apt upgrade. Storagenode always restart

Node Operators troubleshooting
1

Since I’ve done that my storagenode is always restating:
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4b0e483471e1 storjlabs/watchtower “/watchtower storage…” 6 minutes ago Up 6 minutes watchtower
85afd0276f6d storjlabs/storagenode:alpha “/entrypoint” 26 minutes ago Restarting (1) 43 seconds ago storagenode

How I can solve that?
Triyed to stop and start the storagenode. I’ve delete it and create it again but problem is the same

2

You should change the start up storjlabs/storagenode:alpha to storjlabs/storagenode:beta

1 Like
3

logs here:
I’ve that in logs:

2019-12-09T22:42:36.359106069Z 2019-12-09T22:42:36.358Z INFO Configuration loaded from: /app/config/config.yaml
2019-12-09T22:42:36.379618967Z 2019-12-09T22:42:36.379Z INFO Operator email: m
2019-12-09T22:42:36.379762532Z 2019-12-09T22:42:36.379Z INFO operator wallet:
2019-12-09T22:42:37.384926590Z 2019-12-09T22:42:37.384Z FATAL Unrecoverable error {“error”: “Error creating revocation database: revocation database error: boltdb error: timeout\n\tstorjDOTio/storj/storage/boltdb.New:43\n\tstorjDOTio/storj/pkg/revocation.newDBBolt:50\n\tstorjDOTio/storj/pkg/revocation.NewDB:33\n\tstorjDOTio/storj/pkg/revocation.NewDBFromCfg:21\n\tmain.cmdRun:162\n\tstorjDOTio/storj/pkg/process.cleanup.func1.2:278\n\tstorjDOTio/storj/pkg/process.cleanup.func1:296\n\tgithubDOTcom/spf13/cobra.(*Command).execute:826\n\tgithubDOTcom/spf13/cobra.(*Command).ExecuteC:914\n\tgithubDOTcom/spf13/cobra.(*Command).Execute:864\n\tstorjDOTio/storj/pkg/process.Exec:74\n\tmain.main:315\n\truntime.main:203”, “errorVerbose”: “Error creating revocation database: revocation database error: boltdb error: timeout\n\tstorjDOTio/storj/storage/boltdb.New:43\n\tstorjDOTio/storj/pkg/revocation.newDBBolt:50\n\tstorjDOTio/storj/pkg/revocation.NewDB:33\n\tstorjDOTio/storj/pkg/revocation.NewDBFromCfg:21\n\tmain.cmdRun:162\n\tstorjDOTio/storj/pkg/process.cleanup.func1.2:278\n\tstorjDOTio/storj/pkg/process.cleanup.func1:296\n\tgithubDOTcom/spf13/cobra.(*Command).execute:826\n\tgithubDOTcom/spf13/cobra.(*Command).ExecuteC:914\n\tgithubDOTcom/spf13/cobra.(*Command).Execute:864\n\tstorjDOTio/storj/pkg/process.Exec:74\n\tmain.main:315\n\truntime.main:203\n\tmain.cmdRun:164\n\tstorjDOTio/storj/pkg/process.cleanup.func1.2:278\n\tstorjDOTio/storj/pkg/process.cleanup.func1:296\n\tgithubDOTcom/spf13/cobra.(*Command).execute:826\n\tgithubDOTcom/spf13/cobra.(*Command).ExecuteC:914\n\tgithubDOTcom/spf13/cobra.(*Command).Execute:864\n\tstorjDOTio/storj/pkg/process.Exec:74\n\tmain.main:315\n\truntime.main:203”}

replace DOT by .

4

You should try removing the contrainer from docker and try starting a new one. Also what hardware is this running on because I had issues with arm processers after I updated kernals.

5

Same with beta:
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b065047f0645 storjlabs/storagenode:beta “/entrypoint” 44 seconds ago Restarting (1) 11 seconds ago storagenode
4b0e483471e1 storjlabs/watchtower “/watchtower storage…” 18 minutes ago Up 18 minutes watchtower

6

Try removing watchtower and storagenode containters

7

My server:
4.15.0-48-generic #51~16.04.1-Ubuntu SMP Fri Apr 5 12:01:12 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Make a docker rm storagenode and watchtower was already done

8

What does your command to start it look like?

9 
docker run -d --restart unless-stopped -p 28967:28967 \
-e WALLET="0xc" \
-e EMAIL="@gmail.com" \
-e ADDRESS="@IP:28967" \
-e BANDWIDTH="50TB" \
-e STORAGE="0.5TB" \
--mount type=bind,source="/root/.local/share/storj/identity/storagenode",destination=/app/identity \
--mount type=bind,source="/root/storagenode",destination=/app/config \
--name storagenode storjlabs/storagenode:beta
10

Not 100% sure at the moment your startup command looks fine, The only issue I can really think of is ubuntu 16 updating is what caused the problem.

11

I think that too!
I’m using the same start command as before. Problem appear after the upgade

12

I wish I had a better answer for you I run ubuntu 18.04, I havent updated since I started running my node in may the only experience I had was with a pi4 and it pretty much stopped working with docker all together after an update of the kernel.

13

Hello @bundyboss,
Welcome to the forum!
Since you store the identity and data under the root home directory, you must use a sudo to run the docker container.

If you do not, then please re-apply permission for your user for the data folder recursively.

1 Like
14

Im assuming he had to start it in sudo or it would have said permission denied anyways, Least for me it does.

15

The lack of permissions could be a reason. The other way is read-only filesystem or disk inconsistency.
In the last case I would recommend to boot from other media and check the disk for errors.

16

I start it with user root so it’s not a permission problem

17

I’m on a VPS server so can’t boot from other media. For exemple watchtower is working fine. Only Storage node is not working well

18

Any others idea? It was working well for 6 months. No dev to analyse the problem?

19

Have you tried checking HDD for errors as Alexey pointed out? It seems like a permission problem

20

Start as root. so it’s not a right problem!
root@2:~/storagenode# ls -l
total 176
-rw-r–r-- 1 root root 715 May 6 2019 config.yaml
-rw------- 1 root root 262144 Oct 3 21:19 kademlia
-rw------- 1 root root 32768 Nov 20 19:16 revocations.db
drwx------ 7 root root 4096 Nov 20 19:19 storage