Storagenode keeps restarting after update

Node Operators troubleshooting
1

Hi,

this is my first post so I am sorry, if I miss something. My storagenode keeps restarting and doesn’t want to stay up. I tried to fix some corruption in a database, but it still doesn’t work. Here are my logs:

2020-07-06T17:08:07.793Z        INFO    Configuration loaded    {"Location": "/app/config/config.yaml"}
2020-07-06T17:08:07.800Z        INFO    Operator email  {"Address": "***"}
2020-07-06T17:08:07.803Z        INFO    Operator wallet {"Address": "***"}
2020-07-06T17:08:12.131Z        INFO    Telemetry enabled
2020-07-06T17:08:12.186Z        INFO    db.migration    Database Version        {"version": 42}
2020-07-06T17:08:19.610Z        INFO    preflight:localtime     start checking local system clock with trusted satellites' system clock.
2020-07-06T17:08:20.517Z        INFO    preflight:localtime     local system clock is in sync with trusted satellites' system clock.
2020-07-06T17:08:20.518Z        INFO    trust   Scheduling next refresh {"after": "5h4m46.074724979s"}
2020-07-06T17:08:20.520Z        INFO    bandwidth       Performing bandwidth usage rollups
2020-07-06T17:08:20.521Z        INFO    Node 129eZWydKszpJkzqCPo2b8QxcknQJspQSAtaS2H6sJqqfXeQXyU started
2020-07-06T17:08:20.522Z        INFO    Public server started on [::]:28967
2020-07-06T17:08:20.522Z        INFO    Private server started on 127.0.0.1:7778
2020-07-06T17:08:20.525Z        WARN    piecestore:monitor      Disk space is less than requested. Allocating space     {"bytes": 32103596032}
2020-07-06T17:08:20.525Z        ERROR   piecestore:monitor      Total disk space less than required minimum     {"bytes": 500000000000}
2020-07-06T17:08:20.527Z        ERROR   pieces:trash    emptying trash failed   {"error": "pieces error: filestore error: context canceled", "errorVerbose": "pieces error: filestore error: context canceled\n\tstorj.io/storj/storage/filestore.(*blobStore).EmptyTrash:140\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).EmptyTrash:309\n\tstorj.io/storj/storagenode/pieces.(*Store).EmptyTrash:338\n\tstorj.io/storj/storagenode/pieces.(*TrashChore).Run.func1:51\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-07-06T17:08:20.528Z        ERROR   gracefulexit:chore      error retrieving satellites.    {"error": "satellitesdb error: context canceled", "errorVerbose": "satellitesdb error: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*satellitesDB).ListGracefulExits.func1:103\n\tstorj.io/storj/storagenode/storagenodedb.(*satellitesDB).ListGracefulExits:115\n\tstorj.io/storj/storagenode/gracefulexit.(*Chore).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/storj/storagenode/gracefulexit.(*Chore).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func1:56\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-07-06T17:08:20.528Z        ERROR   pieces:trash    emptying trash failed   {"error": "pieces error: filestore error: context canceled", "errorVerbose": "pieces error: filestore error: context canceled\n\tstorj.io/storj/storage/filestore.(*blobStore).EmptyTrash:140\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).EmptyTrash:309\n\tstorj.io/storj/storagenode/pieces.(*Store).EmptyTrash:338\n\tstorj.io/storj/storagenode/pieces.(*TrashChore).Run.func1:51\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-07-06T17:08:20.528Z        ERROR   nodestats:cache Get pricing-model/join date failed      {"error": "context canceled"}
2020-07-06T17:08:20.532Z        ERROR   collector       error during collecting pieces:         {"error": "piece expiration error: context canceled", "errorVerbose": "piece expiration error: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceExpirationDB).GetExpired:39\n\tstorj.io/storj/storagenode/pieces.(*Store).GetExpired:492\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:86\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:55\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(*Service).Run:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func1:56\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-07-06T17:08:20.535Z        ERROR   piecestore:cache        error getting current space used calculation:   {"error": "context canceled; context canceled; context canceled; context canceled; context canceled; context canceled", "errorVerbose": "group:\n--- context canceled\n--- context canceled\n--- context canceled\n--- context canceled\n--- context canceled\n--- context canceled"}
Error: piecestore monitor: disk space requirement not met

Thanks for your help!!!

2

Welcome to the forum @Irgendj0!

How much space did you assign to your node ?

3

Currently, I got 820 GB asssigned to my node

4

check you HDD mounting propertly, looks like it see only 321GB on disk

5

I mount it via fstab. df shows the normal capacity. I don’t know, why only 321 GB show up.

6

Show your docker run command. Remove any personal info from it

7

docker run -d --restart unless-stopped -p 28967:28967 -p 14002:14002 -e WALLET=“" -e EMAIL="” -e ADDRESS="***:28967" -e BANDWIDTH=“3TB” -e STORAGE=“820GB” --mount type=bind,source="/home/pi/Identity/storagenode",destination=/app/identity --mount type=bind,source="/mnt/test",destination=/app/config --name storagenode storjlabs/storagenode:beta

8

You should update your command:
https://documentation.storj.io/setup/cli/storage-node

docker run -d --restart unless-stopped --stop-timeout 300 \
    -p 28967:28967 \
    -p 127.0.0.1:14002:14002 \
    -e WALLET="0xXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" \
    -e EMAIL="user@example.com" \
    -e ADDRESS="domain.ddns.net:28967" \
    -e STORAGE="2TB" \
    --mount type=bind,source="<identity-dir>",destination=/app/identity \
    --mount type=bind,source="<storage-dir>",destination=/app/config \
    --name storagenode storjlabs/storagenode:latest

Also make sure you aren’t using any curly quotes.

9

I had this exact error if I moved all db files. It was fixed by moving the db files starting with “pi” back in place.
I had thought the db files were all regenerated before this happened and perhaps I stopped the storagenode before it was ready

10

Replaced the command, it still shows those errors.

11

I did not move any files. I only did a backup for a corrupted one, according to this :https://support.storj.io/hc/en-us/articles/360029309111-How-to-fix-a-database-disk-image-is-malformed-.

12

Show a screenshot of your command.

13

grafik
grafik757×202 5.45 KB

14

Show output of

df -h

15 
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        15G  8.3G  5.5G  61% /
devtmpfs        459M     0  459M   0% /dev
tmpfs           464M     0  464M   0% /dev/shm
tmpfs           464M   49M  415M  11% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           464M     0  464M   0% /sys/fs/cgroup
/dev/mmcblk0p1  253M   52M  202M  21% /boot
tmpfs            93M     0   93M   0% /run/user/1000
/dev/sda1       916G  840G   30G  97% /mnt/test
16

Which corrupt database file did you fix ?

17

I fixed the “used_serial.db” database.

18

Did you do anything with the piece_spaced_used.db? I think your node may have lost stats of used space and now think that the node has used far less space than it actually has. The remaining space is not enough to start your node and so it crashes before it gets the time to recalculate used space. I’m not sure what to do to fix it though. Maybe artificially raise the used space it that db. But better wait for someone from storj to confirm something like that.

19

You can set the storage2.monitor.minimum-disk-space: to a lower value, save the configuration and restart the node.
It should recalculate the used space.

4 Likes
20

Ahh, thanks I totally forgot about that setting. That’s a much better idea than messing with the db’s.

@Irgendj0 please report back how this worked out!

2 Likes