Node keeps restarting

Node Operators troubleshooting
,
1

What should I do? It started (I suspect so) after log rotation this night, my config for it is:

/storj/node.log {
    missingok
    notifempty
    rotate 12
    compress
    delaycompress
    postrotate
      docker stop -t 300 storagenode && docker start storagenode
    endscript
}

And log is now full of this:

2020-06-28T13:58:23.683Z        ERROR   piecestore:cache        error getting current space used calculation:   {"error": "lstat config/storage/blobs/6r2fgwqz3manwt4aogq343bfkh2n5vvg4ohqqgggrrunaaaaaaaa/mt/5j4jhmbf2ypmrxtibphemlc6edd556mybqizdfdxmuwtey2jwq.sj1: structure needs cleaning; lstat config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/nm/zmvxwysktddbsewigl5q3lluwnoqq5ah54ddepdkgekm3fcxxq.sj1: structure needs cleaning", "errorVerbose": "group:\n--- lstat config/storage/blobs/6r2fgwqz3manwt4aogq343bfkh2n5vvg4ohqqgggrrunaaaaaaaa/mt/5j4jhmbf2ypmrxtibphemlc6edd556mybqizdfdxmuwtey2jwq.sj1: structure needs cleaning\n--- lstat config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/nm/zmvxwysktddbsewigl5q3lluwnoqq5ah54ddepdkgekm3fcxxq.sj1: structure needs cleaning"}
    storj.io/storj/storagenode/pieces.(*CacheService).Run
            /go/src/storj.io/storj/storagenode/pieces/cache.go:56
    storj.io/storj/private/lifecycle.(*Group).Run.func1
            /go/src/storj.io/storj/private/lifecycle/group.go:56
    golang.org/x/sync/errgroup.(*Group).Go.func1
            /go/pkg/mod/golang.org/x/sync@v0.0.0-20200317015054-43a5402ce75a/errgroup/errgroup.go:57
    2020-06-28T13:58:23.818Z        FATAL   Unrecoverable error     {"error": "lstat config/storage/blobs/6r2fgwqz3manwt4aogq343bfkh2n5vvg4ohqqgggrrunaaaaaaaa/mt/5j4jhmbf2ypmrxtibphemlc6edd556mybqizdfdxmuwtey2jwq.sj1: structure needs cleaning; lstat config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/nm/zmvxwysktddbsewigl5q3lluwnoqq5ah54ddepdkgekm3fcxxq.sj1: structure needs cleaning", "errorVerbose": "group:\n--- lstat config/storage/blobs/6r2fgwqz3manwt4aogq343bfkh2n5vvg4ohqqgggrrunaaaaaaaa/mt/5j4jhmbf2ypmrxtibphemlc6edd556mybqizdfdxmuwtey2jwq.sj1: structure needs cleaning\n--- lstat config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/nm/zmvxwysktddbsewigl5q3lluwnoqq5ah54ddepdkgekm3fcxxq.sj1: structure needs cleaning"}
    storj.io/private/process.cleanup.func1
            /go/pkg/mod/storj.io/private@v0.0.0-20200504130741-565a173f6d33/process/exec_conf.go:396
    github.com/spf13/cobra.(*Command).execute
            /go/pkg/mod/github.com/spf13/cobra@v0.0.6/command.go:840
    github.com/spf13/cobra.(*Command).ExecuteC
            /go/pkg/mod/github.com/spf13/cobra@v0.0.6/command.go:945
    github.com/spf13/cobra.(*Command).Execute
            /go/pkg/mod/github.com/spf13/cobra@v0.0.6/command.go:885
    storj.io/private/process.ExecWithCustomConfig
            /go/pkg/mod/storj.io/private@v0.0.0-20200504130741-565a173f6d33/process/exec_conf.go:88
    storj.io/private/process.ExecCustomDebug
            /go/pkg/mod/storj.io/private@v0.0.0-20200504130741-565a173f6d33/process/exec_conf.go:70
    main.main
            /go/src/storj.io/storj/cmd/storagenode/main.go:320
    runtime.main
            /usr/local/go/src/runtime/proc.go:203
2

ive seen this in relation to allocating more usable space than actual available physical diskspace, so tweak your storace parameter correctly.

3

What is the version of your node ? Is it Windows or Linux ?

4

clearly its linux given the person is posting a logrotate configuration file.

5

@binary do not use logrotate outside of docker. use the --log-opt max-size=50m \ in container startup. e.g.:

docker run \
--name=storagenode \
--hostname=9e621e6003a8 \
--env=WALLET= \
--env= \
--env=ADDRESS= \
--env=BANDWIDTH= \
--env=STORAGE= \
--env=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
--env=GOARCH=arm \
-p 28967:28967 \
-p 14002:14002 \
--mount type=bind,source="/mnt/storagenode",destination=/app/identity \
--mount type=bind,source="/mnt/storagenode_appConfig",destination=/app/config \
--restart=unless-stopped \
--log-opt max-size=50m \
--detach=true \
storjlabs/storagenode:latest
6

This usually means you have file system corruption. Try running fsck on the file system your node data is on.

2 Likes
7

I ran xfs_repair - it fixed something but after mounting again I get same problem… Node is still restarting with “structure needs cleaning” error

8

You should do the xfs_repair on unmounted disk.

9

Of course, it doesn’t even run otherwise :slight_smile: However, as I stated, it didn’t help even by fixing some errors.

10

I also noticed it gets broken with FATAL only for piecestore:cache

11

It can’t be true - I allocated 845 GB out of 932 GB total, which leaves recommended 10%.

12

I can only suggest this method: