My storage node on two satellites was suspended

I have got two emails that says my storage node is suspended. I have just installed the node 2 days ago on linux ubuntu server 20.04.

Email: “Your Storage Node on the saltlake Satellite was suspended. You were suspended on 2020-05-19 at 00:43 UTC. You won’t receive any new data on this Satellite until you resolve the issue causing audit failures on your node.”

LOG:

2020-05-19T03:56:46.440Z ERROR orders archiving orders {“error”: “disk I/O error”}
2020-05-19T04:07:48.992Z INFO piecestore download started {“Piece ID”: “NK6HODKFPC3BI3PSVL5FNOYW3FORVGPCETBUSCTKZDOZWUGGD3IQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET_AUDIT”}
2020-05-19T04:07:48.993Z ERROR piecestore download failed {“Piece ID”: “NK6HODKFPC3BI3PSVL5FNOYW3FORVGPCETBUSCTKZDOZWUGGD3IQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET_AUDIT”, “error”: “usedserialsdb error: disk I/O error”, “errorVerbose”: “usedserialsdb error: disk I/O error\n\tstorj. io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj. io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj. io/storj/storagenode/piecestore.(*Endpoint).doDownload:523\n\tstorj. io/storj/storagenode/piecestore.(*drpcEndpoint).Download:471\n\tstorj. io/common/pb.DRPCPiecestoreDescription.Method.func2:995\n\tstorj. io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj. io/common/rpc/rpctracing.(*Handler).HandleRPC:66\n\tstorj. io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj. io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj. io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj. io/drpc/drpcctx.(*Tracker).track:51”}
2020-05-19T04:54:42.676Z INFO bandwidth Performing bandwidth usage rollups
2020-05-19T04:54:42.678Z ERROR bandwidth Could not rollup bandwidth usage {“error”: “disk I/O error”}
2020-05-19T04:54:42.681Z ERROR piecestore:cache error persisting cache totals to the database: {“error”: “piece space used error: disk I/O error”, “errorVerbose”: “piece space used error: disk I/O error\n\tstorj. io/storj/storagenode/storagenodedb.(*pieceSpaceUsedDB).UpdatePieceTotals:174\n\tstorj. io/storj/storagenode/pieces. (*CacheService).PersistCacheTotals:100\n\tstorj. io/storj/storagenode/pieces. (*CacheService).Run.func1:85\n\tstorj. io/common/sync2. (*Cycle).Run:152\n\tstorj. io/storj/storagenode/pieces. (*CacheService).Run:80\n\tstorj. io/storj/private/lifecycle. (*Group).Run.func1:56\n\tgolang. org/x/sync/errgroup.(*Group).Go.func1:57”}
2020-05-19T04:59:37.298Z INFO orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S sending {“count”: 9}
2020-05-19T04:59:37.298Z INFO orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 sending {“count”: 4}
2020-05-19T04:59:37.298Z INFO orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs sending {“count”: 1}
2020-05-19T04:59:37.444Z INFO orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs finished
2020-05-19T04:59:37.851Z INFO orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S finished
2020-05-19T04:59:38.200Z INFO orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 finished
2020-05-19T04:59:38.202Z ERROR orders archiving orders {“error”: “disk I/O error”}

MY DATA FODER:

root@next:/storj/data# ls -l
total 40
-rw------- 1 root root 7701 May 17 17:46 config.yaml
-rw------- 1 root root 32768 May 17 17:54 revocations.db
drwx------ 6 root root 4096 May 17 17:54 storage
-rw------- 1 root root 1430 May 18 06:34 trust-cache.json

There is a disk issue. Check if your disk is properly connected and mounted.

Also check this.

https://documentation.storj.io/resources/faq/linux-static-mount

It is mounted and connected. My storj folder owner is root, is this ok?

My disk:

root@next:~# lsblk -d -fs /dev/sdb
NAME FSTYPE LABEL UUID FSAVAIL FSUSE% MOUNTPOINT
sdb ext4 16378765-1999-4f72-ad0c-07799b1d499e 6.9T 0% /storj

FSTAB: (/etc/fstab)

/dev/disk/by-uuid/825d6c60-c69e-4c87-970e-f143af5807ab / ext4 defaults 0 0
/swap.img none swap sw 0 0
UUID=16378765-1999-4f72-ad0c-07799b1d499e /storj ext4 defaults 0 2
UUID=d09fb5ed-c11c-46b0-a2ed-63924b814850 /next ext4 defaults 0 2

Storj directory:
/storj/data

Show your docker run command and remove any personal info from it.

docker run -d --restart unless-stopped --stop-timeout 300
-p 28967:28967
-p 127.0.0.1:14002:14002
-e WALLET=“0xXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX”
-e EMAIL="XXXXXXX@gmail.com"
-e ADDRESS=“X.X.X.X:28967”
-e STORAGE=“6TB”
–mount type=bind,source="/root/.local/share/storj/identity/storagenode",destination=/app/identity
–mount type=bind,source="/storj/data",destination=/app/config
–name storagenode storjlabs/storagenode:beta

Can you verify your command does not have curly quotes in it ?

The IO errors are a little surprising during these low traffic times. What model HDD are you using? Is anything else running on it, specifically high loads?

It seems to be a HDD error, I am trying a check disk

2 Likes