Error: Error during preflight check for storagenode databases

Hi there,

i tried to update my Storj Node yesterday but i got two errors.

First this: Error: Error during preflight check for storagenode databases: storage node preflight database error: heldamount: expected schema does not match actual: &dbschema.Schema{

I´ve rename the heldamount.db

And then i got this Error

Error: Error during preflight check for storagenode databases: storage node preflight database error: storage_usage: expected schema does not match actual:   &dbschema.Schema{

-  Tables: []*dbschema.Table{
-  s"Name: storage_usage\nColumns:\n\tName: at_rest_total\n\tType: REAL\n\tNullable: false\n\tDefault: \"\"\n\tReference: nil\n\tName: interval_start\n\tType: TIMESTAMP\n\tNullable: false\n\tDefault: \"\"\n\tReference: nil\n\tName: satellite_id\n\tType: BLOB\n\tNullable: false\n\tDefault: \"\"\n\tReference: nil\nPrimaryKey: interval_start satellite_id\nUniques:\n\t",
-  },
+  Tables: nil,
   Indexes: nil,
  }

storj.io/storj/storagenode/storagenodedb.(*DB).Preflight:322
main.cmdRun:190
storj.io/private/process.cleanup.func1.4:359
storj.io/private/process.cleanup.func1:377
github.com/spf13/cobra.(*Command).execute:840
github.com/spf13/cobra.(*Command).ExecuteC:945
github.com/spf13/cobra.(*Command).Execute:885
storj.io/private/process.ExecWithCustomConfig:88
storj.io/private/process.ExecCustomDebug:70
main.main:320
runtime.main:203
2020-07-09T22:48:07.872Z INFO Configuration loaded {"Location": "/app/config/config.yaml"}
2020-07-09T22:48:07.878Z INFO Operator email {"Address": "xxxx"}
2020-07-09T22:48:07.878Z INFO Operator wallet {"Address": "xxxxx"}
2020-07-09T22:48:09.072Z INFO Telemetry enabled
2020-07-09T22:48:09.075Z INFO db.migration Database Version {"version": 42}

I´ve renamed the storage_usage db an restart storj with the same Error and Storj create a new storage_usage db.

Does anyone know the solution?

best regards

Chrysen

Why are you renaming dbs ?

1 Like

I thought that storj create a new db and fix the Problem

1 Like

I fixed IT with :sweat_smile:

Now i got this🤔

2020-07-10T16:33:17.543Z ERROR db Unable to read the disk, please verify the disk is not corrupt

2020-07-10T16:33:17.543Z ERROR piecestore:cache error getting current space used calculation: {“error”: “lstat config/storage/blobs/6r2fgwqz3manwt4aogq343bfkh2n5vvg4ohqqgggrrunaaaaaaaa/3n/3x2rnk74j3rk2qfai5vjc55jxhxyzi6vu7z5cgcsmnkljvqmia.sj1: structure needs cleaning”, “errorVerbose”: “lstat config/storage/blobs/6r2fgwqz3manwt4aogq343bfkh2n5vvg4ohqqgggrrunaaaaaaaa/3n/3x2rnk74j3rk2qfai5vjc55jxhxyzi6vu7z5cgcsmnkljvqmia.sj1: structure needs cleaning\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:714\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:652\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:612\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:258\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:468\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:633\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func1:56\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

2020-07-10T16:33:17.544Z INFO piecestore downloaded {“Piece ID”: “MUSSLFTRWUXQLA2PCL6O4YPZYPUXZSXJLWBFI5GERWHWQXXW23MA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”}
2020-07-10T16:33:17.544Z INFO piecestore downloaded {“Piece ID”: “FUACAQVEHYXAT5ZQY7C54DHAD743O6CUYJ33FP65YNVIJN5LBHAQ”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “GET”}
Error: lstat config/storage/blobs/6r2fgwqz3manwt4aogq343bfkh2n5vvg4ohqqgggrrunaaaaaaaa/3n/3x2rnk74j3rk2qfai5vjc55jxhxyzi6vu7z5cgcsmnkljvqmia.sj1: structure needs cleaning

I’ve delete some folders Weeks ago, not Storj Data. Only other stuff.

What is the version of your node now?

I Use1.6.4 (with beta tag)
Docker Container on Unraid (Slackware based OS)

Does your docker run command follow this template ?

docker run -d --restart unless-stopped --stop-timeout 300 \
    -p 28967:28967 \
    -p 127.0.0.1:14002:14002 \
    -e WALLET="0xXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" \
    -e EMAIL="user@example.com" \
    -e ADDRESS="domain.ddns.net:28967" \
    -e STORAGE="2TB" \
    --mount type=bind,source="<identity-dir>",destination=/app/identity \
    --mount type=bind,source="<storage-dir>",destination=/app/config \
    --name storagenode storjlabs/storagenode:latest

Its Look like this.


name=‘StorjNode-V3-disk5’ --net=‘host’ -e TZ=“Europe/Berlin” -e HOST_OS=“Unraid” -e ‘TCP_PORT_28967’=‘28967’ -e ‘UDP_PORT_28967’=‘28967’ -e ‘WALLET’ xxx’ -e ‘EMAIL’=‘yyy’ -e ‘ADDRESS’=‘111. 111.111.1:28967’ -e ‘BANDWIDTH’=‘2000TB’ -e ‘STORAGE’=‘3900GB’ -e ‘TCP_PORT_14002’=‘14002’ -v ‘/mnt/disk5/storj/storagenode/’:’/app/identity’:‘rw’ -v ‘/mnt/disk5/storj/DATA/’:’/app/config’:‘rw’ ‘storjlabs/storagenode:beta’

zzzzz

I try that. But with many different Error.

My node works finde with my own old config for 8 Month yet.

Mhh

Your node is a ticking time bomb if you keep using the -v mounts. You should also be switching to the latest tag.

This is a pretty clear message and should not be ignored. Please check your file system for errors.

1 Like

I’ve change the Docker (without - v mount)
and i check my disk and Filesystem all ok, i got no errors…

If i Start the Storj Docker i got this:

2020-07-11T09:45:01.316Z INFO Configuration loaded {“Location”: “/app/config/config.yaml”}
2020-07-11T09:45:01.323Z INFO Operator email {“Address”: “xxx”}
2020-07-11T09:45:01.323Z INFO Operator wallet {“Address”: “yyy”}
2020-07-11T09:45:17.132Z INFO Telemetry enabled
2020-07-11T09:45:17.134Z INFO db.migration Database Version {“version”: 42}

2020-07-11T09:45:17.132Z INFO Telemetry enabled
2020-07-11T09:45:17.134Z INFO db.migration Database Version {“version”: 42}

2020-07-11T09:45:18.467Z INFO preflight:localtime start checking local system clock with trusted satellites’ system clock.
2020-07-11T09:45:19.326Z INFO preflight:localtime local system clock is in sync with trusted satellites’ system clock.
2020-07-11T09:45:19.326Z INFO bandwidth Performing bandwidth usage rollups
2020-07-11T09:45:19.326Z INFO Node 12pYbE71MqmW21tLN1U8GoLhVooL7YJG8LuvhQcPU5SeCruEu1s started
2020-07-11T09:45:19.326Z INFO Public server started on [::]:28967
2020-07-11T09:45:19.326Z INFO Private server started on 127.0.0.1:7778
2020-07-11T09:45:19.326Z INFO trust Scheduling next refresh {“after”: “7h7m39.71100414s”}
2020-07-11T09:45:19.326Z INFO preflight:localtime local system clock is in sync with trusted satellites’ system clock.
2020-07-11T09:45:19.326Z INFO bandwidth Performing bandwidth usage rollups
2020-07-11T09:45:19.326Z INFO Node 12pYbE71MqmW21tLN1U8GoLhVooL7YJG8LuvhQcPU5SeCruEu1s started
2020-07-11T09:45:19.326Z INFO Public server started on [::]:28967
2020-07-11T09:45:19.326Z INFO Private server started on 127.0.0.1:7778
2020-07-11T09:45:19.326Z INFO trust Scheduling next refresh {“after”: “7h7m39.71100414s”}
2020-07-11T09:45:35.027Z ERROR db Unable to read the disk, please verify the disk is not corrupt

2020-07-11T09:45:35.027Z ERROR piecestore:cache error getting current space used calculation: {“error”: “lstat config/storage/blobs/6r2fgwqz3manwt4aogq343bfkh2n5vvg4ohqqgggrrunaaaaaaaa/3n/3x2rnk74j3rk2qfai5vjc55jxhxyzi6vu7z5cgcsmnkljvqmia.sj1: structure needs cleaning”, “errorVerbose”: “lstat config/storage/blobs/6r2fgwqz3manwt4aogq343bfkh2n5vvg4ohqqgggrrunaaaaaaaa/3n/3x2rnk74j3rk2qfai5vjc55jxhxyzi6vu7z5cgcsmnkljvqmia.sj1: structure needs cleaning\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:714\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:652\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:612\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:258\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:468\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:633\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func1:56\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

Error: lstat config/storage/blobs/6r2fgwqz3manwt4aogq343bfkh2n5vvg4ohqqgggrrunaaaaaaaa/3n/3x2rnk74j3rk2qfai5vjc55jxhxyzi6vu7z5cgcsmnkljvqmia.sj1: structure needs cleaning

Error: lstat config/storage/blobs/6r2fgwqz3manwt4aogq343bfkh2n5vvg4ohqqgggrrunaaaaaaaa/3n/3x2rnk74j3rk2qfai5vjc55jxhxyzi6vu7z5cgcsmnkljvqmia.sj1: structure needs cleaning

This is yet more signal of file system issues. What did you do? Did you run fsck? What file system are you using?

I use xfs Filesystem and use xfsprogs to repair the Filesystem.

I’ve delete some stuff from HDD Weeks ago but no storj files.
Maybi storj think why i have more space It must be corrupt? …

Its Look like this now Mhh.

I’m not that familiar with xfs specifically, but that clearly shows failed CRC checks and bad blocks. The wording also suggests you ran a diagnostic without repairing. “would rewrite/would reset/would have cleared”

Stop the node, unmount the file system and run xfs_repair on it. Don’t use the -n flag. (I wrote this before I was even sure you used it. But looking at your screenshot, it’s listed at the bottom. The -n flag makes it only check but not fix anything)

Oh you’r right. Wrong Parameter

Now Its Look like this :sweat_smile:

I got no error in storj logs anymore.
I have to wait now and i hope the node has survived

If this was the problem and your node wasn’t disqualified yet, you’ll be fine. You would even be able to recover from suspension.

Edit: Your screenshot still shows the -n flag at the bottom and mentions the no modify flag is set several times. Please run without it.

You may also want to follow up with a SMART check. A good number of bad blocks could be a signal your HDD is starting to fail.

I’ve Start without flag before, and again with - n flag (double Check)

I got this since 40min

Looks good now, but do run that SMART check. You should be able to do that fine while online. If your HDD is indeed starting to fail, the problems will be back. It’s best to know beforehand.

1 Like