Node crashes after restart

not able to fix it! will try later on…

Fixed… No errors now! that file was on a different folder

getting same error! anyone who can help?

Error: Error migrating tables for database on storagenode: migrate: sql: Scan error on column index 0, name "MAX(version)": converting driver.Value type []uint8 ("\xaf,B\x00>\xfc\x82j\xb46\x1fs\xf9ؐ\x94!F\xfe\x0e\xbe\x80g\x86\xf8\xe7\x19\b\x00\x00\x00\x00") to a int64: invalid syntax

Since you cannot recover it, then you need to re-create it:

i cannot find .db-shm and .db-wal extensions! I’ve run the command sudo find -name .db-shm and no output! the other db are in …/storage/ folder!

storagewars@raspberrypi:// $ sudo find /home/storagewars/Hds/HD2/ -maxdepth 1 -iname “.db" -print0 -exec sqlite3 ‘{}’ ‘PRAGMA integrity_check;’ ‘;’
/home/storagewars/Hds/HD2/revocations.dbError: file is not a database
storagewars@raspberrypi:// $ find /home/storagewars/Hds/HD2/storage -maxdepth 1 -iname "
.db” -print0 -exec sqlite3 ‘{}’ ‘PRAGMA integrity_check;’ ‘;’
/home/storagewars/Hds/HD2/storage/piece_spaced_used.dbok
/home/storagewars/Hds/HD2/storage/orders.dbok
/home/storagewars/Hds/HD2/storage/notifications.dbok
/home/storagewars/Hds/HD2/storage/heldamount.dbok
/home/storagewars/Hds/HD2/storage/pieceinfo.dbok
/home/storagewars/Hds/HD2/storage/used_serial.dbok
/home/storagewars/Hds/HD2/storage/reputation.dbok
/home/storagewars/Hds/HD2/storage/info.dbok
/home/storagewars/Hds/HD2/storage/bandwidth.dbok
/home/storagewars/Hds/HD2/storage/secret.dbok
/home/storagewars/Hds/HD2/storage/used_space_per_prefix.dbok
/home/storagewars/Hds/HD2/storage/piece_expiration.dbok
/home/storagewars/Hds/HD2/storage/storage_usage.dbok
/home/storagewars/Hds/HD2/storage/garbage_collection_filewalker_progress.dbok
/home/storagewars/Hds/HD2/storage/satellites.dbok
/home/storagewars/Hds/HD2/storage/pricing.dbok

That’s good. This is mean that your node is not running and didn’t crash (these files are created only when the node is running and deleted when it’s gracefully stopped).

This one you may ignore (it should not be here anyway, it should be one level directory upper, near the config.yaml), you may even delete it, it will be recreated on the node restart.

All others seems doesn’t have issues.

What’s now logs says when you run the node?

Will see when the time let! I’m not able to see it now!

A sábado, 1/06/2024, 07:12, Alexey via Storj Community Forum (official) <storj@literatehosting.com> escreveu:

Do you mean that you do not see errors in your logs?

The only error that i’m able to see is the one that you also view

A sábado, 1/06/2024, 09:40, Alexey via Storj Community Forum (official) <storj@literatehosting.com> escreveu:

I cannot see any errors, they are stored/showed only your side…

I will give a feed when in front of them…

A sábado, 1/06/2024, 13:55, Alexey via Storj Community Forum (official) <storj@literatehosting.com> escreveu:

Please post any errors, if they are appear.

find /home/storagewars/Hds/HD2/storage -maxdepth 1 -iname "*.db" -print0 -exec sqlite3 '{}' 'PRAGMA integrity_check;' ';'
/home/storagewars/Hds/HD2/storage/piece_spaced_used.dbok
/home/storagewars/Hds/HD2/storage/orders.dbok
/home/storagewars/Hds/HD2/storage/notifications.dbok
/home/storagewars/Hds/HD2/storage/heldamount.dbok
/home/storagewars/Hds/HD2/storage/pieceinfo.dbok
/home/storagewars/Hds/HD2/storage/used_serial.dbok
/home/storagewars/Hds/HD2/storage/reputation.dbok
/home/storagewars/Hds/HD2/storage/info.dbok
/home/storagewars/Hds/HD2/storage/bandwidth.dbok
/home/storagewars/Hds/HD2/storage/secret.dbok
/home/storagewars/Hds/HD2/storage/used_space_per_prefix.dbok
/home/storagewars/Hds/HD2/storage/piece_expiration.dbok
/home/storagewars/Hds/HD2/storage/storage_usage.dbok
/home/storagewars/Hds/HD2/storage/garbage_collection_filewalker_progress.dbok
/home/storagewars/Hds/HD2/storage/satellites.dbok
/home/storagewars/Hds/HD2/storage/pricing.dbok

storagewars@raspberrypi:// $ sudo docker logs storagenode22

2024-06-01T15:32:15Z	ERROR	failure during run	{"Process": "storagenode", "error": "Error migrating tables for database on storagenode: migrate: sql: Scan error on column index 0, name \"MAX(version)\": converting driver.Value type []uint8 (\"\\xaf,B\\x00>\\xfc\\x82j\\xb46\\x1fs\\xf9ؐ\\x94!F\\xfe\\x0e\\xbe\\x80g\\x86\\xf8\\xe7\\x19\\b\\x00\\x00\\x00\\x00\") to a int64: invalid syntax\n\tstorj.io/storj/shared/dbutil/txutil.WithTx:53\n\tstorj.io/storj/private/migrate.(*Migration).getLatestVersion:252\n\tstorj.io/storj/private/migrate.(*Migration).Run:182\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:387\n\tmain.cmdRun:98\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:267", "errorVerbose": "Error migrating tables for database on storagenode: migrate: sql: Scan error on column index 0, name \"MAX(version)\": converting driver.Value type []uint8 (\"\\xaf,B\\x00>\\xfc\\x82j\\xb46\\x1fs\\xf9ؐ\\x94!F\\xfe\\x0e\\xbe\\x80g\\x86\\xf8\\xe7\\x19\\b\\x00\\x00\\x00\\x00\") to a int64: invalid syntax\n\tstorj.io/storj/shared/dbutil/txutil.WithTx:53\n\tstorj.io/storj/private/migrate.(*Migration).getLatestVersion:252\n\tstorj.io/storj/private/migrate.(*Migration).Run:182\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:387\n\tmain.cmdRun:98\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:267\n\tmain.cmdRun:100\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:267"}
Error: Error migrating tables for database on storagenode: migrate: sql: Scan error on column index 0, name "MAX(version)": converting driver.Value type []uint8 ("\xaf,B\x00>\xfc\x82j\xb46\x1fs\xf9ؐ\x94!F\xfe\x0e\xbe\x80g\x86\xf8\xe7\x19\b\x00\x00\x00\x00") to a int64: invalid syntax
	storj.io/storj/shared/dbutil/txutil.WithTx:53
	storj.io/storj/private/migrate.(*Migration).getLatestVersion:252
	storj.io/storj/private/migrate.(*Migration).Run:182
	storj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:387
	main.cmdRun:98
	main.newRunCmd.func1:33
	storj.io/common/process.cleanup.func1.4:393
	storj.io/common/process.cleanup.func1:411
	github.com/spf13/cobra.(*Command).execute:983
	github.com/spf13/cobra.(*Command).ExecuteC:1115
	github.com/spf13/cobra.(*Command).Execute:1039
	storj.io/common/process.ExecWithCustomOptions:112
	main.main:34
	runtime.main:267
2024-06-01 15:32:15,543 INFO stopped: storagenode (exit status 1)
2024-06-01 15:32:15,545 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)

storagewars@raspberrypi:// $ sudo docker ps -a
CONTAINER ID   IMAGE                          COMMAND                  CREATED        STATUS                          PORTS                                                                                                                      NAMES
3fa883aed762   storjlabs/storagenode:latest   "/entrypoint --opera…"   2 days ago     Exited (0) About a minute ago                                                                                                                              storagenode22
c0ca3b8c666b   storjlabs/storagenode:latest   "/entrypoint --opera…"   4 months ago   Up 4 days                       127.0.0.1:14003->14002/tcp, 0.0.0.0:28968->28967/tcp, 0.0.0.0:28968->28967/udp, :::28968->28967/tcp, :::28968->28967/udp   storagenode11
c385aa961387   hello-world                    "/hello"                 9 months ago   Exited (0) 9 months ago                                                                                                                                    frosty_fermat
storagewars@raspberrypi:// $ sudo docker rm storagenode22
storagenode22
storagewars@raspberrypi:// $ sudo docker run -d --restart always --stop-timeout 300 -p 28969:28967/tcp -p 28969:28967/udp -p 127.0.0.1:14002:14002 -e WALLET="0x20ba0ed29b38f63cfe96193b1e85365821a7058a" -e EMAIL="unofficialolym@gmail.com" -e ADDRESS="storagewarz.ddns.net:28969" -e STORAGE="1.7TB" --memory=800m --log-opt max-size=50m --log-opt max-file=10 --mount type=bind,source=/home/storagewars/Hds/HD2/ID2/,destination=/app/identity --mount type=bind,source=/home/storagewars/Hds/HD2,destination=/app/config --name storagenode22 storjlabs/storagenode:latest --operator.wallet-features=zksync
WARNING: Your kernel does not support memory limit capabilities or the cgroup is not mounted. Limitation discarded.
111eba2d71bd5e693d7ac6a2270e6acdf823f294b6e35eef7aeb8ed6754b7cac

Same error! Failure during run! port checker reveals port closed!

O errors and 0 bad blocks after running a health check! :upside_down_face:

storagewars@raspberrypi:// $ sudo docker ps -a
CONTAINER ID   IMAGE                          COMMAND                  CREATED          STATUS                    PORTS                                                                                                                      NAMES
111eba2d71bd   storjlabs/storagenode:latest   "/entrypoint --opera…"   16 minutes ago   Up 2 seconds              127.0.0.1:14002->14002/tcp, 0.0.0.0:28969->28967/tcp, 0.0.0.0:28969->28967/udp, :::28969->28967/tcp, :::28969->28967/udp   storagenode22
c0ca3b8c666b   storjlabs/storagenode:latest   "/entrypoint --opera…"   4 months ago     Up 4 days                 127.0.0.1:14003->14002/tcp, 0.0.0.0:28968->28967/tcp, 0.0.0.0:28968->28967/udp, :::28968->28967/tcp, :::28968->28967/udp   storagenode11
c385aa961387   hello-world                    "/hello"                 9 months ago     Exited (0) 9 months ago                                                                                                                              frosty_fermat

I can only suggest to recreate this database

Should all DB be deleted?

A sábado, 1/06/2024, 17:33, Alexey via Storj Community Forum (official) <storj@literatehosting.com> escreveu:

imagem_2024-06-02_000119883

No DB there! One output of the log:

2024-06-01T23:09:01Z	INFO	lazyfilewalker.used-space-filewalker	starting subprocess	{"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-06-01T23:09:01Z	ERROR	lazyfilewalker.used-space-filewalker	failed to start subprocess	{"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "error": "context canceled"}
2024-06-01T23:09:01Z	ERROR	pieces	failed to lazywalk space used by satellite	{"Process": "storagenode", "error": "lazyfilewalker: context canceled", "errorVerbose": "lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:73\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:130\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:704\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-06-01T23:09:01Z	ERROR	piecestore:cache	error getting current used space: 	{"Process": "storagenode", "error": "filewalker: context canceled; filewalker: context canceled; filewalker: context canceled; filewalker: context canceled", "errorVerbose": "group:\n--- filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:74\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:79\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:713\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78\n--- filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:74\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:79\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:713\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78\n--- filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:74\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:79\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:713\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78\n--- filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:74\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:79\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:713\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-06-01T23:09:02Z	ERROR	failure during run	{"Process": "storagenode", "error": "piecestore monitor: disk space requirement not met", "errorVerbose": "piecestore monitor: disk space requirement not met\n\tstorj.io/storj/storagenode/monitor.(*Service).Run:137\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
Error: piecestore monitor: disk space requirement not met
2024-06-01 23:09:02,071 INFO stopped: storagenode (exit status 1)
2024-06-01 23:09:02,074 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)

All DBs should be moved to the backup folder except the corrupted one. The corrupted one should be deleted.
But seems you deleted all databases instead, so the history is now gone. It’s not a problem and do not affect payouts.

This is because all databases were re-created, and there is no info about used space, so the node cannot start properly, because you do not have 500GB free on the disk.
This can be solved by adding/uncommenting option in the config.yaml file:

# how much disk space a node at minimum has to advertise (default 500.00 GB)
storage2.monitor.minimum-disk-space: 0B

Save the config and restart the node.
Or by adding it as a command line argument after the image name in your docker run command:

docker run ...
...
--name storagenode storjlabs/storagenode:latest \
--storage2.monitor.minimum-disk-space=0B

When the used-space-filewalker would finish its job, it will update databases with a correct values and after that you may comment out this option in the config.yaml file if you added/uncommented it before or remove that command line option from your docker run command.
You may check the progress of the filewalkers in your logs: