Database: file is not a database, impossible to know which database is concerned

Hello,

Following a migration of one of my nodes too quickly, some data has obviously been corrupted. My node is running in a Docker container, on a Debian 10 host. When I run docker-compose up, I get the following result:

2021-07-05T17:35:27.838Z INFO Configuration loaded {"Location":"/app/config/config.yaml"}
2021-07-05T17:35:27.857Z INFO Operator email {"Address": "storj@<redacted>"}
2021-07-05T17:35:27.857Z INFO Operator wallet {"Address":"<redacted>"}
Error: Error starting master database on storagenode: database: file is not a database
	storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:323
	storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:305
	storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:281
	storj.io/storj/storagenode/storagenodedb.OpenExisting:248
	main.cmdRun:160
	storj.io/private/process.cleanup.func1.4:363
	storj.io/private/process.cleanup.func1:381
	github.com/spf13/cobra.(*Command).execute:852
	github.com/spf13/cobra.(*Command).executeC:960
	github.com/spf13/cobra.(*Command).execute:897
	storj.io/private/process.ExecWithCustomConfig:88
	storj.io/private/process.ExecCustomDebug:70
	main.main:385
	runtime.main:204

I spent the afternoon scouring the forum for other similar errors, but no solution worked, for the few problems that looked like mine. Among those I found in my search:

Summary

My biggest problem here is that I don’t even know what the “master” database is. I didn’t find it by reading the source code (storj/main.go at 68627e7d803f09390e691024c30a400b95bac73b · storj/storj · GitHub) or with a strace. storagenode doesn’t seem to attempt any database access before raising this error: (see full trace)

Summary
execve("./storagenode", ["./storagenode", "run", "--config-dir", "config", "--identity-dir", "identity", "--metrics.app-suffix=-alpha", "--metrics.interval=30m", "--contact.external-address=node-"..., "--operator.email=storj@<redacted>"..., "--operator.wallet=0x<redacted>"..., "--console.address=:14002", "--storage.allocated-disk-space=1"..., "--operator.wallet-features=zksyn"...], 0x7ffc59b606e8 /* 12 vars */) = 0
arch_prctl(ARCH_SET_FS, 0x1b60d20)      = 0
set_tid_address(0x1b61010)              = 202
brk(NULL)                               = 0x2dc4000
brk(0x2dc6000)                          = 0x2dc6000
mmap(0x2dc4000, 4096, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x2dc4000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9188926000
munmap(0x7f9188926000, 4096)            = 0
sched_getaffinity(0, 8192, [0, 1, 2, 3, 4, 5, 6, 7]) = 64
openat(AT_FDCWD, "/sys/kernel/mm/transparent_hugepage/hpage_pmd_size", O_RDONLY) = 3
read(3, "2097152\n", 20)                = 8
close(3)                                = 0
uname({sysname="Linux", nodename="afce3d7ecb56", ...}) = 0
mmap(NULL, 262144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f91888e7000
mmap(NULL, 131072, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f91888c7000
mmap(NULL, 1048576, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f91887c7000
mmap(NULL, 8388608, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9187fc7000
mmap(NULL, 67108864, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9183fc7000
mmap(NULL, 536870912, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9163fc7000
mmap(0xc000000000, 67108864, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc000000000
mmap(0xc000000000, 67108864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xc000000000
mmap(NULL, 33554432, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9161fc7000
mmap(NULL, 2165768, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9161db6000
mmap(0x7f91888c7000, 131072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f91888c7000
mmap(0x7f9188847000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f9188847000
mmap(0x7f91883cd000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f91883cd000
mmap(0x7f9185ff7000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f9185ff7000
mmap(0x7f9174147000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f9174147000
mmap(NULL, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9161cb6000
mmap(NULL, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9161ca6000
mmap(NULL, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9161c96000
rt_sigprocmask(SIG_SETMASK, NULL, [], 8) = 0
sigaltstack(NULL, {ss_sp=NULL, ss_flags=SS_DISABLE, ss_size=0}) = 0
sigaltstack({ss_sp=0xc000002000, ss_flags=0, ss_size=32768}, NULL) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
gettid()                                = 202
rt_sigaction(SIGHUP, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RT_1 RT_2], NULL, 8) = 0
rt_sigaction(SIGHUP, {sa_handler=0x4738c0, sa_mask=~[RTMIN RT_1 RT_2], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0xf78195}, NULL, 8) = 0
rt_sigaction(SIGINT, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGINT, {sa_handler=0x4738c0, sa_mask=~[RTMIN RT_1 RT_2], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0xf78195}, NULL, 8) = 0
rt_sigaction(SIGQUIT, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGQUIT, {sa_handler=0x4738c0, sa_mask=~[RTMIN RT_1 RT_2], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0xf78195}, NULL, 8) = 0
rt_sigaction(SIGILL, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGILL, {sa_handler=0x4738c0, sa_mask=~[RTMIN RT_1 RT_2], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0xf78195}, NULL, 8) = 0
rt_sigaction(SIGTRAP, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGTRAP, {sa_handler=0x4738c0, sa_mask=~[RTMIN RT_1 RT_2], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0xf78195}, NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, ~[], [], 8)   = 0
rt_sigaction(SIGABRT, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, ~[], [], 8)   = 0
rt_sigaction(SIGABRT, {sa_handler=0x4738c0, sa_mask=~[RTMIN RT_1 RT_2], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0xf78195}, NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigaction(SIGBUS, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGBUS, {sa_handler=0x4738c0, sa_mask=~[RTMIN RT_1 RT_2], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0xf78195}, NULL, 8) = 0
[...]
sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0xf78195}, NULL, 8) = 0
rt_sigaction(SIGRT_29, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGRT_29, {sa_handler=0x4738c0, sa_mask=~[RTMIN RT_1 RT_2], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0xf78195}, NULL, 8) = 0
rt_sigaction(SIGRT_30, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGRT_30, {sa_handler=0x4738c0, sa_mask=~[RTMIN RT_1 RT_2], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0xf78195}, NULL, 8) = 0
rt_sigaction(SIGRT_31, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGRT_31, {sa_handler=0x4738c0, sa_mask=~[RTMIN RT_1 RT_2], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0xf78195}, NULL, 8) = 0
rt_sigaction(SIGRT_32, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGRT_32, {sa_handler=0x4738c0, sa_mask=~[RTMIN RT_1 RT_2], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0xf78195}, NULL, 8) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9161c95000
rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1 RT_2], [], 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RT_1 RT_2], NULL, 8) = 0
mmap(NULL, 143360, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9161c72000
mprotect(0x7f9161c74000, 135168, PROT_READ|PROT_WRITE) = 0
rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1 RT_2], ~[KILL STOP RTMIN RT_1 RT_2], 8) = 0
clone(child_stack=0x7f9161c94ee8, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID|0x400000, parent_tid=[203], tls=0x7f9161c94f38, child_tidptr=0x1b61010) = 203
rt_sigprocmask(SIG_SETMASK, ~[KILL STOP RTMIN RT_1 RT_2], NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 0
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 0
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 0
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 0
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 0
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 0
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 140732910737608
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9161c95000
rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1 RT_2], [], 8) = 0
mmap(NULL, 143360, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9161c4f000
mprotect(0x7f9161c51000, 135168, PROT_READ|PROT_WRITE) = 0
rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1 RT_2], ~[KILL STOP RTMIN RT_1 RT_2], 8) = 0
clone(child_stack=0x7f9161c71ee8, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID|0x400000, parent_tid=[204], tls=0x7f9161c71f38, child_tidptr=0x1b61010) = 204
rt_sigprocmask(SIG_SETMASK, ~[KILL STOP RTMIN RT_1 RT_2], NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x1b2e7f8, FUTEX_WAKE_PRIVATE, 1) = 0
rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1 RT_2], [], 8) = 0
mmap(NULL, 143360, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9161bc9000
mprotect(0x7f9161bcb000, 135168, PROT_READ|PROT_WRITE) = 0
rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1 RT_2], ~[KILL STOP RTMIN RT_1 RT_2], 8) = 0
clone(child_stack=0x7f9161bebee8, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID|0x400000, parent_tid=[206], tls=0x7f9161bebf38, child_tidptr=0x1b61010) = 206
rt_sigprocmask(SIG_SETMASK, ~[KILL STOP RTMIN RT_1 RT_2], NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9161c95000
rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1 RT_2], [], 8) = 0
mmap(NULL, 143360, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9161ba6000
mprotect(0x7f9161ba8000, 135168, PROT_READ|PROT_WRITE) = 0
rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1 RT_2], ~[KILL STOP RTMIN RT_1 RT_2], 8) = 0
clone(child_stack=0x7f9161bc8ee8, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID|0x400000, parent_tid=[207], tls=0x7f9161bc8f38, child_tidptr=0x1b61010) = 207
rt_sigprocmask(SIG_SETMASK, ~[KILL STOP RTMIN RT_1 RT_2], NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
readlinkat(AT_FDCWD, "/proc/self/exe", "/app/storagenode", 128) = 16
fcntl(0, F_GETFL)                       = 0x8002 (flags O_RDWR|O_LARGEFILE)
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
fcntl(1, F_GETFL)                       = 0x8001 (flags O_WRONLY|O_LARGEFILE)
fcntl(2, F_GETFL)                       = 0x8001 (flags O_WRONLY|O_LARGEFILE)
ioctl(1, TCGETS, 0xc000068ebc)          = -1 ENOTTY (Not a tty)
epoll_create1(EPOLL_CLOEXEC)            = 3
pipe2([4, 5], O_NONBLOCK|O_CLOEXEC)     = 0
epoll_ctl(3, EPOLL_CTL_ADD, 4, {EPOLLIN, {u32=28709920, u64=28709920}}) = 0
getrandom("\xee\x10\x9a\xa8\x2d\xff\x53\x1f\x0e\x3b\x5d\xff\xef\x96\xa0\x16", 16, 0) = 16
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x1b30048, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x1b30048, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x1b30048, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x1b30048, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x1b30048, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x1b30048, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x1b30048, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x1b30048, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
[...]
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x1b30048, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x1b30048, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
getrandom("\x5f\xb3\xe0\x55\x14\x2d\x29\x46", 8, 0) = 8
mmap(NULL, 262144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9161b66000
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 25
newfstatat(AT_FDCWD, "config/config.yaml", {st_mode=S_IFREG|0777, st_size=793, ...}, 0) = 0
openat(AT_FDCWD, "/etc/localtime", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "config/config.yaml", O_RDONLY|O_CLOEXEC) = 6
epoll_ctl(3, EPOLL_CTL_ADD, 6, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=1639500056, u64=140262386485528}}) = -1 EPERM (Operation not permitted)
epoll_ctl(3, EPOLL_CTL_DEL, 6, 0xc00035e9dc) = -1 EPERM (Operation not permitted)
fstat(6, {st_mode=S_IFREG|0777, st_size=793, ...}) = 0
read(6, "# path to the certificate chain "..., 1305) = 793
read(6, "", 512)                        = 0
close(6)                                = 0
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x1b62638, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000128148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000128148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
epoll_pwait(3, [], 128, 0, NULL, 1)     = 0
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
getpid()                                = 202
tgkill(202, 204, SIGURG)                = 0
getpid()                                = 202
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 202
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 202
tgkill(202, 204, SIGURG)                = 0
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
sched_yield()                           = 0
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 0
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 0
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 0
futex(0x1b2e7f8, FUTEX_WAKE_PRIVATE, 1) = 0
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 0
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 0
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 0
getpid()                                = 202
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 202
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 202
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 202
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 202
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 202
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 202
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 202
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 202
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 202
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 202
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 202
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 202
tgkill(202, 206, SIGURG)                = 0
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 0
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 0
getpid()                                = 202
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 202
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 202
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 202
tgkill(202, 206, SIGURG)                = 0
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 0
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 0
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 0
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 0
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 0
futex(0x1b2e910, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x1b2e7f8, FUTEX_WAKE_PRIVATE, 1) = 1
getpid()                                = 202
tgkill(202, 206, SIGURG)                = 0
futex(0x1b2e7f8, FUTEX_WAKE_PRIVATE, 1) = 1
epoll_pwait(3, [], 128, 0, NULL, 15578816309150) = 0
mmap(NULL, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9161b33000
mmap(NULL, 262144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9161af3000
futex(0xc00005ed48, FUTEX_WAKE_PRIVATE, 1) = 1
sched_yield()                           = 0
futex(0x1b47460, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x1b2e7f8, FUTEX_WAKE_PRIVATE, 1) = 1
mmap(NULL, 1439992, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9161913000
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=202, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 208
mmap(NULL, 262144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f91618d3000
newfstatat(AT_FDCWD, ".", {st_mode=S_IFDIR|0755, st_size=4096, ...}, 0) = 0
newfstatat(AT_FDCWD, "/app", {st_mode=S_IFDIR|0755, st_size=4096, ...}, 0) = 0
write(2, "2021-07-05T17:47:04.550Z\tINFO\tCo"..., 912021-07-05T17:47:04.550Z	INFO	Configuration loaded	{"Location": "/app/config/config.yaml"}
) = 91
uname({sysname="Linux", nodename="afce3d7ecb56", ...}) = 0
openat(AT_FDCWD, "identity/identity.cert", O_RDONLY|O_CLOEXEC) = 6
epoll_ctl(3, EPOLL_CTL_ADD, 6, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=1639500056, u64=140262386485528}}) = -1 EPERM (Operation not permitted)
epoll_ctl(3, EPOLL_CTL_DEL, 6, 0xc0005be964) = -1 EPERM (Operation not permitted)
fstat(6, {st_mode=S_IFREG|0644, st_size=1618, ...}) = 0
read(6, "-----BEGIN CERTIFICATE-----\nMIIB"..., 2130) = 1618
read(6, "", 512)                        = 0
close(6)                                = 0
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
rt_sigprocmask(SIG_SETMASK, ~[HUP INT QUIT ILL TRAP ABRT BUS FPE SEGV TERM STKFLT CHLD PROF SYS RTMIN RT_1], NULL, 8) = 0
futex(0xc000090148, FUTEX_WAKE_PRIVATE, 1) = 1
rt_sigprocmask(SIG_SETMASK, ~[HUP INT QUIT ILL TRAP ABRT BUS FPE SEGV TERM STKFLT CHLD PROF SYS RTMIN RT_1], NULL, 8) = 0
futex(0xc000128148, FUTEX_WAKE_PRIVATE, 1) = 1
rt_sigprocmask(SIG_SETMASK, ~[HUP INT QUIT ILL TRAP ABRT BUS FPE SEGV TERM STKFLT CHLD PROF SYS RTMIN RT_1], NULL, 8) = 0
futex(0xc000128148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc00005e948, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x1b30048, FUTEX_WAIT_PRIVATE, 0, NULL2021-07-05T17:47:04.575Z	INFO	Operator email	{"Address": "storj@<redacted>"}
2021-07-05T17:47:04.575Z	INFO	Operator wallet	{"Address": "0x<redacted>"}
Error: Error starting master database on storagenode: database: file is not a database
	storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:323
	storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:305
	storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:281
	storj.io/storj/storagenode/storagenodedb.OpenExisting:248
	main.cmdRun:160
	storj.io/private/process.cleanup.func1.4:363
	storj.io/private/process.cleanup.func1:381
	github.com/spf13/cobra.(*Command).execute:852
	github.com/spf13/cobra.(*Command).ExecuteC:960
	github.com/spf13/cobra.(*Command).Execute:897
	storj.io/private/process.ExecWithCustomConfig:88
	storj.io/private/process.ExecCustomDebug:70
	main.main:385
	runtime.main:204
) = ?
+++ exited with 1 +++

If it’s ever useful, I’ve overridden the entrypoint of the docker image to be able to execute exactly the original command prefixed with strace:

cd /app
strace ./storagenode run --config-dir config --identity-dir identity --metrics.app-suffix=-alpha --metrics.interval=30m --contact.external-address=<redacted> --operator.email=storj@<redacted> --operator.wallet=<redacted> --console.address=:14002 --storage.allocated-disk-space=1700GB --operator.wallet-features=zksync

Does anyone have a tip on how to fix my node’s databases?

Additional information:

storagenode version: 1.31.2
$ ./storagenode version
Release build
Version: v1.31.2
Build timestamp: 31 May 21 17:48 UTC
Git commit: 032bc29d77b27e471571a50c5769f38dfc8f8a15

Update trace, with --follow-forks option: https://pastebin.com/tT1Hu3YM
As orders.db is the latest on which stat operation is performed, I think the problem is there… but for the resolution, I don’t know what to do.

There is a more simple way - just check all databases. When you figure out which is irrecoverable corrupted, you can stop storagenode, delete corrupted database, backup all remained databases and start the storagenode, it will recreate all databases, but they would be empty, stop the storagenode and restore databases from backup with overwrite. As result - only missed will be recreated and you will lost only part of statistics.
See How to fix database: file is not a database error – Storj

1 Like

As a last resort, I copied the databases from one of my other nodes to this node. I’m well aware that this may cause problems in the future, but I couldn’t find anything better to make my node available again.

Ofc data is still there, and I can now see successful uploads and downloads. However, the upload speed seems to be much lower than before. I’ll have to see later if this is still the case. In any case, it is much better than a disqualification.

There is no disqualification for lost databases. You will only lost some statistics.

Thanks for your replies.

Is it better to keep my node as it is now, with the databases of a different node, or to start over with clean databases?

I meant, disqualification because node would have been offline for too long.

If I understand correctly, the local databases only store statistics, so replacing them with those of another node should not explain the drop in egress I see?

It’s up on you. What you prefer - wrong data or missed data? :slight_smile:

It’s stat from other storagenode, the current one could have a different egress and graph would be messy. After a month it would be more correct than now.

I will stay with wrong data :grinning_face_with_smiling_eyes:

Stats I was talking about are collected with Prometheus Exporter but I don’t know exactly how the nice value I see in my graph is calculated. I can’t do much more anyway.

Thanks a lot for your help and explanations!

1 Like

Doesn’t matter. The Prometheus exporter likely uses storagenode API to get an information, the storagenode API tooks it from the local database. The local database is from the other node, so the reported stat is wrong.