Fatal error: unexpected signal during runtime execution

Node Operators troubleshooting
, ,
1

What has happened here? Why is the node not able to properly add a log entry and throws exceptions into the docker log?

fatal error: unexpected signal during runtime execution
fatal error: unexpected signal during runtime execution
fatal error: unexpected signal during runtime execution
[signal SIGBUS: bus error code=0x2 addr=0xffff96854000 pc=0xd36f84]

runtime stack:
runtime.throw({0xfca990, 0x2a})
	/usr/local/go/src/runtime/panic.go:1198 +0x54
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:719 +0x230

goroutine 432288 [syscall]:
runtime.cgocall(0xce7fa0, 0x4002ffe6d8)
	/usr/local/go/src/runtime/cgocall.go:156 +0x50 fp=0x4002ffe690 sp=0x4002ffe650 pc=0x4034b0
github.com/mattn/go-sqlite3._Cfunc__sqlite3_step_row_internal(0xffff96739878, 0x4003d49330, 0x4003d49338)
	_cgo_gotypes.go:367 +0x44 fp=0x4002ffe6d0 sp=0x4002ffe690 pc=0xc6a9f4
github.com/mattn/go-sqlite3.(*SQLiteStmt).execSync.func3(0x4000a7a6c0, 0x4003d49330, 0x4003d49338)
	/go/pkg/mod/github.com/mattn/go-sqlite3@v1.14.8/sqlite3.go:1999 +0x68 fp=0x4002ffe710 sp=0x4002ffe6d0 pc=0xc7d628
github.com/mattn/go-sqlite3.(*SQLiteStmt).execSync(0x4000a7a6c0, {0x4005200dc0, 0x4, 0x4})
...

It seems to me that docker has hang up. Restarted docker service solved the issue (on MacOS).

2

The one time I’ve ever seen this problem was when I had a faulty memory chip. I’m surprised you got that much, SIGBUS is usually pretty fatal—like, instant death with no means for recovery/logging.

Usually the right way to defend against this kind of problems is to have a watchdog.

3

Not sure if I get that right - a watchdog notifying about this issue, if it occurs again?

After some additional analysis, for me it looks more like not fully migrated software to the M1 apple chips, as mentioned e.g. here.

As this was extracted from the docker logs of the storagenode, I’m not sure if it is related to the storage node libraries (local/go/src/runtime... GitHub ... sqlite3) or to docker. My feeling is the first one… Hmm.

4

It seems I can solve this issue as well, as we’ve found a proper solution to avoid the issues here:

Btw. @Alexey : it might be helpful for other Mac M1 / docker users to have this information available while setting up a new node: in the MacOS tab here. You can also add, that the latest docker desktop version works well, except it uses limited resources of something like 2 CPUs (only).

6

Added task for the documentation team

1 Like