V1.119.15 on go 1.23.3 crashes with "fatal error: checkptr: converted pointer straddles multiple allocations"

I’ve been running storagenode on macOS for over a year now.
I recently updated the code and built with 1.23 and now after storage node runs for a while, it exits with:

fatal error: checkptr: converted pointer straddles multiple allocations

goroutine 29 gp=0xc000907dc0 m=28 mp=0xc000f60008 [running]:
runtime.throw({0x10466863e?, 0xc0001123f8?})
… (and so on)

I’ve tried rebooting, I’ve also tried a different host too. It seems to be related to going from 1.21 to 1.23 as I tried building earlier versions with 1.23 and got similar issues but 1.21 worked flawlessly.

Now that my node crashes multiple times a day, I’m not sure what to do as while I’m a programmer, I don’t have any experience in golang.

2 Likes

Hello @hchung,
Welcome to the forum!

I shared this post with the team.
Does docker version works for you?

Can you post the full stack trace?

Also which machine is it? ARM64 or AMD64?

Are you building with the latest Go 1.23.4 or some older version? (Run go version to see).

In general, yeah, MacOS should work with Go 1.23… this could be our bug, some dependency bug or a Go compiler bug. Many people in Storj are using macOS as their primary dev machine.

As a workaround, you can build storagenode with Go 1.21:

sh ./scripts/generate-gomod-for-storagenode-go1.20.sh go.mod go.1.20.mod
go build -mod ./go.1.20.mod ./cmd/storagenode

(We use that script to support older EOL Windows versions, until we deprecate support completely)

2 Likes

The issue turns out to be in gosigar. storagenode uses cloudfoundry/gosigar, and in a recent fork, they encountered the same issue:

Since this is used from telemetry, I disabled telemetry as an experiment by setting the metrics.addr to “” and storagenode hasn’t crashed since. It used to crash within a handful of hours and with telemetry disabled it hasn’t crashed with that error yet.

I did briefly try to see if I could build with the other library/version but I don’t know the details about how the build process works and got a dependency error. But since this seems to be working, I hope the team is able to make use of this finding.

3 Likes

I was using go 1.23.3, then tried 1.23.4, then posted here.
It’s arm64, Apple M2.

I tried to post the stack trace and the forum said something about only 2 links in a post.

Thanks. This information is excellent, I’ll try to herd the dependency fixes as necessary.

I’ll let you know when main branch is fixed. It’ll of course take some time to get to the release branch.

3 Likes