What is this error?

Node Operators troubleshooting
1

I don’t think I have seen this error before. What is this trying to tell me?

docker logs storagenode | grep trash
2024-12-21T09:44:40Z    INFO    pieces:trash    emptying trash started  {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
        /go/src/storj.io/storj/storagenode/pieces/trashchore.go:97 +0x228 fp=0x5100cc4 sp=0x5100c40 pc=0x1010450
        /go/src/storj.io/storj/storagenode/pieces/trashchore.go:64 +0x190 fp=0x5100e4c sp=0x5100da8 pc=0x101004c
        /go/src/storj.io/storj/storagenode/pieces/trashchore.go:86 +0x29c fp=0x5102fb8 sp=0x5102e24 pc=0x1010730

When I see this error I also see things like this in the log:

Error: Error opening database on storagenode: while opening memtables error: while opening fid: 46 error: while updating skiplist error: mremap size mismatch: requested: 95010 got: 134217728

And the node restarts. Not enough memory? Badger is on.

2

Never saw this before, passed to the team.

1 Like
3

@jammerdan Is your node on ARM hardware? From what I can see, this is a bug in Badger (or Ristretto) that manifests on ARM.

Do you get any stack trace with the “mremap size mismatch” error?

4

I see this error on an Odroid HC-2 and I think this is indeed an ARM device.

So far I have only grepped for “trash” and saw this as (part) of the output.
Is there any specific thing I have to do to get a stack trace? Like switching to debug level or any specific command?

5

No, it should print it when it got an exception, it look like a long tail of code paths.

6

Then there is no additional stacktrace. The error is showing as is.

7

This indicates that somewhere in your log file there is a stacktrace. You grepped for lines that referenced “trash”, and these lines had “trashchore.go” in them, but they are part of a broader context. Try passing -C 10 or something to see if there is more log line context. If there is, then you’ll want to figure out how to pull out the full stack trace to provide it here (pipe to less or something? export the Docker logs to a file?)

8

I see.
Then there is something more:

Error: Error opening database on storagenode: while opening memtables error: while opening fid: 86 error: while updating skiplist error: mremap size mismatch: requested: 7069 got: 134217728
	storj.io/storj/storagenode/storagenodedb.cachedBlobstore:214
	storj.io/storj/storagenode/storagenodedb.OpenExisting:233
	main.cmdRun:67
	main.newRunCmd.func1:33
	storj.io/common/process.cleanup.func1.4:392
	storj.io/common/process.cleanup.func1:410
	github.com/spf13/cobra.(*Command).execute:983
	github.com/spf13/cobra.(*Command).ExecuteC:1115
	github.com/spf13/cobra.(*Command).Execute:1039
	storj.io/common/process.ExecWithCustomOptions:112
	main.main:34
	runtime.main:271

Error: Error opening database on storagenode: During db.vlog.open error: while truncating last value log file: storage/filestatcache/001186.vlog error: mremap size mismatch: requested: 20 got: 2147483646
	storj.io/storj/storagenode/storagenodedb.cachedBlobstore:214
	storj.io/storj/storagenode/storagenodedb.OpenExisting:233
	main.cmdRun:67
	main.newRunCmd.func1:33
	storj.io/common/process.cleanup.func1.4:392
	storj.io/common/process.cleanup.func1:410
	github.com/spf13/cobra.(*Command).execute:983
	github.com/spf13/cobra.(*Command).ExecuteC:1115
	github.com/spf13/cobra.(*Command).Execute:1039
	storj.io/common/process.ExecWithCustomOptions:112
	main.main:34
	runtime.main:271

Error: Error opening database on storagenode: while opening memtables error: while opening fid: 87 error: while updating skiplist error: mremap size mismatch: requested: 20 got: 134217728
	storj.io/storj/storagenode/storagenodedb.cachedBlobstore:214
	storj.io/storj/storagenode/storagenodedb.OpenExisting:233
	main.cmdRun:67
	main.newRunCmd.func1:33
	storj.io/common/process.cleanup.func1.4:392
	storj.io/common/process.cleanup.func1:410
	github.com/spf13/cobra.(*Command).execute:983
	github.com/spf13/cobra.(*Command).ExecuteC:1115
	github.com/spf13/cobra.(*Command).Execute:1039
	storj.io/common/process.ExecWithCustomOptions:112
	main.main:34
	runtime.main:271

And then lots of these:

runtime: out of memory: cannot allocate 255852544-byte block (627449856 in use)
fatal error: out of memory

goroutine 850 gp=0x40a78c8 m=4 mp=0x3c71088 [running]:
runtime.throw({0x1dc7cf2, 0xd})
	/usr/local/go/src/runtime/panic.go:1023 +0x4c fp=0x3f2cb38 sp=0x3f2cb24 pc=0x56214
runtime.(*mcache).allocLarge(0xb6f94538, 0xf1104d5, 0x1)
	/usr/local/go/src/runtime/mcache.go:236 +0x1c4 fp=0x3f2cb64 sp=0x3f2cb38 pc=0x2b1e4
runtime.mallocgc(0xf1104d5, 0x19f3ed0, 0x1)
	/usr/local/go/src/runtime/malloc.go:1165 +0x708 fp=0x3f2cbac sp=0x3f2cb64 pc=0x216d4
runtime.makeslice(0x19f3ed0, 0xf1104d5, 0xf1104d5)
	/usr/local/go/src/runtime/slice.go:107 +0xa8 fp=0x3f2cbc0 sp=0x3f2cbac pc=0x73bf4
github.com/dgraph-io/ristretto/z.Calloc(...)
	/go/pkg/mod/github.com/dgraph-io/ristretto@v0.1.1/z/calloc_nojemalloc.go:18
github.com/dgraph-io/badger/v4/table.(*Table).decompress(0x3d1e690, 0x684c940)
	/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.0/table/table.go:810 +0x214 fp=0x3f2cc0c sp=0x3f2cbc0 pc=0x1660540
github.com/dgraph-io/badger/v4/table.(*Table).block(0x3d1e690, 0x1e4f, 0x1)
	/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.0/table/table.go:580 +0x394 fp=0x3f2ccc4 sp=0x3f2cc0c pc=0x165ee5c
github.com/dgraph-io/badger/v4/table.(*Iterator).seekHelper(0x4ab0850, 0x1e4f, {0x3e54820, 0x48, 0x48})
	/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.0/table/iterator.go:254 +0x4c fp=0x3f2cce4 sp=0x3f2ccc4 pc=0x165a8ec
github.com/dgraph-io/badger/v4/table.(*Iterator).seekFrom(0x4ab0850, {0x3e54820, 0x48, 0x48}, 0x0)
	/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.0/table/iterator.go:294 +0x12c fp=0x3f2cd28 sp=0x3f2cce4 pc=0x165ab14
github.com/dgraph-io/badger/v4/table.(*Iterator).seek(...)
	/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.0/table/iterator.go:310
github.com/dgraph-io/badger/v4/table.(*Iterator).Seek(0x4ab0850, {0x3e54820, 0x48, 0x48})
	/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.0/table/iterator.go:424 +0x58 fp=0x3f2cd40 sp=0x3f2cd28 pc=0x165b3a8
github.com/dgraph-io/badger/v4.(*levelHandler).get(0x3eea050, {0x3e54820, 0x48, 0x48})
	/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.0/level_handler.go:293 +0x238 fp=0x3f2cdb0 sp=0x3f2cd40 pc=0x16828e0
github.com/dgraph-io/badger/v4.(*levelsController).get(0x3eea000, {0x3e54820, 0x48, 0x48}, {0x0, 0x0, 0x0, {0x0, 0x0, 0x0}, ...}, ...)
	/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.0/levels.go:1610 +0x174 fp=0x3f2ce50 sp=0x3f2cdb0 pc=0x168eaf0
github.com/dgraph-io/badger/v4.(*DB).get(0x40cfb08, {0x3e54820, 0x48, 0x48})
	/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.0/db.go:785 +0x3a8 fp=0x3f2cf2c sp=0x3f2ce50 pc=0x166efa8
github.com/dgraph-io/badger/v4.(*Txn).Get(0x509ad20, {0x4c34c00, 0x40, 0x40})
	/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.0/txn.go:479 +0x2bc fp=0x3f2cfd4 sp=0x3f2cf2c pc=0x16a4e10
storj.io/storj/storagenode/blobstore/statcache.(*BadgerCache).Get(0x438e178, {0x22b8be0, 0x4af7bc0}, {0x4ee0500, 0x20, 0x20}, {0x4c34b40, 0x20, 0x40})
	/go/src/storj.io/storj/storagenode/blobstore/statcache/badger.go:43 +0x140 fp=0x3f2d034 sp=0x3f2cfd4 pc=0x16b420c
storj.io/storj/storagenode/blobstore/statcache.BlobInfo.Stat({{0x22b8dc0, 0x4cbc4b0}, {0x22b8aa0, 0x438e178}}, {0x22b8be0, 0x4af7bc0})
	/go/src/storj.io/storj/storagenode/blobstore/statcache/statcache.go:51 +0xc8 fp=0x3f2d0a8 sp=0x3f2d034 pc=0x16b5134
storj.io/storj/storagenode/blobstore/statcache.(*BlobInfo).Stat(0x4672500, {0x22b8be0, 0x4af7bc0})
	<autogenerated>:1 +0x5c fp=0x3f2d0d4 sp=0x3f2d0a8 pc=0x16b7bcc
storj.io/storj/storagenode/pieces.storedPieceAccess.ModTime({{0x22b8e20, 0x4672500}, {0xff, 0x4, 0x14, 0x76, 0xdb, 0xd7, 0xe5, 0x55, ...}, ...}, ...)
	/go/src/storj.io/storj/storagenode/pieces/store.go:950 +0x40 fp=0x3f2d108 sp=0x3f2d0d4 pc=0x100fe48
storj.io/storj/storagenode/pieces.(*storedPieceAccess).ModTime(0x4cbc4e0, {0x22b8be0, 0x4af7bc0})
	<autogenerated>:1 +0x6c fp=0x3f2d1a4 sp=0x3f2d108 pc=0x1016f74
storj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePiecesToTrash.func3({0x22be578, 0x4cbc4e0})
	/go/src/storj.io/storj/storagenode/pieces/filewalker.go:334 +0x378 fp=0x3f2d364 sp=0x3f2d1a4 pc=0xffa398
storj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces.func1({0x22b8e20, 0x4672500})
	/go/src/storj.io/storj/storagenode/pieces/filewalker.go:70 +0xe8 fp=0x3f2d41c sp=0x3f2d364 pc=0xff7f0c
storj.io/storj/storagenode/blobstore/statcache.(*CachedStatBlobstore).WalkNamespace.func1({0x22b8dc0, 0x4cbc4b0})
	/go/src/storj.io/storj/storagenode/blobstore/statcache/statcache.go:87 +0x84 fp=0x3f2d444 sp=0x3f2d41c pc=0x16b5598
storj.io/storj/storagenode/blobstore/filestore.walkNamespaceWithPrefix({0x22b8be0, 0x4af7e00}, {0x4ee0500, 0x20, 0x20}, {0x3e55860, 0x42}, {0x5968202, 0x2}, 0x4afb9f0)
	/go/src/storj.io/storj/storagenode/blobstore/filestore/dir.go:1026 +0x338 fp=0x3f2d4b8 sp=0x3f2d444 pc=0xfd6254
storj.io/storj/storagenode/blobstore/filestore.(*Dir).walkNamespaceUnderPath(0x3d66210, {0x22b8be0, 0x4af7e00}, {0x4ee0500, 0x20, 0x20}, {0x3e55860, 0x42}, 0x4afb9c0, 0x4afb9f0)
	/go/src/storj.io/storj/storagenode/blobstore/filestore/dir.go:896 +0x30c fp=0x3f2d580 sp=0x3f2d4b8 pc=0xfd4ef0
storj.io/storj/storagenode/blobstore/filestore.(*Dir).walkNamespaceInPath(0x3d66210, {0x22b8be0, 0x4af7e00}, {0x4ee0500, 0x20, 0x20}, {0x430e580, 0xd}, 0x4afb9c0, 0x4afb9f0)
	/go/src/storj.io/storj/storagenode/blobstore/filestore/dir.go:867 +0x1cc fp=0x3f2d610 sp=0x3f2d580 pc=0xfd4894
storj.io/storj/storagenode/blobstore/filestore.(*Dir).WalkNamespace(0x3d66210, {0x22b8be0, 0x4af7da0}, {0x4ee0500, 0x20, 0x20}, 0x4afb9c0, 0x4afb9f0)
	/go/src/storj.io/storj/storagenode/blobstore/filestore/dir.go:860 +0x154 fp=0x3f2d690 sp=0x3f2d610 pc=0xfd437c
storj.io/storj/storagenode/blobstore/filestore.(*blobStore).WalkNamespace(0x40af1b8, {0x22b8be0, 0x4af7d40}, {0x4ee0500, 0x20, 0x20}, 0x4afb9c0, 0x4afb9f0)
	/go/src/storj.io/storj/storagenode/blobstore/filestore/store.go:352 +0x58 fp=0x3f2d6bc sp=0x3f2d690 pc=0xfdc580
storj.io/storj/storagenode/blobstore/statcache.(*CachedStatBlobstore).WalkNamespace(0x3f6e0c0, {0x22b8be0, 0x4af7d40}, {0x4ee0500, 0x20, 0x20}, 0x4afb9c0, 0x4afb9e0)
	/go/src/storj.io/storj/storagenode/blobstore/statcache/statcache.go:86 +0xa8 fp=0x3f2d6e8 sp=0x3f2d6bc pc=0x16b54e4
storj.io/storj/storagenode/pieces.(*BlobsUsageCache).WalkNamespace(0x3cb83c0, {0x22b8be0, 0x4af7d40}, {0x4ee0500, 0x20, 0x20}, 0x4afb9c0, 0x4afb9e0)
	<autogenerated>:1 +0x6c fp=0x3f2d714 sp=0x3f2d6e8 pc=0x101665c
storj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces(0x3cb83f0, {0x22b8be0, 0x4af7d40}, {0xa2, 0x8b, 0x4f, 0x4, 0xe1, 0xb, 0xae, ...}, ...)
	/go/src/storj.io/storj/storagenode/pieces/filewalker.go:58 +0x1c4 fp=0x3f2d7a8 sp=0x3f2d714 pc=0xff7c84
storj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePiecesToTrash(0x3cb83f0, {0x22b8be0, 0x4af7b60}, {0xa2, 0x8b, 0x4f, 0x4, 0xe1, 0xb, 0xae, ...}, ...)
	/go/src/storj.io/storj/storagenode/pieces/filewalker.go:301 +0x76c fp=0x3f2d9e4 sp=0x3f2d7a8 pc=0xff9d7c
storj.io/storj/storagenode/pieces.(*Store).WalkSatellitePiecesToTrash(0x3e6c3c0, {0x22b8be0, 0x4af7b60}, {0xa2, 0x8b, 0x4f, 0x4, 0xe1, 0xb, 0xae, ...}, ...)
	/go/src/storj.io/storj/storagenode/pieces/store.go:600 +0x3fc fp=0x3f2dae4 sp=0x3f2d9e4 pc=0x100b85c
storj.io/storj/storagenode/retain.(*Service).retainPieces(0x3f7a6e8, {0x22b7940, 0x4ecac30}, {{0x3c4e6e0, 0x4b}, {0xa2, 0x8b, 0x4f, 0x4, 0xe1, ...}, ...})
	/go/src/storj.io/storj/storagenode/retain/retain.go:401 +0x884 fp=0x3f2de5c sp=0x3f2dae4 pc=0x104c7e8
storj.io/storj/storagenode/retain.(*Service).Run.func3()
	/go/src/storj.io/storj/storagenode/retain/retain.go:286 +0x1b4 fp=0x3f2dfbc sp=0x3f2de5c pc=0x104b0e0
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.8.0/errgroup/errgroup.go:78 +0x60 fp=0x3f2dfec sp=0x3f2dfbc pc=0x58bccc
runtime.goexit({})
	/usr/local/go/src/runtime/asm_arm.s:859 +0x4 fp=0x3f2dfec sp=0x3f2dfec pc=0x977b0
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 809
	/go/pkg/mod/golang.org/x/sync@v0.8.0/errgroup/errgroup.go:75 +0xa0

goroutine 1 gp=0x3c02128 m=nil [semacquire]:
runtime.gopark(0x2059874, 0x34d4d68, 0x12, 0x5, 0x4)
	/usr/local/go/src/runtime/proc.go:402 +0x104 fp=0x3c54d9c sp=0x3c54d88 pc=0x59bd4
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:408
runtime.semacquire1(0x5008910, 0x0, 0x1, 0x0, 0x12)
	/usr/local/go/src/runtime/sema.go:160 +0x270 fp=0x3c54dc4 sp=0x3c54d9c pc=0x6f3d4
sync.runtime_Semacquire(0x5008910)
	/usr/local/go/src/runtime/sema.go:62 +0x3c fp=0x3c54ddc sp=0x3c54dc4 pc=0x93e84
sync.(*WaitGroup).Wait(0x5008908)
	/usr/local/go/src/sync/waitgroup.go:116 +0x68 fp=0x3c54df8 sp=0x3c54ddc pc=0xa5340
golang.org/x/sync/errgroup.(*Group).Wait(0x5008900)
	/go/pkg/mod/golang.org/x/sync@v0.8.0/errgroup/errgroup.go:56 +0x24 fp=0x3c54e04 sp=0x3c54df8 pc=0x58bb60
storj.io/storj/storagenode.(*Peer).Run(0x3ea2c08, {0x22b7940, 0x50088d0})
	/go/src/storj.io/storj/storagenode/peer.go:1032 +0x354 fp=0x3c54ec4 sp=0x3c54e04 pc=0x1782ea8
main.cmdRun(0x40a5088, 0x4119888)
	/go/src/storj.io/storj/cmd/storagenode/cmd_run.go:127 +0xc50 fp=0x3c556f8 sp=0x3c54ec4 pc=0x17bd248
main.newRunCmd.func1(0x40a5088, {0x4043b28, 0x0, 0x1a})
	/go/src/storj.io/storj/cmd/storagenode/cmd_run.go:33 +0x24 fp=0x3c5570c sp=0x3c556f8 pc=0x17bc5d8
storj.io/common/process.cleanup.func1.4({0x22b8be0, 0x405d740})
	/go/pkg/mod/storj.io/common@v0.0.0-20241107064117-6b3c20bcf19b/process/exec_conf.go:392 +0x11c fp=0x3c55750 sp=0x3c5570c pc=0xd5d574
storj.io/common/process.cleanup.func1(0x40a5088, {0x4043b28, 0x0, 0x1a})
	/go/pkg/mod/storj.io/common@v0.0.0-20241107064117-6b3c20bcf19b/process/exec_conf.go:410 +0x1920 fp=0x3c55de0 sp=0x3c55750 pc=0xd5cbcc
github.com/spf13/cobra.(*Command).execute(0x40a5088, {0x4043a48, 0x1a, 0x1b})
	/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:983 +0xa08 fp=0x3c55ea0 sp=0x3c55de0 pc=0x231638
github.com/spf13/cobra.(*Command).ExecuteC(0x40a4908)
	/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115 +0x44c fp=0x3c55f18 sp=0x3c55ea0 pc=0x231ee8
github.com/spf13/cobra.(*Command).Execute(...)
	/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039
storj.io/common/process.ExecWithCustomOptions(0x40a4908, {0x0, 0x1, 0x1, 0x0, 0x20595a8, 0x3e34330})
	/go/pkg/mod/storj.io/common@v0.0.0-20241107064117-6b3c20bcf19b/process/exec_conf.go:112 +0x1dc fp=0x3c55f40 sp=0x3c55f18 pc=0xd5a1b0
main.main()
	/go/src/storj.io/storj/cmd/storagenode/main.go:34 +0x304 fp=0x3c55fa8 sp=0x3c55f40 pc=0x17bf0c4
runtime.main()
	/usr/local/go/src/runtime/proc.go:271 +0x2fc fp=0x3c55fec sp=0x3c55fa8 pc=0x59628
runtime.goexit({})
	/usr/local/go/src/runtime/asm_arm.s:859 +0x4 fp=0x3c55fec sp=0x3c55fec pc=0x977b0

goroutine 2 gp=0x3c027e8 m=nil [force gc (idle)]:
runtime.gopark(0x2059874, 0x34bcb88, 0x11, 0xa, 0x1)
	/usr/local/go/src/runtime/proc.go:402 +0x104 fp=0x3c6cfd4 sp=0x3c6cfc0 pc=0x59bd4
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:408
runtime.forcegchelper()
	/usr/local/go/src/runtime/proc.go:326 +0xe4 fp=0x3c6cfec sp=0x3c6cfd4 pc=0x599f4
runtime.goexit({})
	/usr/local/go/src/runtime/asm_arm.s:859 +0x4 fp=0x3c6cfec sp=0x3c6cfec pc=0x977b0
created by runtime.init.5 in goroutine 1
	/usr/local/go/src/runtime/proc.go:314 +0x1c

goroutine 18 gp=0x3ca4248 m=nil [GC sweep wait]:
runtime.gopark(0x2059874, 0x34bfa90, 0xc, 0x9, 0x1)
	/usr/local/go/src/runtime/proc.go:402 +0x104 fp=0x3c687c4 sp=0x3c687b0 pc=0x59bd4
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:408
runtime.bgsweep(0x3cb0000)
	/usr/local/go/src/runtime/mgcsweep.go:318 +0x11c fp=0x3c687e4 sp=0x3c687c4 pc=0x3fe2c
runtime.gcenable.gowrap1()
	/usr/local/go/src/runtime/mgc.go:203 +0x28 fp=0x3c687ec sp=0x3c687e4 pc=0x2ff4c
runtime.goexit({})
	/usr/local/go/src/runtime/asm_arm.s:859 +0x4 fp=0x3c687ec sp=0x3c687ec pc=0x977b0
created by runtime.gcenable in goroutine 1
	/usr/local/go/src/runtime/mgc.go:203 +0x74

goroutine 19 gp=0x3ca4368 m=nil [GC scavenge wait]:
runtime.gopark(0x2059874, 0x34c0a08, 0xd, 0xa, 0x2)
	/usr/local/go/src/runtime/proc.go:402 +0x104 fp=0x3c68fb4 sp=0x3c68fa0 pc=0x59bd4
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:408
runtime.(*scavengerState).park(0x34c0a08)
	/usr/local/go/src/runtime/mgcscavenge.go:425 +0x68 fp=0x3c68fc8 sp=0x3c68fb4 pc=0x3d14c
runtime.bgscavenge(0x3cb0000)
	/usr/local/go/src/runtime/mgcscavenge.go:658 +0x60 fp=0x3c68fe4 sp=0x3c68fc8 pc=0x3d884
runtime.gcenable.gowrap2()
	/usr/local/go/src/runtime/mgc.go:204 +0x28 fp=0x3c68fec sp=0x3c68fe4 pc=0x2fef8
runtime.goexit({})
	/usr/local/go/src/runtime/asm_arm.s:859 +0x4 fp=0x3c68fec sp=0x3c68fec pc=0x977b0
created by runtime.gcenable in goroutine 1
	/usr/local/go/src/runtime/mgc.go:204 +0xbc

goroutine 34 gp=0x3d06248 m=nil [finalizer wait]:
runtime.gopark(0x2059714, 0x380526c, 0x10, 0xa, 0x1)
	/usr/local/go/src/runtime/proc.go:402 +0x104 fp=0x3d3878c sp=0x3d38778 pc=0x59bd4
runtime.runfinq()
	/usr/local/go/src/runtime/mfinal.go:194 +0x110 fp=0x3d387ec sp=0x3d3878c pc=0x2ed38
runtime.goexit({})
	/usr/local/go/src/runtime/asm_arm.s:859 +0x4 fp=0x3d387ec sp=0x3d387ec pc=0x977b0
created by runtime.createfing in goroutine 1
	/usr/local/go/src/runtime/mfinal.go:164 +0x5c

goroutine 3 gp=0x3e239e8 m=nil [select]:
runtime.gopark(0x20598a8, 0x0, 0x9, 0x3, 0x1)
	/usr/local/go/src/runtime/proc.go:402 +0x104 fp=0x3c6d6f4 sp=0x3c6d6e0 pc=0x59bd4
runtime.selectgo(0x3c6d7c8, 0x3c6d7bc, 0x0, 0x0, 0x2, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0xaf0 fp=0x3c6d794 sp=0x3c6d6f4 pc=0x6e798
github.com/golang/glog.(*fileSink).flushDaemon(0x34c07c0)
	/go/pkg/mod/github.com/golang/glog@v1.2.2/glog_file.go:349 +0xcc fp=0x3c6d7e4 sp=0x3c6d794 pc=0x16399c8
github.com/golang/glog.init.1.gowrap1()
	/go/pkg/mod/github.com/golang/glog@v1.2.2/glog_file.go:164 +0x28 fp=0x3c6d7ec sp=0x3c6d7e4 pc=0x1638ab4
runtime.goexit({})
	/usr/local/go/src/runtime/asm_arm.s:859 +0x4 fp=0x3c6d7ec sp=0x3c6d7ec pc=0x977b0
created by github.com/golang/glog.init.1 in goroutine 1
	/go/pkg/mod/github.com/golang/glog@v1.2.2/glog_file.go:164 +0x160

goroutine 4 gp=0x3e23b08 m=nil [select]:
runtime.gopark(0x20598a8, 0x0, 0x9, 0x3, 0x1)
	/usr/local/go/src/runtime/proc.go:402 +0x104 fp=0x3c6def4 sp=0x3c6dee0 pc=0x59bd4
runtime.selectgo(0x3c6dfc4, 0x3c6dfb8, 0x0, 0x0, 0x3, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0xaf0 fp=0x3c6df94 sp=0x3c6def4 pc=0x6e798
go.opencensus.io/stats/view.(*worker).start(0x3e6db00)
	/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0xbc fp=0x3c6dfe4 sp=0x3c6df94 pc=0xbfdf98
go.opencensus.io/stats/view.init.0.gowrap1()
	/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x28 fp=0x3c6dfec sp=0x3c6dfe4 pc=0xbfd374
runtime.goexit({})
	/usr/local/go/src/runtime/asm_arm.s:859 +0x4 fp=0x3c6dfec sp=0x3c6dfec pc=0x977b0
created by go.opencensus.io/stats/view.init.0 in goroutine 1
	/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0xa0

goroutine 5 gp=0x40a7328 m=nil [GC worker (idle)]:
runtime.gopark(0x2059728, 0x4008f18, 0x1a, 0xa, 0x0)
	/usr/local/go/src/runtime/proc.go:402 +0x104 fp=0x3c6e790 sp=0x3c6e77c pc=0x59bd4
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xec fp=0x3c6e7ec sp=0x3c6e790 pc=0x32b74
runtime.goexit({})
	/usr/local/go/src/runtime/asm_arm.s:859 +0x4 fp=0x3c6e7ec sp=0x3c6e7ec pc=0x977b0
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x20

goroutine 6 gp=0x40a7448 m=nil [GC worker (idle)]:
runtime.gopark(0x2059728, 0x3cc6000, 0x1a, 0xa, 0x0)
	/usr/local/go/src/runtime/proc.go:402 +0x104 fp=0x3c6ef90 sp=0x3c6ef7c pc=0x59bd4
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xec fp=0x3c6efec sp=0x3c6ef90 pc=0x32b74
runtime.goexit({})
	/usr/local/go/src/runtime/asm_arm.s:859 +0x4 fp=0x3c6efec sp=0x3c6efec pc=0x977b0
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x20

goroutine 50 gp=0x4100008 m=nil [GC worker (idle)]:
runtime.gopark(0x2059728, 0x3d4c048, 0x1a, 0xa, 0x0)
	/usr/local/go/src/runtime/proc.go:402 +0x104 fp=0x3d34790 sp=0x3d3477c pc=0x59bd4
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xec fp=0x3d347ec sp=0x3d34790 pc=0x32b74
runtime.goexit({})
	/usr/local/go/src/runtime/asm_arm.s:859 +0x4 fp=0x3d347ec sp=0x3d347ec pc=0x977b0
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x20

goroutine 20 gp=0x3ca46c8 m=nil [GC worker (idle)]:
runtime.gopark(0x2059728, 0x3d4c060, 0x1a, 0xa, 0x0)
	/usr/local/go/src/runtime/proc.go:402 +0x104 fp=0x3c69790 sp=0x3c6977c pc=0x59bd4
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xec fp=0x3c697ec sp=0x3c69790 pc=0x32b74
runtime.goexit({})
	/usr/local/go/src/runtime/asm_arm.s:859 +0x4 fp=0x3c697ec sp=0x3c697ec pc=0x977b0
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x20

goroutine 35 gp=0x3d06368 m=nil [GC worker (idle)]:
runtime.gopark(0x2059728, 0x3d4c078, 0x1a, 0xa, 0x0)
	/usr/local/go/src/runtime/proc.go:402 +0x104 fp=0x3c6c790 sp=0x3c6c77c pc=0x59bd4
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xec fp=0x3c6c7ec sp=0x3c6c790 pc=0x32b74
runtime.goexit({})
	/usr/local/go/src/runtime/asm_arm.s:859 +0x4 fp=0x3c6c7ec sp=0x3c6c7ec pc=0x977b0
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x20

goroutine 36 gp=0x3d06488 m=nil [GC worker (idle)]:
runtime.gopark(0x2059728, 0x4008f30, 0x1a, 0xa, 0x0)
	/usr/local/go/src/runtime/proc.go:402 +0x104 fp=0x3d38f90 sp=0x3d38f7c pc=0x59bd4
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xec fp=0x3d38fec sp=0x3d38f90 pc=0x32b74
runtime.goexit({})
	/usr/local/go/src/runtime/asm_arm.s:859 +0x4 fp=0x3d38fec sp=0x3d38fec pc=0x977b0
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x20

goroutine 7 gp=0x40a7568 m=nil [GC worker (idle)]:
runtime.gopark(0x2059728, 0x4102000, 0x1a, 0xa, 0x0)
	/usr/local/go/src/runtime/proc.go:402 +0x104 fp=0x3c6f790 sp=0x3c6f77c pc=0x59bd4
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xec fp=0x3c6f7ec sp=0x3c6f790 pc=0x32b74
runtime.goexit({})
	/usr/local/go/src/runtime/asm_arm.s:859 +0x4 fp=0x3c6f7ec sp=0x3c6f7ec pc=0x977b0
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x20

goroutine 51 gp=0x4100128 m=nil [GC worker (idle)]:
runtime.gopark(0x2059728, 0x4102018, 0x1a, 0xa, 0x0)
	/usr/local/go/src/runtime/proc.go:402 +0x104 fp=0x3d34f90 sp=0x3d34f7c pc=0x59bd4
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xec fp=0x3d34fec sp=0x3d34f90 pc=0x32b74
runtime.goexit({})
	/usr/local/go/src/runtime/asm_arm.s:859 +0x4 fp=0x3d34fec sp=0x3d34fec pc=0x977b0
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x20

goroutine 8 gp=0x42e2fc8 m=nil [select]:
runtime.gopark(0x20598a8, 0x0, 0x9, 0x3, 0x1)
	/usr/local/go/src/runtime/proc.go:402 +0x104 fp=0x3c80d24 sp=0x3c80d10 pc=0x59bd4
runtime.selectgo(0x3c80f40, 0x3c80df0, 0x0, 0x0, 0x3, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0xaf0 fp=0x3c80dc4 sp=0x3c80d24 pc=0x6e798
storj.io/monkit-jaeger.(*ThriftCollector).Run(0x405db00, {0x22b7940, 0x3d660c0})
	/go/pkg/mod/storj.io/monkit-jaeger@v0.0.0-20240221095020-52b0792fa6cd/thrift.go:174 +0x2a0 fp=0x3c80fac sp=0x3c80dc4 pc=0xd54f60
storj.io/common/process.cleanup.func1.2()
	/go/pkg/mod/storj.io/common@v0.0.0-20241107064117-6b3c20bcf19b/process/exec_conf.go:351 +0x2c fp=0x3c80fbc sp=0x3c80fac pc=0xd5d800
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.8.0/errgroup/errgroup.go:78 +0x60 fp=0x3c80fec sp=0x3c80fbc pc=0x58bccc
runtime.goexit({})
	/usr/local/go/src/runtime/asm_arm.s:859 +0x4 fp=0x3c80fec sp=0x3c80fec pc=0x977b0
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1
	/go/pkg/mod/golang.org/x/sync@v0.8.0/errgroup/errgroup.go:75 +0xa0

goroutine 9 gp=0x42e30e8 m=nil [select, locked to thread]:
runtime.gopark(0x20598a8, 0x0, 0x9, 0x3, 0x1)
	/usr/local/go/src/runtime/proc.go:402 +0x104 fp=0x3d3bef8 sp=0x3d3bee4 pc=0x59bd4
runtime.selectgo(0x3d3bfd8, 0x3d3bfc0, 0x0, 0x0, 0x2, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0xaf0 fp=0x3d3bf98 sp=0x3d3bef8 pc=0x6e798
runtime.ensureSigM.func1()
	/usr/local/go/src/runtime/signal_unix.go:1034 +0x1b8 fp=0x3d3bfec sp=0x3d3bf98 pc=0x8e7b0
runtime.goexit({})
	/usr/local/go/src/runtime/asm_arm.s:859 +0x4 fp=0x3d3bfec sp=0x3d3bfec pc=0x977b0
created by runtime.ensureSigM in goroutine 1
	/usr/local/go/src/runtime/signal_unix.go:1017 +0xf8

goroutine 10 gp=0x42e3208 m=11 mp=0x411c008 [syscall]:
runtime.notetsleepg(0x3807b48, 0xffffffffffffffff)
	/usr/local/go/src/runtime/lock_futex.go:246 +0x24 fp=0x3d357c0 sp=0x3d357a8 pc=0x1feac
os/signal.signal_recv()
	/usr/local/go/src/runtime/sigqueue.go:152 +0x34 fp=0x3d357e0 sp=0x3d357c0 pc=0x9475c
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:23 +0x14 fp=0x3d357ec sp=0x3d357e0 pc=0x6d99e0
runtime.goexit({})
	/usr/local/go/src/runtime/asm_arm.s:859 +0x4 fp=0x3d357ec sp=0x3d357ec pc=0x977b0
created by os/signal.Notify.func1.1 in goroutine 1
	/usr/local/go/src/os/signal/signal.go:151 +0x28

So I guess the first line says it all:

runtime: out of memory: cannot allocate 255852544-byte block (627449856 in use)
fatal error: out of memory

So I guess I have to disable Badger on them?

9

weirdly I think I am running badger on one ARM node (oracle) without errors.

10

I have disabled Badger on this node for now.

11

@jammerdan And just to confirm, which config item did you disable, and did that make the error stop?

12

Filed at: "mremap size mismatch" when storagenode is run on ARM/ARM64 hosts · Issue #7310 · storj/storj · GitHub

1 Like
13

I removed --pieces.file-stat-cache="badger" from the run command.

I guess I could have also set it to --pieces.file-stat-cache="" ?

Without it I do not see the error anymore.

1 Like
14

Yes, this will also disable the badger cache.