Can I recover from this crash

Alexey · September 8, 2025, 7:19am

It has three containers (I do not remember the exact names):

Init
Setup
Main

You need to check logs of the main container, its name is usually storj.

EddieA · September 8, 2025, 7:33am

That’s the docker logs I posted in my original post.

Cheers.

EddieA · September 8, 2025, 8:46am

By increasing the size of the buffer, and being quick with the select all/copy, here’s the start of the docker logs:

Application Name:storjContainer ID:4352000b07d4feae57b5a8a7863f8e417ffbad80bda07295a4e3c0d4119eb809
2025-09-08 08:24:51.479452+00:00downloading storagenode-updater
2025-09-08 08:24:51.485120+00:00--2025-09-08 08:24:51--  https://version.storj.io/processes/storagenode-updater/minimum/url?os=linux&arch=amd64
2025-09-08 08:24:51.493596+00:00Resolving version.storj.io (version.storj.io)... 34.173.164.90
2025-09-08 08:24:51.549945+00:00Connecting to version.storj.io (version.storj.io)|34.173.164.90|:443... connected.
2025-09-08 08:24:51.765112+00:00HTTP request sent, awaiting response... 200 OK
2025-09-08 08:24:51.765193+00:00Length: 93 [text/plain]
2025-09-08 08:24:51.765215+00:00Saving to: 'STDOUT'
2025-09-08 08:24:51.765230+00:002025-09-08T08:24:51.765230844Z
2025-09-08 08:24:51.765330+00:000K                                                     100%  145M=0s
2025-09-08 08:24:51.765455+00:002025-09-08T08:24:51.765455207Z
2025-09-08 08:24:51.765490+00:002025-09-08 08:24:51 (145 MB/s) - written to stdout [93/93]
2025-09-08 08:24:51.765512+00:002025-09-08T08:24:51.765512521Z
2025-09-08 08:24:51.773615+00:00--2025-09-08 08:24:51--  https://github.com/storj/storj/releases/download/v1.135.5/storagenode-updater_linux_amd64.zip
2025-09-08 08:24:51.791343+00:00Resolving github.com (github.com)... 140.82.116.3
2025-09-08 08:24:51.822612+00:00Connecting to github.com (github.com)|140.82.116.3|:443... connected.
2025-09-08 08:24:52.142136+00:00HTTP request sent, awaiting response... 302 Found
2025-09-08 08:24:52.142211+00:00Location: https://release-assets.githubusercontent.com/github-production-release-asset/128089774/379c984a-b3b3-4828-8bd8-8348a791d1ef?sp=r&sv=2018-11-09&sr=b&spr=https&se=2025-09-08T09%3A02%3A24Z&rscd=attachment%3B+filename%3Dstoragenode-updater_linux_amd64.zip&rsct=application%2Foctet-stream&skoid=96c2d410-5711-43a1-aedd-ab1947aa7ab0&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skt=2025-09-08T08%3A01%3A55Z&ske=2025-09-08T09%3A02%3A24Z&sks=b&skv=2018-11-09&sig=IOQrO13c5Z4m4JWxbl2TijIm0PhrpAekZMBTcErJfW8%3D&jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmVsZWFzZS1hc3NldHMuZ2l0aHVidXNlcmNvbnRlbnQuY29tIiwia2V5Ijoia2V5MSIsImV4cCI6MTc1NzMyMDE5MiwibmJmIjoxNzU3MzE5ODkyLCJwYXRoIjoicmVsZWFzZWFzc2V0cHJvZHVjdGlvbi5ibG9iLmNvcmUud2luZG93cy5uZXQifQ.RHENXiCFecGiysVpWfSnnUyMQ77AC5QnjmNKOODCU5E&response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_amd64.zip&response-content-type=application%2Foctet-stream [following]
2025-09-08 08:24:52.142384+00:00--2025-09-08 08:24:52--  https://release-assets.githubusercontent.com/github-production-release-asset/128089774/379c984a-b3b3-4828-8bd8-8348a791d1ef?sp=r&sv=2018-11-09&sr=b&spr=https&se=2025-09-08T09%3A02%3A24Z&rscd=attachment%3B+filename%3Dstoragenode-updater_linux_amd64.zip&rsct=application%2Foctet-stream&skoid=96c2d410-5711-43a1-aedd-ab1947aa7ab0&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skt=2025-09-08T08%3A01%3A55Z&ske=2025-09-08T09%3A02%3A24Z&sks=b&skv=2018-11-09&sig=IOQrO13c5Z4m4JWxbl2TijIm0PhrpAekZMBTcErJfW8%3D&jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmVsZWFzZS1hc3NldHMuZ2l0aHVidXNlcmNvbnRlbnQuY29tIiwia2V5Ijoia2V5MSIsImV4cCI6MTc1NzMyMDE5MiwibmJmIjoxNzU3MzE5ODkyLCJwYXRoIjoicmVsZWFzZWFzc2V0cHJvZHVjdGlvbi5ibG9iLmNvcmUud2luZG93cy5uZXQifQ.RHENXiCFecGiysVpWfSnnUyMQ77AC5QnjmNKOODCU5E&response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_amd64.zip&response-content-type=application%2Foctet-stream
2025-09-08 08:24:52.149599+00:00Resolving release-assets.githubusercontent.com (release-assets.githubusercontent.com)... 185.199.109.133, 185.199.108.133, 185.199.111.133, ...
2025-09-08 08:24:52.153747+00:00Connecting to release-assets.githubusercontent.com (release-assets.githubusercontent.com)|185.199.109.133|:443... connected.
2025-09-08 08:24:52.172066+00:00HTTP request sent, awaiting response... 200 OK
2025-09-08 08:24:52.172246+00:00Length: 17701456 (17M) [application/octet-stream]
2025-09-08 08:24:52.172320+00:00Saving to: '/tmp/storagenode-updater.zip'
2025-09-08 08:24:52.172345+00:002025-09-08T08:24:52.172345675Z
2025-09-08 08:24:52.175998+00:000K .......... .......... .......... .......... ..........  0% 12.8M 1s
2025-09-08 08:24:52.177868+00:0050K .......... .......... .......... .......... ..........  0% 25.7M 1s
2025-09-08 08:24:52.178727+00:00100K .......... .......... .......... .......... ..........  0% 56.0M 1s
2025-09-08 08:24:52.180295+00:00150K .......... .......... .......... .......... ..........  1% 30.9M 1s
2025-09-08 08:24:52.180826+00:00200K .......... .......... .......... .......... ..........  1% 90.3M 1s
2025-09-08 08:24:52.181867+00:00250K .......... .......... .......... .......... ..........  1% 47.6M 1s

--Snip snip snip --

2025-09-08 08:24:52.347407+00:0017150K .......... .......... .......... .......... .......... 99%  111M 0s
2025-09-08 08:24:52.347776+00:0017200K .......... .......... .......... .......... .......... 99%  131M 0s
2025-09-08 08:24:52.348034+00:0017250K .......... .......... .......... ......               100%  140M=0.2s
2025-09-08 08:24:52.348048+00:002025-09-08T08:24:52.348048905Z
2025-09-08 08:24:52.348209+00:002025-09-08 08:24:52 (96.0 MB/s) - '/tmp/storagenode-updater.zip' saved [17701456/17701456]
2025-09-08 08:24:52.348222+00:002025-09-08T08:24:52.348222198Z
2025-09-08 08:24:52.849796+00:00downloading storagenode
2025-09-08 08:24:53.053919+00:00--2025-09-08 08:24:53--  https://version.storj.io/processes/storagenode/minimum/url?os=linux&arch=amd64
2025-09-08 08:24:53.062122+00:00Resolving version.storj.io (version.storj.io)... 34.173.164.90
2025-09-08 08:24:53.117449+00:00Connecting to version.storj.io (version.storj.io)|34.173.164.90|:443... connected.
2025-09-08 08:24:53.328665+00:00HTTP request sent, awaiting response... 200 OK
2025-09-08 08:24:53.328764+00:00Length: 85 [text/plain]
2025-09-08 08:24:53.328787+00:00Saving to: 'STDOUT'
2025-09-08 08:24:53.328804+00:002025-09-08T08:24:53.328804647Z
2025-09-08 08:24:53.328820+00:000K                                                       100% 89.9M=0s
2025-09-08 08:24:53.328888+00:002025-09-08T08:24:53.328888041Z
2025-09-08 08:24:53.328906+00:002025-09-08 08:24:53 (89.9 MB/s) - written to stdout [85/85]
2025-09-08 08:24:53.328922+00:002025-09-08T08:24:53.328922399Z
2025-09-08 08:24:53.334843+00:00--2025-09-08 08:24:53--  https://github.com/storj/storj/releases/download/v1.135.5/storagenode_linux_amd64.zip
2025-09-08 08:24:53.342884+00:00Resolving github.com (github.com)... 140.82.116.3
2025-09-08 08:24:53.373295+00:00Connecting to github.com (github.com)|140.82.116.3|:443... connected.
2025-09-08 08:24:53.674369+00:00HTTP request sent, awaiting response... 302 Found
2025-09-08 08:24:53.674569+00:00Location: https://release-assets.githubusercontent.com/github-production-release-asset/128089774/95705dce-949d-455b-9423-a43ce25ba18b?sp=r&sv=2018-11-09&sr=b&spr=https&se=2025-09-08T09%3A10%3A55Z&rscd=attachment%3B+filename%3Dstoragenode_linux_amd64.zip&rsct=application%2Foctet-stream&skoid=96c2d410-5711-43a1-aedd-ab1947aa7ab0&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skt=2025-09-08T08%3A10%3A48Z&ske=2025-09-08T09%3A10%3A55Z&sks=b&skv=2018-11-09&sig=EASRiRwZvr054eBLS%2BeC2pr6xm0AlDYUmGKOoEYD%2Bkg%3D&jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmVsZWFzZS1hc3NldHMuZ2l0aHVidXNlcmNvbnRlbnQuY29tIiwia2V5Ijoia2V5MSIsImV4cCI6MTc1NzMyMDE5MywibmJmIjoxNzU3MzE5ODkzLCJwYXRoIjoicmVsZWFzZWFzc2V0cHJvZHVjdGlvbi5ibG9iLmNvcmUud2luZG93cy5uZXQifQ.cQEj08eAo1xPr3shfdbpNe8OJFdsnXQKuO-_S1_k5-w&response-content-disposition=attachment%3B%20filename%3Dstoragenode_linux_amd64.zip&response-content-type=application%2Foctet-stream [following]
2025-09-08 08:24:53.674840+00:00--2025-09-08 08:24:53--  https://release-assets.githubusercontent.com/github-production-release-asset/128089774/95705dce-949d-455b-9423-a43ce25ba18b?sp=r&sv=2018-11-09&sr=b&spr=https&se=2025-09-08T09%3A10%3A55Z&rscd=attachment%3B+filename%3Dstoragenode_linux_amd64.zip&rsct=application%2Foctet-stream&skoid=96c2d410-5711-43a1-aedd-ab1947aa7ab0&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skt=2025-09-08T08%3A10%3A48Z&ske=2025-09-08T09%3A10%3A55Z&sks=b&skv=2018-11-09&sig=EASRiRwZvr054eBLS%2BeC2pr6xm0AlDYUmGKOoEYD%2Bkg%3D&jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmVsZWFzZS1hc3NldHMuZ2l0aHVidXNlcmNvbnRlbnQuY29tIiwia2V5Ijoia2V5MSIsImV4cCI6MTc1NzMyMDE5MywibmJmIjoxNzU3MzE5ODkzLCJwYXRoIjoicmVsZWFzZWFzc2V0cHJvZHVjdGlvbi5ibG9iLmNvcmUud2luZG93cy5uZXQifQ.cQEj08eAo1xPr3shfdbpNe8OJFdsnXQKuO-_S1_k5-w&response-content-disposition=attachment%3B%20filename%3Dstoragenode_linux_amd64.zip&response-content-type=application%2Foctet-stream
2025-09-08 08:24:53.676067+00:00Resolving release-assets.githubusercontent.com (release-assets.githubusercontent.com)... 185.199.109.133, 185.199.108.133, 185.199.111.133, ...
2025-09-08 08:24:53.680276+00:00Connecting to release-assets.githubusercontent.com (release-assets.githubusercontent.com)|185.199.109.133|:443... connected.
2025-09-08 08:24:53.698327+00:00HTTP request sent, awaiting response... 200 OK
2025-09-08 08:24:53.698463+00:00Length: 39751643 (38M) [application/octet-stream]
2025-09-08 08:24:53.698504+00:00Saving to: '/tmp/storagenode.zip'
2025-09-08 08:24:53.698690+00:002025-09-08T08:24:53.698690637Z
2025-09-08 08:24:53.702736+00:000K .......... .......... .......... .......... ..........  0% 11.9M 3s
2025-09-08 08:24:53.704517+00:0050K .......... .......... .......... .......... ..........  0% 25.6M 2s
2025-09-08 08:24:53.706184+00:00100K .......... .......... .......... .......... ..........  0% 29.2M 2s
2025-09-08 08:24:53.707114+00:00150K .......... .......... .......... .......... ..........  0% 49.9M 2s
2025-09-08 08:24:53.708137+00:00200K .......... .......... .......... .......... ..........  0% 47.8M 1s
2025-09-08 08:24:53.709446+00:00250K .......... .......... .......... .......... ..........  0% 37.2M 1s
2025-09-08 08:24:53.710066+00:00300K .......... .......... .......... .......... ..........  0% 81.8M 1s
2025-09-08 08:24:53.710579+00:00350K .......... .......... .......... .......... ..........  1% 96.8M 1s

-- Snip snip snip --

2025-09-08 08:24:54.071530+00:0038700K .......... .......... .......... .......... .......... 99%  134M 0s
2025-09-08 08:24:54.072158+00:0038750K .......... .......... .......... .......... .......... 99% 79.6M 0s
2025-09-08 08:24:54.072465+00:0038800K .......... .........                                  100% 68.7M=0.4s
2025-09-08 08:24:54.072490+00:002025-09-08T08:24:54.072490485Z
2025-09-08 08:24:54.072524+00:002025-09-08 08:24:54 (101 MB/s) - '/tmp/storagenode.zip' saved [39751643/39751643]
2025-09-08 08:24:54.072538+00:002025-09-08T08:24:54.072538441Z
2025-09-08 08:24:54.691296+00:002025-09-08 08:24:54,691 INFO RPC interface 'supervisor' initialized
2025-09-08 08:24:54.691392+00:002025-09-08 08:24:54,691 INFO supervisord started with pid 8
2025-09-08 08:24:55.693387+00:002025-09-08 08:24:55,692 INFO spawned: 'processes-exit-eventlistener' with pid 66
2025-09-08 08:24:55.695034+00:002025-09-08 08:24:55,694 INFO spawned: 'storagenode' with pid 67
2025-09-08 08:24:55.696618+00:002025-09-08 08:24:55,696 INFO spawned: 'storagenode-updater' with pid 68
2025-09-08 08:24:55.746989+00:002025-09-08T08:24:55Z	INFO	Configuration loaded	{"Process": "storagenode-updater", "Location": "/app/config/config.yaml"}
2025-09-08 08:24:55.747065+00:002025-09-08T08:24:55Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "storage.allocated-bandwidth"}
2025-09-08 08:24:55.747104+00:002025-09-08T08:24:55Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "storage.allocated-disk-space"}
2025-09-08 08:24:55.747119+00:002025-09-08T08:24:55Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "operator.wallet-features"}
2025-09-08 08:24:55.747218+00:002025-09-08T08:24:55Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "contact.external-address"}
2025-09-08 08:24:55.747245+00:002025-09-08T08:24:55Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "server.address"}
2025-09-08 08:24:55.747254+00:002025-09-08T08:24:55Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "storage2.piece-scan-on-startup"}
2025-09-08 08:24:55.747274+00:002025-09-08T08:24:55Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "console.address"}
2025-09-08 08:24:55.747281+00:002025-09-08T08:24:55Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "pieces.enable-lazy-filewalker"}
2025-09-08 08:24:55.747288+00:002025-09-08T08:24:55Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "healthcheck.details"}
2025-09-08 08:24:55.747295+00:002025-09-08T08:24:55Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "healthcheck.enabled"}
2025-09-08 08:24:55.747302+00:002025-09-08T08:24:55Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "operator.email"}
2025-09-08 08:24:55.747309+00:002025-09-08T08:24:55Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "pieces.file-stat-cache"}
2025-09-08 08:24:55.747315+00:002025-09-08T08:24:55Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "server.private-address"}
2025-09-08 08:24:55.747321+00:002025-09-08T08:24:55Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "operator.wallet"}
2025-09-08 08:24:55.747328+00:002025-09-08T08:24:55Z	INFO	Anonymized tracing enabled	{"Process": "storagenode-updater"}
2025-09-08 08:24:55.749109+00:002025-09-08T08:24:55Z	INFO	Running on version	{"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.135.5"}
2025-09-08 08:24:55.749239+00:002025-09-08T08:24:55Z	INFO	Downloading versions.	{"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2025-09-08 08:24:55.965628+00:002025-09-08T08:24:55Z	INFO	Current binary version	{"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.135.5"}
2025-09-08 08:24:55.965673+00:002025-09-08T08:24:55Z	INFO	New version is being rolled out but hasn't made it to this node yet	{"Process": "storagenode-updater", "Service": "storagenode"}
2025-09-08 08:24:55.974682+00:002025-09-08T08:24:55Z	INFO	Current binary version	{"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.135.5"}
2025-09-08 08:24:55.974722+00:002025-09-08T08:24:55Z	INFO	New version is being rolled out but hasn't made it to this node yet	{"Process": "storagenode-updater", "Service": "storagenode-updater"}
2025-09-08 08:24:56.922551+00:00unexpected fault address 0x7f6d16be000c
2025-09-08 08:24:56.922607+00:00fatal error: fault
2025-09-08 08:24:56.922629+00:002025-09-08 08:24:56,922 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-09-08 08:24:56.922636+00:002025-09-08 08:24:56,922 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-09-08 08:24:56.922676+00:002025-09-08 08:24:56,922 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-09-08 08:24:56.924678+00:00[signal SIGBUS: bus error code=0x2 addr=0x7f6d16be000c pc=0x484003]
2025-09-08 08:24:56.924698+00:002025-09-08T08:24:56.924698519Z
2025-09-08 08:24:56.924702+00:00goroutine 1 gp=0xc000002380 m=16 mp=0xc006780008 [running]:
2025-09-08 08:24:56.924772+00:00runtime.throw({0x25f2008?, 0xc00114f2e8?})
2025-09-08 08:24:56.924789+00:00/usr/local/go/src/runtime/panic.go:1101 +0x48 fp=0xc00114f2a0 sp=0xc00114f270 pc=0x47aaa8
2025-09-08 08:24:56.924793+00:00runtime.sigpanic()
2025-09-08 08:24:56.924797+00:00/usr/local/go/src/runtime/signal_unix.go:922 +0x10a fp=0xc00114f300 sp=0xc00114f2a0 pc=0x47ca0a
2025-09-08 08:24:56.924800+00:00runtime.memmove()

Lots of stuff cut out

2025-09-08 08:24:56.927475+00:00github.com/dgraph-io/badger/v4.(*DB).monitorCache(0xc000ff4008, 0xc000b34cf0)
2025-09-08 08:24:56.927478+00:00/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.5.0/db.go:473 +0x15d fp=0xc000d0dfc0 sp=0xc000d0def8 pc=0x1bf07dd
2025-09-08 08:24:56.927481+00:00github.com/dgraph-io/badger/v4.Open.gowrap1()
2025-09-08 08:24:56.927484+00:00/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.5.0/db.go:315 +0x25 fp=0xc000d0dfe0 sp=0xc000d0dfc0 pc=0x1bf0065
2025-09-08 08:24:56.927487+00:00runtime.goexit({})
2025-09-08 08:24:56.927490+00:00/usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000d0dfe8 sp=0xc000d0dfe0 pc=0x483161
2025-09-08 08:24:56.927494+00:00created by github.com/dgraph-io/badger/v4.Open in goroutine 1
2025-09-08 08:24:56.927496+00:00/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.5.0/db.go:315 +0xc4d
2025-09-08 08:24:56.931655+00:002025-09-08 08:24:56,931 INFO exited: storagenode (exit status 2; expected)
2025-09-08 08:24:57.934553+00:002025-09-08 08:24:57,934 INFO spawned: 'storagenode' with pid 114
2025-09-08 08:24:57.934628+00:002025-09-08 08:24:57,934 WARN received SIGQUIT indicating exit request
2025-09-08 08:24:57.935049+00:002025-09-08 08:24:57,934 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2025-09-08 08:24:57.935139+00:002025-09-08T08:24:57Z	INFO	Got a signal from the OS: "terminated"	{"Process": "storagenode-updater"}
2025-09-08 08:24:57.936210+00:002025-09-08 08:24:57,936 INFO stopped: storagenode-updater (exit status 0)
2025-09-08 08:24:57.936789+00:002025-09-08 08:24:57,936 INFO stopped: storagenode (terminated by SIGTERM)
2025-09-08 08:24:57.937082+00:002025-09-08 08:24:57,936 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)
2025-09-08 08:24:58.281415+00:002025-09-08 08:24:58,281 INFO RPC interface 'supervisor' initialized
2025-09-08 08:24:58.281450+00:002025-09-08 08:24:58,281 INFO supervisord started with pid 8
2025-09-08 08:24:59.284804+00:002025-09-08 08:24:59,284 INFO spawned: 'processes-exit-eventlistener' with pid 14
2025-09-08 08:24:59.287399+00:002025-09-08 08:24:59,286 INFO spawned: 'storagenode' with pid 15
2025-09-08 08:24:59.289738+00:002025-09-08 08:24:59,289 INFO spawned: 'storagenode-updater' with pid 16
2025-09-08 08:24:59.308803+00:002025-09-08T08:24:59Z	INFO	Configuration loaded	{"Process": "storagenode-updater", "Location": "/app/config/config.yaml"}
2025-09-08 08:24:59.308849+00:002025-09-08T08:24:59Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "pieces.file-stat-cache"}
2025-09-08 08:24:59.308866+00:002025-09-08T08:24:59Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "healthcheck.enabled"}
2025-09-08 08:24:59.308875+00:002025-09-08T08:24:59Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "operator.email"}
2025-09-08 08:24:59.308882+00:002025-09-08T08:24:59Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "server.address"}
2025-09-08 08:24:59.308889+00:002025-09-08T08:24:59Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "operator.wallet-features"}
2025-09-08 08:24:59.308896+00:002025-09-08T08:24:59Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "storage.allocated-bandwidth"}
2025-09-08 08:24:59.308902+00:002025-09-08T08:24:59Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "pieces.enable-lazy-filewalker"}
2025-09-08 08:24:59.308909+00:002025-09-08T08:24:59Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "healthcheck.details"}
2025-09-08 08:24:59.308916+00:002025-09-08T08:24:59Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "server.private-address"}
2025-09-08 08:24:59.308922+00:002025-09-08T08:24:59Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "console.address"}
2025-09-08 08:24:59.308929+00:002025-09-08T08:24:59Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "storage.allocated-disk-space"}
2025-09-08 08:24:59.308935+00:002025-09-08T08:24:59Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "operator.wallet"}
2025-09-08 08:24:59.308942+00:002025-09-08T08:24:59Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "storage2.piece-scan-on-startup"}
2025-09-08 08:24:59.308949+00:002025-09-08T08:24:59Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "contact.external-address"}
2025-09-08 08:24:59.309101+00:002025-09-08T08:24:59Z	INFO	Anonymized tracing enabled	{"Process": "storagenode-updater"}
2025-09-08 08:24:59.310243+00:002025-09-08T08:24:59Z	INFO	Running on version	{"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.135.5"}
2025-09-08 08:24:59.310407+00:002025-09-08T08:24:59Z	INFO	Downloading versions.	{"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2025-09-08 08:24:59.526948+00:002025-09-08T08:24:59Z	INFO	Current binary version	{"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.135.5"}
2025-09-08 08:24:59.526993+00:002025-09-08T08:24:59Z	INFO	New version is being rolled out but hasn't made it to this node yet	{"Process": "storagenode-updater", "Service": "storagenode"}
2025-09-08 08:24:59.535972+00:002025-09-08T08:24:59Z	INFO	Current binary version	{"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.135.5"}
2025-09-08 08:24:59.536014+00:002025-09-08T08:24:59Z	INFO	New version is being rolled out but hasn't made it to this node yet	{"Process": "storagenode-updater", "Service": "storagenode-updater"}
2025-09-08 08:25:00.537423+00:002025-09-08 08:25:00,537 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-09-08 08:25:00.537552+00:002025-09-08 08:25:00,537 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-09-08 08:25:00.537723+00:002025-09-08 08:25:00,537 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-09-08 08:25:01.928646+00:00unexpected fault address 0x7f17019e000c
2025-09-08 08:25:01.928727+00:00fatal error: fault
2025-09-08 08:25:01.930758+00:00[signal SIGBUS: bus error code=0x2 addr=0x7f17019e000c pc=0x484003]
2025-09-08 08:25:01.930785+00:002025-09-08T08:25:01.930785180Z
2025-09-08 08:25:01.930791+00:00goroutine 1 gp=0xc000002380 m=15 mp=0xc001300008 [running]:
2025-09-08 08:25:01.930795+00:00runtime.throw({0x25f2008?, 0xc000f332e8?})
2025-09-08 08:25:01.930802+00:00/usr/local/go/src/runtime/panic.go:1101 +0x48 fp=0xc000f332a0 sp=0xc000f33270 pc=0x47aaa8
2025-09-08 08:25:01.930808+00:00runtime.sigpanic()
2025-09-08 08:25:01.930811+00:00/usr/local/go/src/runtime/signal_unix.go:922 +0x10a fp=0xc000f33300 sp=0xc000f332a0 pc=0x47ca0a
2025-09-08 08:25:01.930814+00:00runtime.memmove()
2025-09-08 08:25:01.930817+00:00/usr/local/go/src/runtime/memmove_amd64.s:117 +0xc3 fp=0xc000f33308 sp=0xc000f33300 pc=0x484003
2025-09-08 08:25:01.930820+00:00github.com/dgraph-io/ristretto/v2/z.(*mmapReader).Read(0xc000dd0a80, {0xc00133f000?, 0x1000, 0x800010000f333a0?})
2025-09-08 08:25:01.930823+00:00/go/pkg/mod/github.com/dgraph-io/ristretto/v2@v2.0.0/z/file.go:100 +0x76 fp=0xc000f33338 sp=0xc000f33308 pc=0x1bc5ad6
2025-09-08 08:25:01.930826+00:00bufio.(*Reader).Read(0xc000b3f800, {0xc00aecb146, 0x16, 0x4783b9?})
2025-09-08 08:25:01.930829+00:00/usr/local/go/src/bufio/bufio.go:245 +0x197 fp=0xc000f33370 sp=0xc000f33338 pc=0x5aa7d7
2025-09-08 08:25:01.930832+00:00github.com/dgraph-io/badger/v4.(*hashReader).Read(0xc00aeda1b0, {0xc00aecb146, 0xc000f333d8?, 0x16})
2025-09-08 08:25:01.930835+00:00/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.5.0/value.go:98 +0x2a fp=0xc000f333a0 sp=0xc000f33370 pc=0x1c2daaa
2025-09-08 08:25:01.930838+00:00io.ReadAtLeast({0x2b33b60, 0xc00aeda1b0}, {0xc00aecb140, 0x1c, 0x1c}, 0x1c)
2025-09-08 08:25:01.930841+00:00/usr/local/go/src/io/io.go:335 +0x91 fp=0xc000f333e8 sp=0xc000f333a0 pc=0x4ba8b1
2025-09-08 08:25:01.930844+00:00io.ReadFull(...)

Lots more stuff cut out

2025-09-08 08:25:01.936990+00:00/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.5.0/db.go:335 +0xe56
2025-09-08 08:25:01.940151+00:002025-09-08 08:25:01,940 INFO exited: storagenode (exit status 2; expected)
2025-09-08 08:25:02.945118+00:002025-09-08 08:25:02,944 INFO spawned: 'storagenode' with pid 62
2025-09-08 08:25:02.945276+00:002025-09-08 08:25:02,945 WARN received SIGQUIT indicating exit request
2025-09-08 08:25:02.946048+00:002025-09-08 08:25:02,945 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2025-09-08 08:25:02.946345+00:002025-09-08T08:25:02Z	INFO	Got a signal from the OS: "terminated"	{"Process": "storagenode-updater"}
2025-09-08 08:25:02.948946+00:002025-09-08 08:25:02,948 INFO stopped: storagenode-updater (exit status 0)
2025-09-08 08:25:05.952324+00:002025-09-08 08:25:05,951 INFO waiting for storagenode, processes-exit-eventlistener to die
2025-09-08 08:25:06.727177+00:00unexpected fault address 0x7fba009e000c
2025-09-08 08:25:06.727247+00:00fatal error: fault
2025-09-08 08:25:06.729304+00:00[signal SIGBUS: bus error code=0x2 addr=0x7fba009e000c pc=0x484003]
2025-09-08 08:25:06.729369+00:002025-09-08T08:25:06.729369406Z
2025-09-08 08:25:06.729394+00:00goroutine 1 gp=0xc000002380 m=15 mp=0xc001186008 [running]:
2025-09-08 08:25:06.729418+00:00runtime.throw({0x25f2008?, 0xc00102b2e8?})
2025-09-08 08:25:06.729494+00:00/usr/local/go/src/runtime/panic.go:1101 +0x48 fp=0xc00102b2a0 sp=0xc00102b270 pc=0x47aaa8
2025-09-08 08:25:06.729551+00:00runtime.sigpanic()
2025-09-08 08:25:06.729575+00:00/usr/local/go/src/runtime/signal_unix.go:922 +0x10a fp=0xc00102b300 sp=0xc00102b2a0 pc=0x47ca0a
2025-09-08 08:25:06.729594+00:00runtime.memmove()
2025-09-08 08:25:06.729616+00:00/usr/local/go/src/runtime/memmove_amd64.s:117 +0xc3 fp=0xc00102b308 sp=0xc00102b300 pc=0x484003
2025-09-08 08:25:06.729635+00:00github.com/dgraph-io/ristretto/v2/z.(*mmapReader).Read(0xc000dee6c0, {0xc0000b6000?, 0x1000, 0x80001000102b3a0?})
2025-09-08 08:25:06.729651+00:00/go/pkg/mod/github.com/dgraph-io/ristretto/v2@v2.0.0/z/file.go:100 +0x76 fp=0xc00102b338 sp=0xc00102b308 pc=0x1bc5ad6
2025-09-08 08:25:06.729668+00:00bufio.(*Reader).Read(0xc000b196e0, {0xc00aeb3126, 0x16, 0x4783b9?})
2025-09-08 08:25:06.729688+00:00/usr/local/go/src/bufio/bufio.go:245 +0x197 fp=0xc00102b370 sp=0xc00102b338 pc=0x5aa7d7
2025-09-08 08:25:06.729706+00:00github.com/dgraph-io/badger/v4.(*hashReader).Read(0xc00aec21b0, {0xc00aeb3126, 0xc00102b3d8?, 0x16})
2025-09-08 08:25:06.729723+00:00/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.5.0/value.go:98 +0x2a fp=0xc00102b3a0 sp=0xc00102b370 pc=0x1c2daaa
2025-09-08 08:25:06.729740+00:00io.ReadAtLeast({0x2b33b60, 0xc00aec21b0}, {0xc00aeb3120, 0x1c, 0x1c}, 0x1c)
2025-09-08 08:25:06.729758+00:00/usr/local/go/src/io/io.go:335 +0x91 fp=0xc00102b3e8 sp=0xc00102b3a0 pc=0x4ba8b1
2025-09-08 08:25:06.729775+00:00io.ReadFull(...)

I gave up here

Cheers.

Alexey · September 9, 2025, 3:29am

I see. Accordingly logs it downloads binaries every time. Do you have a badger cache enabled?
If so, could you please stop the app and remove the badger cache directory (it’s usually in storage/filestatcache), then start the app again?

EddieA · September 9, 2025, 6:54am

That looks like it did the trick. Obviously I’m seeing a gazillion warnings/errors about the data:

2025-09-09T04:40:21Z    WARN    ordersfilestore Corrupted order detected in orders file {"Process": "storagenode", "error": "ordersfile corrupt entry: ordersfile: read config/orders/unsent/unsent-orders-12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs-1757145600000000000.v1: input/output error", "errorVerbose": "ordersfile corrupt entry: ordersfile: read config/orders/unsent/unsent-orders-12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs-1757145600000000000.v1: input/output error\n\tstorj.io/storj/storagenode/orders/ordersfile.(*fileV1).gotoNextEntry:253\n\tstorj.io/storj/storagenode/orders/ordersfile.(*fileV1).ReadOne:164\n\tstorj.io/storj/storagenode/orders.(*FileStore).getUnsentInfoFromUnsentFile:303\n\tstorj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite.func1:257\n\tstorj.io/storj/storagenode/orders.walkFilenamesInPath:465\n\tstorj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite:233\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders:182\n\tstorj.io/storj/storagenode/orders.(*Service).Run.func1:131\n\tstorj.io/common/sync2.(*Cycle).Run:102\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).add.func1:130"}
2025-09-09T04:40:21Z    WARN    ordersfilestore Corrupted order detected in orders file {"Process": "storagenode", "error": "ordersfile corrupt entry: ordersfile: read config/orders/unsent/unsent-orders-12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs-1757145600000000000.v1: input/output error", "errorVerbose": "ordersfile corrupt entry: ordersfile: read config/orders/unsent/unsent-orders-12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs-1757145600000000000.v1: input/output error\n\tstorj.io/storj/storagenode/orders/ordersfile.(*fileV1).gotoNextEntry:253\n\tstorj.io/storj/storagenode/orders/ordersfile.(*fileV1).ReadOne:164\n\tstorj.io/storj/storagenode/orders.(*FileStore).getUnsentInfoFromUnsentFile:303\n\tstorj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite.func1:257\n\tstorj.io/storj/storagenode/orders.walkFilenamesInPath:465\n\tstorj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite:233\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders:182\n\tstorj.io/storj/storagenode/orders.(*Service).Run.func1:131\n\tstorj.io/common/sync2.(*Cycle).Run:102\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).add.func1:130"}

But at least I can start the node again.

Currently I stopped it again waiting for the scrub to finish, which appears to be correcting some files, but not as many as I’d like (so far).

Cheers.