All my nodes died. From the log:
ld-elf.so.1: /usr/local/bin/storagenode: Undefined symbol "__libc_start1@FBSD_1.7"
ld-elf.so.1: /usr/local/bin/storagenode: Undefined symbol "__libc_start1@FBSD_1.7"
ld-elf.so.1: /usr/local/bin/storagenode: Undefined symbol "__libc_start1@FBSD_1.7"
ld-elf.so.1: /usr/local/bin/storagenode: Undefined symbol "__libc_start1@FBSD_1.7"
ld-elf.so.1: /usr/local/bin/storagenode: Undefined symbol "__libc_start1@FBSD_1.7"
ld-elf.so.1: /usr/local/bin/storagenode: Undefined symbol "__libc_start1@FBSD_1.7"
ld-elf.so.1: /usr/local/bin/storagenode: Undefined symbol "__libc_start1@FBSD_1.7"
ld-elf.so.1: /usr/local/bin/storagenode: Undefined symbol "__libc_start1@FBSD_1.7"
ld-elf.so.1: /usr/local/bin/storagenode: Undefined symbol "__libc_start1@FBSD_1.7"
ld-elf.so.1: /usr/local/bin/storagenode: Undefined symbol "__libc_start1@FBSD_1.7"
file /usr/local/bin/storagenode
/usr/local/bin/storagenode: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /libexec/ld-elf.so.1, FreeBSD-style, Go BuildID=2egZI_drur3yOwR2jegB/xLRy2V_1XBW0aFK-fUs3/Uo_otUYmplW-QgsFuuo7/b4GjnYWqD6EohvS-7jqa, with debug_info, not stripped
Do you test on freebsd?
Last working I had seen is 1.142.7. So something broke in-between
@Alexey @bre @Knowledge @heunland @elek @littleskunk @moderators
(I still haven’t updated to using the official storagenode-updater, since it was broken for a while on freebsd, so I use a script to fetch the suggested version ignoring the cursor – hence all nodes died at once. I now reverted all nodes to 1.142.7 and stopped the updater)
Hold on. 148.3 It works on one of the machines, with the same different freebsd jail version.
Investigating.
I have brought this to the attention of our dev team, please update us if you have any other findings
2 Likes
It appears the difference in jails userland. Host is the same in both cases.
The one that works is
# freebsd-version
13.4-RELEASE-p4
Those that broke:
# freebsd-version
13.2-RELEASE-p11
In both cases host kernel is 13.3-RELEASE-p4.
The problem is, both are discontinued releases, and I can’t upgrade to even 13.4. I’m trying to upgrade jail to 13.5 now, with the understanding that it is an unsupported configuration.
I cannot upgrade host – it’s TrueNAS Core, 13.3 is final release.
I’ll need to think what to do about the whole thing.
I’ll report back if 13.5 userland worked, it’s upgrading now.
% sudo iocage upgrade -r 13.5-RELEASE storj
... one eternity later ...
storj successfully upgraded from 13.2-RELEASE-p11 to 13.5-RELEASE-p10!
% sudo iocage exec storj storagenode version
2026-03-04T11:06:09-08:00 INFO Configuration loaded {"process": "storagenode", "location": "/root/.local/share/storj/storagenode/config.yaml"}
2026-03-04T11:06:09-08:00 INFO Anonymized tracing enabled {"process": "storagenode"}
Release build
Version: v1.148.3
Build timestamp: 24 Feb 26 12:40 UTC
Git commit: afdab605689c20d5bbb1781d753deef6f69d3900
% sudo iocage exec storj tail -3 /var/log/storj.log
2026-03-04T11:06:28-08:00 INFO piecestore uploaded {"process": "storagenode", "piece_id": "WXPIFSAQD2XZT63TL265GJTVBNSJHDOZ2ONEBQ4CPOUU66TTALEQ", "satellite_id": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "action": "PUT", "remote_address": "79.127.226.97:37410", "size": 6144}
2026-03-04T11:06:28-08:00 DEBUG piecestore download started {"process": "storagenode", "piece_id": "AVD7MOCEXKAML4AFDCIMPWKQIYPE4ULD426KZ5AK7OIXSMHOYWVQ", "satellite_id": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "action": "GET", "offset": 0, "size": 768, "remote_address": "79.127.219.33:54856"}
2026-03-04T11:06:28-08:00 DEBUG piecestore download started {"process": "storagenode", "piece_id": "KZ75ZKEMKDUX3KHTICP3CYYTWFOALLOUA5QDXOGGPDUSH5MRNXLA", "satellite_id": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "action": "GET", "offset": 0, "size": 10496, "remote_address": "109.61.92.87:34718"}
Phew. 
Worth noting, the correct way for the TrueNAS Core users is to create a new jail from 13.3-RELEASE and migrate your node there. Using jail newer than the host is unsupported and undefined configuration. It may work, until the userland calls unimplemented function from the host kernel. So, use that as a stop-gap workaround.
Possible workaround:
Drop TrueNAS entirely, reinstall new version of FreeBSD; Enable ZFS; Import existing array.
Should be pretty interesting in your power testing as well: No need to run a webserver - might drop a watt or two .)
1 Like
Yes, this is definitely on my todo list. I just thought I have a few more years with the “appliance” software.
Thinking more about it, in the funny times we live, I can just and ai to write me ansible config for the NAS with the configuration I want. Does not need to to be general purpose. Just needs to work fr me.
Well, you just found me next project to tinker with, thank you very much 
2 Likes
….and as the story goes, the stars aligned, The Great Wizzard summoned the Mighty AI assistant (… not sure which one) and together they slayed the Forsaken BlackSluggishDragon.