Release preparation v1.148

Andrii · February 24, 2026, 3:47pm

New release candidate v1.148 is already deployed on QA Satellite

Changelog

General

0ad33da go.mod: bump uplink
c80023e ci: bump to Go 1.25.7
595642c go.mod: remove go.mod stubs from web/*
2294c2b cmd/write-hashtbl: ignore collisions
876c257 shared/dbuilt/dbschema: fix failing TestMigratePostgres
0f737e1 shared/modular: delete modular.Service{}
00d027a ci: add check-zap-fields linter

Satellite

10116bd satellite/metainfo: increase InDelta tolerances for retention charge tests
3677834 satellite/overlay: disable stray node DQ in tests
78b4ee4 satellite/admin: add endpoint to list project members page
25593e7 satellite/{payments,web}: show min retention fee in estimated charges
13cad17 satellite/{satellitedb,console,admin}: reduce DB calls when listing project members
787c14a satellite/admin/ui: add members table to project details
abdfbd4 satellite/admin: include not only ‘owned’ projects on account details page
e06f5cd satellite/{console,web}: simplify object browser pagination
e51cba9 satellite/overlay: allow trusted nodes to bypass difficulty check
b00cfe7 web/satellite: create separate wasm/go.mod and isolate Dockerfile
76e81cd satellite/{console,web}: rework Object Mount page in the UI
4ed0cff satellite/{console,analytics,web}: remove CunoFS/ObjectMount beta form logic
77f14cf web/satellite: add credentials table to Object Mount page
d77118d satellite/console: Update singlewhitelabel config structure
dd8f793 satellite/{entitlements,metainfo}: update AccountLicense structure
16d8788 satellite/{analytics,web}: remove or fix un-triggerable Segment events
6a22ef1 satellite/{analytics,web}: track download prefix event
fc45ac9 satellite/payments/stripe: attempt payment with customer balance
06d9599 satellite/{payments,console}: add endpoint to return failed invoice
724a282 web/satellite: allow retrying of failed invoices
6db4c9e satellite/{console,web}: add support for general SSO providers
7bc82b9 satellite/satellitedb: added new method to atomically update user’s external ID
2bbef94 satellite/{console,web}: add conditional verify email step when linking Storj vs SSO account
e1a3625 satellite/metainfo: prevent CommitObject from ignoring ETag
5c44fe4 satellite/{stripe,db}: fix missing retention remainder db dependency
95df805 satellite/eventing: skip retrying user configuration errors
ec64d2e satellite/metabase: move CommitObject/CommitInlineObject under Adapter interface
1266eb6 satellite/admin: licenses support
5793480 satellite/payments/stripe: fix zap field names
0159c2b satellite/{console,metainfo,web}: allow placements without self-serve details in bucket creation
e04d486 satellite/admin/ui: licensing support
8c95937 satellite/admin: small improvements for the licensing implementation
b068544 satellite/metabase: reset outer-scope variables in Spanner transaction callbacks
bb7e4d9 web/satellite: enable compute for whitelabeled console
dd28384 satellite/{console,web}: do not return error if there’s no failed invoice
5af06ce satellite/repair: repair segment with some invalid EC settings
363d680 satellite/metabase: add callback for bulk object deletes
a1a277d satellite/admin: add create registration token functionality
f04e3b6 satellite/console: add registration token CAPTCHA bypass
060b542 satellite/satellitedb: add more limit and expiration columns to registration_tokens table
4178bf0 satellite/satellitedb: add a method to create reg token with limits and expiration
a5af3c9 satellite/admin: add extended limits and expiration for registration tokens
587acf4 satellite/admin/ui: remove unused code
4c37c00 web/satellite: remove low token balance banner
9f98894 web/satellite: refactor general Billing view component
ee8bd0c web/satellite: add bucket eventing configuration UI
04ac10c satellite/metainfo: add stricter validation for object metadata key length
ae9078c satellite/eventing: remove redundant prefix from eventkit event name
7735aa0 web/satellite/emails: fix logo responsiveness
40d4f29 satellite/admin: loosen pending deletion restriction
42d1261 satellite/{console,web}: fix small session management issues
f1133df web/satellite: remove gradient background from auth layout
9736bf5 satellite/console: unblock inactive member removal
dbd2476 satellite/satellitedb: Update DROP INDEX migration step
1229547 satellite/{console,metainfo}: record remainder charge on bulk object deletes
4dd740a satellite/metabase: refactor bulk object deletes
195bf7b satellite: extract retention remainder logic
a23c3d2 satellite/accounting: send remainder_bytes as part of tally event to BQ
8ce4964 satellite/satellitedb: fixed NULL short_name handling in Get paged members
1a60446 satellite/{web,console}: show Object Mount terms link on sign up page
a00db86 satellite/accounting: report remainder charges to eventkit
7d9112d satellite/{orders,metainfo}: emit inline order accounting event with public project ID
fce7d10 satellite/orders: emit order settlement accounting event with public project ID
7254e07 satellite/{accounting,satellitedb}: emit accounting event with public project ID

Storagenode

c8c38ea storagenode/hashstore: atomic file create error improvement
f7578ef storagenode/hashstore: error if tbl and no logs
acf18b5 storagenode/hashstore: sync directory after all tbl rewrites
a76748e storagenode: account for compaction reserved space in disk usage and stats
afdab60 storagenode/hashstore: fix removing hashtbls

Test

f67d2ea .claude/skills/test-with-spanner: prefer env variable over passing an argument
4fb3a68 testsuite/*: pass in GOPROXY
3bab028 testsuite/*: lower mtu for docker-compose tests

elek · February 25, 2026, 8:26am

This is likely the storagenode release what we will rollout (after some final testing in select)

Note: this is a must have fix, for avoid loosing hashtbl files. (In case you test versions between officially rolled out version + this version)

Will also be included in v1.147.6, and v1.146.10

Ottetal · February 25, 2026, 8:45am

Very interesting! Tell me, have you before skipped multiple versions? I always remember it as being a 2 week continuous rollout.

elek · February 25, 2026, 9:53am

Yes, it happened multiple times. Sometimes because no relevant changes in the release, sometimes because additional testing.

Ottetal · February 25, 2026, 12:19pm

Huh, to reiterate what I said earlier: Very interesting! Thank you for clearing that up, have a good day @elek

karsten · February 25, 2026, 4:49pm

Is there any news about the ‘official’ switch from piecestore to hashstore?

elek · February 25, 2026, 11:16pm

Next step is rolling out the active migration. Starting with low percent on SLC, and checking how does it work.

But first, we prefer to fix a few things (one thing is better cleanup of the empty folders + better monitor):
https://review.dev.storj.tools/c/storj/storj/+/19741

(patch available ==> not in this release)

we are also planning to deprecate “dedicated” style of free space calculation (was marked as EXPERIMENTAL), and start using the normal/default with select nodes. Just to make sure that we don’t have any more space calculation issues.

Final, forced switch can be expected after the active migration…

zip · February 26, 2026, 3:45am

Is this in any way related to the dedicated disk functionality? Because that is working flawlessly compared to free space calculation done by the node from the piecestore era - at least for me it was always way off and was only causing issues, even causing the nodes to crash.
Also when expanding/shrinking the partition/drive, you do not need to change the node config and restart the node with the dedicated disk. You can essentially manage the available space directly from the OS and without any downtime.
So if this is the thing you are planning to drop I would strongly suggest to reconsider.

Andrew · March 4, 2026, 2:20pm

Hello,
It appears that storagenode-updater running in Docker on Linux hasn’t updated our nodes for several months, and they are still on version 1.142.7.
Could you please confirm whether this behavior is expected?
Should we update the nodes manually, or should we wait for the updater to handle it?

Ottetal · March 4, 2026, 4:48pm

Yes, this is written higher in the thread.

This is likely the storagenode release what we will rollout

Versions from 1.142 and up have been released, but not rolled out

Ottetal · March 4, 2026, 7:55pm

@Andrew , what a great timing. My first node just upgraded to 1.148, which brings - wooooow - circles to indicate datapoints on the graph.

It also messed up reporting, but that happens all the time, so I’ll give it a few days, and it should be great again.

Vadim · March 4, 2026, 8:52pm

Is there a plan to improve compaction? today if you have at least one broken log, whole node compaction is broken. Is it possible to implement that at least some workaround that is just skip broken file? until it possible to find any better solution?

Alexey · March 5, 2026, 4:56am

A post was split to a new topic: WARN hashstore mismatched log tail

PieceKeeper · March 4, 2026, 10:35pm

Does an upgrade event not close the node correctly? I got these “unclean shutdown detected” right after the auto update restarted the node. This happened on both of the nodes that were updated so far. Never seen those before.

Alexey · March 5, 2026, 4:55am

The current workaround is to use a write-hashtbl without a --fast flag. Also the root cause needs to be fixed - a normally working filesystem shouldn’t corrupt files.

Alexey · March 5, 2026, 4:58am

hashstore fsck should check those logs for inconsistence and fix them.

PieceKeeper · March 5, 2026, 6:58am

I guess there’s some risk here if the auto-updater shuts a node down in a way that doesn’t allow a clean shutdown—so in-memory state is not saved to disk. So the point isn’t that hashstore fsck can now fix issues after the fact — it’s that the auto-updater shouldn’t cause those problems in the first place.

EddieA · March 5, 2026, 10:28pm

I didn’t see this on the one node of mine that upgraded, which then proceeded to throw errors and restart itself which did then show the unclean shutdown messages, followed by the new mismatched log tail warning:

2026-03-05T20:19:06Z    INFO    Got a signal from the OS: "interrupt"   {"Process": "storagenode"}
2026-03-05T20:19:07Z    INFO    Configuration loaded    {"process": "storagenode", "location": "/app/config/config.yaml"}
2026-03-05T20:19:07Z    INFO    Anonymized tracing enabled      {"process": "storagenode"}
2026-03-05T20:19:07Z    INFO    Operator email  {"process": "storagenode", "address": "xxx@zzz.net"}
2026-03-05T20:19:07Z    INFO    Operator wallet {"process": "storagenode", "address": "0xc43DBD0E344E3a75AA64D917962D06feFB5443Fc"}
2026-03-05T20:19:07Z    INFO    server  existing kernel support for server-side tcp fast open detected  {"process": "storagenode"}
2026-03-05T20:19:07Z    INFO    hashstore       loaded hint file        {"process": "storagenode", "satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "store": "s0", "path": "config/storage/hashstore/121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6/s0/meta/hint-0000000000000125", "writable": [], "largest": 502, "skip": false}
2026-03-05T20:19:07Z    INFO    hashstore       loaded hint file        {"process": "storagenode", "satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "store": "s1", "path": "config/storage/hashstore/121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6/s1/meta/hint-0000000000000174", "writable": [], "largest": 572, "skip": false}
2026-03-05T20:19:08Z    INFO    hashstore       hashstore opened successfully  {"process": "storagenode", "satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "open_time": "385.437307ms", "logs_skipped": 168, "logs_matched": 0, "logs_mismatched": 0}
2026-03-05T20:19:08Z    INFO    hashstore       loaded hint file        {"process": "storagenode", "satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "store": "s0", "path": "config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/meta/hint-000000000000008f", "writable": [], "largest": 7715, "skip": false}
2026-03-05T20:19:13Z    INFO    hashstore       loaded hint file        {"process": "storagenode", "satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "store": "s1", "path": "config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/meta/hint-0000000000000097", "writable": [], "largest": 9051, "skip": false}
2026-03-05T20:19:20Z    INFO    Got a signal from the OS: "terminated"  {"process": "storagenode"}
2026-03-05T20:19:21Z    INFO    hashstore       hashstore opened successfully  {"process": "storagenode", "satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "open_time": "13.777878383s", "logs_skipped": 7549, "logs_matched": 0, "logs_mismatched": 0}
2026-03-05T20:19:21Z    INFO    hashstore       loaded hint file        {"process": "storagenode", "satellite": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "store": "s0", "path": "config/storage/hashstore/12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs/s0/meta/hint-00000000000000dd", "writable": [], "largest": 625, "skip": false}
2026-03-05T20:19:22Z    INFO    hashstore       loaded hint file        {"process": "storagenode", "satellite": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "store": "s1", "path": "config/storage/hashstore/12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs/s1/meta/hint-00000000000000b1", "writable": [], "largest": 614, "skip": false}
2026-03-05T20:19:22Z    INFO    hashstore       hashstore opened successfully  {"process": "storagenode", "satellite": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "open_time": "699.942155ms", "logs_skipped": 327, "logs_matched": 0, "logs_mismatched": 0}
2026-03-05T20:19:22Z    INFO    hashstore       loaded hint file        {"process": "storagenode", "satellite": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "store": "s0", "path": "config/storage/hashstore/1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE/s0/meta/hint-00000000000000a0", "writable": [], "largest": 206, "skip": false}
2026-03-05T20:19:22Z    INFO    hashstore       loaded hint file        {"process": "storagenode", "satellite": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "store": "s1", "path": "config/storage/hashstore/1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE/s1/meta/hint-00000000000000d1", "writable": [], "largest": 303, "skip": false}
2026-03-05T20:19:22Z    INFO    hashstore       hashstore opened successfully  {"process": "storagenode", "satellite": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "open_time": "200.960946ms", "logs_skipped": 60, "logs_matched": 0, "logs_mismatched": 0}
2026-03-05T20:19:22Z    ERROR   version failed to get process version info     {"process": "storagenode", "error": "version checker client: Get \"https://version.storj.io\": context canceled", "errorVerbose": "version checker client: Get \"https://version.storj.io\": context canceled\n\tstorj.io/storj/private/version/checker.(*Client).All:68\n\tstorj.io/storj/private/version/checker.(*Client).Process:89\n\tstorj.io/storj/private/version/checker.(*Service).checkVersion:104\n\tstorj.io/storj/private/version/checker.(*Service).CheckVersion:78\n\tmain.cmdRun:91\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tmain.main:34\n\truntime.main:285"}
2026-03-05T20:19:22Z    INFO    Telemetry enabled       {"process": "storagenode", "instance_id": "127rmeXcTpvdCSFByGFXa56U8ygdGDNPAtcbdd3BMNyYpYZ8txG"}
2026-03-05T20:19:22Z    INFO    Event collection enabled        {"process": "storagenode", "instance_id": "127rmeXcTpvdCSFByGFXa56U8ygdGDNPAtcbdd3BMNyYpYZ8txG"}
2026-03-05T20:19:22Z    ERROR   failure during run      {"process": "storagenode", "error": "Error migrating tables for database on storagenode: migrate: database: info opening file \"config/storage/info.db\" failed: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabaseWithStat:390\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:367\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).Migration.func1:719\n\tstorj.io/storj/private/migrate.(*Migration).Run:169\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:419\n\tmain.cmdRun:100\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tmain.main:34\n\truntime.main:285", "errorVerbose": "Error migrating tables for database on storagenode: migrate: database: info opening file \"config/storage/info.db\" failed: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabaseWithStat:390\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:367\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).Migration.func1:719\n\tstorj.io/storj/private/migrate.(*Migration).Run:169\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:419\n\tmain.cmdRun:100\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tmain.main:34\n\truntime.main:285\n\tmain.cmdRun:102\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tmain.main:34\n\truntime.main:285"}
2026-03-05T20:19:22Z    FATAL   Unrecoverable error     {"process": "storagenode", "error": "Error migrating tables for database on storagenode: migrate: database: info opening file \"config/storage/info.db\" failed: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabaseWithStat:390\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:367\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).Migration.func1:719\n\tstorj.io/storj/private/migrate.(*Migration).Run:169\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:419\n\tmain.cmdRun:100\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tmain.main:34\n\truntime.main:285", "errorVerbose": "Error migrating tables for database on storagenode: migrate: database: info opening file \"config/storage/info.db\" failed: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabaseWithStat:390\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:367\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).Migration.func1:719\n\tstorj.io/storj/private/migrate.(*Migration).Run:169\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:419\n\tmain.cmdRun:100\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tmain.main:34\n\truntime.main:285\n\tmain.cmdRun:102\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tmain.main:34\n\truntime.main:285"}


2026-03-05T20:19:36Z    INFO    Configuration loaded    {"process": "storagenode", "location": "/app/config/config.yaml"}
2026-03-05T20:19:36Z    INFO    Anonymized tracing enabled      {"process": "storagenode"}
2026-03-05T20:19:36Z    INFO    Operator email  {"process": "storagenode", "address": "xxx@zzz.net"}
2026-03-05T20:19:36Z    INFO    Operator wallet {"process": "storagenode", "address": "0xc43DBD0E344E3a75AA64D917962D06feFB5443Fc"}
2026-03-05T20:19:36Z    INFO    server  existing kernel support for server-side tcp fast open detected  {"process": "storagenode"}
2026-03-05T20:19:36Z    INFO    hashstore       loaded hint file        {"process": "storagenode", "satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "store": "s0", "path": "config/storage/hashstore/121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6/s0/meta/hint-0000000000000126", "writable": [502, 493, 481], "largest": 502, "skip": false}
2026-03-05T20:19:36Z    INFO    hashstore       unclean shutdown detected: reconciling logs     {"process": "storagenode", "satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "store": "s0"}
2026-03-05T20:19:36Z    INFO    hashstore       loaded hint file        {"process": "storagenode", "satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "store": "s1", "path": "config/storage/hashstore/121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6/s1/meta/hint-0000000000000175", "writable": [557, 572, 519, 569], "largest": 572, "skip": false}
2026-03-05T20:19:36Z    INFO    hashstore       unclean shutdown detected: reconciling logs     {"process": "storagenode", "satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "store": "s1"}
2026-03-05T20:19:36Z    INFO    hashstore       hashstore opened successfully  {"process": "storagenode", "satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "open_time": "468.098432ms", "logs_skipped": 161, "logs_matched": 7, "logs_mismatched": 0}
2026-03-05T20:19:36Z    INFO    hashstore       loaded hint file        {"process": "storagenode", "satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "store": "s0", "path": "config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/meta/hint-0000000000000090", "writable": [7199, 7514, 7714, 7491, 7715, 7639, 4699, 7248, 7370, 7494, 7532, 7699, 7626, 7574, 7623, 6585, 4068, 7513, 5618, 6808, 5525, 4700, 7702], "largest": 7715, "skip": false}
2026-03-05T20:19:41Z    INFO    hashstore       unclean shutdown detected: reconciling logs     {"process": "storagenode", "satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "store": "s0"}
2026-03-05T20:19:42Z    INFO    hashstore       loaded hint file        {"process": "storagenode", "satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "store": "s1", "path": "config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/meta/hint-0000000000000098", "writable": [8605, 9041, 8983, 8582, 8950, 5862, 7553, 8606, 8413, 9051, 9011, 2714, 5976, 8705, 8740, 8706, 8623, 8715, 7491, 8065, 3204, 6296, 8783, 8766, 2715, 2713, 7434, 8559, 8999, 8788, 8859, 8951], "largest": 9051, "skip": false}
2026-03-05T20:19:49Z    INFO    hashstore       unclean shutdown detected: reconciling logs     {"process": "storagenode", "satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "store": "s1"}
2026-03-05T20:19:51Z    INFO    hashstore       hashstore opened successfully  {"process": "storagenode", "satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "open_time": "14.574218487s", "logs_skipped": 7494, "logs_matched": 55, "logs_mismatched": 0}
2026-03-05T20:19:51Z    INFO    hashstore       loaded hint file        {"process": "storagenode", "satellite": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "store": "s0", "path": "config/storage/hashstore/12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs/s0/meta/hint-00000000000000de", "writable": [457, 534, 592, 624, 428, 565, 584, 528, 625, 483, 572, 522, 272, 504, 474, 418, 434, 502, 380, 619, 447, 620, 608, 406, 492, 283, 469, 615, 558, 547, 616, 423, 581, 595], "largest": 625, "skip": false}
2026-03-05T20:19:51Z    INFO    hashstore       unclean shutdown detected: reconciling logs     {"process": "storagenode", "satellite": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "store": "s0"}
2026-03-05T20:19:55Z    INFO    hashstore       loaded hint file        {"process": "storagenode", "satellite": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "store": "s1", "path": "config/storage/hashstore/12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs/s1/meta/hint-00000000000000b2", "writable": [411, 584, 396, 320, 610, 369, 334, 326, 575, 566, 476, 366, 548, 614, 605, 289, 594, 344, 606, 613, 362, 339, 500, 569, 561, 611, 422, 355, 401], "largest": 614, "skip": false}
2026-03-05T20:19:56Z    INFO    hashstore       unclean shutdown detected: reconciling logs     {"process": "storagenode", "satellite": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "store": "s1"}
2026-03-05T20:19:59Z    WARN    hashstore       mismatched log tail     {"process": "storagenode", "satellite": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "store": "s1", "id": 500, "path": "config/storage/hashstore/12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs/s1/f4/log-00000000000001f4-0000505f", "table": [{"Key":"IXKGLZGWIFXFW6IYBOAM4SY2IZGNOLTPEWDBIZXIRTX5LG3UMOEA","Offset":11599680,"Log":500,"Length":2319872,"Created":20483,"Expires":41150},{"Key":"G3M6AXSXS3HBC5LQEJBNEFV5JWTCZ54LWQPZEOG4WUJ5LGVIUPZQ","Offset":9279744,"Log":500,"Length":2319872,"Created":20483,"Expires":41150},{"Key":"RVP76AACKLTQZNEVJZWWETHIIBHMVGIGMCGOVZCUVUUHYB2PAPJA","Offset":6959808,"Log":500,"Length":2319872,"Created":20483,"Expires":41150},{"Key":"NRYMDK62RNQZ57OJEJ7YW4RK6AI2SVAUDHITLLXUFR7RQBYI3YHQ","Offset":4639872,"Log":500,"Length":2319872,"Created":20483,"Expires":41150},{"Key":"OZMVNGUNJW2YGCZ5IHDESZA2VY2QNDA7EQA4SS4S4CFDGN77IH6Q","Offset":0,"Log":500,"Length":2319872,"Created":20483,"Expires":41150}], "log": [{"Key":"IXKGLZGWIFXFW6IYBOAM4SY2IZGNOLTPEWDBIZXIRTX5LG3UMOEA","Offset":11599680,"Log":500,"Length":2319872,"Created":20483,"Expires":41150},{"Key":"G3M6AXSXS3HBC5LQEJBNEFV5JWTCZ54LWQPZEOG4WUJ5LGVIUPZQ","Offset":9279744,"Log":500,"Length":2319872,"Created":20483,"Expires":41150},{"Key":"RVP76AACKLTQZNEVJZWWETHIIBHMVGIGMCGOVZCUVUUHYB2PAPJA","Offset":6959808,"Log":500,"Length":2319872,"Created":20483,"Expires":41150},{"Key":"NRYMDK62RNQZ57OJEJ7YW4RK6AI2SVAUDHITLLXUFR7RQBYI3YHQ","Offset":4639872,"Log":500,"Length":2319872,"Created":20483,"Expires":41150},{"Key":"URYOOPD6SKXG6PJ4TRNTQYQOJDV6BFMV5KRUFGCH23Q4PNRETY7Q","Offset":2319936,"Log":500,"Length":2319872,"Created":20483,"Expires":41150}]}
2026-03-05T20:19:59Z    INFO    hashstore       hashstore opened successfully  {"process": "storagenode", "satellite": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "open_time": "8.138729906s", "logs_skipped": 264, "logs_matched": 62, "logs_mismatched": 1}
2026-03-05T20:19:59Z    INFO    hashstore       loaded hint file        {"process": "storagenode", "satellite": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "store": "s0", "path": "config/storage/hashstore/1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE/s0/meta/hint-00000000000000a1", "writable": [201, 206, 192, 193, 197, 204, 198, 194, 205, 191, 202], "largest": 206, "skip": false}
2026-03-05T20:19:59Z    INFO    hashstore       unclean shutdown detected: reconciling logs     {"process": "storagenode", "satellite": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "store": "s0"}
2026-03-05T20:20:01Z    INFO    hashstore       loaded hint file        {"process": "storagenode", "satellite": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "store": "s1", "path": "config/storage/hashstore/1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE/s1/meta/hint-00000000000000d2", "writable": [296, 302, 294, 300, 303, 295, 292], "largest": 303, "skip": false}
2026-03-05T20:20:01Z    INFO    hashstore       unclean shutdown detected: reconciling logs     {"process": "storagenode", "satellite": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "store": "s1"}
2026-03-05T20:20:02Z    INFO    hashstore       hashstore opened successfully  {"process": "storagenode", "satellite": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "open_time": "3.516991109s", "logs_skipped": 42, "logs_matched": 18, "logs_mismatched": 0}
2026-03-05T20:20:03Z    INFO    Telemetry enabled       {"process": "storagenode", "instance_id": "127rmeXcTpvdCSFByGFXa56U8ygdGDNPAtcbdd3BMNyYpYZ8txG"}
2026-03-05T20:20:03Z    INFO    Event collection enabled        {"process": "storagenode", "instance_id": "127rmeXcTpvdCSFByGFXa56U8ygdGDNPAtcbdd3BMNyYpYZ8txG"}
2026-03-05T20:20:03Z    INFO    db.migration    Database Version        {"process": "storagenode", "version": 62}
2026-03-05T20:20:03Z    INFO    preflight:localtime     start checking local system clock with trusted satellites' system clock.        {"process": "storagenode"}
2026-03-05T20:20:04Z    INFO    preflight:localtime     local system clock is in sync with trusted satellites' system clock.    {"process": "storagenode"}
2026-03-05T20:20:04Z    INFO    trust   Scheduling next refresh {"process": "storagenode", "after": "7h5m23.144990506s"}
2026-03-05T20:20:04Z    INFO    bandwidth       Persisting bandwidth usage cache to db  {"process": "storagenode"}
2026-03-05T20:20:04Z    INFO    pieces:trash    emptying trash started  {"process": "storagenode", "satellite_id": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2026-03-05T20:20:04Z    INFO    pieces:trash    emptying trash finished {"process": "storagenode", "satellite_id": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "elapsed": "5.29314ms"}
2026-03-05T20:20:04Z    INFO    pieces:trash    emptying trash started  {"process": "storagenode", "satellite_id": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2026-03-05T20:20:04Z    INFO    pieces:trash    emptying trash finished {"process": "storagenode", "satellite_id": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "elapsed": "331.613µs"}
2026-03-05T20:20:04Z    INFO    pieces:trash    emptying trash started  {"process": "storagenode", "satellite_id": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2026-03-05T20:20:04Z    INFO    pieces:trash    emptying trash finished {"process": "storagenode", "satellite_id": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "elapsed": "325.069µs"}
2026-03-05T20:20:04Z    INFO    pieces:trash    emptying trash started  {"process": "storagenode", "satellite_id": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2026-03-05T20:20:04Z    INFO    pieces:trash    emptying trash finished {"process": "storagenode", "satellite_id": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "elapsed": "323.597µs"}
2026-03-05T20:20:04Z    INFO    Node 127rmeXcTpvdCSFByGFXa56U8ygdGDNPAtcbdd3BMNyYpYZ8txG started        {"process": "storagenode"}
2026-03-05T20:20:04Z    INFO    Public server started on [::]:28967     {"process": "storagenode"}
2026-03-05T20:20:04Z    INFO    Private server started on 127.0.0.1:7778       {"process": "storagenode"}

I’m assuming that after that second restart everything is now good and I can let it chug along again in the background.

Cheers.

PieceKeeper · March 5, 2026, 10:47pm

Interesting with the extra restart and that it toke two restarts to produce the unclean messages.

So far I have 5 upgraded and 4 of them had these unclean shutdown detected messages.

Alexey · March 6, 2026, 2:01am

Why it did restart twice? What was the reason?