Release preparation v1.148

[Roberto]

A node on a Qnap potato updated a few hours ago and everything went smoothly

2026-03-06 02:19:42,029 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

2026-03-06T02:19:42Z    INFO    Current binary version    {"process": "storagenode-updater", "service": "storagenode-updater", "version": "v1.148.3"}

2026-03-06T02:19:42Z    INFO    Version is up to date     {"process": "storagenode-updater", "service": "storagenode-updater"}

2026-03-06T02:19:42Z    INFO    server  kernel support for tcp fast open unknown  {"process": "storagenode"}

2026-03-06T02:19:42Z    INFO    hashstoreloaded hint file {"process": "storagenode", "satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "store": "s0", "path": "config/storage/hashstore/121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6/s0/meta/hint-00000000000000c2", "writable": [], "largest": 315, "skip": false}

2026-03-06T02:19:43Z    INFO    hashstoreloaded hint file {"process": "storagenode", "satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "store": "s1", "path": "config/storage/hashstore/121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6/s1/meta/hint-00000000000000b1", "writable": [], "largest": 294, "skip": false}

2026-03-06T02:19:45Z    INFO    hashstorehashstore opened successfully    {"process": "storagenode", "satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "open_time": "2.650736036s", "logs_skipped": 107, "logs_matched": 0, "logs_mismatched": 0}2026-03-06T02:19:45Z    INFO    hashstoreloaded hint file {"process": "storagenode", "satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "store": "s0", "path": "config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/meta/hint-000000000000008e", "writable": [], "largest": 1950, "skip": false}

2026-03-06T02:19:50Z    INFO    hashstoreloaded hint file {"process": "storagenode", "satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "store": "s1", "path": "config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/meta/hint-000000000000009a", "writable": [], "largest": 2602, "skip": false}

2026-03-06T02:19:56Z    INFO    hashstorehashstore opened successfully    {"process": "storagenode", "satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "open_time": "11.336478586s", "logs_skipped": 2027, "logs_matched": 0, "logs_mismatched": 0}2026-03-06T02:19:56Z    INFO    hashstoreloaded hint file {"process": "storagenode", "satellite": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "store": "s0", "path": "config/storage/hashstore/12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs/s0/meta/hint-00000000000000d2", "writable": [], "largest": 839, "skip": false}

2026-03-06T02:20:00Z    INFO    hashstoreloaded hint file {"process": "storagenode", "satellite": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "store": "s1", "path": "config/storage/hashstore/12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs/s1/meta/hint-0000000000000098", "writable": [], "largest": 1005, "skip": false}

2026-03-06T02:20:05Z    INFO    hashstorehashstore opened successfully    {"process": "storagenode", "satellite": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "open_time": "8.961504031s", "logs_skipped": 699, "logs_matched": 0, "logs_mismatched": 0}2026-03-06T02:20:05Z    INFO    hashstoreloaded hint file {"process": "storagenode", "satellite": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "store": "s0", "path": "config/storage/hashstore/1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE/s0/meta/hint-0000000000000078", "writable": [], "largest": 170, "skip": false}

2026-03-06T02:20:06Z    INFO    hashstoreloaded hint file {"process": "storagenode", "satellite": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "store": "s1", "path": "config/storage/hashstore/1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE/s1/meta/hint-000000000000007e", "writable": [], "largest": 174, "skip": false}

2026-03-06T02:20:06Z    INFO    hashstorehashstore opened successfully    {"process": "storagenode", "satellite": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "open_time": "1.232320296s", "logs_skipped": 34, "logs_matched": 0, "logs_mismatched": 0}2026-03-06T02:20:07Z    INFO    Telemetry enabled {"process": "storagenode", "instance_id": "1Kf6y9MyoiSwcvwVKZRtCrp9VbQsMC1Pkhike5Sn1NxrprnnLU"}

2026-03-06T02:20:07Z    INFO    Event collection enabled  {"process": "storagenode", "instance_id": "1Kf6y9MyoiSwcvwVKZRtCrp9VbQsMC1Pkhike5Sn1NxrprnnLU"}

2026-03-06T02:20:07Z    INFO    db.migration      Database Version        {"process": "storagenode", "version": 62}

2026-03-06T02:20:07Z    INFO    preflight:localtime       start checking local system clock with trusted satellites' system clock.    {"process": "storagenode"}

2026-03-06T02:20:08Z    INFO    preflight:localtime       local system clock is in sync with trusted satellites' system clock{"process": "storagenode"}

2026-03-06T02:20:08Z    INFO    Node 1Kf6yxxxxxxxxxxxx started     {"process": "storagenode"}

2026-03-06T02:20:08Z    INFO    Public server started on [::]:28967       {"process": "storagenode"}

2026-03-06T02:20:08Z    INFO    Private server started on 127.0.0.1:7778  {"process": "storagenode"}

2026-03-06T02:20:08Z    INFO    failed to sufficiently increase receive buffer size (was: 800 kiB, wanted: 7168 kiB, got: 3200 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.  {"process": "storagenode"}

2026-03-06T02:20:08Z    INFO    piecemigrate:chore        all enqueued for migration; will sleep before next pooling        {"process": "storagenode", "active": {"12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs": false, "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE": false, "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6": false, "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S": false}, "interval": "10m0s"}

[PieceKeeper]

You can see the whole flow in EddieA’s post above.

[Vadim]

this work around dot work if you have, at least one broken file, it just throw error and end working.

[Alexey]

Yes, but I do not see why the external signal to kill the node is occured?

[Alexey]

I didn’t know. Notified the team.
How did you figure out that it’s stopping to proceed? Did it exit after the error?

[EddieA]

@Alexey

Yes, but I do not see why the external signal to kill the node is occured?

I’m guessing the first one was from the version updater process to tell the main node to restart because it’s version has been updated.

Now why the second one (which is a different signal and appears to be ignored) is another question.

Once any of my other nodes are updated, I’ll check their logs too.

Cheers.

[EddieA]

@EddieA

Once any of my other nodes are updated, I’ll check their logs too.

Another node just updated. All I got this time was this:

2026-03-10T00:08:24-07:00       INFO    Got a signal from the OS: "interrupt"   {"Process": "storagenode"}

Followed by a clean start, with no errors nor a second signal message.

Cheers.

[nitrobass24]

One of my nodes updated today, waiting on the others.
I dont love the new pie chart. Used, Trash, Reclaimable all show as the same color blue and amounts dont show in the legend, so its not quite as useful as it once was, but I am assuming its more accurate?

[Alexey]

Yes, it has more details in expense of a reduced legend, see

@elek

There are too many categories (trash, reserved, reclaimable, etc.). Doesn’t fit easily on the right side. Instead I tried to add categories only to doughnut (move mouse over the two blue parts. There should be tool tips).

Trash represents the trash which couldn’t be deleted, yet.

“dead bytes” are called “reclaimable”, and counted as a separated category, as they can be deleted immediately.

[snorkel]

The dotted graphs are working now on iPhone. Good job! In the past 5 years, scrolling the dashboard on iOs was very annoying, each graph touch would bring the details menu that won’t go away unless you refresh the page.

The donut, on the other hand, is a big downgrade.
Yes, it shows the correct free space now, but you don’t realy know what’s under used space part.
I was using that to compare it with the satellites graph, to check for inconsistencies, the very useful method that helped us spot a lot of problems for the last 5 years.
Now you have to rely on third party apps that can make sense of api’s mumble jumble.
Awful!

[arrogantrabbit]

I still maintain that this is moving in the wrong direction. We don’t need more data in the dashboard. We need much less. We need green “all good” or red “something wrong” with relevant logs surfaced. That’s it.

Better yet — no dashboard. Just send me an email with logs when something breaks. An email. That’s it.

No graphs, no plots, not tables, no nonsense. You know, perfection is archived not when there is nothing more to add, but when there is nothing left to take away.

Right now there is plenty to take away. Pretty much the whole dashboard. Every unnecessary control and data point is friction, bugs, confusion, debugging, and maintenance. It’s extra work for no benefit in return. There is nothing to see in the dashboard that’s is actionable. Nothing. Node either works or it does not. If it does not work well — that’s what telemetry is for, focus on that.

The only use for dashboard today is satisfying idle curiosity. Very dubious thing to spend engineering time developing and maintaining.

[snorkel]

You forget how many bugs we discovered with the dashboard. Just a recent exemple: the unvetted nodes that went unnotticed for years.

[arrogantrabbit]

This is a problem, reinforcing my point actually.

It shall not be on node operators to independently waste time looking on dashboard and reporting bugs.

It is squarely a job of storj QA department: Architecting telemetry and correctness/abnormalities analysis. This is what they are receiving salary for. You can’t hinge your software reliability on unpaid work of volunteers, who don’t even know about said work assignment.

If node operators could find bugs with limited subset of data dashboard awkwardly presents — can you imagine the insights that can be derived from telemetry, where full data over thousands of nodes can be processed? This also will help optimize resource allocation — deprioritizing issues only two operators see and focusing on those affecting majority of nodes.

@snorkel

the unvetted nodes that went unnotticed for years.

This is a great illustration that storj telemetry and qa sucks today. Detecting such cases is trivial and should not have required involvement of multiple operators and lengthy forum discussion.

So yes, thank you for reinforcing my point. Dashboard must go, and storj needs to up their telemetry analysis game. This also will help catch cheaters who sneak around /24 rule, as a side effect

[Toyoo]

@arrogantrabbit

Dashboard must go, and storj needs to up their telemetry analysis game.

I sort of agree with the first, not with the second. Existing QA seems to be good enough for Storj purposes, the network was working well enough even despite nodes unvetted for years Why spend more on QA then?

[Vadim]

Does Amnisty function not working yet?

2026-03-17T23:46:57+02:00 ERROR contact:amnesty failed to get satellite URL for amnesty report {“satellite”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “piece_count”: 2, “error”: “trust: satellite is untrusted”, “errorVerbose”: “trust: satellite is untrusted\n\tstorj.io/storj/storagenode/trust.init:29\n\truntime.doInit1:7670\n\truntime.doInit:7637\n\truntime.main:256”}

[Alexey]

Is it disqualified already?

[Vadim]

not yet, i see that there is some problems, i turnd off node to investigate.

[Alexey]

there is a bug where we amnesty pieces before we get the satellite trust list, but the satellite implementation doesn’t do anything yet, anyway

[nitrobass24]

If you just want logs, then look at your logs. No one is forcing you to look at the dashboard.