Release preparation v1.154

New release candidate v1.154 is already deployed on QA Satellite
Here is list of changes (including reverts for SN)

Changelog

General

  • 0fb6bfb private/apigen: Allow endpoints with non-json responses
  • 6febe81 shared/dbutil: add TiDB implementation enum

Satellite

  • ce735b6 satellite/satellitedb: migrate repair_queue DELETE to DBX
  • 2a86df4 satellite/satellitedb: migrate repair_queue COUNT to DBX
  • 169ab77 web/satellite: create access grants via the satellite API
  • 2941681 satellite/metainfo: unify success and failure trackers into Trackers
  • 3c4e4dc satellite/metainfo: split NodeFailed into NodeCancelled and NodeRetried
  • a1e0882 satellite/{console,satellitedb}: add tenant_whitelabel_configs table
  • 44fb937 private/post,satellite/mailservice: improve emails sent from the satellite
  • 3efec2b satellite/admin: add whitelabel config management API and UI
  • 0f5a3a0 satellite: drop satellitedb RepairQueue, use jobq exclusively
  • 5fdf0ba satellite/metabase: update ListSegments to return checksum
  • 5e91469 satellite/admin: fix linting issues
  • 696ed74 satellite/jobq: prefer ctx.Err() over transport errors in client
  • aa09ab1 satellite/metainfo: add retry tracker to Trackers
  • 5a06b39 satellite/console: create restricted access via public API
  • 1c93629 satellite/jobq: make the Close function nil safe
  • 7b6449d satellite/core: register new projectlimitevens with mud
  • 0031da3 web/satellite,private/apigen: upgrade most of the frontend deps
  • 2865545 web/satellite: make it possible to re-activate account with closed registration
  • 2b72044 web/satellite: don’t compute gzipped chunk sizes during builds
  • 3e2d691 web/satellite: fix password manager autofill on auth forms (#7762)
  • ad418bb web/satellite: move dashboard stats to the Usage page
  • 6b623d7 satellite/nodeselection: alreadySelected should be NodeId instead of SelectedNode
  • 8b18bc3 satellite/accounting/live: fix mon.Task defer and dead error check
  • cc863ea satellite/db: add opt_in_status column to user_settings
  • 04b4d22 satellite/payments: fix usage start and end dates on invoices
  • f0a7453 satellite/satellitedb: skip boundary tally when all in-period tallies are zero

Storagenode

  • 3a6ac1f storagenode/load: add iostat-like disk I/O monitoring via monkit
  • cb5706a storagenode/hashstore: preallocate log files
  • 4abae8d storagenode/hashstore: remove collisions
  • fddfea5 storagenode/hashstore: add config option to require setup
  • 1447d43 storagenode/hashstore: skip log check for missing log files
  • 4d825fd storagenode/hashstore: allow disabling copy_file_range
  • bcab0f8 storagenode/hashstore: Sort api for keys by log position
  • 451b2f9 storagenode/hashstore: refactor Compact to take a struct
  • 4dd0f84 storagenode/hashstore: fix bug with logCollection
  • 7545726 storagenode/monitor: fix PreFlightCheck and DiskSpace for hashstore-only nodes
  • 79cc8b8 Revert “storagenode/monitor: fix PreFlightCheck and DiskSpace for hashstore-only nodes”
  • 3297f78 Revert “storagenode/monitor: include hashstore usage in PreFlightCheck totalUsed”
  • e89ddea Revert “storagenode/monitor: Fix allocated_space for hashstore only storagenodes”
  • 8daef76 Revert “storagenode/monitor: use shared disk report for select runner”

Test

  • 436e18f testsuite: install jobq in tests

Interesting! Maybe this will be a way to expose storage activity metrics up through to our Grafana dashboards (without pulling those number direct from OS)?

No, this is a gimmick that reads from /proc/diskstats on Linux. This does not need to be in the storagenode software. I wish they stopped wasting company resources on gimmicks.

They probably want to use that data for node selection in the select network.

Thank you for making it possible to disable this!

Why ReconstructTable isn’t set to true by default? Isn’t a must?

It’s set to invisible so you are not even supposed to know about it… :wink:

I could see a Select operator: who always wanted to make sure his ‘real’ customers had plenty of disk IO… wanting a node to reject uploads if it detected a disk has been persistently busy. Or maybe having compaction events always look at load first too?

Basically make very sure Storj is only using idle/leftover iops…

What kind of setup could this be? In my opinion storj is the operator and they simply renting complete servers with a bunch of hard drives.

I’m talking about regular Select: where the SOC2 datacenters trade custom payouts for the removal of some restrictions (like no /24 block, so they fill faster). They can reclaim space at any time for regular customers paying more for it (like they rented a dedi / vps / whatever).

I think what you’re talking about (where Storj rents servers themselves) was what they called Surge nodes? I’m don’t think we ever heard how many there were (or if they even still exist). Cool idea though!

They don’t need to fill. Select space is fully paid from day one whether it is filled or not. :wink:

I’m… not sure this is true.

Did that change? I didn’t make up the no-/24-rule-fill-faster stuff: it’s a documented advantage of the program:

Data sets with specific requirements stored on nodes in this program are not subject to the /24 restriction, allowing them to fill their nodes more rapidly

I don’t think so but they also have a “guaranteed usage”.

Ah you may have found the nuance: Select has their own customer pool… but having nodes that are “used” may not mean they’re fully-paid-for.

I guess it could even mean the contract covers minimum payouts? Like if they offered 1PB then at least 10% may be paid for (even before it’s filled)? Above my pay grade :winking_face_with_tongue:

Less payment than global SNOs and only 10% guaranteed? If this was true storj could ditch the entire global network. :money_mouth_face:

They definately could be steering regular/global uploads to Select SNOs with lots of free space: as a way to always be using the lowest-payout storage as possible.

Then when a normal Select customer comes along: willing to pay the higher price: they give them the SOC2 space and simply repair that global-data-in-select-nodes back to the global network.

It would actually be really clever if they’re doing that: great idea :fire:

…don’t exist. Those nodes are operated by storj.

Is this a wording thing? Like they’re really “commercial node operators” that are part of the “Storj Select tier of storage”… so you’d prefer not calling them “Select SNOs” as a generic term?

A third-party owns the datacenter, and the servers, and the network gear. They maintain their own NOC staff, and SOC2 audit reports, and minimum capacity online. They own everything independent of Storj: I think it’s reasonable to call them Select SNOs.

Like… my nodes run the vanilla update logic: so Storj decides what version of the node I’m on, and even controls when they restart for those upgrades. Storj Satellites decide what data gets sent to the node, and when it’s deleted. I could see an argument that the “nodes are operated by Storj”… so maybe I shouldn’t be called a SNO?

And yet… ownership of the environment matters. Which is why everyone here still calls themselves a SNO. So I think Select SNOs exist

Sure. :wink:

I don’t like to call them SNOs because they just doing their normal server and storage business, zero knowledge about running nodes is needed.

Is storage node operating on their hardware with their permission? They are storage node operators by definition.

It’s not some secret club. No specialized knowledge is required to run an executable with three arguments.

I’m sure they will once participating datacenters provide sufficient coverage.