Release preparation v1.62

Andrii · August 19, 2022, 10:57am

New release v1.62 already deployed on QA Satellite
This time there are a lot more changes
List of changes:

Changelog

Satellite

c8b72c8 satellite/{payments/billing,satellitedb}: refactor billing DB
5f6892f satellite/console/consoleweb: early rejection in auth middleware
92be1d8 satellite/payments/stripecoinpayments: storjscan invoice generation
799b159 satellite/reputation: offset write times by random, not by satelliteID
ad37ea4 satellite/{web, console}: login captcha implemented
af6f556 web/satellite: navigation mobile adaptation
f562277 satellite/{web,console}: added string length limits for signup partner and promo params
ec72adb satellite/console: send email when user’s account gets locked
e0c863e web/satellite: update “deposit storj” message to use support link rather than email
c13a470 web/satellite: update dashboard chart data on project switch
30727b9 web/satellite: add data delay info for new project dashboard
d76acda satellite/{web, console}: removed account locked statuses
27f6fbd web/satellite: refactored modals to use common VModal component
e1a85fc web/satellite: added notification when storage limit is exceeded in file browser
81327f6 web/satellite: Fix documentation URL (#5049)
0f3da7f web/satellite: Added token transaction history to new payment methods flow (#4977)
c5abc0d satellite/metainfo: Homogenize object operations logs
e5877fa web/satellite: fix error with “drop files” box in file browser
3f6bf64 satellite/console: Default NewProjectDashboard to “true”
938efd7 satellite/metabase: Don’t stop on error expired & zombie objects
9e64a87 satellite/metabase: measure delete object batch
4c5466a web/satellite: update projects table component
3afd7bc satellite/console: Reset passwords invalidates all current user sessions.
e8b716f satellite/analytics: Added track calls for user behavioral events (#5009)
174d2ec satellite/{payments/billing,satellitedb}: Add retry logic to insert
eca1689 satellite/console: add wallet payments API
b398035 satellite/console/consoleweb: Log trace reqs with INFO
917925b satellite/metabase: delete migrated zombie objects
6cc2052 satellite: fix segment loop observers metrics
a416654 web/satellite: dashboard banners fixed
b7e9d04 web/satellite: common table component applied to file browser
0550731 web/satellite: add S3 credential creation to quickstart dropdown
4418216 Web/satellite: Create Access Flow Modal Refactor (#5015)
f507de6 satellite: in-memory ‘top’-like counter for project/partner
cff8158 satellite/console: add audit logs for failed logins
8b338fb web/satellite: show guide on bucket page
c3e7b14 satellitedb: add salt to projects table
4be219c Web/satellite: Add responsiveness to onboarding flow (#5065)
c552c34 web/satellite: make modals responsive
2abe709 satellite/metabase: add metabase.GetObjectLatestVersion back
d910074 web/satellite Billing screen bug fixes (#5050)
0ec28ca web/satellite: add token funds modal
26bed35 web/satellite: update access grants table
ac24150 satellitedb: add wallet address index to storjscan_wallet table
7299e9f satellite/console: Add missing monkit calls
f596f72 satellite/payments/storjscan: add billing update to chore
8eec61d satellitedb: add timestamp index to billing table
ffd35ec web/satellite: move billing nav item into My Account dropdown
eb54774 web/satellite: fix mobile signup page
c82424c satellite/admin: add Burst field to Update Project Limits params
286f68a web/satellite: fix page content bottom padding
9dcef77 web/satellite: fix for billing screen
4183726 web/satellite: improve access grants empty state
99dee2d web/satellite: fix bucket page guide positioning
0dcc0a9 satellite/reputation: reconfigure lambda and alpha
c59d0d4 satellitedb: add signup_captcha column to users table
874e28e web/satellite: fix inactive buttons on billing page

Storagenode

cf92220 {satellite,storagenode}/gracefulexit: simplify limiter usage
e9692c5 storagenode/gracefulexit: remove unused interface
4fdb81c storagenode/pieces: allow to configure initial piece scan (#5024)
9a539c4 storagenode/storageusage: add interval_end_time, rename interval_start to timestamp
25f8f67 storagenode/nodestats: retrieve storage usage starting from last day of previous month

Test

53db7a8 testsuite/storjscan: add testsuite to test storjscan integration
a6ad86d testsuite/storjscan: add client claim wallet test
379711c testsuite/storjscan: add client payments request test
0036ec9 testsuite/storjscan: add chore test
1f06387 private/apigen,cmd/apigentest: add tests for generated API code

Uplink

04c0e5d cmd/uplink: fix access setup flags setup

General

5a2e348 build: bump timeout for premerge/public
731fecd private/apigen: rework request parameter handling
37a4edb all: reformat comments as required by gofmt 1.19
0698b1e go.mod: bump storj/private version
96b88e2 build: fix conditional run of web builds

Roberto · August 19, 2022, 4:48pm

Great. And now, how do you activate / deactivate? Docker command or change config file?

And what possible problems can there be with a node that has the filewalker turned off?

Thanks

Th3Van · August 19, 2022, 6:33pm

Are there any updates in this version, that somehow breaks fetching (some) data with curl ?

root@backup001:/root# curl -s 127.0.0.1:52000/api/sno/  (works)
{
  "nodeID": "1ko9N9LXiWsY9hcF85eeug2Y6gkxbEppWzNFAFa6n4ZuXrhMgR",
  "wallet": "0x165A1Da6F4d3c0555a9E1f94288A7e17BD14f7d1",
  "walletFeatures": [
    "zksync"
  ],
  "satellites": [
    {
      "id": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo",
      "url": "us2.storj.io:7777",
      "disqualified": null,
      "suspended": null,
      "currentStorageUsed": 5004544
    },
    {
      "id": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE",
      "url": "saltlake.tardigrade.io:7777",
      "disqualified": null,
      "suspended": null,
      "currentStorageUsed": 222742528
    },
    {
      "id": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6",
      "url": "ap1.storj.io:7777",
      "disqualified": null,
      "suspended": null,
      "currentStorageUsed": 286775808
    },
    {
      "id": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S",
      "url": "us1.storj.io:7777",
      "disqualified": null,
      "suspended": null,
      "currentStorageUsed": 1941721344
    },
    {
      "id": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs",
      "url": "eu1.storj.io:7777",
      "disqualified": null,
      "suspended": null,
      "currentStorageUsed": 913410560
    },
    {
      "id": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB",
      "url": "europe-north-1.tardigrade.io:7777",
      "disqualified": null,
      "suspended": null,
      "currentStorageUsed": 181933056
    }
  ],
  "diskSpace": {
    "used": 3559492608,
    "available": 1000000000,
    "trash": 94163200,
    "overused": 2653655808
  },
  "bandwidth": {
    "used": 716437248,
    "available": 0
  },
  "lastPinged": "2022-08-19T20:03:38.159757415+02:00",
  "version": "1.62.0-rc",
  "allowedVersion": "1.24.0",
  "upToDate": true,
  "startedAt": "2022-08-19T19:55:43.196547175+02:00",
  "configuredPort": "50000",
  "quicEnabled": true
}

root@backup001:/root# curl -s 127.0.0.1:52000/api/sno/satellite/12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo | jq
{
  "error": "consoleapi storagenode: console: sql: Scan error on column index 1, name \"24 * (at_rest_total / COALESCE((CAST(strftime('%s', interval_end_time) AS NUMERIC) - CAST(strftime('%s', LAG(interval_end_time) OVER (PARTITION BY satellite_id ORDER BY interval_end_time)) AS NUMERIC)) / 3600, 24))\": converting NULL to float64 is unsupported"
}

And on a side note - @BrightSilence earnings.py script also seems to be having issues, since the node db was upgraded from v53 to v54

root@backup001:/tmp/222# python ./earnings.py /tmp/222
Traceback (most recent call last):
  File "./earnings.py", line 323, in <module>
    for data in con.execute(query):
sqlite3.OperationalError: no such column: interval_start

Th3van.dk

BrightSilence · August 19, 2022, 6:35pm

Thanks for the heads up. I will implement a fix for this when it goes live on production nodes.

littleskunk · August 22, 2022, 2:01pm

That might be a bug on the storage node side. I have reported it to @clement. Let’s see if we can fix this before the new storage node release goes out. To me, it looks like the root cause is that Stefans test satellite is empty and the storage node dashboard has issues displaying that on the storage node dashboard. If I select the QA satellite it looks all fine (besides the known issue with the last 2 days). Both satellites are running the latest version.

BrightSilence · August 22, 2022, 2:14pm

Could probably be fixed by using either NullFloat64 here (I noticed database/sql is already imported)

github.com

storj/storj/blob/9a539c4830bc9b241aaa4f00fb2fad02b54406f3/storagenode/storagenodedb/storageusage.go#L80


      
          
          
	rows, err := db.QueryContext(ctx, query, satelliteID, from.UTC(), to.UTC())
          	if err != nil {
          		return nil, err
          	}
          	defer func() { err = errs.Combine(err, rows.Close()) }()
          
          
	var stamps []storageusage.Stamp
          	for rows.Next() {
          		var satellite storj.NodeID
          		var atRestTotal float64
          		var timestamp time.Time
          
          
		err = rows.Scan(&satellite, &atRestTotal, &timestamp)
          		if err != nil {
          			return nil, err
          		}
          
          
		stamps = append(stamps, storageusage.Stamp{
          			SatelliteID:   satellite,
          			AtRestTotal:   atRestTotal,

Or adding a fallback to 0 using COALESCE around at_rest_total here

github.com

storj/storj/blob/9a539c4830bc9b241aaa4f00fb2fad02b54406f3/storagenode/storagenodedb/storageusage.go#L64


      
          
          
	// the at_rest_total is in bytes*hours, so to find the total number
          	// of hours used to get the at_rest_total, we find the hour difference,
          	// between the interval_end_time of a row and that of the previous row
          	// and divide the at_rest_total by the hour interval and multiply by 24 hours
          	// 24 hours to estimate the value for a 24hour time window.
          	// i.e. 24 * (at_rest_total/hour_difference), where the
          	// hour_difference = current row interval_end_time - previous row interval_end_time
          	// Rows with 0-hour difference are assumed to be 24 hours.
          	query := `SELECT satellite_id,
          					 24 * (at_rest_total / COALESCE((CAST(strftime('%s', interval_end_time) AS NUMERIC) - CAST(strftime('%s', LAG(interval_end_time) OVER (PARTITION BY satellite_id ORDER BY interval_end_time)) AS NUMERIC)) / 3600, 24)),
          					 timestamp
          				FROM storage_usage
          				WHERE satellite_id = ?
          				AND ? <= timestamp AND timestamp <= ?
          				ORDER BY timestamp`
          
          
	rows, err := db.QueryContext(ctx, query, satelliteID, from.UTC(), to.UTC())
          	if err != nil {
          		return nil, err
          	}

I’d do it myself, but I don’t have a go dev environment set up. And I don’t like making untested PR’s (though the COALESCE is a pretty safe bet).

littleskunk · August 22, 2022, 3:23pm

This is already great contribution. Thank you very much. @clement what are your thoughts?

clement · August 22, 2022, 3:49pm

Great suggestion @BrightSilence. I’m adding new test cases to test for such instances and try out your suggestion as a fix.
Though I’m thinking it should already return an empty result set if there is no data for the stefanlite, and this error should not be thrown in the first place.

littleskunk · August 24, 2022, 12:03pm

@BrightSilence the storage node rollout for v1.62.3 should start shortly. The binaries are already available on github. It looks like the storage node release still breaks your script.

It turns out the null values are already wrong on the satellite side. There is a fix available for the satellite but testing the fix would have delayed the satellite release too long. So we fixed just the storage node side to display a used space graph like last release. It is still lacking behind 2 days. Thats why I call it a quick and dirty fix.

At the end it is a risk vs benefit question. How much risk are we willing to accept in order to get the storage node dashboard working. This time we have choosen a low risk approach even if it is only a partial fix.

BrightSilence · August 24, 2022, 12:52pm

Thanks for the heads up, I am aware of this.
This is due to changed column names on the node end. It’s unrelated to the error here discussed here. I already have a change ready and waiting to publish that fixes it (it helps to have a testnet node that updates early). But it would break the script on older node versions, so I’m waiting until rollout begins to publish it.

I’m also working on making some changes to accommodate the new audit scoring threshold. Not sure if that will be in the same version, but it might be. I don’t think I need to do much for this.

littleskunk · August 24, 2022, 2:40pm

There should be a table that tells you about the DB version. Shouldn’t that allow you to just use the old query for any node that isn’t updated and the new query for the nodes that have been updated? The value you need for this decision should in in the info.db. SELECT max(version) FROM versions;

BrightSilence · August 24, 2022, 2:47pm

Yeah, that’s possible, but the script currently works with a single big query. It would require a bit more rework than I’m willing to do for maybe a few weeks of overlap.

I prefer to just see it as an extra incentive to update your node and keep the script more simple. It’s too much code as it is already, haha.

flo · August 24, 2022, 3:11pm

Ready for testing

littleskunk · August 24, 2022, 3:24pm

Yea I am also waiting to download your new version

BrightSilence · August 24, 2022, 3:30pm