How to calculate update progress?

[donald.m.motsinger]

How does the storagenode-updater calculates whether a node should be updated from the nodeID, seed and cursor? And how often does the updater checks version.storj.io?

[arrogantrabbit]

This is the check for whether the node needs to be updated:

storj/common/blob/9815a85cbc32f3fc9c234b35099807119ab5a4e0/version/version.go#L285

Ultimately the cursor magic is checked in

storj/common/blob/9815a85cbc32f3fc9c234b35099807119ab5a4e0/version/version.go#L274

I believe the updater checks every 15 min.

[donald.m.motsinger]

Thanks, but can you translate this to Englishbash, please?

[arrogantrabbit]

It hashes rollout seed and node ID and compares the result with the cursor.

I was going to convert it to bash myself for my replacement updater for FreeBSD, but eventually did not get to it.

Asking Google Gemini to translate it to bash yields this, which is more or less correct on the first glance:

isRolloutCandidate() {
  local nodeID="$1" rolloutSeed="${2[@]:0:32}" rolloutCursor="${2[@]:32}"
  local hash=$(echo -n "$nodeID" | openssl dgst -sha256 -hex -mac "seed:${rolloutSeed}")
  [[ "$hash" <= "$rolloutCursor" ]] && echo "true" || echo "false"
}

[donald.m.motsinger]

I’m a bit confused by this. Why does it take the first 32 characters from the second (and following) arguments for the seed and the remaining characters for the cursor? The seed has more than 32 characters. Apart from this, the openssl command fails with

Error initializing seed:6c261c93e6c41511e900ba09ac726587ba208c841dab5852a1fa76870de34b99 context
4097A8DFA17F0000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:../crypto/evp/evp_fetch.c:349:Global default library context, Algorithm (seed:6c261c93e6c41511e900ba09ac726587ba208c841dab5852a1fa76870de34b99 : 0), Properties (<null>)

here is my simplyfied script

#!/bin/bash
nodeID="$1"
rolloutSeed="$2"
rolloutCursor="$3"
hash=$(echo -n "$nodeID" | openssl dgst -sha256 -hex -mac "seed:${rolloutSeed}")
[[ "$hash" <= "$rolloutCursor" ]] && echo "true" || echo "false"

The last line gives a syntax error too, but this might be because of $hash being empty.

I’m using Ubuntu 22.04 with OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)

Should openssl list -digest-algorithms|grep -i hmac return something?

[donald.m.motsinger]

@raert

echo -n $NODEID | openssl mac -digest sha256 -macopt key:$SEED HMAC

This seems to do something without error message, though it doesn’t seem to be correct. I have nodes with a hash lower than the cursor which are not updated and vice versa. The same applies when I use
-macopt hexkey:$SEED

Here’s the script so far

#!/bin/bash
nodeID="$1"
rolloutSeed=$(curl -s https://version.storj.io|jq -r .processes.storagenode.rollout.seed)
rolloutCursor=$(curl -s https://version.storj.io|jq -r .processes.storagenode.rollout.cursor)
hash=$(echo -n "$nodeID" | openssl mac -digest sha256 -macopt hexkey:${rolloutSeed} HMAC)

echo "cursor $rolloutCursor"
echo "hash   $hash"

[elek]

I didn’t test the script, but do you use the hexadecimal version of the NodeID? I thing the golang version uses the pure binary, while we usually use the base58 representation…

[donald.m.motsinger]

@elek

I didn’t test the script, but do you use the hexadecimal version of the NodeID?

I use what you as a developer give us SNO’s in the dashboard. It looks like it is base58. Do I need to convert it to hex?

I thought it is quite clear from my attempts above that I’m not a developer. I didn’t think that such a simple task would be so complicated…

Ok, with the help of ChatGPT I have this now

#!/bin/bash

base58_to_hex() {
  local base58_string="$1"

  # Decode Base58 string to bytes
  local decoded_bytes=$(echo "$base58_string" | base58 -d | tr -d '\0')

  # Convert bytes to hexadecimal
  local hex_representation=$(xxd -p -c 1000000 <<< "$decoded_bytes")

  echo "$hex_representation"
}


nodeID="$1"
hex_nodeID=$(base58_to_hex $nodeID)
rolloutSeed=$(curl -s https://version.storj.io|jq -r .processes.storagenode.rollout.seed)
rolloutCursor=$(curl -s https://version.storj.io|jq -r .processes.storagenode.rollout.cursor)
hash=$(echo -n "$hex_nodeID" | openssl mac -digest sha256 -macopt hexkey:${rolloutSeed} HMAC)


echo "cursor $rolloutCursor"
echo "hash   $hash"

I will see if this works at the next rollout.

[donald.m.motsinger]

@donald.m.motsinger

I will see if this works at the next rollout.

Now the next rollout started, but this script doen’t seem to do what it should. When I check with the nodeID from a node which received the update, the hash is greater than the cursor.

[Alexey]

You need a little bit more after conversion from BASE58:

• Remove the two leading zeros 00
• Remove everything after the final set of zeros (there should be 7)

[donald.m.motsinger]

I guess you mean the variable $hex_representation in my script. This variable has neither 2 leading zeroes nor 7 zeroes at the end.

[Alexey]

At the end there could be not zeroes. You just need to cut everything after zeroes.
I.e.

The NodeId is 12aTHGbgHd69BeonBQKe27biLUKYAZP1YhsJcHr11GervDPDq8A (from Disqualified without reason)
The base58 decode is 00cfa7cd72c1f7bff4e4f875f1344025904aab0f8f3c9e6e9d9a3c9cf0000000003f8ddddf, so you need to take only cfa7cd72c1f7bff4e4f875f1344025904aab0f8f3c9e6e9d9a3c9cf000000000

[donald.m.motsinger]

@Alexey

so you need to take only cfa7cd72c1f7bff4e4f875f1344025904aab0f8f3c9e6e9d9a3c9cf000000000

Since your example has 9 zeroes at the end, I assume the hex id has a length of 64 characters, correct?

My script above returned cfa7cd72c1f7bff4e4f875f1344025904aab0f8f3c9e6e9d9a3c9cf03f8ddddf0a for this NodeID. Pretty close, but not correct. So I let ChatGPT write me a new script to decode the NodeID.

#!/bin/bash

# Base58 alphabet
BASE58_ALPHABET="123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz"

# Function to convert a Base58 string to a decimal number using bc
base58_to_dec() {
    local base58_str="$1"
    local base58_len="${#BASE58_ALPHABET}"
    local dec_val=0
    local base58_index

    for (( i=0; i<${#base58_str}; i++ )); do
        char="${base58_str:$i:1}"
        base58_index=$(expr index "$BASE58_ALPHABET" "$char")
        base58_index=$((base58_index - 1))
        dec_val=$(echo "$dec_val * $base58_len + $base58_index" | bc)
    done

    echo "$dec_val"
}

# Function to convert a decimal number to a hexadecimal string using bc
dec_to_hex() {
    local dec_val="$1"
    echo "obase=16; $dec_val" | bc
}

# Function to decode Base58 to hexadecimal
decode_base58() {
    local base58_str="$1"
    local dec_val=$(base58_to_dec "$base58_str")
    local hex_val=$(dec_to_hex "$dec_val")

    # Account for leading zeros
    local leading_zeros=$(echo "$base58_str" | grep -o "^1*" | wc -c)
    leading_zeros=$((leading_zeros - 1))
    for ((i = 0; i < leading_zeros; i++)); do
        hex_val="00$hex_val"
    done

    echo "$hex_val" | tr '[:upper:]' '[:lower:]'
}

# Check if the input Base58 string is provided as an argument
if [ -z "$1" ]; then
    echo "Usage: $0 <base58_string>"
    exit 1
fi

BASE58_STRING=$1

# Decode Base58 string and print hexadecimal representation
decoded_hex=$(decode_base58 "$BASE58_STRING"| sed 's/^0*//' |  cut -c 1-64 | head -n1)

rolloutSeed=$(curl -s https://version.storj.io|jq -r .processes.storagenode.rollout.seed)
rolloutCursor=$(curl -s https://version.storj.io|jq -r .processes.storagenode.rollout.cursor)
hash=$(echo -n "$decoded_hex" | openssl mac -digest sha256 -macopt hexkey:${rolloutSeed} HMAC)

echo "cursor $rolloutCursor"
echo "hash   $hash"

For some reason the for loop did introduce a line break near the end into the decoded_hex. As it was after the last zeroes, I just cut it off with head

But the openssl command still seems to create a wrong hash. Some of my already upgraded nodes have a hash greater and some a hash lower than the current cursor.

For the NodeID 12aTHGbgHd69BeonBQKe27biLUKYAZP1YhsJcHr11GervDPDq8A the openssl command creates the hash DADBA98AE4DF87E8868E4381381A37FF2EF948C06FB495860A4BF5EDD24A389C

~$ echo -n "cfa7cd72c1f7bff4e4f875f1344025904aab0f8f3c9e6e9d9a3c9cf000000000" | openssl mac -digest sha256 -macopt hexkey:ac92bbf27e131a4e64ca2ea65696a4678ec8c08d9dcd0a5ca645788642a9d823 HMAC
DADBA98AE4DF87E8868E4381381A37FF2EF948C06FB495860A4BF5EDD24A389C

Where is the error?

[Alexey]

I’m not sure that you need to use a hash of the decoded NodeID, please try to use the decoded NodeID instead.

[donald.m.motsinger]

@Alexey

I’m not sure that you need to use a hash of the decoded NodeID

Why else would there be a seed if it’s not being used?

@Alexey

please try to use the decoded NodeID instead.

Tried with the hexNodeID and the result is similar as expected. Some updated nodes are greather and some are lower than the cursor.

[elek]

While I appreciate the commitment of creating a bash implementation, wondering what is the original motivation of this effort.

• Would it be helpful to support a https://version.storj.io?nodeid=.... call which returns with the current suggested / minimal version for your node?
• Or, Is it acceptable to use a small, static compiled binary which does the conversion (for example: GitHub - elek/storj-id)

@donald.m.motsinger

cfa7cd72c1f7bff4e4f875f1344025904aab0f8f3c9e6e9d9a3c9cf03f8ddddf0a

Note: the last 4 bytes are checksum (sha256, first 4 characters). Your encoding seems to be fine and includes the checksum, the other verison (with zeros) didn’t have it…

[MarviBiene]

Am understanding it right, that the nodes will never upgrade in a random order? They will keep the same order for every rollout?

[donald.m.motsinger]

@elek

While I appreciate the commitment of creating a bash implementation, wondering what is the original motivation of this effort.

I’d like to see which of my nodes should have been updated already and which ones will be updated soon. I just want to put my nodeID into a script or website or … to see if it’s below or above the current cursor.

@elek

Would it be helpful to support a https://version.storj.io?nodeid=.... call which returns with the current suggested / minimal version for your node?

Does this show node specific values or does it show the same info like when you go to https://version.storj.io ?

@elek

Or, Is it acceptable to use a small, static compiled binary which does the conversion (for example: GitHub - elek/storj-id )

I have no idea what to do with those files. In case you haven’t noticed, I’m not a developer.

bash is the only “language” I speak

@elek

Note: the last 4 bytes are checksum (sha256, first 4 characters). Your encoding seems to be fine and includes the checksum, the other verison (with zeros) didn’t have it…

This kind of went over my head too.

Just to recap, I have a nodeID (12aTHGbgHd69BeonBQKe27biLUKYAZP1YhsJcHr11GervDPDq8A) and successfully converted it to hex (cfa7cd72c1f7bff4e4f875f1344025904aab0f8f3c9e6e9d9a3c9cf000000000). Now I want to create the sha256 hash from it with

echo -n "$hex_nodeID" | openssl mac -digest sha256 -macopt hexkey:${rolloutSeed} HMAC

in order to compare it with the current cursor.

But this command seems to give me the wrong result. What would be the correct command? I only struggle at the very last hurdle.

[elek]

Ah, thanks to explain. Now it makes sense.

@donald.m.motsinger

What would be the correct command?

I think you try to openssl mac the string representation of the bytes instead of the raw bytes.

Try echo -n "$hex_nodeID" | xxd -r -p | openssl mac -digest sha256 -macopt hexkey:${rolloutSeed} HMAC

xxd -r -p will convert the hex back to binary what openssl expects…

[elek]

@raert

standard input uses hexadecimal text format.

Interesting. What is the source of this quote? I don’t see such text in my man openssl. Neither in openssl mac help.

I am pretty sure that my openssl uses binary input, because I compared it with the golang implementation:

	id, err := storj.NodeIDFromString("12aTHGbgHd69BeonBQKe27biLUKYAZP1YhsJcHr11GervDPDq8A")
	if err != nil {
		panic(err)
	}
	raw, err := hex.DecodeString("ac92bbf27e131a4e64ca2ea65696a4678ec8c08d9dcd0a5ca645788642a9d823")
	if err != nil {
		panic(err)
	}
	hash := hmac.New(sha256.New, raw)

	_, err = hash.Write(id[:])
	if err != nil {
		panic(err)
	}
	fmt.Println(hex.EncodeToString(hash.Sum([]byte{})))

It prints out 1068360a59fddaae835c74e7239f6f7d826e33471e8fe886b179611cfc9728c6

Same as:

echo -n "cfa7cd72c1f7bff4e4f875f1344025904aab0f8f3c9e6e9d9a3c9cf000000000" | xxd -r -p | openssl mac -digest sha256 -macopt hexkey:ac92bbf27e131a4e64ca2ea65696a4678ec8c08d9dcd0a5ca645788642a9d823 HMAC
1068360A59FDDAAE835C74E7239F6F7D826E33471E8FE886B179611CFC9728C6