Storagenode Operational updates

Node Operators
1
  • This forum topic supposed to give more insights about Storagenode operations
  • Similar to https://status.storj.io/, but focused to SN
  • This may help in case of debugging issues, but shouldn’t be required to be followed for normal operation
  • Please consider opening dedicated thread for specific problems

Current status, notable, known issues:

  • There is an ongoing investigation about not-deleted trash folders, which uses the new structure (date based).
  • Storagenode dashboard may show bigger discrepancies due to using 1024/1000 based units in different ways
  • Storagenodes may have bigger storage overhead because previous BF issues. Multiple successful BF processing might be required

History:

  • 2024-03-22 v1.97.3 rollout has been started
  • 2024-03-23 v1.99.3 rollout has been started
    • storagenode log levels are configurable
  • ~2024-03-28 Bloom filters are not sent out due to a permission problem with temp buckets
  • 2024-04-10 New problem identified: max sized (4100003) bloom filters were not sent out to big nodes
  • 2024-04-14 US1/AP1/EU1 bloom filters are generated successfully and sent out to all nodes
  • 2024-04-14 v1.100.3 rollout has been started
    • trash pieces immediately 52881d2db
    • support big bloom filters 87611d002
    • per-day trash directories 21de0c3fd
    • save-state-resume GC filewalker
  • 2024-04-15 v1.101.3 storagenode rollout has been started (instead of continuing v1.100)
    • save-state-resume feature for GC filewalker 2a8d0d44a
    • important gc file walker fix (don’t use v1.100!)
  • 2024-04-23: v1.102 rollout has been started
  • save-state-resume feature for GC filewalker (0f90f061b)
  • some logging changes
  • 2024-04-24: we started testing 10Mbyte (max size) bloom filters. New enough nodes may receive these bigger BFs. (We are planning to send out backward-compatible, max=4Mb BFs as well, especially for US1.
  • 2024-04-29: Ongoing investigation of not-deleted, date-based trash folders.
26 Likes
3

I would prefer history in ascending time order, to easily follow the events. In order to understand the now, we need to read the past, so it would be naturaly to read the updates in chronological order.

6 Likes
5

I would prefer history in ascending time order, to easily follow

I am fine with both sort orders. But having the latest news on top, helps to understand recent changes faster, IMHO. You don’t need to scroll to down. (Eventually we will have lot of events, but we can prune it to an archive post after a while…).

Also the reverse-time order is more change-log style, where the most recent release is on top. ((But changelage doesn’t contain timestamps, so it can be confusing).

2 Likes
6

Great Job, Storj
Number One of Cloud Distributed Storage

5 Likes
7

I completely agree. I think the label “History” automatically makes us want to see things in chronological order in order to see the progress. While in terms of changelog, a user is going to use the latest version of the software hence showing what is included in the latest version at the top makes sense.

3 Likes
8

New at the top makes the most sense. And thank you for using the YYYY-MM-DD syntax as the ISO intended… and not some bass-ackward Americanism :wink:

5 Likes
9

Or just a sortable table, so everyone can choose his ordering :wink:

10

3 posts were split to a new topic: It seems that the new feature “save-state-resume GC filewalker” isn’t functioning as expected

13

I think that we can use a history of edits for this regards?

14

we are not all from Americas :wink:

1 Like
15

Do you know, how it could be showed on the forum?
Perhaps only the table view, but I’m not sure that it could be sorted without editing…

16

I would suggest that instead of making this into a complete changelog, only the important dates+versions+info should be kept.

Ie:
2024-04-15 v1.101.3 storagenode rollout has been started (instead of continuing v1.100)

  • save-state-resume feature for GC filewalker 2a8d0d44a
  • important gc file walker fix (don’t use v1.100!)

Or in other words, “update to this version asap because we fixed so and so”.

IMHO “regular” SNOs (ie those running a couple of nodes on auto updates) don’t even have to bother with this, unless they notice something wrong (ie a version downgrade). Bigger SNOs (ie those running dozens or hundreds of nodes, manually updating) will be interested in this topic since it gives important information about edge cases (ie bigger nodes and trash).

17

Yes, my idea was to put the important, actual information under Current status, and keep History for posterity…

Probably the warning about v1.100 deserves a line under the current status…

7 Likes