Update Process (Filewalker) + When to update

Node Operators
,
1

Hello,
I have some questions regarding the update process of the docker image.

I am updating manually as it gives me more freedom to update when I have time to maybe also troubleshoot my nodes. From the thread about updating I am getting, that checking for updates every 2 weeks should be fine.

Can someone tell me more about the file walker process which allegedly runs after every update? What is it doing, how much RAM needs it, is it checking every file? How is the file walker handling network drives? (I know not recommended, deal with it ;D)
At the moment I am updating when I really have to because I donā€™t want to stress the systems too much.

Also: This command: curl -s https://version.storj.io | jq .processes.storagenode.minimum.version gives me 1.16.1 @ the moment? What does this mean? Isnā€™t it the latest version available and not the minimum?

Thanks!

PS: I already read this thread: Keeping your node up to date, changes to storage node software! PLEASE READ this thread!

1 Like
2

I think you are on your own there because you are running an unsupported configuration but there may some other ninjas who run it like you.

You should update to 1.16.1, it has gone to all the windows and now to docker too.

1 Like
3

Thanks for your answer!

I know, that network drives are ā€œunsupportedā€, but my question is more in general.

What does the file walker do? Is it running after each and every update? Is it running in between updates?

What about this command? Why is it returning the actual version and not the minimum?

1 Like
4

OPā€™s mistake was admitting theyā€™re not following the ā€œofficialā€ procedure. Now we wonā€™t get any answer for the actual question. Instead, weā€™ll get ten answers telling you how bad it is to not use the official updater.

Sorry for your loss, HeroHann.

3 Likes
5

I still have hope ;D

2 Likes
6

I think it only checks the metadata or presence of files, because on my node it usually only accesses the cache.

I also update manually, but I do it whenever there is a new version available. I think it ok to ā€œstress the systemā€ - better find out that a drive is failing quickly than wait until multiple drives have failed. I also run zpool scrub once a month (it is so by default and I have not disabled it) - again, better to find out that a drive is dying sooner rather than later.

1 Like
7

Thanks for your reply. Have you ever monitored the reads and writes on the drives after an update? Does the file walker have Abigails impact?

I usually wait for a week or so after a new version is available due to new bugs in new versions.

Can you tell me anything about this curl command?

1 Like
8

Iā€™ve been after some more information on this too as my drive seems ā€˜busyā€™ for a week after an update or storagenade service restart.

drive.io.

The fact itā€™s not actually reading or writing much suggests it is checking metadata or file presence like @Pentium100 suggests.

The curl command just reads in the json data from the https://version.storj.io response. Try it in a browser.

1 Like
9

Thanks to you!

Yeah it looks like the file walker is not ā€œdoingā€ much but the drive is at 100%. Metadata as you said. Will try with my network drives.

To the command: Than I donā€™t get what this should be telling me. I thought it would tell me the last supported version, so I know when to update. But instead it tells me the newest version.

Where do I get the info, which is the latest supported version from which I really HAVE TO update?

10

I donā€™t have the official answer but it wouldnā€™t surprise me if 1.16.1 is now the latest and only supported version due to the recent order errors and fixes from the previous 1.15.3 version.

@Alexey might have the official answer.

1 Like
11

When processes.storagenode.minimum.version == latest version you should update else you know the PS part of your post.

12

Maybe I donā€™t get it or I am just dumb.

When do I have to update?
I am running 1.15.3 now and 1.16.1 is out (some people are having problems).
When is my time to absolutely do the switch?

PS part?

13

Since you are not relying on automatic update then you should update when the rollout is 100% i.e. all nodes can update GUI/docker/others.

Post Script.

14

Yeah I SHOULD update but as you might have seen in the forum some people are running into problems. So with everything I own I wait some days / weeks to update if it is not absolutely necessary. Thats why I ask not when I should but rather when I MUST update.

15

yeah same hereā€¦ delayed my update just to wait and see if more problems start flooding the forum.

within the next month or soā€¦ i think the rule is 3 versions behindā€¦ i forget if thats called major or minorā€¦ so on 1.15.3 the latest you should updated would be 1.17.xā€¦ maybe 1.18.x i suppose that depends on how the 3 version difference is countedā€¦ i mean 1.15.3, 1.16.1, 1.17.x would be a range of 3 versions from when it was updated. making 1.18.x when one would get suspendedā€¦ while if one says 1.15.3 + 0.3.0 then it would be 1.18.x that would be the last and 1.19.x that would get one suspendedā€¦

but i would personally try to stay close to the most recent rather than lagging behindā€¦

havenā€™t really looked into all that because i donā€™t plan on getting that far behind.

if there isnā€™t any major issues for people that keep popping up on the forum i expect to update in the next few days.

and was it a suspension thoā€¦ or was it a DQ i forgetā€¦ like i said, you donā€™t want to test that stuff really, especially if the punishment was DQā€¦ i remember getting a bit insulted by it, so may have been DQ

but not really relevant because i donā€™t plan on finding out where that limit isā€¦ even if i may skip 1 version if it turns out to be problematicā€¦ ofc skipping one update makes oneā€™s node slightly different from the norm and might in itself cause problems long termā€¦ in theory atleastā€¦ but who knowsā€¦

the filewalker has so far as iā€™m aware been running on every reboot of a nodeā€¦ not sure if that got changed, i have complained a bit about it.
the filewalker is very iops dependent, thus the slower your storage solution iops the longer it will take.
my 14.5tb node takes a few hours usually, so itā€™s really annoying to troubleshootā€¦
but i got large scale caching and high iops so not to badā€¦ but i have done testing running on 1x hdd iops, 2x hdd worth iops and 3x worth of iops and it basically scales linear.

so in theory on a system of say 30TB with 1x 7200 rpm hdd iops and no caching the filewalker seems to take upwards of 8 hours or more, tho there are some peak utilization in graphā€¦ but it still takes a long time to finishā€¦

there really isnā€™t any fix for it, aside from caching or having good read iops.

a poorly configured raid array of 30TB filled with data doing the filewalker might take a day or more to completeā€¦

1 Like
16

Well, it says on dashboards that 1.13 is the minimum version:
image

I read somewhere that the oldest version would only have a limited support, as in you wouldnā€™t receive anymore data or something like thatā€¦ Canā€™t find it back though.

@HeroHann I think you could delay your updates a bit, for instance you could be one version behind. But I wouldnā€™t go beyond that, as StorjLabs clearly want all nodes to be as up-to-date as possible.

1 Like
17

yeah when you go below the minimum required version ingress will go to zero.

18

The filewalker is a horror on my SMR drives; especially if you are trying to do anything that might cause a large amount of writes (I changed some BTRFS mount settings to disable CoW, and needed to ā€˜applyā€™ to the existing data by copying to a new spot or running a defrag), while drives slowed down to < 10 MB/s total throughputā€¦ Finally finishedā€¦ The cumalitive load of everything was not good; long story is definitely avoid SMR for StorJ if you can, although you can run itā€¦ A RAID might help, I havenā€™t tested that.

1 Like
19

That perfectly describes what it does to my SMR drives too ^^ā€™
It will become worse and worse the more data disks are storing by the way, maybe something should be considered to make this process lighter on drives!

Having several nodes (on several disks obviously) does help too, as it spreads loads on all disks.

1 Like
20

actually raid setup with SMR would be about the sameā€¦ since itā€™s the iops thatā€™s the SMR HDD limitation and with a standard raid array the iops is the same as with a single HDD due to all raid drives writing in harmony with each other or whatever one wants to call it.

the best way to utilize SMR drives will be using multiple smr hdd on multiple nodes and thus the data sent is split across themā€¦ ofc that doesnā€™t make the filewalker any less of a problem, tho since each drive would essentially only have half the data then it should, atleast in theory take about half the time.

so if you have a number of SMR drives making multiple nodes is a pretty good fix, tho when they are full the problem will be the same in regard to running the filewalkerā€¦

raid only increases the write and read bandwidth, not the iopsā€¦ sadlyā€¦ would be wonderful if it did, but yeah bandwidth does increase, giving you HDD MB/s * (Number of drives - Redundancy Equal Drives)
so in a raid 6 with 6 drives you get 4 x base speed because 2 of the drives are the redundancy.

plainly put, as always itā€™s a bit more complex than that.

1 Like