Announcement: major storage node release (potential config changes needed!)

[snorkel]

@peem Yes. You need to wait for startup-piece-scan to finish. It used to show an entrie in the log, but is broken for some and dosen’t.
The other way to check when it is finished is to watch the HDD usage. When it’s 100% piecescan is working. When drops to 20-30% and stays there for more than 15 min, then the scan finished. After the scan finishes, the database is updated with the correct values and the dashboard displayes the reality.

[bre]

The warning was for anyone using inappropriate language in this thread.

We are all adults here.
It is reasonable for adults to know to use appropriate language in the Forum.

One would think it is obvious that cussing or crude references to human biology are not appropriate. If you did not know this, consider this your one time tutorial.

[littleskunk]

@snorkel

That works for who uses config to set parameters. I only use docker run to set everything.

As I mentioned. For running multiple docker nodes I would simplify the config in one central env file and use docker compose to apply that to all storage nodes. That way it is one line to change and one docker compose compand to restart all nodes.

[Alexey]

@digitalfrank

Do I have to do something for the new filesync setting or does it leave everything by default and therefore without inserting anything into the yaml file and therefore automatically takes the new configuration? i.e. the fastest one but with less security for system crashes? Thanks but even reading the post I didn’t understand if something needs to be done to switch to the new quick configuration or if it’s enough to do nothing.

No, the new default is fsync disabled, yes, it could be less secure on unstable systems. If your system didn’t randomly restarts and do not have random power outages, you do not need to do anything.

[Alexey]

Filewalker status

Logs should be at least info level Used space filewalker docker logs storagenode -f 2>&1 | grep "used-space-filewalker" | grep -E "started|finished" GC filewalker docker logs storagenode -f 2>&1 | grep "gc-filewalker" | grep -E "started|finished" Trash-cleanup filewalker docker logs storagenode -f 2>&1 | grep "trash-cleanup-filewalker" | grep -E "started|finished" retain docker logs storagenode -f 2>&1 | grep "retain" collector docker logs storagenode -f 2>&1 | grep "collector" trash docke…

[snorkel]

I have 2 nodes per machine, each machine in different locations. So the centralisation wouldn’t help much.

[snorkel]

I don’t mind cursing and free of speach as long as it’s not addressed to members of this forum directly. We are all grown men/women, we are not kids. But of course, with limits.

[bre]

Enough.

If you need to express yourself in these ways please do it elsewhere and return when you can comport yourself in a way commensurate with the Code of Conduct.

You have escalated now from Reminder status to Warning status.
The next status level is Forfeiture under the “flagrant” category.
The choice is yours.

[bre]

But it’s not a question of what individuals mind.
It is a clear violation of the rules of participation.

We must maintain the standards for everyone equally.
I really hope we can all let this be the end of this issue.
It is a disrespectful waste of everyone’s time.

[ACarneiro]

[bre]

@IsThisOn

But you also should be indulgent with me,

No, that is not possible.
The Forum has rules.
The rules apply to everyone equally.
The rules are clearly written
Code of Conduct

By participating in the Forum, you implicitly agree to abide by the rules, the same as everyone else does.

You have wasted enough of everyone’'s time on this issue.
You know how to act like an adult and I hope you choose to do so here forward.

You have received your last and final warning.
I accept your apology and am glad it will not happen again.

[pdeline06]

@littleskunk

storage2.piece-scan-on-startup: false

Is it current position saved when running filewalker (storage2.piece-scan-on-startup: TRUE) or will scanning start from 0 after the node is restarted?

[jammerdan]

@pdeline06

Is it current position saved when running filewalker

The feature still seems to be under review:

https://review.dev.storj.io/c/storj/storj/+/12806

It seems that the new feature “save-state-resume GC filewalker” isn’t functioning as expected

Quick intro to the development process: We merge all commits after +2 approval of two independent reviewers + green build stroj-gerrit-verify is the first past of the build, we have an other round after the two reviews (pre-merge) usually we cut release branches bi-weekly, unless there are urgent changes Looks like the initial build is passed, so this patch seems to be ready to review. Now, we need two reviewers / all the suggested modification before a release cut. Because the preparation i…

[pdeline06]

This is bad. I’m afraid to even think how long the filewalker process can take for 70 million files. Surely something will happen and the node will restart. And everything will start from 0
In this topic they write that in order not to show terrabytes of garbage in the dashboard, you need to run filewalker to update the node databases. But I can’t imagine how this can be done on nodes with 70 million files without the scan saving function

[jammerdan]

@pdeline06

This is bad.

Correct.
I have turned filewalker off on several nodes because of that.

I don’t know what is halting the completion of the feature and deploy it. I have asked to get this done several times:

Updates on Test Data

Good. SNOs are used to pain. How much space do you need? Speaking for myself I can add space. I am sure other can too. Please solve this: https://review.dev.storj.io/c/storj/storj/+/12806 I have turned off used space calculation on several nodes so they are reporting garbage. We are getting used to get stabbed in the back.

It seems that the new feature “save-state-resume GC filewalker” isn’t functioning as expected

Not good. That feature is really needed to help when a used space filewalker does not finish for whatever reason.

Database locking madness

Iterating of every file not just once in a while but frequently, maybe that was an idea easy and cheap to implement when nodes were at 500GB. But now with nodes at 15TB this is pure madness. And on top the frequent restarts after updates and everything starts all over from the start except when data was kept in RAM only like with the Bloomfilter until recently fixed. Plus independent from the load aka customer usage of the node it gets hammered by filewalkers, database accesses and all. And …

[nerdatwork]

@jammerdan

I don’t know what is halting the completion of the feature and deploy it.

It shows pending code review and

[jammerdan]

@nerdatwork

It shows pending code review

Yes, but no additional information since then. So I don’t know what’s going on with this behind the scenes.

[Dr.Ko]

There are differences in Ingress between 2 nodes running in different locations depending on the version. I think this is because fsync is disabled.

[Roxor]

There’s no extra ingress yet for 1.104 to take advantage of, is there? SLC hasn’t been pushing data since the start of the month… so this must be normal customer traffic… which could coincidentally be closer to your 104 node?

I’m patiently waiting for the torrent of ones and zeros to start falling from the heavens!

[Dr.Ko]

@Roxor

There’s no extra ingress yet for 1.104 to take advantage of, is

I thought it was because of fsync
I guess there is another reason