Weird behavior around nodes

[Various69]

I was following that tutorial Step 1. Understand Prerequisites - Storj Docs
That says minimum 3 down 1 up, recommended 5 down 3 up.

@arrogantrabbit

bufferbloat on upstream. Op needs to enable SQM (fq_codel, smart queues, “gaming mode”, whatever your gateway calls it). Use this test from wired connection to confirm. https://www.waveform.com/tools/bufferbloat

I’m using commercial router, there are no problems with overwhelming.

[Various69]

@snorkel

That 60mbps drops your requests to upload pieces (downloads from client) and audits.
That’s your problem.
When the testing was high, I had a limit of 150mbps that dropped my scores also.

I didn’t notice on router more than 30mb download and 11mb upload. So I think the problem is not there.

[Various69]

@Alexey

This must be executed only once for each identity and never again, otherwise you may destroy the node by making a mistake in the path.

I know that I read whole tutorial from this site:
Step 1. Understand Prerequisites - Storj Docs

@Alexey

I would suggest to move each identity to its disk with data, they are useless without each other anyway, and much less mistakes like mixing identity with a wrong disk in multinode setup

Perhaps for someone but I have my own weirdness and I like to store thinks in one place to have better “view” on them to not make mistake.

@Alexey

Perhaps docker logs already rotated, the local logs driver has 5 files by 20MiB each by default, and 100MiB is not enough for the historic research. I would recommend to either to redirect logs to the file (you would like also to configure a logrotate service) or increase the amount and the logs size for docker.

I’m gonna try that.

[Various69]

From other hand I think I’m gonna nuke that few problematic nodes and make that disks RaidZ to see if there maybe is problem with disks but I can’t see anything wrong in smart or speed test.

[arrogantrabbit]

@Various69

I mean disk bandwidth, read/write.

I figure that that’s what you meant, I tried to hint to you that bandwidth is irrelevant. Only IOPS matter. HDD can sustain about 200 IOPS. Stroj writes a lot of 16k files. 16k*200 = 3.2Mbps with no overhead. That’s the theoretical limit of your HDD. In reality you have metadata to update, and some files are m much smaller.

@Various69

Ok but why saltlake.tardigrade.io:7777 is everywhere online and rest are random?

Because there is no activity on saltlike. It’s a test satellite.

@Various69

standard ddos and ICMP flod protection

Turn it off and check if anything changes.

@Various69

I have full view in mikrotik on what is blocked

You have a view of what microtik considered blocked by design. What else is dropped accidentally you don’t see.

@Various69

15-20% load

20% of what? CPU utilization? Irrelevant. Memory consumption? Irrelevant. Number of connections it tracks at one time? Maybe relevant.

@Various69

I didn’t notice on router more than 30mb download and 11mb upload. So I think the problem is not there.

Again you are conflating iops and bandwidth. 10MB transferred in one session vs 10MB transferred in 10000 sessions 1000 bytes each has vastly different effect and performance requirements.

You claimed that you are IT twice already but I’m baffled I have to explain these basics to you.

Run the buffebloat test. It’s going to be faster than arguing why you should not do it. Confirm you get an A so we can rule this out and move on to other suspects.

@Various69

So I think the problem is not there.

Let me get this straight. You have a problem. Then you explain away and reject every suggestion without testing. What’s your plan?

@Various69

make that disks RaidZ to see if there maybe is problem with disks

Why don’t you check IOPS as suggested in the very beginning instead of randomly messing with system?

@Various69

I can’t see anything wrong in smart or speed test

Again. Throughput vs latency. NOT the same thing.

[nyancodex]

don’t use any kind of RAID

[Julio]

This is pretty funny stuff guys, thanks for the entertainment.
If you don’t marinate your meat in a plastic bag, chew think too highly of yourself!
Seriously, backtrack - reconsider your opinions with that have been proposed here.
A raid Z would just server to conflate and bias read vs. write problems, and be entirely useless - not an option. If you want to ignore it, and that’s your style… maybe raid 10 it… lol, money to burn! Woot.

[andrew2.hart]

@Various69

Third problem is that the 7th node just hang out and container restarts didn’t help, I had to clone/reconfigure node to make it work again

Is it possible that you made a mess of 7 and it made a mess on the other nodes?

Edit oops remove language