Bandwidth utilization comparison thread

[SGC]

actually you can remap the uid and gid i think it’s called…
doing that you can fix the permissions… ofc that does introduce some additional security holes, but can’t really get much worse than what i’m already doing when running it directly on the host…

or thats how i understood it from reading abit about it… i’m sure it’s wrong in more ways than i want to count… but it werks… i think… haven’t exactly done a ton of testing on it yet
just copied a script from it… but i have my zfs permissions setup in that way where the owner gets the rights on stuff they create and then others get read only access… and ofc the lower tier users cannot read root owned… i think … something like that… was setting up some network shares and had trouble so i know i did a good deal of tinkering to get that to run as i wanted…

seems to work well for most stuff… anyways on this link there is a guide to how to make lxc containers access host storage… not sure if its applicable outside of proxmox…
i did these changes…

First the file  `/etc/subuid`  (we allow 1 piece of uid starting from 1005):
root:1005:1
then  `/etc/subgid` :
root:1005:1

https://pve.proxmox.com/wiki/Unprivileged_LXC_containers

tried to add this to the .conf

# uid map: from uid 0 map 1005 uids (in the ct) to the range starting 100000 (on the host), so 0..1004 (ct) → 100000..101004 (host)
lxc.idmap = u 0 100000 1005
lxc.idmap = g 0 100000 1005
# we map 1 uid starting from uid 1005 onto 1005, so 1005 → 1005
lxc.idmap = u 1005 1005 1
lxc.idmap = g 1005 1005 1
# we map the rest of 65535 from 1006 upto 101006, so 1006..65535 → 101006..165535
lxc.idmap = u 1006 101006 64530
lxc.idmap = g 1006 101006 64530

but that made the container not boot, so i removed it and the other changes seemed enough to make it work… i duno if they actually do anything… had also an extra space in the mount configuration in the .conf file

used this one

mp0: /mnt/bindmounts/shared,mp=/shared

another thing i did tho in the proxmox webgui, was going into the container tab
selecting options: selecting features and clicking the edit button in the top…
then adding nesting and keyctl.

you can however add those atleast for proxmox lxc, in the .conf
looks like this when i check it.

features: keyctl=1,nesting=1

in case you wanted to try and get past the 777 issue

my full LXC .conf file ended up like this

arch: amd64
cores: 4
features: keyctl=1,nesting=1
hostname: CT303
memory: 4096
mp0: /nexus/publicshare,mp=/mnt/publicshare
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=DE:41:4a:14:4B:EE,ip=dhcp,type=veth
ostype: ubuntu
rootfs: tank_vms:subvol-303-disk-0,size=4T
swap: 512
unprivileged: 1

oh yeah i forgot to mention some of these settings was to get docker running inside the container also… but i’m sure you know that
and i duno why it works… magic i guess lol
i just copy stuff…

[dragonhogan]

Another good day of egress.
14Aug2020:
Node 1:

Node 2:

[kalloritis]

Node one for you saw a behemoth amount of repair egress- I hope that doesn’t mean we’re seeing some churn of older nodes that have been holding onto some of that data and are now weakening the network.

Fiber node (4mo):

Coax node (3mo):

[callmestu]

That’s a possibility - I’m focusing on the Filecoin “Space Race” recently. I wonder if that’s a draw for folks. It is good for those with large data repos. Even the 100GB/day upload illustrated above is 1/10th of what I was hoping to achieve in a few months of maintaining these boxes. I’m relatively bullish on cloud storage (I mean, my minor in grad school is big data) but we seem to be handling mostly test data, and that is too bad. Customers - where are you?!

[kalloritis]

Well based on some of the anecdotal notations of a few SNO’s that have also submitted their own testing data to tardigrade- there’s a small bevvy of limitations that needs to be solved for before Storj can start to approach even Sia (600-860TiB of 2.2PiB) or Backblaze (multi-Pebibyte) utilization levels.

The one that strikes me, top of mind, is that 1TB is “a lot” as a single customer- which is just not true. At the office, I could dump about 20-40TiB of data to offsite storage and just write that as DR cost, but if I’m kneecap’d at ~1TB, and a bandwidth limit to boot, then I wouldn’t even begin to trial it. For example, a single endpoint’s 3mo of file backups, with dedup&compression, is still ~1.2TiB- before I even start getting to the 1.7TiB of image backups (developement/Graphics design endpoint).

[BrightSilence]

You can request a limit increase. It’s only there so Storj Labs gets a heads up if a customer suddenly wants to upload PB’s of data.

@kalloritis

there’s a small bevvy of limitations that needs to be solved for before Storj can start to approach even Sia (600-860TiB of 2.2PiB)

I have no idea what makes you say that. This is a quote from April.

@john

We have a few PB of test data on the network now, but we expect that to grow to around 8 PB (pre expansion factor) over the next few months.

Storj is several times larger than Sia already. Probably an order of magnitude larger.

[TheMightyGreek]

Node churn is actually a big problem, especially if disqualifications are frequent during GE.

Is there such a thing as an "un-disqualify"?

Please, read the Storage Node Operator Terms and Conditions in This is $10/TB, which Storj Labs should pay to only SNO, when download pieces from storage nodes to the repair job service, then recombine the file and upload all files back to the network, which are charged by the cloud provider of the compute service where is repair job running (all cloud providers charge for egress traffic). So, the total costs of repairing of 7.2TB of data will include the payouts to SNOs and charges for egress…

Alexey did a breakdown of what it costs them to repair data when a node gets disqualified and it’s astronomical !!
I’m actually worried that it might cause BIG problems if many nodes exit the network (not gracefully) in a short period of time. I really hope that they are actively working on a solution (should be one of the top priorities) because at the moment the held amount only covers a fraction of the repair cost.
There is a mention about repair being distributed across the nodes in the whitepaper but I guess that they haven’t gotten around to it yet. That would greatly increase the decentralization of the network as well as reduce the cost of rented servers.

Anyway that’s just my 2 cents on what Storj should work on next. I think @SGC infected me with the “long post with lots of digressions” virus haha

[BrightSilence]

@TheMightyGreek

I really hope that they are actively working on a solution

They are. The biggest reasons for disqualification that can be prevented are:

• Nodes losing access to storage >> https://review.dev.storj.io/c/storj/storj/+/2272
• Nodes failing graceful exit >> Several were fixed already, others are being actively worked on. https://forum.storj.io/t/reproduction-and-solving-issue-with-ge/8377

Furthermore, the repair threshold is currently set very high, partially to test that repair works as intended. But this also leads to much more repair than needed.

[SGC]

i had this happen today… caught it because the color awk log script was running in a window… i really need to setup so alerts lol

Creating identity on storagenode host, seemingly disrupted existing storagenode identity

[image] I was in process of creating a new identity when i ran into this issue… upon completition, or within minutes there off… i cannot say exactly because i didn’t have timestamps on my terminal… tsk tsk rookie mistake i know but i have never seen this issue before and tho my machine is doing a little work… nothing that should cause this… i think… also should be pretty simple to replicate on a computer thats faster than mine in generating keys, if there indeed is a problem… seems…

[shoofar]

@SGC

well many people seem to have the horrible idea that 99.9% uptime and 99.9% data reliability can be reached with consumer grade setups…

Well… that’s kind of idea behind “every Joe” having a node isn’t it?
The horrible idea comes from tardigrade right? with their quote “make a node for fun and profit?”

And as for me and my experience of my node fragility- I am not too keen to shell out spare hundreds of dollars just to make Tardigrade business happy… With ridiculously low egresses? (in my country hundreds of dollars don’t come too easy…)
Unless one is a philanthrope and wants to fund their business… or has own agenda for spare hardware if this project won’t work out.
So I would say - For one’s home-made business expect home-made effects and unfortunately home-made flakiness and through that ->fragility of the network. (one month they are all for it, next they are going on vacation and have to turn off the computer… or even better - they go on vacation and everything crashes (my life’s experience )
Or as my friend did in their new flat - she put a power tester in a wall socket - just to check if there is power - blew up whole vertical skeleton power line - they had to go to the building basement and reset the breakers that took a few hours… imagine you are living with such a “talented” neighbours…
It might have resulted in maybe malformed DBs if no UPS…

I would shell out cash … better yet I would gladly do it through “self funding”, but will not risk own major cash… at least not at the beginning…
This is a conundrum … won’t make cash without more space-> won’t have space without cash

[SGC]

i have made my fair share of bad choices for this… first off i bought used harddrives… because… well they seemed cheap… but long term is certainly not going to end up having been cheap… because the failure rate is … cough an interesting data point that makes me want to throw stuff…

the table i got my server on i bumped on day and basically killed a hdd right there… atleast it was a 3tb one… then ofc the server room isn’t heated yet… so last winter the disk endured -17c
aside from that i wasn’t aware about the whole dew point and temperature thing… or atleast i didn’t take it into consideration before running the server in an unheated room…

so yeah it corroded, so much so that the fan wires literally shock the wires loose and the fan’s stopped workings… then i bought a wrong raid controller and components for 350$ only to replace it with 100$ of better gear within 2 months…

my system is setup to deal well with power outages… because i worry about them even if it’s most likely not even a yearly thing…

so that’s not really my consideration… however i do have a leaky roof way to close to where the server is located…

there will always be problems… only question is if you are bothered by problems or enjoy the challenge of finding solutions.

[dragonhogan]

15Aug2020:
Node 1:

Node 2:

[kalloritis]

Still seeing those super high repair egress, vs June/July, on the slightly older node, and a good bit fuller- but still odd that the month of August has been flowing a lot of repair egress.

Fiber node (4mo)

Coax node (3mo)

[TheMightyGreek]

Looks like egress is increasing significantly, still very low ingress, I’m wondering when they’ll resume testing.

[dragonhogan]

17Aug2020:
Node 1:

Node 2:

[dragonhogan]

Are we still doing this?

18Aug2020:
Node 1:

Node 2:

[TheMightyGreek]

seems like the thread is starting to die down.
We did prove that ingress is almost exactly the same for all nodes. As for egress that will highly depend on the age of the node.

[shoofar]

@TheMightyGreek

As for egress that will highly depend on the age of the node.

Not sure if age is the only factor. I think that the more data your node has, the higher chance of getting egress.
You can have 6 month old 2TB of capacity node that is fully filled. But I doubt that you get more egress than 3 month 6TB node and I would hypothesise that sometimes it can have higher egress, but higher capacity will prevail?

My data for this month, egress is creeping up. Now it is at levels that are quite okey-ish for “per TB” basis.
Although small amount of stored data is not helping…
And what is interesting - log monitoring shows that there is a handful of errors stating that some files are not found… (little unnerving I would say)
Also an observation - as my secondary node gets more and more vetted, the ingress gets lower for primary node (secondary node is evening the ingress more and more)
Summing up ingress and egress for both nodes gives similar results as you guys posted above.
I will try to nerf secondary node free space to see if ingress can be rebalanced.

(blue line is ingress (and should be horizontal/creeping up, not going down), red line egress)

Date	IngressT	EgressT [GB]	StoredT [GB]	egress ‰	egressT	EgressT [kb/s /TB]
02.08.2020	13.3	4.32	1.97	2.19	50.03	25.39
03.08.2020	11.62	3.27	1.98	1.65	37.86	19.12
04.08.2020	19.41	5.96	1.99	2.99	68.98	34.66
05.08.2020	32.14	7.29	2.03	3.59	84.39	41.57
06.08.2020	43.68	6.71	2.06	3.26	77.66	37.7
07.08.2020	49.5	7.8	2.1	3.71	90.23	42.97
08.08.2020	47.37	4.52	2.15	2.1	52.29	24.32
09.08.2020	38.48	7.68	2.17	3.54	88.91	40.97
10.08.2020	39.98	8.38	2.21	3.79	96.98	43.88
11.08.2020	40.28	8.7	2.24	3.88	100.65	44.93
12.08.2020	36.84	9.64	2.27	4.25	111.58	49.15
13.08.2020	35.6	11.84	2.3	5.15	137.06	59.59
14.08.2020	20.09	6.77	2.32	2.92	78.39	33.79
15.08.2020	30.71	15.2	2.34	6.5	175.98	75.2
16.08.2020	27.06	16.17	2.37	6.82	187.18	78.98
17.08.2020	22.69	15.24	2.38	6.4	176.41	74.12
18.08.2020	20.82	16.39	2.4	6.83	189.71	79.04

[TheMightyGreek]

Yes of course, I should have precised that I was talking about nodes with the same amount of data stored.

I started a second node almost two months ago and now that it’s fully vetted (except for stefan-benten) the ingress is almost perfectly split between the two nodes (the most I saw was a 7% difference).
Also my egress permille is almost exactly the same as yours over this period of time.

[SGC]

i think the only path forward is to have some script extract and compile the data for like say 1month at a time… so its nice as each to post / share / compare

seeing some pretty good numbers these days tho