Another good day of egress.
14Aug2020:
Node 1:
Node 2:
Node one for you saw a behemoth amount of repair egress- I hope that doesn’t mean we’re seeing some churn of older nodes that have been holding onto some of that data and are now weakening the network.
Fiber node (4mo):
Coax node (3mo):
That’s a possibility - I’m focusing on the Filecoin “Space Race” recently. I wonder if that’s a draw for folks. It is good for those with large data repos. Even the 100GB/day upload illustrated above is 1/10th of what I was hoping to achieve in a few months of maintaining these boxes. I’m relatively bullish on cloud storage (I mean, my minor in grad school is big data) but we seem to be handling mostly test data, and that is too bad. Customers - where are you?!
1 Like
Well based on some of the anecdotal notations of a few SNO’s that have also submitted their own testing data to tardigrade- there’s a small bevvy of limitations that needs to be solved for before Storj can start to approach even Sia (600-860TiB of 2.2PiB) or Backblaze (multi-Pebibyte) utilization levels.
The one that strikes me, top of mind, is that 1TB is “a lot” as a single customer- which is just not true. At the office, I could dump about 20-40TiB of data to offsite storage and just write that as DR cost, but if I’m kneecap’d at ~1TB, and a bandwidth limit to boot, then I wouldn’t even begin to trial it. For example, a single endpoint’s 3mo of file backups, with dedup&compression, is still ~1.2TiB- before I even start getting to the 1.7TiB of image backups (developement/Graphics design endpoint).
You can request a limit increase. It’s only there so Storj Labs gets a heads up if a customer suddenly wants to upload PB’s of data.
I have no idea what makes you say that. This is a quote from April.
Storj is several times larger than Sia already. Probably an order of magnitude larger.
2 Likes
Node churn is actually a big problem, especially if disqualifications are frequent during GE.
Alexey did a breakdown of what it costs them to repair data when a node gets disqualified and it’s astronomical !!
I’m actually worried that it might cause BIG problems if many nodes exit the network (not gracefully) in a short period of time. I really hope that they are actively working on a solution (should be one of the top priorities) because at the moment the held amount only covers a fraction of the repair cost.
There is a mention about repair being distributed across the nodes in the whitepaper but I guess that they haven’t gotten around to it yet. That would greatly increase the decentralization of the network as well as reduce the cost of rented servers.
Anyway that’s just my 2 cents on what Storj should work on next. I think @SGC infected me with the “long post with lots of digressions” virus haha
2 Likes
They are. The biggest reasons for disqualification that can be prevented are:
- Nodes losing access to storage >> https://review.dev.storj.io/c/storj/storj/+/2272
- Nodes failing graceful exit >> Several were fixed already, others are being actively worked on. https://forum.storj.io/t/reproduction-and-solving-issue-with-ge/8377
Furthermore, the repair threshold is currently set very high, partially to test that repair works as intended. But this also leads to much more repair than needed.
2 Likes
i had this happen today… caught it because the color awk log script was running in a window… i really need to setup so alerts lol
1 Like
Well… that’s kind of idea behind “every Joe” having a node isn’t it?
The horrible idea comes from tardigrade right? with their quote “make a node for fun and profit?”
And as for me and my experience of my node fragility- I am not too keen to shell out spare hundreds of dollars just to make Tardigrade business happy… With ridiculously low egresses? (in my country hundreds of dollars don’t come too easy…)
Unless one is a philanthrope and wants to fund their business… or has own agenda for spare hardware if this project won’t work out.
So I would say - For one’s home-made business expect home-made effects 
and unfortunately home-made flakiness and through that ->fragility of the network. (one month they are all for it, next they are going on vacation and have to turn off the computer… or even better - they go on vacation and everything crashes (my life’s experience 
)
Or as my friend did in their new flat - she put a power tester in a wall socket - just to check if there is power - blew up whole vertical skeleton power line - they had to go to the building basement and reset the breakers 
that took a few hours… imagine you are living with such a “talented” neighbours…
It might have resulted in maybe malformed DBs if no UPS… 
I would shell out cash … better yet I would gladly do it through “self funding”, but will not risk own major cash… at least not at the beginning…
This is a conundrum … won’t make cash without more space-> won’t have space without cash 
2 Likes
i have made my fair share of bad choices for this… first off i bought used harddrives… because… well they seemed cheap… but long term is certainly not going to end up having been cheap… because the failure rate is … cough an interesting data point that makes me want to throw stuff…
the table i got my server on i bumped on day and basically killed a hdd right there… atleast it was a 3tb one… then ofc the server room isn’t heated yet… so last winter the disk endured -17c
aside from that i wasn’t aware about the whole dew point and temperature thing… or atleast i didn’t take it into consideration before running the server in an unheated room…
so yeah it corroded, so much so that the fan wires literally shock the wires loose and the fan’s stopped workings… then i bought a wrong raid controller and components for 350$ only to replace it with 100$ of better gear within 2 months…
my system is setup to deal well with power outages… because i worry about them even if it’s most likely not even a yearly thing…
so that’s not really my consideration… however i do have a leaky roof way to close to where the server is located…
there will always be problems… only question is if you are bothered by problems or enjoy the challenge of finding solutions.
2 Likes
Still seeing those super high repair egress, vs June/July, on the slightly older node, and a good bit fuller- but still odd that the month of August has been flowing a lot of repair egress.
Fiber node (4mo)
Coax node (3mo)
1 Like
Looks like egress is increasing significantly, still very low ingress, I’m wondering when they’ll resume testing.
2 Likes
seems like the thread is starting to die down.
We did prove that ingress is almost exactly the same for all nodes. As for egress that will highly depend on the age of the node.
2 Likes
Not sure if age is the only factor. I think that the more data your node has, the higher chance of getting egress.
You can have 6 month old 2TB of capacity node that is fully filled. But I doubt that you get more egress than 3 month 6TB node and I would hypothesise that sometimes it can have higher egress, but higher capacity will prevail? 
My data for this month, egress is creeping up. Now it is at levels that are quite okey-ish for “per TB” basis.
Although small amount of stored data is not helping…
And what is interesting - log monitoring shows that there is a handful of errors stating that some files are not found… (little unnerving I would say)
Also an observation - as my secondary node gets more and more vetted, the ingress gets lower for primary node (secondary node is evening the ingress more and more)
Summing up ingress and egress for both nodes gives similar results as you guys posted above.
I will try to nerf secondary node free space to see if ingress can be rebalanced.
(blue line is ingress (and should be horizontal/creeping up, not going down), red line egress)
| Date | IngressT | EgressT [GB] | StoredT [GB] | egress ‰ | egressT | EgressT [kb/s /TB] |
|---|---|---|---|---|---|---|
| 02.08.2020 | 13.3 | 4.32 | 1.97 | 2.19 | 50.03 | 25.39 |
| 03.08.2020 | 11.62 | 3.27 | 1.98 | 1.65 | 37.86 | 19.12 |
| 04.08.2020 | 19.41 | 5.96 | 1.99 | 2.99 | 68.98 | 34.66 |
| 05.08.2020 | 32.14 | 7.29 | 2.03 | 3.59 | 84.39 | 41.57 |
| 06.08.2020 | 43.68 | 6.71 | 2.06 | 3.26 | 77.66 | 37.7 |
| 07.08.2020 | 49.5 | 7.8 | 2.1 | 3.71 | 90.23 | 42.97 |
| 08.08.2020 | 47.37 | 4.52 | 2.15 | 2.1 | 52.29 | 24.32 |
| 09.08.2020 | 38.48 | 7.68 | 2.17 | 3.54 | 88.91 | 40.97 |
| 10.08.2020 | 39.98 | 8.38 | 2.21 | 3.79 | 96.98 | 43.88 |
| 11.08.2020 | 40.28 | 8.7 | 2.24 | 3.88 | 100.65 | 44.93 |
| 12.08.2020 | 36.84 | 9.64 | 2.27 | 4.25 | 111.58 | 49.15 |
| 13.08.2020 | 35.6 | 11.84 | 2.3 | 5.15 | 137.06 | 59.59 |
| 14.08.2020 | 20.09 | 6.77 | 2.32 | 2.92 | 78.39 | 33.79 |
| 15.08.2020 | 30.71 | 15.2 | 2.34 | 6.5 | 175.98 | 75.2 |
| 16.08.2020 | 27.06 | 16.17 | 2.37 | 6.82 | 187.18 | 78.98 |
| 17.08.2020 | 22.69 | 15.24 | 2.38 | 6.4 | 176.41 | 74.12 |
| 18.08.2020 | 20.82 | 16.39 | 2.4 | 6.83 | 189.71 | 79.04 |
1 Like
Yes of course, I should have precised that I was talking about nodes with the same amount of data stored.
I started a second node almost two months ago and now that it’s fully vetted (except for stefan-benten) the ingress is almost perfectly split between the two nodes (the most I saw was a 7% difference).
Also my egress permille is almost exactly the same as yours over this period of time.
2 Likes
i think the only path forward is to have some script extract and compile the data for like say 1month at a time… so its nice as each to post / share / compare
seeing some pretty good numbers these days tho
4 Likes
WOW !!! 155GB!!! egress for ONE day!!! OMG!!! 
1 Like