# Updates on Test Data **URL:** \ **Category:** Announcements\ **Created:** [April 30, 2024, 3:36pm UTC](https://forum.storj.io/t/updates-on-test-data/26034 "2024-04-30T15:36:45Z")\ **Posts on this page:** 20\ **Page:** 66
**Author:** ![BrightSilence](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/brightsilence/32/6048_2.png) [@BrightSilence](https://forum.storj.io/u/BrightSilence)\ **Post date:** [June 20, 2024, 10:50pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/1435 "2024-06-20T22:50:38Z")
> [@littleskunk](#): > > And to make your estimation as acurate as possible I can tell you that we have uploaded maybe 75% of our target. That certainly helps. But only for SNOs who had free space during the full testing period. Which happen to be the nodes who likely don’t need to expand right away. Although the overwriting thing kind of throws a wrench into the numbers as well. There’s nothing wrong with mentioning that that is the best way to gauge it, but I still suggest to add estimates. I forgot to mention that what would also help a lot is if GC is more reliably cleaning stuff up. I still have nodes with 25-30% of data being uncollected garbage. I have much more capacity than is being usefully used right now. Should I count that towards the test data as it’s caused by overwritten segments or not? ---
**Author:** ![littleskunk](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/littleskunk/32/63_2.png) [@littleskunk](https://forum.storj.io/u/littleskunk)\ **Post date:** [June 20, 2024, 10:54pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/1436 "2024-06-20T22:54:29Z")
> [@BrightSilence](#): > > I still suggest to add estimates We can’t. I can give you some total numbers. Lets say our target would be to upload 10 PB. Does that mean 10PB\*1.875/20K nodes? That is a TB per node so why has my node grown way more than 1 TB? You see this math doesn’t work. I can’t estimate how big your node will grow. ---
**Author:** ![Roxor](https://storj.bcdn.literatehosting.com/letter_avatar/roxor/32/5_5575768a8748004e209b776fc1b2916d.png) [@Roxor](https://forum.storj.io/u/Roxor)\ **Post date:** [June 20, 2024, 11:10pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/1437 "2024-06-20T23:10:15Z")
The guess I’m going with is I have about 2 more weeks of growth like the last 2 weeks… then all the TTL data will start to wrap… and I’ll level-out. After that I’m back to natural slow growth. Maybe that will fill a drive: we’ll see. 🤞 ---
**Author:** ![Roxor](https://storj.bcdn.literatehosting.com/letter_avatar/roxor/32/5_5575768a8748004e209b776fc1b2916d.png) [@Roxor](https://forum.storj.io/u/Roxor)\ **Post date:** [June 20, 2024, 11:50pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/1438 "2024-06-20T23:50:02Z")
> [@Aitor](#): > > If the network is currently at 50% usage, I don’t quite understand the problem. Is that mystery customer going to use it all? really? If you’re looking at [the graph most of us check](https://storjstats.info/d/storj/storj-network-statistics?orgId=1&from=now-2d&to=now&viewPanel=405): it’s [counting slightly different things](https://forum.storj.io/t/updates-on-test-data/26034/1346) so we’re closer to 2/3rds full (assuming real customer data uses about 2.2x the free space when uploaded). Perhaps when capacity-reservation is complete we’ll be near 3/4? ---
**Author:** ![EricTheRed123](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/ericthered123/32/1124_2.png) [@EricTheRed123](https://forum.storj.io/u/EricTheRed123)\ **Post date:** [June 21, 2024, 12:25am UTC](https://forum.storj.io/t/updates-on-test-data/26034/1439 "2024-06-21T00:25:54Z")
> [@littleskunk](#): > > Lets say our target would be to upload 10 PB Please tell me it’s more than 10 petabytes. ---
**Author:** ![littleskunk](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/littleskunk/32/63_2.png) [@littleskunk](https://forum.storj.io/u/littleskunk)\ **Post date:** [June 21, 2024, 12:54am UTC](https://forum.storj.io/t/updates-on-test-data/26034/1440 "2024-06-21T00:54:28Z")
Sure no problem. It is more than 10 petabytes. ---
**Author:** ![jammerdan](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/jammerdan/32/1121_2.png) [@jammerdan](https://forum.storj.io/u/jammerdan)\ **Post date:** [June 21, 2024, 4:54am UTC](https://forum.storj.io/t/updates-on-test-data/26034/1441 "2024-06-21T04:54:50Z")
> [@Bryanm](#): > > Is a forum post detailing a signed contract even enough, or will SNOs wait until their nodes are actually full before they add more capacity? Obviously these are important questions for us on the capacity planning side. I am wondering if you are aware that there might be other reasons for SNOs not adding capacity? I tried to make that clear in my post: > [@SNO Capacity Planning](https://forum.storj.io/t/sno-capacity-planning/26722/61): > > Currently no because there is no basis for such a planning as almost every number that the nodes provide is not reliable or just wrong. And unfortunately there is only slow progress in fixing this as whenever this is getting mentioned we hear it is “low priority”. Due to various bugs several of my nodes show wrong bandwidth, wrong used space, wrong trash, wrong average used space, wrong estimates, wrong everything… On what basis do you want me to answer your question when I even don’t know ho… From what I see on Github I would have to wait at least until my nodes are on version 1.106 and then maybe get a picture what is currently really used and what’s not. And all the old deleted stuff finally cleaned out. The unreliability of the numbers everywhere but also of the storagenode software and the satellites together with slow rollout of fixes because it is “low priority” is currently my main blocker to add anything. It is also what @BrightSilence has said: > [@BrightSilence](#): > > I forgot to mention that what would also help a lot is if GC is more reliably cleaning stuff up. I still have nodes with 25-30% of data being uncollected garbage. There is a lot on the forum about issues with the databases, space discrepancies, problems with filewalkers, garbage not deleting, nodes restarting and I see all of that on my nodes too. The way this all works with so many issues at the moment is beyond my comprehension. So even if you would make such a forum post today detailing a signed deal I would not add and go for additional capacity or nodes. My suggestion on adding more customers with large space requirements I made here: > [@SNO Capacity Planning](https://forum.storj.io/t/sno-capacity-planning/26722/63): > > Maybe to solve the issues on the node side first before asking customers to do that? I mean the ongoing problems can easily backfire when the nodes can not or will not keep up with such a data flow… Edit: Just to give you an idea what I am talking about: If I look at the average used space over all nodes this gives me only 40% of the space that all my nodes claim they are using. And the trash over all nodes is telling me its size is 1/3 of the reported average used. And this does not add up. Running `du` give me complete different space than the nodes do. Some must have tons of still uncollected garbage on it while some have not correct updated used space databases as filewalkers do not finish. So basically even if I wanted to I could not tell how much space is used altogether, how much trash is there, how much garbage has notbeen collected and on. So it is really impossible to make a decision to add even more terabytes to this obscure situation. Edit2: The resume feature for used space filewalker is also one of the features I need to get the numbers correct. ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [June 21, 2024, 5:00am UTC](https://forum.storj.io/t/updates-on-test-data/26034/1442 "2024-06-21T05:00:31Z")
> [@Solu](#): > > Will the traffic be divided by amount of online nodes in subnet or will the traffic be divided by online nodes in subnet with space available? Only online not full nodes are selected also using the choice of 2 > [@Updates on Test Data](https://forum.storj.io/t/updates-on-test-data/26034/313): > > Ok while you are removing any concurrency limit you might have set I will explain how the node selection actually works. When a segment gets commited to the database it will contain the nodes that have been fast enough and it will be missing the nodes that got long tail canceld. The satellite calculates a success rate for each node with that. The node selection takes that success rate. Instead of 110 total nodes it selectes 220 nodes at first and compares them in pairs and pick the one with th… ---
**Author:** ![gingerbread233](https://storj.bcdn.literatehosting.com/letter_avatar/gingerbread233/32/5_5575768a8748004e209b776fc1b2916d.png) [@gingerbread233](https://forum.storj.io/u/gingerbread233)\ **Post date:** [June 21, 2024, 5:47am UTC](https://forum.storj.io/t/updates-on-test-data/26034/1443 "2024-06-21T05:47:58Z")
They are basically crappy USB-Sticks in a “SSD” Case 😀 At least his USB-Stick is from Samsung… ---
**Author:** ![Solu](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/solu/32/62_2.png) [@Solu](https://forum.storj.io/u/Solu)\ **Post date:** [June 21, 2024, 6:19am UTC](https://forum.storj.io/t/updates-on-test-data/26034/1444 "2024-06-21T06:19:56Z")
I think there is a significant amount of SNO who are willing to upgrade as soon as their disk are filling. But the speed of the expanding depends… mostly the subnet limitation will lead to the “one by one” approach which i already mentioned [here.](https://forum.storj.io/t/sno-capacity-planning/26722/83) So the first question from SNO side is: Do you need just contunously one node with space available at one ip or do you need multiple nodes behind the same ip for distribution? If second, how much nodes makes sense for you? Because from the SNO perspective there is no difference in incoming traffic but only in cost of disks. Currently i see about 1.5 to 2.5TB/day ingress per ip on my nodes, that will fill 18TB disks (16TB usable) at a rate on 8 days, maybe more if some data will be deleted… For me i can keep up the pace and add one more node per week (currently up to 104). The second question is: Do you provide information about the duration of the signed contract? If the contract term was signed for lets say 12 months, the SNO run their math and could decide if it is worth it to spin up even more nodes. Third and final question: How long in advance are we informed before the customer data arrives? ---
**Author:** ![snorkel](https://storj.bcdn.literatehosting.com/letter_avatar/snorkel/32/5_5575768a8748004e209b776fc1b2916d.png) [@snorkel](https://forum.storj.io/u/snorkel)\ **Post date:** [June 21, 2024, 7:22am UTC](https://forum.storj.io/t/updates-on-test-data/26034/1445 "2024-06-21T07:22:17Z")
Samsung says on the cover, but my Syno identifies it as Silicon Motion Taiwan. 🕶 I don’t know who makes the chips and the controller if the digital ID says one thing and the cover says other, if it’s Samsung or just a rebranding. ---
**Author:** ![gingerbread233](https://storj.bcdn.literatehosting.com/letter_avatar/gingerbread233/32/5_5575768a8748004e209b776fc1b2916d.png) [@gingerbread233](https://forum.storj.io/u/gingerbread233)\ **Post date:** [June 21, 2024, 7:24am UTC](https://forum.storj.io/t/updates-on-test-data/26034/1446 "2024-06-21T07:24:39Z")
Maybe you misread Samsung for Smasnug, and it’s just a knock-off too 🙂 ---
**Author:** ![snorkel](https://storj.bcdn.literatehosting.com/letter_avatar/snorkel/32/5_5575768a8748004e209b776fc1b2916d.png) [@snorkel](https://forum.storj.io/u/snorkel)\ **Post date:** [June 21, 2024, 7:28am UTC](https://forum.storj.io/t/updates-on-test-data/26034/1447 "2024-06-21T07:28:30Z")
Ideea for trash optimisation: - the trash is just a safety backup, right? - why do we need to backup all the pieces? - we just need the smallest number of pieces to recreat a segment or a file or whatever. - so let’s store only the smallest number of pieces in trash, by cleaverly choosing the nodes that store those pieces evenly, not to overload some and free up too much on others. This should be part of the bloom filter generation process. ---
**Author:** ![snorkel](https://storj.bcdn.literatehosting.com/letter_avatar/snorkel/32/5_5575768a8748004e209b776fc1b2916d.png) [@snorkel](https://forum.storj.io/u/snorkel)\ **Post date:** [June 21, 2024, 7:30am UTC](https://forum.storj.io/t/updates-on-test-data/26034/1448 "2024-06-21T07:30:32Z")
On 8 sticks bought in sealed retail packages from reputable store, with all the serial numbers, logos, materials etc.? Just buy one and test it. 😉 ---
**Author:** ![arrogantrabbit](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/arrogantrabbit/32/11200_2.png) [@arrogantrabbit](https://forum.storj.io/u/arrogantrabbit)\ **Post date:** [June 21, 2024, 7:39am UTC](https://forum.storj.io/t/updates-on-test-data/26034/1449 "2024-06-21T07:39:44Z")
> [@snorkel](#): > > Samsung says on the cover, but my Syno identifies it as Silicon Motion Taiwan It’s likely has Samsung nand chips (barely passing the QA, bottom of the barrel yield, instead of sending them to landfill they make cheap consumer thumb drives because nobody expects any reliability from them) and silicon motion controllers – because they are cheap. Again, cost drives lack of bothering with custom firmware to report “Samsung” for the final product… it’ garbage all around. Generally, Samsung is great at manufacturing parts – nand and DDR chips mostly, but that’s it. Every single end user product made by Samsung is incoherent trash. These are no exceptions. ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [June 21, 2024, 7:44am UTC](https://forum.storj.io/t/updates-on-test-data/26034/1450 "2024-06-21T07:44:24Z")
> [@snorkel](#): > > we just need the smallest number of pieces to recreat a segment or a file or whatever. Yes… but some operators are deleting the trash manually or disabled it in their config, so the remaining pieces could be not enough to re-create missing pieces. > [@snorkel](#): > > so let’s store only the smallest number of pieces in trash Your node should store only one piece of the segment. So 1 of the current required amount. You cannot store “some” pieces in the trash, protecting only “some” segments. It doesn’t make sense. ---
**Author:** ![BrightSilence](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/brightsilence/32/6048_2.png) [@BrightSilence](https://forum.storj.io/u/BrightSilence)\ **Post date:** [June 21, 2024, 8:10am UTC](https://forum.storj.io/t/updates-on-test-data/26034/1451 "2024-06-21T08:10:27Z")
> [@littleskunk](#): > > We can’t. I can give you some total numbers. Lets say our target would be to upload 10 PB. Does that mean 10PB\*1.875/20K nodes? That is a TB per node so why has my node grown way more than 1 TB? You see this math doesn’t work. I can’t estimate how big your node will grow. You can certainly do a whole lot better than that. I was careful in my message to say subnets with free space. That wouldn’t be close to 20k. And yes, I realize that the choiceofn node selection makes it more difficult, but you could calculate and average and mention that the fastest nodes would get n times more than the average. I’m betting if you do those things, your node will suddenly fall neatly within that range. I’m happy to do my own calculation if at the time you can share the number of subnets with free space, the total amount of data expected to be uploaded and the n you settled on. But it would be better to just provide the results as some guidance. Because many will likely not know how to do that calculation or will simply not bother. > [@jammerdan](#): > > There is a lot on the forum about issues with the databases, space discrepancies, problems with filewalkers, garbage not deleting, nodes restarting and I see all of that on my nodes too. The way this all works with so many issues at the moment is beyond my comprehension. I’m not surprised. This testing patterns is significantly different from how the network used to behave and this is what testing is for, right? So rather than airing my frustration and bashing the current situation, I just stick to calmly informing Storj that this will and does impact my behavior in regards to upgrading. That’s all they really need to know to know they should fix these things. I’m sure they will, but I’m also not surprised that issues popped up under such conditions. > [@snorkel](#): > > This should be part of the bloom filter generation process. This is not possible with how bloom filters work. They just know your node shouldn’t have this piece but aren’t aware of which other nodes have pieces for the same segment. You can put a random file that was never even uploaded through store in the folder and the bloom filter will clean it up too. It’s not a list of files to be deleted, but it’s a cleaver filter that would match all files to keep. ---
**Author:** ![littleskunk](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/littleskunk/32/63_2.png) [@littleskunk](https://forum.storj.io/u/littleskunk)\ **Post date:** [June 21, 2024, 9:03am UTC](https://forum.storj.io/t/updates-on-test-data/26034/1452 "2024-06-21T09:03:43Z")
> [@BrightSilence](#): > > You can certainly do a whole lot better than that. I thought so at the beginning but it turns out any math I can come up with just gets slapped by reality. My grow rate is simply off the scale and I haven’t found a math that would explain that. Instead measuring my grow rate for a day and estimate my size after 30 days seems to be very accurate. > [@BrightSilence](#): > > I’m happy to do my own calculation if at the time you can share the number of subnets with free space, the total amount of data expected to be uploaded and the n you settled on. Sure try your luck. 6K at the moment, 20, 6-8. So just by math this gives us a range of 0.8TB for slow nodes and up to 50TB for fast nodes. And now? If you would be in my situation what kind of numbers (=promises) will you put out there? 0.8TB? That is a bit low isn’t it? But 50TB would also be unfair. But please let me know if you can come up with some numbers that will work better than mine. ---
**Author:** ![Vadim](https://storj.bcdn.literatehosting.com/letter_avatar/vadim/32/5_5575768a8748004e209b776fc1b2916d.png) [@Vadim](https://forum.storj.io/u/Vadim)\ **Post date:** [June 21, 2024, 9:22am UTC](https://forum.storj.io/t/updates-on-test-data/26034/1453 "2024-06-21T09:22:07Z")
So best correct answer is Everyone should calculate it by themselves. take 5or 10 day average added space(not ingress) then you can estimate how much you will get a day and then x30 thats all. ---
**Author:** ![BrightSilence](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/brightsilence/32/6048_2.png) [@BrightSilence](https://forum.storj.io/u/BrightSilence)\ **Post date:** [June 21, 2024, 9:53am UTC](https://forum.storj.io/t/updates-on-test-data/26034/1454 "2024-06-21T09:53:43Z")
> [@littleskunk](#): > > Sure try your luck. 6K at the moment, 20, 6-8. So just by math this gives us a range of 0.8TB for slow nodes and up to 50TB for fast nodes. And now? If you would be in my situation what kind of numbers (=promises) will you put out there? 0.8TB? That is a bit low isn’t it? But 50TB would also be unfair. But please let me know if you can come up with some numbers that will work better than mine. You sound skeptical, but this is actually useful information for me. I think my nodes are tending towards the upper range of performance and now I can decide a little bit what I should do. Probably wouldn’t be worth it to buy more than 2x20TB per IP and I might get rid of half of my IP’s as the test has shown there are diminishing returns with the new node selection. So maybe I’ll hold on to 3 of them and buy 6 20TB HDD’s. Or start with 3 and see how fast they fill. I know buying more is not going to be worth it now. Also, I think the lower range is technically 0TB as it is theoretically possible that you are almost never selected. That said, I wasn’t aware the n number would be this high, so I was indeed expecting smaller ranges. Still, this information is still more valuable than “big deal signed”. [Previous page](https://forum.storj.io/t/updates-on-test-data/26034.md?page=65) [Next page](https://forum.storj.io/t/updates-on-test-data/26034.md?page=67)