# Updates on Test Data **URL:** \ **Category:** Announcements\ **Created:** [April 30, 2024, 3:36pm UTC](https://forum.storj.io/t/updates-on-test-data/26034 "2024-04-30T15:36:45Z")\ **Posts on this page:** 20\ **Page:** 11
**Author:** ![nerdatwork](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/nerdatwork/32/1135_2.png) [@nerdatwork](https://forum.storj.io/u/nerdatwork)\ **Post date:** [May 21, 2024, 7:29am UTC](https://forum.storj.io/t/updates-on-test-data/26034/211 "2024-05-21T07:29:50Z")
Windows node crashed with a stack trace. ```auto 2024-05-21T01:18:10Z ERROR services unexpected shutdown of a runner {"Process": "storagenode", "name": "piecestore:monitor", "error": "piecestore monitor: timed out after 1m0s while verifying readability of storage directory", "errorVerbose": "piecestore monitor: timed out after 1m0s while verifying readability of storage directory\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func1.1:154\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func1:143\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"} 2024-05-21T01:18:11Z FATAL Unrecoverable error {"Process": "storagenode", "error": "manager closed: closed: read tcp 127.0.0.1:49212->127.0.0.1:7778: read: connection reset by peer", "errorVerbose": "manager closed: closed: read tcp 127.0.0.1:49212->127.0.0.1:7778: read: connection reset by peer\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:232"} ``` Part 1 of log ☝ Part 2 👇 > **[stacktrace.txt | Storj](https://link.qa.storjshare.io/s/jxxfwi445kmar3ggvmuxo72jwoaa/rom/stacktrace.txt)** > > Shared content - Storj Above file is just a single line of stack trace that has 1.45 million characters. Discourse has character limit per post of 32k characters. ---
**Author:** ![Julio](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/julio/32/1132_2.png) [@Julio](https://forum.storj.io/u/Julio)\ **Post date:** [May 21, 2024, 8:27am UTC](https://forum.storj.io/t/updates-on-test-data/26034/212 "2024-05-21T08:27:14Z")
Coolio… Getting near my own saturation point… WTF? ![wtf-storj](https://storj-s3.bcdn.literatehosting.net/original/3X/2/0/203e66a1285d9ff7be56ee6662b1642660089d31.jpeg) ---
**Author:** ![BrightSilence](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/brightsilence/32/6048_2.png) [@BrightSilence](https://forum.storj.io/u/BrightSilence)\ **Post date:** [May 21, 2024, 9:09am UTC](https://forum.storj.io/t/updates-on-test-data/26034/213 "2024-05-21T09:09:40Z")
A little note from my end. Some of my nodes are beginning to fill up, but I’m a little hesitant to upgrade space now, since I still have 27TB of trash, which would free up plenty of space for further testing. Are you expecting to fill more than what was removed? If so, I’ll go shopping now, but I’d need some assurances first. ---
**Author:** ![Mitsos](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/mitsos/32/1125_2.png) [@Mitsos](https://forum.storj.io/u/Mitsos)\ **Post date:** [May 21, 2024, 9:13am UTC](https://forum.storj.io/t/updates-on-test-data/26034/214 "2024-05-21T09:13:02Z")
> [@Huge amount of trash data?](https://forum.storj.io/t/huge-amount-of-trash-data/26168/193): > > Thanks to all the cleanup work you shouldn’t have to add any extra drives. For now I would say please don’t order new hard drives and only use what you already have connected and on standby. Next week all the data that was moved into the trash folder this week will be available for new uploads. I am trying to solve this question with math but there are to many variables. I guess we need to wait for the actual uploads to hit our nodes. At that point we should be able to calculate how big our nod… ---
**Author:** ![ACarneiro](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/acarneiro/32/1129_2.png) [@ACarneiro](https://forum.storj.io/u/ACarneiro)\ **Post date:** [May 21, 2024, 9:15am UTC](https://forum.storj.io/t/updates-on-test-data/26034/215 "2024-05-21T09:15:51Z")
Interesting, isn’t it? I suppose it boils down to your disposable income and how much of a drama it would be for you to have another spun-up node with little to no ingress for a while. Personally, I’m waiting for all the trash data to be finally purged from my disks to get a better sense of what’s going on before going shopping. ---
**Author:** ![Mitsos](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/mitsos/32/1125_2.png) [@Mitsos](https://forum.storj.io/u/Mitsos)\ **Post date:** [May 21, 2024, 9:17am UTC](https://forum.storj.io/t/updates-on-test-data/26034/216 "2024-05-21T09:17:36Z")
Personally I’ll still expand just not at the rate I was, until we figure out what the future usage is like. ---
**Author:** ![BrightSilence](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/brightsilence/32/6048_2.png) [@BrightSilence](https://forum.storj.io/u/BrightSilence)\ **Post date:** [May 21, 2024, 9:20am UTC](https://forum.storj.io/t/updates-on-test-data/26034/217 "2024-05-21T09:20:05Z")
Yeah, on the one hand I don’t want to miss out on the test data. On the other, I don’t want to waste money if I don’t have to. But to be fair, I’ve made more than enough since the last expansion to pay for another purchase. So it’ll depend on what I should expect. Also due to the short TTL I wouldn’t be missing on permanent storage anyway. So I’m just pondering my options. Thanks @Mitsos . I guess I did read that but forgot about it. That gives some guidance at least. ---
**Author:** ![ACarneiro](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/acarneiro/32/1129_2.png) [@ACarneiro](https://forum.storj.io/u/ACarneiro)\ **Post date:** [May 21, 2024, 9:20am UTC](https://forum.storj.io/t/updates-on-test-data/26034/218 "2024-05-21T09:20:10Z")
Well, I currently have 8.5TB in trash on one of my machines. That, along with the free space on those nodes will be MORE than enough to accommodate any spikes for the foreseeable future. ---
**Author:** ![ACarneiro](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/acarneiro/32/1129_2.png) [@ACarneiro](https://forum.storj.io/u/ACarneiro)\ **Post date:** [May 21, 2024, 9:25am UTC](https://forum.storj.io/t/updates-on-test-data/26034/219 "2024-05-21T09:25:34Z")
Well… perhaps I’m just a bit of a doubter but… well… growth historically hasn’t really been massive and this recent frenzy is all on the promise of a new client which may or may not come to pass (fingers crossed, by the way!). The history of the last couple of years suggests that reality will be less optimistic than the plans. So long as the network currently has the performance and capacity to onboard big customers I see no reason to expand. I remain somewhat sceptical but following developments with great interest. PS: I hope that didn’t sound too pessimistic. I do believe in the project and have nothing but respect for all the Storj team. ---
**Author:** ![agente](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/agente/32/1137_2.png) [@agente](https://forum.storj.io/u/agente)\ **Post date:** [May 21, 2024, 9:30am UTC](https://forum.storj.io/t/updates-on-test-data/26034/220 "2024-05-21T09:30:12Z")
> [@Julio](#): > > Coolio… Getting near my own saturation point… WTF? what kind of router do you have for manage 7.5gbit (ppoe?) ---
**Author:** ![BrightSilence](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/brightsilence/32/6048_2.png) [@BrightSilence](https://forum.storj.io/u/BrightSilence)\ **Post date:** [May 21, 2024, 9:42am UTC](https://forum.storj.io/t/updates-on-test-data/26034/222 "2024-05-21T09:42:52Z")
> [@IsThisOn](#): > > That is not pessimistic but realistic. I agree. You’ve just perfectly countered my FOMO @ACarneiro 😃 ---
**Author:** ![zip](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/zip/32/7813_2.png) [@zip](https://forum.storj.io/u/zip)\ **Post date:** [May 21, 2024, 9:49am UTC](https://forum.storj.io/t/updates-on-test-data/26034/224 "2024-05-21T09:49:01Z")
> [@Julio](#): > > Coolio… Getting near my own saturation point… WTF? You should probably check the interface bandwidth as these data are now very broken showing tenfold of what the actual interface bandwidth is. ---
**Author:** ![jammerdan](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/jammerdan/32/1121_2.png) [@jammerdan](https://forum.storj.io/u/jammerdan)\ **Post date:** [May 21, 2024, 9:50am UTC](https://forum.storj.io/t/updates-on-test-data/26034/225 "2024-05-21T09:50:28Z")
> [@ACarneiro](#): > > Personally, I’m waiting for all the trash data to be finally purged from my disks I have so much trash on my nodes for multiple reasons too and I would like to see that purged first. ---
**Author:** ![littleskunk](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/littleskunk/32/63_2.png) [@littleskunk](https://forum.storj.io/u/littleskunk)\ **Post date:** [May 21, 2024, 11:28am UTC](https://forum.storj.io/t/updates-on-test-data/26034/226 "2024-05-21T11:28:05Z")
Oh boy. I can’t keep up with all the messages in this thread. > [@BrightSilence](#): > > Are you expecting to fill more than what was removed? Non of these deals are signed yet. → Lets better wait for the customers before making any decisions. Also don’t forget the impact of the TTL. Lets take a TTL of 4 weeks or so. At first you will see an amazing grow rate and might order new hard drives just to find out that after 4 weeks there is almost no growth on your node. → We need to adopt our estimations and take the TTL into account. I don’t know how yet. It is an SQLite DB so a script could run the math for us. > [@snorkel](#): > > I believe their servers of Saltlake are the bottleneck. He said he must tune the db on the sat. I believe we ruled that out yesterday. If we put load on 2 satellites we still hit the same throughput so it can’t be the satellite. It looks more like a storage node limitation. We could try a different node selection. We already designed a new node selection based on upload success rate that could be usefull here. Parts of it are already implemented but it would take some more effort to finish and also test it. Sounds more like this would be further down on our priority list and we will try other methods first. One method that is easy to apply is a different RS setting. I guess that will be one of the tests for today. ---
**Author:** ![nerdatwork](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/nerdatwork/32/1135_2.png) [@nerdatwork](https://forum.storj.io/u/nerdatwork)\ **Post date:** [May 21, 2024, 11:41am UTC](https://forum.storj.io/t/updates-on-test-data/26034/227 "2024-05-21T11:41:36Z")
> [@littleskunk](#): > > I can’t keep up with all the messages Just wanted to bring this to your attention in case it helps. > [@Updates on Test Data](https://forum.storj.io/t/updates-on-test-data/26034/211): > > Windows node crashed with a stack trace. 2024-05-21T01:18:10Z ERROR services unexpected shutdown of a runner {"Process": "storagenode", "name": "piecestore:monitor", "error": "piecestore monitor: timed out after 1m0s while verifying readability of storage directory", "errorVerbose": "piecestore monitor: timed out after 1m0s while verifying readability of storage directory\n\tstorj.io/storj/storagenode/monitor.(\*Service).Run.func1.1:154\n\tstorj.io/common/sync2.(\*Cycle).Run:160\n\tstorj.io/storj… ---
**Author:** ![ACarneiro](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/acarneiro/32/1129_2.png) [@ACarneiro](https://forum.storj.io/u/ACarneiro)\ **Post date:** [May 21, 2024, 11:44am UTC](https://forum.storj.io/t/updates-on-test-data/26034/228 "2024-05-21T11:44:17Z")
> [@littleskunk](#): > > I believe we ruled that out yesterday. If we put load on 2 satellites we still hit the same throughput so it can’t be the satellite. It looks more like a storage node limitation. From my limited understanding, this is a bit of a worry, isn’t it? Any idea if nodes are being held back by IOPS or bandwidth or any other “non-tuneable” constraints? ---
**Author:** ![littleskunk](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/littleskunk/32/63_2.png) [@littleskunk](https://forum.storj.io/u/littleskunk)\ **Post date:** [May 21, 2024, 12:18pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/229 "2024-05-21T12:18:27Z")
> [@ACarneiro](#): > > From my limited understanding, this is a bit of a worry, isn’t it? > Any idea if nodes are being held back by IOPS or bandwidth or any other “non-tuneable” constraints? With the few tests we executed yesterday it looked more like a bandwidth and not IOPS limitation. But don’t worry. It is tuneable for sure. We still have plenty of ideas what to try out next. ---
**Author:** ![ACarneiro](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/acarneiro/32/1129_2.png) [@ACarneiro](https://forum.storj.io/u/ACarneiro)\ **Post date:** [May 21, 2024, 12:26pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/230 "2024-05-21T12:26:53Z")
That’s very interesting. I think many of us “pleb SNOs” still have loads of available domestic bandwidth that didn’t get used. In my particular case, my home nodes are behind 600Mbit upstream and I didn’t go higher than around 40 during yesterday’s test (despite the Success Rate script claiming over 98% successful transfers). My nodes in other locations and countries saw similarly low upstream usage. ---
**Author:** ![BrightSilence](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/brightsilence/32/6048_2.png) [@BrightSilence](https://forum.storj.io/u/BrightSilence)\ **Post date:** [May 21, 2024, 12:35pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/231 "2024-05-21T12:35:28Z")
> [@littleskunk](#): > > Non of these deals are signed yet. → Lets better wait for the customers before making any decisions. Fair enough. I’ll pull my finger off the buy trigger. 😉 > [@littleskunk](#): > > We need to adopt our estimations and take the TTL into account. I don’t know how yet. It is an SQLite DB so a script could run the math for us. Yeah, but the piece size isn’t in the DB, so you don’t get much further than counting the number of pieces about to expire per day. ![image](https://storj-s3.bcdn.literatehosting.net/original/3X/c/4/c49af9dd4150a42531a7ed078f8c8e45ef63b905.png) > [@littleskunk](#): > > One method that is easy to apply is a different RS setting. Really curious about what this would yield. I’ve been looking forward to seeing some changes in that. Segment size might also be a thing to look at, though that would only help if you expect files to be big. > [@ACarneiro](#): > > That’s very interesting. I think many of us “pleb SNOs” still have loads of available domestic bandwidth that didn’t get used. Yeah, it’s peaking at about 9% of what I have available, plenty of room for more. ---
**Author:** ![Roxor](https://storj.bcdn.literatehosting.com/letter_avatar/roxor/32/5_5575768a8748004e209b776fc1b2916d.png) [@Roxor](https://forum.storj.io/u/Roxor)\ **Post date:** [May 21, 2024, 12:51pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/232 "2024-05-21T12:51:24Z")
> [@littleskunk](#): > > It looks more like a storage node limitation. We could try a different node selection. We already designed a new node selection based on upload success rate that could be usefull here. Your team has a lot of interesting challenges: a chance to try new things! I get that the ‘race’ system already biases data towards faster nodes… but I understand you may want to lean harder towards those fast nodes so clients have better interactive performance. I guess that could also be offset the other way to not penalize slower nodes too much: like perhaps the repair system could be biased _towards_ slower nodes… because there’s no customer-visible time pressure on repair traffic. If it takes twice as long to move repair data around: nobody will care. I have no suggestions or comments or criticisms. It can just be exciting to think about creative answers to technical questions. Have fun Storj devs! [Previous page](https://forum.storj.io/t/updates-on-test-data/26034.md?page=10) [Next page](https://forum.storj.io/t/updates-on-test-data/26034.md?page=12)