# Updates on Test Data **URL:** \ **Category:** Announcements\ **Created:** [April 30, 2024, 3:36pm UTC](https://forum.storj.io/t/updates-on-test-data/26034 "2024-04-30T15:36:45Z")\ **Posts on this page:** 20\ **Page:** 77
**Author:** ![nerdatwork](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/nerdatwork/32/1135_2.png) [@nerdatwork](https://forum.storj.io/u/nerdatwork)\ **Post date:** [July 2, 2024, 6:02am UTC](https://forum.storj.io/t/updates-on-test-data/26034/1710 "2024-07-02T06:02:56Z")
I saw multiple lines of `count: 600k` which should update at least 10s of gbs. I had no ingress so the number didnt update at all. I need to further investigate. ```auto 2024-06-29T17:04:56Z INFO collector collect {"Process": "storagenode", "count": 664118} 2024-06-29T22:52:07Z INFO collector collect {"Process": "storagenode", "count": 666053} 2024-06-30T05:29:22Z INFO collector collect {"Process": "storagenode", "count": 667016} 2024-06-30T09:20:18Z INFO collector collect {"Process": "storagenode", "count": 667057} 2024-06-30T11:11:36Z INFO collector collect {"Process": "storagenode", "count": 638589} ``` ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [July 2, 2024, 6:54am UTC](https://forum.storj.io/t/updates-on-test-data/26034/1711 "2024-07-02T06:54:09Z")
Accordingly this issue: > [@Updates on Test Data](https://forum.storj.io/t/updates-on-test-data/26034/1679): > > Not on the nodes… It seems deleting expired TTL pieces does not update the used space stats. I think it’s not updating databases with a new free space. So the workaround would be the same - restart the node with a used-space-filewalker enabled, until the new version would be rolled out. ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [July 2, 2024, 6:55am UTC](https://forum.storj.io/t/updates-on-test-data/26034/1712 "2024-07-02T06:55:48Z")
It’s interesting. It should be deleted. Just like @Vadim said - with the same time intervals (if the TTL was the same though). ---
**Author:** ![snorkel](https://storj.bcdn.literatehosting.com/letter_avatar/snorkel/32/5_5575768a8748004e209b776fc1b2916d.png) [@snorkel](https://forum.storj.io/u/snorkel)\ **Post date:** [July 2, 2024, 7:05am UTC](https://forum.storj.io/t/updates-on-test-data/26034/1713 "2024-07-02T07:05:51Z")
What’s the point? The TTL data is removed daily. The startup piecescan is run only at restart and it takes days. Should we keep restarting the nodes when the FW finished? It’s pointless. Just fix the bug and release those updates quicker. THIS SHOULD BE THE PRIORITY NUMBER ONE! ---
**Author:** ![JoshieGarza](https://storj.bcdn.literatehosting.com/letter_avatar/joshiegarza/32/5_5575768a8748004e209b776fc1b2916d.png) [@JoshieGarza](https://forum.storj.io/u/JoshieGarza)\ **Post date:** [July 2, 2024, 7:19am UTC](https://forum.storj.io/t/updates-on-test-data/26034/1714 "2024-07-02T07:19:06Z")
this could explain why 12 days after reducing the allocated space (from several TBs to 1TB because i want to migrate the SN to another disk faster), the used space is still the same. ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [July 2, 2024, 7:22am UTC](https://forum.storj.io/t/updates-on-test-data/26034/1715 "2024-07-02T07:22:04Z")
I do not think that you need to restart it daily. However, If your node is rendered as full, but you have a free space, you may restart it to make updates. And I do not suggest it as a fix, it’s a workaround until the fix would be rolled out. ---
**Author:** ![flwstern](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/flwstern/32/7821_2.png) [@flwstern](https://forum.storj.io/u/flwstern)\ **Post date:** [July 2, 2024, 7:55am UTC](https://forum.storj.io/t/updates-on-test-data/26034/1716 "2024-07-02T07:55:26Z")
I think you have to do some redesigns alexey. To much issues for everyone. ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [July 2, 2024, 8:16am UTC](https://forum.storj.io/t/updates-on-test-data/26034/1717 "2024-07-02T08:16:24Z")
I know, and there is a work in the team, but lightly exposed right now like > [@Disk usage discrepancy?](https://forum.storj.io/t/disk-usage-discrepancy/24715/84): > > Ah, if we’re talking about the manipulation of the files, then yes, as you rightly point out, we have lots of room for improvement. I was only talking above about the directory traversal piece by itself. Your suggestions are good ones and some we’ve talked about doing before. Some are harder than others. For example, giving up on the temp dir would mean we need some other way to differentiate partial uploads from full, intact blobs. We could try to use a database to keep track of what files a… > [@Has it ever been considered to use in-memory databases?](https://forum.storj.io/t/has-it-ever-been-considered-to-use-in-memory-databases/25353/42): > > It would require a separate chore to move data from the operational database to the historical database, it also will require to redo how this information is handled in the storagenode API (it must combine data from two sources (three if we account both operational databases - in-memory and on the disk, also historic DB) for each metric). it’s as complicated as it sounds. As result - it likely slow down the node and anything what would request these metrics. I do have more hope into this resea… > [@Design draft: a low I/O piece storage](https://forum.storj.io/t/design-draft-a-low-i-o-piece-storage/24864/61): > > Thanks for this design doc @Toyoo, It’s a well written design, I like it. And finally I had some time to collect my thoughts. TLDR; I like the approach, but I am not sure about the implementation. So far I have heard about multiple attempt to improve the piece store. They are usually save the metadata of pieces in a database like system and save the raw data in custom files. (technically it would be possible to store both metadata and data in database, but it’s quite slow. I tried it, and mea… ---
**Author:** ![Roxor](https://storj.bcdn.literatehosting.com/letter_avatar/roxor/32/5_5575768a8748004e209b776fc1b2916d.png) [@Roxor](https://forum.storj.io/u/Roxor)\ **Post date:** [July 2, 2024, 10:34am UTC](https://forum.storj.io/t/updates-on-test-data/26034/1718 "2024-07-02T10:34:29Z")
Storj itself is relatively lightweight: even with current ingress. But seemingly running used-space-filewalker 24x7… 100% iops all-the-time… does get tiring. Good thing I’m a patient man 😊 ---
**Author:** ![Andrew](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/andrew/32/1139_2.png) [@Andrew](https://forum.storj.io/u/Andrew)\ **Post date:** [July 2, 2024, 6:32pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/1719 "2024-07-02T18:32:14Z")
Devs, do you recommend to update to 1.107.3? Or should we wait something? ---
**Author:** ![snorkel](https://storj.bcdn.literatehosting.com/letter_avatar/snorkel/32/5_5575768a8748004e209b776fc1b2916d.png) [@snorkel](https://forum.storj.io/u/snorkel)\ **Post date:** [July 2, 2024, 7:47pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/1720 "2024-07-02T19:47:05Z")
Wait! Because if you recreate the container, it could go down a version and the databases are incompatible. You will end up updating it again or deleting the databases. ---
**Author:** ![pangolin](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/pangolin/32/15685_2.png) [@pangolin](https://forum.storj.io/u/pangolin)\ **Post date:** [July 2, 2024, 7:57pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/1721 "2024-07-02T19:57:46Z")
> [@snorkel](#): > > it could go down a version The version server url is not hard coded. This can be used to avoid such downgrades or to let it install any version you want. ---
**Author:** ![littleskunk](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/littleskunk/32/63_2.png) [@littleskunk](https://forum.storj.io/u/littleskunk)\ **Post date:** [July 2, 2024, 8:13pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/1722 "2024-07-02T20:13:28Z")
> [@Andrew](#): > > Devs, do you recommend to update to 1.107.3? Or should we wait something? We have started the normal rollout procedure. Please be patient and follow the rollout process. If we would all upgrade our nodes by hand we are risking to lose customer data. The rollout procedure is designed to mitigate that risk. If you feel the rush to install it faster please don’t do it on all of your nodes all at once. Better pick a single node first, let it run for a few hours - a day, upgrade like 50% of your nodes , wait again and last but not least the remaining nodes. ---
**Author:** ![PePeR](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/peper/32/1127_2.png) [@PePeR](https://forum.storj.io/u/PePeR)\ **Post date:** [July 2, 2024, 8:34pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/1723 "2024-07-02T20:34:55Z")
Are you running some crazy benchmarks right now? Last 2 days all my nodes restart every few hours ---
**Author:** ![arrogantrabbit](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/arrogantrabbit/32/11200_2.png) [@arrogantrabbit](https://forum.storj.io/u/arrogantrabbit)\ **Post date:** [July 2, 2024, 9:24pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/1724 "2024-07-02T21:24:51Z")
Find out why do they restart and fix it. It’s in no way normal. ---
**Author:** ![PePeR](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/peper/32/1127_2.png) [@PePeR](https://forum.storj.io/u/PePeR)\ **Post date:** [July 2, 2024, 9:30pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/1725 "2024-07-02T21:30:25Z")
I think having 300gb egress of daily repair isn’t normal… ---
**Author:** ![littleskunk](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/littleskunk/32/63_2.png) [@littleskunk](https://forum.storj.io/u/littleskunk)\ **Post date:** [July 2, 2024, 9:30pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/1726 "2024-07-02T21:30:50Z")
> [@PePeR](#): > > Are you running some crazy benchmarks right now? I would say not more crazy than on other days. It should still be the same load. What has changed is the TTL cleanup that now comes on top. Maybe that is too much for your storage node? ---
**Author:** ![PePeR](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/peper/32/1127_2.png) [@PePeR](https://forum.storj.io/u/PePeR)\ **Post date:** [July 2, 2024, 9:33pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/1727 "2024-07-02T21:33:46Z")
probably… I will monitor it for few days and if its gonna keep going Im gonna just change config to full drives to not get more ingress. This should fix it temporarly ---
**Author:** ![Roxor](https://storj.bcdn.literatehosting.com/letter_avatar/roxor/32/5_5575768a8748004e209b776fc1b2916d.png) [@Roxor](https://forum.storj.io/u/Roxor)\ **Post date:** [July 2, 2024, 9:42pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/1728 "2024-07-02T21:42:39Z")
Last few days look like normal 75%-crazy: nothing special. ---
**Author:** ![Andrew](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/andrew/32/1139_2.png) [@Andrew](https://forum.storj.io/u/Andrew)\ **Post date:** [July 2, 2024, 11:29pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/1729 "2024-07-02T23:29:11Z")
Two nodes were updated to version 1.107.3. After the update, **many** records appeared in the logs like this: > 2024-07-02T23:11:27Z WARN collector unable to delete piece {“Process”: “storagenode”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Piece ID”: “DA3NATNM5X6BWSRR6XUSOOB4AIP745ERCBSPV5VDVZXXF5G2TFRA”, “error”: “pieces error: filestore error: file does not exist”, “errorVerbose”: “pieces error: filestore error: file does not exist\n > [tstorj.io/storj/storagenode/blobstore/filestore.(\*blobStore).Stat:122](http://tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).Stat:122)\n\tstorj.io/storj/storagenode/pieces.(\*BlobsUsageCache).pieceSizes:270\n\tstorj.io/storj/storagenode/pieces.(\*Blobs > UsageCache).DeleteWithStorageFormat:250\n\tstorj.io/storj/storagenode/pieces.(\*Store).DeleteSkipV0:355\n\tstorj.io/storj/storagenode/collector.(\*Service).Collect.func1:88\n\tstorj.io/storj/storagenode/storagenodedb.(\*pieceExpirationDB).GetExpired:71\n\tstorj.io/storj/storagenode/pieces.(\*Store).GetExpired:575\n\tstorj.io/storj/storagenode/collector.(\*Service).Collect: > 87\n\tstorj.io/storj/storagenode/collector.(\*Service).Run.func1:65\n\tstorj.io/common/sync2.(\*Cycle).Run:99\n\tstorj.io/storj/storagenode/collector.(\*Service).Run:61\n\tstorj.io/storj/p > rivate/lifecycle.(\*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(\*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(\*Group).Go.func1:78”} Why did they appear? Is this expected? Before the update there was no errors (log grepped by “collect”): > 2024-07-02T09:43:15Z INFO collector collect {“Process”: “storagenode”, “count”: 28} > 2024-07-02T10:43:14Z INFO collector collect {“Process”: “storagenode”, “count”: 18} > 2024-07-02T11:43:44Z INFO collector collect {“Process”: “storagenode”, “count”: 727} > 2024-07-02T12:43:43Z INFO collector collect {“Process”: “storagenode”, “count”: 1027} > 2024-07-02T13:43:22Z INFO collector collect {“Process”: “storagenode”, “count”: 304} > 2024-07-02T14:43:55Z INFO collector collect {“Process”: “storagenode”, “count”: 1356} > 2024-07-02T15:43:42Z INFO collector collect {“Process”: “storagenode”, “count”: 696} > 2024-07-02T16:43:47Z INFO collector collect {“Process”: “storagenode”, “count”: 1075} > 2024-07-02T17:43:38Z INFO collector collect {“Process”: “storagenode”, “count”: 970} > 2024-07-02T18:44:12Z INFO collector collect {“Process”: “storagenode”, “count”: 2015} > 2024-07-02T19:43:43Z INFO collector collect {“Process”: “storagenode”, “count”: 1016} > 2024-07-02T20:43:14Z INFO collector collect {“Process”: “storagenode”, “count”: 13} > 2024-07-02T21:43:14Z INFO collector collect {“Process”: “storagenode”, “count”: 23} > 2024-07-02T22:43:15Z INFO collector collect {“Process”: “storagenode”, “count”: 27} [Previous page](https://forum.storj.io/t/updates-on-test-data/26034.md?page=76) [Next page](https://forum.storj.io/t/updates-on-test-data/26034.md?page=78)