# High Trash usage - 27 TB - no more uploads **URL:** \ **Category:** troubleshooting\ **Tags:** trash\ **Created:** [November 8, 2024, 11:31am UTC](https://forum.storj.io/t/high-trash-usage-27-tb-no-more-uploads/28471 "2024-11-08T11:31:37Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![mcanto73](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/mcanto73/32/1133_2.png) [@mcanto73](https://forum.storj.io/u/mcanto73)\ **Post date:** [November 8, 2024, 11:31am UTC](https://forum.storj.io/t/high-trash-usage-27-tb-no-more-uploads/28471/1 "2024-11-08T11:31:37Z")
Hello, I’m running 5 nodes (with different IP) and I noticed that one has stopped to receive uploads even if with space availanle and I see a lot of Trash space in all my nodes ![node1](https://storj-s3.bcdn.literatehosting.net/original/3X/0/e/0ec87a0aca12531c75967656cdbc4b9841e7fc2c.png) ![node2](https://storj-s3.bcdn.literatehosting.net/original/3X/1/b/1b84ade35d6a7d926f5cb6e85bae83ea18a0db0c.png) ![node3](https://storj-s3.bcdn.literatehosting.net/original/3X/0/3/0395d8a66d8fbdba1539ac8b4d7019543fe3c9b6.png) ![node4](https://storj-s3.bcdn.literatehosting.net/original/3X/2/7/2753b0a747a4ae206c7ff4d85616affa71c06780.png) ![node5](https://storj-s3.bcdn.literatehosting.net/original/3X/1/7/17eaba34a18d1e22673a71681fdb07b97b625552.png) How is it possible that I have something like 27TB used as Trash ? Why I’m not getting uploads in this with 3 TB free : /STORJ5/STORJ\_NFS4 11T 7.4T 3.1T 71% /STORJ The screenshot related to it is node1 and says 0 free space with 7.3TB used, 5.91 TB Trash So, why this huge amount of space is in trash ? Thanks ---
**Author:** ![andrew2.hart](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/andrew2.hart/32/6415_2.png) [@andrew2.hart](https://forum.storj.io/u/andrew2.hart)\ **Post date:** [November 8, 2024, 11:54am UTC](https://forum.storj.io/t/high-trash-usage-27-tb-no-more-uploads/28471/2 "2024-11-08T11:54:04Z")
> [@mcanto73](#): > > STORJ5/STORJ\_NFS4 Is it really NFS? It might be worth checking if you really have that much trash on one of the nodes. Look to see if the folders in the four satellite folders in trash have old dates, older than 7 days ---
**Author:** ![flo82](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/flo82/32/10747_2.png) [@flo82](https://forum.storj.io/u/flo82)\ **Post date:** [November 8, 2024, 1:10pm UTC](https://forum.storj.io/t/high-trash-usage-27-tb-no-more-uploads/28471/3 "2024-11-08T13:10:04Z")
Have you disabled the filewalkers on startup? ---
**Author:** ![mcanto73](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/mcanto73/32/1133_2.png) [@mcanto73](https://forum.storj.io/u/mcanto73)\ **Post date:** [November 8, 2024, 1:32pm UTC](https://forum.storj.io/t/high-trash-usage-27-tb-no-more-uploads/28471/4 "2024-11-08T13:32:42Z")
Hello, why should I do that ? My storj instances are build via docker compose with these parameters: - STORJ\_PIECES\_ENABLE\_LAZY\_FILEWALKER=true - STORJ\_STORAGE2\_PIECE\_SCAN\_ON\_STARTUP=false Best regards ---
**Author:** ![PieceKeeper](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/piecekeeper/32/1125_2.png) [@PieceKeeper](https://forum.storj.io/u/PieceKeeper)\ **Post date:** [November 8, 2024, 4:32pm UTC](https://forum.storj.io/t/high-trash-usage-27-tb-no-more-uploads/28471/5 "2024-11-08T16:32:32Z")
@mcanto73, try docker compose down on one of the nodes and set STORJ\_STORAGE2\_PIECE\_SCAN\_ON\_STARTUP=true to allow the filewalker to run at startup. You can set it back to false afterwards ---
**Author:** ![snorkel](https://storj.bcdn.literatehosting.com/letter_avatar/snorkel/32/5_5575768a8748004e209b776fc1b2916d.png) [@snorkel](https://forum.storj.io/u/snorkel)\ **Post date:** [November 8, 2024, 5:08pm UTC](https://forum.storj.io/t/high-trash-usage-27-tb-no-more-uploads/28471/6 "2024-11-08T17:08:50Z")
There were so many bugs regarding trash in the last months, and your config didn’t help either. To correct the problem, just put the lazzy on false and startup piece scan on true. Stop, rm and start the nodes. Let the scan finish for all sats and wait a few hours before changing back or restart the node (if you want). ---
**Author:** ![EasyRhino](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/easyrhino/32/1133_2.png) [@EasyRhino](https://forum.storj.io/u/EasyRhino)\ **Post date:** [November 8, 2024, 5:12pm UTC](https://forum.storj.io/t/high-trash-usage-27-tb-no-more-uploads/28471/7 "2024-11-08T17:12:16Z")
agree with snorkels advice. you may have missed the excitement over the summer, but there was a HUGE amount of test data that came in, filled up nodes, but now has all been trashed. your trash should have been deleted by now. So turn your filewalker back on (STORJ\_STORAGE2\_PIECE\_SCAN\_ON\_STARTUP=true) and give it a few days to finish. in addition, you can go into your data folders and the “trash” folders and see if you have anything older than 7 days. if you do, you could always delete it manually, but the storj node _should_ be doing it itself. The trash deleting will also probably take multiple days, you are talking millions of files. ---
**Author:** ![mcanto73](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/mcanto73/32/1133_2.png) [@mcanto73](https://forum.storj.io/u/mcanto73)\ **Post date:** [November 8, 2024, 7:05pm UTC](https://forum.storj.io/t/high-trash-usage-27-tb-no-more-uploads/28471/8 "2024-11-08T19:05:52Z")
Hello, I will try to change node parameters. What do you mean with rm? Do you mean manually remove files from trash ? How ? Is there a single folder ? Can this broke something: maybe the node still believe to have this darà? Thanks ---
**Author:** ![snorkel](https://storj.bcdn.literatehosting.com/letter_avatar/snorkel/32/5_5575768a8748004e209b776fc1b2916d.png) [@snorkel](https://forum.storj.io/u/snorkel)\ **Post date:** [November 8, 2024, 9:46pm UTC](https://forum.storj.io/t/high-trash-usage-27-tb-no-more-uploads/28471/9 "2024-11-08T21:46:18Z")
No no no! In order to change a docker command parameter, you stop the container, remove the container and start with new command. Maybe in docker compose is different, I don’t use it. You don’t delete/remove any file! ---
**Author:** ![EasyRhino](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/easyrhino/32/1133_2.png) [@EasyRhino](https://forum.storj.io/u/EasyRhino)\ **Post date:** [November 8, 2024, 9:46pm UTC](https://forum.storj.io/t/high-trash-usage-27-tb-no-more-uploads/28471/10 "2024-11-08T21:46:37Z")
if you go into the trash folder there are four subfolders with really long id’s, that’s one for each storj satellite: [Satellite info (Address, ID, Blobs folder, Hex) - Node Operators / FAQ - Storj Community Forum (official)](https://forum.storj.io/t/satellite-info-address-id-blobs-folder-hex/17183) for our purposes that doesn’t really matter. inside each one, they will be broken down by date. they were already shuffled there by the garbage collection job (taken out of “blobs” and put into ‘trash’) if the date is older than seven days. it shouldn’t be there. maybe the emptying trash job has failed or is just waaay behind. So you shouldn’t _have_ to delete folders older than seven days, but it won’t hurt anything if you do it. And again these may have millions of files so a simple “rm” or windows delete command could take many many hours. ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [November 9, 2024, 2:48am UTC](https://forum.storj.io/t/high-trash-usage-27-tb-no-more-uploads/28471/11 "2024-11-09T02:48:54Z")
@snorkel mean the `docker rm storagenode` command to remove the previously stopped container and then run it back with all your parameters. However, for the docker compose it’s as simple as `docker compose up -d` and it will do everything automatically. ---
**Author:** ![mcanto73](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/mcanto73/32/1133_2.png) [@mcanto73](https://forum.storj.io/u/mcanto73)\ **Post date:** [November 9, 2024, 8:28am UTC](https://forum.storj.io/t/high-trash-usage-27-tb-no-more-uploads/28471/12 "2024-11-09T08:28:05Z")
Hi All, I have deleted all trash files that has more than 7 days. cd /STORJ5/storage/trash for n in $(ls) ; do echo $n; /usr/bin/find $n -type f -mtime +8 -name “\*” -print0 | xargs -r0 rm -v --; done the space moved from 11T 7.4T 3.1T 71% /STORJ → 11T 6.8T 3.7T 65% /STORJ so more or less 0.6 TB was in the Trash unfortunately still no uploads coming and nothing seems to be changed in the node report (see attached file)… I restarted a lot of time, now I disabled lazy and enable piece scan. Thanks > **(attachments)** > > ![node_after_trash_deletion.png](https://storj-s3.bcdn.literatehosting.net/original/3X/5/e/5e5e9f794f15861636dccab466bc1db975cf3e93.png) ---
**Author:** ![nerdatwork](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/nerdatwork/32/1135_2.png) [@nerdatwork](https://forum.storj.io/u/nerdatwork)\ **Post date:** [November 9, 2024, 8:35am UTC](https://forum.storj.io/t/high-trash-usage-27-tb-no-more-uploads/28471/13 "2024-11-09T08:35:25Z")
What is the version of your node with no upload ? ---
**Author:** ![mcanto73](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/mcanto73/32/1133_2.png) [@mcanto73](https://forum.storj.io/u/mcanto73)\ **Post date:** [November 9, 2024, 8:39am UTC](https://forum.storj.io/t/high-trash-usage-27-tb-no-more-uploads/28471/14 "2024-11-09T08:39:25Z")
> [@mcanto73](#): > > 11T 7.4T 3.1T 71% /STORJ it’s 1.114.6 likes other 3 nodes. Another one is 1.115.5 for sure this is not the issue, the problem is that he is reporting 0 free space ! see previous email Best regards ---
**Author:** ![snorkel](https://storj.bcdn.literatehosting.com/letter_avatar/snorkel/32/5_5575768a8748004e209b776fc1b2916d.png) [@snorkel](https://forum.storj.io/u/snorkel)\ **Post date:** [November 9, 2024, 4:05pm UTC](https://forum.storj.io/t/high-trash-usage-27-tb-no-more-uploads/28471/15 "2024-11-09T16:05:12Z")
You won’t see any change until the used space scan finishes + 2 hours (I mean after the scan is finished, wait 2 more hours before doing anything). Restarting dosen’t do anything for the reported space. Only the scan rectifies the dashboard. ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [November 10, 2024, 3:43am UTC](https://forum.storj.io/t/high-trash-usage-27-tb-no-more-uploads/28471/16 "2024-11-10T03:43:17Z")
You need to enable the scan on startup if you disabled it (it’s enabled by default) and restart the node. The used-space-filewalker will calculate the actual usage and will update the databases. by the way, this was a mistake: > [@mcanto73](#): > > for n in $(ls) ; do echo $n; /usr/bin/find $n -type f -mtime +8 -name “\*” -print0 | xargs -r0 rm -v --; done You deleted also newest trash, maybe almost all of it. See ```auto $ sudo stat /mnt/w/storagenode5/storage/trash/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/2024-11-04/3q/cexe7papfxruuyaqdckgifbkhmldnaji2bbkiidqdzc6363f4a.sj1 File: /mnt/w/storagenode5/storage/trash/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/2024-11-04/3q/cexe7papfxruuyaqdckgifbkhmldnaji2bbkiidqdzc6363f4a.sj1 Size: 4864 Blocks: 16 IO Block: 512 regular file Device: 57h/87d Inode: 3659174698552822 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-10-27 14:11:33.583839900 +0300 Modify: 2024-10-27 14:11:33.583839900 +0300 Change: 2024-11-04 05:30:51.999237300 +0300 Birth: - ``` You need to use `-ctime`, not `-mtime`. And you can use the simplest script ```auto for n in $(ls) ; do echo $n; /usr/bin/find $n -type f -ctime +8 -print -delete; done ``` ---
**Author:** ![mcanto73](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/mcanto73/32/1133_2.png) [@mcanto73](https://forum.storj.io/u/mcanto73)\ **Post date:** [November 10, 2024, 6:30am UTC](https://forum.storj.io/t/high-trash-usage-27-tb-no-more-uploads/28471/17 "2024-11-10T06:30:19Z")
Hello, I’m running with these - STORJ\_PIECES\_ENABLE\_LAZY\_FILEWALKER=false - STORJ\_STORAGE2\_PIECE\_SCAN\_ON\_STARTUP=true Still no luck It reports 0 free space and no uploads Best regards ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [November 10, 2024, 6:45am UTC](https://forum.storj.io/t/high-trash-usage-27-tb-no-more-uploads/28471/18 "2024-11-10T06:45:03Z")
Did it finish the scan? ```auto docker logs storagenode 2>&1 | grep "\sused-space" | grep -E "started|completed" ``` ---
**Author:** ![mcanto73](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/mcanto73/32/1133_2.png) [@mcanto73](https://forum.storj.io/u/mcanto73)\ **Post date:** [November 10, 2024, 6:47pm UTC](https://forum.storj.io/t/high-trash-usage-27-tb-no-more-uploads/28471/19 "2024-11-10T18:47:26Z")
Hello, my node is running since 34 hours with lazy filewalker disabled and piece scan enabled storjlabs/storagenode:latest “/entrypoint” 34 hours ago These are my logs after the parameters change: ubuntu@hpool:/STORJ\_LOCAL-5/LOG$ zcat node.log.1.gz | grep “\sused-space” | grep -E “started|completed” 2024-11-09T08:16:05Z INFO pieces used-space-filewalker started {“Process”: “storagenode”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”} 2024-11-09T09:17:17Z INFO pieces used-space-filewalker started {“Process”: “storagenode”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”} 2024-11-09T09:17:17Z INFO pieces used-space-filewalker started {“Process”: “storagenode”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”} 2024-11-09T09:17:17Z INFO pieces used-space-filewalker started {“Process”: “storagenode”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”} 2024-11-09T09:17:25Z INFO pieces used-space-filewalker started {“Process”: “storagenode”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”} 2024-11-09T15:59:59Z INFO pieces used-space-filewalker started {“Process”: “storagenode”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”} 2024-11-09T15:59:59Z INFO pieces used-space-filewalker started {“Process”: “storagenode”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”} 2024-11-09T15:59:59Z INFO pieces used-space-filewalker started {“Process”: “storagenode”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”} 2024-11-09T16:00:08Z INFO pieces used-space-filewalker started {“Process”: “storagenode”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”} ubuntu@hpool:/STORJ\_LOCAL-5/LOG$ cat node.log | grep “\sused-space” | grep -E “started|completed” 2024-11-10T04:39:27Z INFO pieces used-space-filewalker completed {“Process”: “storagenode”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Lazy File Walker”: false, “Total Pieces Size”: 4924677327714, “Total Pieces Content Size”: 4910494774626, “Total Pieces Count”: 27700299, “Duration”: “12h39m18.037612122s”} 2024-11-10T04:39:27Z INFO pieces used-space-filewalker started {“Process”: “storagenode”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”} 2024-11-10T04:45:21Z INFO pieces used-space-filewalker completed {“Process”: “storagenode”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Lazy File Walker”: false, “Total Pieces Size”: 10708280576, “Total Pieces Content Size”: 10701173504, “Total Pieces Count”: 13881, “Duration”: “5m54.47502696s”} 2024-11-10T04:45:21Z INFO pieces used-space-filewalker started {“Process”: “storagenode”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”} 2024-11-10T13:45:17Z INFO pieces used-space-filewalker started {“Process”: “storagenode”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”} 2024-11-10T13:45:33Z INFO pieces used-space-filewalker started {“Process”: “storagenode”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”} 2024-11-10T13:45:41Z INFO pieces used-space-filewalker started {“Process”: “storagenode”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”} 2024-11-10T13:45:41Z INFO pieces used-space-filewalker started {“Process”: “storagenode”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”} 2024-11-10T13:45:41Z INFO pieces used-space-filewalker started {“Process”: “storagenode”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”} 2024-11-10T13:45:47Z INFO pieces used-space-filewalker started {“Process”: “storagenode”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”} I can see a lot of “started” and just a couple of “completed”. Best regards ---
**Author:** ![arrogantrabbit](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/arrogantrabbit/32/11200_2.png) [@arrogantrabbit](https://forum.storj.io/u/arrogantrabbit)\ **Post date:** [November 10, 2024, 7:38pm UTC](https://forum.storj.io/t/high-trash-usage-27-tb-no-more-uploads/28471/20 "2024-11-10T19:38:11Z")
> [@Alexey](#): > > `$(ls)` Never parse output of ls. Never. Especially, if deleting data is involved. Further reading: [ParsingLs - Greg's Wiki](http://mywiki.wooledge.org/ParsingLs) I thought to fix the scriptfor you, but it is a minefield in its entirety. Throw it away, and write a much simpler one. [Next page](https://forum.storj.io/t/high-trash-usage-27-tb-no-more-uploads/28471.md?page=2)