Deleted files not cleaned after more than 2 weeks

Node Operators troubleshooting
1

Hello,
I have a running node that have 3.13TB on trash since at least 2 weeks and never deleted:

image
image523Ă—520 23.1 KB

looking trash folder on my node I see many directory very old still, why ?

How can be solved ?

fabrizio@storjnodes:/SNOUK$ sudo ls -l storage/trash/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa
total 144
drwx------ 9 root root 4096 Jul 14 01:52 2024-07-13
drwx------ 154 root root 4096 Jul 18 21:24 2024-07-14
drwx------ 650 root root 12288 Jul 31 17:38 2024-07-27
drwx------ 78 root root 4096 Aug 5 03:17 2024-08-03
drwx------ 801 root root 20480 Aug 9 16:35 2024-08-07
drwx------ 305 root root 4096 Sep 7 13:02 2024-09-06
drwx------ 360 root root 12288 Sep 11 06:57 2024-09-09
drwx------ 156 root root 4096 Sep 15 08:27 2024-09-11
drwx------ 47 root root 4096 Sep 17 06:25 2024-09-15
drwx------ 86 root root 4096 Sep 22 06:59 2024-09-20
drwx------ 87 root root 4096 Sep 25 02:50 2024-09-22
drwx------ 326 root root 4096 Sep 29 02:55 2024-09-28
drwx------ 348 root root 12288 Oct 1 13:46 2024-09-29
drwx------ 605 root root 12288 Oct 8 03:46 2024-10-02
drwx------ 503 root root 12288 Oct 13 07:43 2024-10-12
drwx------ 44 root root 4096 Oct 16 03:48 2024-10-13
drwx------ 269 root root 4096 Oct 16 20:09 2024-10-16
drwx------ 166 root root 4096 Oct 20 18:15 2024-10-19
drwx------ 28 root root 4096 Oct 24 03:51 2024-10-20
drwx------ 64 root root 4096 Oct 27 18:33 2024-10-24
drwx------ 18 root root 4096 Oct 29 06:42 2024-10-27
fabrizio@storjnodes:/SNOUK$ sudo ls -l storage/trash/qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa
total 0
fabrizio@storjnodes:/SNOUK$ sudo ls -l storage/trash/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa
total 76
drwx------ 502 root root 12288 Sep 28 06:44 2024-09-25
drwx------ 469 root root 12288 Oct 2 01:13 2024-10-01
drwx------ 655 root root 12288 Oct 12 10:00 2024-10-10
drwx------ 814 root root 20480 Oct 19 12:36 2024-10-16
drwx------ 841 root root 20480 Oct 29 22:31 2024-10-29

2

grep your storagenode logs for “empty”, see if the emptying trash process is running, or completed, or failed.

if there is indication in the node that the trash empty failed, you could fix the underlying problem if there is one, restart the node and it should restart the trash emptying.

Can you if your hard disk is under load? (i use atop) It could be taking an exceptionally long time. (3TB could actually take weeks on a slow disk. At least days).

Finally, you can, manually, delete the trash folders older than 7 days old. Some folks did this when there was the deluge of trash from the test data this summer. even the linux rm command can be slow, there is a way to use the find command to pipe into rm to make it a bit faster. No I don’t know the exact syntax.

1 Like
3

Look at node logs. You will see deletions. Deleting files is slow on any filesystem. Leave it be, it wil complete eventually.

4

looking the log i just see:
fabrizio@storjnodes:/$ docker logs --tail 100000000 storagenode | grep empty
2024-10-30T09:05:55Z INFO pieces:trash emptying trash started {“Process”: “storagenode”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”}
2024-10-30T09:05:55Z INFO pieces:trash emptying trash finished {“Process”: “storagenode”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “elapsed”: “58.283589ms”}
2024-10-30T09:05:55Z INFO pieces:trash emptying trash started {“Process”: “storagenode”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”}

5

Be patient. Check back in one week.