# 4 TB node has been full the past few months, still only 2.17 TBm storage paid **URL:** \ **Category:** troubleshooting\ **Tags:** storage, btrfs, web-dashboard\ **Created:** [May 30, 2023, 5:41pm UTC](https://forum.storj.io/t/4-tb-node-has-been-full-the-past-few-months-still-only-2-17-tbm-storage-paid/22821 "2023-05-30T17:41:42Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Sfynx](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/sfynx/32/1121_2.png) [@Sfynx](https://forum.storj.io/u/Sfynx)\ **Post date:** [May 30, 2023, 5:41pm UTC](https://forum.storj.io/t/4-tb-node-has-been-full-the-past-few-months-still-only-2-17-tbm-storage-paid/22821/1 "2023-05-30T17:41:42Z")
The container of my storage node is indeed having 4 TB of data on it, but the payment and average disk space is 1.8 TB lower: Anybody any advice? I hope this picture makes the picture a bit more clear: ![image](https://storj-s3.bcdn.literatehosting.net/original/2X/3/39600d811fe57fc1927d4253fcc57df739e6afa2.png) ---
**Author:** ![zeebo](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/zeebo/32/7300_2.png) [@zeebo](https://forum.storj.io/u/zeebo)\ **Post date:** [May 30, 2023, 6:18pm UTC](https://forum.storj.io/t/4-tb-node-has-been-full-the-past-few-months-still-only-2-17-tbm-storage-paid/22821/2 "2023-05-30T18:18:13Z")
Hmm… Can you tell me what version your storagenode is running? Also, if you have logs, can you look for entries that contain the string “retain”? ---
**Author:** ![Sfynx](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/sfynx/32/1121_2.png) [@Sfynx](https://forum.storj.io/u/Sfynx)\ **Post date:** [May 30, 2023, 7:13pm UTC](https://forum.storj.io/t/4-tb-node-has-been-full-the-past-few-months-still-only-2-17-tbm-storage-paid/22821/3 "2023-05-30T19:13:56Z")
Version is: 1.78.3 But the issue has been there for some time (also in previous versions), just was not sure if my interpretation of average disk space used this month was correct, so waited till the node was full for a month, in any definition the average of a full month full should be the full amount. :-). docker logs |grep retain gives 0 results ---
**Author:** ![zeebo](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/zeebo/32/7300_2.png) [@zeebo](https://forum.storj.io/u/zeebo)\ **Post date:** [May 30, 2023, 7:25pm UTC](https://forum.storj.io/t/4-tb-node-has-been-full-the-past-few-months-still-only-2-17-tbm-storage-paid/22821/4 "2023-05-30T19:25:29Z")
How long do those logs go back? I’d expect a retain call around a couple of times per week. ---
**Author:** ![Sfynx](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/sfynx/32/1121_2.png) [@Sfynx](https://forum.storj.io/u/Sfynx)\ **Post date:** [May 30, 2023, 7:30pm UTC](https://forum.storj.io/t/4-tb-node-has-been-full-the-past-few-months-still-only-2-17-tbm-storage-paid/22821/5 "2023-05-30T19:30:51Z")
less then 24 hours, due to os reboot. ---
**Author:** ![zeebo](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/zeebo/32/7300_2.png) [@zeebo](https://forum.storj.io/u/zeebo)\ **Post date:** [May 30, 2023, 7:39pm UTC](https://forum.storj.io/t/4-tb-node-has-been-full-the-past-few-months-still-only-2-17-tbm-storage-paid/22821/6 "2023-05-30T19:39:25Z")
Dang. That’s not unexpected that there would be no retain calls then. The only thing I can think of is that for some reason the garbage collection process isn’t working for your node. Can you keep watching the logs for lines with “retain” in them? It might take a couple of days, unfortunately. ---
**Author:** ![Sfynx](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/sfynx/32/1121_2.png) [@Sfynx](https://forum.storj.io/u/Sfynx)\ **Post date:** [May 30, 2023, 7:56pm UTC](https://forum.storj.io/t/4-tb-node-has-been-full-the-past-few-months-still-only-2-17-tbm-storage-paid/22821/7 "2023-05-30T19:56:24Z")
I will get back to you when there are some log lines with retain in them. Thanks a lot for responding so far. ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [May 31, 2023, 4:36am UTC](https://forum.storj.io/t/4-tb-node-has-been-full-the-past-few-months-still-only-2-17-tbm-storage-paid/22821/8 "2023-05-31T04:36:19Z")
Hello @Sfynx, Welcome to the forum! Please check all your databases: > **[How to fix a "database disk image is malformed"](https://support.storj.io/hc/en-us/articles/360029309111-How-to-fix-a-database-disk-image-is-malformed-)** > > Problem > Sometimes a Storage Node Operator may encounter the "database disk image is malformed" error in their log. This could happen during unplanned shutdown or reboot. The error indicates that on... What’s filesystem on your disk? ```auto df -T --si ``` ---
**Author:** ![Sfynx](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/sfynx/32/1121_2.png) [@Sfynx](https://forum.storj.io/u/Sfynx)\ **Post date:** [May 31, 2023, 3:46pm UTC](https://forum.storj.io/t/4-tb-node-has-been-full-the-past-few-months-still-only-2-17-tbm-storage-paid/22821/9 "2023-05-31T15:46:56Z")
Thanks for the welcome, FS is btrfs Results for all db checks return a ok: ./info.dbok ./bandwidth.dbok ./orders.dbok ./piece\_expiration.dbok ./pieceinfo.dbok ./piece\_spaced\_used.dbok ./reputation.dbok ./storage\_usage.dbok ./used\_serial.dbok ./satellites.dbok ./notifications.dbok ./heldamount.dbok ./pricing.dbok ./secret.dbok ./notifications.db.dbok Also fsck on filesystem show all is correct. The actual amount on the ‘blobs’ directory is that 4GB… Ow, still no retain in the log’s… so will report back on that when that happens. Thanks for all the help so far. ---
**Author:** ![nyancodex](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/nyancodex/32/1129_2.png) [@nyancodex](https://forum.storj.io/u/nyancodex)\ **Post date:** [May 31, 2023, 4:47pm UTC](https://forum.storj.io/t/4-tb-node-has-been-full-the-past-few-months-still-only-2-17-tbm-storage-paid/22821/10 "2023-05-31T16:47:22Z")
This happened to me before. You should try to recreate all databases. And don’t forget to backup before doing. ---
**Author:** ![Sfynx](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/sfynx/32/1121_2.png) [@Sfynx](https://forum.storj.io/u/Sfynx)\ **Post date:** [June 1, 2023, 7:21am UTC](https://forum.storj.io/t/4-tb-node-has-been-full-the-past-few-months-still-only-2-17-tbm-storage-paid/22821/11 "2023-06-01T07:21:52Z")
Tried that, removed all db’s… let them rebuild, took a while for the data to return, but still the same result. Day 3 still no retain… will keep you all posted. ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [June 1, 2023, 7:32am UTC](https://forum.storj.io/t/4-tb-node-has-been-full-the-past-few-months-still-only-2-17-tbm-storage-paid/22821/12 "2023-06-01T07:32:34Z")
> [@Sfynx](#): > > FS is btrfs This could be a reason, unfortunately. It’s also proven as slow for Storj and have many other issues, see [Topics tagged btrfs](https://forum.storj.io/tag/btrfs) If you already rebuilt databases, then seems you cannot do anything else. You may also check the actual usage (replace to your path): ```auto du -s --si --apparent-size /mnt/storj/storagenode/storage ``` ---
**Author:** ![nyancodex](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/nyancodex/32/1129_2.png) [@nyancodex](https://forum.storj.io/u/nyancodex)\ **Post date:** [June 1, 2023, 10:28am UTC](https://forum.storj.io/t/4-tb-node-has-been-full-the-past-few-months-still-only-2-17-tbm-storage-paid/22821/13 "2023-06-01T10:28:25Z")
for me it took a month, something are scheduled to run monthly ---
**Author:** ![Ottetal](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/ottetal/32/18876_2.png) [@Ottetal](https://forum.storj.io/u/Ottetal)\ **Post date:** [June 6, 2023, 9:38am UTC](https://forum.storj.io/t/4-tb-node-has-been-full-the-past-few-months-still-only-2-17-tbm-storage-paid/22821/14 "2023-06-06T09:38:14Z")
Did you ever get a fix to this problem? I Have the same issue on one of my containers 🙂 ---
**Author:** ![arrogantrabbit](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/arrogantrabbit/32/11200_2.png) [@arrogantrabbit](https://forum.storj.io/u/arrogantrabbit)\ **Post date:** [June 6, 2023, 11:43am UTC](https://forum.storj.io/t/4-tb-node-has-been-full-the-past-few-months-still-only-2-17-tbm-storage-paid/22821/15 "2023-06-06T11:43:17Z")
Do [this](https://forum.storj.io/t/4-tb-node-has-been-full-the-past-few-months-still-only-2-17-tbm-storage-paid/22821/12) and if you see correct size — the problem is large sector size. ---
**Author:** ![Sfynx](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/sfynx/32/1121_2.png) [@Sfynx](https://forum.storj.io/u/Sfynx)\ **Post date:** [June 6, 2023, 4:05pm UTC](https://forum.storj.io/t/4-tb-node-has-been-full-the-past-few-months-still-only-2-17-tbm-storage-paid/22821/16 "2023-06-06T16:05:47Z")
Done that: du -s --si --apparent-size, the results is 4.1 TB too. Problem still remains… it did jump to 2.25 TBm though… no clue why. Still no retain, but i really need to get this docker image to write it’s logs outside the container, every restart i loose the logging. hoping to get some insights from that retain logging. ---
**Author:** ![Sfynx](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/sfynx/32/1121_2.png) [@Sfynx](https://forum.storj.io/u/Sfynx)\ **Post date:** [June 6, 2023, 4:58pm UTC](https://forum.storj.io/t/4-tb-node-has-been-full-the-past-few-months-still-only-2-17-tbm-storage-paid/22821/17 "2023-06-06T16:58:18Z")
logging is fixed, and made a small script when my logs mention retain… Hope this will catch the suggestion of zeebo. I’ll keep you all posted ---
**Author:** ![Sfynx](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/sfynx/32/1121_2.png) [@Sfynx](https://forum.storj.io/u/Sfynx)\ **Post date:** [June 8, 2023, 7:11am UTC](https://forum.storj.io/t/4-tb-node-has-been-full-the-past-few-months-still-only-2-17-tbm-storage-paid/22821/18 "2023-06-08T07:11:19Z")
Three of the six satellites have started done some retain actions: these where in the logs… 2023-06-07T21:45:01.796Z INFO retain Prepared to run a Retain request. {“Process”: “storagenode”, “Created Before”: “2023-04-16T02:09:42.044Z”, “Filter Size”: 29, “Satellite ID”: “12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo”} 2023-06-07T21:45:02.031Z INFO retain Moved pieces to trash during retain {“Process”: “storagenode”, “num deleted”: 0, “Retain Status”: “enabled”} 2023-06-07T23:13:13.245Z INFO retain Prepared to run a Retain request. {“Process”: “storagenode”, “Created Before”: “2023-06-04T05:59:59.694Z”, “Filter Size”: 8625, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”} 2023-06-07T23:13:17.535Z INFO retain Moved pieces to trash during retain {“Process”: “storagenode”, “num deleted”: 0, “Retain Status”: “enabled”} 2023-06-08T06:41:30.642Z INFO retain Prepared to run a Retain request. {“Process”: “storagenode”, “Created Before”: “2023-01-22T20:31:29.101Z”, “Filter Size”: 1422, “Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”} 2023-06-08T06:41:31.906Z INFO retain Moved pieces to trash during retain {“Process”: “storagenode”, “num deleted”: 0, “Retain Status”: “enabled”} ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [June 8, 2023, 7:36am UTC](https://forum.storj.io/t/4-tb-node-has-been-full-the-past-few-months-still-only-2-17-tbm-storage-paid/22821/19 "2023-06-08T07:36:53Z")
So it should remove deleted/expired data from your node. Will hope it will match what’s reported. ---
**Author:** ![Sfynx](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/sfynx/32/1121_2.png) [@Sfynx](https://forum.storj.io/u/Sfynx)\ **Post date:** [June 19, 2023, 4:18pm UTC](https://forum.storj.io/t/4-tb-node-has-been-full-the-past-few-months-still-only-2-17-tbm-storage-paid/22821/20 "2023-06-19T16:18:31Z")
Small update: There still is a discrepancy between total used disk space and average disk space used this month. 5/6 satellites have had a retain thing done so far. only [us1.storj.io](http://us1.storj.io) is missing yet… hopeing that will clean out a lot of files. [Next page](https://forum.storj.io/t/4-tb-node-has-been-full-the-past-few-months-still-only-2-17-tbm-storage-paid/22821.md?page=2)