Disk usage discrepancy?

jammerdan · March 19, 2024, 11:34am

Do you mean this part:

The values are retrieved by storagenode and stored in a local database:

cd /opt/storagenode1/config/storage
sqlite3 storage_usage.db "select timestamp,hex(satellite_id),at_rest_total,interval_end_time,julianday(interval_end_time) from storage_usage order by satellite_id,timestamp desc limit 6"
2024-01-08

elek · March 19, 2024, 11:43am

Yep. There are also more advanced queries in the thread:

(which makes it unnecessary to manually calculate the difference between rows. The interval of rows can be different and should be calculated for actual average usage)

Windows7ge · March 20, 2024, 1:38am

My utilization finally calmed down after I told it to use less space. I’m sitting at 88% now instead of pinned at 100% and storjnode crashing.

I don’t remember what the maximum recommended utilization was. Should I not use more than 85% 90% 95%? I’d like to configure my node to the max.

Alexey · March 20, 2024, 4:47am

What’s the error after crash?
The official recommendation is to have 10% free of a total capacity. But many Community members are going much riskier: something about 100GB free from 12TB capacity (it’s actually dangerous especially in case of possible bugs).

jammerdan · March 20, 2024, 4:52am

Obviously too advanced for me.
I have no idea what this is and what I am supposed to do with the values to get the information I want:

Debugging space usage discrepancies

Adapted usage query on the sqlite-database:

root@STORJ6:/storj/DBs# sqlite3 storage_usage.db "SELECT * FROM (SELECT timestamp, hex(satellite_id) AS satellite, at_rest_total, interval_end_time, julianday(interval_end_time), ROW_NUMBER() OVER (PARTITION BY satellite_id ORDER BY timestamp DESC) AS rown FROM storage_usage) WHERE rown <= 6 ORDER BY satellite,rown"
2024-01-31 00:00:00+00:00|7B2DE9D72C2E935F1918C058CAAF8ED00F0581639008707317FF1BD000000000|1144871492.78699|2024-01-31 12:08:11.655857+00:00|2460341.00569046|1
2024-01-30 00:00:00+00:00|7B2DE9D72C2E935F1918C058CAAF8ED00F0581639008707317FF1BD000000000|1905812703.83945|2024-01-30 22:07:58.305931+00:00|2460340.42220262|2
2024-01-29 00:00:00+00:00|7B2DE9D72C2E935F1918C058CAAF8ED00F0581639008707317FF1BD000000000|1963121158.57989|2024-01-29 22:49:17.618464+00:00|2460339.45089836|3
2024-01-28 00:00:00+00:00|7B2DE9D72C2E935F1918C058CAAF8ED00F0581639008707317FF1BD000000000|1962384981.24134|2024-01-28 22:48:33.406566+00:00|2460338.45038666|4
2024-01-27 00:00:00+00:00|7B2DE9D72C2E935F1918C058CAAF8ED00F0581639008707317FF1BD000000000|1960121297.41671|2024-01-27 22:48:21.61155+00:00|2460337.45025014|5
2024-01-26 00:00:00+00:00|7B2DE9D72C2E935F1918C058CAAF8ED00F0581639008707317FF1BD000000000|1964648129.49649|2024-01-26 22:49:49.495744+00:00|2460336.45126731|6
2024-01-31 00:00:00+00:00|84A74C2CD43C5BA76535E1F42F5DF7C287ED68D33522782F4AFABFDB40000000|366591057285.886|2024-01-31 12:20:56.281846+00:00|2460341.0145403|1
2024-01-30 00:00:00+00:00|84A74C2CD43C5BA76535E1F42F5DF7C287ED68D33522782F4AFABFDB40000000|607945919264.781|2024-01-30 22:21:20.621328+00:00|2460340.43148867|2
2024-01-29 00:00:00+00:00|84A74C2CD43C5BA76535E1F42F5DF7C287ED68D33522782F4AFABFDB40000000|625035496727.823|2024-01-29 23:09:06.727148+00:00|2460339.46466119|3
2024-01-28 00:00:00+00:00|84A74C2CD43C5BA76535E1F42F5DF7C287ED68D33522782F4AFABFDB40000000|625963109207.811|2024-01-28 23:18:00.108378+00:00|2460338.47083458|4
2024-01-27 00:00:00+00:00|84A74C2CD43C5BA76535E1F42F5DF7C287ED68D33522782F4AFABFDB40000000|629843985583.987|2024-01-27 23:24:09.029323+00:00|2460337.4751045|5
2024-01-26 00:00:00+00:00|84A74C2CD43C5BA76535E1F42F5DF7C287ED68D33522782F4AFABFDB40000000|629327864851.166|2024-01-26 23:21:56.485264+00:00|2460336.47357043|6
2024-01-31 00:00:00+00:00|A28B4F04E10BAE85D67F4C6CB82BF8D4C0F0F47A8EA72627524DEB6EC0000000|2550794586300.69|2024-01-31 10:34:11.078242+00:00|2460340.940406|1
2024-01-30 00:00:00+00:00|A28B4F04E10BAE85D67F4C6CB82BF8D4C0F0F47A8EA72627524DEB6EC0000000|6415699699302.59|2024-01-30 23:30:44.978691+00:00|2460340.47968726|2
2024-01-29 00:00:00+00:00|A28B4F04E10BAE85D67F4C6CB82BF8D4C0F0F47A8EA72627524DEB6EC0000000|5058033869119.73|2024-01-29 19:44:39.873507+00:00|2460339.32268373|3
2024-01-28 00:00:00+00:00|A28B4F04E10BAE85D67F4C6CB82BF8D4C0F0F47A8EA72627524DEB6EC0000000|6234512324652.41|2024-01-28 21:51:48.240841+00:00|2460338.41097501|4
2024-01-27 00:00:00+00:00|A28B4F04E10BAE85D67F4C6CB82BF8D4C0F0F47A8EA72627524DEB6EC0000000|5010395282047.32|2024-01-27 18:46:03.322426+00:00|2460337.28198289|5
2024-01-26 00:00:00+00:00|A28B4F04E10BAE85D67F4C6CB82BF8D4C0F0F47A8EA72627524DEB6EC0000000|5018996834657.59|2024-01-26 20:56:45.586654+00:00|2460336.37274985|6
2024-01-31 00:00:00+00:00|AF2C42003EFC826AB4361F73F9D890942146FE0EBE806786F8E7190800000000|2177062238235.85|2024-01-31 10:35:42.833848+00:00|2460340.94146799|1
2024-01-30 00:00:00+00:00|AF2C42003EFC826AB4361F73F9D890942146FE0EBE806786F8E7190800000000|4518726578542.36|2024-01-30 22:41:42.942761+00:00|2460340.44563591|2
2024-01-29 00:00:00+00:00|AF2C42003EFC826AB4361F73F9D890942146FE0EBE806786F8E7190800000000|4658415876735.9|2024-01-29 22:02:22.726171+00:00|2460339.41831859|3
2024-01-28 00:00:00+00:00|AF2C42003EFC826AB4361F73F9D890942146FE0EBE806786F8E7190800000000|5340204565777.73|2024-01-28 21:51:46.287986+00:00|2460338.41095241|4
2024-01-27 00:00:00+00:00|AF2C42003EFC826AB4361F73F9D890942146FE0EBE806786F8E7190800000000|7335583691130.42|2024-01-27 22:17:33.555502+00:00|2460337.4288606|5
2024-01-26 00:00:00+00:00|AF2C42003EFC826AB4361F73F9D890942146FE0EBE806786F8E7190800000000|9712548046971.15|2024-01-26 22:12:46.172614+00:00|2460336.42553441|6

Shab · March 20, 2024, 5:36pm

Hi,
I have 13 TB dedicated to Storj. 9.18TB of which is being used. I have an entire drive dedicated to Storj Data of 15TB. Currently Storj is taking up 14.8TB on the drive yet Storj says it is only taking 9.18TB. Please advise as Storj is consuming much more space than it is saying. I was following another person having the same issue but it went to a different language and translation was not good.

WS 2019
NTFS
intel NUC i9 32GB RAM.

Shab · March 20, 2024, 5:39pm

Other topic I was talking about

JWvdV · March 20, 2024, 5:42pm

OS?
File system?
Hardware?

Shab · March 20, 2024, 9:39pm

WS 2019
NTFS
intel NUC i9 32GB RAM.

JWvdV · March 20, 2024, 10:02pm

Output of:

fsutil fsinfo ntfsInfo {drive-letter}:

Shab · March 20, 2024, 10:17pm

fsutil fsinfo ntfsInfo E:
NTFS Volume Serial Number : 0x7624e1fe24e1c16b
NTFS Version : 3.1
LFS Version : 2.0
Total Sectors : 31,457,243,135 ( 14.6 TB)
Total Clusters : 3,932,155,391 ( 14.6 TB)
Free Clusters : 68,108,221 (259.8 GB)
Total Reserved Clusters : 1,136 ( 4.4 MB)
Reserved For Storage Reserve : 0 ( 0.0 KB)
Bytes Per Sector : 512
Bytes Per Physical Sector : 4096
Bytes Per Cluster : 4096 (4 KB)
Bytes Per FileRecord Segment : 1024
Clusters Per FileRecord Segment : 0
Mft Valid Data Length : 53.95 GB
Mft Start Lcn : 0x00000000000c0000
Mft2 Start Lcn : 0x0000000000000002
Mft Zone Start : 0x00000000bf2f2860
Mft Zone End : 0x00000000bf2f2940
MFT Zone Size : 896.00 KB
Max Device Trim Extent Count : 4096
Max Device Trim Byte Count : 0xffffffff
Max Volume Trim Extent Count : 62
Max Volume Trim Byte Count : 0x40000000

Aitor · March 20, 2024, 11:03pm

In most of the post I read about similar issues, the problem usually is the NTFS and the filewalker.

elek · March 21, 2024, 10:45am

They are explained in the wiki page what I linked (first post in the forum).

I assume you are interested about the exact storage which is maintained by the satellite.

But there is no such value. Satellite shares the bytes*hour usage for a certain period. The average period can be calculated (for that period) which dividing it with the duration of the period (diff between the previous and current end time)

Again: I would recommend to check the post, including the diagram…

jammerdan · March 21, 2024, 5:31pm

That’s not in question for me. I am trying to put that together with the “advanced” query

Debugging space usage discrepancies

Adapted usage query on the sqlite-database:

root@STORJ6:/storj/DBs# sqlite3 storage_usage.db "SELECT * FROM (SELECT timestamp, hex(satellite_id) AS satellite, at_rest_total, interval_end_time, julianday(interval_end_time), ROW_NUMBER() OVER (PARTITION BY satellite_id ORDER BY timestamp DESC) AS rown FROM storage_usage) WHERE rown <= 6 ORDER BY satellite,rown"
2024-01-31 00:00:00+00:00|7B2DE9D72C2E935F1918C058CAAF8ED00F0581639008707317FF1BD000000000|1144871492.78699|2024-01-31 12:08:11.655857+00:00|2460341.00569046|1
2024-01-30 00:00:00+00:00|7B2DE9D72C2E935F1918C058CAAF8ED00F0581639008707317FF1BD000000000|1905812703.83945|2024-01-30 22:07:58.305931+00:00|2460340.42220262|2
2024-01-29 00:00:00+00:00|7B2DE9D72C2E935F1918C058CAAF8ED00F0581639008707317FF1BD000000000|1963121158.57989|2024-01-29 22:49:17.618464+00:00|2460339.45089836|3
2024-01-28 00:00:00+00:00|7B2DE9D72C2E935F1918C058CAAF8ED00F0581639008707317FF1BD000000000|1962384981.24134|2024-01-28 22:48:33.406566+00:00|2460338.45038666|4
2024-01-27 00:00:00+00:00|7B2DE9D72C2E935F1918C058CAAF8ED00F0581639008707317FF1BD000000000|1960121297.41671|2024-01-27 22:48:21.61155+00:00|2460337.45025014|5
2024-01-26 00:00:00+00:00|7B2DE9D72C2E935F1918C058CAAF8ED00F0581639008707317FF1BD000000000|1964648129.49649|2024-01-26 22:49:49.495744+00:00|2460336.45126731|6
2024-01-31 00:00:00+00:00|84A74C2CD43C5BA76535E1F42F5DF7C287ED68D33522782F4AFABFDB40000000|366591057285.886|2024-01-31 12:20:56.281846+00:00|2460341.0145403|1
2024-01-30 00:00:00+00:00|84A74C2CD43C5BA76535E1F42F5DF7C287ED68D33522782F4AFABFDB40000000|607945919264.781|2024-01-30 22:21:20.621328+00:00|2460340.43148867|2
2024-01-29 00:00:00+00:00|84A74C2CD43C5BA76535E1F42F5DF7C287ED68D33522782F4AFABFDB40000000|625035496727.823|2024-01-29 23:09:06.727148+00:00|2460339.46466119|3
2024-01-28 00:00:00+00:00|84A74C2CD43C5BA76535E1F42F5DF7C287ED68D33522782F4AFABFDB40000000|625963109207.811|2024-01-28 23:18:00.108378+00:00|2460338.47083458|4
2024-01-27 00:00:00+00:00|84A74C2CD43C5BA76535E1F42F5DF7C287ED68D33522782F4AFABFDB40000000|629843985583.987|2024-01-27 23:24:09.029323+00:00|2460337.4751045|5
2024-01-26 00:00:00+00:00|84A74C2CD43C5BA76535E1F42F5DF7C287ED68D33522782F4AFABFDB40000000|629327864851.166|2024-01-26 23:21:56.485264+00:00|2460336.47357043|6
2024-01-31 00:00:00+00:00|A28B4F04E10BAE85D67F4C6CB82BF8D4C0F0F47A8EA72627524DEB6EC0000000|2550794586300.69|2024-01-31 10:34:11.078242+00:00|2460340.940406|1
2024-01-30 00:00:00+00:00|A28B4F04E10BAE85D67F4C6CB82BF8D4C0F0F47A8EA72627524DEB6EC0000000|6415699699302.59|2024-01-30 23:30:44.978691+00:00|2460340.47968726|2
2024-01-29 00:00:00+00:00|A28B4F04E10BAE85D67F4C6CB82BF8D4C0F0F47A8EA72627524DEB6EC0000000|5058033869119.73|2024-01-29 19:44:39.873507+00:00|2460339.32268373|3
2024-01-28 00:00:00+00:00|A28B4F04E10BAE85D67F4C6CB82BF8D4C0F0F47A8EA72627524DEB6EC0000000|6234512324652.41|2024-01-28 21:51:48.240841+00:00|2460338.41097501|4
2024-01-27 00:00:00+00:00|A28B4F04E10BAE85D67F4C6CB82BF8D4C0F0F47A8EA72627524DEB6EC0000000|5010395282047.32|2024-01-27 18:46:03.322426+00:00|2460337.28198289|5
2024-01-26 00:00:00+00:00|A28B4F04E10BAE85D67F4C6CB82BF8D4C0F0F47A8EA72627524DEB6EC0000000|5018996834657.59|2024-01-26 20:56:45.586654+00:00|2460336.37274985|6
2024-01-31 00:00:00+00:00|AF2C42003EFC826AB4361F73F9D890942146FE0EBE806786F8E7190800000000|2177062238235.85|2024-01-31 10:35:42.833848+00:00|2460340.94146799|1
2024-01-30 00:00:00+00:00|AF2C42003EFC826AB4361F73F9D890942146FE0EBE806786F8E7190800000000|4518726578542.36|2024-01-30 22:41:42.942761+00:00|2460340.44563591|2
2024-01-29 00:00:00+00:00|AF2C42003EFC826AB4361F73F9D890942146FE0EBE806786F8E7190800000000|4658415876735.9|2024-01-29 22:02:22.726171+00:00|2460339.41831859|3
2024-01-28 00:00:00+00:00|AF2C42003EFC826AB4361F73F9D890942146FE0EBE806786F8E7190800000000|5340204565777.73|2024-01-28 21:51:46.287986+00:00|2460338.41095241|4
2024-01-27 00:00:00+00:00|AF2C42003EFC826AB4361F73F9D890942146FE0EBE806786F8E7190800000000|7335583691130.42|2024-01-27 22:17:33.555502+00:00|2460337.4288606|5
2024-01-26 00:00:00+00:00|AF2C42003EFC826AB4361F73F9D890942146FE0EBE806786F8E7190800000000|9712548046971.15|2024-01-26 22:12:46.172614+00:00|2460336.42553441|6

and what you have said before:

For me it looks like this when I take the first 2 rows:

2024-01-31 00:00:00+00:00|7B2DE9D72C2E935F1918C058CAAF8ED00F0581639008707317FF1BD000000000|1144871492.78699|2024-01-31 12:08:11.655857+00:00|2460341.00569046|1
2024-01-30 00:00:00+00:00|7B2DE9D72C2E935F1918C058CAAF8ED00F0581639008707317FF1BD000000000|1905812703.83945|2024-01-30 22:07:58.305931+00:00|2460340.42220262|2

Byte hours: 1144871492.78699
Interval: 2460341.00569046-2460340.42220262 * 24
Average: 1144871492.78699/14,00370816
= 81754880,90055927 |~ 8,17 TB

jammerdan · March 25, 2024, 9:32am

Another discrepancy on a node: Node says a bit less than 3 TB is used, trash negligible.
Then I wanted to move this to a 4TB partition and it cannot be moved because the 4TB partition is full before the node data is fully moved. Crazy.

daki82 · March 25, 2024, 9:50am

May it be a different cluster size on the new partition?

jammerdan · March 25, 2024, 10:11am

I have not been able to closely investigate.
But cluster sizes, block seizes is all the same.
I’ll have to check if reported node sizes are correct at all. Maybe the node is in fact bigger than what it is saying.

Vicente · March 25, 2024, 11:36am

Delete all databases. I thought that this would recover the capacity of the real node in the dashboard.

The dashboard capacity is not recovered, the node says that it is almost empty, the node is filling up, the node is becoming overloaded.

The node does not stop receiving data and if you do nothing the hard drive will fill up completely in a few weeks.

Is there a way for the node to mark the actual capacity? Can the entry of parts be stopped? Do I consider this node lost?

snorkel · March 25, 2024, 11:51am

Let the Filewalker finish. It will update the database.

daki82 · March 25, 2024, 1:14pm

Uncheck this (red arrow) and wait for it.

This causes extra iops for the node drive. Also slows down filewalker propably.