"Disk Space used this month" dropped

Hey
I found the “disk space used this month” my dashboard did not update since May 4.
I have checked everything but nothing goes wrong.
I’m sure my storagenode is online.
In the last month(April), it update every 12h.

And here is my last 20 log

Blockquote
[root@storj ~]# docker logs fbe92c53a0a5 --tail=20

2020-05-04T14:31:32.227Z INFO piecestore upload canceled {“Piece ID”: “XKEHEC4W6LMCZGZLKHVOUYCZHZRRYXHXPESEM5CFNEVWNOWAB3XA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”}

2020-05-04T14:31:32.966Z INFO piecestore upload canceled {“Piece ID”: “UBCNFWHG67J7MW42JFRBVQ6DUG5HGAOW4YUE2BWFDNOBK2RNPNCA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”}

2020-05-04T14:31:33.069Z INFO piecestore upload canceled {“Piece ID”: “4WMNHSE7KXCLZOGBZDX6MXOQWN6YODBUQ7RCVZ4GIMHNB4GUF6HA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”}

2020-05-04T14:31:33.815Z INFO piecestore upload started {“Piece ID”: “ZXH3SXFCG5MHO7FDRIB3PDLPYELUT2U3C3WFTRX4T25YPA5JFMEQ”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”, “Available Space”: 3108020629376}

2020-05-04T14:31:34.077Z INFO piecestore upload canceled {“Piece ID”: “TOR6DRBLUK63Y6FVGEPWH4NKFWQWJUWHVYS7P3JOWEKIJJYG5U2A”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”}

2020-05-04T14:31:34.129Z INFO piecestore upload started {“Piece ID”: “QJB77IQRMU6UJ5ZGASRP4QWILMGN46TT6HFOYO5RA263BNWTGOTQ”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”, “Available Space”: 3108020629376}

2020-05-04T14:31:34.130Z INFO piecestore upload canceled {“Piece ID”: “S5ASED5FDVROZ5QE7F77SU6PBS773Y3CYKLDR5A6CW3UHBLP6TXA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”}

2020-05-04T14:31:35.360Z INFO piecestore upload canceled {“Piece ID”: “WT5LWPATPGYZRFHUAEVESG6DLCVLNNE7SQFRMXN5XRJHOOAZQFBQ”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”}

2020-05-04T14:31:36.485Z INFO piecestore upload canceled {“Piece ID”: “UEGVW2F7DPNPPPM2TRDH2UAJQFPMFCD6ID7FX5RLC2GFLVLT5WMA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”}

2020-05-04T14:31:36.650Z INFO piecestore upload started {“Piece ID”: “2F6IW54QLFOKSKJ3EMX5OCY64B6HFPD3FBSOV3UF65T2QENOHCUA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”, “Available Space”: 3108020629376}

2020-05-04T14:31:36.791Z INFO piecestore upload canceled {“Piece ID”: “EQQXCBMF2LJOIVLUKWOVEJ5QPGLC6UXRZ4O3GE5YDDQ2V2KEBFFA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”}

2020-05-04T14:31:36.902Z INFO piecestore upload started {“Piece ID”: “2KFC22MWNTCVOKYLY7QS7JKRRBFL63BBHHFYLUQBGP677FMEZ4YQ”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”, “Available Space”: 3108020629376}

2020-05-04T14:31:37.291Z INFO piecestore upload started {“Piece ID”: “H4LRGZLZXNXTFLVVLXBKEA2ENPCPOCSWIZ4TGMW2UFPUGG5AUCDA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”, “Available Space”: 3108020629376}

2020-05-04T14:31:39.430Z INFO piecestore upload started {“Piece ID”: “I5Q25IOBBINUJSRKG2L5QQLIXPT3BTPEN7Y65OPYURAHC5OMF4VA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”, “Available Space”: 3108020629376}

2020-05-04T14:31:39.630Z INFO piecestore upload started {“Piece ID”: “OCJ5WNTYAFMMF2TDDVZPWRS2FFXYVTPIHSDU6XJ5JA7SRDGOCCHQ”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”, “Available Space”: 3108020629376}

2020-05-04T14:31:39.665Z INFO piecestore upload canceled {“Piece ID”: “B2FRHLMHXHMJWAZV2U5IJ2OEKZVCWVKZ7MACOAIBKMRGJ6UVZKMQ”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”}

2020-05-04T14:31:40.485Z INFO piecestore upload started {“Piece ID”: “S3J7YUUUNNLJIEMR62FTZKZWP5KK3ATPHRQA2QKATZ4UYCFLWKUQ”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”, “Available Space”: 3108020629376}

2020-05-04T14:31:42.008Z INFO piecestore upload started {“Piece ID”: “VAYBXNUAUUSSGOGIVZ4K2GYNIB5JMV6SEDAVFAYQA5PNHYNVADPQ”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”, “Available Space”: 3108020629376}

2020-05-04T14:31:42.421Z INFO piecestore upload canceled {“Piece ID”: “GVE7DTL7IAFUXXRBAT632WKLLCXDK2CQYQJHSEQFKXFPS4DNWBXA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”}

2020-05-04T14:31:42.541Z INFO piecestore upload canceled {“Piece ID”: “4CTVWZJGAEJZKSAFEEDWF6DYDB4AKWKTOZI6DDFSW45UWOLLKWYA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”}

1 Like

I have the same behaviour, only for the saltlake sattelite though

Hello everyone,

I am kind of a newbie to Storj even if I have been mining for 3 months. Recently, something weird happened to my node, and I cannot understand what is this problem. See the picture below.

I have a 4TB node running for 3 months, never turn off the node unless the docker needs to be updated, and it is working perfectly for 3 months.

However, on the day before yesterday, I found that the Disk Space Used dropped dramatically from 98 TB*h to 2 TB*h, but the Disk is still occupied (both showed in the dashboard and my computer), and the Ingress Bandwidth is quite low. Therefore, I think the data is not replaced by new data, and the node is still maintaining the old data, which should be counted for the Disk Space Used (in TB*h).

Initially, I suppose the information on the dashboard is not updated, so I wait for another day. As you can see in the picture, the Disk Space Used is 1.73 Tb*h yesterday, even lower than the day before.

How can I check what is the problem? Is anyone have the same problem here?

Thank you in advance for any suggestions.

I don’t think it’s a dashboard problem. Dashboard is using API for rendering data, the same API used for payouts calculations and everything. The problem is that satellites just see 5% of my space allocation online. And that is the space I will be paid for.
I have 3TB constantly online, but satellites see just 3TB for 2hr day (6TBh) instead of 3TB for 24h/day (72TBh).
This will heavily affect payments.

cool, thanks for the reply, appreciate it.

2 Likes

I’m seeing this drop as well on my side, and it’s not caused by vetting process according to earnings calculator:

Sudden drop in demand:

image

2 Likes

To be honest, I don’t trust the dashboard.

I’m collecting stats in Zabbix for all of my nodes and here is 7 days of egress traffic for one node:

image

7 days of storage on the same node:

image

This is fairly representative of all of my nodes. Some have been around since last year and two of them I started running last month.

The dashboard paints a totally different story for this node:

Both of these cannot be true at the same time, and I trust Zabbix’s historical data collection way more than storagenode’s – it is significantly more field-tested, and I have it set to capture a sample once per minute.

From what I’ve seen around the forum, storagenode collects a sample of timed storage usage twice per day (unless that changed in v1.3.3) which can result in significant variation in the displayed data where no such variation actually exists. I don’t know if there is a similar issue with ingress/egress usage.

I it just me or is it like the numbers for the deletions came in on the dashboard graph before on my actual system…

and now i get a graph after two days of near zero at 250TB*h, which doesn’t make any sense i got atleast 5tb… even now i got 5tb
and don’t tell me it’s deleted, because it isn’t at the speed thats being deleted its barely deleting anything at all … so far today its deleted 150k files thats only like 300gb and i have like or had like 6tb
so at current speed it will take 3 days to get below 5tb

so i should get atleast + 120TBh which seems right for the two first days.
then it drops to 9 or 8TB
h pr day, which to me seem like somebody is basically just sitting there putting in numbers… like the system reacted immediately to the data being deleted in the statistics, but the satellites taking a week to delete it all…
and now it jumps back up… to 250TB… looks very very weird…
i mean if the system was adding it all up and the error was fixed, then it should give 3 days in one… but it didn’t it gave me 2 days in one, which means tomorrow it should also give me 2 days in one, to catch up…

but that makes it seem like somebody actually decided the graph should move like that because of people yelling out at large swings…
i’m not saying it’s anything nefarious, nor means anything… just looks odd to me… makes me think there is a wizard of oz…

In my opinion

The saltlake satellite has been adding data at a huge rate. The process that counts up the storage used runs twice a day but has been getting longer and longer runs.
At some point (end of April) this became significant and runs started overlapping, this in turn caused the next run to take longer and longer until, May 3rd, no run were able to complete.
The storj response has been to stop uploading and to delete a lot of the little pieces that make up the data on storagenodes.
So you will see lots of little deletes for May 5th on.

No your node has not deleted TBs of data, yes the graph is out of date.

Pay will be calculated from the data not the graph.

/ In my opinion

i was saying it looks odd that the graph adjusts to whats going to be deleted on my node before it’s actually deleted… and then a bit of ramblings xD

i’m not worried, this is a perfect time for me to move my node yet again and maybe repair my server while the node will fit on my smaller computer.
aside from that this is great for testing how well everything operates, comparing with idle use cases with some … i don’t want to say high activity but highest activity i’ve seen thus far anyways…
i don’t expect us to be able to perform such tests in the future as the network will be more active all the time… and then we are stuck with guessing and what we can gauge from a network for a more consistent load.

and in regards to the pay… well not like i’m actually making any money on this yet hehe xD
the graph is just like ahead, when it should be behind… thats odd

Yes, I’ve noticed your posts are quite long :smiley:

Are you N1 by the way?

Hi,

Disk space used for one of my Storage node (hosting ~2.5-3TB of data) fell down from 75 TBh to 750GBh.

Is that normal?
If I well understand this diagram, it means that a lot of data have been erased on my node and then a lot of new data has been uploaded to it the next day.

Should I be worried about my node health?

Thanks for your help

No your node is probably fine. Check some other threads for full story but saltlake go a bit slow accounting

Thank you, I’m reassured :slight_smile:

In my opinion, the graph for TB*h is the projection of my allocation on the satellite, meaning that file piece must be present in my storage AND must be pointed by the satellite. If the satellite removes the pointer to my file, the piece and its allocation disappears from graphs instantly (and, in my opinion, from payments). The physical deletion of the file piece from my storage is asynchronous and will happen later.
I think that satellites in the past days removed all test data, causing allocation-projection drop, meaning I have now tons of orphan data, not pointed by satellites, not counted in graphs, not counted in payments, that slowly will get deleted in future days.

Just my thoughts.

1 Like

This didn’t happen. It’s merely a reporting issue. TBh is reported back to the nodes by the satellite after calculation is done. This process saw some delay, which is why you’re missing some data.

Additionally, when files are deleted and your node is online, those deletes will be passed on to your node right away. The only way garbage collection will have to kick in is when your node wasn’t online during the deletion of data or for some reason timed out on the initial deletion of data.

Thanks all!

So I suppose it’s also normal if I see my “Disk Space Remaining” being progressively higher day after day?
I would like to make sure that my node is not emptying out because of a problem on my node…

Just check the logs and make sure that uploads are still occurring along with audits and that you aren’t suspended on any satellites and that should be the easiest way to verify.

1 Like

Yes. Saltlake is removing some test data, but it hasn’t removed all of it (yet, anyway).

One of my nodes’ storage usage over the last week:

image

This node’s upload/deletion requests over the same time range (note the sharp increase in deletions in the past ~day):

image

2 Likes