"Disk Space used this month" dropped

Hello, after upgrading to 1.3.3, the “Disk Space used this month” dropped to almost zero.
Checks i did:

  • databases are ok
  • node has been always online, exept on 02/05 from 20:00 to 20:15 for maintenance (grace stopped), constantly monitored by uptime robot
  • os is reporting 90% of disk space is used. The dashboard is reporting the same.
  • logs do not have any error, i can set GET and GET_AUDIT operations

Thanks

2 Likes

Seeing the same behavior here on my nodes. So, I don’t think it’s unusual. Just the current behavior

Yes, but this is affecting the payments. My storage is almost full and it’s paid as empty?

2 Likes

same here… seem it’s just another bug in the new web dashboard.
but hey atleast they got fixed so it doesn’t jump all over the place lol
hopefully they will get it working soon… basically don’t trust anything the web dashboard tells you seems to be the winning attitude… lol

@niccotnt don’t confuse the web dashboard with the actual storagenode information…
i’m sure the underlying data is quite safe and sound… the web dashboard is just doing a poor job of interpreting it… or there is simply a coding error somewhere…

just because the dashboard breaks in your car, doesn’t mean everything else does…

i would be more worried about the errors 1.3.3 seems to give us in the logs… my successrate script says i’m failing audits because of that… tsk tsk
gives a usedserialdb error database is locked

No, but if the check engine light comes on then you should probably take it to the mechanic.

2 Likes

surely you mean the “we want more money” lamp, nah that can come on for all kinds of non critical issues also… xD

I had a brief power outage the other night and was concerned for my node. I checked all the databases for corruption and they all checked “ok.” I have not gotten any log errors related to the databases either. But on my dashboard the “Disk Space Used This Month” metric has precipitously dropped. I have attached a screenshot. This seems inaccurate to me and is causing me concern. Do I have a problem?Capture|690x486

You are not alone, mine is behaving like this too. There are already threads about it:

and also No diskspace used in dashboard graph .

Interesting, after reading that thread I checked the saltlake satellite utilization specifically and that is the one reporting 0bytes*hour. Other satellites seem to be reporting sensible usage statistics. Maybe there is no problem on my end at all. I have disabled write caching to be safe. I was really scared I messed up my node. (Air conditioner blew a fuse causing my node to lose power)

It almost certainly won’t. Space usage isn’t tracked locally, but by the satellites. The data has always been reported back to the nodes with a delay and the last 2 days can have incomplete information. A part of that is that the data is infrequently reported back to the node. Only once or twice a day. Give it a while to catch up. I’m pretty sure it will over the next few days and these days will display the actual usage.

But even if they don’t, the payout will be determined based on what your node stores according to the satellites db’s. So your payout is safe either way.

3 Likes

More out of curiosity, what would you guestimate the probability of a satellite’s db’s getting corrupted? Any idea of preventive measures in place to preserve their integrity?

Same as other, and logs are filled with deletion requests

Me too, only deletes in the logs. About 10 uploadeds today.

I remember they said they will delete stuff from one of the satelite but it looks like it’s more massive then that lol

I have the same behaviour. My saltlake sattelite has dropped to zero space used

If those get corrupted they have bigger problems than node payouts. Since they also store object metadata in satellite db’s, you can be sure those are built to be VERY stable. They also use a proper db for large and hugely parallel database access, since they deal with much more data and many more processes accessing it simultaneously. On the server side there isn’t really a downside to running a database server alongside the software unlike on the storagenode end.

1 Like

Hey
I found the “disk space used this month” my dashboard did not update since May 4.
I have checked everything but nothing goes wrong.
I’m sure my storagenode is online.
In the last month(April), it update every 12h.

And here is my last 20 log

Blockquote
[root@storj ~]# docker logs fbe92c53a0a5 --tail=20

2020-05-04T14:31:32.227Z INFO piecestore upload canceled {“Piece ID”: “XKEHEC4W6LMCZGZLKHVOUYCZHZRRYXHXPESEM5CFNEVWNOWAB3XA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”}

2020-05-04T14:31:32.966Z INFO piecestore upload canceled {“Piece ID”: “UBCNFWHG67J7MW42JFRBVQ6DUG5HGAOW4YUE2BWFDNOBK2RNPNCA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”}

2020-05-04T14:31:33.069Z INFO piecestore upload canceled {“Piece ID”: “4WMNHSE7KXCLZOGBZDX6MXOQWN6YODBUQ7RCVZ4GIMHNB4GUF6HA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”}

2020-05-04T14:31:33.815Z INFO piecestore upload started {“Piece ID”: “ZXH3SXFCG5MHO7FDRIB3PDLPYELUT2U3C3WFTRX4T25YPA5JFMEQ”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”, “Available Space”: 3108020629376}

2020-05-04T14:31:34.077Z INFO piecestore upload canceled {“Piece ID”: “TOR6DRBLUK63Y6FVGEPWH4NKFWQWJUWHVYS7P3JOWEKIJJYG5U2A”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”}

2020-05-04T14:31:34.129Z INFO piecestore upload started {“Piece ID”: “QJB77IQRMU6UJ5ZGASRP4QWILMGN46TT6HFOYO5RA263BNWTGOTQ”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”, “Available Space”: 3108020629376}

2020-05-04T14:31:34.130Z INFO piecestore upload canceled {“Piece ID”: “S5ASED5FDVROZ5QE7F77SU6PBS773Y3CYKLDR5A6CW3UHBLP6TXA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”}

2020-05-04T14:31:35.360Z INFO piecestore upload canceled {“Piece ID”: “WT5LWPATPGYZRFHUAEVESG6DLCVLNNE7SQFRMXN5XRJHOOAZQFBQ”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”}

2020-05-04T14:31:36.485Z INFO piecestore upload canceled {“Piece ID”: “UEGVW2F7DPNPPPM2TRDH2UAJQFPMFCD6ID7FX5RLC2GFLVLT5WMA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”}

2020-05-04T14:31:36.650Z INFO piecestore upload started {“Piece ID”: “2F6IW54QLFOKSKJ3EMX5OCY64B6HFPD3FBSOV3UF65T2QENOHCUA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”, “Available Space”: 3108020629376}

2020-05-04T14:31:36.791Z INFO piecestore upload canceled {“Piece ID”: “EQQXCBMF2LJOIVLUKWOVEJ5QPGLC6UXRZ4O3GE5YDDQ2V2KEBFFA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”}

2020-05-04T14:31:36.902Z INFO piecestore upload started {“Piece ID”: “2KFC22MWNTCVOKYLY7QS7JKRRBFL63BBHHFYLUQBGP677FMEZ4YQ”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”, “Available Space”: 3108020629376}

2020-05-04T14:31:37.291Z INFO piecestore upload started {“Piece ID”: “H4LRGZLZXNXTFLVVLXBKEA2ENPCPOCSWIZ4TGMW2UFPUGG5AUCDA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”, “Available Space”: 3108020629376}

2020-05-04T14:31:39.430Z INFO piecestore upload started {“Piece ID”: “I5Q25IOBBINUJSRKG2L5QQLIXPT3BTPEN7Y65OPYURAHC5OMF4VA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”, “Available Space”: 3108020629376}

2020-05-04T14:31:39.630Z INFO piecestore upload started {“Piece ID”: “OCJ5WNTYAFMMF2TDDVZPWRS2FFXYVTPIHSDU6XJ5JA7SRDGOCCHQ”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”, “Available Space”: 3108020629376}

2020-05-04T14:31:39.665Z INFO piecestore upload canceled {“Piece ID”: “B2FRHLMHXHMJWAZV2U5IJ2OEKZVCWVKZ7MACOAIBKMRGJ6UVZKMQ”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”}

2020-05-04T14:31:40.485Z INFO piecestore upload started {“Piece ID”: “S3J7YUUUNNLJIEMR62FTZKZWP5KK3ATPHRQA2QKATZ4UYCFLWKUQ”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”, “Available Space”: 3108020629376}

2020-05-04T14:31:42.008Z INFO piecestore upload started {“Piece ID”: “VAYBXNUAUUSSGOGIVZ4K2GYNIB5JMV6SEDAVFAYQA5PNHYNVADPQ”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”, “Available Space”: 3108020629376}

2020-05-04T14:31:42.421Z INFO piecestore upload canceled {“Piece ID”: “GVE7DTL7IAFUXXRBAT632WKLLCXDK2CQYQJHSEQFKXFPS4DNWBXA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”}

2020-05-04T14:31:42.541Z INFO piecestore upload canceled {“Piece ID”: “4CTVWZJGAEJZKSAFEEDWF6DYDB4AKWKTOZI6DDFSW45UWOLLKWYA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”}

1 Like

I have the same behaviour, only for the saltlake sattelite though

Hello everyone,

I am kind of a newbie to Storj even if I have been mining for 3 months. Recently, something weird happened to my node, and I cannot understand what is this problem. See the picture below.

I have a 4TB node running for 3 months, never turn off the node unless the docker needs to be updated, and it is working perfectly for 3 months.

However, on the day before yesterday, I found that the Disk Space Used dropped dramatically from 98 TB*h to 2 TB*h, but the Disk is still occupied (both showed in the dashboard and my computer), and the Ingress Bandwidth is quite low. Therefore, I think the data is not replaced by new data, and the node is still maintaining the old data, which should be counted for the Disk Space Used (in TB*h).

Initially, I suppose the information on the dashboard is not updated, so I wait for another day. As you can see in the picture, the Disk Space Used is 1.73 Tb*h yesterday, even lower than the day before.

How can I check what is the problem? Is anyone have the same problem here?

Thank you in advance for any suggestions.