No "Disk Space Used This Month" since yesterday

Node Operators troubleshooting
, ,
1

Hello,
i am currently running 6 Nodes. 5 are on one machine and 1 on a different location. I use the Multinode Dashboard to show all together. Since yesterday the graph “Disk Space Used This Month” show very little.
My Question is if i have to worry about that. What got me wondering is, that it affects all nodes at the same time despite the physical and network seperation. (One in a Datacenter and one at home)
What could cause that?
I checkt log and everything looks normal. I Checkt the node specific Dashboards and there is the same problem (So no Multinode-Dashboard Problem)

image
image918×365 25.4 KB

edit: All Nodes are Linux docker Nodes. 3x v1.78.3 and 3x v1.76.2

Best thanks
marvi

2

Yes i see it also on lot of nodes

3

Yes, same here.

image

4

Same at my place, yesterday and today only reports from us2.storj.io; in total five nodes. So probably nothing to worry about.

6

Here as well, since yesterday. Anyone to confirm what is happening?

7

same here, suddenly

image
image630×623 22.8 KB

image
image579×598 18.6 KB

8

5 satellites have reported back 0 at_rest_total records with a wrong interval_end_time in the storage_usage table.

image
image761×252 11.1 KB

We’ve seen similar issues be resolved automatically on the next reports from the satellites, but this is kind of messing with what the nodes currently show. (Also in my earnings calculator)

9

so we just wait and it will solve itself?

10

Same issue!
I was starting to investigate to avoid a disqualification :joy:

11

Don’t know if it’s related, but today it shows 0 B repair egress
image

Yesterday it was already lower, around 100 MB.
And prior to that it was around 400 MB each day.

Disk space also borked from yesterday.

12

Same here for all 6 nodes, also ingress suddenly went to half of the average daily, egress repair went to zero.
They are probably doing stuff on the satellites, lets hope they can fix it soon.

13

It hasn’t had an impact on segment availability on customer facing satellites at least. Minimum availability across segments on all customer satellites is 51 pieces. Well above the 29 minimum required. However, test satellites look weird… though this is almost certainly a reporting issue and not actual loss of pieces. It’s possible they stopped/paused repair due to something related to this.

image
image811×1206 78 KB

I can’t promise that. All I can say is that it’s not something wrong with your node specifically. And these kinds of errors have resolved itself in the past. Only Storj Labs can confirm whether that would happen now as well.

1 Like
14

Some background jobs are experiencing performance issues, affecting satellite data reporting. The team is actively working on resolving the issue.

13 Likes
15

Hi there,

I am running two nodes in two different locations and I noticed that from yesterday the “Average Disk Space used This Month” became “0 Bytes” on both of them.

Could someone explain why is that happening and how could I possibly fix it?

Thanks a lot in advance for your prompt attention to this matter.

With kind regards,
Angelos Pitsos

17

Thanks mate! I thought is just me!

1 Like
18

until now 5 of 7 tb are back, where rest of used space? :face_with_peeking_eye:

19

At my place it has been fixed now.
If you check out per satellite, you can see that not every satellite has already reported back. Don’t worry, it probably will be ok!

1 Like
20

I tried to see if there was a better place to post this, but I settled on here.

I noticed on storjstats, the min healthy pieces for europe-north-1 and saltlake have tanked.
The stats are the same here too

I’m hoping this is a glitch and someone could enlighten me on what’s going on.

image
image1247×1209 92.5 KB

These are test satellites if my memory serves me right

21

This is already discussed in No "Disk Space Used This Month" since yesterday

1 Like