Sudden drop of bandwidth and disk usage

Node Operators troubleshooting
1

So the last two days i had a massive decrease in bandwidth usage (arround 50%) without changing anything to my node. I saved the logs of the last day in a t1 file (13-14/1) and this is what i saw:

 cat t1 |wc -l
128347
 cat t1|grep ERROR|wc -l
57849  #45% of logs are errors
cat t1|grep ERROR|grep "too many"|wc -l
57434 #almost all of them are caused from the limit of concurrent requests

And now i am in the same situation that i was in my previous post , with the exception that now my disk can’t handle any further increase in the concurrent requests. Having in mind that for the last 10 days my node was working fine: This proves that now requests are really bad distributed over time which means my node is accepting lot of requests in the same time and then no requests for some time…(right?). Also my disk usage has decreased from 0.8 to 0.78TB which is something that took a LONG time to happen (about 2 weeks) and now it was erased in matter of 2 days. My question is:

1)Why do i even accept so many concurrent requests, while i have set a limit
2)What is the real reason bandwidth dropped (if not the distribution of errors?)
3)What else i can do?

2

I think there might be some sort of reconciliation happening. The same happened 6 months ago, trash went up a lot. (today stats)

image

If you look at the stored data, it is currently on a gradual decline

image

Ignoring the spike, In 3 days we have moved from a steadily growing 9,482TB to 9,469TB

6 months ago, over 6 days we saw a gradual decline from 9,197TB to 9,057TB then a sharp drop to 8,990TB, so total deletions of about 200TB

image
image1169×175 24 KB

I was hoping to go over the 3TB this month!!

1 Like
3

And I was hoping we cross 10 PB at last.

1 Like
4

I was hoping to go over 20TB this month :slight_smile:

5

20TB is my realistic 7 year goal :wink:

In 6 months we gained 0.5PB, so at current rate, we should be at 10PB by July 2022

6

Thanks a lot for the info but how is this connected to the bandwidth usage drop?

7

I have same usege drop, so it not only you, it hole system. Just usage go down.
weekly
I dont like it also but it just a reality.

2 Likes
8

We simply need more customers and more usage.

1 Like
9

Ohh ok thanks for that :wink:

10

same here, was afraid it was just me… since i had been tinkering as usual lol

11

Node operators who have been here since Storj got out of beta are closing in on it now. I’m just 370GB short.

3 Likes
12

Finally :confetti_ball::tada:

image
image828×239 36.7 KB

5 Likes
13

:partying_face: :partying_face: :partying_face: :partying_face: :partying_face: :partying_face: :partying_face: :partying_face: :partying_face: :partying_face: :partying_face: :partying_face: :partying_face:

14

Verify that your auto updater it is actually updating and that you are currently running the current version. This happened to me last month and I had a hell of a time figuring it.

My bandwidth has been pretty steady and my disk usage is actually up over the last two weeks.

2 Likes
15

Data is back on the rise, so I think the data purge is now over. Looks like 140TB delete over last 9 days. My Trash ballooned from a normal 45GB to 90GB and is slowly reducing after the 7 day deletion retention

image
image1255×186 22.9 KB

1 Like