Storage size limit decreased but traffic stays the same

Node Operators troubleshooting
,
1

Screenshot 2020-05-07 at 18.13.20
Screenshot 2020-05-07 at 18.13.201248×343 42.1 KB

On Monday I changed my max allocation size as my node is ingesting 250-300GB of data each day, and the node will be full of test data within weeks. I left 500 GB of “headroom”, to allow it to grow a bit.
Change-> restart and dashboard reflected the “Space remaning” instantly.

However my ingest is the same. Nothing has changed. My free space has not changed (Since Monday), still 500 GB but I’m getting 250GB a data. I know apparently the data comes from the satellite for some weird reason so perhaps the PUTs that has failed are still being reported. Just a hunch.

2

There has been significant delete activity from saltlake that started on 5/1. As it deletes data, this creates free space that is then taken up by new uploads. Keep in mind that the default dashboard does not show delete activity but does show ingress, so it can create the illusion that incoming data is “going nowhere” when in fact it’s just filling the free space left when other data was deleted.

3 Likes
3

Yes, that is a plausible explanation. I find it a bit difficult however to see my ingress be 100% the same across the week, i.e the delete would 100% match, ingress, every second. If you delete I’m sure some of that will be in chunks.

The delete vs upload ratio is not in favour of that idea either Im afraid:

PS C:\Users\x> Get-Content "$env:ProgramFiles/Storj/Storage Node/storagenode.log" | Select-String "DELETE" | Measure-Object

Count    : 817196

PS C:\Users\x> Get-Content "$env:ProgramFiles/Storj/Storage Node/storagenode.log" | Select-String "upload" | Measure-Object

Count    : 8688571
4

That’s almost what I am seeing on one of my nodes since the deletions started. Only within the last two days have deletions started to outpace ingress activity:

image
image

Assuming that you have your logs at the default level, “upload” is going to appear twice in two situations:

  1. upload started
  2. uploaded

And:

  1. upload started
  2. upload canceled

So you are not only counting successful uploads twice, you are also including failed uploads (also twice).

Search instead for “uploaded”.

I would also suggest changing your deletion search to be “deleted” as I’m not sure if there are other situations where the word “delete” may show up in the logs.

5

quite interesting that your failed uploads has gone down, upload has not changes at all (mine has by 95%) and your delete has just recently changed.

6

Note that the ratio between successful uploads stays mostly the same. It’s not just failed uploads that went down; all upload activity went down.

7

Seeing the same thing, uploads to delete ratio from my logs is showing a 60/40 uploaded/deleted split and it was closer to 15/85 uploaded/deleted earlier this week.

8

Yes, so that would mean your net effect would be negative, you would increase your free Storj space? But that’s almost not happening. That could indicate that the data that is deleted are only smaller chunks?
In my case, to have zero net effect the deleted chucks would have to be the same size.
Remember I’m getting 250GB, your node seems to be full?

9

I’ve seen almost net 0 on my node for a few days now. Amount of stored data is going down ever so slightly. If your node is storing less you could see a little fewer deletes and be right in that sweet spot.

10

Yes, that’s what starts to happen towards the end of the graph in my post. Before that there were about three days where the deletions and successful uploads tracked almost exactly, which is why you don’t see a dip in the used space graph even though there are a lot of deletions.