1 GB of bandwidth to upload 60MB?

I uploaded a few files which total about 60MB according to the storage meter. However, the bandwidth shows close to 1 GB in allocated and settled bandwidth. Should those two numbers closely match? What would explain such a high difference?

ETA: the upload was done through rclone, if that’s relevant, using S3, and the transfer occurred several days ago, so the bandwidth use has had plenty of time to settle.

Hey @b0rrasca. You should expect to see a single line on the graph up to a certain point. You will only see the lines for settled and allocated diverge for the most recent data points on the graph - before allocated traffic expires.

That said, it is difficult for me to understand your situation given only the information you shared here. I have one clarifying question:

  • Are these 60MB of files the only files you have in any bucket for this project? Or is there other activity you are aware of on this project that you did not mention here?

Another thing that could help - if you DM me or email me (moby@storj.io) with your DCS email and satellite you are using, I can look at your usage numbers in the database and see if I can understand where these numbers on your dashboard are coming from. Without some more information, it is hard for me to determine if this is expected behavior or if it indicates a bug with the graph on the dashboard.

3 Likes

Here’s what I’m seeing. Does this make sense? (apparently, I can only upload once picture, but the blue graph shows exactly 61MB when I hover over)

Yeah, that makes sense. You should expect that allocated > settled in the most recent period in the bandwidth graph, but once allocated+unsettled traffic expires, allocated=settled.
Technically, from a database perspective (if I remember correctly), “allocated” traffic becomes “settled” traffic, so really what you are seeing in that graph is just the “settled” traffic for August 3rd. And the reason you see the same amount in the “allocated” line is mainly to prevent confusion since otherwise you would see “0” for the allocated line on August 3rd.
But perhaps it is more confusing this way.
Did that help? I am happy to answer follow up questions if you have them.

Thanks for the explanation. I think I understand the part about allocated and settled. What I don’t understand is how a 60MB upload would result in almost 1GB of bandwidth.

If you used rclone, it can download files to check their consistency, but it should not be more than uploaded data, unless there were retries.

If you would like me to take a look at our satellite database and analyze the usage associated with your account, feel free to privately message me or email me (moby@storj.io) with your account email as well as the satellite you are on. With that information I should be able to help you understand at the very least what kind of traffic this is (e.g. upload vs download), and we might be able to find some things out from there.

@moby If further discussion happens outside of this thread, will you give us a summary of what was going on when the “issue” is resolved? Might clarify things for other people in the future :slight_smile:

3 Likes

Yes, I plan on putting a summary here if I do any further investigation based on private conversations.

2 Likes

What rclone command did you use? copy, mount, sync…?

One thing worth noting is that in addition to adding files using the command line (using the sync command), I was also testing mounting the remote as a drive in windows (using --vfs-cache-mode writes). I wonder if that may have something to do with it?

Could be. Your file browser application might have downloaded the files to generate thumbnails.

The odd thing is that it would have had to download the entire set of files 15 times to account for the bandwidth use, which doesn’t really make sense, right?

Thumbnails, Windows Defender, other antivirus software if you have any, Windows Indexing Service… A typical Windows installation does this a lot. Though, I’d wonder why these all would work on a networked drive, rclone should mark it as one, so I find it somewhat unlikely that this has happened.

Not a regular Windows user though, so I don’t have direct experience.

Haha, rclone now needs a switch to mark the drive as networked. This wasn’t the case in the past. rclone mount

So it is actually likely this happened.

2 Likes