I have my storj mounted as a encrypted rclone volume. When I watch a video I find that the egress/download shown as consumed is 2-3x the file size. Sometimes the consumed limit does reduce after few days but it is still well over 1.5x-2x of the file size I viewed.
Is there something I can do to fix this or is this normal?
Sometimes I rewind/forward the video, can this be causing the extra consumption?
docs say uploads are not counted towards the limit but I find sometimes it does add to the bandwidth used.
I have rclone set as “vfs-cache-mode=writes” in case that makes any difference.
What’s the point of this? It has no effect on your usecase. Instead, enable full caching.
How are you measuring this?
If you use native integration, your bandwidth consumption will be higher than if you use s3, by design (long tail cancellation: more than necessary bits are downloaded and the as soon as enough are received the rest are cancelled), but not 3x higher.
Having cache enabled for writes only may explain your excessive bandwidths usage if you scrub and rewatch parts of the video.
i’m using vfs-cache-mode=writes because I have limited space on my machine & rclone sometimes does not empty the cache after reading or it takes too much time to empty it even though I’ve set vfs-cache-max-age=0h10m0s.
I checked the storj dashboard, at the start of the month it was 0, then after playing a 4.5gb file it was showing as 13.5gb.
Ok, then the amount of data downloaded will be correct.
This is rather vague and imprecise, those couters don’t update in realtime.
I would suggest using system profiling tools on your OS to measure how much data does rclone end up downloading for that isolated usecase. If you just streamed the file once, the downloaded amount of data should be pretty close to the file size.
is there anyway to get a precise usage from storj because I am on the pro tier so I have set a download limit so that I don’t go above a particular usage?
Native integration will consume more bandwidth (as measured at your router) but you will only be charged and your account will reflect actual amount of useful data downloaded, not the overhead
It’s precise, but not realtime.
I.e. you could have downloaded 1GB from the beginning of the month, but it takes time to get reflected in the UI. So you looked, and it was still showing zero.
Then your rclone downloaded another 1GB, you look at the dashboard and it updated to show total 2.
another question - if I watch say 20% of a 5gb file stored on storj will the bandwidth consumed be calculated as 5gb or 1gb? is there any rclone config (encrypted mode) you recommend for optimization of speed & data consumption?
Rclone does not pre-fetch data, instead it downloads in chunks as you access it. So if you watch 20% of 5GB file sequentially once, the size downloaded will be equal of a total size of chunks necessary to fit the 1GB size.
You can read more about --vfs-read-chunk-size (default is 128MB) and --vfs-read-chunk-size-limit.
But this is really not storj specific.
Enabling full caching mode might help, if your player does not actually read the data sequentially.
Hence the advice to check actual network traffic usage using OS tools. Until you do that – it’s all non-actionable speculations.
This also depends on how will your video player buffer data. It is usually not a big problem, but some players thinking they deal with local storage will proactively generate thumbnails from keyframes to have nicer UI for seeking, and this will trigger downloads.