I’ve been facing some problems with AP1. Uploads aren’t as fast as they used to be. On a 200mbps symmetrical line, I’m getting 40mb up if I’m lucky and as low as 7.5mb otherwise. This is through the browser.
I’ve got the bucket mounted to a linux vps using rclone (native) and on there I’ve been getting input/output errors while reading & writing text files.
My media server on the vps was struggling to play simple SD videos. I reached out to the developer of the software and after seeing my logs he said the storage is too slow and so the media server is not able read through the video file and generate a header file for playback.
Any idea if this is an temporary isolated case or a common issue for the region?
Another quick question: Which city is the AP1 satellite server located in?
I’m not aware of any performance issues, you have to remember that you are transferring pieces from individual nodes here and not from the satellite itself. Of course there can be a little overhead from the satellite, but I don’t think that would necessarily cause the issue (Off the top of my head anyway) Have you tried doing a tracert from your machine to a destination to see if you are seeing latency on the route? Perhaps your router or one at the ISP level is having issues.
In the meantime, I’ll let the team know about your post and see if they have any thoughts and then let you know.
I don’t think its a network issue, because I’ve been able to do everything except play videos out of the bucket. If you prefer, I can run the test and post results, but from a ux perspective, I’m pretty sure its the storage not responding because I also had a file upload (browser) get stuck at 100% for 2-3 hours before I abandoned it.
Another funny thing that happened was I noticed my egress was full, but I really doubt I’ve used up all 150gb of it so soon. I added my bank card to the account and the egress charges have been going up at 1 cent a minute or every 2mins and I’m yet to successfully play a single video file. Even the media server gives me an media offline error. I deleted the stream configs, so now there is 0 chance of any background use. Yet my egress counter is still going now and now I’m beginning to regret adding my bank card to my account.
This hasn’t been a great day. Today was supposed to be the first test screening on our streaming service, after months of development and testing and just so happened that everything went wrong.
I’ll give this a shot again tomorrow, if the egress counter doesn’t surprise me with more charges for services I didn’t use. Sorry about the rant. I’ve been trying to solve this since morning.
Edit: By the time I finished writing this, the egress counter went up by a few more cents, even though there is nothing using storj right now. I even unmounted the rclone. Now I’ve fully shut down the server and hope that it doesn’t keep counting up.
Edit2: It did it. Egress went up, even though my server has been switched off. Somethings not right here. Really not happy about this.
These files were only accessible to the media server through a mounted folder using rclone. No shareable links were ever generated. The only person who has access to these buckets and server is me.
I can create a billing support ticket but I don’t have much hope. I’m extremely disappointed. This has really shaken my trust for Storj because if the egress counter miscalculates data when not a single file was successfully read today, then on busy high traffic days those numbers would be relying on my blind faith. The egress is still going up by the way.
I think your account ussage is runing even when your already not using, because nodes where you downloading files not submitting orders to sattelite instantly, but in some session time with batch of orders(Storj Team i hope I am not wrong, writing as i see on my nodes) as i see it happen once in hour. So sattelite cant also charge instantly. Egres orders can be submitted max in 2h. After that sattelite not accepting it any more.
Normally streaming works great, in fact we are delivering >100k streams per day. While we work with you to sort the issues presented I’m happy to apply a credit to your account to assure you are not charged for unnecessary usage. Please send us the email the account is registered to as well as the project.
After which please provide a link to a video and or a way for us to test what you have hosted on Storj. Happy to host a screen share or whatever works best for you. Please note that the file browser (web-based) is not the ideal tool for large bulk uploads and is more useful for reviewing quickly what is in buckets. We are happy to guide you through everything.
when you played video files, Did you used Storj DSC gateway mt or directly uplinc?
Be aware then if you download directly from nodes this mean that your CPU will decrypt all this in your player. So second question what was CPU load and RAM usage on the player at this time?
if itwas VPS how many cores did it has? and what usage?
Thanks Dominic. I’ll submit a ticket. Appreciate your kind gesture to offer credits but the amount is not a big deal, its the principle. Today its only a few cents, tomorrow when I have active users throughout the day, I’ll have no way to verify if the data is accurate or not.
Instead of a credit, I would prefer if Storj audited the egress calculator and ensures that it works as intended.
Hi Vadim. I was on the free tier and my egress counter was maxed out (I highly doubt this). In hope of getting the stream working, I added my bank card. I tried to play the video 2 times and the media server still showed the same offline error. That’s when I deleted the stream configs and unmounted the rclone folder. After this point there has been no interaction with Storj at all, but it has been steadily going up. So realistically, I should be billed for those 5 mins of activity after upgrading from the free tier, but I have no idea what I’m currently being billed for.
I used rclone with native integration. CPU load has been between 0.2-3% on average. 20% was peak. Ram was maxed out. 5gb. VPS has 2.5 cores, 5 threads.
Billing is accurate but the web-based egress counter settles after 48h due to the differences between settled and allotted bandwidth thus it’s delayed and can max early to prevent cheaters. Long story short when you load up a video we record allotted bandwidth totaling the full video size. If you only play half the video you will only be charged for half the video data after the nodes settle up.
We are doing this at scale today and it working great. We are committed to making sure everything is proper. Excited to get that ticket and dig in.
I wanted to leave a quick update regarding this situation. I’ve been in touch with Dominick via email and also a screenshare/video call earlier this evening. He’s been very patient and has been going above and beyond to learn about the issue and potential workarounds that may solve it.So while we don’t have a solution yet, we’re trying a few options and I’ll be in touch with Dominick during this while.
Thank you Knowledge and Vadim for your help so far and special thanks to Dominick for dedicating his time and attention towards helping me solve this.
That’s very encouraging to hear. Thanks for sharing that.
This may have already been explained to you by now, but the reason there is a delay in the accurate reporting is because the satellite only gives you the information to retrieve the file, but doesn’t immediately know how much is being transferred. The transfers happen directly between you and the storagenodes and are based on incremental bandwidth orders. I won’t bore you with the exact details, but it comes down to your uplink only requesting small incremental parts with signed orders. The storagenodes then submit those orders to the satellite. They have 48 hours to do so, after that the order expires. Your charges are based on those actual orders, not the full file size. So if a node fails to submit the orders for transfers that actually happened, you won’t be charged either.
I don’t know what’s causing the issues you are seeing, but I’m glad it’s being looked into. I hope this extra info helps explain the delay and difference in numbers.
