I noticed a spike in ingress traffic today. Test or costumers activities?
You can see on daily graph: ** Traffic Analysis for Th3Vans storj.dk Server030 ** 1.2gbit spike
Hopefully it was a customer and not a test.
There have been recurring upload spikes around midnight UTC for quite awhile now (which we can see in Th3Vans graphs). The timing of this one aligns… just a bit higher traffic.
Looks like a customer backup job.
No tests in the foreseeable future.
BTW, tests are performed on a Saltlake Satellite, if you are not aware.
2 Likes
I noticed recurring upload spike but today was double the “normal” ingress speed
Most probably customer activity derived from the listing bug:
So as Alexey metioned in that thread, this bug probably caused some jobs like rclone sync would have uploaded more data than usual, in turn causing (most if not all of) the spike we saw.
5 Likes
So all the data we saw in input today will be deleted in 7 days because it was uploaded incorrectly twice? or did I misunderstand?
Even more interesting if it will be billed again to customer. I don’t know what happened to the duplicate upload if it has replaced the existing data or stored additionally to what was already there.
Second consideration: If a customer had setup a mirror sync with Storj and suddenly the Storj folder appears empty, did their files on the destination get deleted? So they might have to download them again from Storj, meaning additional egress cost to them?
1 Like
If it was the only traffic - then perhaps, yes. But I do not believe that it was the only traffic.
The problem was exist since December 19, see Storj DCS Status - Intermittent listing failures
no. It’s a visual issue, nothing is affected except listing. So, the billing works as usual, payouts - too.
What I am talking about is that even if it was only a visual issue, it could be that customer uploaded data gain or downloaded data again. And the question is, if they would would be billed for that. Imagine you mirror sync deletes all your local stuff due to this bug and later you have to download everything again from Storj. Would you like to get billed for that?
they would be billed for any usage, as usual. Nothing has changed. If they have a backup job, it would try to upload the same data - ingress is not billed, the storage is. But it was before, so nothing has changed.
If they have a download something (like a static web hosting), the paths are worked as usual. I have had only 2 tickets for EU1, 1 for US1 and 1 for AP1. And they all resolved. Yes, they were not happy, that the listing stopped to work and we are sorry for that, this is why it was escalated and fixed.
However, we always resolves any issues on the case by case basis. If they have had something like you described, they can submit a support ticket.
But if they would upload the data twice the old gets replaced. So they don’t use more storage. And ingress isn’t billed. To it would change nothing for them, if I understood it right.
1 Like
That is correct, faulty uploads due to the listing bug would be overwritting the old files, and unless the customer had versioning active, then the old files would have been deleted and will be garbage collected from the nodes.
This bug, while frustrating for customers at least, should not incurr in any additional billing as the new uploaded data would overwrite old one and ingress is free (unless versioning is on, which would incurr costs due to the additional versions that would now be stored).
It depends on how they have their sync configured, but this is a possibility and could cause additional egress.
Cost wise there wont be much impact for most of the customer, while some might indeed face some additional fees. The main issue is not that much about additional fees but essentially a disruption to the service overall, which does inturn impact all customers.
1 Like