Insanely High Failed Uploads

[branv]

Until today and yesterday, my upload and download rates have been reasonable. Now, it averages at about a 10% success rate for uploads and 30% for downloads. Is this normal? Has something changed in the network?

Here’s the output of the success script for uploads.
======== UPLOAD ==========
Successful: 41
Rejected: 0
Failed: 265
Acceptance Rate: 100.0%
Success Rate: 13.39%

[Dylan]

At the time, you will notice a lot of audit errors, especially from users running scripts to watch the audit error rate.

The satellite was updated, but the storage nodes not. In order to speed up real downloads we are closing connections faster now. This also affects audits because they are small downloads.

On the storage node side you will see context canceled. Ignore these errors for the moment. On the satellite side they are successful audits and if you want you can query the dashboard API to verify that.

With the storage node update that error should go away.

[kuprov]

I’ve had everything updated for a while:
========== AUDIT =============
Successful: 241
Recoverable failed: 0
Unrecoverable failed: 0
Success Rate Min: 100.000%
Success Rate Max: 100.000%
========== DOWNLOAD ==========
Successful: 319
Failed: 45
Success Rate: 87.637%
========== UPLOAD ============
Successful: 751
Rejected: 0
Failed: 2109
Acceptance Rate: 100.000%
Success Rate: 26.259%
========== REPAIR DOWNLOAD ===
Successful: 0
Failed: 0
Success Rate: 0.000%
========== REPAIR UPLOAD =====
Successful: 0
Failed: 0
Success Rate: 0.000%

There’re a whole a lot of ‘Canceled Context’ errors.

[taak]

where to get this script and how to run this?

[Dylan]

Located here: Script for Audits stat by satellites

[BrightSilence]

That’s a different one, though also very useful. The output @kuprov posted is from this script.

Script: Calculate Success Rates for Audit, Download, Upload, Repair

Windows/Powershell version of the script: Linux/Bash version of the script: Grafana dashboard monitor of the script: [TIG stack] Daily Email Version: Historic Post: I found the current monitoring scripts very difficult to determine trends. And I was feeling like learning some better .bash. I created this script to better measure the ratios of successful transfers over time. I plan to have this run periodically, and send the results to a influxDB for Grafana monitoring. Please share…

[taak]

i got both the scripts and here is the output… the upload success rate is very low approx 26% … how to finetune this ??

========== AUDIT =============
Successful: 665
Recoverable failed: 2
Unrecoverable failed: 0
Success Rate Min: 99.700%
Success Rate Max: 100.000%
========== DOWNLOAD ==========
Successful: 610
Failed: 68
Success Rate: 89.971%
========== UPLOAD ============
Successful: 282
Rejected: 0
Failed: 794
Acceptance Rate: 100.000%
Success Rate: 26.208%
========== REPAIR DOWNLOAD ===
Successful: 0
Failed: 0
Success Rate: 0.000%
========== REPAIR UPLOAD =====
Successful: 0
Failed: 0
Success Rate: 0.000%

[Karama]

Never mind about upload.

Here is mine…

========== AUDIT =============
Successful: 119
Recoverable failed: 0
Unrecoverable failed: 0
Success Rate Min: 100.000%
Success Rate Max: 100.000%
========== DOWNLOAD ==========
Successful: 12
Failed: 0
Success Rate: 100.000%
========== UPLOAD ============
Successful: 5
Rejected: 0
Failed: 423
Acceptance Rate: 100.000%
Success Rate: 1.168%
========== REPAIR DOWNLOAD ===
Successful: 0
Failed: 0
Success Rate: 0.000%
========== REPAIR UPLOAD =====
Successful: 0
Failed: 0
Success Rate: 0.000%

[Alexey]

Don’t use any kind of network connected drives for storage, especially via NFS.
The best way is to directly connect the drive to a device with Storagenode. Please avoid using USB where possible.

[taak]

the current data path is internal hdd and directly mounted in ubuntu…

[Alexey]

Then nothing you can do really. Only move closer to the everyone customer…