Consistent "upload canceled"

Node Operators
, ,
1

Hi everyone. I am currently getting consistent “upload canceled” when the piece size are “65536” and “163840”, the rest are inconsistent. here are some of my logs that i filtered with “upload canceled”.

2021-10-30T14:25:58.742Z	INFO	piecestore	upload canceled	{"Piece ID": "XV3MLA5LWODVCQJYRFXDZJKIBXZLMB22DQ2E6NSFGQ7VXYNIQQ3A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 163840}
2021-10-30T14:35:39.737Z	INFO	piecestore	upload canceled	{"Piece ID": "WTNPZCAYVKQHCNA4DCSC4ZLESF3HMUMAYL635W2VRZA6CDIMTAFQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 65536}
2021-10-30T14:39:02.985Z	INFO	piecestore	upload canceled	{"Piece ID": "QD476LBVPK4IKIAGCYLKRPTPZD53BTOBGNR2U6FRNWOQ2QH4VAUA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Size": 311296}
2021-10-30T14:46:29.637Z	INFO	piecestore	upload canceled	{"Piece ID": "5K74UAZB2HKU5LVOG6J42PPMA3GRF6VGAMPYVGOCAQGGNE2WZMLA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 163840}
2021-10-30T14:47:05.995Z	INFO	piecestore	upload canceled	{"Piece ID": "H5ASJ4CBB6FLQAANZ3BOLMGRGUVF5PIWQ4KOGWSVOAFMHM2GE6PA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 163840}
2021-10-30T14:47:27.740Z	INFO	piecestore	upload canceled	{"Piece ID": "N34WCPVNNN57L6DTVY6F7DPR32XPFXUY2SXU24D57EFYT3ZURIDA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 163840}
2021-10-30T14:48:26.963Z	INFO	piecestore	upload canceled	{"Piece ID": "7IZNCPSEZ237XFBN62BOXBXISS35QDS4ZPDTHKLWFFNZPINLF3PQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 163840}
2021-10-30T14:55:18.361Z	INFO	piecestore	upload canceled	{"Piece ID": "YWJSQNQRQMQ3ILNNUDNQGY6G7EKDOUF5WXJVNVYBSUVDDG5KJBYQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 1581056}
2021-10-30T15:12:43.156Z	INFO	piecestore	upload canceled	{"Piece ID": "XZ56YNDU53C3TKWIOHE25EG5B5JNMKJT7F3PXCGYHBEAF5IMREYA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 65536}
2021-10-30T15:16:35.014Z	INFO	piecestore	upload canceled	{"Piece ID": "SVULQ6G45IZSKX3H355JSWRGRJEH2M3CGSAXFYLHOSQ2ZD776YWA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 65536}
2021-10-30T15:21:00.149Z	INFO	piecestore	upload canceled	{"Piece ID": "SYF3CTPGLYEHAJF5UR5A335KAGFU4YMLYJSEUQFKXH6RKT72CB6Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 65536}
2021-10-30T15:22:07.703Z	INFO	piecestore	upload canceled	{"Piece ID": "RTOMYXFPMDUVK7RSRSHC6CEFWOD6LWGHCDMIMHRA4A2V5XYXQJYA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Size": 65536}
2021-10-30T15:35:17.310Z	INFO	piecestore	upload canceled	{"Piece ID": "SOLWDFJAOOJIB56SDK5TJZUMO4GOWMVY65C6R7BY2PLMJKC3JXAA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 2105344}
2021-10-30T15:38:28.097Z	INFO	piecestore	upload canceled	{"Piece ID": "W4GC4X2KH3NZQNANY2TRJLLCCAHR5DSAHEIONLR2BH4RODDJCSXA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 65536}
2021-10-30T15:40:32.515Z	INFO	piecestore	upload canceled	{"Piece ID": "URCIGJANBY5AKWE7R3X5GLX2G2TTEXI56ORZINEYD7S6GPUFC3VQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 65536}
2021-10-30T15:42:45.319Z	INFO	piecestore	upload canceled	{"Piece ID": "7A6LQ4GMPKHW4R3V6TUYJ7UQ7ESAJKOIP5CP63R3FUE667NZ54XQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 65536}
2021-10-30T16:13:49.287Z	INFO	piecestore	upload canceled	{"Piece ID": "Z73Q3KDE4YFPTN25FGJEGEHSXX5RMANY25WNZZ7Q5SD5QJ5EGYNA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Size": 163840}
2021-10-30T16:22:46.906Z	INFO	piecestore	upload canceled	{"Piece ID": "KJRBW7ZVYZDUGNUJY7YXQJDRABY2T5AA2IRVB7PF36CVQVLCDASQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 65536}
2021-10-30T16:23:00.647Z	INFO	piecestore	upload canceled	{"Piece ID": "ZNXWSUMT4ZEZWWJIZ57FQ4VRRL5F2O4A5F6WE4UMSS52L6UOO6KA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 65536}
2021-10-30T16:41:36.428Z	INFO	piecestore	upload canceled	{"Piece ID": "KJ4Z5AYNXTTOTQE6O4UFID5VMZKHKNPGMJQXT2JGRWZEF5MPFE4A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 65536}
2021-10-30T16:41:44.563Z	INFO	piecestore	upload canceled	{"Piece ID": "KCQL57QSQMSAE5KU452ZVHNSODPIPNJCJM7UYHYED7BGJG4N5QDQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 65536}
2021-10-30T16:49:10.027Z	INFO	piecestore	upload canceled	{"Piece ID": "H5IECJZ5RJTAJQTVIYFLHU27GEQ4MUVI766AS6FY45FHS6RPXXTQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 65536}
2021-10-30T16:52:02.228Z	INFO	piecestore	upload canceled	{"Piece ID": "TGCEQ6CECZ2SFNHNNXZVWVANWTAYGLMD5C3IAXAMQ2X7GDCESTJQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 65536}
2021-10-30T17:00:44.209Z	INFO	piecestore	upload canceled	{"Piece ID": "QDKL4QANQVCI4IS3QWTJDE2RLZPTEJPTKMH7FTW6SBPFF6GHFZDA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 65536}
2021-10-30T17:13:05.874Z	INFO	piecestore	upload canceled	{"Piece ID": "6YGHZTRZVRWVS32CODF3523GQDI5YA3QPZBTMECTADZLX5JNFQQQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 65536}
2021-10-30T17:14:54.325Z	INFO	piecestore	upload canceled	{"Piece ID": "JHS3VQAFALJR7RVK2OOTCJ6BGE4MOI7YKVZRVJA2VJMRKOG2PL6Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 1581056}

I’m not sure if i should be worried or not. please advise. thanks in advance.

2

looking into the logs again, i found out that since day one of my node, there are more piece sizes that are constantly giving me “upload canceled”.

here are the other piece sizes that causing constant “upload canceled”: 794624, 2105344, 1310720, 1318912, 311296 …

size: 65536 screenshot

image
image1610×900 276 KB

size: 163840 screenshot

image
image1617×819 260 KB

you can see in my screenshots for the specific size and it only returns “upload canceled” and there are no successful upload.

i already tried restarting the node, the raspberry pi, and the router, but still im having the same issue. i dont know what else to do. please advise

3

checked my log file for today and tho it’s not all canceled then they are on a very high ratio…
maybe there are other nodes out there which are so fast they basically win these uploads every time… i duno…

long story short i’m also seeing something similar.
my successrates are usually 99%
but for this 65536 size file its like 50% for today.

usually bad successrates means high latency, but i doubt that is it… my latency should be pretty good.

hmmm my successrates for today is pretty bad… only 91% download…
only seems to be for today, the rest of the week it was like 97%

seems like this is something new maybe…
tho i am finding the same pattern of mainly only started and canceled when dealing with 65536byte pieces

@clearing you don’t need to worry about anything, this seems to be a network issue or something…
good catch.

@Alexey check this out, are you seeing the same thing as i am?
i’m getting almost exactly the same results as @clearing

i tracked it back over a month to 1½ in my logs, seems about the same, does seem to maybe get worse later… my logs doesn’t go any further so i cannot say how long it’s been acting like this.

1 Like
5

i know there is a lot of uploads that are around 1k… not sure if its those you are seeing, seems odd that it would say 0 since it seems to be in bytes.

ofc 1k is only 25% of a 4k sector so in avg it would count as zero, even tho it takes 4k, these files seems to be transient and grows into larger files / pieces i assume.

7

Any customers’ behavior is normal. They can choose the segment size as they want.

8 
 grep -n "65536" 2021-10-31* > 65536.log
  ./successrate.sh 65536.log
========== UPLOAD =============
Rejected:              0
Acceptance Rate:       100.000%
---------- accepted -----------
Failed:                16
Fail Rate:             0.825%
Canceled:              1510
Cancel Rate:           77.835%
Successful:            414
Success Rate:          21.340%

then if i check for 163840 byte files
i get a much more reasonable but still kinda low… 92% successrate
not sure whats up with my system today.
the 65536 byte pieces successrates doesn’t seem normal, seems weird…

grep -n "163840" 2021-10-31* > 163840.log
./successrate.sh 163840.log

========== UPLOAD =============
Rejected:              0
Acceptance Rate:       100.000%
---------- accepted -----------
Failed:                38
Fail Rate:             2.018%
Canceled:              1735
Cancel Rate:           92.140%
Successful:            110
Success Rate:          5.842%

hmmm wait a minute thats even worse…
oh right those was the other ones clearance was showing as having bad successrates lol
so if i do the next size
i get a much more reasonable 99.999% successrate as i would expect to see… :slight_smile:

grep -n "181504" 2021-10-31* > 181504.log
./successrate.sh 181504.log

========== UPLOAD =============
Rejected:              0
Acceptance Rate:       100.000%
---------- accepted -----------
Failed:                3
Fail Rate:             0.001%
Canceled:              0
Cancel Rate:           0.000%
Successful:            252005
Success Rate:          99.999%
9

You can check the speed of upload to the lvm simple disk (maybe even lvm raid5/6) and to your zfs raidz/raidz2 disk. Perhaps you would see why is it happen :slight_smile:

I recently tested them and was surprised with results, especially for one disks setups. For RAID-like setups it was even more significant. So, I would like to see your results for confirmation.

10

upload isn’t a problem

./successrate.sh 2021-10-31-sn0001*
========== AUDIT ==============
Critically failed:     0
Critical Fail Rate:    0.000%
Recoverable failed:    0
Recoverable Fail Rate: 0.000%
Successful:            9339
Success Rate:          100.000%
========== DOWNLOAD ===========
Failed:                61
Fail Rate:             0.117%
Canceled:              3367
Cancel Rate:           6.475%
Successful:            48573
Success Rate:          93.408%
========== UPLOAD =============
Rejected:              0
Acceptance Rate:       100.000%
---------- accepted -----------
Failed:                40
Fail Rate:             0.026%
Canceled:              398
Cancel Rate:           0.260%
Successful:            152635
Success Rate:          99.714%
========== REPAIR DOWNLOAD ====
Failed:                0
Fail Rate:             0.000%
Canceled:              0
Cancel Rate:           0.000%
Successful:            47338
Success Rate:          100.000%
========== REPAIR UPLOAD ======
Failed:                0
Fail Rate:             0.000%
Canceled:              0
Cancel Rate:           0.000%
Successful:            20834
Success Rate:          100.000%
========== DELETE =============
Failed:                0
Fail Rate:             0.000%
Successful:            2837
Success Rate:          100.000%

as you can see all the canceled downloads are only within the range of those two file sizes.
like i show in the previous post.

you can’t say its a latency problem, for just those two file sizes and not anything else out of all the many different file sizes used.

and for the record, my slowest read speeds seem to be 33ms
over a 5 minute time period going to do a full hour, i should have a daily log somewhere

bitpool      total_wait     disk_wait    syncq_wait    asyncq_wait
latency      read  write   read  write   read  write   read  write  scrub   trim
----------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
1ns             0      0      0      0      0      0      0      0      0      0
3ns             0      0      0      0      0      0      0      0      0      0
7ns             0      0      0      0      0      0      0      0      0      0
15ns            0      0      0      0      0      0      0      0      0      0
31ns            0      0      0      0      0      0      0      0      0      0
63ns            0      0      0      0      0      0      0      0      0      0
127ns           0      0      0      0      0      0      0      0      0      0
255ns           0      0      0      0      0      0      0      0      0      0
511ns           0      0      0      0      0      0      0      0      0      0
1us             0      0      0      0      0      7      0      0      0      0
2us             0      0      0      0      9    106      8     30      0      0
4us             0      0      0      0     37     40      5    105      0      3
8us             0      0      0      0      2     10      0     20      0      0
16us            0      0      0      0      0      1      0     13      0      0
32us            0      1      0    241      0      0      0     22      0      0
65us            0     24      0  1.20K      0      0      0     47      0      0
131us           3    104      4   1002      0      0      0     90      0      4
262us          32    200     32    299      0      0      0    157      0      7
524us          13    319     12    356      0      0      0    244      0      2
1ms             2    459      2    195      0      0      0    336      0      1
2ms             1    488      1     28      0      0      0    439      0      2
4ms             0    518      0      9      0      0      0    492      0     16
8ms             1    453      1      6      0      0      0    438      0      1
16ms            5    365      5     14      0      0      0    350      0      0
33ms            3    262      3     14      0      0      0    247      0      0
67ms            0    165      0      2      0      0      0    161      0      0
134ms           0     32      0      0      0      0      0     30      0      0
268ms           0      3      0      0      0      0      0      2      0      0
536ms           0      2      0      0      0      0      0      1      0      0
1s              0      1      0      0      0      0      0      1      0      0
2s              0      1      0      0      0      0      0      0      0      0
4s              0      0      0      0      0      0      0      0      0      0
8s              0      0      0      0      0      0      0      0      0      0
17s             0      0      0      0      0      0      0      0      0      0
34s             0      0      0      0      0      0      0      0      0      0
68s             0      0      0      0      0      0      0      0      0      0
137s            0      0      0      0      0      0      0      0      0      0
--------------------------------------------------------------------------------

i think these are the avg times for the different operations.


              capacity     operations     bandwidth    total_wait     disk_wait    syncq_wait    asyncq_wait  scrub   trim
pool        alloc   free   read  write   read  write   read  write   read  write   read  write   read  write   wait   wait
----------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
bitpool     80.2T  18.3T    413  5.63K  3.32M  83.3M   10ms   11ms    8ms  486us    1ms    7us    2ms   11ms   84ms    1ms

please tell me again how my 10ms avg is to slow :smiley:
when i get like 99% + for all other filesizes than 64KiB and whats the other 160KiB
65536 and 163840
it doesn’t make sense, has to be something weird somewhere.

11

but lets check why don’t we…

grep -n "4096" 2021-10-31* > 4096.log
./successrate.sh 4096.log

========== AUDIT ==============
Critically failed:     0
Critical Fail Rate:    0.000%
Recoverable failed:    0
Recoverable Fail Rate: 0.000%
Successful:            0
Success Rate:          0.000%
========== DOWNLOAD ===========
Failed:                0
Fail Rate:             0.000%
Canceled:              0
Cancel Rate:           0.000%
Successful:            0
Success Rate:          0.000%
========== UPLOAD =============
Rejected:              0
Acceptance Rate:       100.000%
---------- accepted -----------
Failed:                0
Fail Rate:             0.000%
Canceled:              0
Cancel Rate:           0.000%
Successful:            7112
Success Rate:          100.000%
========== REPAIR DOWNLOAD ====
Failed:                0
Fail Rate:             0.000%
Canceled:              0
Cancel Rate:           0.000%
Successful:            0
Success Rate:          0.000%
========== REPAIR UPLOAD ======
Failed:                0
Fail Rate:             0.000%
Canceled:              0
Cancel Rate:           0.000%
Successful:            1350
Success Rate:          100.000%
========== DELETE =============
Failed:                0
Fail Rate:             0.000%
Successful:            10
Success Rate:          100.000%

and for 1KiB

grep -n "1024" 2021-10-31* > 1024.log
./successrate.sh 1024.log
========== AUDIT ==============
Critically failed:     0
Critical Fail Rate:    0.000%
Recoverable failed:    0
Recoverable Fail Rate: 0.000%
Successful:            0
Success Rate:          0.000%
========== DOWNLOAD ===========
Failed:                0
Fail Rate:             0.000%
Canceled:              0
Cancel Rate:           0.000%
Successful:            0
Success Rate:          0.000%
========== UPLOAD =============
Rejected:              0
Acceptance Rate:       100.000%
---------- accepted -----------
Failed:                0
Fail Rate:             0.000%
Canceled:              0
Cancel Rate:           0.000%
Successful:            24911
Success Rate:          100.000%
========== REPAIR DOWNLOAD ====
Failed:                0
Fail Rate:             0.000%
Canceled:              0
Cancel Rate:           0.000%
Successful:            0
Success Rate:          0.000%
========== REPAIR UPLOAD ======
Failed:                0
Fail Rate:             0.000%
Canceled:              0
Cancel Rate:           0.000%
Successful:            5056
Success Rate:          100.000%
========== DELETE =============
Failed:                0
Fail Rate:             0.000%
Successful:            0
Success Rate:          0.000%

and so lets do 32KiB

 grep -n "32768" 2021-10-31* > 32768.log
./successrate.sh 32768.log
========== AUDIT ==============
Critically failed:     0
Critical Fail Rate:    0.000%
Recoverable failed:    0
Recoverable Fail Rate: 0.000%
Successful:            0
Success Rate:          0.000%
========== DOWNLOAD ===========
Failed:                0
Fail Rate:             0.000%
Canceled:              0
Cancel Rate:           0.000%
Successful:            0
Success Rate:          0.000%
========== UPLOAD =============
Rejected:              0
Acceptance Rate:       100.000%
---------- accepted -----------
Failed:                0
Fail Rate:             0.000%
Canceled:              0
Cancel Rate:           0.000%
Successful:            6369
Success Rate:          100.000%
========== REPAIR DOWNLOAD ====
Failed:                0
Fail Rate:             0.000%
Canceled:              0
Cancel Rate:           0.000%
Successful:            0
Success Rate:          0.000%
========== REPAIR UPLOAD ======
Failed:                0
Fail Rate:             0.000%
Canceled:              0
Cancel Rate:           0.000%
Successful:            45
Success Rate:          100.000%
========== DELETE =============
Failed:                0
Fail Rate:             0.000%
Successful:            0
Success Rate:          0.000%

lets so something larger then.
Hmmm here we apparently got another size that misbehaves for some reason.
2 MiB

grep -n "2097152" 2021-10-31* > 2097152.log
./successrate.sh 2097152.log
========== AUDIT ==============
Critically failed:     0
Critical Fail Rate:    0.000%
Recoverable failed:    0
Recoverable Fail Rate: 0.000%
Successful:            0
Success Rate:          0.000%
========== DOWNLOAD ===========
Failed:                0
Fail Rate:             0.000%
Canceled:              0
Cancel Rate:           0.000%
Successful:            0
Success Rate:          0.000%
========== UPLOAD =============
Rejected:              0
Acceptance Rate:       100.000%
---------- accepted -----------
Failed:                0
Fail Rate:             0.000%
Canceled:              74
Cancel Rate:           88.095%
Successful:            10
Success Rate:          11.905%
========== REPAIR DOWNLOAD ====
Failed:                0
Fail Rate:             0.000%
Canceled:              0
Cancel Rate:           0.000%
Successful:            0
Success Rate:          0.000%
========== REPAIR UPLOAD ======
Failed:                0
Fail Rate:             0.000%
Canceled:              0
Cancel Rate:           0.000%
Successful:            0
Success Rate:          0.000%
========== DELETE =============
Failed:                0
Fail Rate:             0.000%
Successful:            0
Success Rate:          0.000%

1MiB
all failed… how is that even possible.

grep -n "1048576" 2021-10-31* > 1048576.log
./successrate.sh 1048576.log
========== AUDIT ==============
Critically failed:     0
Critical Fail Rate:    0.000%
Recoverable failed:    0
Recoverable Fail Rate: 0.000%
Successful:            0
Success Rate:          0.000%
========== DOWNLOAD ===========
Failed:                0
Fail Rate:             0.000%
Canceled:              0
Cancel Rate:           0.000%
Successful:            0
Success Rate:          0.000%
========== UPLOAD =============
Rejected:              0
Acceptance Rate:       100.000%
---------- accepted -----------
Failed:                0
Fail Rate:             0.000%
Canceled:              40
Cancel Rate:           100.000%
Successful:            0
Success Rate:          0.000%
========== REPAIR DOWNLOAD ====
Failed:                0
Fail Rate:             0.000%
Canceled:              0
Cancel Rate:           0.000%
Successful:            0
Success Rate:          0.000%
========== REPAIR UPLOAD ======
Failed:                0
Fail Rate:             0.000%
Canceled:              0
Cancel Rate:           0.000%
Successful:            0
Success Rate:          0.000%
========== DELETE =============
Failed:                0
Fail Rate:             0.000%
Successful:            0
Success Rate:          0.000%
12

Hi again, yesterday my node was officially 1 month old :tada: and i’ve also replaced the SMR drive yesterday. :slight_smile:.

However, upon checking my logs today i see the same “upload canceled” in those same piece sizes without any success on it. And to see these repeating “upload canceled”, seems not normal because it is only happening in those specific sizes.

should this be something that deserve a bit attention from storj team? im not the only one experiencing it, even those who have good or if not the best setup are also experiencing this issue too.

i hope someone from storj team can help to take a look on it, provide solution or explain what is happening.

13

I see something similar too. However I am having other problems on the affected nodes currently, so I cannot tell for 100% sure if it is the same cause.

14

image
That’s for 65536 byte pieces. Not on zfs in my setup, but on SHR (basically lvm’ed together mdraid 6’s).

This is what my total looks like:
image

I don’t know why this would happen, but some observations.

  • 65536 = 2^16, which is not unusual to see, but you would expect the original segment size to have these common numbers, not really the piece size. The segment size would be 29 times that. Could there be something inherently inefficient at the node end when pieces are this size? Seems unlikely.
  • Most of these look like they come from the same satellite, maybe it’s just one customers triggering this behavior
  • If all nodes are seeing most of these pieces canceled, that almost necessarily means the customer has actually canceled the whole transfer. Need more sources to confirm.

That’s all I got atm.

1 Like
15

Im not seeing the same I see barely any canceled from US1 But it looks like my log is newer
My total node
image

65536 byte pieces
image

16

thats also what i ended up concluding.
just the 163840 byte pieces / uploads represent 25% of all my canceled uploads over a month.
one thing i did also consider was if it might be related to the atlantic connection… maybe its customers in the states using these sizes and then europe nodes will fail because of the range

17

I have some zero size canceled uploads. Maybe the size on a canceled upload shows how much was uploaded (before it was canceled) rather than the real piece size? That would explain the “round” sizes, like 65536 or 163840.

1 Like
18

i was looking at filesizes because i want to do like successrates on them… but
there are like 7997 different sizes, because its basically 1024 byte + 256 byte steps until 2 Mebibytes.

but yeah 64 Kebibytes and 160 Kebibytes is pretty round numbers, hadn’t thought of that.

so basically you are saying its a math thing and somewhere along the line its rounded which then makes the canceled seem like they are all the same size…

we should be able to test that… by looking at larger and smaller sizes because the rounding will most likely show up in many more spots… like with the 2Mebibytes and 1 Mebibytes which i also saw large canceled at …

they basically all failed tho… which is odd… but maybe its a combination of more things… i would think if a transfer is larger maybe the chance of it being canceled goes up…

19

I was doing some testing on QA testnet in order to get those sizes you would either A need to split larger files to an exact amount in order to repeat those sizes.
Which its pretty easy to repeat it over and over., and cancel is when someone is uploading and cancels it mid way so someone is repeating this process over and over.

Which I was able to do on the testnet QA on my own test node to see what it was actually doing.

2 Likes
20

I am wondering if these “cancels” can affect the node reputation, specially the new nodes which are just building reputations.
because if this is a valid customers’ behavior, then those “cancels” will be considered in determining the nodes’ reputation. which is a bit off if that is coming from the customers’ side.

i also observed but not really sure if its related to the “cancels” or this was just in my node, every time im getting a momentum of successful uploads and downloads, suddenly a piece or two of “upload canceled” come up then the momentum slows down.

21

Reputation is determined by the satellites alone. Customers cancelling a transfer doesn’t impact reputation. Only audit and repair traffic impacts reputation.

3 Likes
22

i have an idea or question. is it possible that the customer are able to see the IP addresses of the nodes? ive read an article somewhere that when downloading, the client/customer will get the pieces directly from the nodes, so im assuming theres a way to check the ip address of those nodes?

my wild idea is, atleast a number of SNOs are participating in this and theyll have a list of their nodes’ IP addresses. A fake customer (bot) will make the upload/download (also the reason why the size is not changing) and those in the IP list will have priority in transacting those files. and the cancels are happening when no ip address was selected from their list.

would this ever be possible to happen? considering the free account plan.