Suddenly lose nearly all races

ptC7H12 · October 13, 2023, 12:18pm

Hello,

I’m suddenly losing nearly every race… You can see here, that since 00:40 there is a huge drop in the bandwidth. All other Nodes are behaving normal. DBs are allready on SSD.

Do somebody have an idea, what it could be?

2023-10-13T12:11:31Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "472QL3ONDPO3BWGOI5QO32CVW5GEMXSH5EUMGJXVP7TMUREDKRFQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 814356045824, "Remote Address": "23.237.232.130:47260"}
2023-10-13T12:11:32Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "CT4LAZBLVFRIX6CEUV3H7JMMI5N6LTZUOYMHHD7B2O54P272TSXA"}
2023-10-13T12:11:32Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "IUBZA7NOBCXKMK6PMD4HUFJ22HBKQVLWPLJYQWBE2JGVTWK332TQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814356033280, "Remote Address": "23.237.191.122:34046"}
2023-10-13T12:11:32Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "ZPIRFT5ZFGCRPMC5JR4MZ5AAICKBL24UHC3BCIXCPGUAAY3QWGLQ"}
2023-10-13T12:11:33Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "X2WOUXLZALTBJL6YASJPBTOSAGED3H4VI3KYSJ564DTJFL4Q4W6Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814355995904, "Remote Address": "5.161.117.79:35806"}
2023-10-13T12:11:33Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "3VNDNGLZ46MU7WDYY54AQPBKTG426EURH7IFBFTRFVDDY5T4BEAQ"}
2023-10-13T12:11:34Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "UZ27LTB65LFXAGUXEJ2K5CSQCZU2UETZEUMX6TNWVH3CXHCLV5XQ"}
2023-10-13T12:11:34Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "VSO2MCE4BLEEIFWNQE6MXTP5BUXVL5SSP2XUXP6UE62ICORUPGUA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814355873536, "Remote Address": "5.161.117.79:50746"}
2023-10-13T12:11:35Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "JUM5H6YAJBBCQCIUV4CLGK7HYN7F7K3I32BT4UQHU32P6PVGM3UA"}
2023-10-13T12:11:35Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "7FZ2HMOP2PVDIVW6RXMAPNSQPHLYJEH4VGZX7OYYO6VOY7O4QTJQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814355792384, "Remote Address": "5.161.117.79:62740"}
2023-10-13T12:11:35Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "P5K64S3ZJ2WWNEEWOGITQ3F3I55BM2N3LBLXZSO53GDLXO5BWLIQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 814355792384, "Remote Address": "50.7.22.194:58682"}
2023-10-13T12:11:35Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "EDUAF6LG4USNN7I6XYEVOJ7AWBHROEE4PE7XMRXKR75GKDZDSVYA"}
2023-10-13T12:11:35Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "HQ5NHLRMPV4LWAMBCUKD2UOH6NR7UYEIF7B5M6ROSH2Q2TD6TZ5Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814355755008, "Remote Address": "5.161.143.41:45314"}
2023-10-13T12:11:36Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "O3JYUHWUHE477CUMJG3S5GZ6LBSQ2URVFPHF6WAHOSRQGXWL2LCA"}
2023-10-13T12:11:36Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "PA7DJ7VQE7VLCB2EIHFJSK5VJFBM6FNRANOZKVSX5C5YOPXEP76Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814355755008, "Remote Address": "23.237.232.146:47468"}
2023-10-13T12:11:36Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "IYHTJJIVMZYCTYQRBWCE6VODDHPIAF7EYB562C4ZWNZVB76IRH3Q"}
2023-10-13T12:11:36Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "NP7TJFYDFTHRS27EWTB6AKLUMWUSUMSJGYMFMP2BGKRH4C6VNWRQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814355718656, "Remote Address": "5.161.211.187:63788"}
2023-10-13T12:11:38Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "EYC4LPUUGBIDC3NR4HY5W63UTWECFRFL3FX5TADJ5ZGNDFMX67WA"}
2023-10-13T12:11:38Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "RILNU37OXMOARY4ARFWKENC6G2AAXMM3E4XFLAURVL5B57N7RCIQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814355681280, "Remote Address": "5.161.220.231:25714"}
^[[A2023-10-13T12:11:46Z        INFO    piecestore      upload canceled {"process": "storagenode", "Piece ID": "EQLKMS6F5JCQGISH5V2TPVLFYMGPL5XTZFC7E74F36L7VCXH7EEA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 196608, "Remote Address": "50.7.230.34:44826"}
2023-10-13T12:11:46Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "IUPPF4PKY6OC3VGIKEF34BAF66JIC5YUUZKCXNS24MQCSBQHWUUQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814355681280, "Remote Address": "23.237.232.146:33408"}
2023-10-13T12:11:46Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "2DSWDFPKLJ7HGEOTOEYLWPYZLTPUIMWKEUEHKIUSVJR7MO7GNJGQ"}
2023-10-13T12:11:46Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "DTZI64FJ6FKEG2P7GU477457KJP2GWSBZTVLHLV4XGJXACBB4Z2A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814355681280, "Remote Address": "5.161.176.200:51184"}
2023-10-13T12:11:46Z    ERROR   piecestore      upload failed   {"process": "storagenode", "Piece ID": "ZXKKVZH5YN3E4BZRRJHNXKEMBG7G3WXCEYYIHQ7WN2PE7VB5QLJA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 131072, "Remote Address": "23.237.232.130:42940"}
^[[B2023-10-13T12:11:46Z        INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "Y2DKHCZ6DJG6DDVY3PQZP2E7EPIVZ63ZJTNOMORMCBALLI6G7LVA"}
2023-10-13T12:11:47Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "5U5S6ROWDDO4NXJURLZYF4NWHQHM3T7OY5SUECF2UL7AJQ56PFNA"}
2023-10-13T12:11:47Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "COGZMZUFDPNRIVXZXJJLRFRXP4536BO2CBXOMUFGJUH2OGGTEATQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814355675904, "Remote Address": "5.161.220.231:34546"}
2023-10-13T12:11:47Z    ERROR   piecestore      upload failed   {"process": "storagenode", "Piece ID": "BCQKNRBYXXHKLQWPIPGMQWJXIXX2AE7WWEPP5SMSL56XQF6VE3WA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 786432, "Remote Address": "216.66.40.82:53650"}
2023-10-13T12:11:48Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "5MAOCAQBSXSK2QVOJG35YL7X3LBK5U6EUWIPBRI52RCKO3FXV3RA"}
2023-10-13T12:11:48Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "GXVXYKRAO5SOARB7IPTVCHPEQWLYFQ6KU5XYFLBVEO35IFGFVCMA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 814355658240, "Remote Address": "50.7.22.66:49266"}
2023-10-13T12:11:48Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "WAT7ZCKW7T7UKTXKYD6WJLIBIUIY24EWUEMFFSGO4R7LFDVE2RUA"}
2023-10-13T12:11:48Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "3TJLOOO4PPYODDY43LBFFAD5LSVYH72YMZNJV3OEKJSZCMXWJSJQ"}
2023-10-13T12:11:48Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "H7BLAWG57IZ4KYXGSRBUHDPCR53RTVHD3CUURIHC7BZKLUJ4IMNA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814355658240, "Remote Address": "23.237.232.146:33958"}
2023-10-13T12:11:48Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "Z2H5RLYBQAS2S56KTC4VQ3ORJXWIIS4AWWDW4F2TXG2JIIESF4WA"}
2023-10-13T12:11:49Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "K4S2J45LATYYMDAYWQ72BEQEXDF2HEW6EWDNCRSWN4NVQK66CUUQ"}
2023-10-13T12:11:49Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "ARTI3R7S4NRSO6ACUL35ZRS7OLGW6PDNH6L5YB5IFTT4HYJDCV2A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814355549952, "Remote Address": "50.7.22.194:59542"}
2023-10-13T12:11:50Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "BR7QKDSNGQY5AGDR7E5Q6MOITSXVZSTJ2MGR25IEDP2ESDBLJZNQ"}
2023-10-13T12:11:50Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "GCFA3BU2MFBI634LHYEDTZWIYEGPD65CI7G4HF6SVQUDHMXXIAGA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814355513344, "Remote Address": "5.161.220.231:31610"}
2023-10-13T12:11:51Z    INFO    piecestore      upload canceled {"process": "storagenode", "Piece ID": "5BXGC6KC3EMHLGOHT7HZ55OKOMQNIRYI46ARVDBRHTBIDGOEQEIQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 589824, "Remote Address": "216.66.40.82:34304"}
2023-10-13T12:11:51Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "AVF456KZFQ3JOVLGC5MVVPNWBQXN6OIGY2XZJ7SPO3UMTGE2BYCQ"}
2023-10-13T12:11:51Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "T2UITSQN6B7HNIV44TXTYNMI3KSMWCAVZOZOQEOEENDB736LQKAQ"}
2023-10-13T12:11:51Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "YBS5FFFIH2QVAJ4JSDLVQ535DD75BKBBUG6IPUIIRFUAOZDX2WEQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814355481856, "Remote Address": "5.161.220.231:25582"}
2023-10-13T12:11:52Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "IR3E7L722INHQQ6WX6RJBA4MKT6H33RFHZM7I224XQ2DNFVUW6NA"}
2023-10-13T12:11:52Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "VD2Q6TTYNTWJQ732XSBD2PZWKWKMJUEJP2YTDQC3EAAUVA32JJKA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814355469824, "Remote Address": "5.161.220.231:31650"}
2023-10-13T12:11:52Z    ERROR   piecestore      upload failed   {"process": "storagenode", "Piece ID": "4XEPQRB7PSMBTM7LU7XVGUBIVAIVF6IWBZ3SNCAE5EA6NGW6Y5XA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 917504, "Remote Address": "38.101.146.225:45146"}
2023-10-13T12:11:52Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "ZOTOZPWFOWWVGCJEOSFE2C2MSZRG3TZOI345ANNZ5BFVXHFAZILQ"}
2023-10-13T12:11:52Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "FKVL6VEFJYQD2QZNAX2RSHH7VLEF4ZKTILH75NCZ6E3KJLCRZULQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814355248896, "Remote Address": "23.237.191.122:56844"}
2023-10-13T12:11:53Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "PIB2FYD7KTLL35MRGVX2VYXSHJYMSBYMMVMEXOEDIB53MHXDG6YQ"}
2023-10-13T12:11:53Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "QV3IZUKX6ZOCDRDQHNVD2ZAYRS3YFLSXFGPHSNRQUNYWOJ4IOYSQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814355227648, "Remote Address": "5.161.143.41:39452"}
2023-10-13T12:11:53Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "E2EXI2W2SUYSE6PFMTFGPZCY4PBQWTLE7ATLPGEC4GKDDQVQQMBQ"}
2023-10-13T12:11:54Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "M2G5RXCUCDH7YKGSG2ZLBD6WDLM3AGDUYR4UOY6LXCZ2EVCHNLQA"}
2023-10-13T12:11:54Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "VISTBF7GDWHUFLJU5KTA33ZMQYBJYXNCSARKXEO2NRT4M4ZSOOLQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 814354970880, "Remote Address": "143.210.249.6:48836"}
2023-10-13T12:11:54Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "PX4WSWQLKM4TBHJCOMZLTPI4RPB2VSOXL7UND2JUJWG3PQ2JZJHQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814354970880, "Remote Address": "23.237.232.130:39270"}
2023-10-13T12:11:54Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "IOVG56CLHN2TPTS55Z54Z5T7WHS5G5JLBHKGK2HYFCXRQOHCYXHA"}
2023-10-13T12:11:54Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "M6EC2723KHOKZF7CK25CIB2YH4VLMORSTUY6NEEYX3OSX233H3KA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 814354965504, "Remote Address": "50.7.230.34:58950"}
2023-10-13T12:11:54Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "MBGXWTIX6KM7ADVJ3EITCL2CVGB2SBT6Y4X575CZO7Y3WL7O7RXQ"}
2023-10-13T12:11:54Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "H3MDT4NKA4PVKYEKTRW73P7M7QJ3QHXQ2LNEKCKUQYUWD3XWLMNA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814354930432, "Remote Address": "5.161.220.231:37344"}
2023-10-13T12:11:55Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "4N7PEBMRECI5GTMVZ2VMOMWHM47H5OFINDF4SLLWOTP4NVHLKMAA"}
2023-10-13T12:11:55Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "AQB4AZ2CJR67AHXIXSPGDJAOQ6NOC35NH47YY6E4YB2DIPQZTHWA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814354930432, "Remote Address": "5.161.220.231:47754"}
2023-10-13T12:11:55Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "HWEQHJWHDDXT36ZSF7LX4RSBNOLW5AICFSCLQDSC2DCJ6OJFBZJQ"}
2023-10-13T12:11:55Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "SMP3NQL6CJJE65N3XQGQRNHW7TVGDJIUR5ZQYP4GMZKMOKUYT6DA"}
2023-10-13T12:11:55Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "BUZ3W5EYYF27J7G44W3EWKFJL5AGMIHSSEPJFDXCABUADMYDM6AA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814354930432, "Remote Address": "50.7.22.194:48664"}
2023-10-13T12:11:55Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "IXD4I422ESWNHZDCVDGM52SG7TKKQF7RREX3JZU4BVHQARQZPWAQ"}
2023-10-13T12:11:55Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "WSH4RX5D6LZYERJ4EOEAJ62FVBJGXDCDHFMRF52PWHDU6K7HUOYQ"}
2023-10-13T12:11:56Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "C74FYAE72L2ZFYIIYYBQIJVZIZ7JCWFN4DQDHPNYF2F64PBDSRUA"}
2023-10-13T12:11:56Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "HN3U4KZ4CLSE5WIKBKVJ4J6J5AQFCOJ6MNGCAHLCAYMD6FPVNLEQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814354887936, "Remote Address": "5.161.143.41:8060"}
2023-10-13T12:11:57Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "GXPRCAAN3JMG5YEVKXGI5EL2T5JANELNQC2TRUSDLS5RW3FVNNAQ"}
2023-10-13T12:11:57Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "KVPI3GKKTF3HDUBY3JVPAWDP3L7ULHAULPMGYIPQBHMPW4J76AIQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814354800896, "Remote Address": "5.161.143.41:29300"}
2023-10-13T12:11:58Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "FILL7SESYIUQMKSPJADN7OLWMNS3MK62ELRNMDQWB4EYYM2DZN6A"}
2023-10-13T12:11:58Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "E7GCPUM37WQ2YB2RB7DVGDA5N7HIQLTE35NAYGVBVQGNZ3AEOXRA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814354768896, "Remote Address": "5.161.143.41:20778"}
2023-10-13T12:11:58Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "7RCKS5NLD335R4RGS4W6HG2TO4LS62HBVT2IYX4LQLVRVJU5DP7A"}
2023-10-13T12:11:58Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "S23ZNHUFOX5IM3MKFZHKZNFJAGD3ZEEZL2TKYVCTWWUSR6GE2UGQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814354766848, "Remote Address": "5.161.192.57:34610"}
2023-10-13T12:11:59Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "R2VIEM5DAOWEJHYYA5CKRGBPXNW7MVLQP4LMCZKAEBKF37NA3BFA"}
2023-10-13T12:11:59Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "LLVID4S22WV63UUOVAKRXRFOGUKRBUNKDG2OFOJ7OTG257RFBX6A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814354766848, "Remote Address": "5.161.220.231:34420"}
2023-10-13T12:11:59Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "MHJMGUUPJTI2ODVEHY72WXVNBYBO6VPTW6H5PQOX3VN7YNP6SKWQ"}
2023-10-13T12:11:59Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "VGWB7XCLABAONSHX66FKG32B4RZYPPB3NORDXYBE4KITJNEMJMRQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 814354766848, "Remote Address": "50.7.22.194:39992"}
2023-10-13T12:11:59Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "3DSUXJAJPKXTNXYIHFZYYRVYSOF4VGKQO3SWCGJQI5MZZYCHFDEA"}
2023-10-13T12:11:59Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "YFMKWSFZC4OBSR3ZNSYZX4QDWAZCSWNDLDGVIGYD4DBADCYUAI7A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814354766848, "Remote Address": "5.161.176.200:49356"}
2023-10-13T12:11:59Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "XQRIDBNGVVLCJO6JAEXGWTDEBDEMUYFRXGOJDI2C23AAQHSWBSLQ"}
2023-10-13T12:11:59Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "HPFOFEKLNHB5WZTZ6JX5GDRTE2JU33IHG3LXFIWF4GMAUE7LYHJQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814354766848, "Remote Address": "5.161.192.57:27682"}
2023-10-13T12:12:00Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "F3VYLVPRN7HKMILLQV4ZCDTF2HZMPVQ6ITNCCZRSMXCMOJAZT4NA"}
2023-10-13T12:12:00Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "SREKWBKQJLS3ZOBKOWPQOCLC6AEDE3QKKTCLINDSDXXB2W26VJVA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814354766848, "Remote Address": "5.161.220.231:57500"}
2023-10-13T12:12:00Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "2SD7SO53H5SJ4Q4FK7DYZCRFYMRWTTL2UUIEZBQPHRKRTC3L2YVQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814354764800, "Remote Address": "23.237.232.130:50644"}
2023-10-13T12:12:00Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "4PZEH525N7FSHULKZCC7PSIMHNE72BS276U27NVBBMSXR3IZPNBQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814354764800, "Remote Address": "5.161.220.231:28142"}
2023-10-13T12:12:00Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "MY2XJAIK76QU4SR3Y2DOCALYBMGO4QGYJ5XXDJ53SMXTN4CU4B6A"}
2023-10-13T12:12:00Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "R3P2YFECYMIHTRWMJI4ODKRU6AV6E23TCYGTBB7LCKQFTNC44GBA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814354764800, "Remote Address": "23.237.191.42:47352"}
2023-10-13T12:12:00Z    INFO    piecestore      download started        {"process": "storagenode", "Piece ID": "OM7JWBC4BHLGPH5V43MGVIDHXAZZQG5EOPVP7JKB3SKHXHSZDJ7Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 12032, "Remote Address": "23.237.232.146:34306"}
2023-10-13T12:12:00Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "IG5T7GXVCNYUJR5QCCNWZ6WCJB34H3T25F42QV6E6CYPPJFSJLEQ"}
2023-10-13T12:12:00Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "VTPVPUEBHLO22DME4JEXOR2ACSZ2WZPNUKBVWN3XXN3OSRAKS2NA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814354764800, "Remote Address": "23.237.232.130:53636"}
2023-10-13T12:12:01Z    INFO    piecestore      upload canceled {"process": "storagenode", "Piece ID": "EA6JQEGCO6QUXR6Z4THX2FPB233JXSW5TLZHUCYFTTZ5SEUW7FCQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 65536, "Remote Address": "5.161.143.41:63414"}
2023-10-13T12:12:01Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "3WGF7SU3APKU7QV7CJP5LTDOGTJBKQRPTMYMTBPJ2OZS45ZXP3YQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT", "Available Space": 814354747392, "Remote Address": "157.211.109.157:39428"}
2023-10-13T12:12:01Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "J2WE7WFJO2FKEJC5HQCTCXGNQL72MNWEMZWOF4HCZEW3OBPGKY6Q"}
2023-10-13T12:12:01Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "UA5OFASZ56U4K73DZDZMAYQOWTMQ6CXYL4N2NPV55OKDB7DTKE5A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814354747392, "Remote Address": "5.161.117.79:54162"}
2023-10-13T12:12:01Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "5D2UHERTHZQQDNVEUBTXFAMR744FZWUDGAB66OQTUIE6RDAK2NJQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT_REPAIR", "Available Space": 814354747392, "Remote Address": "202.83.99.55:35176"}
2023-10-13T12:12:02Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "VIE2DPIBD2CN7IXZO5NCOKWXJKR6H2IDANJ73OKQNXIFD54FLFCQ"}
2023-10-13T12:12:02Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "KUZKCZSNFS5PNW7L26QZ2J3ZLBQTBLWSS253KRYRKLHA4X3QBD4Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814354739968, "Remote Address": "5.161.192.57:29520"}
2023-10-13T12:12:02Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "MKTMEVH35LLWOND46XIRMER4WLQ2NKYEENHZMTF5DVBVBKPNUBKQ"}
2023-10-13T12:12:02Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "IRGDXFYX65EC5Y3Z3ZFZOY4QX4QD2PRSVBB46RI5RMBCFZ3NL6LQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814354738688, "Remote Address": "5.161.211.187:14258"}
2023-10-13T12:12:03Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "3MQKPEXGRJH7UM6J6BNHSMVO4TK66OSIYRYWPOGBRY7T3I4RCTOQ"}
2023-10-13T12:12:03Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "F3AQKL5NU2TGPCADNAU6XPEQ4DQGWYFOTBGEL6LYIEU2QLHUYIZA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814354736640, "Remote Address": "23.237.191.146:36928"}
2023-10-13T12:12:03Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "TCE37VJTGETVEWK4RZPSFHSILCJYKBC6IRLMHY6KN3IK44FP25ZA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814354736640, "Remote Address": "72.83.198.17:58938"}
2023-10-13T12:12:04Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "2L7HBNT6H7UZO7XWKRTDUL67LFYPV4QUYCAQVOXRZRB6MNCSKEPA"}
2023-10-13T12:12:04Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "2QQL5GJQVYDMP5C3XBSYBYCL6IZ5IWV3WI7HSJ3JMHSQVBLLSEEA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814354732288, "Remote Address": "5.161.192.57:45444"}
2023-10-13T12:12:04Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "3BVLSJ7O4MCYZTS2VKAC7T4HJPO56664HGPYAGWP2Y4XGGWMAF7Q"}
2023-10-13T12:12:04Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"process": "storagenode", "Piece ID": "7EE76XCQZI2EYRYMFOP4P65635QFU3EHZFLDMWBE5EXI5GWNEVMQ"}
2023-10-13T12:12:05Z    INFO    piecestore      upload started  {"process": "storagenode", "Piece ID": "QNG55IGIEZEBUO63PS2OZQXD6VZ3CKM6DRZHBGAC2YONDXIDIBQA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 814354550272, "Remote Address": "5.161.220.231:27708"}

daki82 · October 13, 2023, 1:04pm

Could be everything. Setup?

Roberto · October 13, 2023, 1:14pm

How long has the node been running, how old is it… To see the success rate use

Ruskiem · October 13, 2023, 1:22pm

Yea me too, i noticed today, some pices get canceled ~2 minutes after its started.
I noticed a lot upload started in batch each after another, in short time or same timestamp, so a lot got canceled.
From yesterday the traffic increased 2-3 times, both downloads and uploads.
I’m talking based on my best node, in terms of latency and HDD response times, even with that, im observing a lot of canceled now, but dont know if thats normal.
Long time i was not looking into logs.
For example, i noticed same piece started downloading and canceled in 4 times in a row, each after each in consecutive 8 lines, in same time, no idea if that’s normal, for one piece to be asked 4 times by same ip in one time it was from 50.7.22.66.
i even recocgnise same ips as from Your logs: 23.237.232.146
or 50.7.22.xxx

edit: finnaly 19 sec. later that same piece ID got downloaded succesfully again.

After more investigation, i say: it’s not canceled all the time,
last 15 minutes was sporadically cancelled,
There’s was just an episode, of wall of canceled, 20 minutes ago, (it’s not like this all the time nooo, last 1h and 50 minutes, there was only 2 such wall of canceled uploads, no big deal, all other mostly succesfull!
4020 “download started”, and
9420 “upload started” and
only 353 “upload canceled”,
and 557 “download canceled”
phrases over 1h and 50 minutes in this fresh log)
fun to see, the node is losing more % in downloads than uploads, lol.

“downloaded”, in storagenode.log, means egress for nodes?
oh, no, upload is egress.

confused, coz saw that piece first uploaded, and later in log downloaded same piece,
strange my node would first uplaod it somwhere, means its have it already,
and the same piece ID to download later 3 times successfuly, why if You already have it hmm)

ptC7H12 · October 13, 2023, 7:25pm

I know, but Setup did not change… thats the strange, but now it seems to normalize

Uptime is now 46h.

Alexey · October 14, 2023, 2:24am

No. We always uses a customer’s point of view. So, “uploads” it is an ingress to your node, “downloads” it’s an egress from your node.

Alexey · October 14, 2023, 2:26am

The customers behavior cannot be configured on your node. Any their behavior is normal, any usage patterns in the past cannot be assumed for the future.
In short - perhaps some customers testing the system to find bottlenecks or measure the speed or whatever they do. It also possible that some customers distribute software or stream the video.

Ruskiem · October 14, 2023, 4:38am

whaaat? wow, thats now make sense!
but the the “(race lost or connection lost)” in log, after “upload canceled” confused me!
that kind of note should be rather in egress lines!
don’t have such note in “download canceled” lines tho!
i know the race is in both case, but still, only “uploaded canceled” has this lol

Alexey · October 14, 2023, 4:42am

Because when you download from the network the long tail cancellation is much less: the customer starts 35 downloads for their segment when they want only 29, so as soon as first 29 are finished, all others got canceled. So it’s only 6 nodes. Maybe your node is fast enough for downloads