Extremely high ingress + massive node fault

Alexey · June 4, 2024, 2:35am

Yes, it’s required to be enabled and performed if your node have issues like

However, not during the stress test and running a node in the same time.
So, the suggestion to reduce an allocated space and stopping a defrag during migration to a new disk - not a bad advise.
Or you may reduce the allocation (to stop ingress) and try to finish a defrag. However, it will take a lot of time with the running node and perhaps will cause stopping the node due to checks timeouts. You may increase the check timeout and disable a startup scan as suggested during the defrag and reduce the timeout and comment out the added option after the defrag is finished. Or run the defrag while the node is stopped.

surfercool · June 5, 2024, 4:55am

I was also amazed at why the disks were so heavily used. 1.5 TB of bandwidth this month already. I have a RAID array with 5 drives. 64 TB in total (43 TB usable). My internet connection is almost half used at times over 300-400 MBit.
And yes, they are SMR HDDs. But with an SSD 40GB read cache and 10GB write, the hard drives are 80 up to 100% busy. However, the node manages to process more than 40GB per hour without any problems. However, there are very few upload cancellations in the log. The data is written in one go about every 5 minutes. But the hard drives manage this within a few seconds at writespeeds up to 100MB/s.

I suspect that you need a cache. Check out PrimoCache. Test it for 30 days and decide.

Aka985 · June 5, 2024, 4:49pm

Okay, I stoped all nodes, closed Docker, did defragmentation for 2 CMR drives, reset Docker to defaults, and run 2 nodes.

In few hours evething works fine, logs are Ok:

2024-06-05T16:41:58Z    INFO    piecestore      downloaded      {"Process": "storagenode", "Piece ID": "WUFD2C5W5B3GXP7RJP5MZGBPJAP56N5C4J7OPHDS5X4QWR25S7UA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "Offset": 0, "Size": 22016, "Remote Address": "172.17.0.1:39320"}
2024-06-05T16:42:01Z    INFO    piecestore      upload started  {"Process": "storagenode", "Piece ID": "2YITVLXO6JWHZJTZBLJGVQVRFX7S2C7YUAIT6QCFUNLB4275TLRQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:39322", "Available Space": 12303819981350}
2024-06-05T16:42:01Z    INFO    piecestore      uploaded        {"Process": "storagenode", "Piece ID": "2YITVLXO6JWHZJTZBLJGVQVRFX7S2C7YUAIT6QCFUNLB4275TLRQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:39322", "Size": 3328}
2024-06-05T16:42:01Z    INFO    piecestore      upload started  {"Process": "storagenode", "Piece ID": "DOO7M5IF2D4T2XDXHGGWFWRKOLZYVOPZW2HRSWT5DHVCXPYKVJYA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT_REPAIR", "Remote Address": "172.17.0.1:39326", "Available Space": 12303819977510}
2024-06-05T16:42:01Z    INFO    piecestore      uploaded        {"Process": "storagenode", "Piece ID": "DOO7M5IF2D4T2XDXHGGWFWRKOLZYVOPZW2HRSWT5DHVCXPYKVJYA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT_REPAIR", "Remote Address": "172.17.0.1:39326", "Size": 38400}
2024-06-05T16:42:02Z    INFO    piecestore      upload started  {"Process": "storagenode", "Piece ID": "6IWFHVNPX5SCEF5YJTVGXE2IXGKJHNSA73MFMF4L37Y65SQ5XU7A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "172.17.0.1:39264", "Available Space": 12303819938598}
2024-06-05T16:42:03Z    INFO    piecestore      uploaded        {"Process": "storagenode", "Piece ID": "6IWFHVNPX5SCEF5YJTVGXE2IXGKJHNSA73MFMF4L37Y65SQ5XU7A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "172.17.0.1:39264", "Size": 664064}
2024-06-05T16:42:03Z    INFO    piecestore      download started        {"Process": "storagenode", "Piece ID": "AEGAM77I4N44MKKLCOFB7CWC3EE4UCKHHMYQS5SX6Q46PPUPW3UA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 144896, "Size": 217600, "Remote Address": "172.17.0.1:39322"}
2024-06-05T16:42:03Z    INFO    piecestore      upload started  {"Process": "storagenode", "Piece ID": "ZWLZJQPLHAT6EIGL5PAZW7I43KWS5HS4IJXB6ZLMYP55DWOFBUTQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "172.17.0.1:39330", "Available Space": 12303819274022}
2024-06-05T16:42:03Z    INFO    piecestore      upload started  {"Process": "storagenode", "Piece ID": "FL55FOEWLS4OIS2G7NBY4FYGXI4YRLNDG46GAF3J3XZ3WPKL7WTQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:39332", "Available Space": 12303819274022}
2024-06-05T16:42:03Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"Process": "storagenode", "Piece ID": "ZWLZJQPLHAT6EIGL5PAZW7I43KWS5HS4IJXB6ZLMYP55DWOFBUTQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "172.17.0.1:39330"}
2024-06-05T16:42:03Z    INFO    piecestore      uploaded        {"Process": "storagenode", "Piece ID": "FL55FOEWLS4OIS2G7NBY4FYGXI4YRLNDG46GAF3J3XZ3WPKL7WTQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:39332", "Size": 3328}
2024-06-05T16:42:04Z    INFO    piecestore      downloaded      {"Process": "storagenode", "Piece ID": "AEGAM77I4N44MKKLCOFB7CWC3EE4UCKHHMYQS5SX6Q46PPUPW3UA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 144896, "Size": 217600, "Remote Address": "172.17.0.1:39322"}
2024-06-05T16:42:09Z    INFO    piecestore      upload started  {"Process": "storagenode", "Piece ID": "XR522GOTVEY5LZXRLBEE62T6WPPO5OWPQUZEBS7SL2EBRWEBEN3A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "172.17.0.1:39336", "Available Space": 12303819267622}
2024-06-05T16:42:09Z    INFO    piecestore      uploaded        {"Process": "storagenode", "Piece ID": "XR522GOTVEY5LZXRLBEE62T6WPPO5OWPQUZEBS7SL2EBRWEBEN3A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "172.17.0.1:39336", "Size": 145664}
2024-06-05T16:42:09Z    INFO    piecestore      upload started  {"Process": "storagenode", "Piece ID": "CW275UFXN6LAR3X5E2UQ5OP6F2RLH4VPJACMU5XG4OVUCYFWPBBA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:39340", "Available Space": 12303819121446}
2024-06-05T16:42:09Z    INFO    piecestore      uploaded        {"Process": "storagenode", "Piece ID": "CW275UFXN6LAR3X5E2UQ5OP6F2RLH4VPJACMU5XG4OVUCYFWPBBA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:39340", "Size": 3328}
2024-06-05T16:42:10Z    INFO    piecestore      upload started  {"Process": "storagenode", "Piece ID": "SJ3YQTF6HRAEJKZOWO5IU5SJZW7YSK24XKTGVEI5CPQFB4IAAIWQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "172.17.0.1:39264", "Available Space": 12303819117606}
2024-06-05T16:42:10Z    INFO    piecestore      uploaded        {"Process": "storagenode", "Piece ID": "SJ3YQTF6HRAEJKZOWO5IU5SJZW7YSK24XKTGVEI5CPQFB4IAAIWQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "172.17.0.1:39264", "Size": 2304}
2024-06-05T16:42:11Z    INFO    piecestore      upload started  {"Process": "storagenode", "Piece ID": "GG2I2J2CI5DLMBUC34OK4P4YXO7Q6J5RFNYEDM3P77S5GNXHDG3Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "172.17.0.1:39344", "Available Space": 12303819114790}
2024-06-05T16:42:11Z    INFO    piecestore      uploaded        {"Process": "storagenode", "Piece ID": "GG2I2J2CI5DLMBUC34OK4P4YXO7Q6J5RFNYEDM3P77S5GNXHDG3Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "172.17.0.1:39344", "Size": 1024}
2024-06-05T16:42:12Z    INFO    piecestore      download started        {"Process": "storagenode", "Piece ID": "ZI45EOSUV4DOQCW67ZOJ44DN2CBSANZIYKP6E7Z5MCT3I5A7LJNA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 181248, "Remote Address": "172.17.0.1:39232"}
2024-06-05T16:42:13Z    INFO    piecestore      upload started  {"Process": "storagenode", "Piece ID": "4QHZITJTOCXQ4IGJDQOSVQ6T3WV3FHNIXVL5WHGSHP33SS76LYTA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "172.17.0.1:39348", "Available Space": 12303819113254}
2024-06-05T16:42:13Z    INFO    piecestore      download started        {"Process": "storagenode", "Piece ID": "6SAOQZXJKVVU6MBTJTARXZN5EU27M5L263UDIWKMHFJBAUP5ZDDA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 7168, "Remote Address": "172.17.0.1:39352"}
2024-06-05T16:42:13Z    INFO    piecestore      uploaded        {"Process": "storagenode", "Piece ID": "4QHZITJTOCXQ4IGJDQOSVQ6T3WV3FHNIXVL5WHGSHP33SS76LYTA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "172.17.0.1:39348", "Size": 2304}
2024-06-05T16:42:13Z    INFO    piecestore      downloaded      {"Process": "storagenode", "Piece ID": "6SAOQZXJKVVU6MBTJTARXZN5EU27M5L263UDIWKMHFJBAUP5ZDDA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 7168, "Remote Address": "172.17.0.1:39352"}
2024-06-05T16:42:13Z    INFO    piecestore      downloaded      {"Process": "storagenode", "Piece ID": "ZI45EOSUV4DOQCW67ZOJ44DN2CBSANZIYKP6E7Z5MCT3I5A7LJNA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 181248, "Remote Address": "172.17.0.1:39232"}
2024-06-05T16:42:15Z    INFO    piecestore      upload started  {"Process": "storagenode", "Piece ID": "RA43HSWUA2O4EGPO7AZ2XST66I2NNYOTZ5FIJIPGNO2G5Y32C3NQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:39094", "Available Space": 12303819110438}
2024-06-05T16:42:15Z    INFO    piecestore      uploaded        {"Process": "storagenode", "Piece ID": "RA43HSWUA2O4EGPO7AZ2XST66I2NNYOTZ5FIJIPGNO2G5Y32C3NQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:39094", "Size": 3328}
2024-06-05T16:42:16Z    INFO    piecestore      download started        {"Process": "storagenode", "Piece ID": "EGQQSEWAHRC2KXRSKI3HNL5QEVGTRSSEJ7U7XQ6CWVLFFWNGVOXA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 12288, "Remote Address": "172.17.0.1:39356"}
2024-06-05T16:42:16Z    INFO    piecestore      downloaded      {"Process": "storagenode", "Piece ID": "EGQQSEWAHRC2KXRSKI3HNL5QEVGTRSSEJ7U7XQ6CWVLFFWNGVOXA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 12288, "Remote Address": "172.17.0.1:39356"}
2024-06-05T16:42:16Z    INFO    piecestore      download started        {"Process": "storagenode", "Piece ID": "AQKMCTZDSJZDQ3ZAGI3YZVRP6LPRDOL24LGPDJYL25KL2YKOENAA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 10240, "Remote Address": "172.17.0.1:39360"}
2024-06-05T16:42:16Z    INFO    piecestore      downloaded      {"Process": "storagenode", "Piece ID": "AQKMCTZDSJZDQ3ZAGI3YZVRP6LPRDOL24LGPDJYL25KL2YKOENAA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 10240, "Remote Address": "172.17.0.1:39360"}
2024-06-05T16:42:17Z    INFO    piecestore      download started        {"Process": "storagenode", "Piece ID": "JDXQ3EJAFQW7ZTN5UG6VGJS67D5W3QBE5NOH4VOJBWR5SZHHB62Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 181248, "Remote Address": "172.17.0.1:38646"}
2024-06-05T16:42:18Z    INFO    piecestore      upload started  {"Process": "storagenode", "Piece ID": "VAOACSVEYGEABHOBRROKS2F5V5LJRARCU3SCOGEOXM5T66E7ZV3A", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:39094", "Available Space": 12303819106598}
2024-06-05T16:42:18Z    INFO    piecestore      uploaded        {"Process": "storagenode", "Piece ID": "VAOACSVEYGEABHOBRROKS2F5V5LJRARCU3SCOGEOXM5T66E7ZV3A", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:39094", "Size": 249856}
2024-06-05T16:42:18Z    INFO    piecestore      upload started  {"Process": "storagenode", "Piece ID": "LWK7HNSJRMX6ND6RXVH7AGFYJWAZ24AB2CE7JZGWMLP72NGYQW2A", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:39144", "Available Space": 12303818856230}
2024-06-05T16:42:18Z    INFO    piecestore      uploaded        {"Process": "storagenode", "Piece ID": "LWK7HNSJRMX6ND6RXVH7AGFYJWAZ24AB2CE7JZGWMLP72NGYQW2A", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:39144", "Size": 3328}
2024-06-05T16:42:18Z    INFO    piecestore      upload started  {"Process": "storagenode", "Piece ID": "JH3NEROELGXI65432HCG6WFFYT7AEZ65NQL34OATO6SS4MCUXY3A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "172.17.0.1:39364", "Available Space": 12303818852390}
2024-06-05T16:42:18Z    INFO    piecestore      uploaded        {"Process": "storagenode", "Piece ID": "JH3NEROELGXI65432HCG6WFFYT7AEZ65NQL34OATO6SS4MCUXY3A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "172.17.0.1:39364", "Size": 51968}
2024-06-05T16:42:19Z    INFO    piecestore      download started        {"Process": "storagenode", "Piece ID": "DBZXGTQNQY4QS7LX3IKEWARK5CDRK46S7ZC6FDKNF343DQTM2JNQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 1024, "Remote Address": "172.17.0.1:39368"}
2024-06-05T16:42:19Z    INFO    piecestore      downloaded      {"Process": "storagenode", "Piece ID": "DBZXGTQNQY4QS7LX3IKEWARK5CDRK46S7ZC6FDKNF343DQTM2JNQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 1024, "Remote Address": "172.17.0.1:39368"}
2024-06-05T16:42:20Z    INFO    piecestore      downloaded      {"Process": "storagenode", "Piece ID": "JDXQ3EJAFQW7ZTN5UG6VGJS67D5W3QBE5NOH4VOJBWR5SZHHB62Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 181248, "Remote Address": "172.17.0.1:38646"}
2024-06-05T16:42:20Z    INFO    piecestore      upload started  {"Process": "storagenode", "Piece ID": "EKVVHOMWIVFPM7BAAY5R3U3XEKEC3QZYB732SIEMSVTMWOCPC5PA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:39376", "Available Space": 12303818799910}
2024-06-05T16:42:20Z    INFO    piecestore      uploaded        {"Process": "storagenode", "Piece ID": "EKVVHOMWIVFPM7BAAY5R3U3XEKEC3QZYB732SIEMSVTMWOCPC5PA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:39376", "Size": 249856}
2024-06-05T16:42:21Z    INFO    piecestore      download started        {"Process": "storagenode", "Piece ID": "65WJ4LJVIDKOP36XE3ZDICUNNCAT26VI5SXF7FQ42SY4DTS2F36A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "Offset": 0, "Size": 181504, "Remote Address": "172.17.0.1:39380"}
2024-06-05T16:42:21Z    INFO    piecestore      upload started  {"Process": "storagenode", "Piece ID": "I2TL646M5LSZFGNI5GHKVZIAQ6OGDWBBK4NQJLRLMKXWRX3F4WWA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "172.17.0.1:39382", "Available Space": 12303818549542}
2024-06-05T16:42:21Z    INFO    piecestore      uploaded        {"Process": "storagenode", "Piece ID": "I2TL646M5LSZFGNI5GHKVZIAQ6OGDWBBK4NQJLRLMKXWRX3F4WWA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "172.17.0.1:39382", "Size": 9216}
2024-06-05T16:42:21Z    INFO    piecestore      downloaded      {"Process": "storagenode", "Piece ID": "65WJ4LJVIDKOP36XE3ZDICUNNCAT26VI5SXF7FQ42SY4DTS2F36A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "Offset": 0, "Size": 181504, "Remote Address": "172.17.0.1:39380"}
2024-06-05T16:42:22Z    INFO    piecestore      upload started  {"Process": "storagenode", "Piece ID": "E5NBZQMT3IBV3WXQEB4KB5FHCFKFA672XJVGO6NCKVVCZIX6YC6A", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Remote Address": "172.17.0.1:39372", "Available Space": 12303818539814}

But after few hours both get down with very strange logs. I cant insert it all here, because it is huge, but this is it’s end:

One HDD Seagate EXOS ST18000NM000J-2TV103 : 18000,2 GB, another one is WDC WD80PURZ-85YNPY0 : 8001,5 GB. Both are CMR

Whats is this now?

nerdatwork · June 5, 2024, 4:56pm

The long error line is a stack trace. You can look for lines above the start of this stack trace.

Aka985 · June 5, 2024, 5:13pm

It starts like this:

Do you mean I shall request more than 50 lines of log?

nerdatwork · June 5, 2024, 6:10pm

It would surprise you but its just 1 line in the log that is around 1.4 million characters long that looks like 50 lines in the command prompt. Try to look for the lines above the start of this stack trace.

Aka985 · June 5, 2024, 6:32pm

100 lines log request give same picture. Is some log file available with all the logs?

nerdatwork · June 5, 2024, 7:15pm

Yes, since you didnt redirect log to file it will be under /var/lib/docker/containers'. There you would see container names. Inside the container you will see <container name>-json.log file which is your log.

Alexey · June 6, 2024, 3:23am

Need a head of the exception not the tail. The excerpt from the stack-trace looking very similar to having issues with the RAM (you need to test it or replace).

To get a full line search for FATAL (PowerShell),

docker logs storagenode 2>&1 | sls fatal -context 1,2

EasyRhino · June 6, 2024, 5:12pm

probably not related to the crash error, but if a SMR disk is having trouble handling ingress, you can modify the config.yaml and modify storage2.max-concurrent-requests: to something really low. like 8 or less.

buchette · June 6, 2024, 5:44pm

wow you did not lie, massive traffic. My 2 cheap USB HDD were completely congestionned, i moved the databases to the system SSD, and it’s now runnning OK.

On the other hand my 3 other datacenter grade HDD (ST12000NM0127 ) so called “refurbished” but new in reality from amazon are delivering amazing iops performance.

Aka985 · June 7, 2024, 7:00pm

At the end looks heavy defragmentation helped. Node was off durin the process. Atleast node is online second day and I proceed to copy it to CMR HDD. Also I reduced an allocated space till it’s moving will be finished.

Thanks to Everybody for support!

Regards,
Alexander

Aka985 · June 10, 2024, 7:30am

No, it is NOT. During weekeng several nodes again gone offline. Reset of docker help fot a while. Is it so, what 3 10tb nodes under Docker under Win10 is not viable? I am about to stop all this and forget about Storj.

pangolin · June 10, 2024, 1:42pm

Storj has changed a lot. When I started my first node years ago it was install and forget. Only checked payments once a month. Now it needs permanent care and some knowledge in networking and storage.

I decided to accept that, to learn the things it needs and to devote the time it needs. But I understand for many people it might be the time to stop now.

Aka985 · June 10, 2024, 3:07pm

If it would be checklist what to do, but not. I am asking - doing, it doesn’t help, asking again… Looks everething I could, I have done.

Do somebody have big size nodes under docker under Win? How much per motherboard? One per motherboard works fine, no problem, sometimes 2-3 works ok. But 4 and more not.

pangolin · June 10, 2024, 3:22pm

How about moving databases to ssd? Have you tried this advice?

Aka985 · June 10, 2024, 3:40pm

No, I asked about this option, but as I understood what was not a solution. Which db to be moved?

пн, 10 июн. 2024 г., 18:32 pangolin via Storj Community Forum (official) <storj@literatehosting.com>:

pangolin · June 10, 2024, 4:19pm

All databases.

stop node
move all files with extension .db to a ssd folder
make storage2.database-dir: pointing to that folder
start node

Aka985 · June 10, 2024, 4:34pm

Do you mean that DB’s (exclude temporary files)?

And then on SSD move files here:

And open config.yaml

Correct?

pangolin · June 10, 2024, 4:39pm

Yes. But stop node first, shm and wal files should not be copied. And remove # from storage2.database-dir.