Hello Sirs,
Last couple of weeks I have extremely high ingres on my nodes
I do not know is it related or not, but nodes become down twice a day or more often almost on each PC. I have 6 PC, 4-5 node each 10-18Tb CMR. All defragmentated. It looks like a disaster? like a virus.
Do somebody have same problems? Any ideas about what to do?
Regards,
Alexander
Okay, I got in general. And what shall I do to avoid nodes fault?
Regards, Alexander
I have 30 nodes and nothing node is gone down. Verify your configuration because you have many problems. explain your configuration and we can help you.
System: Win10, Docker 2.1.0.5, HDD WD 5.4Tb SMR
I run defragmentation on Friday, status were fragmented 4%. It is still working and only 56% done. Standard defragmentation setup ones a week, but looks it doesnt work
Node health is critical:
Logs immidiatly after node start looks normal , except one error:
C:\Users\aka>docker logs --tail 50 storagenodeD1.12
2024-06-03T10:52:11Z INFO piecestore upload started {"Process": "storagenode", "Piece ID": "O7SMWYJQQURREB5VZPUE3R5DHZ7GGVE7O7S3Z5VFOSZAVLSSKRTA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:58052", "Available Space": 387006544318}
2024-06-03T10:52:11Z INFO piecestore upload started {"Process": "storagenode", "Piece ID": "LHJJRDH4HGWWFNMOXWGHHUEWWUUSL5P76BXLDAYC74N7T6AMJR7A", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:58066", "Available Space": 387006544318}
2024-06-03T10:52:11Z INFO piecestore download canceled {"Process": "storagenode", "Piece ID": "YCNBNZFBOZB7MJY6ZOYQ7SBMAQCKL4CP5RG4WFHZWFLS5HRXFSCQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 181248, "Remote Address": "172.17.0.1:58200"}
2024-06-03T10:52:11Z INFO piecestore upload canceled (race lost or node shutdown) {"Process": "storagenode", "Piece ID": "IQ6Q5D7SNLANFRARZEUKKJOQW2A6UBAQP6YYPIIHKJSTQOPRJ23A", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:58126"}
2024-06-03T10:52:11Z INFO piecestore upload canceled (race lost or node shutdown) {"Process": "storagenode", "Piece ID": "QSUA6EMQ6BMWXFI6I7POVZPKQDJOQ6GJKS3Q6HBGWGM4FBV3YE4A", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:58088"}
2024-06-03T10:52:11Z INFO piecestore upload canceled (race lost or node shutdown) {"Process": "storagenode", "Piece ID": "TQCZC2UEAANEV24DKDC4KFSTWQ74UEKPDZZLSAROC3B2O3IWNO4A", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:58130"}
2024-06-03T10:52:11Z INFO piecestore upload canceled (race lost or node shutdown) {"Process": "storagenode", "Piece ID": "VUIEFUZVV6YU32JC2VWSZVNOVZ2KJA5AOUWWJZ23CVJRVTBHMKEA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:57984"}
2024-06-03T10:52:11Z INFO piecestore upload canceled (race lost or node shutdown) {"Process": "storagenode", "Piece ID": "HJ6ZTO4JO2KL2KSREV36NKA4UUIMCGSECFVHSN5SLDXO4PF7Z6UA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:58104"}
2024-06-03T10:52:11Z INFO piecestore upload canceled (race lost or node shutdown) {"Process": "storagenode", "Piece ID": "7EQ7DXEFRC275GG3UVHH2RKNTBECU64M6JHVVPANE6GUJU5NQX3Q", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:58154"}
2024-06-03T10:52:11Z INFO piecestore upload canceled (race lost or node shutdown) {"Process": "storagenode", "Piece ID": "7JIQQ2VGEFHINFF6LYMUODDGXLDCL3FDF7YGTZE6M4TD3NHEWHNA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:58062"}
2024-06-03T10:52:11Z INFO piecestore upload canceled (race lost or node shutdown) {"Process": "storagenode", "Piece ID": "ZTSXYNRFM6BRA5Y54NWTOLG45PRPRNGB6XOULOFS3DJXISP4VMQQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:58144"}
2024-06-03T10:52:12Z INFO piecestore upload canceled (race lost or node shutdown) {"Process": "storagenode", "Piece ID": "35OGRBBKJQO46GFQEYDY6ZIGQQ6JFIJE4TSJYKPPXGNHSGQMORPQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:58076"}
2024-06-03T10:52:12Z INFO piecestore upload canceled (race lost or node shutdown) {"Process": "storagenode", "Piece ID": "O7SMWYJQQURREB5VZPUE3R5DHZ7GGVE7O7S3Z5VFOSZAVLSSKRTA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:58052"}
2024-06-03T10:52:12Z INFO piecestore upload canceled (race lost or node shutdown) {"Process": "storagenode", "Piece ID": "N5LPNOXL35XUY77RIKWUG6HE5RPXD2OHMFLTERBF3H6H7WH2GVHQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:57992"}
2024-06-03T10:52:12Z INFO piecestore upload canceled (race lost or node shutdown) {"Process": "storagenode", "Piece ID": "LHJJRDH4HGWWFNMOXWGHHUEWWUUSL5P76BXLDAYC74N7T6AMJR7A", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:58066"}
2024-06-03T10:52:14Z ERROR piecestore upload failed {"Process": "storagenode", "Piece ID": "FNWC7HKR42WGVDQWSCL7BFRFU3PDGW3IBKI4TZNKQUET3JW2JETQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:58214", "Size": 196608, "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:229"}
2024-06-03T10:52:14Z INFO orders.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE sending {"Process": "storagenode", "count": 567}
2024-06-03T10:52:14Z INFO orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S sending {"Process": "storagenode", "count": 215}
2024-06-03T10:52:15Z INFO orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S finished {"Process": "storagenode"}
2024-06-03T10:52:15Z INFO orders.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE finished {"Process": "storagenode"}
2024-06-03T10:52:31Z WARN console:service unable to get Satellite URL {"Process": "storagenode", "Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "error": "console: trust: satellite is untrusted", "errorVerbose": "console: trust: satellite is untrusted\n\tstorj.io/storj/storagenode/trust.init:29\n\truntime.doInit1:6740\n\truntime.doInit:6707\n\truntime.main:249"}
2024-06-03T10:52:31Z WARN console:service unable to get Satellite URL {"Process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "error": "console: trust: satellite is untrusted", "errorVerbose": "console: trust: satellite is untrusted\n\tstorj.io/storj/storagenode/trust.init:29\n\truntime.doInit1:6740\n\truntime.doInit:6707\n\truntime.main:249"}
2024-06-03T10:52:33Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "ZNLXINR4UO4AUDHWEC77M5CDDNDM7RTHY4EVW5R7K4FL5DZPQOQQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Offset": 0, "Size": 25600, "Remote Address": "172.17.0.1:58228"}
2024-06-03T10:52:34Z INFO piecestore upload started {"Process": "storagenode", "Piece ID": "GIV3JCFEPX5YPXMMJRLALZZLO64YETTJWMIROOKDQWP7AULYLPZA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:58052", "Available Space": 387005793214}
2024-06-03T10:52:34Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "ZNLXINR4UO4AUDHWEC77M5CDDNDM7RTHY4EVW5R7K4FL5DZPQOQQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Offset": 0, "Size": 25600, "Remote Address": "172.17.0.1:58228"}
2024-06-03T10:52:36Z INFO piecestore uploaded {"Process": "storagenode", "Piece ID": "GIV3JCFEPX5YPXMMJRLALZZLO64YETTJWMIROOKDQWP7AULYLPZA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:58052", "Size": 249856}
2024-06-03T10:52:44Z INFO piecestore upload started {"Process": "storagenode", "Piece ID": "MODU5XBPNJBXPVXECW5HYYLB3DZNOLNEZZN2EGV7XP6J3LRD6P5Q", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:58234", "Available Space": 387005542846}
2024-06-03T10:52:45Z INFO piecestore upload canceled (race lost or node shutdown) {"Process": "storagenode", "Piece ID": "MODU5XBPNJBXPVXECW5HYYLB3DZNOLNEZZN2EGV7XP6J3LRD6P5Q", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:58234"}
2024-06-03T10:52:51Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "24OVB5CT44DXWOIW7L52ZTXMPZH2FEBDSIPIOEGSWF6GR72EYQJA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 9728, "Remote Address": "172.17.0.1:58238"}
After node fault:
C:\Users\aka>docker logs --tail 50 storagenodeD1.12
2024-06-03T10:52:12Z INFO piecestore upload canceled (race lost or node shutdown) {"Process": "storagenode", "Piece ID": "N5LPNOXL35XUY77RIKWUG6HE5RPXD2OHMFLTERBF3H6H7WH2GVHQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:57992"}
2024-06-03T10:52:12Z INFO piecestore upload canceled (race lost or node shutdown) {"Process": "storagenode", "Piece ID": "LHJJRDH4HGWWFNMOXWGHHUEWWUUSL5P76BXLDAYC74N7T6AMJR7A", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:58066"}
2024-06-03T10:52:14Z ERROR piecestore upload failed {"Process": "storagenode", "Piece ID": "FNWC7HKR42WGVDQWSCL7BFRFU3PDGW3IBKI4TZNKQUET3JW2JETQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:58214", "Size": 196608, "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:229"}
2024-06-03T10:52:14Z INFO orders.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE sending {"Process": "storagenode", "count": 567}
2024-06-03T10:52:14Z INFO orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S sending {"Process": "storagenode", "count": 215}
2024-06-03T10:52:15Z INFO orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S finished {"Process": "storagenode"}
2024-06-03T10:52:15Z INFO orders.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE finished {"Process": "storagenode"}
2024-06-03T10:52:31Z WARN console:service unable to get Satellite URL {"Process": "storagenode", "Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "error": "console: trust: satellite is untrusted", "errorVerbose": "console: trust: satellite is untrusted\n\tstorj.io/storj/storagenode/trust.init:29\n\truntime.doInit1:6740\n\truntime.doInit:6707\n\truntime.main:249"}
2024-06-03T10:52:31Z WARN console:service unable to get Satellite URL {"Process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "error": "console: trust: satellite is untrusted", "errorVerbose": "console: trust: satellite is untrusted\n\tstorj.io/storj/storagenode/trust.init:29\n\truntime.doInit1:6740\n\truntime.doInit:6707\n\truntime.main:249"}
2024-06-03T10:52:33Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "ZNLXINR4UO4AUDHWEC77M5CDDNDM7RTHY4EVW5R7K4FL5DZPQOQQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Offset": 0, "Size": 25600, "Remote Address": "172.17.0.1:58228"}
2024-06-03T10:52:34Z INFO piecestore upload started {"Process": "storagenode", "Piece ID": "GIV3JCFEPX5YPXMMJRLALZZLO64YETTJWMIROOKDQWP7AULYLPZA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:58052", "Available Space": 387005793214}
2024-06-03T10:52:34Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "ZNLXINR4UO4AUDHWEC77M5CDDNDM7RTHY4EVW5R7K4FL5DZPQOQQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Offset": 0, "Size": 25600, "Remote Address": "172.17.0.1:58228"}
2024-06-03T10:52:36Z INFO piecestore uploaded {"Process": "storagenode", "Piece ID": "GIV3JCFEPX5YPXMMJRLALZZLO64YETTJWMIROOKDQWP7AULYLPZA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:58052", "Size": 249856}
2024-06-03T10:52:44Z INFO piecestore upload started {"Process": "storagenode", "Piece ID": "MODU5XBPNJBXPVXECW5HYYLB3DZNOLNEZZN2EGV7XP6J3LRD6P5Q", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:58234", "Available Space": 387005542846}
2024-06-03T10:52:45Z INFO piecestore upload canceled (race lost or node shutdown) {"Process": "storagenode", "Piece ID": "MODU5XBPNJBXPVXECW5HYYLB3DZNOLNEZZN2EGV7XP6J3LRD6P5Q", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:58234"}
2024-06-03T10:52:51Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "24OVB5CT44DXWOIW7L52ZTXMPZH2FEBDSIPIOEGSWF6GR72EYQJA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 9728, "Remote Address": "172.17.0.1:58238"}
C:\Users\aka>docker logs --tail 50 storagenodeD1.12
2024-06-03T12:55:24Z INFO piecestore upload started {"Process": "storagenode", "Piece ID": "4K7IWVLBAUU6TGFWFOJQF5LGBSWPONFYIPHVNXUM23VUVAJWFFIQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:48612", "Available Space": 386975924158}
2024-06-03T12:55:27Z INFO piecestore upload canceled (race lost or node shutdown) {"Process": "storagenode", "Piece ID": "QBR7U6BO3KRZWZGHSGQFGWVHDNREMPWIQGBW6BQSJOKGPCSHQC4A", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:48084"}
2024-06-03T12:55:27Z INFO piecestore upload canceled (race lost or node shutdown) {"Process": "storagenode", "Piece ID": "7BYJHVAHAPYUGGACIFAAKY7DX6L56GPNEDF5U4IENA6AEKEULLNA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:48488"}
2024-06-03T12:55:27Z INFO piecestore upload canceled (race lost or node shutdown) {"Process": "storagenode", "Piece ID": "UW6MUU2BU5N5QRVQNLMKMU3M7JVTQ2R3IEL6XAD3NFKYNNOOK4XQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:48154"}
2024-06-03T12:55:27Z INFO piecestore upload canceled (race lost or node shutdown) {"Process": "storagenode", "Piece ID": "D5QDKG7EHZODCQVY63RMZ6LOM66W7K7347LNMJ42UXZAIDBKSHMA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:48318"}
2024-06-03T12:55:27Z INFO piecestore upload canceled (race lost or node shutdown) {"Process": "storagenode", "Piece ID": "4K7IWVLBAUU6TGFWFOJQF5LGBSWPONFYIPHVNXUM23VUVAJWFFIQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:48612"}
2024-06-03T12:55:27Z INFO piecestore upload canceled (race lost or node shutdown) {"Process": "storagenode", "Piece ID": "PDTBZ4PW2K2EZNNOYIJO5RCFYZOMS3B5QGTCH5HPPI6IT7DRC74Q", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:47694"}
2024-06-03T12:55:29Z INFO piecestore upload started {"Process": "storagenode", "Piece ID": "STIJXZFAKAOEZD2JJBR6KAS5DEQ2RSIU4HUDFKNGXWNSEXR4E5OQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:48510", "Available Space": 386975654590}
2024-06-03T12:55:30Z INFO piecestore uploaded {"Process": "storagenode", "Piece ID": "STIJXZFAKAOEZD2JJBR6KAS5DEQ2RSIU4HUDFKNGXWNSEXR4E5OQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:48510", "Size": 3328}
2024-06-03T12:55:30Z INFO piecestore upload started {"Process": "storagenode", "Piece ID": "TDEWPYY7DXI42FJCKDHOGSEZQAM5VIVFJDYFTFLL64QAMFJWGCIQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:48068", "Available Space": 386975650750}
2024-06-03T12:55:30Z INFO piecestore upload started {"Process": "storagenode", "Piece ID": "JER3XZGKV4OIS5AMRS4WZ4VOFDKBM22DNGLBWAJBRMD64X3JZ3HQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:48474", "Available Space": 386975650750}
2024-06-03T12:55:31Z INFO piecestore uploaded {"Process": "storagenode", "Piece ID": "JER3XZGKV4OIS5AMRS4WZ4VOFDKBM22DNGLBWAJBRMD64X3JZ3HQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:48474", "Size": 249856}
2024-06-03T12:55:31Z INFO piecestore uploaded {"Process": "storagenode", "Piece ID": "TDEWPYY7DXI42FJCKDHOGSEZQAM5VIVFJDYFTFLL64QAMFJWGCIQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:48068", "Size": 249856}
2024-06-03T12:55:31Z INFO piecestore downloaded {"Process": "storagenode", "Piece ID": "PDSAYAIPTK3BLYE2UDWH3EIC4B2LQQRMEHNKJGWW267EODN3P7ZA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_REPAIR", "Offset": 0, "Size": 58112, "Remote Address": "172.17.0.1:48606"}
2024-06-03T12:55:31Z INFO piecestore upload started {"Process": "storagenode", "Piece ID": "AHUF6GXSZPXKT3JKRYAP4UJOH3IXGHOQJEOZ6Y22DMZQ7GPY6V6Q", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:48616", "Available Space": 386975150014}
2024-06-03T12:55:31Z INFO piecestore uploaded {"Process": "storagenode", "Piece ID": "AHUF6GXSZPXKT3JKRYAP4UJOH3IXGHOQJEOZ6Y22DMZQ7GPY6V6Q", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:48616", "Size": 3328}
2024-06-03T12:55:32Z INFO piecestore upload started {"Process": "storagenode", "Piece ID": "Q7BWWRSN6XE6WOYYL5VSSJMSMQU2QCRB2OXTIWMHSPOJ4OMNHG6A", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:48622", "Available Space": 386975146174}
2024-06-03T12:55:36Z INFO piecestore upload started {"Process": "storagenode", "Piece ID": "5CSCAMSRBFKPHRB6WA7453QE4FTUX3GQI4WNATWXHTLB5XOKCKLQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:48624", "Available Space": 386975146174}
2024-06-03T12:55:36Z INFO piecestore upload canceled (race lost or node shutdown) {"Process": "storagenode", "Piece ID": "Q7BWWRSN6XE6WOYYL5VSSJMSMQU2QCRB2OXTIWMHSPOJ4OMNHG6A", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:48622"}
2024-06-03T12:55:36Z INFO piecestore uploaded {"Process": "storagenode", "Piece ID": "5CSCAMSRBFKPHRB6WA7453QE4FTUX3GQI4WNATWXHTLB5XOKCKLQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:48624", "Size": 249856}
2024-06-03T12:55:37Z INFO piecestore upload started {"Process": "storagenode", "Piece ID": "6BCIGPRLW6IAWBJXWNG5OL6V7WN4WAS4KMVLFTS3P55N277ZZQJA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:48628", "Available Space": 386974645438}
2024-06-03T12:55:38Z INFO piecestore upload canceled (race lost or node shutdown) {"Process": "storagenode", "Piece ID": "6BCIGPRLW6IAWBJXWNG5OL6V7WN4WAS4KMVLFTS3P55N277ZZQJA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:48628"}
2024-06-03T12:55:40Z INFO piecestore upload started {"Process": "storagenode", "Piece ID": "ICWIHDWH7X32YEHLFBO5CIOZGFQMMVL6LSOHX5F6TX7WDEEGK42Q", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:48318", "Available Space": 386974395070}
2024-06-03T12:55:41Z INFO piecestore uploaded {"Process": "storagenode", "Piece ID": "ICWIHDWH7X32YEHLFBO5CIOZGFQMMVL6LSOHX5F6TX7WDEEGK42Q", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "172.17.0.1:48318", "Size": 3328}
2024-06-03T12:55:46Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "PNFEZK65UOG27CWNIK3VG7LXVP5B3DCLNIELWP7IVWIG5H3YRZ6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_REPAIR", "Offset": 0, "Size": 2319360, "Remote Address": "172.17.0.1:48632"}
2024-06-03T12:55:48Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "ORQRQHO5JYSTNSAR36FLKPPXYEMPY4AF42EWRNXAQUAGDNMRIHAA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 8704, "Remote Address": "172.17.0.1:48634"}
2024-06-03T12:55:51Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "RQ75X65ATQLWFH6ASFXHFAVNBHXHJSOJ32PWNXGSTOD5W5OE54ZA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 190464, "Remote Address": "172.17.0.1:48650"}
2024-06-03T12:55:52Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "74QB2TSI6BL46ONTCQFOSUL7HB4K5DN2B3XYBL5QOPSAGO5EF3WA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 3840, "Remote Address": "172.17.0.1:48590"}
2024-06-03T12:55:54Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "RGLW7UQBEJ54R5D7F4ZF5QEZXKYW5RMM2LD35L2HEBRYAB2RWPXQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 2560, "Remote Address": "172.17.0.1:48658"}
2024-06-03T12:55:56Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "7SI4J2CQEAAXDQBEMXGL46TDPAUWECETZVONVZFDXYZEKIZSS6FA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET_REPAIR", "Offset": 0, "Size": 8448, "Remote Address": "172.17.0.1:48666"}
2024-06-03T12:56:07Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "CRT352AT54JCQB6KGTSLXYDJOOIMSSTQMMP2E3IUGBCSKL56HNPQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 8704, "Remote Address": "172.17.0.1:48680"}
2024-06-03T12:56:15Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "4GZCNPW4P4CRCACPPZDG4DVRPIF3OVZXNA6AWVKYCJGBY5LTR2BA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 60928, "Remote Address": "172.17.0.1:48692"}
2024-06-03T12:56:24Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "4LGTTFSOR5YZEIQHS5APQVWUMPD7ISASLMSOXBWTVPE3BYETHDVA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 8960, "Remote Address": "172.17.0.1:48720"}
2024-06-03T12:56:29Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "YPAFVP7ZDASAHPI7RWVOX7FDDAJLQYZELTKKR5HSEMIK36RF666Q", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET_REPAIR", "Offset": 0, "Size": 17408, "Remote Address": "172.17.0.1:48730"}
2024-06-03T12:56:32Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "3567VFRZ7C6DAT5VLWCKBCBXKLXXGLJYFKDE5OX3MRHZ6KYCRKOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 13056, "Remote Address": "172.17.0.1:48740"}
2024-06-03T12:56:35Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "SLUKMSFDBYG4XMHAZD2CUUTXEFUJDVGV764BH2JIHZZBMCAZG6KA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 360704, "Remote Address": "172.17.0.1:48748"}
2024-06-03T12:56:35Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "RT7H7XZ6OVGHVPEUBP6RMNQL2LHTRYNW64WB2RQZZKVC5NIWGQOA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET_REPAIR", "Offset": 0, "Size": 5632, "Remote Address": "172.17.0.1:48752"}
2024-06-03T12:56:42Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "HC66AVTS57G6GD5M7DEFJ67Z5U2VHIM53D7FRY3PEKBQT7Q5DOUA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 22784, "Remote Address": "172.17.0.1:48766"}
2024-06-03T12:56:50Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "DSUM6BOYEMY6RGPHTW2TH6M32752RYIMZRYFQT64QTST6G2PVPZQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_REPAIR", "Offset": 0, "Size": 2319360, "Remote Address": "172.17.0.1:48802"}
2024-06-03T12:56:54Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "WXP7KTPU6FW4ENR5M2ITIDSXQX62ZXIOGJZPV6HQVO7C22NVVQMA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 487168, "Remote Address": "172.17.0.1:48058"}
2024-06-03T12:56:58Z WARN console:service unable to get Satellite URL {"Process": "storagenode", "Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "error": "console: trust: satellite is untrusted", "errorVerbose": "console: trust: satellite is untrusted\n\tstorj.io/storj/storagenode/trust.init:29\n\truntime.doInit1:6740\n\truntime.doInit:6707\n\truntime.main:249"}
2024-06-03T12:56:58Z WARN console:service unable to get Satellite URL {"Process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "error": "console: trust: satellite is untrusted", "errorVerbose": "console: trust: satellite is untrusted\n\tstorj.io/storj/storagenode/trust.init:29\n\truntime.doInit1:6740\n\truntime.doInit:6707\n\truntime.main:249"}
2024-06-03T12:57:02Z WARN console:service unable to get Satellite URL {"Process": "storagenode", "Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "error": "console: trust: satellite is untrusted", "errorVerbose": "console: trust: satellite is untrusted\n\tstorj.io/storj/storagenode/trust.init:29\n\truntime.doInit1:6740\n\truntime.doInit:6707\n\truntime.main:249"}
2024-06-03T12:57:02Z WARN console:service unable to get Satellite URL {"Process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "error": "console: trust: satellite is untrusted", "errorVerbose": "console: trust: satellite is untrusted\n\tstorj.io/storj/storagenode/trust.init:29\n\truntime.doInit1:6740\n\truntime.doInit:6707\n\truntime.main:249"}
2024-06-03T12:57:04Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "QUA7ZMREYZYGIFUSRQXXPRLIITJ3H5CYN6WVOSYD4VNC4TGI3OIA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET_REPAIR", "Offset": 0, "Size": 5120, "Remote Address": "172.17.0.1:48826"}
2024-06-03T12:57:24Z ERROR services unexpected shutdown of a runner {"Process": "storagenode", "name": "piecestore:monitor", "error": "piecestore monitor: timed out after 1m0s while verifying readability of storage directory", "errorVerbose": "piecestore monitor: timed out after 1m0s while verifying readability of storage directory\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func1.1:154\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func1:143\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-06-03T12:57:24Z INFO lazyfilewalker.trash-cleanup-filewalker subprocess exited with status {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "status": -1, "error": "signal: killed"}
2024-06-03T12:57:24Z ERROR pieces:trash emptying trash failed {"Process": "storagenode", "error": "pieces error: lazyfilewalker: signal: killed", "errorVerbose": "pieces error: lazyfilewalker: signal: killed\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:85\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkCleanupTrash:187\n\tstorj.io/storj/storagenode/pieces.(*Store).EmptyTrash:419\n\tstorj.io/storj/storagenode/pieces.(*TrashChore).Run.func1.1:84\n\tstorj.io/common/sync2.(*Workplace).Start.func1:89"}
2024-06-03T12:57:24Z INFO lazyfilewalker.used-space-filewalker subprocess exited with status {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "status": -1, "error": "signal: killed"}
2024-06-03T12:57:24Z ERROR pieces failed to lazywalk space used by satellite {"Process": "storagenode", "error": "lazyfilewalker: signal: killed", "errorVerbose": "lazyfilewalker: signal: killed\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:85\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:130\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:704\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
I don’t know why you are using windows defrag ! you are killing your node…
Your issue is the SMR disk drive, it’s nearly full, so i would expect every WRITE to be requiring a really slow resliver, combined with the defrag it will just die (seen as non-responsive, hung node, cancelled uploads / downloads)
I would stop the defrag for a start… either from defrag tool, or scheduled tasks if you set it up there.
As for saving the node, given the recent changes to the satellite node selection SMR disks are pretty much useless… Even if you do get it working, the node selection will hardly pick your node as it will be slow on write, and you will find long term your node will only be getting repair traffic, and maybe 2-3% of upload.
Linux nodes have tools that will allow the node to be saved, lvmcache or bcachefs or ZFS L2arc to name a few…
Windows, you can use Storage Spaces Direct with an SSD cache, but current implementation uses a scheduled task to move hot / cold data to and from the SSD, it’s not really usage scenario for SNO’s…
Maybe others will know how to save node…
#edit : As a moment, you’ve not put the model of WD disk - The firmware is designed to re-organize the data an resliver when there is minimal disk activity. If you are able to completely stop all access to the disk, defrag and node for a few hours it might give the drive a chance to recover. You will know if it’s working as you will hear the HDD clicking away as it sorts itself out, after a few hours or once the data noise from hdd finishes, start the node back up again.
CP 
I am not advanced for Linux or Win Server. It’s too compligated. I trying to copy this node to CMR Seagate EXOS X18 ST18000NM000J, but it goes really slow. Ofcourse not together with defragmentation. But anyway it will not be finished before node will be disqualifyed. What if I will extend WD HDD with sone other 1Tb HDD or 250Gb SSD? Can it help? Or maybe move somу database to SSD?
Model of WD is WDC WD60EFAX-68JH4N1 : 6001,1 GB
Regards,
Alexander
I understand 
but Linux while initially looks scary, you get use to how flexible it is…Maybe one day you will feel like using it, but I agree now is not the time to change operating system.
So If it was me, trying to save that node I would;
priority is to get online score back to 100%, that takes 30 days, so this won’t be quick ! but quicker then re-vetting a node for 6+ months.
stop all defrags on that drive, stop any copies to another disk, they need to wait for online score.
edit your node launch script for docker for that node;
a) set the STORAGE allocated to 1TB (replacing the 5.4TB you currently have), don’t worry the node doesn’t suddenly get rid of all the data, it takes months ! the aim is to stop any more data coming in and overloading the node.
b) disable the piece scan on startup, find the nodes config.yaml it will be in the directory with all the node data, add this to the file and save;
storage2.piece-scan-on-startup: false
now start the node, and leave it alone until the online score has recovered.
once online score above 70%, start the copy again to the new CMR disk
once the node data copy, put the Storage allocation back to 10TB, and comment out the piece scan on startup using # at beginning of line on the new disk.
#Warning : Obviously don’t run the node on the old disk, once the node moved to new disk else it will instantly ban node, only run node in one place, on new disk once all data is copied and you updated all the config.
Hope that helps, If anyone notices glaring issues, point them out 
CP
Thank you so much for such detailed reply)
I did according to your instruction:
Added command to the end of config.yaml
And setup 1Tb node size:
Log after node start, something looks wrong:
C:\Users\aka>docker logs --tail 50 storagenodeD1.12
2024-06-03T16:12:32Z INFO Operator email {"Process": "storagenode", "Address": "7437493@gmail.com"}
2024-06-03T16:12:32Z INFO Operator wallet {"Process": "storagenode", "Address": "0xE158e01cDb77F9f220d8359335Fe0b75799829AA"}
2024-06-03T16:12:33Z INFO server kernel support for tcp fast open unknown {"Process": "storagenode"}
2024-06-03T16:12:33Z INFO Current binary version {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.104.5"}
2024-06-03T16:12:33Z INFO Version is up to date {"Process": "storagenode-updater", "Service": "storagenode"}
2024-06-03T16:12:33Z INFO Current binary version {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.104.5"}
2024-06-03T16:12:33Z INFO Version is up to date {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-06-03T16:12:33Z INFO Telemetry enabled {"Process": "storagenode", "instance ID": "12dha8yhAHeYTyCFdRm12YCQYZQwsX6BWUGksUcsdxrRUupNArF"}
2024-06-03T16:12:33Z INFO Event collection enabled {"Process": "storagenode", "instance ID": "12dha8yhAHeYTyCFdRm12YCQYZQwsX6BWUGksUcsdxrRUupNArF"}
2024-06-03 16:12:34,524 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-06-03 16:12:34,525 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-06-03 16:12:34,525 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-06-03T16:12:47Z INFO db.migration Database Version {"Process": "storagenode", "version": 57}
2024-06-03T16:12:59Z INFO preflight:localtime start checking local system clock with trusted satellites' system clock. {"Process": "storagenode"}
2024-06-03T16:12:59Z INFO preflight:localtime local system clock is in sync with trusted satellites' system clock. {"Process": "storagenode"}
2024-06-03T16:12:59Z INFO piecestore:cache Startup piece scan omitted by configuration {"Process": "storagenode"}
2024-06-03T16:12:59Z INFO Node 12dha8yhAHeYTyCFdRm12YCQYZQwsX6BWUGksUcsdxrRUupNArF started {"Process": "storagenode"}
2024-06-03T16:12:59Z INFO Public server started on [::]:28967 {"Process": "storagenode"}
2024-06-03T16:12:59Z INFO Private server started on 127.0.0.1:7778 {"Process": "storagenode"}
2024-06-03T16:12:59Z INFO failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details. {"Process": "storagenode"}
2024-06-03T16:12:59Z INFO trust Scheduling next refresh {"Process": "storagenode", "after": "4h48m14.832544677s"}
2024-06-03T16:12:59Z INFO bandwidth Persisting bandwidth usage cache to db {"Process": "storagenode"}
2024-06-03T16:12:59Z INFO pieces:trash emptying trash started {"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-06-03T16:12:59Z INFO lazyfilewalker.trash-cleanup-filewalker starting subprocess {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-06-03T16:12:59Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "CZ6LMZ2SRCVGTVPVIAKWTSXPRBN7GJR5D56ZOA6AVUOPLDQCFSDA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 11776, "Remote Address": "172.17.0.1:35688"}
2024-06-03T16:12:59Z ERROR piecestore download failed {"Process": "storagenode", "Piece ID": "CZ6LMZ2SRCVGTVPVIAKWTSXPRBN7GJR5D56ZOA6AVUOPLDQCFSDA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 11776, "Remote Address": "172.17.0.1:35688", "error": "untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup us1.storj.io: operation was canceled", "errorVerbose": "untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup us1.storj.io: operation was canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).VerifyOrderLimitSignature:140\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:62\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:621\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:167\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:109\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:157\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-06-03T16:12:59Z WARN piecestore:monitor Used more space than allocated. Allocated space is {"Process": "storagenode", "bytes": 1000000000000}
2024-06-03T16:13:00Z WARN console:service unable to get Satellite URL {"Process": "storagenode", "Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "error": "console: trust: satellite is untrusted", "errorVerbose": "console: trust: satellite is untrusted\n\tstorj.io/storj/storagenode/trust.init:29\n\truntime.doInit1:6740\n\truntime.doInit:6707\n\truntime.main:249"}
2024-06-03T16:13:00Z WARN console:service unable to get Satellite URL {"Process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "error": "console: trust: satellite is untrusted", "errorVerbose": "console: trust: satellite is untrusted\n\tstorj.io/storj/storagenode/trust.init:29\n\truntime.doInit1:6740\n\truntime.doInit:6707\n\truntime.main:249"}
2024-06-03T16:13:00Z INFO lazyfilewalker.trash-cleanup-filewalker subprocess started {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-06-03T16:13:00Z INFO lazyfilewalker.trash-cleanup-filewalker.subprocess trash-filewalker started {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "dateBefore": "2024-05-27T16:12:59Z", "Process": "storagenode"}
2024-06-03T16:13:00Z INFO lazyfilewalker.trash-cleanup-filewalker.subprocess Database started {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode"}
2024-06-03T16:13:00Z INFO lazyfilewalker.trash-cleanup-filewalker.subprocess trash-filewalker completed {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "numKeysDeleted": 0, "Process": "storagenode", "bytesDeleted": 0}
2024-06-03T16:13:00Z INFO lazyfilewalker.trash-cleanup-filewalker subprocess finished successfully {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-06-03T16:13:00Z INFO pieces:trash emptying trash finished {"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "elapsed": "1.0054617s"}
2024-06-03T16:13:00Z INFO pieces:trash emptying trash started {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-06-03T16:13:00Z INFO lazyfilewalker.trash-cleanup-filewalker starting subprocess {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-06-03T16:13:00Z INFO lazyfilewalker.trash-cleanup-filewalker subprocess started {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-06-03T16:13:00Z INFO lazyfilewalker.trash-cleanup-filewalker.subprocess trash-filewalker started {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode", "dateBefore": "2024-05-27T16:13:00Z"}
2024-06-03T16:13:00Z INFO lazyfilewalker.trash-cleanup-filewalker.subprocess Database started {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}
2024-06-03T16:13:00Z INFO lazyfilewalker.trash-cleanup-filewalker.subprocess trash-filewalker completed {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "bytesDeleted": 0, "numKeysDeleted": 0, "Process": "storagenode"}
2024-06-03T16:13:01Z INFO lazyfilewalker.trash-cleanup-filewalker subprocess finished successfully {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-06-03T16:13:01Z INFO pieces:trash emptying trash finished {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "elapsed": "33.6162ms"}
2024-06-03T16:13:01Z INFO pieces:trash emptying trash started {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-06-03T16:13:01Z INFO lazyfilewalker.trash-cleanup-filewalker starting subprocess {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-06-03T16:13:01Z INFO lazyfilewalker.trash-cleanup-filewalker subprocess started {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-06-03T16:13:01Z INFO lazyfilewalker.trash-cleanup-filewalker.subprocess trash-filewalker started {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode", "dateBefore": "2024-05-27T16:13:01Z"}
2024-06-03T16:13:01Z INFO lazyfilewalker.trash-cleanup-filewalker.subprocess Database started {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode"}
2024-06-03T16:13:10Z INFO piecestore download started {"Process": "storagenode", "Piece ID": "RBP4WVW7JT4PNYHIAU7M3N2X4MRXXI6YABRQ6E7CWZ527LT7DRCQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 8448, "Remote Address": "172.17.0.1:35746"}
2024-06-03T16:13:11Z INFO piecestore download canceled {"Process": "storagenode", "Piece ID": "RBP4WVW7JT4PNYHIAU7M3N2X4MRXXI6YABRQ6E7CWZ527LT7DRCQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 8448, "Remote Address": "172.17.0.1:35746"}
It is always 100% load:
Is it ok?
Guys, sorry to interrupt you), any ideas about my topic? Did I done everething right?
Node logs look ok 
The D: at 100% is not ok…
Can I just check, you only have 1 node on the D:, and that’s the one we are working with ?
If that’s the case, there is something else using the D: and killing disk - you did stop the Defrag and the copy didn’t you ?
Can you post an updated last 50 lines of the node log please.
Only one node. Defrag and copy is off. If I stop Doker, load become zero. I just checked, node is down. Looks I shall kill it and do not use this SMR drive anymore. But this is not the end of the story. I will proceed tomorrow.
Thans a lot for your support!
Windows and Docker is a pain. I gave up on that, using now Storj-Node-Toolbox to run multiple windows nodes. No problems any more.
Is this sime kind of conteiner software instead of Docker? Is it for Win?
No containers. Its just a helper to install multiple nodes as windows services.
Here it is:
C:\Users\aka>docker logs --tail 50 storagenodeD1.12
2024-06-03T18:43:42Z INFO Got a signal from the OS: "terminated" {"Process": "storagenode-updater"}
2024-06-03 18:43:42,225 INFO stopped: storagenode-updater (exit status 0)
2024-06-03 18:43:43,226 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)
2024-06-03 18:43:44,788 INFO Set uid to user 0 succeeded
2024-06-03 18:43:44,792 INFO RPC interface 'supervisor' initialized
2024-06-03 18:43:44,792 INFO supervisord started with pid 1
2024-06-03 18:43:45,794 INFO spawned: 'processes-exit-eventlistener' with pid 10
2024-06-03 18:43:45,795 INFO spawned: 'storagenode' with pid 11
2024-06-03 18:43:45,796 INFO spawned: 'storagenode-updater' with pid 12
2024-06-03T18:43:45Z INFO Configuration loaded {"Process": "storagenode-updater", "Location": "/app/config/config.yaml"}
2024-06-03T18:43:45Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "console.address"}
2024-06-03T18:43:45Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "operator.email"}
2024-06-03T18:43:45Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "operator.wallet"}
2024-06-03T18:43:45Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "storage.allocated-disk-space"}
2024-06-03T18:43:45Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "storage.allocated-bandwidth"}
2024-06-03T18:43:45Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "healthcheck.details"}
2024-06-03T18:43:45Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "storage2.piece-scan-on-startup"}
2024-06-03T18:43:45Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "healthcheck.enabled"}
2024-06-03T18:43:45Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "contact.external-address"}
2024-06-03T18:43:45Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "operator.wallet-features"}
2024-06-03T18:43:45Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "server.address"}
2024-06-03T18:43:45Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "server.private-address"}
2024-06-03T18:43:45Z INFO Anonymized tracing enabled {"Process": "storagenode-updater"}
2024-06-03T18:43:45Z INFO Running on version {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.104.5"}
2024-06-03T18:43:45Z INFO Downloading versions. {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-06-03T18:43:45Z INFO Configuration loaded {"Process": "storagenode", "Location": "/app/config/config.yaml"}
2024-06-03T18:43:45Z INFO Anonymized tracing enabled {"Process": "storagenode"}
2024-06-03T18:43:45Z INFO Operator email {"Process": "storagenode", "Address": "7437493@gmail.com"}
2024-06-03T18:43:45Z INFO Operator wallet {"Process": "storagenode", "Address": "0xE158e01cDb77F9f220d8359335Fe0b75799829AA"}
2024-06-03T18:43:45Z ERROR failure during run {"Process": "storagenode", "error": "Error opening database on storagenode: database: pricing opening file \"config/storage/pricing.db\" failed: unable to open database file: no such file or directory\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:364\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:341\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:316\n\tstorj.io/storj/storagenode/storagenodedb.OpenExisting:281\n\tmain.cmdRun:65\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:267", "errorVerbose": "Error opening database on storagenode: database: pricing opening file \"config/storage/pricing.db\" failed: unable to open database file: no such file or directory\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:364\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:341\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:316\n\tstorj.io/storj/storagenode/storagenodedb.OpenExisting:281\n\tmain.cmdRun:65\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:267\n\tmain.cmdRun:67\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:267"}
Error: Error opening database on storagenode: database: pricing opening file "config/storage/pricing.db" failed: unable to open database file: no such file or directory
storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:364
storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:341
storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:316
storj.io/storj/storagenode/storagenodedb.OpenExisting:281
main.cmdRun:65
main.newRunCmd.func1:33
storj.io/common/process.cleanup.func1.4:393
storj.io/common/process.cleanup.func1:411
github.com/spf13/cobra.(*Command).execute:983
github.com/spf13/cobra.(*Command).ExecuteC:1115
github.com/spf13/cobra.(*Command).Execute:1039
storj.io/common/process.ExecWithCustomOptions:112
main.main:34
runtime.main:267
2024-06-03 18:43:45,871 INFO exited: storagenode (exit status 1; not expected)
2024-06-03T18:43:46Z INFO Current binary version {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.104.5"}
2024-06-03T18:43:46Z INFO Version is up to date {"Process": "storagenode-updater", "Service": "storagenode"}
2024-06-03T18:43:46Z INFO Current binary version {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.104.5"}
2024-06-03T18:43:46Z INFO Version is up to date {"Process": "storagenode-updater", "Service": "storagenode-updater"}
Ok that node is not working now, it’s complaining it can’t find the database on the disk - looks like it could be in a restart loop, probably why your online has become low;
I wonder if that disk is failing…
Last try, stop the node on it, and run a check disk to see if errors, from command prompt;
chkdsk D: /f /r /x
but this one might not make it, sorry 
I was under the impression we want defrag on win/ntfs nodes - saw quite some discussion/topics about this not too long ago?