Storagenode suspended, but has been online for the last few weeks

[Ottetal]

Hello lads,

Last weekend I celebrated a milestone - I’ve moved all of my nodes from Windows to Linux. Huge decrease in consumed power - great!

One of my nodes though, 12JTcEWGuLHEDRvXjq7dpXLCGkDf2PE8sHxDqSarJZaKQRLBqYG, was just suspended on all satellites this night. I’ve included a section of it’s log, filtered to only warning and above. As you can see, while it does have a few errors (as to be expected), most of the log is the node checking it’s version every 15 minutes.

Here is also a screenshot of the nodes page. As can be seen, it was moved from the end of February to the mid of march. I’m sure the actual move was a bit faster, but I was on a businesstrip, and could not manually take the node online. Online stat is decreasing every day

StorJ-05-Win10  | 2025-03-24T03:59:16Z  INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
StorJ-05-Win10  | 2025-03-24T03:59:16Z  INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.124.6"}
StorJ-05-Win10  | 2025-03-24T03:59:16Z  INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
StorJ-05-Win10  | 2025-03-24T03:59:16Z  INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.124.6"}
StorJ-05-Win10  | 2025-03-24T03:59:16Z  INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
StorJ-05-Win10  | 2025-03-24T04:02:10Z  ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "DLDS4TJIEWVFFJSHZUVX5VZLE5D6AAZDMOUYIZG6I57NKGV7S6KA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 579840, "Remote Address": "79.127.219.35:51720", "error": "hashstore: file does not exist", "errorVerbose": "hashstore: file does not exist\n\tstorj.io/storj/storagenode/hashstore.(*DB).Read:333\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).Reader:298\n\tstorj.io/storj/storagenode/piecestore.(*MigratingBackend).Reader:180\n\tstorj.io/storj/storagenode/piecestore.(*TestingBackend).Reader:105\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:676\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
StorJ-05-Win10  | 2025-03-24T04:09:56Z  ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "DLYIW6PF45NFVKCY2XIZBZ3MGUMH32XTJMYLYTCUMEU4FM4DBNQA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 8960, "Remote Address": "79.127.201.219:48024", "error": "hashstore: file does not exist", "errorVerbose": "hashstore: file does not exist\n\tstorj.io/storj/storagenode/hashstore.(*DB).Read:333\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).Reader:298\n\tstorj.io/storj/storagenode/piecestore.(*MigratingBackend).Reader:180\n\tstorj.io/storj/storagenode/piecestore.(*TestingBackend).Reader:105\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:676\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
StorJ-05-Win10  | 2025-03-24T04:14:16Z  INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
StorJ-05-Win10  | 2025-03-24T04:14:17Z  INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.124.6"}
StorJ-05-Win10  | 2025-03-24T04:14:17Z  INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
StorJ-05-Win10  | 2025-03-24T04:14:17Z  INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.124.6"}
StorJ-05-Win10  | 2025-03-24T04:14:17Z  INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
StorJ-05-Win10  | 2025-03-24T04:19:45Z  ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "KAU7NGSAE7HONY5BG3N6J76B4RKD6FEWUYMYZTGAWZGMV4OFMXTA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 578816, "Size": 73728, "Remote Address": "79.127.201.219:49970", "error": "write tcp 172.18.0.2:28967->79.127.201.219:49970: use of closed network connection", "errorVerbose": "write tcp 172.18.0.2:28967->79.127.201.219:49970: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:428\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:489\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData.func1:860\n\tstorj.io/storj/storagenode/piecestore.withTimeout[...]:1138\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData:858\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func7:758\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
StorJ-05-Win10  | 2025-03-24T04:29:16Z  INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
StorJ-05-Win10  | 2025-03-24T04:29:16Z  INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.124.6"}
StorJ-05-Win10  | 2025-03-24T04:29:16Z  INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
StorJ-05-Win10  | 2025-03-24T04:29:17Z  INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.124.6"}
StorJ-05-Win10  | 2025-03-24T04:29:17Z  INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
StorJ-05-Win10  | 2025-03-24T04:31:31Z  ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "NZRXNPYEP3DRQGQF5FCGU3QQENQMXUHNFQNB5DYPDYU7RU6IEGAQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 290048, "Remote Address": "192.234.196.162:1712", "error": "write tcp 172.18.0.2:28967->192.234.196.162:1712: write: broken pipe", "errorVerbose": "write tcp 172.18.0.2:28967->192.234.196.162:1712: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:428\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:489\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData.func1:860\n\tstorj.io/storj/storagenode/piecestore.withTimeout[...]:1138\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData:858\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func7:758\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
StorJ-05-Win10  | 2025-03-24T04:39:26Z  ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "VMLLLHSR3RLXN4L7PBERJ55N53O2LB7EMZKFCQMDHRXICPCIQVWQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 181248, "Remote Address": "79.127.219.42:55446", "error": "write tcp 172.18.0.2:28967->79.127.219.42:55446: use of closed network connection", "errorVerbose": "write tcp 172.18.0.2:28967->79.127.219.42:55446: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:428\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:489\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData.func1:860\n\tstorj.io/storj/storagenode/piecestore.withTimeout[...]:1138\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData:858\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func7:758\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
StorJ-05-Win10  | 2025-03-24T04:39:27Z  ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "5MOMGN5K3BWICNSBH42D5SUV2Q63TLQTRWJMV4WMHQPBWABXXVRA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 181248, "Remote Address": "79.127.226.104:39216", "error": "write tcp 172.18.0.2:28967->79.127.226.104:39216: use of closed network connection", "errorVerbose": "write tcp 172.18.0.2:28967->79.127.226.104:39216: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:428\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:489\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData.func1:860\n\tstorj.io/storj/storagenode/piecestore.withTimeout[...]:1138\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData:858\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func7:758\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
StorJ-05-Win10  | 2025-03-24T04:44:16Z  INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
StorJ-05-Win10  | 2025-03-24T04:44:17Z  INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.124.6"}
StorJ-05-Win10  | 2025-03-24T04:44:17Z  INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
StorJ-05-Win10  | 2025-03-24T04:44:17Z  INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.124.6"}
StorJ-05-Win10  | 2025-03-24T04:44:17Z  INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
StorJ-05-Win10  | 2025-03-24T04:53:25Z  ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "DLUZ4ARIRUWGMQ4FFRSATYF4BKS725EIEJH4HJJWDFR7JGIIQKKQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 8704, "Remote Address": "109.61.92.71:55242", "error": "hashstore: file does not exist", "errorVerbose": "hashstore: file does not exist\n\tstorj.io/storj/storagenode/hashstore.(*DB).Read:333\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).Reader:298\n\tstorj.io/storj/storagenode/piecestore.(*MigratingBackend).Reader:180\n\tstorj.io/storj/storagenode/piecestore.(*TestingBackend).Reader:105\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:676\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
StorJ-05-Win10  | 2025-03-24T04:59:16Z  INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
StorJ-05-Win10  | 2025-03-24T04:59:16Z  INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.124.6"}
StorJ-05-Win10  | 2025-03-24T04:59:16Z  INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
StorJ-05-Win10  | 2025-03-24T04:59:16Z  INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.124.6"}
StorJ-05-Win10  | 2025-03-24T04:59:16Z  INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
StorJ-05-Win10  | 2025-03-24T05:04:10Z  ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "XRGM3BMYB6D6GDZAUUSNWA64T7Q4MECOOAIS3CKBZRTT5DY2T4CA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 759808, "Size": 1559296, "Remote Address": "45.86.229.103:58026", "error": "write tcp 172.18.0.2:28967->45.86.229.103:58026: use of closed network connection", "errorVerbose": "write tcp 172.18.0.2:28967->45.86.229.103:58026: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:428\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:489\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData.func1:860\n\tstorj.io/storj/storagenode/piecestore.withTimeout[...]:1138\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData:858\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func7:758\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
StorJ-05-Win10  | 2025-03-24T05:14:16Z  INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
StorJ-05-Win10  | 2025-03-24T05:14:16Z  INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.124.6"}
StorJ-05-Win10  | 2025-03-24T05:14:16Z  INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
StorJ-05-Win10  | 2025-03-24T05:14:16Z  INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.124.6"}
StorJ-05-Win10  | 2025-03-24T05:14:16Z  INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
StorJ-05-Win10  | 2025-03-24T05:27:09Z  ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "WACAGJPQVD7P5QXQ2KMKNYVF7BC2A4FW6FB2AMISVV3JHLZCHLJA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 11520, "Remote Address": "109.61.92.65:45102", "error": "hashstore: file does not exist", "errorVerbose": "hashstore: file does not exist\n\tstorj.io/storj/storagenode/hashstore.(*DB).Read:333\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).Reader:298\n\tstorj.io/storj/storagenode/piecestore.(*MigratingBackend).Reader:180\n\tstorj.io/storj/storagenode/piecestore.(*TestingBackend).Reader:105\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:676\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
StorJ-05-Win10  | 2025-03-24T05:29:16Z  INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
StorJ-05-Win10  | 2025-03-24T05:29:16Z  INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.124.6"}
StorJ-05-Win10  | 2025-03-24T05:29:16Z  INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
StorJ-05-Win10  | 2025-03-24T05:29:16Z  INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.124.6"}
StorJ-05-Win10  | 2025-03-24T05:29:16Z  INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
StorJ-05-Win10  | 2025-03-24T05:43:40Z  ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "PC2QWOMOGHGZXVPBMJADB3QOS2JKJONZG3WFQS73LQHPKT5637NA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_AUDIT", "Offset": 80640, "Size": 256, "Remote Address": "34.86.215.8:61552", "error": "hashstore: file does not exist", "errorVerbose": "hashstore: file does not exist\n\tstorj.io/storj/storagenode/hashstore.(*DB).Read:333\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).Reader:298\n\tstorj.io/storj/storagenode/piecestore.(*MigratingBackend).Reader:180\n\tstorj.io/storj/storagenode/piecestore.(*TestingBackend).Reader:105\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:676\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
StorJ-05-Win10  | 2025-03-24T05:44:16Z  INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
StorJ-05-Win10  | 2025-03-24T05:44:17Z  INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.124.6"}
StorJ-05-Win10  | 2025-03-24T05:44:17Z  INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
StorJ-05-Win10  | 2025-03-24T05:44:17Z  INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.124.6"}
StorJ-05-Win10  | 2025-03-24T05:44:17Z  INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
StorJ-05-Win10  | 2025-03-24T05:45:25Z  ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "DL4AKHNLQI65WYENXVSA4F5YPNBNWTIBV7WGRRMDOZDOPKTX6HXQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 34304, "Remote Address": "79.127.205.231:37472", "error": "hashstore: file does not exist", "errorVerbose": "hashstore: file does not exist\n\tstorj.io/storj/storagenode/hashstore.(*DB).Read:333\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).Reader:298\n\tstorj.io/storj/storagenode/piecestore.(*MigratingBackend).Reader:180\n\tstorj.io/storj/storagenode/piecestore.(*TestingBackend).Reader:105\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:676\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
StorJ-05-Win10  | 2025-03-24T05:47:08Z  ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "DLEWHZ3IIUZI75KR6A4M3FFLF2MWVXTIRLFCBQII4GFLRMJQF4NA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 512, "Remote Address": "121.127.47.25:55442", "error": "hashstore: file does not exist", "errorVerbose": "hashstore: file does not exist\n\tstorj.io/storj/storagenode/hashstore.(*DB).Read:333\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).Reader:298\n\tstorj.io/storj/storagenode/piecestore.(*MigratingBackend).Reader:180\n\tstorj.io/storj/storagenode/piecestore.(*TestingBackend).Reader:105\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:676\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
StorJ-05-Win10  | 2025-03-24T05:54:17Z  ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "DLNCF3WNM2KNRWSBVELIYPHQQN2XUTUEQXTFRABBRZ4OMAPKLXQA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 9216, "Remote Address": "109.61.92.79:57358", "error": "hashstore: file does not exist", "errorVerbose": "hashstore: file does not exist\n\tstorj.io/storj/storagenode/hashstore.(*DB).Read:333\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).Reader:298\n\tstorj.io/storj/storagenode/piecestore.(*MigratingBackend).Reader:180\n\tstorj.io/storj/storagenode/piecestore.(*TestingBackend).Reader:105\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:676\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
StorJ-05-Win10  | 2025-03-24T05:57:31Z  ERROR   piecestore      upload internal error   {"Process": "storagenode", "error": "manager closed: read tcp 172.18.0.2:28967->76.130.182.226:33728: read: connection timed out", "errorVerbose": "manager closed: read tcp 172.18.0.2:28967->76.130.182.226:33728: read: connection timed out\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:230"}
StorJ-05-Win10  | 2025-03-24T05:57:31Z  ERROR   piecestore      upload failed   {"Process": "storagenode", "Piece ID": "GPCBWO3XUEHFLOHF47LECMWZVS65WEFZCD77IRTH2AUXAR2TENPA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "76.130.182.226:33728", "Size": 65536, "error": "manager closed: read tcp 172.18.0.2:28967->76.130.182.226:33728: read: connection timed out", "errorVerbose": "manager closed: read tcp 172.18.0.2:28967->76.130.182.226:33728: read: connection timed out\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:230"}
StorJ-05-Win10  | 2025-03-24T05:58:38Z  ERROR   piecestore      upload internal error   {"Process": "storagenode", "error": "manager closed: closed: read tcp 172.18.0.2:28967->79.127.226.103:41524: read: connection reset by peer", "errorVerbose": "manager closed: closed: read tcp 172.18.0.2:28967->79.127.226.103:41524: read: connection reset by peer\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:230"}
StorJ-05-Win10  | 2025-03-24T05:59:16Z  INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
StorJ-05-Win10  | 2025-03-24T05:59:16Z  INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.124.6"}
StorJ-05-Win10  | 2025-03-24T05:59:16Z  INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
StorJ-05-Win10  | 2025-03-24T05:59:16Z  INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.124.6"}
StorJ-05-Win10  | 2025-03-24T05:59:16Z  INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
StorJ-05-Win10  | 2025-03-24T06:03:12Z  ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "DLYIW6PF45NFVKCY2XIZBZ3MGUMH32XTJMYLYTCUMEU4FM4DBNQA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 8960, "Remote Address": "156.146.43.229:45298", "error": "hashstore: file does not exist", "errorVerbose": "hashstore: file does not exist\n\tstorj.io/storj/storagenode/hashstore.(*DB).Read:333\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).Reader:298\n\tstorj.io/storj/storagenode/piecestore.(*MigratingBackend).Reader:180\n\tstorj.io/storj/storagenode/piecestore.(*TestingBackend).Reader:105\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:676\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
StorJ-05-Win10  | 2025-03-24T06:03:46Z  ERROR   piecestore      upload internal error   {"Process": "storagenode", "error": "manager closed: read tcp 172.18.0.2:28967->71.233.229.94:58316: read: connection timed out", "errorVerbose": "manager closed: read tcp 172.18.0.2:28967->71.233.229.94:58316: read: connection timed out\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:235"}
StorJ-05-Win10  | 2025-03-24T06:03:46Z  ERROR   piecestore      upload failed   {"Process": "storagenode", "Piece ID": "B4YT3XVJM4RLCY7DMTPM7GCHYKFAUVR7BOUQAYLK7KKSGWV5MH7Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "71.233.229.94:58316", "Size": 311296, "error": "manager closed: read tcp 172.18.0.2:28967->71.233.229.94:58316: read: connection timed out", "errorVerbose": "manager closed: read tcp 172.18.0.2:28967->71.233.229.94:58316: read: connection timed out\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:235"}
StorJ-05-Win10  | 2025-03-24T06:05:44Z  ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "OL5AHN7ZG3T4YISAQMQDR2ARSCX3CD3RXDZUWFRK6QEXYAU5ABHQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 1477120, "Size": 63232, "Remote Address": "79.127.226.104:55502", "error": "write tcp 172.18.0.2:28967->79.127.226.104:55502: use of closed network connection", "errorVerbose": "write tcp 172.18.0.2:28967->79.127.226.104:55502: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:428\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:489\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData.func1:860\n\tstorj.io/storj/storagenode/piecestore.withTimeout[...]:1138\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData:858\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func7:758\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
StorJ-05-Win10  | 2025-03-24T06:14:16Z  INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
StorJ-05-Win10  | 2025-03-24T06:14:16Z  INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.124.6"}
StorJ-05-Win10  | 2025-03-24T06:14:16Z  INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
StorJ-05-Win10  | 2025-03-24T06:14:16Z  INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.124.6"}
StorJ-05-Win10  | 2025-03-24T06:14:16Z  INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
StorJ-05-Win10  | 2025-03-24T06:15:26Z  ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "DLJTXY3VDHPCHVUKDMLQ6UBC24R2FG522XZJQUNQXMA7EFVLEBGQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 9728, "Remote Address": "79.127.201.217:33428", "error": "hashstore: file does not exist", "errorVerbose": "hashstore: file does not exist\n\tstorj.io/storj/storagenode/hashstore.(*DB).Read:333\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).Reader:298\n\tstorj.io/storj/storagenode/piecestore.(*MigratingBackend).Reader:180\n\tstorj.io/storj/storagenode/piecestore.(*TestingBackend).Reader:105\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:676\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
StorJ-05-Win10  | 2025-03-24T06:16:22Z  ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "DLYIW6PF45NFVKCY2XIZBZ3MGUMH32XTJMYLYTCUMEU4FM4DBNQA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 8960, "Remote Address": "109.61.92.77:41332", "error": "hashstore: file does not exist", "errorVerbose": "hashstore: file does not exist\n\tstorj.io/storj/storagenode/hashstore.(*DB).Read:333\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).Reader:298\n\tstorj.io/storj/storagenode/piecestore.(*MigratingBackend).Reader:180\n\tstorj.io/storj/storagenode/piecestore.(*TestingBackend).Reader:105\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:676\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
StorJ-05-Win10  | 2025-03-24T06:16:46Z  ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "CRMMDNA5ONKMYBL3MJWRKHSOJCCH543WY2ZDKRDDAG42527JWTPA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 457472, "Size": 1861632, "Remote Address": "79.127.226.104:47840", "error": "write tcp 172.18.0.2:28967->79.127.226.104:47840: use of closed network connection", "errorVerbose": "write tcp 172.18.0.2:28967->79.127.226.104:47840: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:428\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:489\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData.func1:860\n\tstorj.io/storj/storagenode/piecestore.withTimeout[...]:1138\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData:858\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func7:758\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
StorJ-05-Win10  | 2025-03-24T06:24:04Z  ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "DLZWZJREDHO74V5GMDPBMOKHJBAEY7WAIMFCHC7ABCSF3NTCDDTA", "Satellite ID":

[Vadim]

I do not see anything criminal in logs, so i think it just because your online score droped and satellite discovered this. Sattelites also lagging in time. so just be online.
Also what hardware do you have, and how much power consumption droped?

[Ottetal]

It might be. It’s the only node that got suspended, so I’ll try to keep it online for the entirety of march and beyond; let’s see what happens.

Storage is hosted on a Synology rs3617xs, that’s presenting 8x20TB disks over ISCSI.
Compute is handled by a 12400f with 128GB of RAM, running VMware VMs.

After converting all 14 Windows machines, average CPU consumption dropped from 80% to 15%.
Idle RAM for dropped from 1.5ishGB from my highly trimmed Windows VMs to ~200Megs in my Debian machines. All VMs still have 8GB of RAM assigned - they now just get to keep a lot more in cache.

Disk utilization have dropped a wee bit as well, but I can’t attribute that to NTFS vs EXT4 differences - I was running a less than optimal setup before.

[Vadim]

I realy do not understand why you used 14 windows machines to run nodes, it can be made all in 1. But i am glad, you got ride of overcomplicating, but you now running it in 14 debian machines? or all in one?
I run 19 nodes on one windows E5-2680 and it use like 3-5% only

[Julio]

Welp… make sure it does not go down anymore, or investigate STAT. Otherwise, by your graph, it’ obvious it should start rocketing up in a few days, and reach 100% by say 10 of april. Pretty normal - don’t sweat it. 40% is about 12-13 (dependant upon any other intermittent outages) days offline, so say about March 28/29 it’ll start going up.

2 cents,
Julio

[nerdatwork]

@Ottetal

suspended on all satellites

… because your online score went below 60%. Keep it online and it will recover.

[Ottetal]

Why my setup is as it is is outside the scope of this post.

But now that you ask, StorJ has always been the secondary goal of my homelabbing. I’ve written more about it here. I’ve built what I want to automate against windows, and am now moving my nodes to Linux to automate against them.

Starting a new node, worth it in 2024? Also some setup questions

Hosting StorJ started off as a hobby of mine. I was working as a VMware administrator at the time, and I wanted/needed a home enviornment that could mimic not only the servers I had at work, but also the workload induced by guest OSes that was actually doing something StorJ for me was the perfect candidate - it had network activity, it had disk activity and I was free to choose OS to work on. I used my platform to learn a lot about VMware automation as well as windows automation. Gradually, …

[holocronology]

Are you using DDNS?

If yes, what DDNS provider?

[Ottetal]

No DDNS. Fixed Public IP

[Ottetal]

Update: Online score is rising again, after dropping until last night. I’ll keep the node online, and report back in this thread when it’s at 100% again.

[Alexey]

I have a prediction : “after 30 days of online your online score would be 100% But be aware - each downtime will require an additional 30 days to recover” (some mystical sound here).

[Ottetal]

Noooooooooooo way! Impossible

Happy I-Joined-The-Forum-Today Anniversary my friend