Fatal Error on my Node / Timeout after 1 min

No problem. It just seemed very odd that you did not want the logs!

I have run the chkdsk, as recommended from another post and I did post the reults of running it - Is that what you mean by optimize the filesystem or do you mean something else?

Which entry am I meant to be adjusting?

storage2.monitor.verify-dir-writable-interval: 5m0s (because the log shows verifying writability of storage directory)?

storage2.monitor.verify-dir-readable-interval - changed to 1:30

I do not have the other 2 in my config - only the following:

storage2.monitor.verify-dir-writable-interval: 5m0s

Is that this?

storage2.monitor.verify-dir-writable-interval: 5m0s

Set them like in this thread:
https://forum.storj.io/t/questions-about-readability-and-writability-intervals-and-timeouts/25034?u=snorkel

1 Like

I have turned on Storage2.piece-scan-on-startup, and after weeks, my trash is now 3.82 TB (and still decreasing) instead of 6.5 TB. It is working with the lazy file walker all the time. Seems to be working correctly.

1 Like

Hello, since 3 days my node cant run for more than a few hours without crashing… It was working very fine until then.

Always the same error lines before it happens :

2024-08-18T21:25:18+02:00	ERROR	piecestore	upload internal error	{"error": "manager closed: read tcp 192.168.1.10:28967->5.161.77.66:35026: wsarecv: Une connexion existante a dû être fermée par l’hôte distant.", "errorVerbose": "manager closed: read tcp 192.168.1.10:28967->5.161.77.66:35026: wsarecv: Une connexion existante a dû être fermée par l’hôte distant.\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:235"}
2024-08-18T21:25:18+02:00	ERROR	piecestore	upload failed	{"Piece ID": "XEKVLHG3KV6EOEDB6EWVGANJ2SECMFG25PTLR36LYAEW4XH3TSDA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT_REPAIR", "Remote Address": "5.161.77.66:35026", "Size": 65536, "error": "manager closed: read tcp 192.168.1.10:28967->5.161.77.66:35026: wsarecv: Une connexion existante a dû être fermée par l’hôte distant.", "errorVerbose": "manager closed: read tcp 192.168.1.10:28967->5.161.77.66:35026: wsarecv: Une connexion existante a dû être fermée par l’hôte distant.\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:235"}
2024-08-18T21:25:18+02:00	ERROR	piecestore	upload internal error	{"error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:230"}
2024-08-18T21:25:18+02:00	ERROR	piecestore	upload failed	{"Piece ID": "7GJD362NDHCBS4JR3Q73JK2JQLWPGC2IABBLPJAGGLMP53RELAIA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "109.61.92.78:58310", "Size": 524288, "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:230"}
2024-08-18T21:25:18+02:00	ERROR	piecestore	upload internal error	{"error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:230"}
2024-08-18T21:25:18+02:00	ERROR	piecestore	upload failed	{"Piece ID": "XYOMSNK7TOGWTX2HDXJ7OSG3XA4JY76XJ22BMMTJ2EPGOHFVT6LQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "79.127.205.235:54372", "Size": 786432, "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:230"}
2024-08-18T21:25:32+02:00	ERROR	piecestore	upload internal error	{"error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:235"}
2024-08-18T21:25:32+02:00	ERROR	piecestore	upload failed	{"Piece ID": "RNIZORW7HWGIMNHAGKPNI7YEFY6ZLVABMFSXOEBMNWQW5TUS5Y5Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT_REPAIR", "Remote Address": "5.161.124.130:62816", "Size": 589824, "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:235"}
2024-08-18T21:25:53+02:00	ERROR	piecestore	download failed	{"Piece ID": "VXMXQGCKHWCVKOJIHYK5R6DVNF5TTW7EG56ITR3KH63MG2XOFAJA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 8960, "Remote Address": "79.127.201.212:37140", "error": "write tcp 192.168.1.10:28967->79.127.201.212:37140: use of closed network connection", "errorVerbose": "write tcp 192.168.1.10:28967->79.127.201.212:37140: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:428\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:489\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData.func1:921\n\tstorj.io/storj/storagenode/piecestore.withTimeout[...]:1197\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData:919\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func7:819\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-08-18T21:25:53+02:00	ERROR	piecestore	upload internal error	{"error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:230"}
2024-08-18T21:25:53+02:00	ERROR	piecestore	upload failed	{"Piece ID": "JTXP5A6WL4WOL4BGBMOTAXQH2XGWJT2BSZBWRRET2CRIAPN5TX7Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.201.210:41412", "Size": 196608, "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:230"}
2024-08-18T21:25:53+02:00	ERROR	piecestore	upload internal error	{"error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:230"}
2024-08-18T21:25:53+02:00	ERROR	piecestore	upload failed	{"Piece ID": "EAITOKVKTAHJGWZAFS2U6SCFPI26UUBTBZ27F6HDIFIX4FJV57GQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Remote Address": "79.127.201.211:38478", "Size": 458752, "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:230"}
2024-08-18T21:25:55+02:00	ERROR	piecestore	upload internal error	{"error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:230"}
2024-08-18T21:25:55+02:00	ERROR	piecestore	upload failed	{"Piece ID": "ET37FOPXABFMLN453GK7UFKH5YALRELFYFYOPDPHWATSTB2Z3LHQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.226.101:37526", "Size": 65536, "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:230"}
2024-08-18T21:26:14+02:00	ERROR	piecestore	download failed	{"Piece ID": "L4GN22FI5OONCQSOB36K7LNGXS7MM6CCDXRKSXUT3RFGN2GMJG3A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 7936, "Remote Address": "109.61.92.66:35040", "error": "write tcp 192.168.1.10:28967->109.61.92.66:35040: use of closed network connection", "errorVerbose": "write tcp 192.168.1.10:28967->109.61.92.66:35040: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:428\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:489\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData.func1:921\n\tstorj.io/storj/storagenode/piecestore.withTimeout[...]:1197\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData:919\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func7:819\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-08-18T21:26:15+02:00	ERROR	piecestore	upload internal error	{"error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:230"}
2024-08-18T21:26:15+02:00	ERROR	piecestore	upload failed	{"Piece ID": "XAZ4CJQAMIXNYUWZ4DQ3AMBUW3MHBV67Z4LKQF7MWYWBKVCR2QPQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.226.98:45470", "Size": 196608, "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:230"}
2024-08-18T21:27:58+02:00	ERROR	piecestore	download failed	{"Piece ID": "W5K6R6GPTPENRHA47X5OJXW6AC5WI3XUKGFX43QH4I4WEDVXDZYQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 7936, "Remote Address": "79.127.201.212:57330", "error": "manager closed: read tcp 192.168.1.10:28967->79.127.201.212:57330: wsarecv: Une connexion existante a dû être fermée par l’hôte distant.", "errorVerbose": "manager closed: read tcp 192.168.1.10:28967->79.127.201.212:57330: wsarecv: Une connexion existante a dû être fermée par l’hôte distant.\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:211\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:230"}
2024-08-18T21:28:49+02:00	ERROR	services	unexpected shutdown of a runner	{"name": "piecestore:monitor", "error": "piecestore monitor: timed out after 1m0s while verifying writability of storage directory", "errorVerbose": "piecestore monitor: timed out after 1m0s while verifying writability of storage directory\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func2.1:175\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func2:164\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-08-18T21:28:49+02:00	ERROR	gracefulexit:chore	error retrieving satellites.	{"error": "satellitesdb: context canceled", "errorVerbose": "satellitesdb: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*satellitesDB).ListGracefulExits.func1:200\n\tstorj.io/storj/storagenode/storagenodedb.(*satellitesDB).ListGracefulExits:212\n\tstorj.io/storj/storagenode/gracefulexit.(*Service).ListPendingExits:59\n\tstorj.io/storj/storagenode/gracefulexit.(*Chore).AddMissing:55\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tstorj.io/storj/storagenode/gracefulexit.(*Chore).Run:48\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-08-18T21:29:03+02:00	ERROR	pieces	used-space-filewalker failed	{"Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Lazy File Walker": false, "error": "filewalker: context canceled", "errorVerbose": "filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:74\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:79\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkAndComputeSpaceUsedBySatellite:728\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:81\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-08-18T21:29:03+02:00	ERROR	piecestore:cache	encountered error while computing space used by satellite	{"error": "filewalker: context canceled", "errorVerbose": "filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:74\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:79\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkAndComputeSpaceUsedBySatellite:728\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:81\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "SatelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-08-18T21:29:03+02:00	ERROR	pieces	used-space-filewalker failed	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Lazy File Walker": false, "error": "filewalker: context canceled", "errorVerbose": "filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:74\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:79\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkAndComputeSpaceUsedBySatellite:728\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:81\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-08-18T21:29:03+02:00	ERROR	piecestore:cache	encountered error while computing space used by satellite	{"error": "filewalker: context canceled", "errorVerbose": "filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:74\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:79\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkAndComputeSpaceUsedBySatellite:728\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:81\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-08-18T21:29:03+02:00	ERROR	retain	retain pieces failed	{"cachePath": "D:\\STORJ/retain", "error": "retain: filewalker: context canceled", "errorVerbose": "retain: filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:74\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePiecesToTrash:181\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePiecesToTrash:579\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces:379\n\tstorj.io/storj/storagenode/retain.(*Service).Run.func2:265\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-08-18T21:29:03+02:00	ERROR	piecestore:cache	error getting current used space for trash: 	{"error": "filestore error: failed to walk trash namespace 7b2de9d72c2e935f1918c058caaf8ed00f0581639008707317ff1bd000000000: context canceled", "errorVerbose": "filestore error: failed to walk trash namespace 7b2de9d72c2e935f1918c058caaf8ed00f0581639008707317ff1bd000000000: context canceled\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).SpaceUsedForTrash:272\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:100\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-08-18T21:29:04+02:00	ERROR	failure during run	{"error": "piecestore monitor: timed out after 1m0s while verifying writability of storage directory", "errorVerbose": "piecestore monitor: timed out after 1m0s while verifying writability of storage directory\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func2.1:175\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func2:164\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-08-18T21:29:04+02:00	FATAL	Unrecoverable error	{"error": "piecestore monitor: timed out after 1m0s while verifying writability of storage directory", "errorVerbose": "piecestore monitor: timed out after 1m0s while verifying writability of storage directory\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func2.1:175\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func2:164\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}

Any idea ?

https://forum.storj.io/search?q=timed%20out%20after%201m0s%20while%20verifying%20writability%20order%3Alatest

1 Like