Fatal Error on my Node

[Alexey]

@IgRost

2023-11-08T16:40:19+03:00	FATAL	Unrecoverable error	{"error": "piecestore monitor: timed out after 1m0s while verifying writability of storage directory", "errorVerbose": "piecestore monitor: timed out after 1m0s while verifying writability of storage directory\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func2.1:176\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func2:165\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}

Fatal Error on my Node

Please search for fatal errors, not upload/download errors. this is a culprit, you need to increase this timeout see

[gnegnus]

Hello,
I run a new node from Italy since 12 october 2023.
Everything was going fine since 3/4 days ago.

Now every morning my node shutdown with this fatal error.

2023-11-22T08:14:33+01:00 ERROR piecestore upload failed {“Piece ID”: “TMXS24UQQT2LX4G42G5QWC6C7BT2CMBKIBQRXSM6VJHKUCYKNHSQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “manager closed: unexpected EOF”, “errorVerbose”: “manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226”, “Size”: 262144, “Remote Address”: “5.161.143.41:55470”}
2023-11-22T08:17:24+01:00 ERROR services unexpected shutdown of a runner {“name”: “piecestore:monitor”, “error”: “piecestore monitor: timed out after 1m0s while verifying writability of storage directory”, “errorVerbose”: “piecestore monitor: timed out after 1m0s while verifying writability of storage directory\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func2.1:176\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func2:165\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”}
2023-11-22T08:17:35+01:00 FATAL Unrecoverable error {“error”: “piecestore monitor: timed out after 1m0s while verifying writability of storage directory”, “errorVerbose”: “piecestore monitor: timed out after 1m0s while verifying writability of storage directory\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func2.1:176\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func2:165\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”}

I’ve setup storagenode service to restart automatically and now I’m going offline because after restart everything works fine.

I’ve an IBM Microserver Gen8 with 4x16TB in RAID 5.
OS: VMWare ESXi 7 running only one VM - Windows Server 2022 that is hosting Storj.
The file data are in a VMDK disk inside the RAID5 array.

I’m hosting only 1.19 TB of data

Any suggestion?
Thanks in advance to all.

[Alexey]

@gnegnus

ERROR services unexpected shutdown of a runner {“name”: “piecestore:monitor”, “error”: “piecestore monitor: timed out after 1m0s while verifying writability of storage directory”, “errorVerbose”: “piecestore monitor: timed out after 1m0s while verifying writability of storage directory

Fatal Error on my Node

Please search for fatal errors, not upload/download errors. this is a culprit, you need to increase this timeout see

[a.j.thiart]

Hi all. My node doesn’t stay online. I constantly have to restart it. What can I change in the config file to have it stay online? Please help. My satellites are getting suspended

[tylkomat]

The error message suggests that your storage directory is not writable. Therefore the node stops working. Where is your storage directory located? Is it a local hard drive, connected via USB or something else?

[daki82]

Easy, read here

Fatal Error on my Node

Hello Everyone, a week ago i got several errors on my main node. Maybe someone can help me to solve the problem. The Node runs on Windows 10. Thanks for your Help. 2023-03-27T05:11:25.897+0200 INFO piecestore download started {Piece ID: 7WMQ364BMF672RLAQWDD76IQ6EORNXMYXIL5PGN46I4VUT4P4JXA, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: GET, Offset: 110336, Size: 469504, Remote Address: 184.104.224.98:48702} 2023-03-27T05:11:25.916+0200 ERROR services unexp…

@Alexey i like to move it move-it

And here…

Windows Storj node best practices Wiki

i had recently problems, so i configured the windows service to restart after 2minutes more than the checking interval of uptimerobot, so it get the downtime registered when sleeping and missing the alarm. eg: restart after 23 min and uptimerobot checks every 21 min. so i could minimize the data loss after 4h offline. automatic updates are not affected by this, an continue normaly. (restart service after 1min maybe to short)

[Alexey]

Fatal Error on my Node

Please search for fatal errors, not upload/download errors. this is a culprit, you need to increase this timeout see

[thelastspark]

Hi all,

I have a ~7tb node that’s been running without issue, but recently started to have the following entries in the log:

2023-12-19T07:44:40-05:00	ERROR	services	unexpected shutdown of a runner	{"name": "piecestore:monitor", "error": "piecestore monitor: timed out after 1m0s while verifying writability of storage directory", "errorVerbose": "piecestore monitor: timed out after 1m0s while verifying writability of storage directory\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func2.1:176\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func2:165\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-12-19T07:44:40-05:00	ERROR	gracefulexit:chore	error retrieving satellites.	{"error": "satellitesdb: context canceled", "errorVerbose": "satellitesdb: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*satellitesDB).ListGracefulExits:192\n\tstorj.io/storj/storagenode/gracefulexit.(*Service).ListPendingExits:59\n\tstorj.io/storj/storagenode/gracefulexit.(*Chore).AddMissing:58\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tstorj.io/storj/storagenode/gracefulexit.(*Chore).Run:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-12-19T07:45:07-05:00	ERROR	piecestore	error sending hash and order limit	{"error": "context canceled"}
2023-12-19T07:45:09-05:00	FATAL	Unrecoverable error	{"error": "piecestore monitor: timed out after 1m0s while verifying writability of storage directory", "errorVerbose": "piecestore monitor: timed out after 1m0s while verifying writability of storage directory\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func2.1:176\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func2:165\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}

2023-12-19T08:27:11-05:00	ERROR	services	unexpected shutdown of a runner	{"name": "piecestore:monitor", "error": "piecestore monitor: timed out after 1m0s while verifying writability of storage directory", "errorVerbose": "piecestore monitor: timed out after 1m0s while verifying writability of storage directory\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func2.1:176\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func2:165\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-12-19T08:27:59-05:00	ERROR	piecestore	error sending hash and order limit	{"error": "context canceled"}
2023-12-19T08:27:59-05:00	ERROR	piecestore	error sending hash and order limit	{"error": "context canceled"}
2023-12-19T08:28:01-05:00	FATAL	Unrecoverable error	{"error": "piecestore monitor: timed out after 1m0s while verifying writability of storage directory", "errorVerbose": "piecestore monitor: timed out after 1m0s while verifying writability of storage directory\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func2.1:176\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func2:165\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}

It seems to keep working after, one time I needed to restart my machine to fix it - not sure what could be causing this?

P.S.
Here is the crystal disl info of the drive

[Roxor]

If you still have free space… did the drive go read-only? Your Windows Event Log may show something.

[thelastspark]

it’s a windows 10 machine - where/which log should I check? And yea, there’s still enough free space on it:

[Roxor]

I’d open Event Viewer, and look in the Custom Views → Administrative Event, and Windows Logs → System logs.

Those logs are often full of crap that you don’t really have to worry about: but maybe the System one will show errors related to your disk?

[thelastspark]

Honestly I didn’t see anything there that would indicate the disk being involved…

[arrogantrabbit]

Amount of disk IO in the resource monitor? Disk response latency when this happens?

Looks like your disk can’t keep up when IO slightly increases. Look into PrimoCache.

Also run check disk to verify filesystem integrity.

[thelastspark]

which flags should I run for the checkdisk? It’s a standalone drive just for the node, not the filesystem.

The IO load seems to be fine?

[daki82]

Its related to slow disk subsystem or fragmentation.

Did you ever defrag the node?
Also various solutions here:

Fatal Error on my Node

Hello Everyone, a week ago i got several errors on my main node. Maybe someone can help me to solve the problem. The Node runs on Windows 10. Thanks for your Help. 2023-03-27T05:11:25.897+0200 INFO piecestore download started {Piece ID: 7WMQ364BMF672RLAQWDD76IQ6EORNXMYXIL5PGN46I4VUT4P4JXA, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: GET, Offset: 110336, Size: 469504, Remote Address: 184.104.224.98:48702} 2023-03-27T05:11:25.916+0200 ERROR services unexp…

Locks like you defrag the D drive atm.

[arrogantrabbit]

@thelastspark

The IO load seems to be fine?

Looks fine right now. But you are not seeing issues right now, right? You may want to monitor IOPs and latency on the disk long term (e.g. with performance monitor) and when the issue reproduces review what was the load at the time issue has occurred.

Because your disk can service very small number of IOPs (around 200) it will be absolutely fine under that and faceplant when load goes over. Storagenode load is variable.

You may want to move databases to a separate drive, ideally SSD, if you haven’t done so yet. This will remove massive amount of IOPs.

@thelastspark

which flags should I run for the checkdisk?

Stop the node and

chkdsk z: /f

@daki82

Did you ever defrag the node?

This produces a very small improvement if any, and is not worth wasting time doing.

[daki82]

@arrogantrabbit

This produces a very small improvement if any, and is not worth wasting time doing.

Any proof for that? And why do some people here recommend it?
NTFS with nodes mess this up over time, pushing seek time up, when walking the millions of files.

[thelastspark]

I am defragging a different drive atm, but they are completely independent. And my cpu usage is below 25%

[thelastspark]

The databases are on a different SSD, so the drives only deal with the data itself. I’ll run the check disk command now though

[daki82]

Ok, if its without errors, maybe consider:

-Defragmentation (or just do an analyse to see how bad it is)
-increasing the timeouts (had to do that too at around 7TB)