{name: piecestore:monitor, error: piecestore monitor: timed out after 1m0s while verifying writability of storage directory, errorVerbose: piecestore monitor: timed out after 1m0s while verifying writability of storage directory\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func2.1:150\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func2:146\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75}
i have the timeout error as well. but not the graceful exit eighter. so far no solution, after restart of the service it runs again fine for some hours. disc check done, no errors. you have 100%audit too?
tought big logfile may the cause, but its obviously not. nothing changed exept set the useable space from 9 to 10 tb, hdd has 12(10.9) with 5.5 full
My Logfile was to big so iI deleted it and create a new one.
I have 15 nodes and it is the only one with the problem. I Created a Script wich restarts the complete VM of this Node when the error appear.
I saw that my HDD has only 1.65 TB in use but suddenly on the HDD is like 3TB in use.
This error mean that your disk have issues - storagenode cannot write on it.
I suggest to stop the storagenode service either from the Services applet or from the elevated PowerShell:
Stop-Service storagenode
then check the cable and power supply of your disk. If it’s an external USB disk, it must have an external power supply.
After that run in the PowerShell or Command prompt as an Administrator (assuming that you disk has letter D:):
chkdsk /f d:\
Perhaps you should run it several times until all errors will be fixed.
After that you may try to start the storagenode service either from the Services applet, or from the elevated PowerShell:
As far as I understand it’s not gone yet. If there is no errors detected, then it must be a cable or power.
If you checked them (unplug and plug again while storagenode is not running), you may run your node back and see - did it help or not.
If it did not help, you probably need to replace the cable.
I don´t think it is the Cable but I will replace it when im back home.
Just for understanding my setup:
One windows Server Ryzen 1700 with 5 internal HDDs and one m.2 SSD on it. on the SSD is the Windows System with VMWate Workstation on it. On VMWare Workstation are 5 VMS. The VM Store is on the m.2 SSD. In every VM is one HDD Mounted as E Partition. This Partition is the Data Store for the node. And on my first one I got the Problem.
Hmm I think I got the update to version v1.75.2 at the same time when the node starts to get the problem. But not sure. I Have serveral Nodes with this version and they don´t have an problem.
2023-03-28T13:25:02.622+0200 FATAL Unrecoverable error {“error”: “piecestore monitor: timed out after 1m0s while verifying writability of storage directory”, “errorVerbose”: “piecestore monitor: timed out after 1m0s while verifying writability of storage directory\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func2.1:150\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func2:146\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”}
since my setup is kompletely different it has to be something else… but my node goes down slowly and no cure in sight. restarting the service after 1min is now configured.
none of the drives are full or missbehave. no screeching, just usage stopps. when the runner times out.
can use them normaly for data, no bsod, no errors from windows. drive is exclusive for storj atm.
tested my bigabyte brix last summer with high cpu load and ram tests. environment and temps normal. its just the runner who kills the background service.
Possibly, but at the moment I don’t have a lot of corresponding data from other node operators to determine if the issue is widespread or not. This error has been seen in the past, and it is usually because drives are full or have errors. Your other nodes are not having this issue, correct?