Fatal Error on my Node / Timeout after 1 min

Node Operators troubleshooting
, ,
206

any chances that you use a SMR / 2,5"drive?

1 Like
207

Hi.My node is a new one.The first 3 days was offline several times a day.I did not notice that was offline after a few minutes working.I found now this from my log.

2023-04-04T22:39:00.181+0300 FATAL Unrecoverable error {“error”: “piecestore monitor: timed out after 1m0s while verifying readability of storage directory”, “errorVerbose”: “piecestore monitor: timed out after 1m0s while verifying readability of storage directory\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func1.1:137\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func1:133\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”}

I can not understand the problem.Any help?Thanks.

208

can you write about your setup also, windows, diker?
hardrare setup? sata? usb hdd? RAM?

209

Ok i have a mini pc AMD Ryzen 7 3750H with Radeon Vega Mobile Gfx 2.30 GHz,
16GB RAM, Windows11 i run GUI node in an external SSD(USB Stick).

210

USB stick? how big is this stick? i mean 100 gb, 2tb? you got it from china?

211

Your usb seems to be not accessible during readability test hence the error. You need to check the USB cable or change USB port.

212

Lots of sno have this problem.
Only common thing so far is the filesystem ntfs.
Wich is probably spammed with small files in the data and the trash.
Set the node in windows to restart itself when the client stops. This keeps the node alive (don t know for how long)
Or downgrade to 1.74.1
Also stopping ingress via suggesting the node is full. (Capacity in yaml to used space seems to help helped me for 28h so far)

I suspect that moving to/ trash+deletion + ingress handling takes sometimes more than 1min.

They implemented an 1m timeout check wich does not work properly on windows.
I had perfect scores until the update. Now everybody tells me my hardware is to slow/broken. But it isnt.
Do drivescan if you want or if audit score is affected.

See: fatal error on my node post

213

I will ask also, did you turned off defragmentation, if not it work automatically, and make all things slower.

214

well, i have now in windows to restart,started to work for 3 minutes,and then nothing.Node is online but doing nothing.

Your Storage Node Stats

STATUS

Online

QUIC

OK

UPTIME

1h 50m

LAST CONTACT

1h 47m ago

VERSION

v1.75.2

PERIOD

April

Last contact before 1h47min.Is it normal.I try to find 1.74.1 but i can not untill now.And Vadim how to turn of defragment in windows11?I have some more HDDs 2TB.I 'll try them too and after that i 'll try Raspberry.Did any body check Raspberry4 8MB RAM?

215

write defrag in start menu, it will find it, open it, in below section will be schedule, turn it off.

216

Show last 10 lines of your log How do I check my logs? - Storj Node Operator Docs

217

This is log:

PS C:\Users\makis> Get-Content “$env:ProgramFiles/Storj/Storage Node/storagenode.log” -Tail 20 -Wait
2023-04-05T21:25:34.575+0300 INFO piecestore upload canceled {“Piece ID”: “ONU2K5RMRDFINHNG2XY2XVVH6JZ6STSYUJOUZJCCJSX
LE2SQGITQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 0, “Remote
Address”: “5.161.76.29:48410”}
2023-04-05T21:25:34.575+0300 INFO piecestore upload canceled {“Piece ID”: “SJXNUOLJUJ544GTYQD2DQXT5DOWEGRDLGTHMYGGYEBN
JHEPBRFLQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 0, “Remote
Address”: “184.104.224.98:40624”}
2023-04-05T21:25:34.576+0300 INFO piecestore upload canceled {“Piece ID”: “TSFDPDTOLUJKHLD6LG2YPPK3BF6BGPQ4ZDUFX2N2CFX
53FXGMF4Q”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 0, “Remote
Address”: “5.161.76.29:48732”}
2023-04-05T21:25:34.576+0300 INFO piecestore upload canceled {“Piece ID”: “W7N3YWO6GYMTNTOTM37TF7WNXMZJHOAFC7LSG64LKYR
O7A6C6M2A”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 0, “Remote
Address”: “216.66.40.82:63762”}
2023-04-05T21:25:34.575+0300 INFO piecestore upload canceled {“Piece ID”: “CUDM7B2L5ESFGOX4CJKBLZDOWHEDC5UJADKZGKAJ5J4
GAUBIAB7Q”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 0, “Remote
Address”: “202.61.236.143:54172”}
2023-04-05T21:25:34.575+0300 INFO piecestore upload canceled {“Piece ID”: “PQD4BJ33IBCYKNN5GBCWRVOF5LLKNLDGUXI2SIKLUFO
WLPP3PSNQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 0, “Remote
Address”: “47.184.58.67:36840”}
2023-04-05T21:25:34.575+0300 INFO piecestore upload canceled {“Piece ID”: “4VTXTTJ4ZD2WHUDHD4NJPVIY3OCQXJH6YR7IJBDEUUH
WZAOLYKTA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 0, “Remote
Address”: “5.161.110.148:45366”}
2023-04-05T21:25:34.575+0300 INFO piecestore upload canceled {“Piece ID”: “Y2ZUXMPSCLZO2UROUCTX5ZQHO23UNKP56MLGGCFQ73M
YK2K5RT3A”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 0, “Remote
Address”: “5.161.193.0:26050”}
2023-04-05T21:25:34.576+0300 INFO piecestore upload canceled {“Piece ID”: “S6SYS2HHXOXYO4A633ING53ILNMAPQF7GAK5LO3WL43
AFVPSJSYA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 0, “Remote
Address”: “195.201.29.23:20598”}
2023-04-05T21:25:34.576+0300 INFO piecestore upload canceled {“Piece ID”: “J6QNHDK7CRQ565ZG6H2XLM2OLCO7ELO6G6SQN64Z6CZ
PZONEHOAA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 0, “Remote
Address”: “216.66.40.82:63746”}
2023-04-05T21:25:34.576+0300 INFO piecestore upload canceled {“Piece ID”: “PVO5T3LK4WUUG46JSXO6Q4MGVW37QERTCOH3NC6OXQK
WGKBM3VLQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 0, “Remote
Address”: “5.161.76.29:61404”}
2023-04-05T21:25:34.576+0300 INFO piecestore upload canceled {“Piece ID”: “2YUAWZ2MUQT7AL7K3C2ZIINF7CT3IPOTHHI4BMAHOYH
UDAUIUNLA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 0, “Remote
Address”: “216.66.40.82:63744”}
2023-04-05T21:25:34.576+0300 INFO piecestore upload canceled {“Piece ID”: “YYBDBBOPA4ISZF23DILUYQHNARNFS64NSK5LWYLWWRP
22C2UKQEQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 0, “Remote
Address”: “216.66.40.82:39512”}
2023-04-05T21:25:34.577+0300 INFO piecestore uploaded {“Piece ID”: “BMOR3UJFCZRSCAQQGNCOD6GGFSAZ5BEN6NH6JZCZLIYWBC7S4U
KA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 4864, “Remote Add
ress”: “5.161.76.29:10114”}
2023-04-05T21:25:34.594+0300 INFO piecestore upload canceled {“Piece ID”: “QROTLZAC4RJQ2KLQQUB6JY73NSK3XXXC2BDEKPE7QXO
L7R36AIEA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 0, “Remote
Address”: “128.140.5.141:39826”}
2023-04-05T21:25:34.594+0300 INFO piecestore upload canceled {“Piece ID”: “EETJPLQQGDJVP7A77J4JYKSO65URFQANJBCWRQMTVAM
2E7P37B7A”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 0, “Remote
Address”: “184.104.224.99:26436”}
2023-04-05T21:25:34.594+0300 INFO piecestore upload canceled {“Piece ID”: “W2CDKA2ZNXJF5IY4CCHXH7KWB65LIYF6TNFWNBLYWK5
44SARSGAQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 0, “Remote
Address”: “216.66.40.82:10586”}
2023-04-05T21:25:34.594+0300 INFO piecestore upload canceled {“Piece ID”: “HFACR7NXN7NV2O72FYELZ6WXNYAUQYS2NKPD5KO53AH
HDUMJ2KFA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 0, “Remote
Address”: “216.66.40.83:25216”}
2023-04-05T21:25:34.599+0300 INFO piecestore uploaded {“Piece ID”: “YS2TOI7PAT7VPTIWXR3E5AZ543AUBRZXX3YR3SNJMU5GXIXBHR
HA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “PUT”, “Size”: 1536, “Remote Add
ress”: “72.52.83.202:64362”}
2023-04-05T21:25:47.147+0300 FATAL Unrecoverable error {“error”: “piecestore monitor: timed out after 1m0s while verif
ying readability of storage directory”, “errorVerbose”: “piecestore monitor: timed out after 1m0s while verifying read
ability of storage directory\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func1.1:137\n\tstorj.io/common/sync2
.(*Cycle).Run:160\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func1:133\n\tgolang.org/x/sync/errgroup.(*Group
).Go.func1:75”}

218

This is the log:

PS C:\Users\makis> Get-Content “$env:ProgramFiles/Storj/Storage Node/storagenode.log” -Tail 20 -Wait
2023-04-05T21:25:34.575+0300 INFO piecestore upload canceled {“Piece ID”: “ONU2K5RMRDFINHNG2XY2XVVH6JZ6STSYUJOUZJCCJSX
LE2SQGITQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 0, “Remote
Address”: “5.161.76.29:48410”}
2023-04-05T21:25:34.575+0300 INFO piecestore upload canceled {“Piece ID”: “SJXNUOLJUJ544GTYQD2DQXT5DOWEGRDLGTHMYGGYEBN
JHEPBRFLQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 0, “Remote
Address”: “184.104.224.98:40624”}
2023-04-05T21:25:34.576+0300 INFO piecestore upload canceled {“Piece ID”: “TSFDPDTOLUJKHLD6LG2YPPK3BF6BGPQ4ZDUFX2N2CFX
53FXGMF4Q”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 0, “Remote
Address”: “5.161.76.29:48732”}
2023-04-05T21:25:34.576+0300 INFO piecestore upload canceled {“Piece ID”: “W7N3YWO6GYMTNTOTM37TF7WNXMZJHOAFC7LSG64LKYR
O7A6C6M2A”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 0, “Remote
Address”: “216.66.40.82:63762”}
2023-04-05T21:25:34.575+0300 INFO piecestore upload canceled {“Piece ID”: “CUDM7B2L5ESFGOX4CJKBLZDOWHEDC5UJADKZGKAJ5J4
GAUBIAB7Q”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 0, “Remote
Address”: “202.61.236.143:54172”}
2023-04-05T21:25:34.575+0300 INFO piecestore upload canceled {“Piece ID”: “PQD4BJ33IBCYKNN5GBCWRVOF5LLKNLDGUXI2SIKLUFO
WLPP3PSNQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 0, “Remote
Address”: “47.184.58.67:36840”}
2023-04-05T21:25:34.575+0300 INFO piecestore upload canceled {“Piece ID”: “4VTXTTJ4ZD2WHUDHD4NJPVIY3OCQXJH6YR7IJBDEUUH
WZAOLYKTA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 0, “Remote
Address”: “5.161.110.148:45366”}
2023-04-05T21:25:34.575+0300 INFO piecestore upload canceled {“Piece ID”: “Y2ZUXMPSCLZO2UROUCTX5ZQHO23UNKP56MLGGCFQ73M
YK2K5RT3A”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 0, “Remote
Address”: “5.161.193.0:26050”}
2023-04-05T21:25:34.576+0300 INFO piecestore upload canceled {“Piece ID”: “S6SYS2HHXOXYO4A633ING53ILNMAPQF7GAK5LO3WL43
AFVPSJSYA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 0, “Remote
Address”: “195.201.29.23:20598”}
2023-04-05T21:25:34.576+0300 INFO piecestore upload canceled {“Piece ID”: “J6QNHDK7CRQ565ZG6H2XLM2OLCO7ELO6G6SQN64Z6CZ
PZONEHOAA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 0, “Remote
Address”: “216.66.40.82:63746”}
2023-04-05T21:25:34.576+0300 INFO piecestore upload canceled {“Piece ID”: “PVO5T3LK4WUUG46JSXO6Q4MGVW37QERTCOH3NC6OXQK
WGKBM3VLQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 0, “Remote
Address”: “5.161.76.29:61404”}
2023-04-05T21:25:34.576+0300 INFO piecestore upload canceled {“Piece ID”: “2YUAWZ2MUQT7AL7K3C2ZIINF7CT3IPOTHHI4BMAHOYH
UDAUIUNLA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 0, “Remote
Address”: “216.66.40.82:63744”}
2023-04-05T21:25:34.576+0300 INFO piecestore upload canceled {“Piece ID”: “YYBDBBOPA4ISZF23DILUYQHNARNFS64NSK5LWYLWWRP
22C2UKQEQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 0, “Remote
Address”: “216.66.40.82:39512”}
2023-04-05T21:25:34.577+0300 INFO piecestore uploaded {“Piece ID”: “BMOR3UJFCZRSCAQQGNCOD6GGFSAZ5BEN6NH6JZCZLIYWBC7S4U
KA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 4864, “Remote Add
ress”: “5.161.76.29:10114”}
2023-04-05T21:25:34.594+0300 INFO piecestore upload canceled {“Piece ID”: “QROTLZAC4RJQ2KLQQUB6JY73NSK3XXXC2BDEKPE7QXO
L7R36AIEA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 0, “Remote
Address”: “128.140.5.141:39826”}
2023-04-05T21:25:34.594+0300 INFO piecestore upload canceled {“Piece ID”: “EETJPLQQGDJVP7A77J4JYKSO65URFQANJBCWRQMTVAM
2E7P37B7A”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 0, “Remote
Address”: “184.104.224.99:26436”}
2023-04-05T21:25:34.594+0300 INFO piecestore upload canceled {“Piece ID”: “W2CDKA2ZNXJF5IY4CCHXH7KWB65LIYF6TNFWNBLYWK5
44SARSGAQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 0, “Remote
Address”: “216.66.40.82:10586”}
2023-04-05T21:25:34.594+0300 INFO piecestore upload canceled {“Piece ID”: “HFACR7NXN7NV2O72FYELZ6WXNYAUQYS2NKPD5KO53AH
HDUMJ2KFA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 0, “Remote
Address”: “216.66.40.83:25216”}
2023-04-05T21:25:34.599+0300 INFO piecestore uploaded {“Piece ID”: “YS2TOI7PAT7VPTIWXR3E5AZ543AUBRZXX3YR3SNJMU5GXIXBHR
HA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “PUT”, “Size”: 1536, “Remote Add
ress”: “72.52.83.202:64362”}
2023-04-05T21:25:47.147+0300 FATAL Unrecoverable error {“error”: “piecestore monitor: timed out after 1m0s while verif
ying readability of storage directory”, “errorVerbose”: “piecestore monitor: timed out after 1m0s while verifying read
ability of storage directory\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func1.1:137\n\tstorj.io/common/sync2
.(*Cycle).Run:160\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func1:133\n\tgolang.org/x/sync/errgroup.(*Group
).Go.func1:75”}

219

are you able to read the files normally in windows file explorer? Like navigate down the folders and try to grab a random blob and copy it.

I wonder if the USB drive is either stuck in a write protect mode or somehow got damaged or corrupted (unsafe shutdown, etc)

1 Like
220

if it is something like 2TB USB stick then can just throw it away, I was dummy to buy several sticks with 2tb. yes there is 2 TB inside but all you write there will be damaged. I made serface scan for it and several first bloks was OK and then all other was RED

221

Additional problem here is that once the node had this error, it does not shutdown in a clean way. I cannot stop it and not even kill it:

Error response from daemon: Could not kill running container, cannot remove - tried to kill container, but did not receive an exit event

Logs show INFO waiting for storagenode, processes-exit-eventlistener to die

I don’t think this is how it should be. Now only ways left are to try to restart docker and to hope it will clean up properly or to manually clean the containers or to reboot. That’s not good.

222

same problem here. USB hdd 12tb not SMR. win11.

223

When the node cannot be stopped this is clear indication of hardware issue, there is no other way. It’s not because of check (which is actually a regular write or read as for piece), it because of hanging. When there was no timeout this problem was just hidden, now it’s visible.

So, I can assume you experience it with docker node too. What is OS?

1 Like
224

Seems USB not performant enough. The recommendation is the same

225

Premise, I have nodes with sas and sata hdd too. No problem on that but…
Focus of the storj project is to use the excess space of individual small node operators. I think almost half of these small users may have a usb external hdd. It’s important to keep easy to be a storj op with grandma hardware. You don’t want professional operators and first advice is not to buy new hardware for storj so we need to fix crash with slow hardware. Btw this op will lost every race but storj soul will be unaffected :slight_smile:

2 Likes