Start getting an error on one node few days ago manager closed: unexpected EOF", "**error**Verbose": "manager closed: unexpected EOF

Node Operators troubleshooting
1

Hi

Start receiving some error and audit degradation

grep error:

|

2024-05-26T21:49:05Z|ERROR|piecestore|upload failed|{Process: storagenode, Piece ID: PMEISDSU66P7KH2M5T5AEW4ZZERS4YZDL4XAL4WUXM67JFYT5YQA, Satellite ID: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE, Action: PUT, Remote Address: 79.127.223.130:50492, Size: 851968, error: manager closed: unexpected EOF, errorVerbose: manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:229}|
|---|---|---|---|---|
|2024-05-26T21:49:10Z|ERROR|piecestore|upload failed|{Process: storagenode, Piece ID: OYOHBKGQU6LMAER2P3ADYMWPYUOPPPD3EZL5GUCFTTCW5L5WAFIQ, Satellite ID: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE, Action: PUT, Remote Address: 79.127.205.229:59432, Size: 589824, error: manager closed: unexpected EOF, errorVerbose: manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:229}|
|2024-05-26T21:49:20Z|WARN|contact:service|Your node is still considered to be online but encountered an error.|{Process: storagenode, Satellite ID: 121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6, Error: contact: failed to ping storage node using QUIC, your node indicated error code: 0, rpc: quic: timeout: no recent network activity}|
|2024-05-26T21:49:29Z|WARN|console:service|unable to get Satellite URL|{Process: storagenode, Satellite ID: 12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo, error: console: trust: satellite is untrusted, errorVerbose: console: trust: satellite is untrusted\n\tstorj.io/storj/storagenode/trust.init:29\n\truntime.doInit1:6740\n\truntime.doInit:6707\n\truntime.main:249}|
|2024-05-26T21:49:29Z|WARN|console:service|unable to get Satellite URL|{Process: storagenode, Satellite ID: 12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB, error: console: trust: satellite is untrusted, errorVerbose: console: trust: satellite is untrusted\n\tstorj.io/storj/storagenode/trust.init:29\n\truntime.doInit1:6740\n\truntime.doInit:6707\n\truntime.main:249}|
|2024-05-26T21:49:30Z|ERROR|piecestore|upload failed|{Process: storagenode, Piece ID: E2R35A2ROOVHJOGFMIHJJXZK36RCDHJBKLQK52DXLRJJCCGJSI3A, Satellite ID: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE, Action: PUT, Remote Address: 109.61.92.72:60414, Size: 458752, error: manager closed: unexpected EOF, errorVerbose: manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:229}|
|2024-05-26T21:49:33Z|ERROR|piecestore|download failed|{Process: storagenode, Piece ID: 77SMFVOPWPDVVCHBM6QPAZ3P2EZHSX7SYPM57MBBABGCHPQKLG6A, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: GET, Offset: 0, Size: 2319104, Remote Address: 79.127.219.43:43484, error: write tcp 172.17.0.3:28967->79.127.219.43:43484: use of closed network connection, errorVerbose: write tcp 172.17.0.3:28967->79.127.219.43:43484: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:409\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:470\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData.func1:863\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22}|
|2024-05-26T21:49:43Z|ERROR|piecestore|upload failed|{Process: storagenode, Piece ID: NU52G7OPYHYH4TIHHZKVQBDA76NGKGRXHZBTRX3NR36FRJVVPHIQ, Satellite ID: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE, Action: PUT, Remote Address: 109.61.92.70:56210, Size: 1966080, error: manager closed: unexpected EOF, errorVerbose: manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:229}|
|2024-05-26T21:49:50Z|ERROR|piecestore|upload failed|{Process: storagenode, Piece ID: PC75IXI2XVPELQ5UGZGRK45I7FBBPD3J3C4AHYASWCXVKZCAJUAA, Satellite ID: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE, Action: PUT, Remote Address: 79.127.219.43:42978, Size: 1245184, error: manager closed: unexpected EOF, errorVerbose: manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:229}|
|2024-05-26T21:49:59Z|ERROR|piecestore|upload failed|{Process: storagenode, Piece ID: 2UVAYCYHRMLPZ5BGRJNEJJNZZQFLFYN433FL4GO2AVY3DJUTJOAQ, Satellite ID: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE, Action: PUT, Remote Address: 79.127.201.213:46620, Size: 720896, error: manager closed: unexpected EOF, errorVerbose: manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:229}|
|2024-05-26T21:50:10Z|ERROR|piecestore|upload failed|{Process: storagenode, Piece ID: TFA5BP5F74PCCHGYIU3ULKYMREM5QP64OJGT5KL6O4CB7OUV4WZA, Satellite ID: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE, Action: PUT, Remote Address: 109.61.92.81:33008, Size: 1769472, error: manager closed: unexpected EOF, errorVerbose: manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:229}|
|2024-05-26T21:50:27Z|ERROR|piecestore|upload failed|{Process: storagenode, Piece ID: O23NJPXYH4NCTXOPNEOWWIGZ3QSBK36PGJRVR2DQRZW6EDXRTK6Q, Satellite ID: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE, Action: PUT, Remote Address: 79.127.205.233:43616, Size: 1376256, error: manager closed: unexpected EOF, errorVerbose: manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:229}|

Снимок экрана 2024-05-27 в 00.58.40
Снимок экрана 2024-05-27 в 00.58.401710×646 46.4 KB

Node id 12V1TJKCePWYMpecgPy52EQMm9f6nknDGufjMLdbGuuYEHMPrHp

It seems that the node are not fast enough but actually didn’t know.

Any ideas how to solve it?

Also there is a problem with QUIC Misconfigured asset already about it and the results was in the linux issue. also don’t know how to solve it. it happed few month ago and present till now.

2

This is a usual long tail cancelation - your node is too slow for these customers, so they cancels transfers to it.
You cannot be close to everyone customer in the world, so this is normal.

However, this also could be a result of the overloaded router and it drops connections.

3

Just leaves the question why it’s an error.

1 Like
4

I’m mainly worried about the audit. don’t know why it goes down. I have other logs, external ones. there are frequent interruptions of the open port. but I can’t figure out whether the problem is with the internet or the node itself.

5

Do you have a DDOS protection in your router? What brand is it BTW?

Any other satellite errors in the log?

1 Like
6

Не ростелеком провайдер? Началось такое же с 21 мая.

7

Actually it is difficult to ddos asus ax89x. But who knows. Beeline :honeybee: provider

8

У меня начались проблемы только с us1 спутником. Любая попытка связи с ним заканчивалась tcp reset. Ну и постоянно горящий красным quic. Победил только вчера, причина в провайдере у меня была.

9

Вот такая пика выросла как данные пошли.

изображение
изображение822×399 11.2 KB

10

It’s not, but aside from that: ASUS had implemented your problem:

1000023635
10000236351920×916 113 KB

See firewall page.
Don’t enable that DoS prevention. Otherwise your online score will go down, because the frequency of polls is being seen as DDoS.

I’m not, because it’s 100% as you can see. I’m worried about your online score. Voor that will be solved of you change your router settings.

1 Like
11

That’s the peak of the test from STORJ and update to the newest version. Hasn’t to do with anything else on your side.

I see the same spike, without any changes at my side:

1000023637
1000023637969×904 72.3 KB

Can’t be, his online score is 78. Meaning he lost 22%. Since you lose at maximum 3.03% a day of your online score, the problem must have existed before 20th of May.
Besides, the date you mention is as far I know also the starting date of the network testing with test data.

1 Like
12

I very doubt it, that would mean your provider would block UDP. Don’t think so.

1 Like
13

Just because the cancelation may happen in any moment and depending on where the code was executing would produce a different error stacks.
We used a graceful messages from the libuplink in the past, but to speedup a process we decided to remove that, so libuplink do not spent time on notification of each node about a cancelation and just closes the connection ASAP.

14

Yeah, that’s fine. But probably hasn’t to turn up as an error on the SNO-side, because it isn’t one; just working as intended, must be, …

15

In fact, knowing this provider, it is quite possible (it is similar to the monopolist Comcast in the US, but also its majority stake is in the hands of the government).

16

it seams it help … thx :+1:

1 Like