Manager closed: unexpected EOF

[ptC7H12]

Hello,
a few days ago the Ingress of my Node suddenly dropped from 18GB to 10GB a day.

Now I looked into the logs and found many of these Errors:

2023-09-19T21:46:45+02:00       ERROR   piecestore      upload failed   {"Piece ID": "YPYXMPXGBSPLDYKQP7WHIZLJ6QNT2UXZ7IYPT6IJOHYLS3OPQHNA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:231", "Size": 532480, "Remote Address": "107.6.94.219:42626"}
2023-09-19T21:46:57+02:00       ERROR   piecestore      upload failed   {"Piece ID": "ZF4P322HWK2R2PSIIP5APHSLHA3YCGCD3QCSLOFFBJXIPDPGPITA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:231", "Size": 311296, "Remote Address": "107.6.94.219:47656"}
2023-09-19T21:47:30+02:00       ERROR   piecestore      upload failed   {"Piece ID": "LEPGMQLMAHNWIDVUXDR3R52V3BOQOIO7PAMDBEZ4CUUK2O34TXEA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:231", "Size": 794624, "Remote Address": "107.6.94.219:56614"}
2023-09-19T21:49:14+02:00       ERROR   piecestore      upload failed   {"Piece ID": "F6OP5EUPUSFAREFEYYA5VMUNGK4JEDVV5KBMOPQOYEGJ66I375FQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:231", "Size": 1843200, "Remote Address": "107.6.94.219:53226"}
2023-09-19T21:49:19+02:00       ERROR   piecestore      upload failed   {"Piece ID": "JDFWGPERQB24CMX3BNTEEWGVDP3I7JMURS747H4IGHBQ4DA2VA2Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 851968, "Remote Address": "72.52.83.202:43448"}
2023-09-19T21:49:27+02:00       ERROR   piecestore      upload failed   {"Piece ID": "MEBXEFTOW3WWGO7EJM3XHHG3SYKBYQVVU64QR6XIHR4BS3TEYF6Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:231", "Size": 1843200, "Remote Address": "107.6.94.219:60428"}
2023-09-19T21:53:10+02:00       ERROR   piecestore      upload failed   {"Piece ID": "H4TGFB64MIME2B65NBLRIW4KPSV4JR7U4DHV7Y2ES4PIQFSCZXFQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 655360, "Remote Address": "5.161.117.79:11242"}
2023-09-19T21:53:11+02:00       ERROR   piecestore      upload failed   {"Piece ID": "NCZI4XA74I3NL4NNVMCF2MHQ25H5GOBD4ZQIZNTXLCVWU2HT545A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:231", "Size": 1843200, "Remote Address": "107.6.94.219:50296"}
2023-09-19T21:54:19+02:00       ERROR   piecestore      upload failed   {"Piece ID": "JO53GJOJWR5GCIZ4X7DRCT2HOEWJBTIKK2QF2NWY2MVKCIAGWKDA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:231", "Size": 532480, "Remote Address": "51.81.244.151:49312"}
2023-09-19T22:02:30+02:00       ERROR   piecestore      upload failed   {"Piece ID": "3F52YEIZO4TJ53JVG2UCKGDKXFA5XZMAJR2LC47M5AOJJESXZZQA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:231", "Size": 1843200, "Remote Address": "107.6.94.219:44316"}
2023-09-19T22:07:57+02:00       ERROR   piecestore      upload failed   {"Piece ID": "JL744DHUADXJZ6PDUOPXJY6OZN5IG6VW43JJQTVJ5VVAZLNKFGTA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 131072, "Remote Address": "5.161.77.10:30558"}
2023-09-19T22:09:58+02:00       ERROR   piecestore      upload failed   {"Piece ID": "QXBOVWC6UDMULXDA4WVIWIUPPFZ334KOUSI66S32AVMBYVOL67HA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:231", "Size": 311296, "Remote Address": "51.81.244.151:36116"}
2023-09-19T22:12:30+02:00       ERROR   piecestore      upload failed   {"Piece ID": "3GK4UXBPBSTB6JIJ3UHNERN24QQNPQWWDFPPDOH4Z2PR6S7XU44Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 327680, "Remote Address": "72.52.83.202:31701"}

Can you please help me?

Regards

[pcresumen]

Hello @ptC7H12,

In the following post explain what this error that you are having means.

Error “Action”: “PUT”, “error”: “unexpected EOF”

Hi @RJGSpace Let’s break down the error line… This means the error occurred when trying to access (read/write) a file. Data was coming from the Storj network to your node When trying to save the file to your blobs folder the node received an end of file (EOF) during the save process. This means that your node was not quick enough to save the piece from the network so the upload was cancelled half way through the transfer. On it’s own this is a normal part of the way Storj operates, uplo…

[ptC7H12]

Hi,
I had found this topic, but as the ERROR is not complete the same, I thought that I should ask…

But If this is the case, why I is my Node loosing so many races from one day to another - and about 10GB/day?
Other Nodes on same Host do not have this errors (I’ve got 4 Nodes on same host 2 with this error and 2 without).

[pcresumen]

Check that your disks are CMR instead of SMR. If your disks are SMR the more they fill the free space the slower they will become and the more races you will lose.

Also check the health of the disk, that it is not degrading

If you are doing multinode, make sure each node is on a different HDD.

To also improve performance is to move the databases to an SSD disk. Below I leave you the posts that explain how to move the databases

How to move DB’s to SSD on Docker

How to move DB’s to SSD on Docker Before you beginning, please make sure that your SSD has good endurance (MLC is preferred), I personally recommend using SSD mirror. look into the official documentation and make sure that you are using –mount type=bind parameter in your docker run string Prepare a folder with mounted SSD outside of <storage-dir> from the official documentation. (it your folder with pieces) Add a new mont string to your docker run string: Now we have: docker run -d --restar…

Move databases on Windows storagenode

Hi guys. So, I troubleshooted a lot on one of my nodes, and disk activity is always at 100%. The HDD just can´t keep up with the demand (:P) it´s a WD Blue but it gets almost occupied writing in databases. Can I move the databases to my quicker system drive? If so, how?

[ptC7H12]

Thank you for your explaination, but all your points descibe my Setup Except of DBs on SSD.
Nevertheless it does not explain, why it suddenly appeard…

[pcresumen]

For a couple of months now, the average I have on my multinode host is ~10GB/day. The days when it goes up more it is because the other node is full and then the ingress is doubled from 10 to 20 GB. I attached the image of the node that has free space.

There will always be uploads that fail, there will be more days, other days less, it all depends on the speed of the other nodes.

Also keep in mind that the more nodes there are, the less income we will have, because the more will be distributed

If you have healthy discs and everything works well, I wouldn’t give it importance.

[ptC7H12]

Thank you, I will observe it and maybe move DBs to SSD and then observe more