Suspension Score Drop - No Failed Audits

thelastspark · November 17, 2023, 2:00pm

Hi all,

I have a fairly new node that has near 100% uptime, but a reported drop in suspension score as seen here:

[Album] imgur.com

I checked my logs as recommended here: https://support.storj.io/hc/en-us/articles/360042257912-Suspension-mode

There weren’t any lines with AUDIT nor GET_REPAIR, the only ones were PUT_REPAIR

[Album] imgur.com

2023-10-21T02:58:07-04:00	ERROR	piecestore	upload failed	{"Piece ID": "WC73WKQDLOTUTWPIGCTCRVU576JQJJND3JKXXEEZORLGJ4Y6VDAA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 524288, "Remote Address": "23.237.191.146:37480"}
2023-10-21T02:58:08-04:00	ERROR	piecestore	upload failed	{"Piece ID": "UFO63ZVAK33JK6X5HDXSODPLV46P2FZU6Q62BUK6Z4F7FNQZRAPQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT_REPAIR", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:231", "Size": 1310720, "Remote Address": "23.88.105.177:42748"}
2023-10-21T02:59:17-04:00	ERROR	piecestore	upload failed	{"Piece ID": "7XHFTR6LGT7DHQJ4GMFZEE2T7GKADZTHDGQ7THGY3QJ7CRVGOOTQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:231", "Size": 1318912, "Remote Address": "18.132.35.254:36014"}
2023-10-21T03:02:59-04:00	ERROR	piecestore	upload failed	{"Piece ID": "3IVEQXQWG3EKNCRBPTGL6UMVWWDF353W42JSIZCWVY5GA72PDBVQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 1572864, "Remote Address": "50.7.230.34:56490"}
2023-10-21T03:03:49-04:00	ERROR	piecestore	upload failed	{"Piece ID": "DWRH7PBIGRPMAJUHCR5JV4TVSQY5MAMVLTYIQC3HABJRWNVZC5ZQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:231", "Size": 1056768, "Remote Address": "13.229.75.169:47190"}
2023-10-21T03:12:30-04:00	ERROR	piecestore	upload failed	{"Piece ID": "SSRV6OX62UA4DPBDKQA7IHDLFY3OVRX273NESC255QHJCMZXR6GQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 65536, "Remote Address": "5.161.192.57:22672"}
2023-10-21T03:17:55-04:00	ERROR	piecestore	upload failed	{"Piece ID": "J6UYOHPBXTJODCW6XP4QRA23PZE2HXNM574RFSN7PCUQG2TI4ZGQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 262144, "Remote Address": "5.161.192.57:59940"}
2023-10-21T03:20:59-04:00	ERROR	piecestore	upload failed	{"Piece ID": "4KZFDUTCD3RQWPDBGWHTTESMPZZLLZVVI44Q4K2TLA7ABGXUCB6A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 196608, "Remote Address": "5.161.192.57:52786"}
2023-10-21T03:21:38-04:00	ERROR	piecestore	upload failed	{"Piece ID": "QWFBMAT65OFVPBPTEYYYFU6PWVDOP75BTKXBJ3FDVBNQSYXM2A5Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:231", "Size": 1056768, "Remote Address": "18.132.35.254:37004"}
2023-10-21T03:26:35-04:00	ERROR	piecestore	upload failed	{"Piece ID": "3CKE7XLMMYUDIWQTQ5KNYHUKH4K5NHV4USV7YPNVLSK27GTR4NMQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:231", "Size": 1318912, "Remote Address": "18.132.35.254:49068"}
2023-10-21T03:28:36-04:00	ERROR	piecestore	upload failed	{"Piece ID": "MF3LKN3TRDHBXQN4T24KXUDU5I3MWR6X3RVYYAUQCY7DTSO5ZHCQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 393216, "Remote Address": "5.161.77.10:16240"}
2023-10-21T03:28:36-04:00	ERROR	piecestore	upload failed	{"Piece ID": "D2LY4KXSIB72JLPNN563OO7UW4I3EOYKMV4IW6K54BCYZCRK4TDQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 262144, "Remote Address": "5.161.143.41:35692"}
2023-10-21T03:32:40-04:00	ERROR	piecestore	upload failed	{"Piece ID": "JVPT5XZT3KSAAQVSUDBABGIZ3DNIBTYMXTGITTXVZ5IQHJUICT7Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 2031616, "Remote Address": "64.46.111.190:33606"}
2023-10-21T03:32:52-04:00	ERROR	piecestore	upload failed	{"Piece ID": "3ZBMNNUSBYID5AHA26ZCKPLARKL2GR32F47KI6TFYOX3HT23FCPQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:231", "Size": 1056768, "Remote Address": "18.132.35.254:54194"}
2023-10-21T03:35:52-04:00	ERROR	piecestore	upload failed	{"Piece ID": "AFKQ4GDLRTZ7OXCLLWAFFI5YCVWZ3FQPHAKUBYC3YZA6YWJDHUOQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 1769472, "Remote Address": "116.202.211.83:55636"}
202

Any advice what to look into/double check?

System Specs:
OS: Win10
RAM: 32gb ddr3
HDD: Seagate 16TB HDD Exos X16 (ST16000NM001G)
CPU: AMD A10-7860K Radeon R7

Knowledge · November 17, 2023, 2:42pm

Those are normal errors when your node loses the race for the file. This wouldn’t impact your score. You are dealing with just the “Online” portion which means your node was offline at some point. Perhaps your ISP was performing maintenance and you didn’t notice your connection was down. Assuming it is rising and not falling, you should recover fairly quickly. If it is still falling, than you have some kind of connectivity issues.

daki82 · November 17, 2023, 3:41pm

any FATAL errors?

sometimes its just an bad dyndns.

thelastspark · November 17, 2023, 4:50pm

The online part was me taking it down to do some of my own investigation so it doesn’t just plummet in case it was something simple like the drive cable got loose… so the online will go back up for sure

thelastspark · November 17, 2023, 4:51pm

I have File Walker disabled, but GC hitting it? Maybe… did your suspension score go back up?

thelastspark · November 17, 2023, 4:52pm

Nope, the only errors are basically what you see in that snippit, really just my node loosing races (its on a wifi so it’s expected)

Alexey · November 18, 2023, 3:26am

Perhaps it was a temporary issue for GET_AUDIT or GET_REPAIR. However, you should see it in the logs, unless they are deleted from time to time.
If your node passes audits, the suspension score should grow.
I do not think that PUT_REPAIR can affect an suspension score.
Try to use PowerShell:

sls "get_audit|get_repair" "C:\Program Files\Storj\Storage Node\storagenode.log" | sls "error|failed" | select -last 10

thelastspark · November 18, 2023, 5:16pm

Gave that a shot, nothing gets listed so I don’t think it’s that

Alexey · November 19, 2023, 3:38am

Has the suspension score increased?

thelastspark · November 24, 2023, 1:34am

Thanks for checking up - and yea the score improved. However, I regularly get alerts that my score keeps changing, you can see it in my emails here:

Yet they almost always recover shortly after that… so now I’m confused what could be causing this…

Alexey · November 24, 2023, 4:12am

They always caused by response on audit request with unknown error (i.e. it’s at least online).
Known errors are:

file not found, i/o error, etc.
piece is corrupted (this is not showing in the logs)
3x timeout by 5 min each for the same piece

All known errors affects the audit score, otherwise it will affect a suspension score instead.

daki82 · November 24, 2023, 3:35pm

Maybe the mainboard driver, can be updated or even bios.(general advice)

Did you check for fragmentation?
Maybe if its an iops problem move databases to an spare ssd?

Wich is the system drive?

thelastspark · November 27, 2023, 7:32pm

If a piece is corrupted, it won’t say it found a corrupted piece?

I’m sure my files are fine since it always bounces back so maybe iops… I doubt it though since I have all the DB’s on a seperate SSD, and there are 3 nodes (1 drive each) behind 1 IP so the load is split

thelastspark · November 27, 2023, 7:33pm

The system drive (win10) is an ssd, the dbs are on SSD while each of the nodes has its own HDD. 3 nodes behind 1 ip.

daki82 · November 27, 2023, 8:31pm

Ok. What are the other 2 drives and how are they all connected to the pc?

Alexey · November 28, 2023, 3:22am

no, the satellite does not report the result of checking each piece. But the audit score will be affected. Since it’s not affected, but suspension score is - your node answered on audit request with an unknown error.

thelastspark · November 29, 2023, 5:48pm

All are internal, sata connection so all are treated exactly the same

thelastspark · November 29, 2023, 5:48pm

no way to know what that error is? other than sls "get_audit|get_repair" "C:\Program Files\Storj\Storage Node\storagenode.log" | sls "error|failed" | select -last 10 which didn’t return anything…

Alexey · November 30, 2023, 5:15am

Only to check it with the previous log copy. If your log is located in this path and you didn’t delete the previous one, it should have some errors or warnings related to audits.
Please try also

sls "get_audit|get_repair" "C:\Program Files\Storj\Storage Node\storagenode.log" | sls "database|i/o" | select -last 10

daki82 · November 30, 2023, 11:13am

Im not sure if the Motherboard plays a role there, but even if, it would be only on cheap models.