Troubleshoot yellow warning for suspension score

zeyacry · November 12, 2020, 1:53pm

I would like to know how to troubleshoot my suspension score issue.

I am running on Raspberry pi 4 4GB.

deathlessdd · November 12, 2020, 2:04pm

You should check your logs and see if you see anything having issues.

zeyacry · November 12, 2020, 2:17pm

Do you think this is the cause of the issue?

2020-11-12T14:12:37.545Z WARN ordersfilestore Corrupted order detected in orders file {“error”: “ordersfile corrupt entry: ordersfile: unexpected EOF”, “errorVerbose”: “ordersfile corrupt entry: ordersfile: unexpected EOF\n\tstorj.io/storj/storagenode/orders/ordersfile.(*fileV0).ReadOne:92\n\tstorj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite.func1:239\n\tpath/filepath.walk:360\n\tpath/filepath.walk:384\n\tpath/filepath.Walk:406\n\tstorj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite:193\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrdersFromFileStore:389\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders:183\n\tstorj.io/storj/storagenode/orders.(*Service).Run.func1:134\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2020-11-12T14:12:37.634Z WARN ordersfilestore Corrupted order detected in orders file {“error”: “ordersfile corrupt entry: proto: pb.OrderLimit: illegal tag 0 (wire type 0)”, “errorVerbose”: “ordersfile corrupt entry: proto: pb.OrderLimit: illegal tag 0 (wire type 0)\n\tstorj.io/storj/storagenode/orders/ordersfile.(*fileV0).ReadOne:98\n\tstorj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite.func1:239\n\tpath/filepath.walk:360\n\tpath/filepath.walk:384\n\tpath/filepath.Walk:406\n\tstorj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite:193\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrdersFromFileStore:389\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders:183\n\tstorj.io/storj/storagenode/orders.(*Service).Run.func1:134\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
panic: runtime error: makeslice: len out of range [recovered]
panic: runtime error: makeslice: len out of range [recovered]
panic: runtime error: makeslice: len out of range [recovered]
panic: runtime error: makeslice: len out of range

goroutine 963 [running]:
github.com/spacemonkeygo/monkit/v3.newSpan.func1(0x0)
/go/pkg/mod/github.com/spacemonkeygo/monkit/v3@v3.0.7-0.20200515175308-072401d8c752/ctx.go:147 +0x2e0
panic(0x8b0378, 0xa65400)
/usr/local/go/src/runtime/panic.go:969 +0x118
github.com/spacemonkeygo/monkit/v3.newSpan.func1(0x0)
/go/pkg/mod/github.com/spacemonkeygo/monkit/v3@v3.0.7-0.20200515175308-072401d8c752/ctx.go:147 +0x2e0
panic(0x8b0378, 0xa65400)
/usr/local/go/src/runtime/panic.go:975 +0x3c4
github.com/spacemonkeygo/monkit/v3.newSpan.func1(0x230e0d8)
/go/pkg/mod/github.com/spacemonkeygo/monkit/v3@v3.0.7-0.20200515175308-072401d8c752/ctx.go:147 +0x2e0
panic(0x8b0378, 0xa65400)
/usr/local/go/src/runtime/panic.go:975 +0x3c4
storj.io/storj/storagenode/orders/ordersfile.(*fileV0).ReadOne(0x20ee740, 0x0, 0x0, 0x0)
/go/src/storj.io/storj/storagenode/orders/ordersfile/v0.go:89 +0xe8
storj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite.func1(0x222ed20, 0x6a, 0xa7c2a0, 0x22f8b40, 0x0, 0x0, 0x0, 0x0)
/go/src/storj.io/storj/storagenode/orders/store.go:239 +0x40c
path/filepath.walk(0x222ed20, 0x6a, 0xa7c2a0, 0x22f8b40, 0x21f5bec, 0x0, 0x0)
/usr/local/go/src/path/filepath/path.go:360 +0x2fc
path/filepath.walk(0x220d4c0, 0x14, 0xa7c2a0, 0x2514ab0, 0x21f5bec, 0x0, 0x677be0)
/usr/local/go/src/path/filepath/path.go:384 +0x204
path/filepath.Walk(0x220d4c0, 0x14, 0x24a1bec, 0x0, 0x2438550)
/usr/local/go/src/path/filepath/path.go:406 +0xe8
storj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite(0x2224840, 0xa7a4d0, 0x22b2180, 0x2818c991, 0xbfe36e95, 0x19702ee4, 0x6, 0x106ac80, 0x24be5e0, 0xa6a0fc8c, …)
/go/src/storj.io/storj/storagenode/orders/store.go:193 +0x198
storj.io/storj/storagenode/orders.(*Service).sendOrdersFromFileStore(0x2227c20, 0xa7a4d0, 0x22b2000, 0x2818c991, 0xbfe36e95, 0x19702ee4, 0x6, 0x106ac80)
/go/src/storj.io/storj/storagenode/orders/service.go:389 +0x314
storj.io/storj/storagenode/orders.(*Service).SendOrders(0x2227c20, 0xa7a650, 0x20106a0, 0x2818c991, 0xbfe36e95, 0x19702ee4, 0x6, 0x106ac80)
/go/src/storj.io/storj/storagenode/orders/service.go:183 +0x13c
storj.io/storj/storagenode/orders.(*Service).Run.func1(0xa7a650, 0x20106a0, 0xa7a650, 0x20106a0)
/go/src/storj.io/storj/storagenode/orders/service.go:134 +0x84
storj.io/common/sync2.(*Cycle).Run(0x22013e0, 0xa7a4d0, 0x20a0ea0, 0x2010480, 0x0, 0x0)
/go/pkg/mod/storj.io/common@v0.0.0-20201014090530-c4af8e54d5c4/sync2/cycle.go:92 +0x134
storj.io/common/sync2.(*Cycle).Start.func1(0x9afb8c, 0x0)
/go/pkg/mod/storj.io/common@v0.0.0-20201014090530-c4af8e54d5c4/sync2/cycle.go:71 +0x34
The Go Programming Language(0x248a4b0, 0x2312ae0)
/go/pkg/mod/golang.org/x/sync@v0.0.0-20200625203802-6e8e738ad208/errgroup/errgroup.go:57 +0x50
created by The Go Programming Language
/go/pkg/mod/golang.org/x/sync@v0.0.0-20200625203802-6e8e738ad208/errgroup/errgroup.go:54 +0x50

zeyacry · November 12, 2020, 2:20pm

My node is currently v1.16.1.

deathlessdd · November 12, 2020, 2:23pm

Is your hard drive full? There has been quite a few having the same issues you can search the error.

zeyacry · November 12, 2020, 2:46pm

I think the storage is ok.

pi@raspberrypi:~ $ sudo df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda 7.3T 1.2T 5.7T 18% /mnt/storj2

pi@raspberrypi:~ $ free -h
total used free shared buff/cache available
Mem: 3.7Gi 347Mi 2.5Gi 202Mi 869Mi 3.1Gi
Swap: 99Mi 0B

Pac · November 12, 2020, 5:30pm

What does top report? Is your system overloaded with regards to iowait and/or cpu load?

Alexey · November 12, 2020, 10:00pm

Please, search for errors related to GET_AUDIT:

Error with orders is not good too, but it will not affect the suspension score.

zeyacry · November 13, 2020, 3:04pm

I do run other background tasks on my rpi. But I think this should be normal.

zeyacry · November 13, 2020, 3:05pm

I didn’t get anything.

zeyacry · November 13, 2020, 3:12pm

I realised that the uptime never exceed 30s. When i run the following command, it automatically exits.

sudo docker exec -it storagenode /app/dashboard.sh --color

peem · November 13, 2020, 4:59pm

What is your command output

docker ps

peem · November 13, 2020, 5:10pm

This error requires action, you need the help of a professional like @Alexey

Pac · November 13, 2020, 8:15pm

If you run other things then I couldn’t tell. But on my Pi dedicated to Storj, a “usual” load-average these days is around 0.60~0.90, and the io/wait is around 0~10.

Your values are way higher but it might be normal depending on what other things are running.
Could be worth checking your disks are not struggling (this said, when a disk really struggles to a point where everything crumbles down, the load-average usually skyrockets which isn’t your case).

Alexey · November 13, 2020, 8:49pm

Please, search for kills

journalctl | grep "shim reaped"

And also monitor the load

docker stats

zeyacry · November 14, 2020, 1:04am

The status always reset within seconds.

There are a list of entry which spans over 6 hours.

zeyacry · November 14, 2020, 1:54am

@Alexey Do you think if it helps by removing the orders older than 48 hours?

Alexey · November 14, 2020, 4:31am

Yes, of course, they will not be accepted anyway. Also, use a workaround to find and remove broken orders.

zeyacry · December 5, 2020, 10:07am

@deathlessdd @Pac @Alexey @peem

Thanks for giving your insights on this issue. My node is working normally after updated to version 1.7.4.

Cheers!

f2hex · June 13, 2022, 5:24pm

I was suspended with no audit errors (checked for GET_AUDIT failures) and no other relevant errors in the logs. The system runs perfectly (27% docker loads), system and kernel logs are clean.
However the dashboard reports Audit yellow warnings and the node was suspended.
Any hints?
Thanks