Troubleshoot yellow warning for suspension score

I would like to know how to troubleshoot my suspension score issue.

I am running on Raspberry pi 4 4GB.

You should check your logs and see if you see anything having issues.

Do you think this is the cause of the issue?

2020-11-12T14:12:37.545Z WARN ordersfilestore Corrupted order detected in orders file {“error”: “ordersfile corrupt entry: ordersfile: unexpected EOF”, “errorVerbose”: “ordersfile corrupt entry: ordersfile: unexpected EOF\n\*fileV0).ReadOne:92\n\*FileStore).ListUnsentBySatellite.func1:239\n\tpath/filepath.walk:360\n\tpath/filepath.walk:384\n\tpath/filepath.Walk:406\n\*FileStore).ListUnsentBySatellite:193\n\*Service).sendOrdersFromFileStore:389\n\*Service).SendOrders:183\n\*Service).Run.func1:134\n\*Cycle).Run:92\n\*Cycle).Start.func1:71\n\*Group).Go.func1:57”}
2020-11-12T14:12:37.634Z WARN ordersfilestore Corrupted order detected in orders file {“error”: “ordersfile corrupt entry: proto: pb.OrderLimit: illegal tag 0 (wire type 0)”, “errorVerbose”: “ordersfile corrupt entry: proto: pb.OrderLimit: illegal tag 0 (wire type 0)\n\*fileV0).ReadOne:98\n\*FileStore).ListUnsentBySatellite.func1:239\n\tpath/filepath.walk:360\n\tpath/filepath.walk:384\n\tpath/filepath.Walk:406\n\*FileStore).ListUnsentBySatellite:193\n\*Service).sendOrdersFromFileStore:389\n\*Service).SendOrders:183\n\*Service).Run.func1:134\n\*Cycle).Run:92\n\*Cycle).Start.func1:71\n\*Group).Go.func1:57”}
panic: runtime error: makeslice: len out of range [recovered]
panic: runtime error: makeslice: len out of range [recovered]
panic: runtime error: makeslice: len out of range [recovered]
panic: runtime error: makeslice: len out of range

goroutine 963 [running]:
/go/pkg/mod/ +0x2e0
panic(0x8b0378, 0xa65400)
/usr/local/go/src/runtime/panic.go:969 +0x118
/go/pkg/mod/ +0x2e0
panic(0x8b0378, 0xa65400)
/usr/local/go/src/runtime/panic.go:975 +0x3c4
/go/pkg/mod/ +0x2e0
panic(0x8b0378, 0xa65400)
/usr/local/go/src/runtime/panic.go:975 +0x3c4*fileV0).ReadOne(0x20ee740, 0x0, 0x0, 0x0)
/go/src/ +0xe8*FileStore).ListUnsentBySatellite.func1(0x222ed20, 0x6a, 0xa7c2a0, 0x22f8b40, 0x0, 0x0, 0x0, 0x0)
/go/src/ +0x40c
path/filepath.walk(0x222ed20, 0x6a, 0xa7c2a0, 0x22f8b40, 0x21f5bec, 0x0, 0x0)
/usr/local/go/src/path/filepath/path.go:360 +0x2fc
path/filepath.walk(0x220d4c0, 0x14, 0xa7c2a0, 0x2514ab0, 0x21f5bec, 0x0, 0x677be0)
/usr/local/go/src/path/filepath/path.go:384 +0x204
path/filepath.Walk(0x220d4c0, 0x14, 0x24a1bec, 0x0, 0x2438550)
/usr/local/go/src/path/filepath/path.go:406 +0xe8*FileStore).ListUnsentBySatellite(0x2224840, 0xa7a4d0, 0x22b2180, 0x2818c991, 0xbfe36e95, 0x19702ee4, 0x6, 0x106ac80, 0x24be5e0, 0xa6a0fc8c, …)
/go/src/ +0x198*Service).sendOrdersFromFileStore(0x2227c20, 0xa7a4d0, 0x22b2000, 0x2818c991, 0xbfe36e95, 0x19702ee4, 0x6, 0x106ac80)
/go/src/ +0x314*Service).SendOrders(0x2227c20, 0xa7a650, 0x20106a0, 0x2818c991, 0xbfe36e95, 0x19702ee4, 0x6, 0x106ac80)
/go/src/ +0x13c*Service).Run.func1(0xa7a650, 0x20106a0, 0xa7a650, 0x20106a0)
/go/src/ +0x84*Cycle).Run(0x22013e0, 0xa7a4d0, 0x20a0ea0, 0x2010480, 0x0, 0x0)
/go/pkg/mod/ +0x134*Cycle).Start.func1(0x9afb8c, 0x0)
/go/pkg/mod/ +0x34*Group).Go.func1(0x248a4b0, 0x2312ae0)
/go/pkg/mod/ +0x50
created by*Group).Go
/go/pkg/mod/ +0x50

My node is currently v1.16.1.

Is your hard drive full? There has been quite a few having the same issues you can search the error.

I think the storage is ok.

pi@raspberrypi:~ $ sudo df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda 7.3T 1.2T 5.7T 18% /mnt/storj2

pi@raspberrypi:~ $ free -h
total used free shared buff/cache available
Mem: 3.7Gi 347Mi 2.5Gi 202Mi 869Mi 3.1Gi
Swap: 99Mi 0B

What does top report? Is your system overloaded with regards to iowait and/or cpu load?

Please, search for errors related to GET_AUDIT:

Error with orders is not good too, but it will not affect the suspension score.

I do run other background tasks on my rpi. But I think this should be normal.


I didn’t get anything. :thinking:


I realised that the uptime never exceed 30s. When i run the following command, it automatically exits.

sudo docker exec -it storagenode /app/ --color

What is your command output

docker ps

This error requires action, you need the help of a professional like @Alexey

If you run other things then I couldn’t tell. But on my Pi dedicated to Storj, a “usual” load-average these days is around 0.60~0.90, and the io/wait is around 0~10.

Your values are way higher but it might be normal depending on what other things are running.
Could be worth checking your disks are not struggling (this said, when a disk really struggles to a point where everything crumbles down, the load-average usually skyrockets which isn’t your case).

Please, search for kills

journalctl | grep "shim reaped"

And also monitor the load

docker stats

The status always reset within seconds.

There are a list of entry which spans over 6 hours.

@Alexey Do you think if it helps by removing the orders older than 48 hours?

Yes, of course, they will not be accepted anyway. Also, use a workaround to find and remove broken orders.

@deathlessdd @Pac @Alexey @peem

Thanks for giving your insights on this issue. My node is working normally after updated to version 1.7.4.


1 Like