I would like to know how to troubleshoot my suspension score issue.
I am running on Raspberry pi 4 4GB.
You should check your logs and see if you see anything having issues.
Do you think this is the cause of the issue?
2020-11-12T14:12:37.545Z WARN ordersfilestore Corrupted order detected in orders file {“error”: “ordersfile corrupt entry: ordersfile: unexpected EOF”, “errorVerbose”: “ordersfile corrupt entry: ordersfile: unexpected EOF\n\tstorj.io/storj/storagenode/orders/ordersfile.(*fileV0).ReadOne:92\n\tstorj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite.func1:239\n\tpath/filepath.walk:360\n\tpath/filepath.walk:384\n\tpath/filepath.Walk:406\n\tstorj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite:193\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrdersFromFileStore:389\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders:183\n\tstorj.io/storj/storagenode/orders.(*Service).Run.func1:134\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2020-11-12T14:12:37.634Z WARN ordersfilestore Corrupted order detected in orders file {“error”: “ordersfile corrupt entry: proto: pb.OrderLimit: illegal tag 0 (wire type 0)”, “errorVerbose”: “ordersfile corrupt entry: proto: pb.OrderLimit: illegal tag 0 (wire type 0)\n\tstorj.io/storj/storagenode/orders/ordersfile.(*fileV0).ReadOne:98\n\tstorj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite.func1:239\n\tpath/filepath.walk:360\n\tpath/filepath.walk:384\n\tpath/filepath.Walk:406\n\tstorj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite:193\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrdersFromFileStore:389\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders:183\n\tstorj.io/storj/storagenode/orders.(*Service).Run.func1:134\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
panic: runtime error: makeslice: len out of range [recovered]
panic: runtime error: makeslice: len out of range [recovered]
panic: runtime error: makeslice: len out of range [recovered]
panic: runtime error: makeslice: len out of rangegoroutine 963 [running]:
github.com/spacemonkeygo/monkit/v3.newSpan.func1(0x0)
/go/pkg/mod/github.com/spacemonkeygo/monkit/v3@v3.0.7-0.20200515175308-072401d8c752/ctx.go:147 +0x2e0
panic(0x8b0378, 0xa65400)
/usr/local/go/src/runtime/panic.go:969 +0x118
github.com/spacemonkeygo/monkit/v3.newSpan.func1(0x0)
/go/pkg/mod/github.com/spacemonkeygo/monkit/v3@v3.0.7-0.20200515175308-072401d8c752/ctx.go:147 +0x2e0
panic(0x8b0378, 0xa65400)
/usr/local/go/src/runtime/panic.go:975 +0x3c4
github.com/spacemonkeygo/monkit/v3.newSpan.func1(0x230e0d8)
/go/pkg/mod/github.com/spacemonkeygo/monkit/v3@v3.0.7-0.20200515175308-072401d8c752/ctx.go:147 +0x2e0
panic(0x8b0378, 0xa65400)
/usr/local/go/src/runtime/panic.go:975 +0x3c4
storj.io/storj/storagenode/orders/ordersfile.(*fileV0).ReadOne(0x20ee740, 0x0, 0x0, 0x0)
/go/src/storj.io/storj/storagenode/orders/ordersfile/v0.go:89 +0xe8
storj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite.func1(0x222ed20, 0x6a, 0xa7c2a0, 0x22f8b40, 0x0, 0x0, 0x0, 0x0)
/go/src/storj.io/storj/storagenode/orders/store.go:239 +0x40c
path/filepath.walk(0x222ed20, 0x6a, 0xa7c2a0, 0x22f8b40, 0x21f5bec, 0x0, 0x0)
/usr/local/go/src/path/filepath/path.go:360 +0x2fc
path/filepath.walk(0x220d4c0, 0x14, 0xa7c2a0, 0x2514ab0, 0x21f5bec, 0x0, 0x677be0)
/usr/local/go/src/path/filepath/path.go:384 +0x204
path/filepath.Walk(0x220d4c0, 0x14, 0x24a1bec, 0x0, 0x2438550)
/usr/local/go/src/path/filepath/path.go:406 +0xe8
storj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite(0x2224840, 0xa7a4d0, 0x22b2180, 0x2818c991, 0xbfe36e95, 0x19702ee4, 0x6, 0x106ac80, 0x24be5e0, 0xa6a0fc8c, …)
/go/src/storj.io/storj/storagenode/orders/store.go:193 +0x198
storj.io/storj/storagenode/orders.(*Service).sendOrdersFromFileStore(0x2227c20, 0xa7a4d0, 0x22b2000, 0x2818c991, 0xbfe36e95, 0x19702ee4, 0x6, 0x106ac80)
/go/src/storj.io/storj/storagenode/orders/service.go:389 +0x314
storj.io/storj/storagenode/orders.(*Service).SendOrders(0x2227c20, 0xa7a650, 0x20106a0, 0x2818c991, 0xbfe36e95, 0x19702ee4, 0x6, 0x106ac80)
/go/src/storj.io/storj/storagenode/orders/service.go:183 +0x13c
storj.io/storj/storagenode/orders.(*Service).Run.func1(0xa7a650, 0x20106a0, 0xa7a650, 0x20106a0)
/go/src/storj.io/storj/storagenode/orders/service.go:134 +0x84
storj.io/common/sync2.(*Cycle).Run(0x22013e0, 0xa7a4d0, 0x20a0ea0, 0x2010480, 0x0, 0x0)
/go/pkg/mod/storj.io/common@v0.0.0-20201014090530-c4af8e54d5c4/sync2/cycle.go:92 +0x134
storj.io/common/sync2.(*Cycle).Start.func1(0x9afb8c, 0x0)
/go/pkg/mod/storj.io/common@v0.0.0-20201014090530-c4af8e54d5c4/sync2/cycle.go:71 +0x34
The Go Programming Language(0x248a4b0, 0x2312ae0)
/go/pkg/mod/golang.org/x/sync@v0.0.0-20200625203802-6e8e738ad208/errgroup/errgroup.go:57 +0x50
created by The Go Programming Language
/go/pkg/mod/golang.org/x/sync@v0.0.0-20200625203802-6e8e738ad208/errgroup/errgroup.go:54 +0x50
My node is currently v1.16.1.
Is your hard drive full? There has been quite a few having the same issues you can search the error.
I think the storage is ok.
pi@raspberrypi:~ $ sudo df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda 7.3T 1.2T 5.7T 18% /mnt/storj2pi@raspberrypi:~ $ free -h
total used free shared buff/cache available
Mem: 3.7Gi 347Mi 2.5Gi 202Mi 869Mi 3.1Gi
Swap: 99Mi 0B
What does top report? Is your system overloaded with regards to iowait and/or cpu load?
Please, search for errors related to GET_AUDIT:
Error with orders is not good too, but it will not affect the suspension score.
I didn’t get anything. 
I realised that the uptime never exceed 30s. When i run the following command, it automatically exits.
sudo docker exec -it storagenode /app/dashboard.sh --color
What is your command output
docker ps
This error requires action, you need the help of a professional like @Alexey
If you run other things then I couldn’t tell. But on my Pi dedicated to Storj, a “usual” load-average these days is around 0.60~0.90, and the io/wait is around 0~10.
Your values are way higher but it might be normal depending on what other things are running.
Could be worth checking your disks are not struggling (this said, when a disk really struggles to a point where everything crumbles down, the load-average usually skyrockets which isn’t your case).
Please, search for kills
journalctl | grep "shim reaped"
And also monitor the load
docker stats
@Alexey Do you think if it helps by removing the orders older than 48 hours?
Yes, of course, they will not be accepted anyway. Also, use a workaround to find and remove broken orders.
@deathlessdd @Pac @Alexey @peem
Thanks for giving your insights on this issue. My node is working normally after updated to version 1.7.4.
Cheers!
1 Like
I was suspended with no audit errors (checked for GET_AUDIT failures) and no other relevant errors in the logs. The system runs perfectly (27% docker loads), system and kernel logs are clean.
However the dashboard reports Audit yellow warnings and the node was suspended.
Any hints?
Thanks