After upgrading to version 1.16.1, my node stopped working correctly. Raspberry pi 4

Hello.
There was a problem with the automatic update to version 1.16.1.

When I start the dashboard, I see that my storage node is working for ≈ 20 seconds, after which it immediately reboots.

My Web Dashboard has stopped working.

Device: raspberry pi 4

I would be grateful for your help.
I want to deal with this situation.

sudo docker logs --tail 20 storagenode
storj.io/common/sync2.(Cycle).Run(0x2d80c00, 0xa7a4d0, 0x2df5c20, 0x2fb3b90, 0x0, 0x0)
/go/pkg/mod/storj.io/common@v0.0.0-20201014090530-c4af8e54d5c4/sync2/cycle.go:92 +0x134
storj.io/common/sync2.(Cycle).Start.func1(0x2e15f00, 0x0)
/go/pkg/mod/storj.io/common@v0.0.0-20201014090530-c4af8e54d5c4/sync2/cycle.go:71 +0x34
golang.org/x/sync/errgroup.(Group).Go.func1(0x2f9b590, 0x2e3fb20)
/go/pkg/mod/golang.org/x/sync@v0.0.0-20200625203802-6e8e738ad208/errgroup/errgroup.go:57 +0x50
created by golang.org/x/sync/errgroup.(Group).Go
/go/pkg/mod/golang.org/x/sync@v0.0.0-20200625203802-6e8e738ad208/errgroup/errgroup.go:54 +0x50
2020-11-13T15:06:58.232Z INFO Configuration loaded {“Location”: “/app/config/config.yaml”}
2020-11-13T15:06:58.236Z INFO Operator email {“Address”: "
"}
2020-11-13T15:06:58.236Z INFO Operator wallet {“Address”: "
**"}
2020-11-13T15:06:59.848Z INFO Telemetry enabled
2020-11-13T15:07:02.135Z INFO db.migration Database Version {“version”: 46}
2020-11-13T15:07:03.858Z INFO preflight:localtime start checking local system clock with trusted satellites’ system clock.
2020-11-13T15:07:04.809Z INFO preflight:localtime local system clock is in sync with trusted satellites’ system clock.
2020-11-13T15:07:04.809Z INFO bandwidth Performing bandwidth usage rollups
2020-11-13T15:07:04.810Z INFO Node 126DHrtFzedo7cd64v4QeT9qx6YHnXW2B3xW7wTj7kJn93Pm1BG started
2020-11-13T15:07:04.811Z INFO Public server started on [::]:28967
2020-11-13T15:07:04.811Z INFO Private server started on 127.0.0.1:7778
2020-11-13T15:07:04.812Z INFO trust Scheduling next refresh {“after”: “10h4m51.238044147s”}

You need to give it more time to finish starting up after an update.

My storage node restarts automatically.
Do I somehow need to force it to run longer?

Can you show a bigger log, and what does docker ps say?

Please check if you have many unsent orders in your orders/unsent folder.

For this check, I just have to check the folder along the path /mnt/storagenode/orders ?

Yes, just check how many are in the orders/unsent folder.

What is your command output

docker logs storagenode 2>&1 | head -n 50
Summary

pi@raspberrypi:~ $ sudo docker logs storagenode 2>&1 | head -n 50
2020-11-13T14:23:37.722Z INFO Configuration loaded {“Location”: “/app/config/config.yaml”}
2020-11-13T14:23:37.727Z INFO Operator email {“Address”: “rudenkopostbox@gmail.com”}
2020-11-13T14:23:37.727Z INFO Operator wallet {“Address”: “0xB4E924Bc7ea2Dcc1c391f99349505B1E50876337”}
2020-11-13T14:23:38.824Z INFO Telemetry enabled
2020-11-13T14:23:38.940Z INFO db.migration Database Version {“version”: 46}
2020-11-13T14:23:40.192Z INFO preflight:localtime start checking local system clock with trusted satellites’ system clock.
2020-11-13T14:23:41.179Z INFO preflight:localtime local system clock is in sync with trusted satellites’ system clock.
2020-11-13T14:23:41.179Z INFO trust Scheduling next refresh {“after”: “5h22m5.024019837s”}
2020-11-13T14:23:41.181Z INFO bandwidth Performing bandwidth usage rollups
2020-11-13T14:23:41.183Z INFO Node 126DHrtFzedo7cd64v4QeT9qx6YHnXW2B3xW7wTj7kJn93Pm1BG started
2020-11-13T14:23:41.183Z INFO Public server started on [::]:28967
2020-11-13T14:23:41.184Z INFO Private server started on 127.0.0.1:7778
2020-11-13T14:23:57.492Z WARN ordersfilestore Corrupted order detected in orders file {“error”: “ordersfile corrupt entry: proto: can’t skip unknown wire type 6”, “errorVerbose”: “ordersfile corrupt entry: proto: can’t skip unknown wire type 6\n\tstorj.io/storj/storagenode/orders/ordersfile.(*fileV0).ReadOne:98\n\tstorj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite.func1:239\n\tpath/filepath.walk:360\n\tpath/filepath.walk:384\n\tpath/filepath.Walk:406\n\tstorj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite:193\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrdersFromFileStore:389\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders:183\n\tstorj.io/storj/storagenode/orders.(*Service).Run.func1:134\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2020-11-13T14:23:57.917Z WARN ordersfilestore Corrupted order detected in orders file {“error”: “ordersfile corrupt entry: ordersfile: unexpected EOF”, “errorVerbose”: “ordersfile corrupt entry: ordersfile: unexpected EOF\n\tstorj.io/storj/storagenode/orders/ordersfile.(*fileV0).ReadOne:92\n\tstorj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite.func1:239\n\tpath/filepath.walk:360\n\tpath/filepath.walk:384\n\tpath/filepath.Walk:406\n\tstorj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite:193\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrdersFromFileStore:389\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders:183\n\tstorj.io/storj/storagenode/orders.(*Service).Run.func1:134\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
panic: runtime error: makeslice: len out of range [recovered]
panic: runtime error: makeslice: len out of range [recovered]
panic: runtime error: makeslice: len out of range [recovered]
panic: runtime error: makeslice: len out of range

goroutine 897 [running]:
github.com/spacemonkeygo/monkit/v3.newSpan.func1(0x0)
/go/pkg/mod/github.com/spacemonkeygo/monkit/v3@v3.0.7-0.20200515175308-072401d8c752/ctx.go:147 +0x2e0
panic(0x8b0378, 0xa65400)
/usr/local/go/src/runtime/panic.go:969 +0x118
github.com/spacemonkeygo/monkit/v3.newSpan.func1(0x0)
/go/pkg/mod/github.com/spacemonkeygo/monkit/v3@v3.0.7-0.20200515175308-072401d8c752/ctx.go:147 +0x2e0
panic(0x8b0378, 0xa65400)
/usr/local/go/src/runtime/panic.go:975 +0x3c4
github.com/spacemonkeygo/monkit/v3.newSpan.func1(0x1ad2600)
/go/pkg/mod/github.com/spacemonkeygo/monkit/v3@v3.0.7-0.20200515175308-072401d8c752/ctx.go:147 +0x2e0
panic(0x8b0378, 0xa65400)
/usr/local/go/src/runtime/panic.go:975 +0x3c4
storj.io/storj/storagenode/orders/ordersfile.(*fileV0).ReadOne(0x1ad2578, 0x0, 0x0, 0x0)
/go/src/storj.io/storj/storagenode/orders/ordersfile/v0.go:106 +0x208
storj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite.func1(0x1fc2700, 0x69, 0xa7c2a0, 0x1fbeb40, 0x0, 0x0, 0x0, 0x0)
/go/src/storj.io/storj/storagenode/orders/store.go:239 +0x40c
path/filepath.walk(0x1fc2700, 0x69, 0xa7c2a0, 0x1fbeb40, 0x1cfbbec, 0x0, 0x0)
/usr/local/go/src/path/filepath/path.go:360 +0x2fc
path/filepath.walk(0x1cb67c0, 0x14, 0xa7c2a0, 0x1baaf30, 0x1cfbbec, 0x0, 0x677be0)
/usr/local/go/src/path/filepath/path.go:384 +0x204
path/filepath.Walk(0x1cb67c0, 0x14, 0x1c74bec, 0x0, 0x1c7c270)
/usr/local/go/src/path/filepath/path.go:406 +0xe8
storj.io/storj/storagenode/orders.(*FileStore).ListUnsentBySatellite(0x1a8ca40, 0xa7a4d0, 0x18fa660, 0x536e11dc, 0xbfe3c39f, 0x992f5855, 0x4, 0x106ac80, 0x1b80000, 0xa6a1ef6c, …)
/go/src/storj.io/storj/storagenode/orders/store.go:193 +0x198
storj.io/storj/storagenode/orders.(*Service).sendOrdersFromFileStore(0x19bf050, 0xa7a4d0, 0x18fa4e0, 0x536e11dc, 0xbfe3c39f, 0x992f5855, 0x4, 0x106ac80)
/go/src/storj.io/storj/storagenode/orders/service.go:389 +0x314
storj.io/storj/storagenode/orders.(*Service).SendOrders(0x19bf050, 0xa7a650, 0x1d165b0, 0x536e11dc, 0xbfe3c39f, 0x992f5855, 0x4, 0x106ac80)
/go/src/storj.io/storj/storagenode/orders/service.go:183 +0x13c
storj.io/storj/storagenode/orders.(*Service).Run.func1(0xa7a650, 0x1d165b0, 0xa7a650, 0x1d165b0)
/go/src/storj.io/storj/storagenode/orders/service.go:134 +0x84

You need the help of a professional like @Alexey

Sounds a bit like this issue (although it was thought fixed in 1.15.3): Raspberry Pi4 - Node crashes since today... weird GO error

At the moment the workaround is still valid:

this new update sure didn’t go smooth, i haven’t even updated yet, because looks like there are many people with trouble.

@LorezAR

As far as I can see the problem is the same that I reported in Raspberry Pi4 - Node crashes since today... weird GO error but this time is in another place of the code and if I’m not mistaken it only applies to order files v0; currently we are using v1, so the issue should only impact to nodes with deprecated/old order files and the tendency is to be less over the time.

However, to make even more sure that the problem comes from there I’d like to know wich architecture is your operative system (OS). While your hardware supports 64 bits architecture if your OS is 32, the machine operates on 32 bits and then it faces this issue.

Meanwhile, I’m commenting with one of my mates who has been deeply working in this part to see if a minor patch that I bear in mind makes sense.

Our apologies for all the troubles and thank you (all SNOs community) very much for your collaboration.

2 Likes

Hey. I was able to fix the reboot issue. the dashboard shows that everything is in order. But the web panel doesn’t work.


ports are open, I have not changed anything in the router settings

I am using the 32 bit version of the OS.
I think the patch would be very helpful.

Please, try to open it on the same PC where is node running.
If you want to open it on other location, please, use this guide: https://documentation.storj.io/resources/faq/how-to-remote-access-the-web-dashboard

running 64bit is literally 10 times faster for some tasks… and there is basically no practical downsides…
unless if you have something that cannot run anything other than 32bit or if the software needs 32bit… then it’s a complete waste to even consider using 32bit…

ofc without it you wouldn’t have helped find this bug, so there is that advantage i guess… xD

most stuff can run 64bit today, just saying if you hardware can, then you really should…

1 Like

The issue with 64 bit on Pi’s is it is still in alpha/beta status, so you have to hunt it down. Currently there are two methods:

  1. Modify boot.config for 32 bit system, with 64 bit kernel. This method has gotten easier since the rpi-update tool does most if the work now.
  2. Download 64 bit beta ISO for 64 bit software and kernel. This will likely at some point be officially released, but is currently intended for development and testing (aka things may break and be hard to revert); with that said, I’ve been running it for over 4 months without any issues.