Nodes constantly restarting randomly few times per day last month

Node Operators troubleshooting
, ,
1

Hi all,

I am facing to nodes constantly restarting randomly few times per day last month.

I have checked logs during the time of restart:

image
image1920×742 242 KB

Docker containers does not seems to by overloaded (CPU/RAM):

image
image2204×600 134 KB

Here is HW configuration of SYNOLOGY NAS:

image
image1319×206 27.9 KB

Disks are in single coinfiguration (no RAID at all) and BRFS file system.

Configuration parameters when (re)creating nodes:

-p 28977:28967/tcp -p 28977:28967/udp -p 14011:14002 -e WALLET=“0xbaxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx” -e EMAIL="xxxxx.xxxx@gmail.com" -e ADDRESS=“xxx.xxx.xx.xxx:28977” -e STORAGE=“15TB” --mount type=bind,source=“/volume11/STORJ_10/Identity/storagenode”,destination=/app/identity --mount type=bind,source=“/volume11/STORJ_10”,destination=/app/config --name STORJ_10 storjlabs/storagenode:latest --filestore.write-buffer-size 4MiB --pieces.write-prealloc-size 4MiB

Any idea what can be wrong or can be tweaked?

Thank you

Wigo

2

Search these posts for your “verifying readability” error: it usually indicates slow or overloaded disks. I’m not sure Synology CPU/RAM reports include iowait or filesystem cache stats: as it looks like you have 11 nodes on a 4-core 16GB Atom system… and with recent testing ingress you may be pushing it. And if a DS1817+ only holds 8 HDDs then some of your nodes must be doubled-up?

You can extend the timeout: which may avoid the restarts but isn’t really fixing the problems. You can’t win races if your HDDs are responsive for a minute at a time :wink:

1 Like
3

Hi Roxor,

So do you recommend maybe merge two disks/nodes to RAID 0 (performance) and lower number of nodes? For some of them it could be possible but for some of them not.

Wigo

4

Sell the Synology and buy some used Supermicro 2U server.
It will be much more powerful with much more RAM, you will get 4 more drive slots and you will have money left to buy at least one new drive.
It also might consume more power, but the electricity is getting cheaper these days, so that wouldn’t be a problem I guess. And if you throw some solar at it, it will be a no brainer.

5

Hi Zip,

This is not option now because there is running more services and I am happy with synology so far with increased RAM and 10Gbit NIC. I would prefer more powerfull CPU but so far so good.
I have already Expansion Unit and all filled with HDDs so I am stay now with Synology :wink: And of course this is running in closet at home in lobby so need something enduser like no server rack build :slight_smile:

6

If you sell the expansion unit along with the Syno, then you will be able to buy 24 bay 4U one and two new drives.
There is also this Synology OS clone that can run on x86 platforms, so you won’t lose anything if you are not willing to switch the OS.
To run comfortably, each node should have at least 8GBs of RAM, and I don’t think on this vendor locked platform you can install that much of a RAM.
And at some point it will get EOL and EOS and you won’t be able to install any other OS on that thing. At that point it will loose most of its value.
So get rid of that thing while you can and before they will release a software update that will ban third party drives.

1 Like
7

Please don’t.

  1. This will be RAID0, zero resilience: 1 disk failure and both nodes are gone.
  2. Since it would be a one disk for the system, running two nodes on one disk is against ToS and also they will inevitable affect each other. So everything would become even worse.

You may add an SSD cache layer to improve the filesystem performance. If you can’t, then the only way is to increase a readability check timeout and readability check interval a little bit.
BTRFS is not good for storagenode without an SSD cache layer. It would be better to migrate to ext4 if you do not plan to add an SSD.