Synology Docker crash: failed to initialize logging driver

Node Operators
1

So purchased a Synology DS1520+ to replace my 10 yr old ReadyNAS. Let me tell you I’ve been extremely disappointed with the OS proprietary packaging and utilities. This problem being one of many with the fact they decided logs should go into a database and not flat files…

Anyways, after running my container for about 2 weeks, it suddenly exited and would not restart. The one message I found said:

Start container storjlabs-storagenode1 failed: {“message”:“failed to initialize logging driver: database is locked”}.

After fighting it for an hour or so, I finally found this which said to simply copy the container and restart. How to fix Synology Docker: failed to initialize logging driver: database is locked – TechOverflow

That did fix it, for 2 more weeks and then it happened again. The second time I was able to remediate more quickly, but my reputation is still getting dinged for the downtime. Opened a case with Synology, and they seemed to of deemed it interesting enough for escalation. In the meantime, has anyone else seen this?

2

Im Running about 30 Nodes on some Snyology-Stations.
I sometimes had this Problem, too.
It seems to occur in Case of Hard reboots or a Lot of IO-Load.

I use the CLI and Docker-Compose files for about 2 years now.
Over all relatively stable.

3

So here’s the reply from Synology:

After a further investigation by our developers, they found that the container has additional parameters from it’s github link that show it should be run over terminal, rather than in the UI.
We suggested you to run the container via command but not UI to have a complete parameters introduced into it, and so the container can run properly.

OK. This smells a little funny. Obviously I can’t expect to have a ssh terminal open 24/7 to run a docker instance. I’m assuming I could start it detached but then how do you autostart the docker instances after a reboot?

4 

--restart=always

is what you need :slight_smile:

1 Like
5

Thanks, saw that in the sample and did that in a script. Still think their answer was wrong. We’ll know in 2 weeks.

6

Any feedback / response from Synology?

Sometimes, I’m getting some container in some kind of “freezed” state, when I can’t even remove them…

Wondering, if this is the same issue.

7

This could be related to a hardware issues, like problems with disk or memory.
When this happen do you have a free RAM?

8

I think sth made the process to shutdown unexpectedly which caused the db to be locked. Maybe RAM leakage as @Alexey mentioned. Check the logs for the reason why it stopped, not the reason why it couldn’t restart.

9

I have 64GB RAM.
Usually have more than 20GB free.

Can’t check logs, because when the issue happens - can’t see any logs from that container.

10

I never ran nodes from Syno UI. I always used putty/SSH/CLI to install and run the nodes as root (sudo su). You don’t have to keep the terminal open ;). Just run the node and close the terminal (putty or whatever) by typing exit 2 times if you were in sudo su. Check my post for the commands.
The terminal is just a way to send commands to the linux kernel; the commands and programs started don’t live in that terminal, like in a windows program.
Never had a locked db or sudden crash. But my hardware is new; maybe if you have old or sh hardware, there could be something defective.
https://forum.storj.io/t/my-docker-run-commands-for-multinodes-on-synology-nas/22034

11

how about write out storj logs to file?