Can't set up config.yaml before a new node starts

Today I wanted to set up a new node. As usual, I made sure the signed identity files are placed in the right directory, I copied my config.yaml template and filled it with info for the new storage node, and started docker container. But…

2020-10-10T19:48:57.611Z        INFO        Configuration loaded        {"Location": "/app/config/config.yaml"}                                                                                                                                                                                                                              
2020-10-10T19:48:57.657Z        INFO        Operator email        {"Address": "…"}              
2020-10-10T19:48:57.658Z        INFO        Operator wallet        {"Address": "…"}                                                                                                                                                                                                                 
Error: Error starting master database on storagenode: group:                                                                                                                                                                                                                                                                                 
--- stat config/storage/blobs: no such file or directory                                                       
--- stat config/storage/temp: no such file or directory                                                                                                                                                                                                                                                                                      
--- stat config/storage/garbage: no such file or directory                                                                                                                                                                                                                                                                                   
--- stat config/storage/trash: no such file or directory                                                                                                                                                                                                                                                                                      

Only when I removed the config.yaml it actually started.

Why couldn’t I use my template for config.yaml to set up a new node?

Please, use the storagenode setup instead. Because all needed info, include databases will be created only with this command.
The storagenode run will not create them to prevent the start from scratch if there is something missed - either config.yaml or databases or folders tree (this useful when your disk accidentally unmounted for example).
If it is a docker version - then there should not be any info - the destination folder must be empty because of

It’s quite unusual to use a configuration file as a flag this way. From my sysadmin experience the usual practice is to drop configuration files for a service in the right places, only then install/run the actual service—this is for example what is done on Debian, as all services are already being started as part of apt install expecting their configuration to already be set up. This simplifies automated deployments.

But ok. If that’s the Storj way, then it would at least be nice to have it documented.

It’s documented: https://documentation.storj.io/setup/cli/storage-node#storage-node-concepts
We do not require to create a config.yaml in advance as you can see.
If you follow the documentation - you will succeed with setup.
The Linux installer and updater would be available soon.

1 Like

I don’t see any documentation regarding config.yaml there. The only mention of this file is here: https://documentation.storj.io/resources/faq/redirect-logs And this is why I need config.yaml.

Yes, it’s a tweak after setup. You are right there is no any mention of config.yaml at the setup process.
I’ll create a ticket in the documentation project to add a sentence like “your destination folder must be empty for a new node”, since it’s not obvious as turned out.

Please make sure that you explicitly mention that not even config.yaml can be there, this will be very helpful for sysadmins used to usual conventions regarding service configuration.

The usual convention - to install the service then configure. In most cases you even will not have a folder or configs before the service got installed.
But you can configure before the installation, if service is allowing to do so.
The storagenode container do not allowing to do so.

How the destination can be empty and have a file in the same time?
But ok, I will add this that empty mean that folder must not have anything in it, neither files nor folders.

Update: as turned out, we actually can have a different folders in the storage destination, but you should have an empty storage subfolder (or do not have at all) and no files in the folder, especially config.yaml.

So, reformulated this way:

If you setup a brand new node, make sure that destination folder for storage doesn’t have a config.yaml or/and non-empty storage folder inside, otherwise the storagenode container could fail to start.

Could you please check - Is it clear enough?

Depends on the system, probably. The Debian practice is to drop configuration files where they’re supposed to, then apt install service. This is because (1) the postinst scripts will detect missing files and only if they’re not there, they open configuration dialogs, (2) the services usually are started by the postinst scripts. As I wrote before, this simplifies automated deployments.

In other words, the existence of a configuration file is a flag for the service being configured or not, not for the service being already deployed.

It is a good documentation practice to not only provide a raw description of a procedure, but also point out its interactions with other practices. The Storj procedure is explicitly at odds with some sysadmin practices, hence it is a good idea to explicitly state the conflict.

Looks fine to me, thank you!

Could you please, show that in the Debian documentation, as a reference? For honestly I didn’t heard such requirement anywhere, unless you build the software from the source.

This is defined in the Debian Policy, section 9.3.3.1:

The default behaviour is to enable autostarting your package’s daemon.

BTW, maybe the best way would be to provide a command-line switch for the log.output setting?

We do not provide the corresponding daemons currently! Everything that you use currently is a binary without a daemon.
The tooling you are complaining about not working as expected are not built/released yet!!

We are currently working on the necessary set of tooling to install a systemd service that also walks you through the setup. It will be shipped as a simple .deb to install it.

1 Like

And it is not what I am complaining about. I wanted to automate creating new nodes on my own without waiting for official Debian packages and found a hurdle with the configuration file, that’s it.

Have you tried --log.output stderr ?

Sorry, I do not see requirements to create a config file before installing the service from the package manager.
Scripts and symlinks are far away from the text config file in a YAML format.

Eh, it’s a consequence of the fact that dæmons are started automatically. If they start automatically, when are you going to configure them?

But we do not talking about daemons!
In your excerpt the init system is already installed, when you configure what it should run, places scripts and symlinks on scripts to do so.

In our case it is a user-space application. The daemon in our case the only docker which we do not maintain.

Also, you can install the ssh server for example and then configure it. Perhaps it would work, if you make all configurations before install, but this is a rare case and this is why I did not see the requirement to do so.

Ok, it seems that I won’t be able to convince you, nor you won’t be able to convince me. I give up, sorry.

Thank you, this indeed works. Maybe the FAQ page should be edited to use this switch, as opposed to editing the config.yaml file?

You are the first one, who asked.
I do not see the use case for that except your specific situation.
However, for everyone interested what commands they could use, there is an article: https://documentation.storj.io/resources/faq/other-commands