How to set up a non-Docker node using systemd services with the official native storagenode binary

fmoledina · November 9, 2020, 3:49pm

For sure. I’ll update the OP, included amd64 instructions by default, and update the URLs to 1.16.1 binaries.

11FaTuSh11 · November 9, 2020, 5:24pm

It turned out to run the node on CentOS 7 + VestaCP and ubuntu 20.04. But unfortunately, ha ha ha I started a 9GB node and threw an error that the minimum is 500GB. And everything works. thanks . PS Please correct all of the above in a separate template (tutorial) for installation.

fmoledina · November 9, 2020, 9:57pm

I’ll stand by the fact that the guide was correct for what it was originally written for . Having said that, it’s now been updated for an amd64 installation.

deathlessdd · November 9, 2020, 10:12pm

You did a great job I didn’t use it but I can see it would work perfect for a new user.

madest · November 26, 2020, 7:51pm

if I need to run 2 or 3 nodes, do I need to make service files for each node? or are there other ways?

deathlessdd · November 26, 2020, 8:33pm

Yes I would think you have to setup a service for each node.

fmoledina · December 4, 2020, 3:14am

Hello, I have a method for this but I wanted to get through an update cycle to make sure it worked before I posted it.

I’m running two separate nodes with two separate binaries storagenode1 and storagenode2, and then the individual systemd service files for each node. Both nodes use a common storagenode-updater binary.

If you already have a node installed as per the OP instructions, set up the second node as follows.

Prerequisites

Set up identities, port forwards, etc, for your second node.
Mount appropriate storage mediums at /mnt/storagenode1 and /mnt/storagenode2.

Method for two nodes

Copy the storagenode binaries and systemd service files:

cd /opt/storagenode/bin
mv storagenode storagenode1
cp storagenode1 storagenode2
cd /etc/systemd/system
mv storagenode.service storagenode1.service
mv storagenode-updater.service storagenode1-updater.service
cp storagenode1.service storagenode2.service
cp storagenode1-updater.service storagenode2-updater.service

Contents of /etc/systemd/system/storagenode1.service:

# This is a SystemD unit file for the Storage Node
# To configure:
# - Update the user and group that the service will run as (User & Group below)
# - Ensure that the Storage Node binary is in /usr/local/bin and is named storagenode (or edit the ExecStart line
#   below to reflect the name and location of your binary
# - Ensure that you've run setup and have edited the configuration appropriately prior to starting the
#   service with this script
# To use:
# - Place this file in /etc/systemd/system/ or wherever your SystemD unit files are stored
# - Run systemctl daemon-reload
# - To start run systemctl start storagenode1

[Unit]
Description  = Storage Node service
After        = syslog.target network.target

[Service]
Type         = simple
User         = storj-storagenode
Group        = storj-storagenode
ExecStart    = /opt/storagenode/bin/storagenode1 run --config-dir "/mnt/storagenode1/config"
Restart      = on-failure
NotifyAccess = main

[Install]
Alias        = storagenode1
WantedBy     = multi-user.target

The systemd service /etc/systemd/system/storagenode2.service is the same except replace storagenode1 with storagenode2 in the binary, --config-dir option, and systemd alias.

Contents of /etc/systemd/system/storagenode1-updater.service:

# To configure:
# - Update the user and group that the service will run as (User & Group below)
# - Ensure that the Storage Node binary is in /usr/local/bin and is named storagenode (or edit the ExecStart line
#   below to reflect the name and location of your binary
# - Ensure that you've run setup and have edited the configuration appropriately prior to starting the
#   service with this script
# To use:
# - Place this file in /etc/systemd/system/ or wherever your SystemD unit files are stored
# - Run systemctl daemon-reload
# - To start run systemctl start storagenode1-updater

[Unit]
Description  = Storage Node Updater service
After        = syslog.target network.target

[Service]
Type         = simple
User         = storj-storagenode
Group        = storj-storagenode
ExecStart    = /opt/storagenode/bin/storagenode-updater run --config-dir "/mnt/storagenode1/config" --binary-location "/opt/storagenode/bin/storagenode1" --service-name "storagenode1"
Restart      = on-failure
NotifyAccess = main

[Install]
Alias        = storagenode1-updater
WantedBy     = multi-user.target

The systemd service /etc/systemd/system/storagenode2-updater.service is the same except replace storagenode1 with storagenode2 for the --config-dir, --binary-location, and --service-name options, and change the systemd alias to storagenode2-updater. Note that the binary /opt/storagenode/bin/storagenode-updater remains common for both services.

When complete, run systemctl daemon-reload.

Run the node and updater services for each separate node as indicated in the OP.

ACarneiro · December 13, 2020, 11:44am

Stupid question, but where are the logs kept when running the binary? Been trying to run success rate but have no idea where to point it for logs…

stefanbenten · December 13, 2020, 12:47pm

You should be able to find it in the journal, via journalctl -xe and possibly adding -u storagenode1

deathlessdd · December 13, 2020, 2:32pm

You should change the config to point the log to a file.

fmoledina · December 13, 2020, 4:29pm

You can point the log to a file as @deathlessdd suggests. If you use systemd, you can periodically export the entire log into a file and run that through successrate.sh as @stefanbenten suggests.

To output the entire journal:

journalctl --no-pager -u storagenode > storagenode.log
successrate.sh storagenode.log

deathlessdd · December 13, 2020, 6:46pm

It would be alot easier to just set the config file to save the log as a file instead.

Like here

 # can be stdout, stderr, or a filename
log.output: "/volume2/storj/storj/node.log"

You would just restart the storagenode service after editing the config. Make sure to edit, uncomment the config as well.

ACarneiro · December 13, 2020, 11:15pm

Thank you, everyone
That is obviously what I should have done instead of pestering you all, sorry about the brain fart

crossplatformdev · December 19, 2020, 6:44am

Tested on 1.18.1, instructions of the 1st post apply and works.

I personally put the binary executables on /mnt/storagenode/bin/ and the services on /mnt/storagenode/service_units/ ,

and linked them to /etc/systemd/system/

before running

systemctl enable storagenode
systemctl enable storagenode-updater

and then reboot.

Notice you have to update binary paths on service unit files!!

This way in case of an OS reinstall, is a lot easier to bring back the node online =)

Can someone put all the commands altogether in a bash script and upload it to gist?

Thank you!

fmoledina · February 18, 2021, 5:07pm

@Alexey, can you make this a wiki? I’d like to update the systemd service files to include Restart=always as per discussion in the Tech Preview thread:

Thanks!

will.topping · February 18, 2021, 9:34pm

should we change to “always” on the config for both services or just the updater service?

fmoledina · February 18, 2021, 9:46pm

From the other thread, @badfrog indicated that he changed it to always for both the node and updater. I would think that it’s more important for the node to restart upon update. I would do it for both services.

will.topping · February 18, 2021, 11:16pm

Ive tried to set this up but come across a similar issue to @11FaTuSh11 but the reason is different I think? Ive tried reapplying “chown -R storj-storagenode:storj-storagenode /mnt/storagenode” Any ideas?

Feb 18 23:06:16 storj-node2 storagenode[4269]: Error: piecestore monitor: error verifying location and/or readability of storage directory: open /mnt/storagenode/config/storage/storage-dir-verification: no such file or directory
Feb 18 23:06:16 storj-node2 systemd[1]: storagenode.service: Main process exited, code=exited, status=1/FAILURE
Feb 18 23:06:16 storj-node2 systemd[1]: storagenode.service: Failed with result ‘exit-code’.
Feb 18 23:06:16 storj-node2 systemd[1]: storagenode.service: Scheduled restart job, restart counter is at 185.

fmoledina · February 19, 2021, 12:16am

Can you confirm the following?

That /mnt/storagenode exists and that this is where you want to store your data (i.e. hard drive mounted here)
That you’ve run the storagenode setup --config-dir ... --identity-dir ... step

If so, what is the output of ls -la /mnt/storagenode/config and ls -la /mnt/storagenode/identity?

will.topping · February 19, 2021, 12:25am

I may have messed up the permissions as I copied the files to the wrong location, do these look ok?

total 56
drwx------ 4 storj-storagenode storj-storagenode 4096 Feb 19 00:20 .
drwxr-xr-x 6 storj-storagenode storj-storagenode 4096 Feb 18 23:37 …
-rw------- 1 storj-storagenode storj-storagenode 8699 Feb 18 23:55 config.yaml
drwx------ 4 storj-storagenode storj-storagenode 4096 Feb 18 22:49 orders
-rw------- 1 storj-storagenode storj-storagenode 32768 Feb 19 00:20 revocations.db
drwx------ 6 storj-storagenode storj-storagenode 4096 Feb 19 00:20 storage
-rw------- 1 storj-storagenode storj-storagenode 1420 Feb 19 00:20 trust-cache.json

/mnt/storagenode/identity:
total 32
drwxr-xr-x 2 storj-storagenode storj-storagenode 4096 Feb 18 23:58 .
drwxr-xr-x 6 storj-storagenode storj-storagenode 4096 Feb 18 23:37 …
-rw-r–r-- 1 storj-storagenode storj-storagenode 558 Feb 18 23:58 ca.1606473117.cert
-rw-r–r-- 1 storj-storagenode storj-storagenode 1088 Feb 18 23:58 ca.cert
-rw------- 1 storj-storagenode storj-storagenode 241 Feb 18 23:58 ca.key
-rw-r–r-- 1 storj-storagenode storj-storagenode 1100 Feb 18 23:58 identity.1606473117.cert
-rw-r–r-- 1 storj-storagenode storj-storagenode 1630 Feb 18 23:58 identity.cert
-rw------- 1 storj-storagenode storj-storagenode 241 Feb 18 23:58 identity.key