Filewalker status

Node Operators
,
1

How can I see the status in Filewalker? So how far he has already gone (%)

2

I’m not sure the used-space filewalker (the one run on startup) even knows its progress: it may be blindly traversing directories until it runs out of them?

You may have to look at the start/stop log entries from previous runs to gauge a ‘normal’ runtime… then look at the start time of the current run to estimate %-complete?

3

I have a pi4 and a 14tb HDD and it takes a really long time. That’s why I asked about progress

4

Yeah indeed we need to be able to track progress somehow.

1 Like
5

Logs should be at least info level

Used space filewalker

Task: Calculates disk usage by scanning all pieces and updates the local node databases afterwards with these values

docker logs storagenode -f 2>&1 | grep "used-space-filewalker" | grep -E "started|finished"

GC filewalker

Task: Based on a Bloom filter it marks pieces as expired for removal by the retain process

docker logs storagenode -f 2>&1 | grep "gc-filewalker" | grep -E "started|finished"

Trash-cleanup filewalker

Task: Deletes pieces from trash after trash grace period

docker logs storagenode -f 2>&1 | grep "trash-cleanup-filewalker" | grep -E "started|finished"

retain

Task: Moves expired pieces to trash

docker logs storagenode -f 2>&1 | grep "retain"

collector

Task: Removes pieces that were uploaded with an expiry date without moving them to trash first

Details

Config:

# how frequently expired pieces are collected
# collector.interval: 1h0m0s

docker logs storagenode -f 2>&1 | grep "collector"

trash

docker logs storagenode -f 2>&1 | grep "pieces:trash"

How to check progress more granularly

5 Likes
6

I get that retain moves expired pieces to trash and what GC marks for deletion after bloom filter.
But what does collector do?

7

removes expired pieces right away. These pieces were uploaded with an expiration date.

8

Is it that in the config.yaml:

# how frequently expired pieces are collected
# collector.interval: 1h0m0s

?

9

yes, this one. You may also check it in your logs.

10

Why is it required to run every hour?
I believe expiry date is set by date not by the hour?

So wouldn’t it be enough to run it once a day?

11

Because it should be deleted to date specified by the customer. However, you may change this interval too.

12

Yes, date, but does it mean the customer can set the hour too?

Example: If customer cannot specify that a file shall be deleted a 4 pm why run collector at 5 pm?

13

Ok, I was not precise enough, up to 1 minute, see

I believe even up to a second. But I guess it will be removed from the metadata on the satellite in exact time, but the node likely would not be so precise.

14

I see. Then it makes sense indeed.

15

But this reads only the addressed pieces, right? Not the entire drive…

16

Yes, it should. I didn’t figure out the whole process yet, but it looks like it going through a database and delete only expired pieces, listed there.
At least it doesn’t take a lot of time accordingly logs.

17

how is the comand for docker ?
my config.yaml post this in the node Docker logs .
Must i config in config.yaml or in Docker ?

2024-02-27T07:24:12Z INFO Configuration loaded {“Process”: “storagenode-updater”, “Location”: “/app/config/config.yaml”}
2024-02-27T07:24:12Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “console.address”}
2024-02-27T07:24:12Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “operator.wallet-features”}
2024-02-27T07:24:12Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “storage.allocated-disk-space”}
2024-02-27T07:24:12Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “server.address”}
2024-02-27T07:24:12Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “healthcheck.enabled”}
2024-02-27T07:24:12Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “collector.interval”}
2024-02-27T07:24:12Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “operator.wallet”}
2024-02-27T07:24:12Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “storage.allocated-bandwidth”}
2024-02-27T07:24:12Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “operator.email”}
2024-02-27T07:24:12Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “server.private-address”}
2024-02-27T07:24:12Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “contact.external-address”}
2024-02-27T07:24:12Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “healthcheck.details”}
2024-02-27T07:24:12Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “storage2.database-dir”}
2024-02-27T07:24:12Z INFO Invalid configuration file value for key {“Process”: “storagenode-updater”, “Key”: “log.stack”}
2024-02-27T07:24:12Z INFO Anonymized tracing enabled {“Process”: “storagenode-updater”}
2024-02-27T07:24:12Z INFO Running on version {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.96.6”}
2024-02-27T07:24:12Z INFO Downloading versions. {“Process”: “storagenode-updater”, “Server Address”: “https://version.storj.io”}
2024-02-27T07:24:12Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode”, “Version”: “v1.96.6”}
2024-02-27T07:24:12Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode”}
2024-02-27T07:24:12Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.96.6”}
2024-02-27T07:24:12Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”}
2024-02-27 07:24:13,582 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-02-27 07:24:13,582 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-02-27 07:24:13,582 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

18

You can ignore these messages. They show up since config.yaml is shared between 2 processes.

1 Like
19

You may convert any config.yaml parameter to a command line argument by adding -- before the parameter name, removing : (or replacing it to =) and adding the resulted command line argument after the image name in your docker run command.
You may also check what else parameters you can change (and what are their default values) by this command:

docker exec -it storagenode ./storagenode setup --help

Of course you may convert any command line argument back to the config.yaml file parameter as well by removing the leading --, and replacing = to : (or placing it at the end of the parameter name without leading spaces), and add/update the resulted parameter in your config.yaml file without leading spaces, save the config and restart the node.
Please note, the command line arguments have a precedence above parameters in the config.yaml file.

1 Like
20

Storagenode ver 1.101.3.
The trash walker it’s called: pieces:trash. You missed the final “s”.

The used-space-filewalker, I believe, now it’s called piecestore:cache. I found an entry about it beeing bypassed by the run command; I use:

--pieces.enable-lazy-filewalker=false \
--storage2.piece-scan-on-startup=false

Does it reffers to something else?

1 Like