Should I be nervous or will it get online

[tola5]

Hi my docker give me this log and the web interface say it a old version and it offline and there no data it come after a update on my Unraid . can see there go some data ind but don’t know or it stoj there get them . are there any thing too be nervous for or are it only a question off time before it UpToDate

2024-03-02 21:22:55,554 WARN received SIGQUIT indicating exit request
2024-03-02 21:22:55,554 INFO waiting for storagenode, processes-exit-eventlistener to die
2024-03-02 21:22:55,555 INFO exited: storagenode (exit status 1; not expected)
2024-03-02 21:22:56,557 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)
2024-03-02 21:25:39,106 INFO Set uid to user 0 succeeded
2024-03-02 21:25:39,109 INFO RPC interface 'supervisor' initialized
2024-03-02 21:25:39,109 INFO supervisord started with pid 1
2024-03-02 21:25:40,111 INFO spawned: 'processes-exit-eventlistener' with pid 11
2024-03-02 21:25:40,112 INFO spawned: 'storagenode' with pid 12
2024-03-02 21:25:40,113 INFO spawned: 'storagenode-updater' with pid 13
2024-03-02T21:25:40+01:00       INFO    Configuration loaded    {"Process": "storagenode-updater", "Location": "/app/config/config.yaml"}
2024-03-02T21:25:40+01:00       INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "operator.wallet-features"}
2024-03-02T21:25:40+01:00       INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "console.address"}
2024-03-02T21:25:40+01:00       INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-bandwidth"}
2024-03-02T21:25:40+01:00       INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "healthcheck.details"}
2024-03-02T21:25:40+01:00       INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "operator.wallet"}
2024-03-02T21:25:40+01:00       INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-disk-space"}
2024-03-02T21:25:40+01:00       INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "operator.email"}
2024-03-02T21:25:40+01:00       INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "contact.external-address"}
2024-03-02T21:25:40+01:00       INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "healthcheck.enabled"}
2024-03-02T21:25:40+01:00       INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.private-address"}
2024-03-02T21:25:40+01:00       INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage2.database-dir"}
2024-03-02T21:25:40+01:00       INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.address"}
2024-03-02T21:25:40+01:00       INFO    Invalid configuration file value for key        {"Process": "storagenode-updater", "Key": "log.stack"}
2024-03-02T21:25:40+01:00       INFO    Invalid configuration file value for key        {"Process": "storagenode-updater", "Key": "log.caller"}
2024-03-02T21:25:40+01:00       INFO    Invalid configuration file value for key        {"Process": "storagenode-updater", "Key": "log.encoding"}
2024-03-02T21:25:40+01:00       INFO    Anonymized tracing enabled      {"Process": "storagenode-updater"}
2024-03-02T21:25:40+01:00       INFO    Running on version      {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.95.1"}
2024-03-02T21:25:40+01:00       INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-03-02T21:25:40+01:00       INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.95.1"}
2024-03-02T21:25:40+01:00       INFO    New version is being rolled out but hasn't made it to this node yet     {"Process": "storagenode-updater", "Service": "storagenode"}
2024-03-02T21:25:40+01:00       INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.95.1"}
2024-03-02T21:25:40+01:00       INFO    New version is being rolled out but hasn't made it to this node yet     {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-03-02 21:25:41,677 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-03-02 21:25:41,677 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-03-02 21:25:41,677 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

[arrogantrabbit]

Not sure what you are asking.

Your node is not reachable, logs will have more information.

Check your port forwarding and ip addresses.

[tola5]

here are my log not sure what part you need the link are to my one drive
node.log

have not change ip or port

[Alexey]

Hello @tola5,
Welcome back!

Your node have several issues:

2024-03-02T23:10:45+01:00	ERROR	piecestore	failed to add bandwidth usage	{"process": "storagenode", "error": "bandwidthdb: database is locked", "errorVerbose": "bandwidthdb: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:913\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:542\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}

This error mean that your disk cannot keep up the upload and the database updates simultaneously, so you likely need to move databases to SSD.

and something is requested to stop the node:

2024-03-02T23:10:47+01:00	INFO	Got a signal from the OS: "terminated"	{"process": "storagenode"}

all other errors after that are a consequence of the stopping node. I didn’t see a final error, but I guess that it’s a FATAL one.
You may check:

docker logs storagenode 2>&1 | grep FATAL | tail

if you redirected logs to the file, then you need to check that file instead, i.e.

cat /mnt/storj/storagenode/node.log | grep FATAL | tail

replace the path /mnt/storj/storagenode/node.log with the actual one in this command.

[tola5]

Hi thx
The dbs are on my m2
It nothing with im running a old version .
My web interface say
Your node is outdated. Please update to:

V
But can’t get too update it

[Alexey]

Then I do not know, why it can be locked, if it’s on SSD. I would suggest to check the configuration: in your docker run command you should have a separate binding for the databases and should have an option --storage2.database-dir pointing to the this binding inside the container, see How to move DB’s to SSD on Docker
For example:

How to move DB’s to SSD on Docker

Yes docker run -d ... \ ... --mount type=bind,source=/mnt/storj/storagenode/identity,destination=/app/identity \ --mount type=bind,source=/mnt/storj/storagenode,destination=/app/config \ --mount type=bind,source=/mnt/ssd/storagenode/dbs,destination=/app/dbs \ ... storjlabs/storagenode:latest \ --storage2.database-dir=dbs

Do you have mentioned errors in the logs?

[tola5]

yes my config have

storage2.database-dir: “”

storage2.database-dir: “dbs”
and with error are it the version you talk abut ?
I get this but it the log from the docker cant see it in the log storj write too file
the web interface say it a old version also but not allow too update .
as it say "New version is being rolled out but hasn’t made it to this node yet " how long do that take

2024-03-03T08:56:24+01:00       INFO    Running on version      {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.95.1"}
2024-03-03T08:56:24+01:00       INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-03-03T08:56:24+01:00       INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.95.1"}
2024-03-03T08:56:24+01:00       INFO    New version is being rolled out but hasn't made it to this node yet     {"Process": "storagenode-updater", "Service": "storagenode"}
2024-03-03T08:56:24+01:00       INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.95.1"}
2024-03-03T08:56:24+01:00       INFO    New version is being rolled out but hasn't made it to this node yet     {"Process": "storagenode-updater",

[Alexey]

No, you need to search for errors in your redirected logs. The docker logs storagenode will return only logs from the storagenode-updater in this case.

[tola5]

Failed to add bandwidth usage / bandwidth: database is locked

Hi all, been getting this error and from searching, i found this and repaired bandwidth.db. https://forum.storj.io/t/database-error-error-piecestore-failed-to-add-bandwidth-usage/3044 I went ahead and repaired bandwidth.db using this guide and it took about a day to repair: https://support.storj.io/hc/en-us/articles/360029309111-How-to-fix-a-database-disk-image-is-malformed But now, restarted the node and I still see the same error, help! 2023-04-21T13:57:15.885Z INFO piecestore download st…

from this “If all else fails, you can just delete the database. It is not needed. The storagenode process will just create a new one. You may be missing some info in the dashboard but functionality will not be affected.”
do I need all the files ind the db or can I remove some and get storj too run out the old out loosing my host data

[Alexey]

You can remove them instead of repairing, yes. You will lose the stat and history. But you need to follow this guide:

How to fix database: file is not a database error

[tola5]

Try too fix it follow the guide out luck . Now I remove the db file there had problems. And now up work one more . Thx so much for the help

[Alexey]

I hope it would help. Right now it doesn’t look like a root cause.

[00riddler]

Had the same issue this morning.
Restarting the node did not help.
After several hours it was running normal again.

[Alexey]

Please check your logs for FATAL errors.

[tola5]

Thx also not when the corrupt database was on the m2

[tola5]

are there a way too get db storge data stats as I understand it only graphic the .db file and it will not use more data ind what it allowed ind the config file. but are there a way too look up what are trash and what are data of the storge allowed .

[Alexey]

The allocation you specified with the -e STORAGE= option in your docker run, the actually used space you may check by the command:

du --si -d 1 /mnt/storj/storagenode/storage/

where /mnt/storj/storagenode/ is a path to the data location.

[tola5]

Thx do you know or there any plans too have a other way data work so when there problem with the database all info don’t ge5 show wrong

[Alexey]

Some
https://forum.storj.io/t/disk-usage-discrepancy/24715/20

Disk usage discrepancy?

The node that I’m investigating is indeed a VM, but the storage system should be well fast enough with <1ms latency. I hear that there is possibly a bug with bloom filter cleanup on large nodes (>14M pieces). This node has 31M pieces for the us satellite and may be affected. The bloom filter filling up for large nodes (and thus becoming ineffective by allowing all pieces to remain) makes sense. Lets just hang on and more info will come.

Disk usage discrepancy?

Both commands have no output whatsoever: [image]

Disk usage discrepancy?

OK so since the last garbage collection gave me the “success” result, there seemed to be another process that is actually moving the bulk of the pieces to trash. Since the “success” exit of the garbage collector filewalker process my node has accumulated 700GB of additional trash, and it’s still going. Not sure what process this is, but it might be retain since I see in the logs it has initiated one for EU1 but not exited yet.

[tola5]

Thx will the filewalker update the datebase then after some time with the real amounts off data ? .
And second question the web interfaces think it only habe use 130GB out off 13TB but it more like 8TB will it over write the data or give other problems when it hit the 13TB but mot on the web interfaces.

Thx for all the help