Node shows tiny ingress only

[jammerdan]

I have more than one node that show tiny ingress only, while ingress is flowing in.

Node is not suspended and space is available:

Logs showing that ingress is happening.

docker logs --since "2024-05-21T00:00:00Z" storagenode | grep uploaded | wc -l
36087

It is impossible that there are 36087 successful uploads and the ingress is only 30 Bytes.

[Alexey]

Do you have errors related to databases?
Also, if your node is on 1.104.5, the bandwidth database is updated differently:

@jtolio

So, in v1.104.x+, the storage node graphs are the amount of bandwidth ordered, not necessarily used, whereas prior to v1.104.x, the graphs were the amount of bandwidth actually used.

[jammerdan]

Let me check. What should I grep for? “ERROR” ?

[Alexey]

yes, both ERROR|error (or use a case insensitive grep -i command) and database.

[Ruskiem]

or WARN, or FATAL

[jammerdan]

I am seeing some database is locked errors but the last one is from 18/05.

@Ruskiem

or WARN, or FATAL

No FATALs

[Alexey]

@jammerdan

I am seeing some database is locked errors but the last one is from 18/05.

do you have non-error records after that related to databases?

[jammerdan]

No, it looks like the last entries with databases are from 18/05 with errors but after that there is nothing.

[Alexey]

I mean entries in your log file. It should report when it modifying the database to add a usage.
Search for bandwidth.

[jammerdan]

Yes what I have said.
These are the errors where I have database is locked:

2024-05-18T23:10:10Z    ERROR   piecestore      failed to add bandwidth usage   {"Process": "storagenode", "error": "bandwidthdb: database is locked"

But this is the last one in the logs. So none for 19th/20th/21.

[Alexey]

Please show 10 last results for bandwidth clause.

[jammerdan]

You mean logs grepped with “bandwidth”?

[Alexey]

Yes, only with that single word.

[jammerdan]

docker logs storagenode | grep --max-count 10 bandwidth
2024-05-21T03:10:13Z    INFO    bandwidth       Persisting bandwidth usage cache to db  {"Process": "storagenode"}
2024-05-21T04:10:13Z    INFO    bandwidth       Persisting bandwidth usage cache to db  {"Process": "storagenode"}
2024-05-21T05:10:13Z    INFO    bandwidth       Persisting bandwidth usage cache to db  {"Process": "storagenode"}
2024-05-21T06:10:13Z    INFO    bandwidth       Persisting bandwidth usage cache to db  {"Process": "storagenode"}
2024-05-21T07:10:13Z    INFO    bandwidth       Persisting bandwidth usage cache to db  {"Process": "storagenode"}
2024-05-21T08:10:13Z    INFO    bandwidth       Persisting bandwidth usage cache to db  {"Process": "storagenode"}
2024-05-21T09:10:13Z    INFO    bandwidth       Persisting bandwidth usage cache to db  {"Process": "storagenode"}
2024-05-21T10:10:13Z    INFO    bandwidth       Persisting bandwidth usage cache to db  {"Process": "storagenode"}
2024-05-21T11:10:13Z    INFO    bandwidth       Persisting bandwidth usage cache to db  {"Process": "storagenode"}
2024-05-21T12:10:13Z    INFO    bandwidth       Persisting bandwidth usage cache to db  {"Process": "storagenode"}

[Alexey]

So, records are submitted to the database.
Then I guess that the reason is what’s currently accounted as ingress.

[jammerdan]

I see today is different.
The node that showed only 30B for the current day yesterday shows 50GB for todays ingress so far.
But as for yesterday as said, when I see appr. 40k successfull uploads the ingress normally cannot be 30 bytes.

@Alexey

So, records are submitted to the database.

Maybe records did not get submitted to the database yesterday?

[Alexey]

Accordingly your logs they are submitted:

@jammerdan

2024-05-21T12:10:13Z INFO bandwidth Persisting bandwidth usage cache to db {"Process": "storagenode"}

[jammerdan]

Yes you are right. I missed the date.

These are the last 10 from today backwards:

docker logs storagenode | grep bandwidth | tail -10
2024-05-21T23:10:13Z    INFO    bandwidth       Persisting bandwidth usage cache to db  {"Process": "storagenode"}
2024-05-22T00:10:13Z    INFO    bandwidth       Persisting bandwidth usage cache to db  {"Process": "storagenode"}
2024-05-22T01:10:13Z    INFO    bandwidth       Persisting bandwidth usage cache to db  {"Process": "storagenode"}
2024-05-22T02:10:13Z    INFO    bandwidth       Persisting bandwidth usage cache to db  {"Process": "storagenode"}
2024-05-22T03:10:13Z    INFO    bandwidth       Persisting bandwidth usage cache to db  {"Process": "storagenode"}
2024-05-22T04:10:13Z    INFO    bandwidth       Persisting bandwidth usage cache to db  {"Process": "storagenode"}
2024-05-22T05:10:13Z    INFO    bandwidth       Persisting bandwidth usage cache to db  {"Process": "storagenode"}
2024-05-22T06:10:13Z    INFO    bandwidth       Persisting bandwidth usage cache to db  {"Process": "storagenode"}
2024-05-22T07:10:13Z    INFO    bandwidth       Persisting bandwidth usage cache to db  {"Process": "storagenode"}
2024-05-22T08:10:13Z    INFO    bandwidth       Persisting bandwidth usage cache to db  {"Process": "storagenode"}

So ok we can tell data has been submitted today and yesterday. But yesterday it showed only 30 bytes.

[Alexey]

Does it show the correct value for yesterday now?

[jammerdan]

I don’t know if it is correct, but it seemed to have picked up some data afterwards: