Discrepancy between nodes ingresses

Just some context before getting to the issue:

  • All nodes are vetted;
  • All nodes on the same machine, same IP;
  • All nodes on Docker;
  • Databases on an SSD, separated from data.

Data is month to date, but it’s the same every month. Node 2, 3 and 4 received (roughly) 50GB each, while Node 1 is on 20GB (MTD In).

Node 1 and 2 are the same type of HDD (WD Gold 8TB). They are only used for Storj, nothing else does I/O on them. I’d say that HDD is not the issue as Node 4 is on a WD Green that I still struggle to understand how can be still alive. From my monitoring I also see no issue for the HDD itself.

Node 1 and 2 are both receiving data from:

  • AP1
  • EU1
  • Saltlake
  • US1

Same amount of satellites.

The only difference is that Node 1 is very old and already underwent some HDD migrations (it is 57 months old on US1, it joined in the early days of V3).

Iiuc, the four nodes should have more or less the same amount of ingress if there is free HDD space (as they are on the same machine, I also struggle to see a resource issue).

Is there anything I can check (logs, things inside databases, etc.) to understand what is happening on Node 1? Or does anyone know what could be happening here?

Edit, also, from this:

It looks like it is not receiving data from US1. But I see US1 using earnings.py. This is somewhat confusing.

Reopening as I probably found the issue, it is not Storj directly.

Trying to access details on the first node, US1:

Node 2:

Happens just for US1 on Node 1, nowhere else. It also skips to Saltlake/All nodes depending on where you were before, does not show US1.

And I see this message in the response: consoleapi storagenode: console: bandwidthdb: database disk image is malformed

Solved by following: https://support.storj.io/hc/en-us/articles/360029309111-How-to-fix-a-database-disk-image-is-malformed-

Sorry for the monologue. This error was probably there for years, I only noticed now.

1 Like

No need to apologize. Your post may help someone else who would face similar issue :slight_smile: