1.144.4 version error

@elek version 1.144 have major error, it not update new income data.
it only updates it after restart.
So it write disk full till the end.

So my nodes in half week after restart gain over 10tb.
I see that average usage by satellite is rising, but not data amount.
I see that it deletes old data.

Is it return not updated data via API too? Like in the multinode dashboard?

it not return via SNO Api also. I dont know what multinode dashboard uses to collect data, as i dont use it. I use web API to aggregate my from 106 nodes.

The multinode dashboard uses DRPC, like a satellite, so it shows what’s the satellite seeing.

is there any documents how to install multinode in windows?
do i need to install it like service?

It’s absolutely the same for any platform:

You may also use a docker version, but I doubt that you want it :slight_smile:
Perhaps a useful hack for you would be to export connection parameters for every node to the json files, then form an json array to import them at once.

The format:

Are you sure, that you used 1.144.4?
It’s not published yet.
Do you maybe meant 1.143.4?

I use 144.3 it is 3 weeks old. I use it for a week
I see Devs made some corrections already.
I will try 1.144.4 and in parallel investigate multinode

No important fix in 1.144.4.

You don’t need multinode, all information is already available via localhost:debugport/metrics or SNO rest API.

Some important metrics (from /metrics):

  • used_space: the used space calculated by the actual space reporter (either dedicated or shared)
  • satellite_usage: this is the useful bytes according to the satellite (doesn’t include trash, overhead). This is calculated on satellite side (but shared with storagenode)
  • hashstore{field="LenLogs") cumulative size of log files
  • hashstore{field="LenSet"} The useful data in the DB of the hashstore. Lenlogs can be bigger, as it can also contains dead data (until following compactions)

it not update new income data.
it only updates it after restart.
So it write disk full till the end.

This is not what we experience with our nodes, and recent change should have affect only on the UI.

Wondering if you run active migration, it is possible that that’s the one which is filling the disk?

Hello.
I see there only this parameters in API
“diskSpace”:{“used”:638563713546,“available”:2861436286454,“overused”:0,“allocated”:3500000000000,“trash”:78851443904,“reclaimable”:0}
Or it is by each sattelite section?

All my nodes are migrated some months ago.

Used this is what i use to monitor amount from all nodes together.
when node works it go dawn all the time, after restart it gaining amount.

I now made update from 1.144.3 to 1.144.4 also it made restart to all nodes at same time.
before update it shown that i have 380TB used after now i have 391TB used

Did you mean this in /metrics

TYPE satellite_usage gauge

satellite_usage{satellite=“_121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”,scope=“storj_io_storj_storagenode_nodestats”,field=“used_bytes”} 1.1911087189899876e+11
satellite_usage{satellite=“_12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”,scope=“storj_io_storj_storagenode_nodestats”,field=“used_bytes”} 2.3631619668956416e+12
satellite_usage{satellite=“_12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”,scope=“storj_io_storj_storagenode_nodestats”,field=“used_bytes”} 8.45987884702436e+11

I do not understand how to translate it to usage node has 3.6TB
there is no other fileds with this name.
I found some data in
mon/stats with this name also
satellite_usage,satellite=12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs,scope=storj.io/storj/storagenode/nodestats used_bytes=845987884702.436035
satellite_usage,satellite=121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6,scope=storj.io/storj/storagenode/nodestats used_bytes=119110871898.998764
satellite_usage,satellite=12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S,scope=storj.io/storj/storagenode/nodestats used_bytes=2363161966895.641602

this show that i have 3328 GB but dashboard show that today i have 3.49TB reported usage by sattelite

Hello.

Today i made one node restart, before node shown that it has free space, after it show that it has over usage, difference is 30GB in about 2 days. I sended you a pictures and metrict before and after restart to support, asked to forward it to you.

On added picture on left is before restart on right after
here we see that node do not see rising from ingress after previuse overusage was deleted.

here on picture we see ingress on 7-8.01 was alsmost flat, as i restarted node on 7 and there was over usage. after overusege was deleted some old files it got free space, and ingress starts again, but it not stop because node not register new data added.

and today after new restart we see that there is overusage again.