The overusage is gone after trash removal and the node receives ingress again.
So this one did not get stuck with no ingress:
ND:
MND:
But we see the difference in free space. The MND shows the total space free on disk while the ND shows what is free for the node. This time the value seems consistent with the other values.
I have a node similar to this. The ingress will come to a point when some data will be sent to trash again, then no more ingress until trash is empty. Or I can say that the Used space will somehow keeps itself at certain level, not growing up anymore.
Here we go again. What shall I do?
No ingress for this new month so far:
Despite having 2.5TB free space on disk and node dashboard shows
The logs show downloads only.
I have not yet restarted. I believe as soon as I restart uploads would resume.
Its set to 1 TB and the numbers add up, so its normal and no problems at all?
But it says free 34.83 GB? So shouldnât it receive uploads?
You may receive uploads. What numbers on the MDB?
And do you have âless than requestedâ in your logs?
And the DDNS hostname is updated? Do you have an ability to see when is it happened?
P.S.
Suddenly without me doing anything uploads have resumed:
I have not restarted the node.
All I was able to see is that there was no ingress from the start of the month until a couple of minutes ago.
So that means roughly 6 hours. I have no explanation why the node did not receive ingress during that time. It was online all the time and downloads had been requested.
And in the logs you didnât see any âuploadâ?
I have no explanation except that your node was not selected for uploads, it could be a result of a short unavailability.
Did you check the audit history? Did it miss at least one?
Ok. I have no explanation at this time too.
Could it be that the node became full and then trash was cleared?
How long would it take after a trash clearence the satellite would select a node again for uploads? Could that take several hours?
Edit:
I think here is something:
2024-03-01T04:32:15Z INFO pieces:trash emptying trash started
I have that for all satellites.
Uploads did resume at:
2024-03-01T06:32:32Z INFO piecestore uploaded
Thatâs quite exactly 2 hours after trash emptying. Could it take that long until the satellites start sending data again?
Itâs possible. The node is checked in on the satellites every hour by default, but the stats (from the satellites) is updated roughly every 12h.
However, if the piece:trash chore tooks as long as 6h, then this information would not be updated so long.
Maybe it wasnât that long.
Maybe the node got full some time yesterday and remained full until trash emptying started today at 04:32.
I donât know how long this did take but maybe it was done at 6 and node got checked in shortly after and resumed uploads.
However the node dashboard displayed 34 GB free⌠But I donât know since when.
Here we go again:
Node dashboard:
The node is not suspended, online above 99% on all satellites.
Host has free space also.
What shall I check. Node is getting only egress currently, no uploads in the log.
Still no uploads only downloads.
And still no ingressâŚ
@Alexey
Any idea what else to check. It seems that again only a full restart will recover this node from this condition. So anything I can check before restart?
Still nothing:
I have checked the logs. This is the last line with upload in it:
2024-03-30T18:04:04Z INFO piecestore uploaded
After that only download messages. No other messages like errors or something around that time.
Why is there no ingress since then? This seems to be a serious flaw in the code. I cannot check every node every day if it has stopped receiving ingress despite having enough space and being not suspended or something.
Here is how the nodes is doing:
How much free space does it have (physically)?
Can you open its listening address and port as itâs specified in the config on you phone using a mobile internet (not WiFi)?
It has 650GB free space.
I can access it from outside through configured address and port:
{
"Statuses": null,
"Help": "To access Storagenode services, please use DRPC protocol!",
"AllHealthy": true
}
I donât know for sure but I believe as soon as I fully restart this node it will receive ingress again.
I have not done that yet as it would have wiped the logs.