2 nodes no more ingress

[jammerdan]

I have had similar issues in the past.
But back then a node restart did resolve it.

Not this time. I have had 2 nodes that did not see ingress anymore. Both had space available and were not suspended as far as I have seen.
The second node magically resolved the issue and is currently receiving uploads again. Not the first node:
(Audit/Suspension/Online):
ap1: 100.00% 100.00% 92.69%
eu1: 100.00% 100.00% 94.47%
us1: 99.93% 100.00% 96.48%
saltlake: 100.00% 100.00% 92.83%

I have already restarted it twice and it did not help, no uploads for at least more than 2 hours now. What else could I check or do?

[Vadim]

what logs show? sometimes it think that it is full.

[jammerdan]

@jammerdan

The second node magically resolved the issue and is currently receiving uploads again. Not the first node:

It seems that now after some additional wait the first node has magically resolved the issue as well. I can see uploads again. I have no idea why ingress initially has stopped and why it is back now.

[jammerdan]

Unfortunately I am seeing the very same issue again.

I had it that restart helped, sometimes it did not.

Logs show many uploads, then ceasing, after restart uploads are resumed:

2025-10-08T16:49:02Z    INFO    piecestore      uploaded
2025-10-08T16:49:02Z    INFO    piecestore      uploaded
2025-10-08T16:49:02Z    INFO    piecestore      uploaded
2025-10-08T16:49:02Z    INFO    piecestore      upload canceled
2025-10-08T16:49:02Z    INFO    piecestore      uploaded
2025-10-08T16:49:02Z    INFO    piecestore      uploaded
2025-10-08T16:49:03Z    INFO    piecestore      uploaded
2025-10-08T16:49:03Z    INFO    piecestore      uploaded
2025-10-08T16:49:04Z    INFO    piecestore      uploaded
2025-10-08T16:49:04Z    INFO    piecestore      uploaded
2025-10-08T16:49:04Z    INFO    piecestore      uploaded
2025-10-08T16:49:04Z    INFO    piecestore      uploaded
2025-10-08T16:49:04Z    INFO    piecestore      uploaded
2025-10-08T16:49:05Z    INFO    piecestore      uploaded
2025-10-08T16:49:05Z    INFO    piecestore      uploaded
2025-10-08T16:49:06Z    INFO    piecestore      uploaded
2025-10-08T16:49:06Z    INFO    piecestore      uploaded
2025-10-08T16:49:07Z    INFO    piecestore      uploaded
2025-10-08T16:49:08Z    INFO    piecestore      uploaded
2025-10-08T16:49:09Z    INFO    piecestore      uploaded
2025-10-08T16:49:09Z    INFO    piecestore      uploaded
2025-10-08T16:49:09Z    INFO    piecestore      uploaded
2025-10-08T16:49:10Z    INFO    piecestore      uploaded
2025-10-08T16:49:10Z    INFO    piecestore      uploaded
2025-10-08T16:49:11Z    INFO    piecestore      uploaded
2025-10-08T16:49:11Z    INFO    piecestore      uploaded
2025-10-08T16:49:11Z    INFO    piecestore      uploaded
2025-10-08T16:49:11Z    INFO    piecestore      uploaded
2025-10-08T16:49:11Z    INFO    piecestore      uploaded
2025-10-08T16:49:12Z    INFO    piecestore      uploaded
2025-10-08T16:49:12Z    INFO    piecestore      uploaded
2025-10-08T16:49:12Z    INFO    piecestore      uploaded
2025-10-08T16:49:13Z    INFO    piecestore      uploaded
2025-10-08T16:49:14Z    INFO    piecestore      uploaded
2025-10-08T16:49:14Z    INFO    piecestore      uploaded
2025-10-08T16:49:14Z    INFO    piecestore      uploaded
2025-10-08T16:49:15Z    INFO    piecestore      uploaded
2025-10-08T16:49:15Z    INFO    piecestore      uploaded
2025-10-08T16:49:15Z    INFO    piecestore      uploaded
2025-10-08T16:49:15Z    INFO    piecestore      uploaded
2025-10-08T16:49:15Z    INFO    piecestore      uploaded
2025-10-08T16:49:16Z    INFO    piecestore      uploaded
2025-10-08T16:49:16Z    INFO    piecestore      uploaded
2025-10-08T16:49:17Z    INFO    piecestore      uploaded
2025-10-08T16:49:19Z    INFO    piecestore      uploaded
2025-10-08T16:49:19Z    INFO    piecestore      uploaded
2025-10-08T16:49:20Z    INFO    piecestore      uploaded
2025-10-08T16:49:21Z    INFO    piecestore      uploaded
2025-10-08T16:49:21Z    INFO    piecestore      uploaded
2025-10-08T16:49:21Z    INFO    piecestore      uploaded
2025-10-08T16:49:21Z    INFO    piecestore      uploaded
2025-10-08T16:49:21Z    INFO    piecestore      uploaded
2025-10-08T16:49:21Z    INFO    piecestore      uploaded
2025-10-08T16:49:22Z    INFO    piecestore      uploaded
2025-10-08T16:49:23Z    INFO    piecestore      uploaded
2025-10-08T16:49:24Z    INFO    piecestore      uploaded
2025-10-08T16:49:24Z    INFO    piecestore      uploaded
2025-10-08T16:49:24Z    INFO    piecestore      uploaded
2025-10-08T16:49:25Z    INFO    piecestore      uploaded
2025-10-08T16:49:25Z    INFO    piecestore      uploaded
2025-10-08T16:49:26Z    INFO    piecestore      uploaded
2025-10-08T16:49:27Z    INFO    piecestore      uploaded
2025-10-08T16:49:27Z    INFO    piecestore      uploaded
2025-10-08T16:49:27Z    INFO    piecestore      uploaded
2025-10-08T16:49:27Z    INFO    piecestore      uploaded
2025-10-08T16:49:27Z    INFO    piecestore      uploaded
2025-10-08T16:49:28Z    INFO    piecestore      uploaded
2025-10-08T16:49:28Z    INFO    piecestore      upload canceled
2025-10-08T16:49:29Z    INFO    piecestore      uploaded
2025-10-08T16:49:29Z    INFO    piecestore      uploaded
2025-10-08T16:49:29Z    INFO    piecestore      uploaded
2025-10-08T16:49:32Z    INFO    piecestore      uploaded
2025-10-08T16:49:32Z    INFO    piecestore      uploaded
2025-10-08T16:49:32Z    INFO    piecestore      uploaded
2025-10-08T16:49:32Z    INFO    piecestore      uploaded
2025-10-08T16:49:33Z    INFO    piecestore      uploaded
2025-10-08T16:49:33Z    INFO    piecestore      uploaded
2025-10-08T16:49:33Z    INFO    piecestore      uploaded
2025-10-08T16:49:34Z    INFO    piecestore      uploaded
2025-10-08T16:49:35Z    INFO    piecestore      uploaded
2025-10-08T16:49:36Z    INFO    piecestore      uploaded
2025-10-08T16:49:36Z    INFO    piecestore      uploaded
2025-10-08T16:49:36Z    INFO    piecestore      uploaded
2025-10-08T16:49:37Z    INFO    piecestore      uploaded
2025-10-08T16:49:37Z    INFO    piecestore      uploaded
2025-10-08T16:49:38Z    INFO    piecestore      uploaded
2025-10-08T16:49:39Z    INFO    piecestore      uploaded
2025-10-08T16:49:39Z    INFO    piecestore      uploaded
2025-10-08T16:49:40Z    INFO    piecestore      uploaded
2025-10-08T16:49:40Z    INFO    piecestore      uploaded
2025-10-08T16:49:40Z    INFO    piecestore      uploaded
2025-10-08T16:49:41Z    INFO    piecestore      uploaded
2025-10-08T16:49:41Z    INFO    piecestore      uploaded
2025-10-08T16:49:42Z    INFO    piecestore      uploaded
2025-10-08T16:49:43Z    INFO    piecestore      uploaded
2025-10-08T16:49:44Z    INFO    piecestore      uploaded
2025-10-08T16:49:44Z    INFO    piecestore      uploaded
2025-10-08T16:49:44Z    INFO    piecestore      uploaded
2025-10-08T16:49:44Z    INFO    piecestore      uploaded
2025-10-08T16:49:45Z    INFO    piecestore      uploaded
2025-10-08T16:49:45Z    INFO    piecestore      uploaded
2025-10-08T16:49:47Z    INFO    piecestore      uploaded
2025-10-08T16:49:47Z    INFO    piecestore      uploaded
2025-10-08T16:49:47Z    INFO    piecestore      uploaded
2025-10-08T16:49:47Z    INFO    piecestore      uploaded
2025-10-08T16:49:48Z    INFO    piecestore      uploaded
2025-10-08T16:49:48Z    INFO    piecestore      uploaded
2025-10-08T16:49:48Z    INFO    piecestore      uploaded
2025-10-08T16:49:49Z    INFO    piecestore      uploaded
2025-10-08T16:49:49Z    INFO    piecestore      uploaded
2025-10-08T16:49:50Z    INFO    piecestore      uploaded
2025-10-08T16:49:51Z    INFO    piecestore      uploaded
2025-10-08T16:49:51Z    INFO    piecestore      uploaded
2025-10-08T16:49:52Z    INFO    piecestore      uploaded
2025-10-08T16:49:52Z    INFO    piecestore      uploaded
2025-10-08T16:49:53Z    INFO    piecestore      uploaded
2025-10-08T16:49:56Z    INFO    piecestore      uploaded
2025-10-08T16:49:56Z    INFO    piecestore      uploaded
2025-10-08T16:49:57Z    INFO    piecestore      uploaded
2025-10-08T16:49:57Z    INFO    piecestore      uploaded
2025-10-08T16:49:57Z    INFO    piecestore      uploaded
2025-10-08T16:49:57Z    INFO    piecestore      uploaded
2025-10-08T16:49:58Z    INFO    piecestore      uploaded
2025-10-08T16:49:59Z    INFO    piecestore      uploaded
2025-10-08T16:49:59Z    INFO    piecestore      uploaded
2025-10-08T16:49:59Z    INFO    piecestore      uploaded
2025-10-08T16:49:59Z    INFO    piecestore      uploaded
2025-10-08T16:50:00Z    INFO    piecestore      uploaded
2025-10-08T16:50:00Z    INFO    piecestore      uploaded
2025-10-08T16:50:00Z    INFO    piecestore      uploaded
2025-10-08T16:50:00Z    INFO    piecestore      uploaded
2025-10-08T16:50:01Z    INFO    piecestore      uploaded
2025-10-08T16:50:02Z    INFO    piecestore      uploaded
2025-10-08T16:50:03Z    INFO    piecestore      uploaded
2025-10-08T16:50:03Z    INFO    piecestore      uploaded
2025-10-08T16:50:03Z    INFO    piecestore      uploaded
2025-10-08T16:50:04Z    INFO    piecestore      uploaded
2025-10-08T16:50:04Z    INFO    piecestore      uploaded
2025-10-08T16:50:05Z    INFO    piecestore      uploaded
2025-10-08T16:50:06Z    INFO    piecestore      upload canceled
2025-10-08T16:50:06Z    INFO    piecestore      upload canceled
2025-10-08T16:50:06Z    INFO    piecestore      upload canceled
2025-10-08T16:50:06Z    INFO    piecestore      upload canceled
2025-10-08T16:50:07Z    INFO    piecestore      uploaded
2025-10-08T16:50:07Z    INFO    piecestore      uploaded
2025-10-08T16:50:08Z    INFO    piecestore      uploaded
2025-10-08T16:50:08Z    INFO    piecestore      uploaded
2025-10-08T16:50:09Z    INFO    piecestore      uploaded
2025-10-08T16:50:09Z    INFO    piecestore      uploaded
2025-10-08T16:50:10Z    INFO    piecestore      uploaded
2025-10-08T16:50:10Z    INFO    piecestore      uploaded
2025-10-08T16:50:11Z    INFO    piecestore      uploaded
2025-10-08T16:50:12Z    INFO    piecestore      uploaded
2025-10-08T16:50:12Z    INFO    piecestore      uploaded
2025-10-08T16:50:12Z    INFO    piecestore      uploaded
2025-10-08T16:50:13Z    INFO    piecestore      uploaded
2025-10-08T16:50:13Z    INFO    piecestore      uploaded
2025-10-08T16:50:13Z    INFO    piecestore      uploaded
2025-10-08T16:50:14Z    INFO    piecestore      uploaded
2025-10-08T16:50:14Z    INFO    piecestore      uploaded
2025-10-08T16:50:14Z    INFO    piecestore      uploaded
2025-10-08T16:50:15Z    INFO    piecestore      upload canceled
2025-10-08T16:50:15Z    INFO    piecestore      uploaded
2025-10-08T16:50:17Z    INFO    piecestore      uploaded
2025-10-08T16:50:17Z    INFO    piecestore      uploaded
2025-10-08T16:50:18Z    INFO    piecestore      uploaded
2025-10-08T16:50:18Z    INFO    piecestore      uploaded
2025-10-08T16:50:18Z    INFO    piecestore      uploaded
2025-10-08T16:50:18Z    INFO    piecestore      uploaded
2025-10-08T16:50:19Z    INFO    piecestore      uploaded
2025-10-08T16:50:19Z    INFO    piecestore      uploaded
2025-10-08T16:50:20Z    INFO    piecestore      uploaded
2025-10-08T16:50:20Z    INFO    piecestore      uploaded
2025-10-08T16:50:20Z    INFO    piecestore      uploaded
2025-10-08T16:50:20Z    INFO    piecestore      uploaded
2025-10-08T16:50:21Z    INFO    piecestore      uploaded
2025-10-08T16:50:21Z    INFO    piecestore      upload canceled
2025-10-08T16:50:22Z    INFO    piecestore      uploaded
2025-10-08T16:50:22Z    INFO    piecestore      uploaded
2025-10-08T16:50:22Z    INFO    piecestore      uploaded
2025-10-08T16:50:22Z    INFO    piecestore      uploaded
2025-10-08T16:50:23Z    INFO    piecestore      uploaded
2025-10-08T16:50:23Z    INFO    piecestore      uploaded
2025-10-08T16:50:23Z    INFO    piecestore      uploaded
2025-10-08T16:50:24Z    INFO    piecestore      uploaded
2025-10-08T16:50:24Z    INFO    piecestore      uploaded
2025-10-08T16:50:25Z    INFO    piecestore      uploaded
2025-10-08T16:50:25Z    INFO    piecestore      uploaded
2025-10-08T16:50:25Z    INFO    piecestore      uploaded
2025-10-08T16:50:27Z    INFO    piecestore      uploaded
2025-10-08T16:50:29Z    INFO    piecestore      uploaded
2025-10-08T16:50:30Z    INFO    piecestore      uploaded
2025-10-08T16:50:30Z    INFO    piecestore      uploaded
2025-10-08T16:50:30Z    INFO    piecestore      uploaded
2025-10-08T16:50:30Z    INFO    piecestore      uploaded
2025-10-08T16:50:31Z    INFO    piecestore      uploaded
2025-10-08T16:50:31Z    INFO    piecestore      uploaded
2025-10-08T16:50:31Z    INFO    piecestore      uploaded
2025-10-08T16:50:32Z    INFO    piecestore      uploaded
2025-10-08T16:50:32Z    INFO    piecestore      uploaded
2025-10-08T16:50:32Z    INFO    piecestore      uploaded
2025-10-08T16:50:33Z    INFO    piecestore      uploaded
2025-10-08T16:50:33Z    INFO    piecestore      uploaded
2025-10-08T16:50:33Z    INFO    piecestore      uploaded
2025-10-08T16:50:33Z    INFO    piecestore      uploaded
2025-10-08T16:50:33Z    INFO    piecestore      uploaded
2025-10-08T16:50:34Z    INFO    piecestore      uploaded
2025-10-08T16:50:34Z    INFO    piecestore      uploaded
2025-10-08T16:50:34Z    INFO    piecestore      uploaded
2025-10-08T16:50:34Z    INFO    piecestore      uploaded
2025-10-08T16:50:35Z    INFO    piecestore      uploaded
2025-10-08T16:50:35Z    INFO    piecestore      uploaded
2025-10-08T16:50:36Z    INFO    piecestore      uploaded
2025-10-08T16:50:36Z    INFO    piecestore      uploaded
2025-10-08T16:50:36Z    INFO    piecestore      uploaded
2025-10-08T16:50:37Z    INFO    piecestore      uploaded
2025-10-08T16:50:37Z    INFO    piecestore      uploaded
2025-10-08T16:50:37Z    INFO    piecestore      uploaded
2025-10-08T16:50:37Z    INFO    piecestore      uploaded
2025-10-08T16:50:37Z    INFO    piecestore      uploaded
2025-10-08T16:50:39Z    INFO    piecestore      uploaded
2025-10-08T16:50:39Z    INFO    piecestore      uploaded
2025-10-08T16:50:39Z    INFO    piecestore      uploaded
2025-10-08T16:50:39Z    INFO    piecestore      uploaded
2025-10-08T16:50:39Z    INFO    piecestore      uploaded
2025-10-08T16:50:40Z    INFO    piecestore      uploaded
2025-10-08T16:50:40Z    INFO    piecestore      uploaded
2025-10-08T16:50:41Z    INFO    piecestore      uploaded
2025-10-08T16:50:41Z    INFO    piecestore      uploaded
2025-10-08T16:50:41Z    INFO    piecestore      uploaded
2025-10-08T16:50:42Z    INFO    piecestore      uploaded
2025-10-08T16:50:42Z    INFO    piecestore      uploaded
2025-10-08T16:50:42Z    INFO    piecestore      uploaded
2025-10-08T16:50:42Z    INFO    piecestore      uploaded
2025-10-08T16:50:43Z    INFO    piecestore      uploaded
2025-10-08T16:50:43Z    INFO    piecestore      uploaded
2025-10-08T16:50:45Z    INFO    piecestore      uploaded
2025-10-08T16:50:46Z    INFO    piecestore      uploaded
2025-10-08T16:50:46Z    INFO    piecestore      uploaded
2025-10-08T16:50:47Z    INFO    piecestore      uploaded
2025-10-08T16:50:47Z    INFO    piecestore      uploaded
2025-10-08T16:50:47Z    INFO    piecestore      uploaded
2025-10-08T16:50:48Z    INFO    piecestore      uploaded
2025-10-08T16:50:48Z    INFO    piecestore      uploaded
2025-10-08T16:50:48Z    INFO    piecestore      uploaded
2025-10-08T16:50:48Z    INFO    piecestore      uploaded
2025-10-08T16:50:48Z    INFO    piecestore      uploaded
2025-10-08T16:50:49Z    INFO    piecestore      uploaded
2025-10-08T16:50:51Z    INFO    piecestore      uploaded
2025-10-08T16:50:51Z    INFO    piecestore      uploaded
2025-10-08T16:50:51Z    INFO    piecestore      uploaded
2025-10-08T16:50:51Z    INFO    piecestore      uploaded
2025-10-08T16:50:51Z    INFO    piecestore      uploaded
2025-10-08T16:50:51Z    INFO    piecestore      uploaded
2025-10-08T16:50:51Z    INFO    piecestore      uploaded
2025-10-08T16:50:51Z    INFO    piecestore      uploaded
2025-10-08T16:50:51Z    INFO    piecestore      uploaded
2025-10-08T16:50:52Z    INFO    piecestore      uploaded
2025-10-08T16:50:52Z    INFO    piecestore      uploaded
2025-10-08T16:50:52Z    INFO    piecestore      uploaded
2025-10-08T16:50:52Z    INFO    piecestore      uploaded
2025-10-08T16:50:52Z    INFO    piecestore      uploaded
2025-10-08T16:50:52Z    INFO    piecestore      uploaded
2025-10-08T16:50:53Z    INFO    piecestore      uploaded
2025-10-08T16:50:53Z    INFO    piecestore      uploaded
2025-10-08T16:50:53Z    INFO    piecestore      uploaded
2025-10-08T16:50:53Z    INFO    piecestore      uploaded
2025-10-08T16:50:53Z    INFO    piecestore      uploaded
2025-10-08T16:50:54Z    INFO    piecestore      uploaded
2025-10-08T16:50:54Z    INFO    piecestore      uploaded
2025-10-08T16:50:54Z    INFO    piecestore      uploaded
2025-10-08T16:50:54Z    INFO    piecestore      uploaded
2025-10-08T16:50:55Z    INFO    piecestore      uploaded
2025-10-08T16:50:56Z    INFO    piecestore      uploaded
2025-10-08T16:50:56Z    INFO    piecestore      uploaded
2025-10-08T16:50:56Z    INFO    piecestore      uploaded
2025-10-08T16:50:56Z    INFO    piecestore      uploaded
2025-10-08T16:50:58Z    INFO    piecestore      uploaded
2025-10-08T16:50:58Z    INFO    piecestore      uploaded
2025-10-08T16:50:58Z    INFO    piecestore      uploaded
2025-10-08T16:50:58Z    INFO    piecestore      uploaded
2025-10-08T16:50:59Z    INFO    piecestore      uploaded
2025-10-08T16:50:59Z    INFO    piecestore      uploaded
2025-10-08T16:51:00Z    INFO    piecestore      uploaded
2025-10-08T16:51:00Z    INFO    piecestore      uploaded
2025-10-08T16:51:00Z    INFO    piecestore      upload canceled
2025-10-08T16:51:01Z    INFO    piecestore      uploaded
2025-10-08T16:51:01Z    INFO    piecestore      uploaded
2025-10-08T16:51:01Z    INFO    piecestore      uploaded
2025-10-08T16:51:02Z    INFO    piecestore      uploaded
2025-10-08T16:51:02Z    INFO    piecestore      uploaded
2025-10-08T16:51:03Z    INFO    piecestore      uploaded
2025-10-08T16:51:03Z    INFO    piecestore      uploaded
2025-10-08T16:51:04Z    INFO    piecestore      uploaded
2025-10-08T16:51:04Z    INFO    piecestore      uploaded
2025-10-08T16:51:05Z    INFO    piecestore      uploaded
2025-10-08T16:51:07Z    INFO    piecestore      uploaded
2025-10-08T16:51:08Z    INFO    piecestore      uploaded
2025-10-08T16:51:10Z    INFO    piecestore      uploaded
2025-10-08T16:51:11Z    INFO    piecestore      uploaded
2025-10-08T16:51:11Z    INFO    piecestore      uploaded
2025-10-08T16:51:12Z    INFO    piecestore      uploaded
2025-10-08T16:51:12Z    INFO    piecestore      uploaded
2025-10-08T16:51:13Z    INFO    piecestore      uploaded
2025-10-08T16:51:13Z    INFO    piecestore      uploaded
2025-10-08T16:51:14Z    INFO    piecestore      uploaded
2025-10-08T16:51:15Z    INFO    piecestore      uploaded
2025-10-08T16:51:16Z    INFO    piecestore      uploaded
2025-10-08T16:51:16Z    INFO    piecestore      uploaded
2025-10-08T16:51:16Z    INFO    piecestore      uploaded
2025-10-08T16:51:16Z    INFO    piecestore      uploaded
2025-10-08T16:51:16Z    INFO    piecestore      uploaded
2025-10-08T16:51:17Z    INFO    piecestore      uploaded
2025-10-08T16:51:17Z    INFO    piecestore      uploaded
2025-10-08T16:51:17Z    INFO    piecestore      uploaded
2025-10-08T16:51:18Z    INFO    piecestore      uploaded
2025-10-08T16:51:19Z    INFO    piecestore      uploaded
2025-10-08T16:51:20Z    INFO    piecestore      uploaded
2025-10-08T16:51:20Z    INFO    piecestore      uploaded
2025-10-08T16:51:21Z    INFO    piecestore      uploaded
2025-10-08T16:51:21Z    INFO    piecestore      uploaded
2025-10-08T16:51:21Z    INFO    piecestore      uploaded
2025-10-08T16:51:21Z    INFO    piecestore      uploaded
2025-10-08T16:51:22Z    INFO    piecestore      uploaded
2025-10-08T16:51:22Z    INFO    piecestore      uploaded
2025-10-08T16:51:22Z    INFO    piecestore      uploaded
2025-10-08T16:51:24Z    INFO    piecestore      uploaded
2025-10-08T16:51:25Z    INFO    piecestore      uploaded
2025-10-08T16:51:25Z    INFO    piecestore      uploaded
2025-10-08T16:51:26Z    INFO    piecestore      uploaded
2025-10-08T16:51:26Z    INFO    piecestore      uploaded
2025-10-08T16:51:28Z    INFO    piecestore      uploaded
2025-10-08T16:51:28Z    INFO    piecestore      uploaded
2025-10-08T16:51:29Z    INFO    piecestore      uploaded
2025-10-08T16:51:29Z    INFO    piecestore      uploaded
2025-10-08T16:51:30Z    INFO    piecestore      uploaded
2025-10-08T16:51:30Z    INFO    piecestore      uploaded
2025-10-08T16:51:30Z    INFO    piecestore      uploaded
2025-10-08T16:51:30Z    INFO    piecestore      uploaded
2025-10-08T16:51:31Z    INFO    piecestore      uploaded
2025-10-08T16:51:31Z    INFO    piecestore      uploaded
2025-10-08T16:51:33Z    INFO    piecestore      uploaded
2025-10-08T16:51:33Z    INFO    piecestore      uploaded
2025-10-08T16:51:33Z    INFO    piecestore      uploaded
2025-10-08T16:51:33Z    INFO    piecestore      uploaded
2025-10-08T16:51:34Z    INFO    piecestore      uploaded
2025-10-08T16:51:34Z    INFO    piecestore      uploaded
2025-10-08T16:51:34Z    INFO    piecestore      uploaded
2025-10-08T16:51:35Z    INFO    piecestore      uploaded
2025-10-08T16:51:36Z    INFO    piecestore      uploaded
2025-10-08T16:51:37Z    INFO    piecestore      uploaded
2025-10-08T16:51:38Z    INFO    piecestore      uploaded
2025-10-08T16:51:39Z    INFO    piecestore      uploaded
2025-10-08T16:51:39Z    INFO    piecestore      uploaded
2025-10-08T16:51:40Z    INFO    piecestore      uploaded
2025-10-08T16:51:40Z    INFO    piecestore      uploaded
2025-10-08T16:51:40Z    INFO    piecestore      uploaded
2025-10-08T16:51:44Z    INFO    piecestore      uploaded
2025-10-08T16:51:44Z    INFO    piecestore      uploaded
2025-10-08T16:51:46Z    INFO    piecestore      uploaded
2025-10-08T16:51:47Z    INFO    piecestore      uploaded
2025-10-08T16:51:58Z    INFO    piecestore      uploaded
2025-10-08T16:52:00Z    INFO    piecestore      uploaded
2025-10-08T16:52:01Z    INFO    piecestore      uploaded
2025-10-08T16:52:04Z    INFO    piecestore      uploaded
2025-10-08T16:52:04Z    INFO    piecestore      uploaded
2025-10-08T16:52:05Z    INFO    piecestore      uploaded
2025-10-08T16:52:10Z    INFO    piecestore      uploaded
2025-10-08T16:52:59Z    INFO    piecestore      uploaded
2025-10-08T16:53:05Z    INFO    piecestore      uploaded
2025-10-08T16:53:06Z    INFO    piecestore      uploaded
2025-10-08T16:55:21Z    INFO    piecestore      uploaded
2025-10-08T17:55:27Z    INFO    piecestore      uploaded
2025-10-08T19:01:59Z    INFO    piecestore      uploaded
2025-10-08T20:03:33Z    INFO    piecestore      uploaded

Between 16:55:21Z and 20:03:33Z only 2 uploads!
Then restart and it resumed immediately at its original frequency:

2025-10-08T20:47:08Z    INFO    piecestore      upload canceled
2025-10-08T20:47:08Z    INFO    piecestore      uploaded
2025-10-08T20:47:08Z    INFO    piecestore      uploaded
2025-10-08T20:47:10Z    INFO    piecestore      uploaded
2025-10-08T20:47:11Z    INFO    piecestore      uploaded
2025-10-08T20:47:11Z    INFO    piecestore      uploaded
2025-10-08T20:47:13Z    INFO    piecestore      uploaded
2025-10-08T20:47:14Z    INFO    piecestore      uploaded
2025-10-08T20:47:14Z    INFO    piecestore      uploaded
2025-10-08T20:47:15Z    INFO    piecestore      uploaded
2025-10-08T20:47:15Z    INFO    piecestore      uploaded
2025-10-08T20:47:16Z    INFO    piecestore      uploaded
2025-10-08T20:47:18Z    INFO    piecestore      uploaded
2025-10-08T20:47:18Z    INFO    piecestore      uploaded
2025-10-08T20:47:19Z    INFO    piecestore      uploaded
2025-10-08T20:47:20Z    INFO    piecestore      uploaded
2025-10-08T20:47:22Z    INFO    piecestore      uploaded
2025-10-08T20:47:22Z    INFO    piecestore      uploaded
2025-10-08T20:47:23Z    INFO    piecestore      uploaded
2025-10-08T20:47:24Z    INFO    piecestore      uploaded
2025-10-08T20:47:24Z    INFO    piecestore      uploaded
2025-10-08T20:47:24Z    INFO    piecestore      uploaded
2025-10-08T20:47:24Z    INFO    piecestore      uploaded
2025-10-08T20:47:25Z    INFO    piecestore      uploaded
2025-10-08T20:47:27Z    INFO    piecestore      uploaded
2025-10-08T20:47:29Z    INFO    piecestore      uploaded
2025-10-08T20:47:29Z    INFO    piecestore      uploaded
2025-10-08T20:47:31Z    INFO    piecestore      uploaded
2025-10-08T20:47:32Z    INFO    piecestore      upload canceled
2025-10-08T20:47:32Z    INFO    piecestore      uploaded
2025-10-08T20:47:32Z    INFO    piecestore      uploaded
2025-10-08T20:47:33Z    INFO    piecestore      uploaded
2025-10-08T20:47:33Z    INFO    piecestore      uploaded
2025-10-08T20:47:33Z    INFO    piecestore      uploaded
2025-10-08T20:47:34Z    INFO    piecestore      uploaded
2025-10-08T20:47:35Z    INFO    piecestore      uploaded
2025-10-08T20:47:35Z    INFO    piecestore      uploaded
2025-10-08T20:47:35Z    INFO    piecestore      uploaded
2025-10-08T20:47:36Z    INFO    piecestore      uploaded
2025-10-08T20:47:37Z    INFO    piecestore      uploaded
2025-10-08T20:47:38Z    INFO    piecestore      uploaded
2025-10-08T20:47:38Z    INFO    piecestore      uploaded
2025-10-08T20:47:39Z    INFO    piecestore      uploaded
2025-10-08T20:47:39Z    INFO    piecestore      uploaded
2025-10-08T20:47:39Z    INFO    piecestore      uploaded
2025-10-08T20:47:39Z    INFO    piecestore      uploaded
2025-10-08T20:47:39Z    INFO    piecestore      uploaded
2025-10-08T20:47:41Z    INFO    piecestore      uploaded
2025-10-08T20:47:41Z    INFO    piecestore      uploaded
2025-10-08T20:47:41Z    INFO    piecestore      uploaded
2025-10-08T20:47:41Z    INFO    piecestore      uploaded
2025-10-08T20:47:42Z    INFO    piecestore      uploaded
2025-10-08T20:47:42Z    INFO    piecestore      uploaded
2025-10-08T20:47:44Z    INFO    piecestore      uploaded
2025-10-08T20:47:44Z    INFO    piecestore      uploaded
2025-10-08T20:47:45Z    INFO    piecestore      uploaded
2025-10-08T20:47:46Z    INFO    piecestore      uploaded
2025-10-08T20:47:46Z    INFO    piecestore      uploaded
2025-10-08T20:47:47Z    INFO    piecestore      uploaded
2025-10-08T20:47:48Z    INFO    piecestore      uploaded
2025-10-08T20:47:50Z    INFO    piecestore      uploaded
2025-10-08T20:47:50Z    INFO    piecestore      uploaded
2025-10-08T20:47:50Z    INFO    piecestore      uploaded
2025-10-08T20:47:50Z    INFO    piecestore      upload canceled
2025-10-08T20:47:50Z    INFO    piecestore      uploaded
2025-10-08T20:47:52Z    INFO    piecestore      uploaded
2025-10-08T20:47:52Z    INFO    piecestore      uploaded
2025-10-08T20:47:52Z    INFO    piecestore      uploaded
2025-10-08T20:47:54Z    INFO    piecestore      uploaded
2025-10-08T20:47:54Z    INFO    piecestore      uploaded
2025-10-08T20:47:55Z    INFO    piecestore      upload canceled
2025-10-08T20:47:56Z    INFO    piecestore      uploaded
2025-10-08T20:47:57Z    INFO    piecestore      uploaded
2025-10-08T20:47:58Z    INFO    piecestore      uploaded
2025-10-08T20:47:58Z    INFO    piecestore      uploaded
2025-10-08T20:47:58Z    INFO    piecestore      uploaded
2025-10-08T20:47:58Z    INFO    piecestore      uploaded
2025-10-08T20:47:59Z    INFO    piecestore      uploaded
2025-10-08T20:47:59Z    INFO    piecestore      uploaded
2025-10-08T20:48:00Z    INFO    piecestore      upload canceled
2025-10-08T20:48:00Z    INFO    piecestore      uploaded
2025-10-08T20:48:00Z    INFO    piecestore      uploaded
2025-10-08T20:48:00Z    INFO    piecestore      uploaded
2025-10-08T20:48:01Z    INFO    piecestore      upload canceled
2025-10-08T20:48:01Z    INFO    piecestore      uploaded
2025-10-08T20:48:01Z    INFO    piecestore      uploaded
2025-10-08T20:48:02Z    INFO    piecestore      uploaded
2025-10-08T20:48:02Z    INFO    piecestore      uploaded
2025-10-08T20:48:02Z    INFO    piecestore      uploaded
2025-10-08T20:48:03Z    INFO    piecestore      uploaded
2025-10-08T20:48:03Z    INFO    piecestore      uploaded
2025-10-08T20:48:03Z    INFO    piecestore      uploaded
2025-10-08T20:48:04Z    INFO    piecestore      uploaded
2025-10-08T20:48:04Z    INFO    piecestore      uploaded
2025-10-08T20:48:04Z    INFO    piecestore      uploaded
2025-10-08T20:48:04Z    INFO    piecestore      uploaded
2025-10-08T20:48:05Z    INFO    piecestore      uploaded
2025-10-08T20:48:05Z    INFO    piecestore      uploaded
2025-10-08T20:48:05Z    INFO    piecestore      uploaded
2025-10-08T20:48:05Z    INFO    piecestore      uploaded
2025-10-08T20:48:06Z    INFO    piecestore      uploaded
2025-10-08T20:48:07Z    INFO    piecestore      uploaded
2025-10-08T20:48:08Z    INFO    piecestore      uploaded
2025-10-08T20:48:08Z    INFO    piecestore      uploaded
2025-10-08T20:48:08Z    INFO    piecestore      uploaded
2025-10-08T20:48:10Z    INFO    piecestore      uploaded
2025-10-08T20:48:11Z    INFO    piecestore      uploaded
2025-10-08T20:48:11Z    INFO    piecestore      uploaded
2025-10-08T20:48:11Z    INFO    piecestore      uploaded
2025-10-08T20:48:11Z    INFO    piecestore      uploaded
2025-10-08T20:48:12Z    INFO    piecestore      uploaded
2025-10-08T20:48:13Z    INFO    piecestore      uploaded
2025-10-08T20:48:14Z    INFO    piecestore      uploaded
2025-10-08T20:48:14Z    INFO    piecestore      uploaded
2025-10-08T20:48:14Z    INFO    piecestore      uploaded
2025-10-08T20:48:14Z    INFO    piecestore      uploaded
2025-10-08T20:48:15Z    INFO    piecestore      uploaded
2025-10-08T20:48:15Z    INFO    piecestore      uploaded
2025-10-08T20:48:16Z    INFO    piecestore      uploaded
2025-10-08T20:48:17Z    INFO    piecestore      uploaded
2025-10-08T20:48:17Z    INFO    piecestore      uploaded
2025-10-08T20:48:18Z    INFO    piecestore      uploaded
2025-10-08T20:48:19Z    INFO    piecestore      uploaded
2025-10-08T20:48:19Z    INFO    piecestore      uploaded
2025-10-08T20:48:20Z    INFO    piecestore      uploaded
2025-10-08T20:48:20Z    INFO    piecestore      uploaded
2025-10-08T20:48:21Z    INFO    piecestore      uploaded
2025-10-08T20:48:21Z    INFO    piecestore      uploaded
2025-10-08T20:48:21Z    INFO    piecestore      uploaded
2025-10-08T20:48:21Z    INFO    piecestore      uploaded
2025-10-08T20:48:21Z    INFO    piecestore      uploaded
2025-10-08T20:48:22Z    INFO    piecestore      uploaded
2025-10-08T20:48:22Z    INFO    piecestore      uploaded
2025-10-08T20:48:22Z    INFO    piecestore      uploaded
2025-10-08T20:48:23Z    INFO    piecestore      uploaded
2025-10-08T20:48:24Z    INFO    piecestore      uploaded
2025-10-08T20:48:24Z    INFO    piecestore      uploaded
2025-10-08T20:48:24Z    INFO    piecestore      uploaded
2025-10-08T20:48:24Z    INFO    piecestore      uploaded
2025-10-08T20:48:26Z    INFO    piecestore      uploaded
2025-10-08T20:48:26Z    INFO    piecestore      uploaded
2025-10-08T20:48:27Z    INFO    piecestore      uploaded
2025-10-08T20:48:28Z    INFO    piecestore      uploaded
2025-10-08T20:48:29Z    INFO    piecestore      uploaded
2025-10-08T20:48:29Z    INFO    piecestore      uploaded
2025-10-08T20:48:29Z    INFO    piecestore      uploaded
2025-10-08T20:48:29Z    INFO    piecestore      uploaded

[Alexey]

Are they on piecestore or hashstore (I believe, that the name of the service has not changed, so it will print as piecestore even if it’s on hashstore)?

[jammerdan]

@Alexey

Are they on piecestore or hashstore

On hashtstore. When I had this problem before, of course it was piecestore but it sorted itself out at the end so that it was no longer an issue.
Now with hashstore I am seeing this issue more frequently again.

[Vadim]

have you tried delete databases? May be error it there, node think it is full, depside dashboard show. It absolutely possible that dashboard and node take data from different places.

[jammerdan]

@Vadim

have you tried delete databases?

No, not yet. I’d be surprised if that would be the case that a restart would fix it as it a restart does not alter the databases.

However I may try it on this one, because it just happened with another node. Needless to say that there is plenty of room on the disk left.

2025-10-09T19:45:38Z    INFO    piecestore      uploaded
2025-10-09T19:45:39Z    INFO    piecestore      uploaded
2025-10-09T19:45:39Z    INFO    piecestore      uploaded
2025-10-09T19:45:44Z    INFO    piecestore      uploaded
2025-10-09T19:45:44Z    INFO    piecestore      upload canceled
2025-10-09T19:45:44Z    INFO    piecestore      uploaded
2025-10-09T19:45:44Z    INFO    piecestore      uploaded
2025-10-09T19:45:47Z    INFO    piecestore      upload canceled
2025-10-09T19:45:48Z    INFO    piecestore      uploaded
2025-10-09T19:45:48Z    INFO    piecestore      upload canceled
2025-10-09T19:45:49Z    INFO    piecestore      uploaded
2025-10-09T19:45:49Z    INFO    piecestore      uploaded
2025-10-09T19:45:51Z    INFO    piecestore      uploaded
2025-10-09T19:45:52Z    INFO    piecestore      uploaded
2025-10-09T19:45:52Z    INFO    piecestore      uploaded
2025-10-09T19:45:53Z    INFO    piecestore      uploaded
2025-10-09T19:45:53Z    INFO    piecestore      uploaded
2025-10-09T19:45:55Z    INFO    piecestore      uploaded
2025-10-09T19:45:56Z    INFO    piecestore      uploaded
2025-10-09T19:45:57Z    INFO    piecestore      uploaded
2025-10-09T19:45:59Z    INFO    piecestore      uploaded
2025-10-09T19:45:59Z    INFO    piecestore      uploaded
2025-10-09T19:46:03Z    INFO    piecestore      uploaded
2025-10-09T19:46:03Z    INFO    piecestore      uploaded
2025-10-09T19:46:04Z    INFO    piecestore      uploaded
2025-10-09T19:46:05Z    INFO    piecestore      uploaded
2025-10-09T19:46:08Z    INFO    piecestore      uploaded
2025-10-09T19:46:13Z    INFO    piecestore      uploaded
2025-10-09T19:46:14Z    INFO    piecestore      uploaded
2025-10-09T19:46:15Z    INFO    piecestore      uploaded
2025-10-09T19:46:18Z    INFO    piecestore      uploaded
2025-10-09T19:46:20Z    INFO    piecestore      uploaded
2025-10-09T19:46:35Z    INFO    piecestore      uploaded
2025-10-09T19:46:41Z    INFO    piecestore      uploaded
2025-10-09T19:47:13Z    INFO    piecestore      upload canceled
2025-10-09T20:12:20Z    INFO    piecestore      uploaded
2025-10-09T21:22:08Z    INFO    piecestore      uploaded
2025-10-09T22:23:32Z    INFO    piecestore      uploaded
2025-10-09T23:25:13Z    INFO    piecestore      uploaded
2025-10-10T00:32:27Z    INFO    piecestore      uploaded
2025-10-10T01:50:54Z    INFO    piecestore      uploaded

Now with this node, even a restart did not resume uploads.

[Alexey]

Could you please give me a NodeID, if it’s still happening now, because if you fix, I wouldn’t be able to see anything unusual.

[jammerdan]

@Alexey

because if you fix

There seems no real fix. Removing databases did not resolve it. Only way is to increase available space.
To be clear: Disk has space left. Node dashboard/API is showing tens of GBs free and no overusage.
I have increased the available space to an absurdly high value and after a restart it received ingress again immediately. Reducing the space again to the previous value made ingress halt again.
I have added now some additional space in the config and it is running receiving ingress.

Could it be that some data, either used space or free space calculation is not correct anymore with hashstore? What I am wondering is, if the node still thinks it has a lot of space left, where does the signal for the satellite come from so it stop uploads?

[Alexey]

Hashstore uses a probabilistic model to estimate a used space and calculate a free space in the allocation.
Similar to suggested there:

On a statistical approach for the used space file walker

We’ve recently got a large number of problems that require a full run of a used space file walker, and we have a large number of posts complaining about its duration. We have also seen that its duration is also a source of problems—not all changes to storage while the file walker is running are accounted. One key observation that was made is that the measured values of disk space do not have to be fully accurate, and it should be acceptable to have a decent approximation. This suggests that a s…

Seems the easy fix is to enable a dedicated disk, or use overprovisioning, the node would stop at 5GB of free space on the disk anyway. But it’s an emulation of a dedicated disk in this case, so why to make several hops?

[jammerdan]

@Alexey

Hashstore uses a probabilistic model to estimate a used space and calculate a free space in the allocation.

So used space and free space are only a guess?

Maybe that wouldn’t be so bad, if it would not send out contrary signals. To the satellite it signals it is full, to the SNO it signals there is plenty of space left. This does not make sense.

[jammerdan]

Ok, that’s really bad. I have just checked another node where the dashboard was telling me there is still like 250GB free. In reality, no ingress on that node anymore. After increasing space, ingress is back.

[jammerdan]

And found another node lying at me that it has almost 400 GB space left, but ingress halted at some time today.

[arrogantrabbit]

Looks like a dashboard bug. It shall show space available as minimum of the two.

[jammerdan]

@arrogantrabbit

Looks like a dashboard bug.

I cannot tell which one is the source of truth. Either way, there shouldn’t be contradicting signals.

I have just found 4 more nodes that have not received ingress since the start of the day despite free space showing hundreds of GB as remaining. Now I have to check if increasing space fixes it.

[Alexey]

This is perhaps related to some suspected bug in the single node dashboard:
https://forum.storj.io/t/hashstore-migration-guide/30658/159?u=alexey

Could you please check with the multinode dashboard this behavior:
https://forum.storj.io/t/hashstore-migration-guide/30658/154?u=alexey

[Vadim]

I found now that 1 of my node gone out of sense completely

How it cold be overused and free space at same time?
I deleted prefix db but it not helped.
HDD is totaly full

Update> Even after deletion of all db, picture is the same, so looks like node math is broken

[jammerdan]

A new day begins and again there are several nodes that stopped ingress despite appearing to have enough free space left.

@Alexey

Could you please check with the multinode dashboard this behavior:

I was able to do that now:
Node dashboard:

Multinode dashboard (same node):

If the multinode dashboard is correct no wonder the node believes it is full and does not get any ingress. However with 2Tb allocated and 1.64TB in use, the 6.25GB as free remaining space is not correct either.
It seems like the node dashboard shows the correct value for remaining but the value that the multinode dashboard uses takes precedence.

Anyways, something seems totally off.

[Alexey]

Then it’s exactly the problem, which is recently discovered:

Single node and multinode dashboards shows different values after migration to hashstore

How is this possible? There is free space and also overused: [image] It is a single node’s dashboard which is migrated to hasstore.