I will try to explain in a different way.
I understand, that you set 23TB in your STORAGE option, but your volume is actually have only 21.1TB and there are only 100GB free.
When you start the container, it will check the actual free space (in your case it’s 100GB + 21TB of used space) and warn you, that you trying to allocate more (23TB) than you have (21.1TB), so node set the allocation to 21.1TB instead of 23TB and will not update it until the next restart.
From the other side, the node checks the free space in the allocation (21.1TB) on each upload (and report this for the each piece in your logs), but doesn’t update the allocation, even if you added 100GB more, so the node will stop ingress if there is less than 500MB of free space in the allocation (21.1TB), not on the disk (21.2TB).
You have 2 exits from this situation:
- or use thin provisioned LVM volumes (you will “lie” to the OS that you have 23TB, while add only 21.1TB of the actual space), but you need to closely monitor a usage, otherwise if the volume would be full, there is a probability of the volume corruption.
- or restart the node after you extended the normal volume with additional space.
Please also note, the thin provisioned volumes works noticeable slower than normal volumes (the same is true for COW enabled volumes, i.e. when you made a snapshot, but there are tricks as well).