Alexey
March 29, 2024, 8:19am
9
Это широко обсуждается в уже приведённом треде. Кратко - процессы filewalkers должны завершаться успешно для каждого сателлита до перезапуска узла. Если сканирование не завершилось и узел был перезапущен (по любой причине), уже накопленная информация теряется и узел начинает сканирование с нуля.
Looks to be a bug, garbage_collection_pieces_count and garbage_collection_pieces_skipped are swapped when lazy file walker is used (default).
This should fix the problem:
https://review.dev.storj.io/c/storj/storj/+/12076
Until that, you can check the value of ..._skipped
Not: it’s still not very useful metrics, as the value is updated after any execution. If two satellites have totally different values, the first one may not be visible, only by max/min fields.
I am considering to add the s…
Related issue:
TLDR; with high number of segments on one node, the bloom filter is less effective as it should be. Fix is on the way.
The easiest way to check if you are affected is counting the blobs in the storage directory ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa or v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa folders (US or EU satellite ).
If you have at least 14M blobs (in one satellite folder) and see big discrepancies, feel free to post your numbers as comment on …
В качестве промежуточного решения можно отключить lazy mode:
Please try to disable lazy filewalker, it will turn on the normal (with a normal priority):
# run garbage collection and used-space calculation filewalkers
# as a separate subprocess with lower IO priority (default true)
pieces.enable-lazy-filewalker: false
save the config and restart the node. If you use docker, you may provide this flag after the image name: --pieces.enable-lazy-filewalker=false