Some of you might have noticed a higher amount of garbage. This has multiple reasons. Satellites didn’t send any bloom filters for a week. The job that generates the boom filter was running but couldn’t upload the bloom filters to storj. The shared project we are using for this was hitting the li…

Turns out there is also a bug on the storage node dashboard. Left side:_ ~3 740 000 000 000 displayed as 3.41 TB (base 1024) Right side: ~3 990 000 000 000 displayed as 3.99 TB (base 1000) Because the dashboard shows the numbers wrong it looks like 580 GB unpaid space. In reality the unpaid space…

Unfortunately not the case for the node I’ve been monitoring for this. It seems this run cleaned up even less (which is to be expected since the node still receives new data, so the max size will get less sufficient over time). I won’t bother you too much, so I’ll just post my results after the last…

[image] BrightSilence: Let me know if you need my node ID to test the large bloom filters. Yep, thanks the help. We will update the forum when we are ready for testing. First we need to generate 10Mb files. The current generator uses ~100Gb memory and requires >5 days to be completed. We nee…

I took a while to run the grep, but mine is also smaller. Same 4099953 size. So I don’t think it means the smaller one was sufficient. 2024-04-22T02:06:19Z INFO retain Prepared to run a Retain request. {"Process": "storagenode", "cachePath": "config/retain", "Created Before": "2024-04-…

Can this work be spread to a network of computers? Like in Folding @ home? Or it can be run only on the satellites servers?

It can run only on the satellite. Well not exactly on the same machine. The point is it has access to sensitive information and requires a trusted environment. We can split the work over multiple servers and scale it that way.

If it helps anyone, on my oldest node I see this; the retain chore ran for 7.5 hours on 12TB node, Synology, 18GB RAM, Exos drive, node v.1.101.3 2024-04-21T23:35:21Z INFO retain Prepared to run a Retain request. {"Process": "storagenode", "cachePath": "config/retain", "Created Before": "2024-04-10…

[image] elek: First we need to generate 10Mb files. The current generator uses ~100Gb memory and requires >5 days to be completed. We need to find out the right resource settings… So it takes some time. You’ve nerdsniped me. Please admit that this was your goal. Some sample code for computin…

[image] BrightSilence: [image] I was wondering why the garbage collected more than a week ago is still in trash, but then I remembered reading the migration to the new trash structure would initially put all trash in a single folder with the current date. This was kind of an unfortunate timin…

Updated my code with a more complex (better, but uglier) approach to concurrency. Tested it on AWS’ i3en.24xlarge, managed to run the computations for simulated 25k nodes and 20400631382 pieces (approx. 10% of what us1.storj.io has now). Runtime was 21 minutes. Memory use below 5 GB for the whole pr…

[image] Toyoo: Runtime was 21 minutes. Memory use below 5 GB for the whole process, though the rest of 700 GB was used for caches and buffers, likely helped. I/O was at ~7 GB/s at the peak suggesting a single good consumer NVMe drive would do. I only understand about 25% of what your talking …

Storj Community Forum (official)

Current situation with garbage collection

Node Operators

garbage-collector, storagenode

show post in topic

Home
Categories
Guidelines
Terms of Service
Privacy Policy

Powered by Discourse, best viewed with JavaScript enabled