All my nodes have started showing this warn (endless messages) as soon as they were updated to the new version. All of them have 100% audit score, i’ve never seen this before. Anyone else have this messages ? 2024-07-03T22:06:27Z WARN collector unable to delete piece {“Process”: “sto…

I understand the behaviour in general. But: Something must have changed as the logfiles are getting spammed MASSIVELY since the last update, which wasn’t the case before. So my read is - something changed, not a client delete behaviour. It started on both nodes after the update to v1.114.6. - and it…

The fix to log level - to make if INFO and not print stack.

[image] tankmann: Something must have changed as the logfiles are getting spammed MASSIVELY since the last update, which wasn’t the case before. It sounds like the increase in log activity may be related to the new TTL flat file storage that was automatically enabled in v1.113. With the recen…

Yeah thought the same… hence also let it run. It actually stopped now, it just took way way way way longer than expected: [image] More then 3 days, almost 80 hours. Back to normal on one node. The other one is still going through it…

two nodes produced 100 GB of WARN logs in a short period of time. This happened after update to 1.114.6: {“L”:“WARN”,“T”:“2024-10-18T22:00:54Z”,“N”:“collector”,“M”:“unable to delete piece”,“Process”:“storagenode”,“Satellite ID”:“12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”,“Piece ID”:“PK5X…

Storagenode file deletion worker threads: [image]

Thank you! It’s clear. I used to have a 60 GB dataset in bzip2 that after recompressing into xz went well below 10 GB. Not here though, I strongly suspect you store 2 GB of piece IDs in that file :rofl:

[image] arrogantrabbit: The fix to log level - to make if INFO and not print stack. [image] edo: I agree it might be worth considering adjusting the log level for these specific messages, as suggested by many already, but a more extensive fix doesn’t appear to be needed at this stage. P…

You may use a custom log level for this chore, see --log.custom-level or change it temporary via debug port: Guide to debug my storage node, uplink, s3 gateway, satellite .

This is an old problem. I use this log custom level to get rid of it. [image] --log.custom-level FAQ New version: docker run ... \ --log-driver json-file \ --log-opt max-size=10m \ --log-opt max-file=3 \ --name storagenode storjlabs/storagenode:latest \ --log.level=…

1.107.3 Error: collector unable to delete pieces

Node Operators FAQ

snorkel October 19, 2024, 5:27am 41

This is an old problem. I use this log custom level to get rid of it.