Deleted pieces will be sent to trash by default

[jammerdan]

I see. Does this mean these deletions, almost all of them, are handled by Garbage Collection then? Meaning that final deletion from trash is performed by Garbage Collection?

[Alexey]

Currently yes. I do not know is it permanent or not.

[jammerdan]

Does Garbage Collection run 1 time a day only?
I am currently trying to investigate an issue of extremely high load on more than one node in the morning. Could it be that Garbage Collection kicks off around 0 UTC and does a mass delete then?

[Alexey]

storagenode setup --help | grep collector
--collector.interval duration                              how frequently expired pieces are collected (default 1h0m0s)

[jammerdan]

Is this the setting for garbage collection?

[Alexey]

There is no other options as far as I can see.

[jammerdan]

I had seen this setting, but was unsure as is referring to expired pieces and not deleted pieces. So I am not sure if this is the interval that moves pieces to trash or the interval that deletes pieces from trash.

[BrightSilence]

@jammerdan

Does this mean these deletions, almost all of them, are handled by Garbage Collection then?

@Alexey

Currently yes

That doesn’t sound right. I definitely see normal deletes still in my node logs. It looks like they are still processed when the delete happens, but are just moved to trash. I’m pretty sure GC still only captures removals that were somehow missed.

@Alexey

There is no other options as far as I can see.

I think garbage collection timing is determined by the satellite. The node would need to receive a bloom filter from the satellite in order to proceed. That takes some processing time on the satellite, so effectively the satellite determines when the node can do clean up.

[jammerdan]

Any clue what storagenode operation could cause load early morning?

[BrightSilence]

Not from the top of my head, but you can use this to find out.

Guide to debug my storage node, uplink, s3 gateway, satellite

We have integrated a package called monkit in all storage nodes. It is very usefull for debuging. By default it will open a random port. In order to connect to the debug port you will need to: Add debug.addr: ":5999" to the config file. Now the port is fixed and will not change on every restart. Add -p 127.0.0.1:5999:5999 to the docker run command. Ready for action! You can request the following informations: curl localhost:5999/mon/funcs Find out how long a function like GetExpired need…

/mon/ps end point to see what’s running
/mon/funcs could also get you info on runtime of specific processes.

[jammerdan]

Wow, first time I am seeing this. Yes maybe that will help to find out what is going on.

[jammerdan]

If moving to trash is and may remain the default behavior, is this still a copy transaction instead of a move like it was mentioned here, resulting in more disk i/o than necessary:

Moving data to the trash folder results in a lot of disk IO

It still is the same - 31780 be/4 root 6.68 M/s 6.75 M/s 0.00 % 34.54 % ./storagenode run - Moving to trash is done cp old new; rm old instead of mv old new, which may be OK for a single file (still, WHY) but not OK for mass moves like now.