Moving data to the trash folder results in a lot of disk IO

It looks like moving data to the “trash” folder is done by copying the file there and then deleting the original. This results in a lot of disk IO because all of that data needs to be copied to the new location.

Why this is not done as a simple “move” operation that would just change the file system metadata and be much faster?

2 Likes

In my experience copy then delete is safer than cut/paste in case pc crashes. I think same logic might be applied here.

I’m also getting a lot of disk IO which I assume is for the same reason. I’m also assuming that in a real world situation the data moved to trash would be more of a steady stream so would have less impact?

Thoughts?

1 Like

My nodes on BTRFS collected garbage for the first time about 4-5 days. It looks like it will be repeated every time. Read 2 MB = 40-60 MB Write

1 Like

This is how it looked for my node:

I am hoping fro an answer from Storj - was this done on purpose (and if so, why) or did it somehow happened (which would mean that hopefully it would be fixed in the future)?

There probably won’t be 180GB of garbage moved at once in production though, so even if this stays, maybe it won’t be so bad.

Is most your processing bound to a single CPU too?

@nerdatwork Umm, no. If the PC crashes while copying you have a half-written chunk of garbage lying around. Modern logging file systems don’t break when a cross-directory rename gets interrupted by a power failure.
Even if they did, the window for damage would be a lot smaller (also, it’s a file that’s moved to the trash anyway, so the damage is kindof very limited).
Copying files is not free, the superfluous copy slows down the node.

3 Likes

This seems to be a really slow HD, 20MB/s?

It was 20MB/s read, 40MB/s write. Also, hard drives are usually not that optimized copying data from one place of the drive to another (I guess I could mount separate array as the “trash” folder, but it could lead to complications.

not understand, why need moove files at all, just mark them as trash and no need to moove them.
After when need to delet just delete them.

2 Likes

I have also noticed this as well putting unnecessary load on the hard drive to copy files to another folder. Should be a more efficient way to do this is it possible to just mark a file to be deleted instead of copying to another folder. If someone was running this node on there windows machine its going to slow there system down alot. Its going to create a unpleasant performance drop over all to there PC.

1 Like

Same here, thinking about wiping nodes if this won’t be fixed in future…

I don’t know any filesystem which has a permission bit “mark as deleted”. Even if there were, it has to work on all filesystem the same way.

I don’t understand why the files are not moved. A move within the same filesystem is an atomic operation, at least under *nix. There is no risk of data loss.

3 Likes

no one speak about data loss, do not know how it made but looks like it not take atomic time, like if it copied entierly. But i might be wrong.
Under mark, i meant like change extension from sj1 to trash

1 Like

It still is the same -

31780 be/4 root        6.68 M/s    6.75 M/s  0.00 % 34.54 % ./storagenode run -

Moving to trash is done cp old new; rm old instead of mv old new, which may be OK for a single file (still, WHY) but not OK for mass moves like now.

2 Likes

Or even .sj1.trash

Someone remove the 20 char thing already.

seems like a very inefficient & expensive way to do this. 18 hours to send 1.2T to the trashbin

Worthy to note that this was also on a 10Gb iSCSI to a 4xRAID10 with SSD caching.

I was hit by this recently as well, which is even more mysterious given that my nodes have been up pretty much 24/7 for weeks, so there shouldn’t even be a need for any GC – my nodes wouldn’t’ve missed any delete instructions.

This approach to storing trash is causing thrashing just like others have experienced. Please fix this.

This was a bit of a special scenarion with a large amount of zombie segments being cleaned up at the same time on the satellites end. Normally GC would only have to deal with very small amounts.

1 Like

But why do it like that? What’s wrong with the mv function provided by the OS?

6 Likes