0.28.4 Disk activity is crazy

Since new release a few mins ago my disk activity is going crazy. I wasn’t expecting this from the release notes, is all ok?

You can downgrade to v0.28.2. It will have the same disk activity. If you want to see which process it is you might want to take a look at: Guide to debug my storage node

Disc activity is big, as you geting and loading lot of data.

I think that’s garbage collection… Mine spent the last day or two and I watched the trash folder increase in size. It does eventually end. Just have to let it run. I don’t think it impacted writing and reading of files. all seemed to upload and download as usual.

During today i have 80 GB of bandwidth on veetted node. and day is not ended yet

My node suffered from the same craziness: 7 hours of ≈7MB/s reads at ≈400 reads/s. Now it’s over and I’m sad I had no chance to look at what was the culprit function :frowning:

High disk activity via StorJ would mean more upload and download data from your SNO.

My monitoring suggests there’s no change in the uplink/downlink traffic. Also 400 qps reads (!) is not what I would expect from downloads.

$ curl localhost:7777/mon/funcs
...
[8151419892526493396] storj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite
  parents: 9111472579214935817
  current: 0, highwater: 1, success: 1, errors: 0, panics: 0
  success times:
    0.00: 6h47m6.77714944s
    0.10: 6h47m6.77714944s
    0.25: 6h47m6.77714944s
    0.50: 6h47m6.77714944s
    0.75: 6h47m6.77714944s
    0.90: 6h47m6.77714944s
    0.95: 6h47m6.77714944s
    1.00: 6h47m6.77714944s
    avg: 6h47m6.776749459s
    ravg: 6h47m6.77714944s

So it’s SpaceUsedTotalAndBySatellite. I wonder if there’s a way to make this process less stressing to the hardware.

Not necessarily, I find upload/download data has little impact on disk usage. I find the garbage cleanup to be really intense as it has to scan the drive for all the files it is cleaning up and that’s a disk intensive process with high file counts and 1000+ directories. 300GB got moved to trash and that took about a day and a half. Also during node startup there is a lot of disk activity as storj loads the file database. Once that’s over it seems to calm down. I’m running the windows version which is much less disk intensive than the docker version was on my system.

These are long overdue cleanups and I suspect in the future with regular file deletions disk usage should drop dramatically and cleanups will be shorter in time.

3 Likes