I recently switched my RPi 4B’s system disk to a dedicated SSD because I crashed my 3rd SD card in 3 years, so I thought it was time to have a more robust solution ^^’
I also took this opportunity to re-install everything on Raspbian 64bit (had been on 32bit so far).
Everything seems to run smoothly. In fact, the filewalker seems to be WAY faster now (a few hours instead of the previously agonizing 36+ hours of file browsing…)
That’s great, but I was wondering what could explain such an improvement?
Storage disks did not change, the only changes are:
Firmware update, so the RPi 4B can boot from USB
An SSD instead of the SD card for the Operating System
Raspbian 64bit
(software updates I guess, as I reinstalled everything from scratch)
Did the node software improve the filewalker mechanism recently?
don’t think the other stuff helped much… because the filewalker needs to read all the metadata on the hdd… but 32bit to 64bit sure makes some stuff run a lot faster, so that can’t have hurt…
also the filewalker is a bit hit and miss, sometimes it seems to run forever and other times it takes no time at all…
as if it can resume or update based upon changes since last time it ran or something…
i duno, aside from that its not easy to get wise on the filewalker.
generally you see the filewalker finishing faster if it has been run recently … or maybe if it has run and hasn’t finished… not sure which tho
or maybe both
There was some change where the file walker used to have to open all files for reading and read some header, and now it’s sufficient to stat() for file system metadata. But this was a long time ago.
Curiously I can also report that after I recently added some RAM (changed from 2GB to 16GB) to my Storj box (hosting ~13TB of data), the file walker got a decent speed-up. Can’t measure it exactly without log entries though. So maybe in your case it is faster swap space that helps?
I’ve been told this yeah, not sure how it could make I/O operations go faster though but… maybe ^^
That makes sense I think, because most of the metadata related to the file system must still be in RAM.
Adding RAM is always a good thing as Linux knows what to do with it and caches many usefull stuff, like file system metadata, right?
I have no swap on my RPi (it’s disabled). So unless Linux does use the OS disk for caching some other stuff via other means, I don’t think the SSD is helping in anyway during the filewalker process, but I might be totally wrong
I’ve seen no-one talking about such a long filewalker process, I’m not sure what was wrong with my setup. I do have extremely cheap hardware (the worst being a 5TB 2.5" SMR HDD), but still…
And even though I’m glad it seems like everything works better now, I really don’t get what fixed it
Yeah I cant say I have extremely expensive hardware either, But I did switch from SDs to a SSD on rpi4s and I used Ubuntu 64bit. My rpi4 kept killing my SD cards so I had to move to a SSD for long term. I did post about it a while back though but no one seemed to agree with me. Apperently I was the only one having those issues with SD cards.
Oh! Well, I can’t remember if I was part of the ones disagreeing, but I do agree now
I hope switching to SSD & 64bit OS will make things better for me too. It has already it seems
ill just say it again… there should be SD cards that actually use regular NAND cells
but they also cost a lot more… ofc it might be a temp issue… NAND and flash cells in general doesn’t like to much heat.
I don’t think I’ve ever noticed anything take that long upon node start up. The drives seem busy when nodes start and I’ve never timed it, but certainly less than an hour.
