Came the time that my 2TB (full for 3 months now), 1 year old RPI just no longer “cuts” it.
I was “losing the race” all the time with success rates hovering 11%
I decided to move it to Windows GUI and a new 3TB WD RED NAS HDD.
All good and running now, but let me advise anyone that wants to do the same:
Thank you Storj team, as the RPI was a Signature Box I won on Twitter, working now as a Private NAS, and content moved to a HP Workstation Z400, but never leaving SNO´s…7 so far, all green
You can also try rsync --delete -r --size-only to speed up the additional runs, and you only need to shutdown the node just before the final rsync, which should be only a few changes so not take too long.
I also run a raspi node and it also isn’t cutting it so far as winning 100% of uploads.
Also one other thought, maybe easier in the future to just put the drive into the other machine and then copy it over instead of rsyncing over network.
It wasn´t an option, as my PI drive was in EXT4 and Windows is in NTFS
I did use the --delete from source option, but even so, Pi is very limited regarding I/O operations.
I even investigated a possible delay between file transfers EXT4-NTFS…Pi is just slow.
I checked both systems:
Pi with NMon and Task Manager in Windows during RSync operation:
NMon was showing IO with 99% and Task Manager 27% for the HDD drives.
The bottleneck was (and is) the Expansion Board on the Pi…
It was enough in May 2019 for SNO, but not now. There´s serious and fears competitors with real machines on Storj.
I always chuckle when I see that signature box. That was a great joke/giveaway. I think the RPI 3B+ was just slightly too underpowered. RPI4 already fares better. Good to see you found another use for it though!
It was last years April fools joke, except they made it real.
The entire thing was a reference to Silicon Valley (the TV series). In which tech CEO Gavin Belson introduced a storage box called the Box III: Gavin Belson Signature Edition. I would post an image here, but their joke around it is kind of NSFW, so just google it if you want.
These boxes (presumably without the Shawn Wilkinson Signature Edition logo) were used in early testing of the network. And they used this April fools joke to give some away. Very cool.
So I have exactly the same setup on a PI 3 and my current success rate is 98% on downloads compared to a success rate of 99% on my DL380 server.
I wouldn’t describe the PI as being underpowered but would question more the network connectivity you have. The PI is on a 1G internet connection so it is limited to what download rates it can supply due to the PI3 ethernet being on the USB 2 but running speed tests on it I can only get 98Mb/s out of it which is what I would expect.
I mean… kind of. Yes it negotiates a 1G connection, but it can’t really serve speeds far over 200mbit or so due to how it shares resources.
But like you said, many people have made it work quite well for them. Which always makes me chuckle at HA server setups with RAID, UPS and failover ISPs.
After extensive troubleshooting and testing, I´ve come to the conclusion that, yes, Internet connection plays an important role on the Pi 3B+ performance.
When I had 30/10, the performance was bad, without ever “filling” I/O.
Then I upgraded to 100/60 and then I/O was 100% all the time and losing the race for pieces!!!
The bottleneck is NOT on the HDD itself or the internet connection, it´s in the X830 Storage Board as it only deals with USB2 transfer rate regardless.
I´ve seen that Pi4 is by far better, until 4x faster, so it may perform better.
Anyway, I had my Z400 working 24/7 nevertheless, so I just migrated Pi node to it.
Bare in mind that “uploads” is what matters the most for profit, not downloads.
In my Pi, I had 98% success as well, the upload success rate was the real issue!
I think you may have it backwards. It’s from the clients perspective, so an upload is ingress traffic, which is not paid. It’s downloads that are paid (egress traffic).