My machine has been regularly freezing for 5-10 seconds for a few days now. It seems to work fine again when I stop the storj service. Also, looking at the storj log, every single upload that is started is subsequently cancelled.
Does anyone know what the issue could be?
I’m having to stop the storj service for now, as my machine is totally unusable with it running.
Can you provide your computer specs ? CPU, Ram, HDD size.
How is your HDD connected?
None of the resources appear to be excessively in use and it’s a high spec machine.
Main drive is an M.2 SSD, storj node is using an Seagate ST8000DM004 Barracuda - 8 TB internal hard drive connected using 6gb/s SATA.
The only slightly odd thing is the my Disk is permenantly showing 50% usage with Storj being the main culprit. If I stop Storj then Disk drops to 0%, as expected and generally never goes above 5%.
Go to the Performance tab and check disk usage of Seagate ST8000DM004 Barracuda - 8 TB
Your node lost the race hence you see upload canceled and its an expected outcome. However do check logs if you get some uploads or not.
ST8000DM004
This is an SMR drive, meaning that some kind of pauses or drive slowing down are to be expected. Whether this is the actual cause of your problem, I don’t really know how to diagnose.
What model is your SSD?
Wait, they’re using SMR on other drives than the archive range?! Seriously surprised to see this. In many cases these drives need to write much more data than what actually needs to be written because writing one track overwrites the next. These drives should really only be used for archival purposes as they are incredibly slow with lots of random writes, which Storj needs quite a bit of.
Shame on Seagate to use this tech without clearly mentioning it for Desktop models. I guess they assume this is now mostly used as a secondary disk for storage.
I suspect SMR drives should still be fine for Storj as long as the database files are on a different storage.
They aren’t though. And moving them is not supported atm.
Is moving the database likely to be supported in the future? I’m guessing it’d be a fairly minor change to make it configurable? Obviously automatically moving the database is more work, but manually configurable with manual moving of the files while the node is stopped I assume is pretty easy?
So far it doesn’t seem Storjlab is very open to that idea. It’s partially understandable as you’re actually adding an additional HDD failure risk on top of the data disk. So it would increase the risk of node failure.
This idea has been posted before here: https://ideas.storj.io/ideas/V3-I-121
You could upvote it there.
Makes sense - although if the idea behind storj is for people to lease out spare space on hard drives, often those will be large SMR drives so if that is the problem, then this will become more and more of an issue as users come on board. It wouldn’t be too bad if it didn’t affect the whole machine, but it literally becomes unusable. I don’t really understand why it has such a knock on effect.
I’ve had lots of periods with 100% disk usage for quite some time, so I’d consider this normal behaviour due to garbage collection or something along those lines.
However your system shouldn’t freeze because of a busy hard drive unless you are running something vital from said drive.
What else do you use that HDD for?
When I said earlier today that I bought an enterprise level 8TB, I also referred to a SMR drive. The days when these were only used for “cold storage” high capacity drives is long gone, it seems.
Using less platters + SMR to achieve a certain capacity is the more energy-efficient way I presume and the negative effects connected to that should be less noticable nowadays compared to, say, 5 years ago.
I’m guessing it’d be a fairly minor change to make it configurable?
It takes a two-line patch to hard-code database locations to somewhere else, if you’re willing to recompile the code after each upgrade (ie. you’re on your own).
However your system shouldn’t freeze because of a busy hard drive unless you are running something vital from said drive.
I agree, that’s why I wrote I’m not sure if that would cause freezes. I can imagine a scenario where database writes invoke system-level syncs as opposed to a single file system syncs, and this would lead to system-level freezes. But frankly I would expect sqlite to be better than that? It should be possible to test this hypothesis using a tool like nosync.
It takes a two-line patch to hard-code database locations to somewhere else, if you’re willing to recompile the code after each upgrade (ie. you’re on your own).
Would someone be able to build me the latest release with this change? With the path set to C:\Program Files\Storj\Database. This would save me from having to set up a dev environment that I have no knowledge of.
We’re currently speculating on the cause, this would help narrow that down.
It’s not used for anything else. I bought it purely to test Storj as it seems like a worthy project that should be supported and I have machines running 24/7 so I’m in a good place to do that.
@willyc i think this might be a perfect use case for changing the config.yaml
i would suggest something like this, with a setting of 3 or 4 to begin with, then see how it runs.
the problem with your drive is lack of IOPS
also the problem might get much worse if you have very limited free space on the drive.
I’ve done that. I’ll see how that runs for now. I still think moving the DB would be an interesting test.
edit: I’ve seen the word “uploaded” three times in the first 2 minutes. A rough scan looks like I’d get a successful upload about twice a per hour previously. So it looks like something’s improved. I’ll keep monitoring.
Sorry, there is no change of this kind in the public repository. All I stated is that this is possible.
