To be fair my Seagate Drives aren’t the best. But I don’t see a point of giving them an easier workload. They will have to deal with this or they will get replaced with something that can deal with it. It is as simple as that. They are in the system to make as much money as possible before they die and if I can get 120$ in a single month I will take it. I am sure even these bad Seagate drives will still survive another year or so and that would be enough payout to replace them all.
“CDN style” storing data need a different price than long term TTL. Furthermore we need to mark this kind of data differently and divide nodes in groups. Long term TTL nodes (slow) and fast nodes for fast TTL. Paid differently of course. We all started with potatoes (hey… you told us!)
This is what happen in large datacenters… some data still stored on tapes others with SSD caching. Truck filled with potatoes and bullet trains with fresh food. Someone will prefer to earn less with less stress on bandwidth… others will setup bullets
IMO people who do lot of reading can get the gist of the sentence or read between the lines. But its easy to misunderstand someone when you are low on caffeine or sleep.
I have no idea. I also have 1 WD Ultrastar and 1 Toshiba drive in the mix. We will see which one dies first. Problem with the Seagates is that they have a high error rate in the first months. Not a big deal. Send them in and get a replacement. Might be acceptable for a storage node. I haven’t thought about these details. For now I just keep them running as long as they make it.
During the test, I see an increase in reported data by the satellites and the really used data on the disks.
I also see many disks, that have to handle so many uploads that done filewalkers apparently starve because they take to long to finish. Also, the storagenodes are being restarted automatically every 1-2 days (I don’t know why…).
Other people also getting this problem?
How can I prevent the (unwanted) auto restarts of the storagenodes? They don’t seem have to do with updates or overt errors.
Shouldn’t it be wiser to disable lazy filewalkers?
Now do your calculations with a TTL of 1 day (or less) and that’s how I started the conversation. I just wanted to know if STORJ is aware of this “attack vector”. Then your 20TB will be written every week if you have a 1 Gbit/s connection.
The drive will still have other data on it. If a drive is 70% full, then only 30% of it can be re-written (assuming that 70% is long-term data).
You are describing a theoretical attack scenario where an attacker has an infinite (there is no such thing) upload capacity, is willing (and has the capital) to pay however long he/she/it needs to keep the network full (which it will not be, but let’s go with the flow) because even storing the data for 1ns still takes money, but it will actually live on the drive for as long as the expired-cleanup chore is going through the loop (so further decreasing the damage to the hardware), and all of this is happening without anyone (either storj or SNOs) noticing anything about it.
In reality (which is often far far away from theory) the tiny ToS letters to the client will come in effect: Do not abuse the network. Storj bans the attacker, we have a massive bonfire, dancing around it and chanting “so long spammer!” and the world moves on.
As I said, the sun will shine tomorrow and there is absolutely nothing anyone can do about it.
Technically, if an owner controls a c2/botnet, this could be possible. The black hat effectively would have no cost for pushing data in the network - cost for storj data would be active. But significant enough to warrant deterrence?
If MTBF is significantly greater above a 300TB workload per year, this likely is a concern that may be valid to protect the distributed nature of the network. Most are likely using non enterprise drive. The network will repair, but if farmers only earned enough to buy the next drive, they might quit.
I’m diamond hold’ing - so I believe in the project, but just want to highlight that to bring awareness.
That doesn’t solve your problem. Garbage collection would still be very expensive.
Better solution is to look into improving your file system. I am using ZFS with enough memory to cache the metadata in it. I also have an SSD so that this metadata cache survives a restart. With that garbage collection takes just minutes. Similar performance should be possible with ext4 + inode cache. So pick one and try it out while you can.
