doesn’t help, data is transferred between the customers and the nodes, not through the satellite.
So actually it’s better to be close to the customers. But it would not be easy to be close to every customer in the world.
Well in the southern Hemisphere we are close to neither, which is why the storage service is very very slow down here and profit poor as well.
Hi Unique,
How much ingress are you getting in a month? What’s your node setup like?
Two nodes on same Lan, ARM architecture, Linux, docker, SATA 7200 HD’s. Averaging 18 to 20 GB per day ingress (9 to 10 each). Trash unfortunately has been averaging more than that lately. I have 400GB less storage than I add in October.
I have the same amount across three nodes (the same /24 subnet), nodes are located in Russia, St. Petersburg.
so seems unrelated to your location.
1 Like
I have nodes in the northeast (US) and average about 800 GB/mo ingress not accounting for deleted data. Each are run on enterprise grade hardware with fiber connections and have been online since the start of V3. Totally worth uprooting your life and moving there if your looking to make a few extra bucks a month from Storj. lol
As for raid/zpools, since I already had zfs pools without any data yet I started on those. Figured the redundancy would be good… and early on it was fine, but as the nodes get larger and having multiple nodes on one pool, it started becomming a bottleneck at times. Plus, good luck resilvering a new drive after a failure… no really! In reality they perform much better separating them onto single disks as Storj recommends. The issue is really the database IOPS, not bandwidth. If the database is getting hit hard the bandwidth can be next to nothing, but the disk gets pegged at 100% and latency begins creeping up. This could probably be dealt with by just moving the database to a separate SSD pool but I haven’t tried that. I instead moved the nodes to separate drives. I decided to do this since redundancy isn’t needed for Storj anyway, and by dropping my own redundancy I pick up that extra storage space.
I’m sure we’re all thinking along the sames lines in terms of using raid arrays for this, but from my experience it just isn’t worth it. If you want “redundancy” in terms of the monthly income, just run smaller nodes on multiple drives and upgrade as needed. This way, if a drive fails you only loose a fraction of the data as opposed to having a large node and loosing all of it. Your nodes are still small enough, but you’ll see when you get upwards of 20+ TB on a single pool. After the trouble I had, I couldn’t even imagine filling a 60 TB pool. Just thought I’d throw that out there since once you hit that 20 TB mark your pretty much commited to that pool.
6 Likes
this is not good. Why did you started several nodes on the same pool? They will not get more than a one node.
It is better to move them to a SSD drive, zfs could be slow for them.
nothing unusual. This is one of the reason why running multiple nodes on the one drive (the zfs pool is actually behave as a one drive) is not a good idea.
good move
exactly!
1 Like
You need to get out more. St Petersburg is practically in Europe, not down the bottom of the world.
The point is, even if it’s almost Europe it has the same usage as a node in Australia.
Sure, see the below specs. Its a complete overkill, but this is a hobby for me and I’m experimenting with multiple projects. System is setup in RAID-5 which gives me a max capacity around 170TB. The reason I went with this was because it gives me more flexibility for future projects. As of right now, I have two StorJ nodes running on different subnets to maximize data. Using VMware to run multiple machines.
PowerEdge R730xd
2x Xeon v4 2.1Ghz 16 Core
128GB RAM
12x16TB Seagate NAS HDD
2x4TB Samsung SSD
1 Like
Seagate NAS HDDs (Ironwolf, Ironwolf pro) are more expensive than enterprise Exos HDDs. I don’t know why anyone whould pay more for weaker hardware. Maybe the “best for NAS” expression motivates customers to not search farword.
You really have TWELVE drives in a RAID 5? You know that if 2 drives have a failure at the same time, all of your data is lost? That also includes the RAID-rebuild/ resync after swap a faulty drive. Your resync would be very stressing all other drives, so the possible failure of a second drive is very likely.
@snorkel Thank you captain obvious.
@node_operator0815 I’ve had a similar setup running for 8 years with 1 drive failure. Yes 2 drives failing is a possibility that I’m willing to risk.
This is obviously a common fear mongering not accurate. If that was the case, you would be seeing disk failures after every other periodic scrub. This does not happen in reality. With periodic scrub enabled you need to have to experience two complete drive failures or at the same sector within a short period between scrubs. This is too unlikely to worry about. You have other system component that can fail with higher probability. RAID6 on such as small arrays has significant performance and cost hit, while providing second order improvement, that is rather irrelevant.
Seems you flipped to the other side too far. It is of course more likely for an HDD to fail during heavy load times than during light load times. And rebuilding an array as large as 12x16TB takes a really long time. Depending on your RAID setup, the entire process might fail on a single unrecoverable read error during rebuild. With those settings, arrays of this size are almost certain to fail during rebuild. This tends to be the default setting for hardware RAID. If you can adjust this behavior to just accept the loss/corruption as a result of URE during rebuild, make sure you do.
That said, in general running RAID5 on such a large array is not recommended.
Have a look at this post and the linked article:
1 Like
At my old company, we ran two servers on RAID 5 since 1997 for a very early (cloud) application in the Record Center industry. That ran 24/7 until 2022 when it was finally migrated to Azure and off local hardware. We had drive failures here and there, but we never lost more than one drive at a time in 25 years. We did keep bare metal backups as a contingency in the event of failure. Tape drive! But we never had to go to them. We did lose motherboards a couple times, and PSU failure (Even with redundant PSU’s) happened. But two drives at once, for whatever reason, did not. Not say it can’t happen, it obviously can and there are better RAID configurations these days to manage those kind of failures. I certainly wouldn’t run RAID-5 today. Not unless there was some specific use case that required it for optimal performance. I can’t imagine what that would be though.
2 Likes
My question is why not just create multiple nodes?
I don’t understand if your subnets and just in local network or not,
but if you can have access to multiple ip’s, multiple nodes will bring a guarantee for increasing profits…
My understanding is that 1 node on 1 subnet will receive the same amount of data as 100 nodes on the same subnet. Limiting the amount of nodes to 1 per subnet is easier to manage and maintain.
You can also use this to check: Neighbors
I would like to note that the subnet restriction is for a /24 (CIDR) subnet, respectively 255.255.255.0
This refers to the public IP.