I’m a little wary to expand right now, even though I don’t have enough nodes with free space to use all my IP’s. The main reason is that at the moment uncollected garbage and trash account for almost a third of my used capacity. Something still isn’t working right there. I will probably expand when all nodes are close to full and wait with expanding more until that issue is resolved. I’m also a little worried about what will happen with performance if expiration of the TTL data kicks in combined with large GC runs still needed. This is still an untested load scenario.
A side note: With the new node selection, it may not be all that useful to keep working with multiple IP’s as added load on the system reduces the chance node selection would pick my nodes. It still helps, but not as much. That’s a good thing in general. Though I wanted to keep those IP’s for collecting data for the earnings estimator for different sized and different ages of nodes. However, it’s near impossible now to make a good estimation as node performance can have a big impact. It’d be awesome if we could get average stats on ingress per node with free space.
So it’s been a little over a month in testing, can we get any update if a client signed/is getting ready to sign, any kind of numbers (ie we are 30% there wrt requirements), or even what the reduction in throughput was after filling the 3000 nodes?
I’m not looking at a mountain of data, just a spec of dust will do.
I don’t quite understand. The satellite knows the right amount of data stored per node. So why is it reporting a wrong stat to the node? Is there a difference?
You might have misread that somewhere. I explained why we are going to plan on adding surge capacity so that nobody has to take unessesary risks. That also covers adding another ISP line. With the surge capacity we are going to buy time so that the network can adopt to what ever the new situation will be.
So you expect that we need to run the surge capacity more permanent? That wouldn’t be an issue. We can run the surge nodes for as long as needed. The costs are low enough for that.
The node selection will look like this. Lets say 95% of the pieces will get uploaded to the public network and 5% of the pieces to the surge nodes. Or what every distribution we need. The numbers don’t really matter here. All that matters is that we will mix them in with our public network in order to boost throughput.
There will be no rebalancing. The pieces that the surge nodes will store will stay there for 30 days. Instead we just reduce the number of new pieces they are getting. A complete shutdown is possible by setting the free space to 0 and just wait 30 days until they are empty.
Give us more potatoes. Thats fine. The benchmark tests are telling us that the potatoes still have decent throughput and will help. This is not a question of hardware. Even an Pi5 has a great time these days. There is no need for datacenter grade hardware. Run what ever you feel works best.
Common can we maybe stop this discussion? I undestand that you have to try to ask for more money. I would do this as well. At the same time I look at my node and the payout is already way higher than I thought possible this year. I can operate my node with that payout just fine. I understand some nodes might have too high costs for bandwidth. Thats ok. We can’t make this a net profit for all nodes. We only have to make it a profit for most nodes like for example my own storage node. Potato nodes are welcome.
I know the costs, I wasn’t suggesting you keep running it forever. I was suggesting that if one of the “big clients” has already signed up and “will be” is coming into effect, it’s better if the SNOs hear about it. If the client hasn’t signed up yet, that’s where “may be” is coming into effect.
Storj labs is reacting on what the numbers are telling it (ie projected sales, network growth, nodes being added/removed). Us SNOs are reacting on what crumbs of information we end up being fed. I think we have a right to be a bit more skeptical about any "will be"s.
Sorry, I have to disappoint you…
I have explored the logs of many of these nodes that state only to be up for some hours. But for unknown reason, if I look back at the time which they claim to be up for, there is invariable a moment of version-check. But sometimes apparently the version check can’t be fullfilled / server isn’t reachable, and the uptime is being reset in one or another way. I even don’t see a real restart of the storagenode.
Ok got it. It will be a bit short notice. I expect just a few days between signing the contracts and the uploads hitting our nodes. So with the surge nodes we can buy time for the network to adopt to the new situation.
Yes. I am a node operator myself and fully agree on that. I have communicated that point to the stakeholders already. If we are lucky they might make a commitment to take the risk of no deal signed from the table. If the deals are getting signed we get a higher payout and if the deals are not getting signed we also get a higher payout. That would also allow us to start growing in advance. That commitment isn’t on me so lets wait and see how it will look like.
Could you please open a new thread for that and ask [@] elek how he did it? To my knowledge he managed to get a storage node down to a few minutes file walker with an inode cache. I don’t know the details but he can share his knowledge.
I have this exact same problem on some nodes only:
My trash is empty but its not updating the trash to the node, so at the moment is thinking i have 6.5 TB in trash. Im monitoring the logs now and lets see what it says. But nodes on the same storage does not have this issue. And filewalkers are finished successfully.
Thank you. I have a few old 6TB drives that I’ve spun up and got ready to start new nodes when you pull the trigger. If they fill up I’ll buy some decent SAS ones for more permanent use.
Will there be any changes to vetting in order to speed up the process of bringing new nodes online? I think you alluded to that earlier on but not sure if that is actually “a thing”.
