Moving the performance tests to a new satellite

Wow! Can you confirm that now it’s FINAL information? :crossed_fingers:t2:

Please read above again, no final date yet.
I just gave my personal estimate scoping the work.

1 Like

Please note the citation. I am interested about payouts and shutdown/repurposing, not exact dates :slight_smile:

Your quote was preceeded by

Emphasis added by me

I think your question was answered.

2 Likes

Nope. There is no fix in v1.9.5. In fact I was able to reproduce one way to get a false GE result with v1.9.5

“Денег нет [за GE из-за DQ], но вы держитесь!” :rofl: :+1:t2:
Sorry, Russian speakers will understand the beauty of this phrase and why I wrote it like that.
If translated, it will be something like: “We know about the problem and do not solve it for a very long time, since it is profitable to distribute disqualification to everyone who try to make GE”.
Problem will fixed SoonTM? :rofl:

Wrong discussion. Instead we could talk about why it takes so long to fix it and how everyone of us could help to speed up the process.

2 Likes

I would like to help, how can I help?

The simplest solution seems to me to be to remove the test garbage from the north. After all, it is because of this that many have run out of place :wink:

As for the help … I see, you ignore problems (I don’t see any confirmation like: yes, we have a problem and we are working to solve it) and error messages for months, don’t answer questions, the support team doesn’t help in any way. What kind of help are we talking about? You not to accept problems and confirm problems, I do not see any customer orientation, and why I should help you? what a reason?
If people really heed help they ask about it, something like: “hey community! We have a problem and we really need help with it!”. But what we can see now? only silence and ignoring…
So you say: “Wrong discussion”
I can say: “wrong opinion”

In this case, the project has no future.

I don’t see how this is related to graceful exit. Besides that wouldn’t you be the first one that will complain about storage nodes deleting 500GB and after that not filling up again? On my storage node I wouldn’t like to see that.

End of conversation. Next time please think about a positive intention. If you claim that I ignore problems I will go ahead and ignore the claim. There is no point for me in arguing against that. I don’t see how that would help. Next time you could just ask what the current state is instead of claiming that I would ignore the problem.

3 Likes

Thank you for asking. I would say there are 2 things that we should do.

  1. Try to find out what is going wrong with graceful exit.
  2. What would help to address a similar bug a bit faster next time.

Let’s start with the first point.
The developer team found out that a few storage nodes are running into timeout issues when they submit graceful exit results to the satellite. There is no retry in place. The fix is not finished yet: https://review.dev.storj.io/c/storj/storj/+/2335
I don’t believe that this is the only bug we might have so I have started testing some other edge cases. What happens if a storage node has 1 corrupted piece? We would expect that the storage node can still finish graceful exit but for some reason an entire batch will fail. As far as I know Stefans satellite is running with a higher batch size which would explain why that is happening.
Another edge case that I haven’t tested is missing piece. We might see similar mistakes as with 1 corrupted piece.
These tests can be executed in storj-sim. A good bug description with all the steps to reproduce it will help the developer team.
Any other edge cases that are worth testing? Basically any help is appreciated. If you have an idea what we should test just let me know. If you are able to execute that test yourself that would be even better.

Now to the second point. You might wonder if a simple storj-sim test is all we need why does it take so long to get there? Good question. I don’t know the answer. What do we need to simplify it? Should I write down a guide on how to use storj-sim? Currently I am not happy with my own storj-sim setup and I have some ideas how the ideal setup would look like. We could take that route to make sure next time we can test stuff more quickly. Idealy the community is able to track down bugs without having to wait for help from the developers. Every idea how to get to that point is welcome.

@Alexey I believe you want to move that part into a new thread soon :slight_smile:

2 Likes

Insignificant losses (payment of only 25-50%) for storage are justified, since this data is rubbish that is almost never repaired ($) and does not generate enough outgoing traffic ($$). For this reason, I’ve already removed a decent amount of terabytes of north.

Not you as person, but you as Storj Labs company.
-Hey give us parallel segments upload because now its slow… no reaction
-Hey open bugbounty for all… no reaction
-Hey, I need more space in targirade… rejected
-Hey… no reaction

It’s 75%+ of the payout actually. And I really don’t see why you would complain about getting 4x as much as you would have if you only got customer data. Please don’t get rid of it. I for one enjoy the extra pay a lot.

“No reaction” is also completely disingenuous. I’ve seen reactions on all topics you mention here on the forums and you yourself have referred to the reactions you received from support on those topics as well. You may not be satisfied with the responses you got, but your requests weren’t ignored.

2 Likes

Duplicati and rclone have an option for that and both are available with native tardigrade integrations. There are a few more but let’s start with these 2 apps just to showcase that you can upload segments in parallel.

It turns out that is not as easy as expected. My concern is that the community is unable to look into our internal bugs. You would file bugs and expect a bounty for that without knowing that it is a duplicate. Anway beside my personal concerns there was a response here in the forum. It was from JT if I am not mistaken.

Correct it was rejected and I kind of understand the decision. How can we make sure you will pay the bill at the end? At the moment we don’t have a clear answer for that but we are working on some ways to minimize the risk. By the end of August, the situation might have changed and you can request a limit increase again.

I didn’t read “Hey”. You started the conversation with claiming that we would ignore graceful exit bugs because it increases our profit. That is wrong on multiple levels. If you expect negative intends everywhere you will always find what you are looking for. Next time expect positive intends and the hole conversation will shift into the direction that we all would like to see here.

5 Likes

There is a simple explanation for my solution - optimization. On production satellites, data extraction accounts for approximately 15% of storage, while in the north it is close to 1-2%. The choice is obvious - I delete low-income data and take more profitable ones in their place.

Hmmm… I wonder why Google doesn’t ask such questions or Hetzner? Want more? Get it!

Good point. Btw the reason for setting up a new satellite was that we managed to get saltlake to the state that we wanted to see. We learned a lot out of that test. At the end we had to make a decision. Should we continue uploading to saltlake and risk that we crash it or should we setup a new satellite maybe with some different options and use it for comparision. The risk of crashing it was the main reason why we setup a new satellite at the end.

1 Like

I’m wondering why didn’t just restart Stefan? The percentage of payments at which it was 100% … So, I think this question will not be answered for obvious reasons.

Hetzner has limits as well. If you want more you will have to buy a bigger package and fun fact. Hetzner lets you pay for the package in advance. You get charged at the beginning of the contract.

You can do the same with tardigrade. If you pay STORJ tokens in advance we will basically accept any limit increase as long as it is within that token balance.

Google is a good example. The issue for us is that there is an incentive to cheat the system. I would like to setup an account with a 50 TB limit and use that to increase my storage node payout. This incentive is making a hugh difference between us and google.

2 Likes

Please look at:

My request already rejected.

That’s true for dedicated server, but hetzner.cloud dont use prepay model (I use cloud+dedicated).

For you, this is also not a big problem, since you pay the node operators not with real money, but with a coin without actual collateral.