Some questions about test uploads and tests

I recently joined storj and running my first ~2Tb setup. It’s been fun so far. Following the official setup guide my experience was quick and smooth, the docs are great!
I was going to extend my setup but wanted to give it a try first and now I have doubts. I’m sure my node is running without issues, though traffic is low and unstable. Since then I found a few things I’d like to clarify about storj:

  • What are Storj-test-uploads and how are they managed vs customer uploads?
  • Are there customer uploads at all at this point in time?
  • Is there anything like “Storj-test-upload to Customer-upload” ratio? Enforced or eventual?
  • Is there “Total-test-upload-size to Total-storj-nodes-size” ratio?
  • What are this ratios if any, do they change, how and why?
  • How come traffic is so low while testing the network and having test uploads in action? Do you expect the same or lower levels of traffic post product launch? (I’d expect a proper test would have same or higher then expected rates.)
  • It seems --storage.allocated-bandwidth parameter has no impact on amount of traffic the node receives and all nodes receive similar traffic. Can you please confirm?

I’m not receiving almost any traffic last 12+ hours. This has happened before and I go checking my node each time to find it’s OK and then I learn here that you are running some tests etc.

  • What are these tests? (I can’t imagine any possible tests that require stopping uploads for 12 hours.)
  • It sounds like this are planned tests and this is expected. How come there is no announcement/warning for this work in advance so that SNOs know what to expect.

In general it seems the project is lacking transparency about what’s currently going on and what to expect short/long-term. Things like global network stats would also help. Also https://storj.io/storage-node-estimator/ is fairly useless since the main factor is traffic rate and there’s no control/visibility for SNOs. It looks like it currently heavily depends on storj test uploads and this would need to be clarified for estimator to make sense.

i am also looking for answers of these questions

It doesn’t matter if it’s test data or customer data, you get paid the same for it. Also, customer data at this stage is going to be customers building out tools and testing them. So their data would also be test data. Best not to think of it like test and not-test. With beta, the data is the data.

For the rest, we are just now switching to Beta. We are not in production yet. So, having these metrics and such that you want are simply not ready yet. There will be an improved dashboard for the SNO’s to use in the future. When it’s ready, it’ll be released and include additional data to assist you in operating your node properly.

3 Likes

Thanks for you answer.

Well I think it does. Storj traffic rate is justified by “supply and demand” but this doesn’t play well with test uploads. At least without details of what they are and how they compare to customer traffic. If they represent majority of data then Storj effectively has a valve to manipulate payouts instead of the advertised open-market “supply and demand” approach. This kind of speculation appears due to lack of transparency so I’m asking questions to clarify what is actually happening.

The reason I bring this up though is not the payouts I’m looking to get, my first payout will be well below 5$ so not significant. I’m interested in this project and I got the node running to support the project by contributing to tests and to better understand it. Payouts are only relevant to me because I think Storj success strongly depends on being attractive to both customers and SNOs.

While we are now in beta (congrats!) I’d still want to get answers to the above even if they are not relevant moving forward. These are still relevant though:

  • Do you expect the same or lower levels of traffic post product launch? (I’d expect a proper test would have same or higher then expected rates.)
  • It seems --storage.allocated-bandwidth parameter has no impact on amount of traffic the node receives and all nodes receive similar traffic. Can you please confirm?
  • What are these tests? (I can’t imagine any possible tests that require stopping uploads for 12 hours.)
  • It sounds like this are planned tests and this is expected. How come there is no announcement/warning for this work in advance so that SNOs know what to expect?
  • What features/stats can we expect to be added?
  • Is there an ETA for the improved dashboard release?

Hello, in addition to this excellent input from @knowledge the team is also discussing your topic today with the Product Manager. I’ll be sure to update later on with an update on what develops as we discuss it Internally. Thank you for being here

1 Like

Again, we’re in beta. Tests are run to test things. They might upload 5 gig, they might upload 5 TB. It depends on what element they are testing. They are not testing your ability to get data, they are testing network features, logging, auditing, and the like. The developers know what they are looking for, but those tests may be turned on and off whenever they need to gather some data. It’s not a formal process.

Traffic post launch is dependent on customers. We will see test data exist even at launch, as developers will continue to utilize Storj and provide test data. Further, Storj will continue to work on and innovate the product. So, test data will continue indefinitely.

On a long enough timeline, nodes should receive about the same amount of data if their reputation is the same, their downtime is equal, their Internet speed is the same, and their CPU and response time is equal. In other words, it varies.

1 Like

I recommend you to read this blog for details:

Since tests started i get only fatal errors and zero data… i would like to think that this is normal… node is up and looks fine…
Any confirmations? That this is normal

Please, give me your NodeID and please, post your dashboard

I have similar, here is my data, please check.

2019-08-22T09:33:25.783Z INFO Configuration loaded from: /app/config/config.yaml
2019-08-22T09:33:25.921Z INFO Operator email: be*******029@gmail.com
2019-08-22T09:33:25.922Z INFO operator wallet: 0x02d4c1Ed3FC0B471901ebb567bc88604C1C69671
2019-08-22T09:33:26.885Z INFO version running on version v0.17.0
2019-08-22T09:33:27.287Z INFO db.migration Latest Version {“version”: 14}
2019-08-22T09:33:27.287Z INFO vouchers Checking vouchers
2019-08-22T09:33:27.288Z INFO bandwidth Performing bandwidth usage rollups
2019-08-22T09:33:27.300Z INFO Node 12WrdTVFhFngTNhnLo8wR9LJXNgjDx6idPrLy3sRTNwcf1ss21N started
2019-08-22T09:33:27.301Z INFO Public server started on [::]:28967
2019-08-22T09:33:27.301Z INFO Private server started on 127.0.0.1:7778
2019-08-22T09:33:27.301Z INFO vouchers Requesting voucher {“satellite”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”}
2019-08-22T09:33:27.433Z INFO version running on version v0.17.0
2019-08-22T09:33:27.653Z INFO vouchers Voucher request denied. Vetting process not yet complete
2019-08-22T09:33:59.977Z ERROR server rpc error: code = PermissionDenied desc = info requested from untrusted peer
2019-08-22T09:39:13.847Z ERROR server rpc error: code = PermissionDenied desc = info requested from untrusted peer
2019-08-22T09:40:54.596Z INFO piecestore:monitor Remaining Bandwidth {“bytes”: 6781043480832}
2019-08-22T09:44:33.119Z ERROR server rpc error: code = PermissionDenied desc = info requested from untrusted peer
2019-08-22T09:48:28.096Z INFO version running on version v0.17.0
2019-08-22T09:49:48.989Z ERROR server rpc error: code = PermissionDenied desc = info requested from untrusted peer
2019-08-22T09:54:57.704Z ERROR server rpc error: code = PermissionDenied desc = info requested from untrusted peer
2019-08-22T10:00:08.721Z ERROR server rpc error: code = PermissionDenied desc = info requested from untrusted peer
2019-08-22T10:03:28.029Z INFO version running on version v0.17.0
2019-08-22T10:05:26.390Z ERROR server rpc error: code = PermissionDenied desc = info requested from untrusted peer
2019-08-22T10:10:45.302Z ERROR server rpc error: code = PermissionDenied desc = info requested from untrusted peer
2019-08-22T10:15:59.566Z ERROR server rpc error: code = PermissionDenied desc = info requested from untrusted peer
2019-08-22T10:18:28.008Z INFO version running on version v0.17.0
2019-08-22T10:21:18.493Z ERROR server rpc error: code = PermissionDenied desc = info requested from untrusted peer
2019-08-22T10:26:37.792Z ERROR server rpc error: code = PermissionDenied desc = info requested from untrusted peer
2019-08-22T10:31:45.704Z ERROR server rpc error: code = PermissionDenied desc = info requested from untrusted peer
2019-08-22T10:33:27.359Z INFO bandwidth Performing bandwidth usage rollups
2019-08-22T10:33:28.163Z INFO version running on version v0.17.0
2019-08-22T10:36:56.867Z ERROR server rpc error: code = PermissionDenied desc = info requested from untrusted peer
2019-08-22T10:42:12.456Z ERROR server rpc error: code = PermissionDenied desc = info requested from untrusted peer

@Knowledge I appreciate your answers but somehow they don’t help clear things:

Who are they? Surely there’s a track record or monitoring of this uploads at Storj that can be published?


VS

So which is it?
etc…

Ideally I was looking for answers in form of raw network stats, hopefully the new dashboard will have them. It seems not many SNOs are interested to clear these questions and I’m fine with details I got for now. Thanks again!

@Alexey Thanks, indeed I found some answers in this blog post.

@brandon this is the post we have been discussing together

While I would love to have more insights in current test scenarios and reasons for it, I wouldn’t want to put the burden of having to publish that information every time there is a change on the developers and testers. Furthermore it could actually taint the results if SNO’s for some reason change things based on what is published.

Customers aren’t going to tell SNO’s what they’re doing on the network. I see no reason why storjlabs testers should either. What’s needed is good information about your node and it’s standing. That’s already being built. When we have that information, the rest is no longer needed to determine whether no traffic is an issue.

3 Likes

@BrightSilence that’s a fair point but I don’t completely agree.
Storj is a company that’s building a commercial product that heavily relies on SNO community. As such I believe it’s important that Storj are hones and transparent with the community as early as possible so that we can all build long-term trust.

In this specific case while on alpha tests, the claim of traffic being dependant on “supply and demand” sets certain expectations for SNOs. From reading blog posts and forum my initial understanding was that Storj already allocate storage space to customers depending on available capacity on the network and keep it below ~50% of total to ensure stability. Tests being a small subset of traffic. In this scenario SNOs can expect that their payouts depend on demand, and want to be assured that traffic is fairly distributed according to known rules. I think the last bit is critical for building community and while there are known rules there’s currently no way to make sure they are applied fairly. This may turn people away from Storj and this is where real-time network-wide stats would help build trust in the community.

Now I learned the above scenario is what we can expect in future (hopefully) but this was simply not the case until this point in time. It appears majority of space was allocated by Storj tests and they decided to utilise 2 out of 8 PB available on the network (1/4 ratio). While this is a good approach to testing it sets different expectations. If Storj was more clear about this decisions as they happened I believe it would again help build a stronger community.

I don’t think it would cause any overhead to make network stats public as I’m sure Storj already have them (I really hope so) and use internally. It’s just a matter of exposing them.

As to SNOs changing things, I don’t see what they could possibly change as they don’t have much control other then taking their nodes down. I believe being open with community only helps growing it and working through temporary problems together.

That being said I’m not trying to accuse Storj of anything. Firstly because I only recently started following the project and set my node running. I may have missed or misread some info leading me to wrong conclusions. Secondly I don’t think Storj was ever trying to deceive the community in any way. It’s a complex technical project where miscommunication can happen and things get missed. I do believe there’s space for improvement but the blog post linked above is a good sign, it helped me clear things and gain more trust. I’m looking forward to see this project succeed.

1 Like

I find no issue with Storjlabs transparency. The code is open source, there are publicly available documents that outline development progress and the whitepaper and blog posts go in depth on how everything works. If you hang around on the forums or in the community channel you’ll quickly find out that for now most tests are done by Storjlabs. This is not a secret and openly communicated regularly.

I’m not sure where you got these numbers, but I read pretty much everything they publish and as mentioned before it has never been suggested that the majority of traffic at this moment is customer traffic.

That said, I can perhaps say a few things I could derive as an outsider about what’s going on. Keep in mind, I don’t have any inside info and I could be wrong about some of these things. Tests during alpha phase were mostly meant to prove the network is capable. It’s about verifying features work as intended and testing speed, capacity, reliability etc. Since it is near impossible to verify whether the claimed available space by SNO’s is actually available, those tests included filling up the network to prove this for at least the smaller nodes (hence the difference between proven capacity and claimed capacity).

Since more beta customers are being onboarded now and beta comes with promises of higher speeds and more reliability it is a good idea to remove some data from nodes that were filling up. This would guarantee the best experience for the most important entities in the entire endeavor; the customers.

I imagine a secondary reason for tests is giving SNO’s an incentive to join or stick around. This is a balancing game, since customers are just starting out with the first few tests, there is not much customer data and traffic yet, so to incentivise SNO’s more tests may be needed right now than into the future. But this is a constant balancing game and it wouldn’t be efficient to inform SNO’s with every minor change. And the tests are likely to change a lot based on what feature tests are needed, what capacity is available and the level of usage by customers.

I think it’s just a matter of priorities. Getting the best experience for customers would be number one. This means, speed, stability and reliability. Everything else is secondary. But even when looking at priorities for SNO’s the first thing would be to give SNO’s better insights into their own nodes standing, which is something that’s being worked on hard right now. Keep in mind that this software is still actively being developed on and everything that has to be built takes time away from other components.

One of the most important things being learned right now is what node churn is like in different scenarios. Your one example is probably one of the most important examples of why you don’t want to influence SNO behavior based on anything that won’t be available in a production scenario. Such as traffic consistency and availability.

I feel like I should end by saying that I don’t disagree with you and would love to have this same information available. I just don’t think it is of the highest priority right now and I don’t think the info that could be provided about expected network load right now is something that can still be done in a production scenario. In which case I don’t think any development effort should be wasted on it.

3 Likes

Howdy - Jocelyn, the community manager for Storj here. I just wanted to pop in and say that I appreciate the constructive and useful discussion I am seeing here. Its really great to see people asking good questions and also to see how respectful its been. Thank you! I dont want to hijack the convo as its developing, but I just had to pay you all a compliment.

4 Likes