Node Operator Fireside Discussion

Announcements
42

@john , @Knowledge any updates to the status of questions and video?

1 Like
43

The link is in this post upthread in case anyone missed it.
After filling out the form you should be able to watch the video.

44

@bre - the Privacy and compliance webinar is separate. We are after the zoom meeting we held with @John, @knowledge @jtolio and also the questions further up the chain here that were not answered.After we had the fireside meeting with those above the webinar was held - but it has been posted first - which i also don’t understand.

1 Like
45

That’s on me for asking about it in a this topic. But yeah, we’re still waiting on the video for the node operator fireside discussion. As well as the additional answers to questions in this topic.

1 Like
46

Being the COO, John is VERY busy. We have the questions that weren’t answered lined up, he just needs to find some time to work through them. Please be patient as he gets pulled in a lot of directions. We are going to get you the results here. We discuss it in our meetings.

As for the video, there may be an issue there as it may actually not have been recorded. We assumed that the host of the conference (None of the three of us in the video) was either manually recording, or it recorded automatically. But it “appears” that might not be the case. I don’t have access to the host account, so JG or someone else on the team would have to confirm that we either don’t have a recording, or that we do and they need to store it (On Storj!) so we can access it.

In the future if I’m hosting or involved with public meetings, I’m going to record the meetings with the software I use for capturing podcasts rather than rely on Zoom recording, because the Zoom video format for recordings leaves a lot to be desired in terms of quality, and I can be sure there is a copy for release and have ready access to it for immediate release.

7 Likes
47

That’s unfortunate, but if the video is indeed lost, it would be nice if the questions answered in it could be addressed in a short text response as well. That said, I definitely understand that the life of a COO is a busy one.

3 Likes
48

I’m getting to the end of the questions. Should have this wrapped up tomorrow.

I’ll try to also publish answers to the questions we covered during the Fireside chat as well as the ones we missed.

This won’t be the last Fireside chat. We’ve got Town Hall coming up in a couple weeks then we’ll get another Fireside chat on the calendar.

I’m sorry to hear about the video, too. We’ll be sure to check that in advance for the next one.

Thanks for the patience!

7 Likes
49

Ok, this took longer to get posted than I’d hoped, but here are the questions/answers. I am posting the ones we covered in the Fireside chat also. Thanks @jt and @knowledge for the help pulling these together. I’ll keep an eye out for follow ups. We’ll host more Fireside Chats in the near future and @Knowledge has some other ideas cooking to keep the lines of communication even more open.

Questions covered in the Fireside Chat:

  • How much customer data is on the network?
    • All data on production satellites is customer data
      • US1, AP1 and EU1
      • We’re at a few PB and growing
    • Public API and how to use it
    • Trends in growth
      • Increase in higher bandwidth use cases
    • We store test data to make sure supply exceeds demand, but this is largely only happening on the “saltlake” and “europe-north-1” Satellites.
      *** What use cases are successful**
    • We’re seeing early traction with video, software distribution /large file transfer, cloud native apps and backup
  • What’s going on with all those tiny files
    • Our pricing makes us attractive to MSPs, many of whom use tools like Veeam and others that generate a lot of small files. There are a number of excellent backup tools that we’ve been recommending that pack tiny files into larger blocks ex. Hashbackup or Ahsay.
  • What’s going on with Filebase
    • Filebase is our most successful partner and have been seeing great traction. They’ll be offering a solution for tiny files soon.
  • How’s performance doing
    • Performance is amazing and will continue to improve. The QUIC project and some work on round trips will be impactful in the near term.
  • I can see used space and traffic is slowly increasing. Can you share some background information about the customers that are causing that?
    • Watch for case studies as they’re published on the website and more details coming during the Town Hall
  • I am worried about the repair traffic that I see on my storage node. That doesn’t look healthy. Can you tell us more about how frequently a file needs to get repaired? Is it getting out of hand right now?
    • We have the repair threshold set to a higher rate than the data science would suggest based on our Reed Solomon settings. We loaded up a bunch of test data and we’re seeing a wave of repair as that levels out. Overall churn is low and within an acceptable level
  • Payment. Always a great topic. Is there a L2 exchange available or soon available?
    • Nothing concrete yet but we will let the community know as soon as we hear something
  • Why is the customer pricing different from the storage node pricing? Shouldn’t it be the same? (I want to hear that I can count on the current storage node pricing for a few more month and don’t have to worry about the upcoming pricing discussion yet)
    • We’re in growth mode and we are priced to win. As we grow, we need a stable supply of nodes to ensure a great experience. We don’t plan on making any significant changes to node payouts in the near future. You can count on the current storage node pricing for a few more month and don’t have to worry about the upcoming pricing discussion yet
  • Email alerts. I was getting disqualified without any email warning. (I would love to see a grafana endpoint that lets the community write their own email alerts)
    • We agree that an email alert endpoint for writing your own integration is a good idea. We’re going to place this idea on our roadmap. So far, we have relied on our marketing outreach tool for emails like this and so reception has been spotty (if you unsubscribe, etc). This does seem like a core email to send (like password recovery or something) and we should make it much more robust.
  • What’s the status of the web browser-compatible JS client library 1 idea?
    • There’s two ways to approach a web browser-compatible JS client library.
      • The first way is to try and get all of our Uplink code compiled into web assembly and loaded into the browser. This is a challenge, largely due to browser security policies and certificate trust with nodes. If we want your browser to directly talk to nodes, browser security policies really restrict what we can do. We have some work arounds (such as having all storage nodes allocate trusted signed certificates either from Let’s Encrypt or through a deal like Plex made with Digicert). This is possible, but will take a lot of coordination.
      • The second way is to make a sort of Uplink-lite, where the browser does encryption/decryption, but a server such as one of our gateways essentially proxies encrypted data to nodes. This is something we’re planning, and it is likely we will make an Uplink-lite first. The benefit of an Uplink-lite is it will essentially only do encryption, so it will be much smaller.
    • We’ll keep everyone updated on the above, but as it stands, it’s likely we’ll hit approach 2 before approach 1, but if someone in the community is passionate to work on either, we’ll definitely take help!
  • When will multi-node dashboards be possible?
    • That code is fairly far along, but it is not prioritized for delivery in the near term. It’s a great candidate for the open development process initiative if there is anyone who wants to take on the work from the community.
  • A while back there was some talk about possibly tuning RS numbers, any updates on this?
    • We have done a bunch of work around R/S values, but there turned out to be much more low hanging fruit in terms of improvements to performance and stability. R/S numbers impact a ton of things from durability to repair to performance to node storage efficiency to cost and on and on. We have to be very thoughtful about those changes because of the wide range of potential impacts.
  • On the topic of decentralization, I know Storj originally had plans to allow independent satellite operators. What’s Storj’s plan/roadmap/timeline for third-party satellite operators?
    • This one of the very few things yet to be delivered from the V3 whitepaper and is an active topic for planning and development. We’ll have more to share soon
  • What can nodes do to help the network?
    • Spread the word about the benefits of decentralized cloud storage to help with demand generation
    • Avoid node churn
      • One disk per node
      • Keep that high uptime going!
  • What are the expectations about network growth over the coming months? A year? Multiple years?
    • We’re in growth mode and have found product market fit in a number of key use cases. We’re seeing channel partners like Filebase demonstrating accelerating growth. We have the network data API in place to provide transparency into the current state and trajectory. We’re really excited by what we’re seeing right now and the potential of the network!

Not covered in the Fireside Chat

  • What about a native Synology application?
    • We haven’t produced a Synology node application but would welcome a community contribution. If you want to build it, we’re happy to help
  • Are we going to see an easy data transfer solution for customers with large data similar to AWS Snowball or Wasabiball?
    • We are currently operating hosted gateways in multiple colocation hardware in Equinix facilities. We have access to Equinix Fabric and the ability to leverage lower cost transfer via direct connections. We are not currently working on a hardware-based data transfer solution but have discussed a number of different options.
  • Are we going to see a common proxy for more performant implementation of linksharing. Ideally this would help Storj to go back and link to videos hosted on their own Storj DCS instead of Youtube.
    • We’re working on a number of performance enhancements related to linksharing and already we’ve seen significant improvements. We will continue to optimize our edge services, and are working to implement BGP routing, which will address the need.
  • Why does Storj not accept Paypal as payment option for Storj DCS storage and are there any plans for it?
    • We use Stripe for credit card payments and don’t have Paypal on the roadmap at this point. We are working on refactoring billing and payments to reduce friction around payments, especially with STORJ.
  • What are Storj’s thoughts or plans on localization of the websites and resources to attract more customers outside of native English speaking countries?
    • We’re heavily focused on getting the product optimized and localization is a ways down the road for us. We will get there!
  • It looks like S3 compatable Gateway-MT is really popular ! however as these exit points are Geo-Located in specific locations ( I think 2-3 datacentres ? ) Are there any plans in place to optimise node selection further ? as currently nodes most respondent to those datacenters are preferred due to quicker network links ?
    • We are researching smarter ways to do node selection as well as exposing node selection as a feature in uplink (ex. Geofencing data)
  • We recently saw that in the US combating cybercrime and mitigating through zero trust architectures and decentralization became a top agenda. How will it affect Storj and Storj DCS usage with and without compliance certification as mentioned above?
    • We are continually optimizing for increased security and privacy and have a current project underway related to compliance related to data storage. We look forward to sharing our progress in the next few weeks.
  • Are there any plans to make a much more easily accessible public issue tracker, that SNO / customers can use to log issues / feature requests ?
  • Could the node software ensure that it automatically refuses ingress requests whenever the underlying storage system (HDD, SSD, …) cannot keep up? Is that feature considered?
    • There are a number of features coming with the rollout of the QUIC project that should require nodes to do less to keep up with more, but nothing specifically as described here. Nodes that are going too slow relative to others are currently addressed at the uplink side via long-tail cancellation. Nodes used to have a feature where operators could specify a maximum number of concurrent requests, but that actually caused more capacity problems than it solved. We would rather the network as a whole continue to make progress with each request with the resources it has as opposed to refusing requests beyond some ideal amount of concurrency.
  • Would like to hear something about ethereum scaling solutions you are keeping an eye on.
    • We’re working primarily with zkSync but have also looked at avalabs, tezos and a few others.
  • Does Storj plan to accept L2 payments?
    • We’re working on a refactor to token payments and this is on the list.
  • How much does Storj currently owe to SNO’s who have not signed up for L2 payments and haven’t met the threshold for L1?
    • Any individual operator can see their held amount, but we don’t currently publish this statistics in the aggregate.
  • How can Storj consider small SNO’s in any way profitable considering you need to wait up to a year for a L1 payment and L2 Exchanges are yet to exist?
    • We’re working on the L2 payout situation and will be sharing a proposal related to an effort to make the L2 payout situation better in the near term.
  • What trends are you seeing with inbound data? Is Storj becoming more popular?
    • The public network data API and community dashboard provide transparent data on network usage. We’re seeing steady growth and customers, particularly in video-related use cases, data distribution, cloud native apps and backup.
  • Are there any quantifiable growth expectations or any larger customer expected to make use of Storj or at least estimates, hints or rumors?
    • The API is the best we can do for now, but we appreciate having the community as a force multiplier to get the word out about Storj DCS.
  • Can we get a breakdown of how much traffic as percentage of total traffic is now passing through Gateway-MT compared to uplink or Developer API
    • We’ll look into whether we can add this to the API, but both methods are up/downloading from SNOs
  • Is there any update on: Tuning audit scoring ?
    • No update yet but we’ll have one in the near future
  • Is there any update on: Refining Audit Containment to Prevent Node Disqualification for Being Temporarily Unresponsive ?
    • No update yet but we’ll have one in the near future
  • What is the development status / priority on nodes as repair workers?
    • This is still a ways away. It’s still in the design phase
  • What is the development status / priority on scale on demand for popular files?
    • This is still a ways away. It’s still in the design phase
  • Can we get an update on how the Storj CockroachDB deployment went ? specifically how the latency of transaction between geo-satellite locations, plus anything else learnt from deployment :slight_smile:
    • I’ll see if we can get a blog post on this. Overall, it’s been good
  • Do problems like Trouble with UDP 1 affect customer experience in any measurable way?
    • This does not appear to have been something that impacts the customer experience, but we are very focused on ensuring customers have a great experience
  • How many nodes are currently in production?
  • When looking at the Storj Github, what does Storj think about the level of Pull requests being seen from the community ? Is this something that Storj is happy with ? What does the Dev team think about this ?
  • What is the status on compliance/audits/certification for Storj DCS (e.g. like Wasabi: GDPR, HIPAA, CJIS, FERPA, MPA, etc. 2)
    • We have an active project underway and will have an update to share in the near future
  • Now even President Biden was talking about decentralization and zero trust architecture. But on Storj social media this marketing support from the White House finds no mention, or I did not see it. Twitter, Reddit and Medium appear rather sleepy or even dormant than actively used for marketing. Is that going to be changed?
    • We covered that topic in article by Ben Golub, recent and upcoming webinars, an upcoming blog post and expect something on the topic from one of the analysts covering decentralization in the next few week’s.
  • Why did Paul Ford leave Storj 7? Why was there no announcement contrary to Storj’s transparency claim? And what does it mean for the Storj product, the developer relations, marketing, community relations and the potential danger of competition with Chia (in case they might starting to use their capacity as storage, maybe even through Storjs own Open Source software)?
    • Paul left for a new opportunity and we wish him success in his new adventure. We typically don’t make an announcement when people choose to leave the organization and don’t comment on HR matters. We’re excited about where the marketing organization, including developer relations, marketing, community relations are going and will have a lot to share in the coming weeks. We don’t expect competition from Chia as a storage provider.
  • Jocelyn was heavily involved in the forum community but since her departure that involvement doesn’t seem to have been replicated. Any plans to change that?
    • We are working on making the development process more open and getting more engagement with the community from a broader cross section of the company. If you have ideas on how we can be more involved or different ways to engage, we’re very much open to suggestions or feedback.
15 Likes
50

@john, unless there is something more in the works than what is contained in @jtolio 's proposal there is nothing that addresses the fact that L2 tokens are nearly useless and have been ever since Storj implemented that “solution”. Right now it appears L2/zksync addresses issues on the Storj side but none on the SNO side. Yes, it is a transfer of token from Storj to a wallet controlled by the SNO. But, what can we do with those tokens again?

It was mentioned in the zoom meeting that the average node life is currently 9 months - but I didn’t see that reflected in the answers above. Can you confirm this to be the case? If correct, why isn’t this a concern for Storj since hard drives should be going for far longer than 9 months. If true, isn’t this a possible issue with SNO’s leaving the project?

3 Likes
52

Was this posted somewhere?

53

No, they claim it wasn’t recorded.

54

Ah well, it happens. Thanks the reply.

55

To be fair, many many things can go wrong when hosting a service, no just the drives… So my bet is that hard drives may not be the only thing explaining such a node churn. If I was not monitoring my nodes, they would have been gone for good a long time ago because of DNS issues, errors within /etc/fstab, RPi not providing enough power to HDDs, processes being killed by the OOM killer because of a lack of RAM, and the last thing that struck my setup is that the RPi official power-supply simply died, just like that…

Currently, my disks are still fine! But oh boy, my nodes could have been disqualified a long time ago if I wasn’t fostering’em!! :sweat_smile: