Hello Storj Community,
We are excited to announce some changes that we have begun to implement regarding how we are using test data on the network!
What is changing
We are currently seeing exponential growth in our paid customer data and expect that trend to continue based on our sales pipeline. Because of this we want to ensure that we are sending timely demand signals to the node community so that the capacity and other resources we need for our growing customer base is available when they need it. As we are experiencing this growth we believe that our two limiting factors will be capacity and IOPS. In order to secure resources and to scale test, we have decided to begin to upload additional data to the SLC satellite essentially “reserving capacity” for our customer growth a few months before we anticipate it will arrive. At times we may also use this method to test synthetic bandwidth load as well.
We have prospective customers with a number of different usage patterns and several of them have significant requirements for volume of writes and ingress throughput. Many of the synthetic load tests we’re conducting are designed to simulate and validate those customer use cases.
How will this be implemented
To accomplish this we plan to analyze our anticipated data growth and begin to continuously upload synthetic data, with the anticipated customer segment sizes, to the SLC satellite with a TTL. Because of the TTL, the data will expire and be removed on the node side without the need for garbage collection. In this way we are testing our upload throughput, reserving capacity, and automatically deleting data as it is being replaced by paid customer data.
As previously mentioned when we announced the discontinuation of the free tier, We have also already been removing old data from our production satellites composed mostly of old abandoned accounts as well as free tier accounts that were closed by users following the conversion to free trial on our production satellites. We plan to increase the rate at which we are doing this in order to transfer all non-paid data to the SLC satellite and clean up the production satellites. As a consequence of this, nodes will see an increase in the amount of garbage they are storing in the short term.
Other items of note
We hope that this process will enable us to scale up the capacity we need as required by our growth. In addition to that we may also send out announcements asking for more nodes to be brought online in specific locations as required.
We also anticipate the need to continually improve many aspects of the network and the storage node code and experience. Things including better scaling our garbage collection processes, and improving our IOPS efficiency (thanks to our community for getting involved and suggesting design docs). We are excited to continue to work with all of you as we grow the network together.
Actions for node operators
We don’t expect that this load is temporary, and may in fact be the new normal. Please take steps to correct issues your nodes may have keeping up with this load, and as always, adding more nodes can also help reduce the per-node load on the network. We will continue to be active here assisting, gathering data to help inform further improvements.
Thank you for all your continued support and contribution to this endeavor.
The Storj Team
