It will be deleted, but with a garbage collector. On slow and/or overcomplicated setups the garbage collection already throws issues.
This test data is:
Ok that I get. It just sounded like thereās a sudden need to reserve a large amount of additional space in anticipation of new customer data. I donāt see why load tests and additional data reservation canāt be done in combination with hanging on to old data a little longer. I just saw ~25% of all my held data trashed and it sounds like thereās still more coming. Thatās a lotā¦ and now just re-reserving all that sounds a bit counter intuitive. But no biggie, just seems a bit pointless.
Well my setup is neither slow or over complicated. I run enterprise servers (56 CPU cores / 512 GB RAM, 36 drive bays) which is overkill for Storj, yet Storj nodes can still cripple them when filewalker / garbage collection runs especially under high general load. Databases, orders, logs (which are not as significant as people seem to think) all run on NVME drives, no cache, but when every disk drive (18TB CMRs) is pegged at 95-100% with only a couple mbps throughput wellā¦ thatās kind of a major bottleneck. Without multiple drive controllers they canāt run even close to all 36 drives without locking up. I imagine small SNOs have never run into this, but start stacking drives on as Storj scales and I think people are going to start to notice real quick how bad the IO issue really is when their systems start locking up because the controller canāt even keep upā¦ just sayin.
Edit: So I have since been reading over some other threads (havenāt been around in a bit) and it appears this issue is really starting to become a problem. Storj doesnāt even have a real heavy load yet and is so bottlenecked from IO limitations that nodes are just crashing. This will only get worse. I suggest Storj get this under control before onboarding to many large customers or itāll really hurt their reputation. This level of IO is not sustainable large scale and it will just cripple peoples hardware and shorten the lifespan of their drives.
yea, we need to clone Alex, feed Ai with allex posts, and storjās Docs, and before asking a new tech question, You have to chat with bot first ;D
Yes. Easilyā¦
Use what you have is the mantra.
I started Storj on what I hadā¦ and it crippled it regularly. So I set up overkill dedicated systems for Storjā¦ and it crippled them too. Although at least theyāre āstableā for now.
You canāt hang on to clients data if you said you delete it. You canāt use it for your own purposes and tests.
Test data is the way to go if you want to test something or reserve space in advance.
I donāt agree too much with the approach though; they use just the Saltlake for space reservation, but many SNO GE from it.
Better reserve space with the data satellites instead.
Than they were not properly configured or overkill. I didnāt had any problems with mine. I ran 2 nodes per 1 Synology with celeron and 18GB RAM. Never had a sudden restart or crash. But I followed the recommendations; use ext4, use SATA connection, no RAID, no VM in VM in VM, 1 node per drive, etc. Storj is a good representation of KISS principle. 
Salt Lake is the testing Satellite for a reason. The team is testing IOPS. You can always exclude that satellite from your nodes if you dont want to be part of the testing.
You didāt understood me. I agree with testing on SL. I was refering to space reservation. But I think it didnāt started for the moment, canāt remember the anouncement.
SNOs that voluntarily opt out of receiving any traffic from SaltLake did so knowing that this is our testing Satellite, we will not make changes to our testing methods nor will we start using production satellites for testing just for the convenience of SNOs that now may be missing out on some usage on their node(s) due to their own prior decision not to participate in any benefits from getting data or egress from the SaltLake Satellite. This applies for all types of testing, regardless what the goal of the testing may be.
I never understood why some SNOs were exiting Saltlake. I for one would like to know if my node couldnāt handle the IOPS of upcoming customer use cases and act accordingly. So far Iām all good luckily.
because for years the test satellites didnāt generate any egress and this was when the payout rates for egress where greater than storage. this was an advantage on full nodes
In my case, if they can handle the load when it comes on to production satellites then good, if not, too bad Iām out, as I am not willing to spend in better hardware for this project anyway.
Short update:
We are looking into specific usecases right now based on customers that are in our sales pipeline but, we donāt have signed deals yet. So we canāt make any promises right now. Please keep that in mind when reading the following:
Right now to prepare for this growth we are working to scale up the amount of IOPs and bandwidth on the network. We are less concerned about capacity. We have to cleanup almost all unpaid data and maybe even a big chunk of testdata. The challenge with capacity is that cleaning up space takes up to 3 weeks to generate bloom filter, move data to trash and finally have the space available again. Depending on how fast these customers would like to upload the data we need to free up space now. That will upset a few storage nodes so we use SLC to fill the gap. That is the best balance we can offer right now. Replace data that needs up to 3 weeks to clean up with data that has a short TTL and can be replaced in a shorter timespan. If we have the time for perfection we could schedule the TTL ahead of time to make it a seamless transition from testdata to customer data.
Regarding IOPs and bandwidth, we are running the numbers right now about how many IOPs and bandwidth we will need per storage node. We have a simulator ready that tests out the maximum throughput of a storage node. We identified a few places that can be optimized to increase the performance. Up next we will implement some of these optimizations and run the simulator again. We will repeat this process in order to iterate and hit our target.
It sound like if we have several nodes on machene, NVME buffer will be very good for reads
Your comparing a Synology with 2 nodes with full-fledged enterprise servers running many more than 2 nodes. This is not even a close comparison. I wouldnāt expect you to have issues with only 2 nodes. Read my previous post for context and take note of where I saidā¦
Has this test begun yet or is it restricted to certain regions? I only see a few MB ingress on my nodes with available space on the dashboards.
I think so too. Its probably to match where the clients would be uploading so the same SNOs can test their IOPs and bandwidth
Today i see more that weak point is more Satellite than nodes. Even under this todays low loads there is not consistency in statistics. this mean that there is not enough resources for databases. As there is lot of work for filter creation, auditsā¦ even if processing made buy other servers, databases are working always for input data to this processes. It is all only my assumption, no offence.
Thatās also true for some extend (we do not have exact numbers to share at the moment, but no one will leave us in upset (except the extreme stretched setupsā¦ butā¦ well)).
because itās likely static, and this is not the expected pattern. Also deletion would take weeks instead of hours (if we need it).
Likely VMs, right? This is what Iām calling an overcomplicated. Because your have so many options, how you would pass a storage to there and the impact is significant - up to 20x of what it could have without useless virtualization, especially with a foreign guest OSes like Windows under any Linux Hypervisor, include, but not limited to VMWare, Proxmox and KVM, and the worst one software QEMU or even worse the VirtualBOX.
Yep. Perhaps you should not run so much if your system is limited? (since you have issues).
ok. But why my 3 nodes are not crashing? 2 are Docker for Windows Desktop (and this is a worst setup) and one Windows GUI, the Windows platform is not good for a high IOPS anytime. But itās working!
I do not have any VMs, any RAID, the simple (bad) NTFS and nothing else, and what is worse, the docker ones are uses WSL2, so disks are connected through a network protocol.
Thanks for the technical update, we appreciate the time you take to keep us informed 
I still have some concerns on this testing strategy, primarily the testing on production ! I know you will say you are using the Saltlake Satellite, but as you mention you are testing uploads and that hits nodes which run the production satellites as well - this sort of test in my view should of been done on storj-up or on the QA satellites, it will be concerning for many Enterprise customers to see this, not to mention existing customers - anyway, itās done now.
Your quote implies that we should be expecting some minimum IOPS and Bandwidth requirements for nodes to be defined, and also a change from the geo-ip and ip subnet based node selection ? Sorry if Iāve made the wrong assumption.
I would agree with this, but please update the requirements and TOS for running a node, the message has already been run what you have, however people on enterprise disks and storage systems are naturally going to be better than a rpi3 nodeā¦
we suspect that the problem is largely the storage node network - link
Also, thereās lots of changes in around monitoring Storage nodes, and the disk subsystem monitoring - again if you are going to start doing that outside Monkit, it would be nice to let SNOās know so we can opt in / opt out of this 
this change introduces a mechanism for the satellite to keep track of which nodes successfully received pieces during segment uploads and prefer those nodes during the selection process - link
Again, not complaining, my nodes can take all you can throw at them, as they are tightly rate limited, however I feel sad for new operators, or seasoned veterans who have always followed the āuse what you haveā, and might now find that they are slower then the many Enterprise grade operators in the public network, and for which the node usage might be impacted.
I guess Summary:
Please update the Terms of Use, and Node Requirements based on the feedback from this testing, so node operators aware of what specifications they need.
If there is going to be any non-anonymous profiling of nodes, and their hardware - this should be an opt in choice.
If you are going to change how nodes are selected, outside the /24 IP restriction, then please announce those changes in advance - I know you will get moaning, but is will be better than sudden surprise in my view.
CP