usualy big node operators have more effort and possibilities to maintain and protect their setups, like with UPS than small ones.
1 Like
I haven’t actually seen any analysis as to how much correlation is too much. I suspect Storj internally has analyses like that, given they were investigating the impact of the Russia invasion, but I don’t think we can make statements like that blindly.
Besides, with the current state of network, independence of failures is already traded off for longevity of nodes by people who migrate nodes between locations, e.g. when they can no longer host a node at some place. Longevity is also a desired feature.
1 Like
Well, I’m not making them blindly. And @peter_linder also mentioned they made some calculations. I did the same a while ago. All the data you need is available. Lowest availability segments are in the stats, we know how many nodes there are. The only assumption I had to make is that availability is evenly distributed between the max of 80 and the lowest number (which is certainly a very pessimistic assumption).
You can get the exact chances of taking even a single segment down with a bit of calculation on those numbers. I suggest you give it a try yourself.
I am pretty good at probability theory. I’m not very good at meteorology though. What kind of assumption do you put for the number of nodes taken down by a lightning strike?
I’m not saying you can predict events, but you can calculate whether say taking down a 100 nodes (IP ranges) would be a problem would cause a problem.
I have a few VPS with excess storage still filling, every VPS has it’s own /24 subnet.
I found that if a lot of people start doing it the subnets of the datacenter will be shared, same for someone using a commercial VPN, at some point people is going to start sharing subnets.
I think I’m not breaking any rules by using my VPS excess capacity. That’s the whole idea behind Storj in fact.
I’d get worried if a huge corporation starts concentrating petabytes of data and they go down, but normal people with a bunch of IPs? The network is going to be too big for an individual to cause trouble, but still good if we keep an eye.
That calculation would not answer the question stated above. It would answer a question similar to the stated above, but not the same.
But yeah, let’s try. The network currently consists of 16.5k nodes and stores 914M segments. Let’s consider a case where this means 165 SNOs, each managing 100 nodes that were already vetted over the whole lifetime of all 914M segments, had free disk space and exactly the same latency/speed to the uploader (so that their chances for selection were equal), pretended to the network over their whole life they are pairwise independent by using separate /24 blocks, yet turn out to have fully correlated failures. Let’s also assume that all segments are at their median health, around 64. Then a single lightning taking down one such SNO with their 100 nodes means probability of this much to lose at least one segment. And, well, WolframAlpha doesn’t want to compute that and I’m too lazy to think of approximations to deal with even this kind of simplified assumptions.
Now: older segments had less nodes to choose from, after few years many of their segments are stored on older hardware. The nodes are not equally close to uploaders. Segments are at different health. All these factors increase the risk, because probabilities work with geometric means, not arithmetic means.
The thing here is, this case reminds me of the subprime mortgage crisis, where similar assumptions were made. The biggest mistake was the assumption of each tranche of mortgages being supposed to contain mortgages whose risks were independent. The model and numbers are clearly different for Storj, but the lesson is that without a careful analysis you can’t make statements like the one I expressed worry about.
I’m not following your calculation here. But there are many ways to calculate these things. I think there is a simpler approach. I assumed a worst case scenario of all pieces being at the lowest availability of 52, so only 24 pieces would need to be lost to lose the segment.
Just calculate the chance that at least 24 pieces end up on a set of 100 nodes operated on different subnets by the same operator (or within the same ISP, pick your fault scenario, it doesn’t really matter, only the amount of nodes does).
Subtract from one to get the chance that a segment would survive, raise to the power of 914000000 to get the chance that all segments survive. Subtract the result from on to get the chance of failure of a single segment or more.
This one works just fine on wolfram alpha: 1-(1-((100/16500)^24)*(52 choose 24))^914000000 - Wolfram|Alpha
Resulting in: 2.35 × 10^-30
I think we’re ok considering that this is the absolute worst case scenario of every segment being on the repair threshold.
Edit: btw, I get that this is technically not correct (100/16500)^24 should technically be 100/16500 * 99/16499 * 98/16498 * … * 77/16477. But that would only lower chances more. And wolfram alpha stops working if I do it the right way. 1-(1-(product((100-i)/(16500-i)) for i = 0 to 23)*(52 choose 24))^914000000 - Wolfram|Alpha
1 Like
You know, after thinking about this problem over night, the worst what might happen is probably the following scenario. User uploads a piece, and the fastest 100 nodes all belong to the same SNO (because, let say, he operates within the same ISP as the user). The saving grace is that using VPNs usually increase latency, but again, only Storj has real, empirical data that would allow any kind of estimation how dangerous this situation is in the current state of things.
That doesn’t change the equation much, because the satellite still picks 110 nodes at random to participate in that range. So in my formula it would only change the number of permutations. 1-(1-((100/16500)^24)*(110 choose 24))^914000000 - Wolfram|Alpha
9 orders of magnitude more likely. But this literally assumes that ALL pieces end up transferring to this SNO the fastest, winning all races and THEN drop to 52 availability. It’s really the worst of the worst case scenarios and assuming all segments are in that worst case scenario. And we’re still talking about 5.8 * 10^-21
Edit: and subtracting full nodes and aggressively rounding down: 1-(1-((100/11000)^24)*(110 choose 24))^914000000 - Wolfram|Alpha
Yeah, sorry, still not convinced, but I admit I’m too lazy to think about it and follow your reasoning. So, let’s maybe have this conversation another time.
This is similar to the result that I got.
2 Likes
it is not possible, because sattelite gives this 100 nodes from different around the world, with some algoritm. if it geofeatured then it possibilities is bigger.
Are you using VPN/OpenVPN-Clients as Dockers on your server and on top of them STORJ-Dockers?
Can’t get a OpenVPN-Client to run at the moment. 
My nodes are running without any use of docker or any type of VPN.
I’m using the storagenode bin file released by Storj on GitHub https://github.com/storj/storj/releases/tag/v1.68.2 → storagenode_linux_amd64.zip
Using the terminal multiplexer screen, the bin file are executed in separate screens, one for every node, combined with a config file for that particular node.
root@server030:~# screen -list
There are screens on:
2458567.SN-107-30107 (2022-12-22 23:31:15) (Detached)
2457093.SN-106-30106 (2022-12-22 23:17:28) (Detached)
2451356.SN-105-30105 (2022-12-22 23:03:41) (Detached)
2445370.SN-104-30104 (2022-12-22 22:49:53) (Detached)
2443843.SN-103-30103 (2022-12-22 22:36:06) (Detached)
2442298.SN-102-30102 (2022-12-22 22:22:19) (Detached)
2440803.SN-101-30101 (2022-12-22 22:08:31) (Detached)
2430531.SN-100-30100 (2022-12-22 21:54:44) (Detached)
2429104.SN-099-30099 (2022-12-22 21:40:57) (Detached)
2427171.SN-098-30098 (2022-12-22 21:27:09) (Detached)
2424076.SN-097-30097 (2022-12-22 21:13:22) (Detached)
2416166.SN-096-30096 (2022-12-22 20:59:35) (Detached)
2412347.SN-095-30095 (2022-12-22 20:45:47) (Detached)
2410923.SN-094-30094 (2022-12-22 20:32:00) (Detached)
2409316.SN-093-30093 (2022-12-22 20:18:13) (Detached)
2406167.SN-092-30092 (2022-12-22 20:04:25) (Detached)
2397547.SN-091-30091 (2022-12-22 19:50:38) (Detached)
2396230.SN-090-30090 (2022-12-22 19:36:51) (Detached)
2394712.SN-089-30089 (2022-12-22 19:23:04) (Detached)
2393283.SN-088-30088 (2022-12-22 19:09:16) (Detached)
2382932.SN-087-30087 (2022-12-22 18:55:29) (Detached)
2381507.SN-086-30086 (2022-12-22 18:41:42) (Detached)
2380110.SN-085-30085 (2022-12-22 18:27:54) (Detached)
2378679.SN-084-30084 (2022-12-22 18:14:07) (Detached)
2370949.SN-083-30083 (2022-12-22 18:00:20) (Detached)
2366950.SN-082-30082 (2022-12-22 17:46:32) (Detached)
2365569.SN-081-30081 (2022-12-22 17:32:45) (Detached)
2364047.SN-080-30080 (2022-12-22 17:18:58) (Detached)
2362636.SN-079-30079 (2022-12-22 17:05:10) (Detached)
2352315.SN-078-30078 (2022-12-22 16:51:23) (Detached)
2350929.SN-077-30077 (2022-12-22 16:37:36) (Detached)
2349541.SN-076-30076 (2022-12-22 16:23:48) (Detached)
2348301.SN-075-30075 (2022-12-22 16:10:01) (Detached)
2338069.SN-074-30074 (2022-12-22 15:56:14) (Detached)
2336831.SN-073-30073 (2022-12-22 15:42:26) (Detached)
2335518.SN-072-30072 (2022-12-22 15:28:39) (Detached)
2334258.SN-071-30071 (2022-12-22 15:14:52) (Detached)
2326857.SN-070-30070 (2022-12-22 15:01:05) (Detached)
2322813.SN-069-30069 (2022-12-22 14:47:17) (Detached)
2321413.SN-068-30068 (2022-12-22 14:33:30) (Detached)
2320067.SN-067-30067 (2022-12-22 14:19:42) (Detached)
2318632.SN-066-30066 (2022-12-22 14:05:55) (Detached)
2308441.SN-065-30065 (2022-12-22 13:52:08) (Detached)
2307164.SN-064-30064 (2022-12-22 13:38:20) (Detached)
2305769.SN-063-30063 (2022-12-22 13:24:33) (Detached)
2304558.SN-062-30062 (2022-12-22 13:10:46) (Detached)
2294280.SN-061-30061 (2022-12-22 12:56:58) (Detached)
2293007.SN-060-30060 (2022-12-22 12:43:11) (Detached)
2291668.SN-059-30059 (2022-12-22 12:29:24) (Detached)
2290372.SN-058-30058 (2022-12-22 12:15:36) (Detached)
2284306.SN-057-30057 (2022-12-22 12:01:49) (Detached)
2278918.SN-056-30056 (2022-12-22 11:48:02) (Detached)
2277613.SN-055-30055 (2022-12-22 11:34:14) (Detached)
2276287.SN-054-30054 (2022-12-22 11:20:27) (Detached)
2275020.SN-053-30053 (2022-12-22 11:06:40) (Detached)
2264830.SN-052-30052 (2022-12-22 10:52:52) (Detached)
2263478.SN-051-30051 (2022-12-22 10:39:05) (Detached)
2261936.SN-050-30050 (2022-12-22 10:25:18) (Detached)
2260519.SN-049-30049 (2022-12-22 10:11:30) (Detached)
2250230.SN-048-30048 (2022-12-22 09:57:43) (Detached)
2248939.SN-047-30047 (2022-12-22 09:43:56) (Detached)
2247581.SN-046-30046 (2022-12-22 09:30:08) (Detached)
2246184.SN-045-30045 (2022-12-22 09:16:21) (Detached)
2241656.SN-044-30044 (2022-12-22 09:02:34) (Detached)
2234661.SN-043-30043 (2022-12-22 08:48:46) (Detached)
2233379.SN-042-30042 (2022-12-22 08:34:59) (Detached)
2232023.SN-041-30041 (2022-12-22 08:21:12) (Detached)
2230777.SN-040-30040 (2022-12-22 08:07:24) (Detached)
2220608.SN-039-30039 (2022-12-22 07:53:37) (Detached)
2219061.SN-038-30038 (2022-12-22 07:39:50) (Detached)
2217501.SN-037-30037 (2022-12-22 07:26:02) (Detached)
2216270.SN-036-30036 (2022-12-22 07:12:15) (Detached)
2206883.SN-035-30035 (2022-12-22 06:58:28) (Detached)
2204805.SN-034-30034 (2022-12-22 06:44:40) (Detached)
2203572.SN-033-30033 (2022-12-22 06:30:53) (Detached)
2202256.SN-031-30031 (2022-12-22 06:17:06) (Detached)
2201029.SN-030-30030 (2022-12-22 06:03:18) (Detached)
2190795.SN-029-30029 (2022-12-22 05:49:31) (Detached)
2189606.SN-028-30028 (2022-12-22 05:35:44) (Detached)
2188359.SN-027-30027 (2022-12-22 05:21:56) (Detached)
2187160.SN-026-30026 (2022-12-22 05:08:09) (Detached)
2176990.SN-025-30025 (2022-12-22 04:54:22) (Detached)
2175710.SN-024-30024 (2022-12-22 04:40:34) (Detached)
2174117.SN-023-30023 (2022-12-22 04:26:47) (Detached)
2172543.SN-022-30022 (2022-12-22 04:12:59) (Detached)
2163630.SN-021-30021 (2022-12-22 03:59:12) (Detached)
2160155.SN-020-30020 (2022-12-22 03:45:25) (Detached)
2156877.SN-019-30019 (2022-12-22 03:31:37) (Detached)
2154451.SN-018-30018 (2022-12-22 03:17:50) (Detached)
2152885.SN-017-30017 (2022-12-22 03:04:03) (Detached)
2142371.SN-016-30016 (2022-12-22 02:50:15) (Detached)
2140831.SN-015-30015 (2022-12-22 02:36:28) (Detached)
2139139.SN-014-30014 (2022-12-22 02:22:41) (Detached)
2137329.SN-013-30013 (2022-12-22 02:08:53) (Detached)
2126845.SN-012-30012 (2022-12-22 01:55:06) (Detached)
2125363.SN-011-30011 (2022-12-22 01:41:19) (Detached)
2123727.SN-010-30010 (2022-12-22 01:27:31) (Detached)
2122077.SN-009-30009 (2022-12-22 01:13:44) (Detached)
2114121.SN-008-30008 (2022-12-22 00:59:56) (Detached)
2110047.SN-007-30007 (2022-12-22 00:46:09) (Detached)
2108422.SN-006-30006 (2022-12-22 00:32:22) (Detached)
2106631.SN-005-30005 (2022-12-22 00:18:34) (Detached)
2104818.SN-004-30004 (2022-12-22 00:04:47) (Detached)
2094000.SN-003-30003 (2022-12-21 23:51:00) (Detached)
2092283.SN-002-30002 (2022-12-21 23:37:12) (Detached)
4152811.pts-1.server030 (2022-10-12 19:04:37) (Detached)
106 Sockets in /run/screen/S-root.
Th3van.dk
But how do you manage to provide the many different IPs into one machine?
That Screen looks also very good. This might also be be an install option.
And with the Screens you connect to the VPS? Do you have any guide to find such?
Usually when you create OpenVPN-Client-Docker you do the same, as you have on the VPS the OpenVPN-Server installed and setup by yourself aswell. But this screens might be alot easier and way less overhead, therefore more perfomance.
- Screen are only used internally on the main server.
- VPS are for proxying the TCP traffic to the main server.
Th3Van.dk
Any reason to not use the docker image and just run the binary manually uysing screen?
Which FS do you use for so many nodes? Any optimization there?
There has been some work done at this thread to optimize how the storage node uses the filesystem. It may be helpful for you running so many nodes (even on separate disks)