You are trying to estimate the silent majority without knowing the distribution, a “tail” of a distribution. Tails are sneaky though.
I can add to your notes as a data point that last time I checked, my position when sorting Storj transfers was around 200, and yet I do maintain 77 nodes right now (with only a few that have spare capacity).
We can approach this problem from the other side, asking: how many nodes on average do large node operations maintain if half of all operators have only one node. This means: 4500÷2=2250 singleton nodes, and 22000-2250 nodes belonging to one of 2250 multi-node operators, i.e. 8.7 nodes per person on average. Not that much.
Or maybe let’s have a estimate where ½ of node operators have one node, ¼ have two, and ¼ of them have more (getting closer to a power-law distribution here). This means: 2250 nodes in the first group, 2250 nodes in the second group belonging to 1125 operators, and on average 15.5 nodes belong to the third group. Still well below my 77 nodes, and below your assumpton of 20 nodes for top operators.
Or maybe let’s do 5 groups with ½, ¼, ⅛, 1/16, and 1/16 of all wallets, and with 1, 2, 3, 4, and >4 nodes. The top 280 operators then maintain on average 34.2 nodes. Sounds still plausible to me.
Unless Storj publishes more data, we can’t assume much here.
When I first learned how big some of the large SNOs were… I was initially impressed by the number of HDDs they controlled. As I learned more… I realized any yahoo can buy lots of hard drives: and that I should be impressed by the number of IPs they controlled.
Trash is deleted 7 days after it is collected.
I know there have been some issues regarding trash cleanup for some people, but I am not up to date with them so Im not sure what the status is.
We have customers who uploads with a TTL for years (usually this is backups), and also customers who do not use a TTL and uses data on their sites, also customers who stores their data and rarely download or delete it - usual home media usage (you usually do not review all your albums, right? unless some photo app would show you a “what was on that day a year ago”).
We are using a TTL for the test performance data to do not wait for a GC, that’s doesn’t mean that we are switching to TTL only.
If we’ve all been on 1.104.5 for awhile (or should have been)… is this now all true about trash?
Everybody should still have only 4 satellite directories in files/config/storage/trash (pmw6t*, qstuy*, ukfu6* and v4wee*) because the other two were dropped long ago
In those files/config/storage/trash/[satellite] directories we should no longer have any [11]->[zz] subdirectories
Instead, inside those files/config/storage/trash/[satellite] directories we should only see timestamped subdirecties (like today I have 2024-05-18, 2024-05-19, 2024-05-20, 2024-05-22 and 2024-05-25 for ukfu6*). And if things are working properly with deletion: we should never see any of those dated directories with a name older than 8-days-from-today
Those files/config/storage/trash/[satellite]/[date] directories are now where we expect to see all the [11]->[zz] subdirectories
If this is all true… then I think after I’ve been running 1.104.5 for at least 8 days it should be safe (or even recommended) to manually delete any leftover files/config/storage/trash/[satellite]/[11]->[zz] subdirectories?
ls /mnt/node1/storagenode/storage/trash/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/2024-05-19/
22 3o 5c 6w ak c6 ds fg h2 io kc lw nk p6 qs sg u2 vo xc yw
23 3p 5d 6x al c7 dt fh h3 ip kd lx nl p7 qt sh u3 vp xd yx
24 3q 5e 6y am ca du fi h4 iq ke ly nm pa qu si u4 vq xe yy
25 3r 5f 6z an cb dv fj h5 ir kf lz nn pb qv sj u5 vr xf yz
26 3s 5g 72 ao cc dw fk h6 is kg m2 no pc qw sk u6 vs xg z2
27 3t 5h 73 ap cd dx fl h7 it kh m3 np pd qx sl u7 vt xh z3
2a 3u 5i 74 aq ce dy fm ha iu ki m4 nq pe qy sm ua vu xi z4
2b 3v 5j 75 ar cf dz fn hb iv kj m5 nr pf qz sn ub vv xj z5
2c 3w 5k 76 as cg e2 fo hc iw kk m6 ns pg r2 so uc vw xk z6
2d 3x 5l 77 at ch e3 fp hd ix kl m7 nt ph r3 sp ud vx xl z7
2e 3y 5m 7a au ci e4 fq he iy km ma nu pi r4 sq ue vy xm za
2f 3z 5n 7b av cj e5 fr hf iz kn mb nv pj r5 sr uf vz xn zb
2g 42 5o 7c aw ck e6 fs hg j2 ko mc nw pk r6 ss ug w2 xo zc
2h 43 5p 7d ax cl e7 ft hh j3 kp md nx pl r7 st uh w3 xp zd
2i 44 5q 7e ay cm ea fu hi j4 kq me ny pm ra su ui w4 xq ze
2j 45 5r 7f az cn eb fv hj j5 kr mf nz pn rb sv uj w5 xr zf
2k 46 5s 7g b2 co ec fw hk j6 ks mg o2 po rc sw uk w6 xs zg
2l 47 5t 7h b3 cp ed fx hl j7 kt mh o3 pp rd sx ul w7 xt zh
2m 4a 5u 7i b4 cq ee fy hm ja ku mi o4 pq re sy um wa xu zi
2n 4b 5v 7j b5 cr ef fz hn jb kv mj o5 pr rf sz un wb xv zj
2o 4c 5w 7k b6 cs eg g2 ho jc kw mk o6 ps rg t2 uo wc xw zk
2p 4d 5x 7l b7 ct eh g3 hp jd kx ml o7 pt rh t3 up wd xx zl
2q 4e 5y 7m ba cu ei g4 hq je ky mm oa pu ri t4 uq we xy zm
2r 4f 5z 7n bb cv ej g5 hr jf kz mn ob pv rj t5 ur wf xz zn
2s 4g 62 7o bc cw ek g6 hs jg l2 mo oc pw rk t6 us wg y2 zo
2t 4h 63 7p bd cx el g7 ht jh l3 mp od px rl t7 ut wh y3 zp
2u 4i 64 7q be cy em ga hu ji l4 mq oe py rm ta uu wi y4 zq
2v 4j 65 7r bf cz en gb hv jj l5 mr of pz rn tb uv wj y5 zr
2w 4k 66 7s bg d2 eo gc hw jk l6 ms og q2 ro tc uw wk y6 zs
2x 4l 67 7t bh d3 ep gd hx jl l7 mt oh q3 rp td ux wl y7 zt
2y 4m 6a 7u bi d4 eq ge hy jm la mu oi q4 rq te uy wm ya zu
2z 4n 6b 7v bj d5 er gf hz jn lb mv oj q5 rr tf uz wn yb zv
32 4o 6c 7w bk d6 es gg i2 jo lc mw ok q6 rs tg v2 wo yc zw
33 4p 6d 7x bl d7 et gh i3 jp ld mx ol q7 rt th v3 wp yd zx
34 4q 6e 7y bm da eu gi i4 jq le my om qa ru ti v4 wq ye zy
35 4r 6f 7z bn db ev gj i5 jr lf mz on qb rv tj v5 wr yf zz
36 4s 6g a2 bo dc ew gk i6 js lg n2 oo qc rw tk v6 ws yg
37 4t 6h a3 bp dd ex gl i7 jt lh n3 op qd rx tl v7 wt yh
3a 4u 6i a4 bq de ey gm ia ju li n4 oq qe ry tm va wu yi
3b 4v 6j a5 br df ez gn ib jv lj n5 or qf rz tn vb wv yj
3c 4w 6k a6 bs dg f2 go ic jw lk n6 os qg s2 to vc ww yk
3d 4x 6l a7 bt dh f3 gp id jx ll n7 ot qh s3 tp vd wx yl
3e 4y 6m aa bu di f4 gq ie jy lm na ou qi s4 tq ve wy ym
3f 4z 6n ab bv dj f5 gr if jz ln nb ov qj s5 tr vf wz yn
3g 52 6o ac bw dk f6 gs ig k2 lo nc ow qk s6 ts vg x2 yo
3h 53 6p ad bx dl f7 gt ih k3 lp nd ox ql s7 tt vh x3 yp
3i 54 6q ae by dm fa gu ii k4 lq ne oy qm sa tu vi x4 yq
3j 55 6r af bz dn fb gv ij k5 lr nf oz qn sb tv vj x5 yr
3k 56 6s ag c2 do fc gw ik k6 ls ng p2 qo sc tw vk x6 ys
3l 57 6t ah c3 dp fd gx il k7 lt nh p3 qp sd tx vl x7 yt
3m 5a 6u ai c4 dq fe gy im ka lu ni p4 qq se ty vm xa yu
3n 5b 6v aj c5 dr ff gz in kb lv nj p5 qr sf tz vn xb yv
Personally: I deleted everything not sticking to the proper structure (ie non-migrated directories) manually. I don’t recommend this unless you know what you are doing and don’t want to wait until it’s properly fixed. Used-space needs to be run after you manually delete trash, so keep that in mind.
I believe that if the modification date of each file is in the past at least of 7 days or more it’s safe to delete them manually. However, I would prefer to do not touch anything inside the data location without a strong and mandatory reasons.
I cannot check this myself, my nodes do not have anything older than 7 days there…
What I have noticed (been running 1.104.1 since the day it came out): If the folders fail to migrate to date-based ones, they don’t get automatically deleted even after the required days pass. If they are migrated they are deleted on the 8th day trash-cleanup run. If you are @jammerdan your node is possessed and nothing of this applies
Everyone seem to be complaining about how much data went into trash, I seem to be one of the lucky who didn’t have to take that big a loss so to say. So probably also had something to do with the age of your nodes (mine are 1½ year or less), I assume.
But the 2½TB I lost over a week ago, has been compensated in the meantime by never-seen-before upload speeds (usually 1-2TB/month, I did have that in one week now).
