Of my 25 nodes, four seem to keep the old style trash folders although already having being upgraded to 1.104.5 for over a week now. Actually I already saw the first new style folders being deleted (2024-05-18 for each satellite).
So I decided to fix it on these nodes myself, and wanted to share my solution for those in need for the same solution:
People think its just you because you have been very vocal about it but there are plenty of SNOs that would have this issue, me included.
I have to say this out of concern that repeating the issue over and over makes you being categorized as a complainer/whiner. All the good you have done before the complaints will be overlooked. You very well know that the right people have taken note of your issue. If there is a solution then it will be deployed otherwise lets go by the manual route to fixing the issue. Given that a manual solution is available like deleting the trash folders outside date folders.
If nobody complains, but only one, you can’t blame people of wrongful thinking. It’s good to complain, to let devs know the situation in the field.
Me, I didn’t had any problems on my nodes, even on the slowest one, and seeing that only jammerdan was complaining, I though, like many others, he must have some issues with his setup.
[...wd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa]# ls -a
. 2m 3b 3w 4l 5a 5v 6k 77 7u aj b6 bt ci d5 ds eh f4 fr gg h3 hq if j2 jp ke kz lo md my nn oc ox pm qb qw rl sa sv tk u7 uu vj w6 wt xi y5 ys zh
.. 2n 3c 3x 4m 5b 5w 6l 7a 7v ak b7 bu cj d6 dt ei f5 fs gh h4 hr ig j3 jq kf l2 lp me mz no od oy pn qc qx rm sb sw tl ua uv vk w7 wu xj y6 yt zi
2024-05-22 2o 3d 3y 4n 5c 5x 6m 7b 7w al ba bv ck d7 du ej f6 ft gi h5 hs ih j4 jr kg l3 lq mf n2 np oe oz po qd qy rn sc sx tm ub uw vl wa wv xk y7 yu zj
22 2p 3e 3z 4o 5d 5y 6n 7c 7x am bb bw cl da dv ek f7 fu gj h6 ht ii j5 js kh l4 lr mg n3 nq of p2 pp qe qz ro sd sy tn uc ux vm wb ww xl ya yv zk
23 2q 3f 42 4p 5e 5z 6o 7d 7y an bc bx cm db dw el fa fv gk h7 hu ij j6 jt ki l5 ls mh n4 nr og p3 pq qf r2 rp se sz to ud uy vn wc wx xm yb yw zl
24 2r 3g 43 4q 5f 62 6p 7e 7z ao bd by cn dc dx em fb fw gl ha hv ik j7 ju kj l6 lt mi n5 ns oh p4 pr qg r3 rq sf t2 tp ue uz vo wd wy xn yc yx zm
25 2s 3h 44 4r 5g 63 6q 7f a2 ap be bz co dd dy en fc fx gm hb hw il ja jv kk l7 lu mj n6 nt oi p5 ps qh r4 rr sg t3 tq uf v2 vp we wz xo yd yy zn
26 2t 3i 45 4s 5h 64 6r 7g a3 aq bf c2 cp de dz eo fd fy gn hc hx im jb jw kl la lv mk n7 nu oj p6 pt qi r5 rs sh t4 tr ug v3 vq wf x2 xp ye yz zo
27 2u 3j 46 4t 5i 65 6s 7h a4 ar bg c3 cq df e2 ep fe fz go hd hy in jc jx km lb lw ml na nv ok p7 pu qj r6 rt si t5 ts uh v4 vr wg x3 xq yf z2 zp
2a 2v 3k 47 4u 5j 66 6t 7i a5 as bh c4 cr dg e3 eq ff g2 gp he hz io jd jy kn lc lx mm nb nw ol pa pv qk r7 ru sj t6 tt ui v5 vs wh x4 xr yg z3 zq
2b 2w 3l 4a 4v 5k 67 6u 7j a6 at bi c5 cs dh e4 er fg g3 gq hf i2 ip je jz ko ld ly mn nc nx om pb pw ql ra rv sk t7 tu uj v6 vt wi x5 xs yh z4 zr
2c 2x 3m 4b 4w 5l 6a 6v 7k a7 au bj c6 ct di e5 es fh g4 gr hg i3 iq jf k2 kp le lz mo nd ny on pc px qm rb rw sl ta tv uk v7 vu wj x6 xt yi z5 zs
2d 2y 3n 4c 4x 5m 6b 6w 7l aa av bk c7 cu dj e6 et fi g5 gs hh i4 ir jg k3 kq lf m2 mp ne nz oo pd py qn rc rx sm tb tw ul va vv wk x7 xu yj z6 zt
2e 2z 3o 4d 4y 5n 6c 6x 7m ab aw bl ca cv dk e7 eu fj g6 gt hi i5 is jh k4 kr lg m3 mq nf o2 op pe pz qo rd ry sn tc tx um vb vw wl xa xv yk z7 zu
2f 32 3p 4e 4z 5o 6d 6y 7n ac ax bm cb cw dl ea ev fk g7 gu hj i6 it ji k5 ks lh m4 mr ng o3 oq pf q2 qp re rz so td ty un vc vx wm xb xw yl za zv
2g 33 3q 4f 52 5p 6e 6z 7o ad ay bn cc cx dm eb ew fl ga gv hk i7 iu jj k6 kt li m5 ms nh o4 or pg q3 qq rf s2 sp te tz uo vd vy wn xc xx ym zb zw
2h 34 3r 4g 53 5q 6f 72 7p ae az bo cd cy dn ec ex fm gb gw hl ia iv jk k7 ku lj m6 mt ni o5 os ph q4 qr rg s3 sq tf u2 up ve vz wo xd xy yn zc zx
2i 35 3s 4h 54 5r 6g 73 7q af b2 bp ce cz do ed ey fn gc gx hm ib iw jl ka kv lk m7 mu nj o6 ot pi q5 qs rh s4 sr tg u3 uq vf w2 wp xe xz yo zd zy
2j 36 3t 4i 55 5s 6h 74 7r ag b3 bq cf d2 dp ee ez fo gd gy hn ic ix jm kb kw ll ma mv nk o7 ou pj q6 qt ri s5 ss th u4 ur vg w3 wq xf y2 yp ze zz
2k 37 3u 4j 56 5t 6i 75 7s ah b4 br cg d3 dq ef f2 fp ge gz ho id iy jn kc kx lm mb mw nl oa ov pk q7 qu rj s6 st ti u5 us vh w4 wr xg y3 yq zf
2l 3a 3v 4k 57 5u 6j 76 7t ai b5 bs ch d4 dr eg f3 fq gf h2 hp ie iz jo kd ky ln mc mx nm ob ow pl qa qv rk s7 su tj u6 ut vi w5 ws xh y4 yr zg
Same here and also some leftovers from april:
[...aogq343bfkh2n5vvg4ohqqgggrrunaaaaaaaa]# ls -a
. .. 2024-04-18
But look, there are several different manifestations of the issue.
You know when @thepaul wasn’t even aware of additional ones:
I don’t know if this is onlyl one issue with the same cause as I see several different manifestations where each might require a different approach to get rid of it. So one thing is to make devs aware of the wide range of weird outcomes in the trash.
That’s one thing. The other thing is that I would like to know if Storj is going to offer an automatic solution to fix this. I mean they would have to deal with all those variations that I am not sure if there is an easy fix (Well, there is).
All we have seen so far is the suggestion from @clement to manually sort it out:
But what we don’t have is a word from Storj that this is going to be fixed, so don’t touch it, there is no issue on Github that somebody is working on it or when to follow progress.
And again the worst thing is happening: Trash does not get deleted, blocking new ingress on full nodes. And the case that garbage accumulates without getting deleted is the one big issue from recent problems and we hoped we don’t see something like this again soon.
It’s requirement for the running xargs, see man xargs.
You may either quote it like this \; or use a literal strings like ';', just \; is shorter than ';'.
When I ran this it worked well on a couple of nodes, but on one it left subfolders behind:
ls /storage/trash/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa
2024-05-25 2024-05-27 2024-05-29 au os
Which is weird because both were empty.
I get that feiling that maybe if one of the filewalkers is going over the files maybe they cannot be accessed or deleted. Maybe something like this was the reason why we are in this situation in the first place.
Now after running the command a second time, the 2 leftovers are gone.
Thinking of it, when the folders, which are not empty, get directly removed, this will not change the databases, correct? So the free amounts would stay the same?
So wouldn’t it be better instead of removing the subfolders to move them into a new date folder with some very old date. That way they would get deleted with the next trash chore and then the databases would get updated as well?
The idea would be to move all 2char subfolders into a very old (1970-01-01) subfolder that would get purged 100% on the next trash chore.
As I am not a coder I cannot guarantee that it will work. But I tried on 2 nodes and they moved all 2character subfolders into a 1970-01-01 date folder.
So maybe it does work.
I can tell you it won’t work.
Because the move shouldn’t be conditional.
But more important is that it’s unnecessary complicating stuff. Since it’s inherently slower, because in the end those directories have to be deleted anyways. Your proposal is first cluttering the meta data a bit. Than you’re hoping it’s working the way you suppose it works, but you have to wait some days to see that happen.
I’m just relying on the used space filewalker and I made sure, only to run it after the node was upgraded to 1.104.5 for over a week.