Same:
trash/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/2024-04-13/2024-04-13/2024-04-13/2a
Does not look right.
1 Like
I meant a storagenode update, like from v.1.99.3 → v1.100.3. which has its rollout started.
As well as the bloom filter from yesterday.
Now when my node is processing a bloom filter and receives an update. My guess is it will restart then and the processing of the bloom filter will not complete.
We changed recently, how we will store trash:
why?
But why have nested date folders?
Because of this: GC bloom filter should be stored on disk in case of node restart · Issue #6725 · storj/storj · GitHub
This is just an idea. It was never considered it as a solution, sorry, if I confused you.
I do not implement things (yet), so this is only a proposal.
I guess you read the description of the issue? I believe, It should answer this question.
I thought it should save the filter and continue when it was interrupted?
@Alexey Yes, that’s what I said.
Roughly the same time. It took 12 hours to send out all the bloom filters.
See the commit below. Recent improvements make it possible to:
- save bloom filter locally (and start from the beginning if it’s not finished).
- resume deletion process
1 Like
This looks right to you?
storage/trash/qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa/2024-04-13/2024-04-13/2024-04-13/2024-04-13/2024-04-13/2024-04-13/2024-04-13
How many more nested folders can I expect under it?
This is live In version 1.99.3 already?
1 Like
Nested date folders in trash makes it very easy to purge old trash data. You don’t need to check each trash files on by one for the deletion time (which is the creation date of the trashed file). You can just delete folders which are old enough…
Not a primary goal, but it also makes it easier to debug what happened…
3 Likes
I don’t understand the need for nested folders even after reading the github issue. In my mind the process should be this: on the first update to a version that uses the date folders, assume that all files under trash were saved before today’s date (which they have, otherwise something else is wrong), and store them in today’s trash folder. Any files trashed tomorrow should go in tomorrow’s trash folder and so on.
I don’t think we even need separate “top level” folders for every satellite (read whole paragraph before commenting). It doesn’t matter if a file was from US1 or EU1. If its 7 days are gone, that file will be deleted regardless. If US1 blooms today and EU1 blooms tomorrow, the files will be deleted with 1 day difference since they will be stored under 2024-04-12/US1 and 2024-04-13/EU1, or am I missing something? Saves using separate “top level” directories for the satellites and prevents nesting dates. Has the “top level” directory aged enough to be deleted? If yes, delete it.
1 Like
Right. But we see multi-nested folders with the same date:
/storage/trash/qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa/2024-04-13/2024-04-13/2024-04-13/2024-04-13/2024-04-13/2024-04-13/2024-04-13/2a
This appears weird. But if this is how it should be then everything is fine.
2 Likes
Adding to that:
/storage/trash/2024-04-13/qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa/
That’s what it should be (see my previous comment).
Can you check if any of the directories under /storage/trash contain a file named .trash-uses-day-dirs-indicator?
EDIT: And a second question: how many entries with migrating trash namespace to use per-day directories do you see in your logs?
drwx------ 3 node11 node11 4096 Apr 12 19:38 pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa
drwx------ 3 node11 node11 4096 Apr 12 19:38 qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa
-rw-r--r-- 1 node11 node11 23 Apr 12 19:38 .trash-uses-day-dirs-indicator
drwx------ 4 node11 node11 4096 Apr 13 04:14 ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa
drwx------ 3 node11 node11 4096 Apr 12 19:38 v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa
Version v1.101.3
Digging further in:
root@store2 ~ # ls /home/node11/storj/storagenode/storage/trash/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/2024-04-12/
22/ 2t/ 3m/ 4f/ 56/ 5x/ 6q/ 7j/ ac/ b3/ bu/ cn/ dg/ e7/ ey/ fr/ gk/ hd/ i4/ iv/ jo/ kh/ la/ lz/ ms/ nl/ oe/ p5/ pw/ qp/ ri/ sb/ t2/ tt/ um/ vf/ w6/ wx/ xq/ yj/ zc/
23/ 2u/ 3n/ 4g/ 57/ 5y/ 6r/ 7k/ ad/ b4/ bv/ co/ dh/ ea/ ez/ fs/ gl/ he/ i5/ iw/ jp/ ki/ lb/ m2/ mt/ nm/ of/ p6/ px/ qq/ rj/ sc/ t3/ tu/ un/ vg/ w7/ wy/ xr/ yk/ zd/
24/ 2v/ 3o/ 4h/ 5a/ 5z/ 6s/ 7l/ ae/ b5/ bw/ cp/ di/ eb/ f2/ ft/ gm/ hf/ i6/ ix/ jq/ kj/ lc/ m3/ mu/ nn/ og/ p7/ py/ qr/ rk/ sd/ t4/ tv/ uo/ vh/ wa/ wz/ xs/ yl/ ze/
25/ 2w/ 3p/ 4i/ 5b/ 62/ 6t/ 7m/ af/ b6/ bx/ cq/ dj/ ec/ f3/ fu/ gn/ hg/ i7/ iy/ jr/ kk/ ld/ m4/ mv/ no/ oh/ pa/ pz/ qs/ rl/ se/ t5/ tw/ up/ vi/ wb/ x2/ xt/ ym/ zf/
26/ 2x/ 3q/ 4j/ 5c/ 63/ 6u/ 7n/ ag/ b7/ by/ cr/ dk/ ed/ f4/ fv/ go/ hh/ ia/ iz/ js/ kl/ le/ m5/ mw/ np/ oi/ pb/ q2/ qt/ rm/ sf/ t6/ tx/ uq/ vj/ wc/ x3/ xu/ yn/ zg/
27/ 2y/ 3r/ 4k/ 5d/ 64/ 6v/ 7o/ ah/ ba/ bz/ cs/ dl/ ee/ f5/ fw/ gp/ hi/ ib/ j2/ jt/ km/ lf/ m6/ mx/ nq/ oj/ pc/ q3/ qu/ rn/ sg/ t7/ ty/ ur/ vk/ wd/ x4/ xv/ yo/ zh/
2a/ 2z/ 3s/ 4l/ 5e/ 65/ 6w/ 7p/ ai/ bb/ c2/ ct/ dm/ ef/ f6/ fx/ gq/ hj/ ic/ j3/ ju/ kn/ lg/ m7/ my/ nr/ ok/ pd/ q4/ qv/ ro/ sh/ ta/ tz/ us/ vl/ we/ x5/ xw/ yp/ zi/
2b/ 32/ 3t/ 4m/ 5f/ 66/ 6x/ 7q/ aj/ bc/ c3/ cu/ dn/ eg/ f7/ fy/ gr/ hk/ id/ j4/ jv/ ko/ lh/ ma/ mz/ ns/ ol/ pe/ q5/ qw/ rp/ si/ tb/ u2/ ut/ vm/ wf/ x6/ xx/ yq/ zj/
2c/ 33/ 3u/ 4n/ 5g/ 67/ 6y/ 7r/ ak/ bd/ c4/ cv/ do/ eh/ fa/ fz/ gs/ hl/ ie/ j5/ jw/ kp/ li/ mb/ n2/ nt/ om/ pf/ q6/ qx/ rq/ sj/ tc/ u3/ uu/ vn/ wg/ x7/ xy/ yr/ zk/
2d/ 34/ 3v/ 4o/ 5h/ 6a/ 6z/ 7s/ al/ be/ c5/ cw/ dp/ ei/ fb/ g2/ gt/ hm/ if/ j6/ jx/ kq/ lj/ mc/ n3/ nu/ on/ pg/ q7/ qy/ rr/ sk/ td/ u4/ uv/ vo/ wh/ xa/ xz/ ys/ zl/
2e/ 35/ 3w/ 4p/ 5i/ 6b/ 72/ 7t/ am/ bf/ c6/ cx/ dq/ ej/ fc/ g3/ gu/ hn/ ig/ j7/ jy/ kr/ lk/ md/ n4/ nv/ oo/ ph/ qa/ qz/ rs/ sl/ te/ u5/ uw/ vp/ wi/ xb/ y2/ yt/ zm/
2f/ 36/ 3x/ 4q/ 5j/ 6c/ 73/ 7u/ an/ bg/ c7/ cy/ dr/ ek/ fd/ g4/ gv/ ho/ ih/ ja/ jz/ ks/ ll/ me/ n5/ nw/ op/ pi/ qb/ r2/ rt/ sm/ tf/ u6/ ux/ vq/ wj/ xc/ y3/ yu/ zn/
2g/ 37/ 3y/ 4r/ 5k/ 6d/ 74/ 7v/ ao/ bh/ ca/ cz/ ds/ el/ fe/ g5/ gw/ hp/ ii/ jb/ k2/ kt/ lm/ mf/ n6/ nx/ oq/ pj/ qc/ r3/ ru/ sn/ tg/ u7/ uy/ vr/ wk/ xd/ y4/ yv/ zo/
2h/ 3a/ 3z/ 4s/ 5l/ 6e/ 75/ 7w/ ap/ bi/ cb/ d2/ dt/ em/ ff/ g6/ gx/ hq/ ij/ jc/ k3/ ku/ ln/ mg/ n7/ ny/ or/ pk/ qd/ r4/ rv/ so/ th/ ua/ uz/ vs/ wl/ xe/ y5/ yw/ zp/
2i/ 3b/ 42/ 4t/ 5m/ 6f/ 76/ 7x/ aq/ bj/ cc/ d3/ du/ en/ fg/ g7/ gy/ hr/ ik/ jd/ k4/ kv/ lo/ mh/ na/ nz/ os/ pl/ qe/ r5/ rw/ sp/ ti/ ub/ v2/ vt/ wm/ xf/ y6/ yx/ zq/
2j/ 3c/ 43/ 4u/ 5n/ 6g/ 77/ 7y/ ar/ bk/ cd/ d4/ dv/ eo/ fh/ ga/ gz/ hs/ il/ je/ k5/ kw/ lp/ mi/ nb/ o2/ ot/ pm/ qf/ r6/ rx/ sq/ tj/ uc/ v3/ vu/ wn/ xg/ y7/ yy/ zr/
2k/ 3d/ 44/ 4v/ 5o/ 6h/ 7a/ 7z/ as/ bl/ ce/ d5/ dw/ ep/ fi/ gb/ h2/ ht/ im/ jf/ k6/ kx/ lq/ mj/ nc/ o3/ ou/ pn/ qg/ r7/ ry/ sr/ tk/ ud/ v4/ vv/ wo/ xh/ ya/ yz/ zs/
2l/ 3e/ 45/ 4w/ 5p/ 6i/ 7b/ a2/ at/ bm/ cf/ d6/ dx/ eq/ fj/ gc/ h3/ hu/ in/ jg/ k7/ ky/ lr/ mk/ nd/ o4/ ov/ po/ qh/ ra/ rz/ ss/ tl/ ue/ v5/ vw/ wp/ xi/ yb/ z2/ zt/
2m/ 3f/ 46/ 4x/ 5q/ 6j/ 7c/ a3/ au/ bn/ cg/ d7/ dy/ er/ fk/ gd/ h4/ hv/ io/ jh/ ka/ kz/ ls/ ml/ ne/ o5/ ow/ pp/ qi/ rb/ s2/ st/ tm/ uf/ v6/ vx/ wq/ xj/ yc/ z3/ zu/
2n/ 3g/ 47/ 4y/ 5r/ 6k/ 7d/ a4/ av/ bo/ ch/ da/ dz/ es/ fl/ ge/ h5/ hw/ ip/ ji/ kb/ l2/ lt/ mm/ nf/ o6/ ox/ pq/ qj/ rc/ s3/ su/ tn/ ug/ v7/ vy/ wr/ xk/ yd/ z4/ zv/
2o/ 3h/ 4a/ 4z/ 5s/ 6l/ 7e/ a5/ aw/ bp/ ci/ db/ e2/ et/ fm/ gf/ h6/ hx/ iq/ jj/ kc/ l3/ lu/ mn/ ng/ o7/ oy/ pr/ qk/ rd/ s4/ sv/ to/ uh/ va/ vz/ ws/ xl/ ye/ z5/ zw/
2p/ 3i/ 4b/ 52/ 5t/ 6m/ 7f/ a6/ ax/ bq/ cj/ dc/ e3/ eu/ fn/ gg/ h7/ hy/ ir/ jk/ kd/ l4/ lv/ mo/ nh/ oa/ oz/ ps/ ql/ re/ s5/ sw/ tp/ ui/ vb/ w2/ wt/ xm/ yf/ z6/ zx/
2q/ 3j/ 4c/ 53/ 5u/ 6n/ 7g/ a7/ ay/ br/ ck/ dd/ e4/ ev/ fo/ gh/ ha/ hz/ is/ jl/ ke/ l5/ lw/ mp/ ni/ ob/ p2/ pt/ qm/ rf/ s6/ sx/ tq/ uj/ vc/ w3/ wu/ xn/ yg/ z7/ zy/
2r/ 3k/ 4d/ 54/ 5v/ 6o/ 7h/ aa/ az/ bs/ cl/ de/ e5/ ew/ fp/ gi/ hb/ i2/ it/ jm/ kf/ l6/ lx/ mq/ nj/ oc/ p3/ pu/ qn/ rg/ s7/ sy/ tr/ uk/ vd/ w4/ wv/ xo/ yh/ za/ zz/
2s/ 3l/ 4e/ 55/ 5w/ 6p/ 7i/ ab/ b2/ bt/ cm/ df/ e6/ ex/ fq/ gj/ hc/ i3/ iu/ jn/ kg/ l7/ ly/ mr/ nk/ od/ p4/ pv/ qo/ rh/ sa/ sz/ ts/ ul/ ve/ w5/ ww/ xp/ yi/ zb/
No nested dates, still not optimal.
Edit: Not running info logs.
Sorry, this was a question to @jammerdan and @flo to debug the multiply nested date directories.
2 Likes
it’s in /storage/trash
One per satellite.
I have it only on one node currently. That is strange.
I don’t know how this works but maybe there is some wrong coding? Like it tells GC to create the new folder under the last folder, not under the top folder.
1 Like
Looks ok then. If, for some reason, the node was not able to create that file, migration procedure would run multiple times for a single satellite, nesting existing directories. But it seems this is not what is happening with your node.
No. It doesn’t look right. There should be only one folder for deleted items per date.