Huh, I wasn’t aware of those. That would definitely break trash collection. That means the per-day dir transition ran more than once, but it’s not clear how that could happen. If you haven’t deleted them yet, could you show the timestamps on:
/storage/trash/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/2024-04-13/storage/trash/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/2024-04-13/2024-04-13/storage/trash/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/2024-04-13/2024-04-13/2024-04-13/storage/trash/.trash-uses-day-dirs-indicator
(preferably with stat(1) or something like it to show subsecond precision)?
If the higher layers and the indicator file all have birth times within a few microseconds of each other, then perhaps there are multiple things “opening” the blobstore layer at the same time for some reason.
It’s 100% safe for you, the operator, as long as you know things like: you haven’t put anything in there yourself, there aren’t any bind mounts under that point, etc. For the part of the storagenode software, it has to avoid deleting things that aren’t expected, in case of configuration mistakes or a bug causing confusion about where important files are.