Large amounts of trash on europe-west-1

Recently it seems europe-west-1 has moved a large amount of data to trash. Over 400GB on my node.
The pieces seem to be mostly from the period of june 2019 - October 2019.

The entire thing seems very similar to a cleanup that happened a while ago on stefan benten’s satellite, where zombie segments were cleaned up on the satellite end. However, I can’t be sure it’s the same thing and just wanted to check if others are seeing this too.

For reference, the europe-west-1 folder is: v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa

You’ll almost certainly only see this issue if your node existed prior to September 2019.

1 Like

Over the weekend my node also lost 200GB, I didn’t check from which satellite this data was. What I did notice was that the cli dashboard was still showing the full space, while the web dashboard reflected the 200GB.

1 Like

This is expected behavior. The CLI dashboard doesn’t separate out trash. Since the data is still taking up space on the disk.

Thanks for sharing your confirmation though. You can easily check if it’s the same satellite by checking the size of the trash folder as mentioned in the first post.

Almost 250 GB that were moved to trash here.

1 Like

111 GB in trash for europe-west but my node was only 4 TB during the time of june 19 - oct 19.
2nd node with less than 1TB at the time also got 97 GB in trash for europe-west.

1 Like

49 GB moved to trash from europe-west-1,

happened last night i think it was… sure did make the node activity spike a bit, was fun to actually see it do a bit of work and actually eat some cpu time… even if it was for a brief few minutes :smiley:

another thing of node… i usually always have about a few gb in trash… after my new slog device and thus improved write latency, my trash seems to have reduced down to the MB level, nothing conclusive… but found it interesting… was just thinking about that shortly before trash blew up lol

Yeah, prior to this I saw the trash on my node drop to the MB level as well, so I’m not sure that would be related.

Anyway, it looks like this was indeed a satellite side cleanup. So nothing to worry about. More free space for making money! :slight_smile:

i guess storj’s tinkering or the small nodes complaining about disk space used for trash paid off…
did happen very close to when i got the slog device working right… but not easy to tell, would be nice if there was like some sort of node avg data point one could compare to… ofc just creating that is kinda tricky.

right because new data gives more egress than old that … not sure the difference is that huge tho… but meh duno…

figured that the more data the better, who knows that kind of interesting things might be concluded from it…

sure would be a sizable deletion if this happened on most nodes… i mean if this was across most nodes and what 100-200 gb avg maybe… thats 600-1200 tb deleted and even if we super low ball the avg then its like easy past 100tb

can we see that anywhere… the amounts of data on the network currently?

Pre NASA cleanup. :grin:
Nah, just kidding.

2 Likes

I saw a similar trend on my node that’s been around since Aug 2019. ~80GB was moved to the europe-west-1 trash.

1 Like

Well, you’d have to go by the number of nodes that were available at the time these pieces were uploaded. That was quite a bit lower than today. I don’t know exactly, maybe 2000-3000. So yeah, probably a few hundred TB. In the scale of what Storj has been uploading over time it’s still peanuts, which I guess speaks for the size of the network. I doubt anybody will be sorely missing this data. Besides, this data was almost certainly never downloaded, no longer owned by customers and only paid for static storage. Better to make room for actual used data I’d say.

yeah it’s almost quite difficult to understand just how much data this is we are talking about…

like one can have a few years of fairly high quality video on 4 tb maybe even 2tb with h265
so a few hundred tb… thats like 50-100 years of HD quality streaming… or so… lol
boggles the mind… not going to be long before storage capacity like this could literally without the shadow of a doubt contain consciousness… in pural

i guess we are only waiting for the computation / coding or bandwidth before ai rises from the ashes

We are waiting for large scale Tardigrade customers to finally hook up some huge drives.

1 Like

Confirmed, my 4TB node dumped 140 GB into the europe-west trash folder some time in the last 24 hours.

Speaking of, anyone else see this new AI powered SSD? This looks like some next level $hit.

I have absolutely no use case for anything like this at the moment, but I was super tempted to back it.

wendel from level1 tech’s the youtube channel seems to like it, if memory serves…
very interesting stuff… basically an ssd for ssd’s to improve their speed, but yeah very next level stuff
ofc it can be used to improve the speed of any storage medium or most anyways…

basically it does smart caching and thus will basically be able to predict what comes next and thus can reach even closer to zero latency than was previously possible, ofc the use cases can be quite difficult… some workloads would gain from it while others might barely even see a change…

most likely also does FPGA, however QLC chips can all to my understanding run as SLC so that part is… a bit standard, really its the whole smart thing one wants… and really imo wouldn’t one want that a place where it’s a bit more upgradable… but certain pretty cool stuff
almost considered it for my SLOG upgrade… but didn’t think it would be possible to get on a limited time schedule, and state of the art is always pretty expensive… and then the whole…
put up the smart on the device… tho i have seem some amazing smart storage, processing storage, cpu storage or whatever its called when they put high end processing directly into the each harddrive in a large storage setup and then do like live math on the storage or process the data without using the cpu…

might have run into it when watching something about the 1square mile telescope or whatever its called… think they are partly developing stuff like that for their storagemedium to when the telescope goes onlike in a few years… i could also imagine doing processing directly on the drives might be pretty effective… ofc upgrading it seems… tricky … but then again doesn’t need much processing to do stuff like searching… and instead of having all search data go through the cpu, then keeping stuff like that internally in the drives makes perfect sense.

i think i saw a setup that could adapt even if one removed like 70% of the storage it would just get slower because of how it was storing it in something like erasure coding or something…
i forget where the limit was before it would crash… but was very impressive stuff.

alas i digress
yeah fuzedrive looks pretty cool, and wendel sure seemed impressed, haven’t had time to really dig into the project, even if the tech is cool… the companies like nikola sure did manage to take a potential big business and just crash and burn it in a spectacular way… and there are many such pitfalls… i wouldn’t expect support on such devices like fuzedrive in 5 years… ofc it maybe super popular and have support in 50… requires some serious deep dives to predict such stuff… but cool non the less…

Is someone running a storage node since 22.09 22:00 UTC+2 without a restart? Mine has restarted and now I can’t look into monkit.

i’m only at 101hr currently… i would think @dragonhogan would be a good bet if he is still around… his uptime is usually pretty great if memory serves.
i’ve been tinkering to much lately

I have two nodes that have been up since 2020-09-15 13:00 UTC without restarts. One of the nodes has only existed since that time. Although I’ve never tried anything relating to monkit. Both docker v1.12.3.

1 Like

What I am interested in is the mon/stats output especially piecedeleter-queue-full

If you haven’t specified the debug port the storage node will open a random port. Changing it now would require a restart and that means the data is lost. If you haven’t specified the port you might need to connect to the docker container and find out which port it is. You will not be able to reach the port from outside the docker container.