Updates on Test Data

[nerdatwork]

I saw multiple lines of count: 600k which should update at least 10s of gbs. I had no ingress so the number didnt update at all. I need to further investigate.

2024-06-29T17:04:56Z    INFO    collector       collect {"Process": "storagenode", "count": 664118}
2024-06-29T22:52:07Z    INFO    collector       collect {"Process": "storagenode", "count": 666053}
2024-06-30T05:29:22Z    INFO    collector       collect {"Process": "storagenode", "count": 667016}
2024-06-30T09:20:18Z    INFO    collector       collect {"Process": "storagenode", "count": 667057}
2024-06-30T11:11:36Z    INFO    collector       collect {"Process": "storagenode", "count": 638589}

[Alexey]

Accordingly this issue:

Updates on Test Data

Not on the nodes… It seems deleting expired TTL pieces does not update the used space stats.

I think it’s not updating databases with a new free space.
So the workaround would be the same - restart the node with a used-space-filewalker enabled, until the new version would be rolled out.

[Alexey]

It’s interesting. It should be deleted. Just like @Vadim said - with the same time intervals (if the TTL was the same though).

[snorkel]

What’s the point? The TTL data is removed daily. The startup piecescan is run only at restart and it takes days. Should we keep restarting the nodes when the FW finished? It’s pointless. Just fix the bug and release those updates quicker. THIS SHOULD BE THE PRIORITY NUMBER ONE!

[JoshieGarza]

this could explain why 12 days after reducing the allocated space (from several TBs to 1TB because i want to migrate the SN to another disk faster), the used space is still the same.

[Alexey]

I do not think that you need to restart it daily. However, If your node is rendered as full, but you have a free space, you may restart it to make updates.

And I do not suggest it as a fix, it’s a workaround until the fix would be rolled out.

[flwstern]

I think you have to do some redesigns alexey. To much issues for everyone.

[Alexey]

I know, and there is a work in the team, but lightly exposed right now
like

Disk usage discrepancy?

Ah, if we’re talking about the manipulation of the files, then yes, as you rightly point out, we have lots of room for improvement. I was only talking above about the directory traversal piece by itself. Your suggestions are good ones and some we’ve talked about doing before. Some are harder than others. For example, giving up on the temp dir would mean we need some other way to differentiate partial uploads from full, intact blobs. We could try to use a database to keep track of what files a…

Has it ever been considered to use in-memory databases?

It would require a separate chore to move data from the operational database to the historical database, it also will require to redo how this information is handled in the storagenode API (it must combine data from two sources (three if we account both operational databases - in-memory and on the disk, also historic DB) for each metric). it’s as complicated as it sounds. As result - it likely slow down the node and anything what would request these metrics. I do have more hope into this resea…

Design draft: a low I/O piece storage

Thanks for this design doc @Toyoo, It’s a well written design, I like it. And finally I had some time to collect my thoughts. TLDR; I like the approach, but I am not sure about the implementation. So far I have heard about multiple attempt to improve the piece store. They are usually save the metadata of pieces in a database like system and save the raw data in custom files. (technically it would be possible to store both metadata and data in database, but it’s quite slow. I tried it, and mea…

[Roxor]

Storj itself is relatively lightweight: even with current ingress. But seemingly running used-space-filewalker 24x7… 100% iops all-the-time… does get tiring. Good thing I’m a patient man

[Andrew]

Devs, do you recommend to update to 1.107.3? Or should we wait something?

[snorkel]

Wait! Because if you recreate the container, it could go down a version and the databases are incompatible. You will end up updating it again or deleting the databases.

[pangolin]

@snorkel

it could go down a version

The version server url is not hard coded. This can be used to avoid such downgrades or to let it install any version you want.

[littleskunk]

@Andrew

Devs, do you recommend to update to 1.107.3? Or should we wait something?

We have started the normal rollout procedure. Please be patient and follow the rollout process. If we would all upgrade our nodes by hand we are risking to lose customer data. The rollout procedure is designed to mitigate that risk. If you feel the rush to install it faster please don’t do it on all of your nodes all at once. Better pick a single node first, let it run for a few hours - a day, upgrade like 50% of your nodes , wait again and last but not least the remaining nodes.

[PePeR]

Are you running some crazy benchmarks right now? Last 2 days all my nodes restart every few hours

[arrogantrabbit]

Find out why do they restart and fix it. It’s in no way normal.

[PePeR]

I think having 300gb egress of daily repair isn’t normal…

[littleskunk]

@PePeR

Are you running some crazy benchmarks right now?

I would say not more crazy than on other days. It should still be the same load.

What has changed is the TTL cleanup that now comes on top. Maybe that is too much for your storage node?

[PePeR]

probably… I will monitor it for few days and if its gonna keep going Im gonna just change config to full drives to not get more ingress. This should fix it temporarly

[Roxor]

Last few days look like normal 75%-crazy: nothing special.

[Andrew]

Two nodes were updated to version 1.107.3. After the update, many records appeared in the logs like this:

2024-07-02T23:11:27Z WARN collector unable to delete piece {“Process”: “storagenode”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Piece ID”: “DA3NATNM5X6BWSRR6XUSOOB4AIP745ERCBSPV5VDVZXXF5G2TFRA”, “error”: “pieces error: filestore error: file does not exist”, “errorVerbose”: “pieces error: filestore error: file does not exist\n
tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).Stat:122\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:270\n\tstorj.io/storj/storagenode/pieces.(*Blobs
UsageCache).DeleteWithStorageFormat:250\n\tstorj.io/storj/storagenode/pieces.(*Store).DeleteSkipV0:355\n\tstorj.io/storj/storagenode/collector.(*Service).Collect.func1:88\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceExpirationDB).GetExpired:71\n\tstorj.io/storj/storagenode/pieces.(*Store).GetExpired:575\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:
87\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:65\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/storj/storagenode/collector.(*Service).Run:61\n\tstorj.io/storj/p
rivate/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}

Why did they appear? Is this expected?

Before the update there was no errors (log grepped by “collect”):

2024-07-02T09:43:15Z INFO collector collect {“Process”: “storagenode”, “count”: 28}
2024-07-02T10:43:14Z INFO collector collect {“Process”: “storagenode”, “count”: 18}
2024-07-02T11:43:44Z INFO collector collect {“Process”: “storagenode”, “count”: 727}
2024-07-02T12:43:43Z INFO collector collect {“Process”: “storagenode”, “count”: 1027}
2024-07-02T13:43:22Z INFO collector collect {“Process”: “storagenode”, “count”: 304}
2024-07-02T14:43:55Z INFO collector collect {“Process”: “storagenode”, “count”: 1356}
2024-07-02T15:43:42Z INFO collector collect {“Process”: “storagenode”, “count”: 696}
2024-07-02T16:43:47Z INFO collector collect {“Process”: “storagenode”, “count”: 1075}
2024-07-02T17:43:38Z INFO collector collect {“Process”: “storagenode”, “count”: 970}
2024-07-02T18:44:12Z INFO collector collect {“Process”: “storagenode”, “count”: 2015}
2024-07-02T19:43:43Z INFO collector collect {“Process”: “storagenode”, “count”: 1016}
2024-07-02T20:43:14Z INFO collector collect {“Process”: “storagenode”, “count”: 13}
2024-07-02T21:43:14Z INFO collector collect {“Process”: “storagenode”, “count”: 23}
2024-07-02T22:43:15Z INFO collector collect {“Process”: “storagenode”, “count”: 27}