16TB full node, version 1.105.? to 1.107.3 database migration took over 70 minutes.
Running on Intel Xeon, databases are on some cheap 512GB SATA SSD, piece_expiration.db was 17GB before the migration.
This should be deployed with caution, as it will probably take much much longer on nodes having databases on HDDs.
2024-07-03T01:06:19+02:00 INFO db.migration.58 Remove unused trash column {"Process": "storagenode"}
2024-07-03T01:26:27+02:00 INFO db.migration.59 Remove unused deletion_failed_at column {"Process": "storagenode"}
2024-07-03T01:44:42+02:00 INFO db.migration.60 Overhaul piece_expirations {"Process": "storagenode"}
2024-07-03T02:18:45+02:00 INFO db.migration Database Version {"Process": "storagenode", "version": 60}
And also seeing significant amounts of "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist" messages as the trash-cleanup-filewalker is running on the node.
Just a guess, but this seems related to the fact that some of the initial space reservation data was overwritten due to insufficent randomness.
It would make sense that these pieces were deleted by the GC, as we experienced a couple of weeks ago, and now that the TTL is reached as defined in the db, its unable to find these previously deleted files.
If this is the case, the warnings should not be cause for concern.
On the other hand, if we start to experience elevated amounts of reservation data being deleted by GC instead of TTL, it could mean that there is an underlying issue with the TTL collector and its not correctly deleting the files (because it cant find them).
Out of curiosity I started a saltlake only node today. This seems to be what the new node selection algo likes, it runs like crazy. Not a big surprice.
If you want to see those, turn on debug logging. I see them though I’m starting to think debug logging is a bad idea as those writes are now a significant part of the io load for GC and TTL deletes.
That is never a good idea. A test data only strategy sounds good short term but will backfire long term. The size of this node is limited (TTL) and fully dependent on storj itself. Sure it sounds like we will get additional testdata uploads for up to 3 months but one way or the other I would expect this income source to be lowered and ideally replaced if not outmatched by real customer uploads. At that point you would be looking at the wrong satellite.
And you will have to wait for vetting, in the same time when we get all the juicy customer data. Smart approach.
Believe me, I thought about every strategy, including this one. This one is bad.
You will only see your node shrinking, and we will have the same occupation.
Just wanted to provide a quick update. For the upcoming holiday / long weekend we are going to pause the uploads for a few days. The repair queue has grown a bit and we don’t want to be paging engineers over the holiday. We will re-evaluate on Monday and determine next steps from there and will let you know
I guess this is because of the new node selection algorithm. When serving only one satellite there is a better chance to be constantly selected and the benefit seems to be bigger than the penalty for not serving the other 3 satellites.
Next test will be 4 nodes per HDD but only one satellite each. BTW: This should not be a problem for data security (TOS) because satellites are independent.
Ah… a few days to experience natural free-range organic traffic again: I had forgotten what it was like! Artisanal ones and zeroes from a small selection of actual paying customers!
A reminder that the “new normal” is not yet normal…
