This sounds interesting.
However in my config the storage directory destination is /app/storage. As the temp folder is located in the storage folder shouldn’t the destination therefore be /app/storage/temp?
I just want to make sure the node will find and use the new temp location.
Nvm. Tried it and received failed upload errors.
With
it seems to work. At least I don’t see failed uploads now. Now let’s wait and check if this is going well.
When I check the temp folder on the disk, I can still see new partial files being created.
So it does not seem to work as it should.
They will show up there, these files are however stored in RAM then swap in case of tmpfs.
You mean they are supposed to show up in the temp folder on the disk although they get stored in RAM? This does not sound right.
But even if that is how it should be, then after removal of the container, the should disappear. But I can still see them. I have just tested it and there are numerous new partial files with current timestamp and they remain there even after removal of the container. I can even open the partial files.
When I use --mount type=tmpfs,destination=/app/storage/temp then I get ERROR piecestore upload failed with invalid cross-device link when it tries to rename/move the partial file as sj1 piece.
Correct, this way it is transparent to the application. I use the same for storagenode logs and if I reboot the system the logs are gone.
Not sure how it works in Docker, but maybe you would have to restart the whole host/Docker for the files to be gone.
Also, if you had files on drive at that location before you mounted tmpfs, then after unmounting tmpfs the original files at that location stored on drive will be displayed.
Yes, but then I should no longer see the files that have been created while the tmpfs was mounted, right?
But I still see those files even after removal of the container and as said I can even open them. So they are still present.
Correct.
This shouldn’t be the case.
There is more about how it works in Docker here: tmpfs mounts | Docker Docs. From the text it looks like it only works on Linux hosts however, so if you are using Windows maybe this doesn’t work there.
No I am talking about Linux too.
Maybe my issue is this. I have already a bind mount for /app/storage in the docker run command. Now when I add a tmpfs mount for /app/storage/temp or /app/config/storage/temp, maybe it is not supposed to work as there is already a bind mount for the storage destination?
Oh, this is interesting. I assumed libuplink already can handle this case, as the same observed behavior might be caused by simple data loss on node’s HDD. File might have been truncated after a bad shutdown, or simply some garbage substituted for metadata.
This is also interesting. Unless a node would be shut down uncleanly, the files would still be removed as part of the upload procedure if the node knows the upload was canceled or failed. As such, the only leftover files would be due to pending uploads during an unclean shutdown, which would be maybe tens of megabytes. Insignificant at node’s scale, and certainly not worth correcting for if it requires more I/O.
The node wouldn’t have to do it at all. GC would do it as part of normal operation, as file names wouldn’t be matched by the bloom filter.
I can’t verify this now, but IIRC not during the file walker process unless a piece is to be trashed.
Sure. Though, the idea is that if we take an assumption that all partial uploads ever will always be stored in RAM, with no option to have partial uploads stored on disk, then we can change the structure of data on disk to make it more IOPS-efficient.
That’s a quality of implementation issue. A basic measure is the fact that a node restart will free up RAM of all pending uploads. Another measure, which is not the default now, but can be turned on even now, is upload speed—no upload should take more than few minutes, after all.
You can’t, the rename() syscall doesn’t work across file systems. The storage node code would have to explicitly copy the file, this is not implemented.
2 Likes
@Toyoo is right- don’t use a tempfs for the partial uploads directory. The benefit of keeping the storage and temp directories on the same device is that once blobs are complete, they can be moved into the live storage directory atomically.
1 Like
Seems I was wrong:
Yes, but no problem. It’s always worth a try to see if the node behavior can be improved.
1 Like
Then I don’t know what process @thepaul is referring to when claiming that temp files get deleted on startup.
My node is around 9TB, but the space statistics are not correct.
I recently deleted all the .db files because there was an error in some of the files.
What can I do ?
investigate why the filewalk does not finish.?
provide logs? try this example here.
3 Likes
I was wrong, as some people pointed out. There isn’t such a thing. Maybe I’m remembering something from the past or something we planned to do but haven’t yet.
You’re right, the uplink can handle that case, as long as there are enough other input streams. I didn’t think that through.
This is what I’d expect also, but I’ve heard we have people who end up with massive temp directories containing gigabytes of detritus. It’s not clear to me whether there is a bug in the node code that fails to delete temp files when an upload fails, or if those nodes are restarting a lot more frequently than I think they should. I also don’t know how widespread of a problem that is; maybe it’s nothing to worry about.
Overall, I am intrigued by your suggestion and I’ll be eager to read your storage architecture proposal, if you get a chance to put it together.
2 Likes
Had this temp/*.partial problem (detected) in June 2023, since then, its gone. (just checked)
I think people notice this propably old problem way later as it happened, or only when the node has undetected bigger problems, that cause constant restarts, unplausible stats, or downtime.
like the 1min timeout problem that so many sno had. and some have.
maybe depends on the configuration of the node( databases on ssd??). (write buffer size? there was a change from 128kib to 4Mib) afaik
Would be interesting to know wich version was running at the time the .partial was “leftover”.
Accordingly storagenode setup --help | grep buffer it’s still 128KiB
--filestore.write-buffer-size memory.Size in-memory buffer for uploads (default 128.0 KiB)