A couple of days ago (about 2 days after i went from 1.99.3 to 1.101.3) my node started crashing roughly every hour and has not stopped since. Logs showed a lot of this error "Unrecoverable error {“error”: “piecestore monitor: timed out after
1m0s while verifying readability of storage directory”, “errorVerbose”: “piecestore monitor: timed out after 1m0s while verifying readability of storage directory\n\tstorj.io/storj/storagenode/monitor(*Service).Run.func1.1:154\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func1:143\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}
"
I adjusted the timeouts a little higher and there are no new errors in the logs, but the crashes have continued for the last 16 hours. Things i have noticed are a high number of files in my temp directory (475K), but most are only from the last couple of days. I also noticed the memory just keeps climbing for storagenode program for storj. I have seen it get as high as 8 GB or so, which was below my allotment of 24 GB. I bumped the memory up to 120 GB just to give it plenty of room, but that hasn’t helped.
Setup:
Server 2022 VM. 10 cores, 120 GB memory. Data is iSCSi back to a Truenas server. I have stopped Storj but still can’t run a chkdsk on that drive as it says something is still using it. I have started the vm in safe mode and checked resource monitor and i don’t see anything accessing that drive, and i swear i was able to run chkdsk late last year when i was having an issue and it worked fine, so maybe theres something there.
Last few days the ingress traffic septupled, and your storage cannot sustain the IO pressure.
You may want to ensure you have all the low hanging fruit configuration in place (no sync writes or databases off the main storage, no atime updates anywhere, etc)
On the iSCISI on windows: NTFS is horrible for high IO load. Adding ram does not help, as it would for literally any other OS on earth. You have truenas – why dont you run the node directly there? What’s the point of adding VM, windows, NTFS, and iscsi into the mix? Doing so you are also preventing the filesystem on TrueNAS to make optimizations – it only sees opaque blob of data with random access, where there is no distinction between metadata and other data fetches – which drives most of the performance optimizations.
Honestly, i have been running my node for a couple of years and the Scale option was newer as was the built in app at the time i started in on this project, so i steered clear and went with what i was more familiar with. As of late i have been strongly considering trying to find a way to do this, but have not figured out how to go about it, shy of GE and start a new node once this one is done with. I just don’t want to start all over given how long it takes to get anywhere.
At this point am i better off trying to get over the transition hurdle (whether thats trying to move everything or just GE and start over), or focus energy on getting the current platform stable?
I don’t think it will be easy to stabilize the current setup if the current load persists.
I would start migrating the data. It will take few weeks but it’s doable. You would robocopy files to a dataset several times (via your windows vm), and then stop the node, do a final robocopy run with a delete option, and start node on a nas.
I believe scale runs k8, I would run storj provided docker container there. (I’m running Core, and simply run storagenode directly in jail. I have tried scale a few times, I don’t like it. I don’t think it’s ready and/or stable enough)
I currently run Core on my production unit (I have scale for my snapshot replication destination). I assume to use my current disks, i need to get the data off them first (set up a new iscsi maybe with a couple spare disks to move the data over to), then set up the new core/scale option with the current disks? I am beyond a newb when it comes to docker/k8s. Is one or the other more user friendly to learning. Or is one more stable given i will be behind the curve when it comes to self supporting.
Do the core and scale options work off a standard dataset that i can transfer data into from outside of truenas, or is it some other form of private data pool?
Part of the other issue now is also trying to move 10 TB of data with the entire node crashing every hour. This may very well become a GE situation.
Are these all settings within the storj config file? As far as the storage pool goes, nothing else access the drives from TN or from Windows. In TN this pool is set to Standard for Sync, LZ4 for compression, and dedup is off. My database is currently on the pool drive. Would moving that over to the c: drive for now take any load off? I feel like i need to get this stablized to a degree before i can get the data shuffled around. This seems like death by a thousand cuts at nearly 1.5 hours a day downtime from the reboots. My online scores are starting to drop.
And perhaps you need to increase a check timeout again. Please note, in case of readable check timeout you need also increase the readable check interval too, because they both 1m0s by default.
You may also have a writable check timeouts too, so just make sure that you modifying the correct one.
For the migration it’s better to use a simple SMB/CIFS or NFS share from your NAS dataset, because you likely cannot simple migrate from iSCSI virtual disk to a dataset.
You may setup a node using TrueChart on Scale, just need to provide a custom paths to your identity and data before the start of the application. I would also disable a port forwarding in case of mistake to do not disqualify your migrated node. When you will make sure that the app is running and use the provided paths, e.g. without duplicated blobs subfolder in a wrong place, you may enable the port forwarding. Please note, the docker version expects all data in the storage subfolder, but you must provide a parent folder to the docker version (it will append storage automatically to the provided path to see its data), this is the main difference between Windows GUI and docker setups.
See Migrating from Windows GUI installation to Docker CLI - Storj Docs
It turned out Page file was enabled for my storage drive and that was keeping chkdsk from running. I removed the page file and was able to run chkdsk. No errors were found, but i also noticed i was not crashing anymore. So far everything has been running for about 11 hours without incident. I want to tweak a little more before trying to migrate away from this set up. The one thing I have to do is move the databases to the c:. I understand the process but am not 100% sure of which files to get. My data path is e:. Do I take all of the files from the root of e:? I have txt, tmp, database file, db-wal file, db-shm file, and file, as types. Do i just take the Data Base File and leave the other db and random types behind?
Apparently i shouldn’t have spoken. An hour after this post the system resumed crashing every hour. I have seen some people mention issues with the temp folder. Is 500,000 files normal? These are mostly from the last few days
You need to move only files with the *.db extension, if you want to move databases. And this must be performed, when the node is stopped (the service storagenode must be stopped), and you need to change the option
to the path where is now your database files are located before the start of the service.
No. It just means that your node is restarted too much during attempts of upload something to your node. If these files are older than 48h you can remove them while the node is stopped.
Thanks for the clarification on which files. A day after my last post everything became stable again so I got scanfile running and so far no more crashes and my other problem of the large data discrepancy is slowly fixing itself. Once that’s done I will move the db files and then start working on trying to migrate off of my current setup.
At this point I am going to attribute my issues to an unusually high load and my setup inadequate to handle the high read/writes needed to keep up.
Interesting thing happened when i moved the .db files. I stopped the storj service. I moved all of the .db files (leaving all of the .db-wal and shm and others behind) to a new location on the c: drive. I updated the location in the config file and uncommented it so it would be active. I then saved it and started the service. The service started and nearly instantly all of those files i left behind were recreated in the new location and disappeared from the old, aside from garbage_collection_filewalker_progress.db-shm and .db-wal. Those 2 files were recreated in the new location, but also have to older versions left in the previous directory. The odd part is my disk usage items. I was around 15.5 TB used and 369 gb in the trash days ago before it seemed like filewalker started working and closing the gap between Disk Space used this month and Disk space used (difference was around 5 TB). This morning before the moving of the DB files, Used was at 14.25 TB and Trash was just over 2 TB. After restarting the service, Used jumped back up to 15.41 TB and trash dropped back to 369 GB. The dashboard seems to be happy, but those numbers seemingly jumping back to much older numbers has me questioning if everything is actually ok.