Yep, this is another time zone. Node was again restart 40 min ago. I calculated this and got this time - docker logs --tail 30000 storagenode | grep “ERROR”
Did you check resource graphs and docker (not container) logs? Something could also be in system logs (e.g. a/car/log/messages when process is forcefully killed).
What is your storage configuration, filesystem, and how much ram does the device have?
Side note — when posting logs, don’t post screenshots, they are impossible to read. Instead, paste the text itself between tripple backticks, like so:
Yes you right. I got file system ext4 errors. I made fsck there was about 30 errors, (fixed)sure after that docker rm storagenode and now after start node again got 100% busy i/o wait
This is normal on start. If you had enough ram, most metadata would have been cached in a first few minutes and iops load on drive would reduce.
This is not nearly enough for anything. You want to have enough free ram for caching to fit at least metadata in its entirety. If you run storj in separate VM then that 1GB is all it has, no caching is possible, and with a single drive you will immediately hit iops limit, storage node will be buffering, run out of ram and die. this won’t work.
If you run it in a container with shared ram — this could allow more efficient memory usage across containers. Give the vm it at least 8, ideally 16-24GB or ram.
If you run it in a container with shared ram — this could allow more efficient memory usage across containers. Give the vm it at least 8, ideally 16-24GB or ram.
yes I may give more ram but 2 questions:
How much ram need for 14.5 TB node?
How everyone runs storj nodes with raspberry on 1-2 gb ram?
I’m not an expert on ext4, but you would want to calculate amount of metadata that is needed for the number of files that will be stored, and provide at least that. Plus what storagenode process requires itself (under a gigabyte IIRC)
I don’t know anyone who does it beyond proof of concept and/or very small node and/or using some kind of SSD caching.
There is no magic, it’s very simple. Your hdd can serve 200 IOPS tops. This is theoretical maximum.
Now, every upload and download involves reading and writing metadata. If metadata is not cached and is located on the same HDD — this uses up valuable IOPS. Let’s say, generously, using up just half. This means your node can service 100 uploads and downloads per second combined, until hdd can no longer keep up. In this case node tries to buffer writes, waiting for IO, and eating up ram, further taking away from disk cache, making things worse and ultimately is killed by the OS (you still need to confirm that this is the culprit in your case).
There are ways to reduce the extra IO to minimum (remove sync, access time updates, etc; search forum, it has been discussed many time for variety of OSes and file systems) but ultimately, the more files your node stores, the larger amount of reads it must be able to serve, and less resources are left for writes.
Ideally, you want HDDs to only serve large sequential IO and small random IO shall be served from RAM or SSD.
You may want to try to configure your VM with 8GB of ram and run FreeBSD with ZFS pool comprised of your HDD and an SSD as a special device to exclusively service small files and metadata. This will allow you to scale much further on a single HDD than otherwise would have been possible with other filesystems.
For ext4 and no RAID and no disk/filesystem corruption and no virtualization, 1GB of RAM should be enough, however, if you can add more - please do it.
In the normal circumstances it uses a small amount of RAM, however, OS itself requires RAM to work and RAM for caching. So either do not use VM and run a docker container directly on the host, or add more RAM to the VM.
From my observations, ext4 metadata cache for typical Storj node files take around 1 GB of RAM per 1 TB of stored data. A theoretical estimate on this based on the size of metadata (around 300 bytes per file: inode + direntry, around 1 M to 3 M files per 1 TB, plus some overhead for data not laid out in contiguous way).
You can make it more efficient if you reduce the size of an inode at file system creation time—though this parameter cannot be changed if you already have data on your file system. If you don’t have enough RAM, you should look for some other means of caching metadata, for example an SSD-based cache, otherwise you risk running out of IOPS as your node grows.
