Hello Team, I totally changed the HDD and still see storj container crashing. I get the following errors when the container fails.
2021-10-25T06:40:42.527Z ERROR services unexpected shutdown of a runner {“name”: “piecestore:monitor”, “error”: “piecestore monitor: error verifying location and/or readability of storage directory: open config/storage/storage-dir-verification: transport endpoint is not connected”, “errorVerbose”: “piecestore monitor: error verifying location and/or readability of storage directory: open config/storage/storage-dir-verification: transport endpoint is not connected\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func1.1:133\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func1:130\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
fatal error: unexpected signal during runtime execution
[signal SIGBUS: bus error code=0x2 addr=0xffff64d95000 pc=0xcf3048]
Have you lost a previous data?
If not, how is it happened that you node missed this file?
If you want to start from scratch, you should remove a previous identity (because if you lost data, the identity will be disqualified as soon as you bring it online), generate a new one, sign it with a new authorization token and start with clean storage, following the guide: Storage Node - Node Operator
@Alexey Alexey, I copied all the data from the old disk to new disk. These errors did not pop up for more than 4-days. They started after that. Only way I can bring back the storjnode is to reload the whole system.
Is it a Raspberry Pi? If so, I would recommend to move the identity folder to the disk with data and flash a fresh OS image to your SD-card, then setup docker and static mount, then try to start storagenode.
@Alexey after digging deeper, I found the issue with out of memory. When the system is out of memory, it is killing mount process. I have 1GB RAM on this device. This started happening with 1.41.2 version. I would like to go back to an older version where the container was not taking much memory. Please let me know where I can find the version history of the containers.
oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=cron.service,mems_allowed=0,global_oom,task_memcg=/system.slice/mnt-storagenode.mo>
Oct 27 23:00:10 pine64 kernel: Out of memory: Killed process 400 (mount.ntfs) total-vm:797052kB, anon-rss:475884kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:1596kB oom_s>
Oct 27 23:00:10 pine64 kernel: oom_reaper: reaped process 400 (mount.ntfs), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Oct 27 22:37:06 pine64 smartd[1355]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 47 to 49
Oct 27 23:00:12 pine64 systemd[1]: mnt-storagenode.mount: A process of this unit has been killed by the OOM killer.
It’s not recommended and can destroy your databases.
And the actual problem is using NTFS on Linux. Do not do that.
Even if you figure out how to revert:
it will be killed anyway because NTFS is way slower than ext4 on Linux, thus have a higher RAM usage.
you likely lose your databases due a wrong scheme.
your ingress likely would be stopped because the software version would be 2 version behind.
I would recommend to convert your NTFS to ext4. Stop the node, backup data, format disk to ext4 and restore data back.
You can also backup your data online: How do I migrate my node to a new device? - Storj Docs, so the downtime would be minimized.
You can do that gradually on the same disk in edge case, but requires a lot more knowledge.
I can describe, how to do, but you may lose data on any point. if you decide to go this dangerous way, I would recommend to do not use GParted for that, use LVM instead. See Moving from Windows to Ubuntu and back
The backup-restore is much more robust method.