The same mysterious thing kept happening to me for the last few days. My node is running on unraid and previously I have removed one disc from the array. The data from the removed disc was properly migrated and there was no data inconsistency after migration. I found the following error records in my container log
2023-11-23T02:18:26-08:00 WARN console:service unable to get Satellite URL {“process”: “storagenode”, “Satellite ID”: “12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo”, “error”: “console: trust: satellite is untrusted”, “errorVerbose”: “console: trust: satellite is untrusted\n\tstorj.io/storj/storagenode/trust.init:29\n\truntime.doInit1:6740\n\truntime.doInit:6707\n\truntime.main:249”}
You need to stop and remove the container, unmount the disk, check and fix errors, then mount it back and try to run the container using all your parameters (you must not run it with SETUP step, otherwise you may destroy it, use the normal full docker run command instead), then check your logs.
If it’s still produce the same error - check permissions. If you use --user $(id -u):$(id -g) in your docker run command, make sure that the user $(id -u):$(id -g) is the owner of data location and has full permissions.
Hi Alexey. Please forgive me, but I’m new to Linux and not very proficient with CLI and Docker. Are there somewhere more detailed instructions for these steps?
The provided error is a usual context canceled error - your node was too slow and loosed the race for pieces.
It cannot be close to the everyone customer in the world, so this is normal.
I suspect I have skipped that step since I was young and foolish then and considered it wasn’t necessary if you run the node on docker. How can I check for the status of the mount, and if my suspicion is right how should I correct it?
Try to follow this guide, it has all necessary steps.
Moreover now I suspect that you tried to mount a device to the docker container. This will not work. You must mount the device to the mount point in your OS, format the drive to ext4 and then (only then) run the docker container.
To do not mount manually every time when you reboot the system, you need to make this mount automatic and permanent, as described in the guide above.
Ok, my update so far. I used all the little free time I had in the last few days and could not make the node forget the unused satellites and clean the data. After the recent update, the node became more stable restarting every few hours instead of every 20 minutes. As suggested, I decided to deal with file corruption first and then move the node to a dedicated mounted disk. ZFS tool indicated this (I’m using unRaid with an array of individual ZFS discs):
I tried to repair or remove these 3 folders without any luck. I believe all that is left for me to do is to transfer the node to a new unassigned device (disk). I’m waiting for a new drive to arrive in a couple of days. One problem here - it seems like unRaid doesn’t offer a formatting option for xt4. And another question, what is the best way to move the node to a new disc?
Do you know how much data in these folders?
Try to copy these folders to some other place, then copy back. After that try to run the scrub again.
If you want to migrate to a new disk, you may use this instruction:
You did not say that you use Unraid. It supports formatting to xfs by default and mounting of ext4 disk, so for Unraid you seems may format either to xfs or to zfs. zfs is a good choice, but without redundancy (parity or mirroring) it cannot resist corruption and is slower than xfs and ext4. If you don’t use zfs features, you might be better off using xfs.
you need to run the command
please note the --force flag - it will remove data. This command will exit, when it finish its work.
perhaps the docker package is not installed?
Are you use a WSL2 under Windows? If so, you need to unpause the Docker desktop app, it going to a sleep mode from time to time… very annoying
do you mean that you opened a terminal to the container? If so, your command should not include docker exec -it, you need to run the command directly, i.e.:
No, because you don’t seem to be aware in which environment you’re executing the code.
When you attach to the docker, your current directory is the /app directory. In that directory, you do have a ‘config’ and ‘identity’ folder on which your /mnt/user/storj_01/(…)-folders have been mounted. These folders are reachable by the names they have in your docker environment, but not by using the /mnt/user/storj_01…
Essentially this means, you shouldn’t adapt the code. I used this:
