Generally there are many knowledgeable users on here. I am sure somebody will be able to tell you exactly where to look on a TrueNAS.
Its the weekend so you will have to wait till others see your post 
Thank you all so much for the help and encouragement!
I’ve fully shut my node down to be sure I’m not online. That gives me 30 days to work on this without stressing.
Hey, @Alexey: able to give me a hand with this? Know TrueNAs well enough to problem solve it with me?
The node’s data folders are located in the dataset, which your TrueNAS used to store the application’s data. It’s not hardcoded, so you need to search it.
You may execute the command in the shell:
df -T --si
It should show mount points and you can locate, where your dataset is mounted. Next you need to navigate to the folder with node’s data.
For example, for the dataset with name data, the output may look like:
Filesystem Type Size Used Avail Use% Mounted on
udev devtmpfs 819M 0 819M 0% /dev
tmpfs tmpfs 201M 8.1M 193M 5% /run
boot-pool/ROOT/22.12.2 zfs 115G 2.9G 112G 3% /
tmpfs tmpfs 1.1G 107k 1.1G 1% /dev/shm
tmpfs tmpfs 105M 0 105M 0% /run/lock
tmpfs tmpfs 1.1G 13k 1.1G 1% /tmp
boot-pool/grub zfs 112G 8.7M 112G 1% /boot/grub
data zfs 127G 132k 127G 1% /mnt/data
data/ix-applications zfs 127G 132k 127G 1% /mnt/data/ix-applications
data/ix-applications/k3s zfs 127G 141M 127G 1% /mnt/data/ix-applications/k3s
data/ix-applications/docker zfs 129G 2.2G 127G 2% /mnt/data/ix-applications/docker
data/ix-applications/catalogs zfs 127G 59M 127G 1% /mnt/data/ix-applications/catalogs
data/ix-applications/releases zfs 127G 132k 127G 1% /mnt/data/ix-applications/releases
data/ix-applications/default_volumes zfs 127G 132k 127G 1% /mnt/data/ix-applications/default_volumes
data/ix-applications/releases/storj zfs 127G 132k 127G 1% /mnt/data/ix-applications/releases/storj
data/ix-applications/releases/storj/volumes zfs 127G 132k 127G 1% /mnt/data/ix-applications/releases/storj/volumes
data/ix-applications/releases/storj/charts zfs 127G 394k 127G 1% /mnt/data/ix-applications/releases/storj/charts
data/ix-applications/releases/storj/volumes/ix_volumes zfs 127G 132k 127G 1% /mnt/data/ix-applications/releases/storj/volumes/ix_volumes
data/ix-applications/releases/storj/volumes/ix_volumes/ix_data zfs 127G 263k 127G 1% /mnt/data/ix-applications/releases/storj/volumes/ix_volumes/ix_data
data/ix-applications/releases/storj/volumes/ix_volumes/ix_identity zfs 127G 263k 127G 1% /mnt/data/ix-applications/releases/storj/volumes/ix_volumes/ix_identity
...
So, your data for that example should be in /mnt/data/ix-applications/releases/storj/volumes/ix_volumes:
[-]$ ls -l /mnt/data/ix-applications/releases/storj/volumes/ix_volumes
total 17
drwxr-xr-x 4 apps apps 7 Sep 30 18:28 ix_data
drwxr-xr-x 2 apps apps 8 May 1 01:48 ix_identity
ix_data should contain all folders, include blobs and trash.
HI, Alexey:
Thank you for working through this with me. Sorry about the delay getting back to this.
So, I have no directories within /mnt/Main/ix-applications/releases/storj/volumes/ix_volumes
I have attached a screenshot, so you know I am not making error in your directions.
Please show the output of
df -T --si | grep storj
You may copy the text from the web UI and paste it here between two new strings with three backticks, like this:
```
text output here
```
Is it possible, that you specified a custom paths manually for the identity and data in the app configuration?
You may check that
sudo docker ps | grep storj
you will get a container ID, then use it to figure out used volumes:
docker inspect replace-me-with-container-id -f '{{.Mounts}}'
or in the web UI
Yes, I did add custom paths for data and identity–the videos I used to help my through setup recommended this. You can see it here:
admin@truenas[/mnt/Main/apps/storj-node/config/storage]$ df -T --si | grep storj
Main/apps/storj-node zfs 29T 263k 29T 1% /mnt/Main/apps/storj-node
Main/apps/storj-node/config zfs 30T 1.1T 29T 4% /mnt/Main/apps/storj-node/config
Main/ix-applications/releases/storj zfs 29T 263k 29T 1% /mnt/Main/ix-applications/releases/storj
Main/apps/storj-node/config-auto-2023-08-27_00-00-clone zfs 30T 633G 29T 3% /mnt/Main/apps/storj-node/config-auto-2023-08-27_00-00-clone
Main/ix-applications/releases/storj/volumes zfs 29T 263k 29T 1% /mnt/Main/ix-applications/releases/storj/volumes
Main/ix-applications/releases/storj/charts zfs 29T 394k 29T 1% /mnt/Main/ix-applications/releases/storj/charts
Main/apps/storj-node/identity zfs 29T 263k 29T 1% /mnt/Main/apps/storj-node/identity
Main/ix-applications/releases/storj/volumes/ix_volumes zfs 29T 263k 29T 1% /mnt/Main/ix-applications/releases/storj/volumes/ix_volumes
So is the next step to follow jammerdan’s advice above and hunt down a piece ID, for example here:
2023-09-30T16:46:03Z ERROR piecestore download failed {“process”: “storagenode”, “Piece ID”: “6QTWSBJFU42FKD4S52KXI5JG4P2M5SOD5YWXH7VBJXH7ICS6GCLA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET_AUDIT”, “Offset”: 884224, “Size”: 0, “Remote Address”: “34.146.139.227:36776”, “error”: “file does not exist”, “errorVerbose”: “file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:75\n\tstorj.io/storj/storagenode/piecestore.
It isn’t in the trash folder. Not sure how to begin searching elsewhere.
Then your node is really lost files.
Have you uninstalled the app in the past?
Did you specify your own paths when installing the application or did you change them later?
Are these paths on zfs pool too?
I do not believe I even uninstalled the app, but I am not certain about whether I changed paths. I am only one year into Linux, and 3 months into TrueNAS, so much was merely blindly following videos/article.
Yes, this app is within a ZFS pool/dataset. Would it have been better to build it without ZFS array?
And I believe you had some criticisms of the Chart image. If I have to start over, should I build without using True Charts image? I know I can also create my own docker containers.
If you already have a zfs pool, you may use it. You may also use True Charts app, it’s up on you. You would just have some complications if you would like to customize something, but in general it’s ok.
I suspect that initially you perhaps used a default path for data, then changed it to a new one and part of data have been leaved in the old location, thus your node has been disqualified.
Yes, that seems quite possible. At the time, the various options were totally unclear to me. Having worked with them now in other apps and such, it all makes much more sense.
Buy anyhow, so it is definitely time to start over? No salvaging this node, right? Just delete and begin again?
If it’s not disqualified on all satellites, you may keep it running, the remained satellites will still pay you for used storage and bandwidth.
But if it’s disqualified on all satellites then you need to start over - stop the app, delete all data, provide a new authorization token and start it again, if data is absent the app should generate an identity, sign it with a new authorization token and prepare all needed folders structure in the data location.
OK, seems best to just start from scratch without these issues, and also so I can take advantage of all satellites. I was only at 1Tb, so easy to start again. Thanks!
I got my new node node up. And it appears, from the dashboard, to be working fine.
What are the main steps I should take to monitor the health of my data, so I’m sure to catch problems before being disqualified this time? Maybe link to an earlier post on this?
Thank you for your patience.
You may use [Tech Preview] Email alerts with Grafana and Prometheus or just periodically check logs for failed audits or changes in audit/suspension scores on your dashboard.