Disqualification & Suspension

Generally there are many knowledgeable users on here. I am sure somebody will be able to tell you exactly where to look on a TrueNAS.

Its the weekend so you will have to wait till others see your post :slight_smile:

Thank you all so much for the help and encouragement!

I’ve fully shut my node down to be sure I’m not online. That gives me 30 days to work on this without stressing.

Hey, @Alexey: able to give me a hand with this? Know TrueNAs well enough to problem solve it with me?

The node’s data folders are located in the dataset, which your TrueNAS used to store the application’s data. It’s not hardcoded, so you need to search it.
You may execute the command in the shell:

df -T --si

It should show mount points and you can locate, where your dataset is mounted. Next you need to navigate to the folder with node’s data.
For example, for the dataset with name data, the output may look like:

Filesystem                                                          Type      Size  Used Avail Use% Mounted on
udev                                                                devtmpfs  819M     0  819M   0% /dev
tmpfs                                                               tmpfs     201M  8.1M  193M   5% /run
boot-pool/ROOT/22.12.2                                              zfs       115G  2.9G  112G   3% /
tmpfs                                                               tmpfs     1.1G  107k  1.1G   1% /dev/shm
tmpfs                                                               tmpfs     105M     0  105M   0% /run/lock
tmpfs                                                               tmpfs     1.1G   13k  1.1G   1% /tmp
boot-pool/grub                                                      zfs       112G  8.7M  112G   1% /boot/grub
data                                                                zfs       127G  132k  127G   1% /mnt/data
data/ix-applications                                                zfs       127G  132k  127G   1% /mnt/data/ix-applications
data/ix-applications/k3s                                            zfs       127G  141M  127G   1% /mnt/data/ix-applications/k3s
data/ix-applications/docker                                         zfs       129G  2.2G  127G   2% /mnt/data/ix-applications/docker
data/ix-applications/catalogs                                       zfs       127G   59M  127G   1% /mnt/data/ix-applications/catalogs
data/ix-applications/releases                                       zfs       127G  132k  127G   1% /mnt/data/ix-applications/releases
data/ix-applications/default_volumes                                zfs       127G  132k  127G   1% /mnt/data/ix-applications/default_volumes
data/ix-applications/releases/storj                                 zfs       127G  132k  127G   1% /mnt/data/ix-applications/releases/storj
data/ix-applications/releases/storj/volumes                         zfs       127G  132k  127G   1% /mnt/data/ix-applications/releases/storj/volumes
data/ix-applications/releases/storj/charts                          zfs       127G  394k  127G   1% /mnt/data/ix-applications/releases/storj/charts
data/ix-applications/releases/storj/volumes/ix_volumes              zfs       127G  132k  127G   1% /mnt/data/ix-applications/releases/storj/volumes/ix_volumes
data/ix-applications/releases/storj/volumes/ix_volumes/ix_data      zfs       127G  263k  127G   1% /mnt/data/ix-applications/releases/storj/volumes/ix_volumes/ix_data
data/ix-applications/releases/storj/volumes/ix_volumes/ix_identity  zfs       127G  263k  127G   1% /mnt/data/ix-applications/releases/storj/volumes/ix_volumes/ix_identity

So, your data for that example should be in /mnt/data/ix-applications/releases/storj/volumes/ix_volumes:

[-]$ ls -l /mnt/data/ix-applications/releases/storj/volumes/ix_volumes          
total 17
drwxr-xr-x 4 apps apps 7 Sep 30 18:28 ix_data
drwxr-xr-x 2 apps apps 8 May  1 01:48 ix_identity

ix_data should contain all folders, include blobs and trash.

HI, Alexey:
Thank you for working through this with me. Sorry about the delay getting back to this.

So, I have no directories within /mnt/Main/ix-applications/releases/storj/volumes/ix_volumes

I have attached a screenshot, so you know I am not making error in your directions.

Please show the output of

df -T --si | grep storj

You may copy the text from the web UI and paste it here between two new strings with three backticks, like this:

text output here

Is it possible, that you specified a custom paths manually for the identity and data in the app configuration?

You may check that

sudo docker ps | grep storj

you will get a container ID, then use it to figure out used volumes:

docker inspect replace-me-with-container-id -f '{{.Mounts}}'

or in the web UI

Yes, I did add custom paths for data and identity–the videos I used to help my through setup recommended this. You can see it here:

admin@truenas[/mnt/Main/apps/storj-node/config/storage]$ df -T --si | grep storj
Main/apps/storj-node                                                           zfs        29T  263k   29T   1% /mnt/Main/apps/storj-node
Main/apps/storj-node/config                                                    zfs        30T  1.1T   29T   4% /mnt/Main/apps/storj-node/config
Main/ix-applications/releases/storj                                            zfs        29T  263k   29T   1% /mnt/Main/ix-applications/releases/storj
Main/apps/storj-node/config-auto-2023-08-27_00-00-clone                        zfs        30T  633G   29T   3% /mnt/Main/apps/storj-node/config-auto-2023-08-27_00-00-clone
Main/ix-applications/releases/storj/volumes                                    zfs        29T  263k   29T   1% /mnt/Main/ix-applications/releases/storj/volumes
Main/ix-applications/releases/storj/charts                                     zfs        29T  394k   29T   1% /mnt/Main/ix-applications/releases/storj/charts
Main/apps/storj-node/identity                                                  zfs        29T  263k   29T   1% /mnt/Main/apps/storj-node/identity
Main/ix-applications/releases/storj/volumes/ix_volumes                         zfs        29T  263k   29T   1% /mnt/Main/ix-applications/releases/storj/volumes/ix_volumes

So is the next step to follow jammerdan’s advice above and hunt down a piece ID, for example here:

2023-09-30T16:46:03Z ERROR piecestore download failed {“process”: “storagenode”, “Piece ID”: “6QTWSBJFU42FKD4S52KXI5JG4P2M5SOD5YWXH7VBJXH7ICS6GCLA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET_AUDIT”, “Offset”: 884224, “Size”: 0, “Remote Address”: “”, “error”: “file does not exist”, “errorVerbose”: “file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:75\n\tstorj.io/storj/storagenode/piecestore.

It isn’t in the trash folder. Not sure how to begin searching elsewhere.

Then your node is really lost files.
Have you uninstalled the app in the past?
Did you specify your own paths when installing the application or did you change them later?
Are these paths on zfs pool too?

I do not believe I even uninstalled the app, but I am not certain about whether I changed paths. I am only one year into Linux, and 3 months into TrueNAS, so much was merely blindly following videos/article.

Yes, this app is within a ZFS pool/dataset. Would it have been better to build it without ZFS array?

And I believe you had some criticisms of the Chart image. If I have to start over, should I build without using True Charts image? I know I can also create my own docker containers.

If you already have a zfs pool, you may use it. You may also use True Charts app, it’s up on you. You would just have some complications if you would like to customize something, but in general it’s ok.
I suspect that initially you perhaps used a default path for data, then changed it to a new one and part of data have been leaved in the old location, thus your node has been disqualified.

Yes, that seems quite possible. At the time, the various options were totally unclear to me. Having worked with them now in other apps and such, it all makes much more sense.

Buy anyhow, so it is definitely time to start over? No salvaging this node, right? Just delete and begin again?

If it’s not disqualified on all satellites, you may keep it running, the remained satellites will still pay you for used storage and bandwidth.
But if it’s disqualified on all satellites then you need to start over - stop the app, delete all data, provide a new authorization token and start it again, if data is absent the app should generate an identity, sign it with a new authorization token and prepare all needed folders structure in the data location.

OK, seems best to just start from scratch without these issues, and also so I can take advantage of all satellites. I was only at 1Tb, so easy to start again. Thanks!

I got my new node node up. And it appears, from the dashboard, to be working fine.

What are the main steps I should take to monitor the health of my data, so I’m sure to catch problems before being disqualified this time? Maybe link to an earlier post on this?

Thank you for your patience.

You may use [Tech Preview] Email alerts with Grafana and Prometheus or just periodically check logs for failed audits or changes in audit/suspension scores on your dashboard.