Pieces error: filestore error: chmod on FreeNAS (FreeBSD)

Storgeez · April 3, 2020, 11:24pm

I didn’t manage to install tmux nor screen, will have to give it some more time another time. Used CTRL + C to shut it down.
I launched SN with space full to verify it’s working before allowing uploads, it seemed to be working correctly. I then increased the allocated space, restarted it and errors ensued.

2020-04-04T00:01:15.668+0200 ERROR piecestore upload failed {“Piece ID”: “NITMZSLA65RODFL6US6VNG4M7ZQIQ5MVVJCGDAV7FDTKO2AWNBQQ”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”, “error”: “pieces error: filestore error: chmod /Storj/Storj-v3-data/storage/temp/blob-477299112.partial: operation not permitted”, “errorVerbose”: “pieces error: filestore error: chmod /Storj/Storj-v3-data/storage/temp/blob-477299112.partial: operation not permitted\n\tstorj.io/storj/storage/filestore.(*blobWriter).Commit:105\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit.func1:130\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit:198\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:419\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:988\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:199\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:173\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:124\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:161\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}

I ran it as root. What am I doing wrong?

Alexey · April 4, 2020, 1:46pm

Please, show the result of the command

df -h

Storgeez · April 4, 2020, 1:55pm

For the data path I presume:

Filesystem Size Used Avail Capacity Mounted on
/mnt/dataset/Storj 6.4T 3.3T 3.1T 52% [restricted]

Alexey · April 4, 2020, 2:17pm

But your node uses a different path

Symlinks will not work

Storgeez · April 4, 2020, 2:19pm

I’m sorry, that was the mountpoint df, the /Storj/Storj-v3-data/storage goes after the /mnt/dataset/Storj. It is the same path, I just ran df outside the nested directory. I.e. full path is /mnt/dataset/Storj/Storj-v3-data/.

Alexey · April 4, 2020, 2:27pm

You should specify a full absolute path in the storagenode config.yaml

Storgeez · April 4, 2020, 2:30pm

I didn’t specify the path in the config.yaml, I used the command parameter. I cannot put “/mnt/dataset/Storj” part as that doesn’t exist because the nested dataset is mounted under /Storj so /Storj/Storj-v3-data path contains the node’s data.

Alexey · April 4, 2020, 2:32pm

ls -l /Storj
ls -l /Storj/Storj-v3-data

Storgeez · April 4, 2020, 2:35pm

ls -l /Storj
total 157772
drwxrwxr-x+ 2 nobody wheel 3 Apr 3 19:37 Backup
drwxrwxr-x+ 2 nobody wheel 8 Apr 3 19:17 Identity
drwxrwxr-x+ 3 nobody wheel 7 Apr 4 13:31 Storj-v3-data

ls -l /Storj/Storj-v3-data
total 203
-rwxrwxr-x+ 1 root wheel 4842 Apr 4 01:22 config.yaml
-rwxrwxr-x+ 1 nobody wheel 524288 Oct 3 2019 kademlia
-rwxrwxr-x+ 1 nobody wheel 32768 Apr 4 01:22 revocations.db
drwxrwxr-x+ 6 nobody wheel 43 Apr 4 01:22 storage
-rwxrwxr-x+ 1 root wheel 1202 Apr 4 13:31 trust-cache.json

Alexey · April 4, 2020, 2:38pm

Try to chown the data recursively

su
chown root:wheel -R /Storj/Storj-v3-data
exit

Storgeez · April 4, 2020, 2:39pm

chown root:wheel -R /Storj/Storj-v3-data
chown: -R: No such file or directory

Alexey · April 4, 2020, 2:43pm

chown -R root:wheel /Storj/Storj-v3-data

Storgeez · April 4, 2020, 2:44pm

Ok that seems to be doing something, am I supposed to stop the node first?

Alexey · April 4, 2020, 2:45pm

yes, of course, you should stop it before processing

Storgeez · April 4, 2020, 2:46pm

Ok, it’s running now.
I thought root had access to everything regardless of permissions…

Hmm, this will probably mess up all the permissions. Should I change ACL in FreeNAS perhaps? This will lead to some weird inconsistencies probably.

Storgeez · April 4, 2020, 4:44pm

Command completed, I re-enabled uploads, started the node again, same error.
I don’t understand why it would be able to access the databases and old files but not new files.

I just updated to 1.1.1, same thing.

Alexey · April 5, 2020, 11:00am

What is a filesystem on that drive?

gpart show

I’m not sure, is the double mount points work normally.
Please, show the result of the command:

ls -l /Storj/Storj-v3-data/storage/
ls -l /Storj/Storj-v3-data/storage/temp/

Storgeez · April 5, 2020, 4:38pm

File system is ZFS.

gpart show
=> 40 33554352 da0 GPT (16G)
40 1024 1 freebsd-boot (512K)
1064 33553320 2 freebsd-zfs (16G)
33554384 8 - free - (4.0K)

=> 6 2929721333 da1 GPT (11T)
6 122 - free - (488K)
128 4194304 1 freebsd-swap (16G)
4194432 2925526907 2 freebsd-zfs (11T)

=> 6 2929721333 da2 GPT (11T)
6 122 - free - (488K)
128 4194304 1 freebsd-swap (16G)
4194432 2925526907 2 freebsd-zfs (11T)

ls -l /Storj/Storj-v3-data/storage/
total 1415977
-rwxrwxr-x+ 1 root  wheel   12976128 Apr  5 18:37 bandwidth.db
-rwxrwxr-x+ 1 root  wheel      32768 Apr  5 18:37 bandwidth.db-shm
-rwxrwxr-x+ 1 root  wheel    4272472 Apr  5 18:38 bandwidth.db-wal
drwxrwxr-x+ 7 root  wheel          7 Feb 11 23:32 blobs
drwxrwxr-x+ 2 root  wheel          2 Apr  5 09:42 garbage
-rwxrwxr-x+ 1 root  wheel      32768 Apr  4 20:07 heldamount.db
-rwxrwxr-x+ 1 root  wheel      32768 Apr  4 20:25 heldamount.db-shm
-rwxrwxr-x+ 1 root  wheel      32992 Apr  4 20:09 heldamount.db-wal
-rwxrwxr-x+ 1 root  wheel      16384 Apr  4 20:07 info.db
-rwxrwxr-x+ 1 root  wheel      32768 Apr  4 20:09 info.db-shm
-rwxrwxr-x+ 1 root  wheel      32992 Apr  4 20:09 info.db-wal
-rwxrwxr-x+ 1 root  wheel      24576 Apr  4 20:07 notifications.db
-rwxrwxr-x+ 1 root  wheel      32768 Apr  4 20:25 notifications.db-shm
-rwxrwxr-x+ 1 root  wheel      32992 Apr  4 20:09 notifications.db-wal
-rwxrwxr-x+ 1 root  wheel  284594176 Apr  5 18:30 orders.db
-rwxrwxr-x+ 1 root  wheel      32768 Apr  5 18:37 orders.db-shm
-rwxrwxr-x+ 1 root  wheel    4775112 Apr  5 18:38 orders.db-wal
-rwxrwxr-x+ 1 root  wheel      77824 Apr  4 20:07 piece_expiration.db
-rwxrwxr-x+ 1 root  wheel      32768 Apr  5 18:10 piece_expiration.db-shm
-rwxrwxr-x+ 1 root  wheel     222512 Apr  5 18:09 piece_expiration.db-wal
-rwxrwxr-x+ 1 root  wheel      24576 Apr  4 20:07 piece_spaced_used.db
-rwxrwxr-x+ 1 root  wheel      32768 Apr  5 10:19 piece_spaced_used.db-shm
-rwxrwxr-x+ 1 root  wheel      78312 Apr  5 10:19 piece_spaced_used.db-wal
-rwxrwxr-x+ 1 root  wheel  298348544 Apr  4 20:05 pieceinfo.db
-rwxrwxr-x+ 1 root  wheel      32768 Apr  4 20:09 pieceinfo.db-shm
-rwxrwxr-x+ 1 root  wheel      82432 Apr  4 20:09 pieceinfo.db-wal
-rwxrwxr-x+ 1 root  wheel      40960 Dec 21 20:04 piecestore.db
-rwxrwxr-x+ 1 root  wheel      20480 Apr  4 20:07 reputation.db
-rwxrwxr-x+ 1 root  wheel      32768 Apr  5 16:09 reputation.db-shm
-rwxrwxr-x+ 1 root  wheel     280192 Apr  5 16:09 reputation.db-wal
-rwxrwxr-x+ 1 root  wheel      32768 Apr  4 20:07 satellites.db
-rwxrwxr-x+ 1 root  wheel      32768 Apr  4 20:09 satellites.db-shm
-rwxrwxr-x+ 1 root  wheel      32992 Apr  4 20:09 satellites.db-wal
-rwxrwxr-x+ 1 root  wheel     188416 Apr  4 20:07 storage_usage.db
-rwxrwxr-x+ 1 root  wheel      32768 Apr  5 08:10 storage_usage.db-shm
-rwxrwxr-x+ 1 root  wheel     127752 Apr  5 08:10 storage_usage.db-wal
drwxrwxr-x+ 2 root  wheel         59 Apr  5 18:36 temp
drwxrwxr-x+ 7 root  wheel          7 Feb 17 10:47 trash
-rwxrwxr-x+ 1 root  wheel  836177920 Apr  5 18:32 used_serial.db
-rwxrwxr-x+ 1 root  wheel      32768 Apr  5 18:37 used_serial.db-shm
-rwxrwxr-x+ 1 root  wheel    4198312 Apr  5 18:38 used_serial.db-wal


ls -l /Storj/Storj-v3-data/storage/temp/
total 2149
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:59 blob-000485805.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:55 blob-017361873.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:39 blob-017537556.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:51 blob-018429450.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:54 blob-042589485.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 17 01:58 blob-046855275.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:51 blob-086189483.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:59 blob-086190598.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:39 blob-102558000.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:41 blob-105394579.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 17 01:58 blob-117721929.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:44 blob-130247545.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:52 blob-136379054.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:36 blob-197737267.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:43 blob-201162509.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:37 blob-211132967.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:42 blob-285268644.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:35 blob-287808978.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:54 blob-289089566.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:39 blob-334264456.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:58 blob-340473992.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:35 blob-342025499.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:41 blob-367852793.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:59 blob-385635506.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:43 blob-440469886.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 21:03 blob-443892924.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:36 blob-471086374.partial
-rwxrwxr-x+ 1 root  wheel  2317568 Mar 20 21:32 blob-513123078.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:51 blob-515922099.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:39 blob-527677284.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:54 blob-563948521.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:53 blob-568083212.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 21:02 blob-590152417.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:51 blob-599315983.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:39 blob-606637217.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:54 blob-606789759.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:51 blob-637737441.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:38 blob-673755872.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:43 blob-706148624.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:44 blob-708122033.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:41 blob-725010464.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 21:02 blob-742164337.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 21:00 blob-789120829.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 21:00 blob-818698489.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:58 blob-836690761.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:41 blob-839144587.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:56 blob-900544023.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:42 blob-902549127.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:36 blob-908738596.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:44 blob-926831902.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:56 blob-948052032.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:50 blob-958010757.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:43 blob-960955575.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 17 01:58 blob-979294462.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 17 01:58 blob-979799654.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:55 blob-981116332.partial
-rwxrwxr-x+ 1 root  wheel        0 Dec 16 20:35 blob-987675253.partial

Alexey · April 6, 2020, 8:27am

Seems you have used ACLs https://www.freebsd.org/doc/handbook/fs-acl.html
Please, set them to the full access or remove completely with

setfacl -kR /Storj/Storj-v3-data

For ref: https://forums.freebsd.org/threads/operation-not-permitted-when-chmod-with-root.73133/

Storgeez · April 6, 2020, 12:05pm

This is why I asked this.

I didn’t know it would restrict only SOME functionality. Weird stuff this non-Windows stuff is.
What I did was, instead of changing all the permissions again, to go to the Storj dataset options and set the ACL mode from RESTRICTED to PASSTHROUGH. This, if I understand correctly, will no longer block partially incompatible permission changes, i.e. it will allow any file mode changes. Node seems to be fully working now.

Thanks @Alexey for figuring that out!