thus it is a wrong assumption, Storj here is out of scope. The standard is ext4 and no RAID. Storj doesn’t force you to use any recommendations. This is your decision, only yours.
I see you still must be misunderstanding me, I was referring filesystems performance comparison testing on behalf of storj as an entity about which I got a feeling you were asking me twice very recently.
I keep reading these threads: but it seems to boil down to in-everything-but-the-most-exotic-cases the IO has to make it to the HDD eventually… so there’s only so much a clever filesystem can do. So do Ext4/NTFS, one drive per node. Maybe put the DBs on SSD because that’s safe (as they can disappear and your stats will be wrong but the node will be OK). And if you do set up something exotic (like ZFS metadata SSD with embedded small-file support) you better use something like triple mirroring because if you lose it you lose everything on the HDDs as well.
Are people just trying to speed up filewalker? Regular day-to-day node IO seems reasonable already without anything fancy.
2 Likes
I’d say it boils down to curiosity. The topic is ‘What is the best file system for a Storj node?’ And now it depends on how you look at it. Generally, it’s probably Ext4/NTFS; however, in absolute terms, is it really? I don’t think ZFS is exotic. Actually, maybe ZFS on Solaris, or Gfarm on PowerPC, or maybe some of those new Chinese filesystems. But ZFS, it comes with Ubuntu and is packaged by TrueNAS Scale. Anyway, I am genuinely curious about @arrogantrabbit 's opinion on XFS and Lustre. He usually is very much spot on.
1 Like
Just to chime in. I have 7 nodes running on single disk LVM’s with XFS each on a LSI SAS2008 controller. My biggest node is on a 6TB disk with 5.5TB assigned to the node and 3.8TB filled. Databases are on the same disk. I defragment the databases and log files daily. Filewalker takes only a few minutes.
# grep -i filewalker /mnt/storagenode2/node/node.log
2024-02-01T16:55:37Z INFO lazyfilewalker.gc-filewalker starting subprocess {"process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-02-01T16:55:37Z INFO lazyfilewalker.gc-filewalker subprocess started {"process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-02-01T16:55:37Z INFO lazyfilewalker.gc-filewalker.subprocess Database started {"process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "process": "storagenode"}
2024-02-01T16:55:37Z INFO lazyfilewalker.gc-filewalker.subprocess gc-filewalker started {"process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "process": "storagenode", "createdBefore": "2024-01-22T22:00:10Z", "bloomFilterSize": 508487}
2024-02-01T16:56:32Z INFO lazyfilewalker.gc-filewalker.subprocess gc-filewalker completed {"process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "piecesCount": 850464, "piecesSkippedCount": 0, "process": "storagenode"}
2024-02-01T16:56:32Z INFO lazyfilewalker.gc-filewalker subprocess finished successfully {"process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-02-02T17:59:16Z INFO lazyfilewalker.gc-filewalker starting subprocess {"process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-02-02T17:59:16Z INFO lazyfilewalker.gc-filewalker subprocess started {"process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-02-02T17:59:17Z INFO lazyfilewalker.gc-filewalker.subprocess Database started {"process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "process": "storagenode"}
2024-02-02T17:59:17Z INFO lazyfilewalker.gc-filewalker.subprocess gc-filewalker started {"process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "process": "storagenode", "createdBefore": "2024-01-29T17:59:59Z", "bloomFilterSize": 342271}
2024-02-02T18:00:50Z INFO lazyfilewalker.gc-filewalker.subprocess gc-filewalker completed {"process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "process": "storagenode", "piecesCount": 568538, "piecesSkippedCount": 0}
2024-02-02T18:00:50Z INFO lazyfilewalker.gc-filewalker subprocess finished successfully {"process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-02-04T03:16:52Z INFO lazyfilewalker.gc-filewalker starting subprocess {"process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-02-04T03:16:52Z INFO lazyfilewalker.gc-filewalker subprocess started {"process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-02-04T03:16:53Z INFO lazyfilewalker.gc-filewalker.subprocess Database started {"process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "process": "storagenode"}
2024-02-04T03:16:53Z INFO lazyfilewalker.gc-filewalker.subprocess gc-filewalker started {"process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "process": "storagenode", "createdBefore": "2024-01-30T17:59:59Z", "bloomFilterSize": 973771}
2024-02-04T03:27:07Z INFO lazyfilewalker.gc-filewalker.subprocess gc-filewalker completed {"process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "process": "storagenode", "piecesCount": 1687692, "piecesSkippedCount": 0}
2024-02-04T03:27:07Z INFO lazyfilewalker.gc-filewalker subprocess finished successfully {"process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-02-09T22:06:03Z INFO lazyfilewalker.gc-filewalker starting subprocess {"process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-02-09T22:06:03Z INFO lazyfilewalker.gc-filewalker subprocess started {"process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-02-09T22:06:04Z INFO lazyfilewalker.gc-filewalker.subprocess Database started {"process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "process": "storagenode"}
2024-02-09T22:06:04Z INFO lazyfilewalker.gc-filewalker.subprocess gc-filewalker started {"process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "process": "storagenode", "createdBefore": "2024-02-05T17:59:59Z", "bloomFilterSize": 340936}
2024-02-09T22:07:47Z INFO lazyfilewalker.gc-filewalker.subprocess gc-filewalker completed {"process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "process": "storagenode", "piecesCount": 569917, "piecesSkippedCount": 0}
2024-02-09T22:07:47Z INFO lazyfilewalker.gc-filewalker subprocess finished successfully {"process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
What’s used space for each of those satellites?
see my example:
My server runs fine @Alexey . I just wanted to say that XFS runs fine here.
# du -hd 1 /mnt/storagenode2/node/storage/blobs/
1.7T /mnt/storagenode2/node/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa
290G /mnt/storagenode2/node/storage/blobs/qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa
1.3T /mnt/storagenode2/node/storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa
3.2T /mnt/storagenode2/node/storage/blobs/
2 Likes