Major Bump to Ingress Bandwidth

Nope, I do not have this information. You may track the stat on this site: Storj Stats, or use the source: GitHub - storj/stats: Storj Public Network Statistics.

1 Like

Hiya countryman. What’s the reasoning for moving to XFS? :slight_smile:

I just got done moving all my nodes from NFTS on Windows to EXT4 on Linux - don’t give me any ideas!

1 Like

Davs Ottetal :wink: (Nice to meet you Ottetal)

I’m using XFS due to my own personal experiences, working with EXT3/4 and XFS for some years now.
And since storjnodes now has changed to larger and fewer files due to hashstore, I wanted to try XFS to see, if that would have any impact on nodes performance.

ChatGPT’s take on it :

How do you compare ext4 vs. xfs in this scenario :

  • On single ubuntu server server
  • CPU : 4 x Intel(R) Xeon(R) CPU E5-4669 v4 @ 2.20GHz
  • RAM : 48 x 64GB LRDIMM DDR4 2400 MHz
  • Controller : 3 x Broadcom SAS 9480-8i8e ctrl.
  • HDD : 132 x 7200 RPM, SAS Seagate spindle HDD’s, that varies between 20TB to 24TB
  • 132 x Storj Nodes (this is software)
  • Each node has about 4000 files varies between 1 GB to 2 GB
  • Only one node per SAS HDD’s

Thanks! With this very specific and performance-critical setup, let’s dive into EXT4 vs XFS for your Storj node-per-HDD layout:


:gear: Your Setup Summary

Component Details
Server Single Ubuntu server
CPU 4 × Xeon E5-4669 v4 (88 cores total)
RAM 3TB (48 × 64GB DDR4)
Disks 132 × 7200 RPM SAS HDDs (20–24TB each)
Controllers 3 × Broadcom SAS 9480-8i8e
Workload 132 Storj nodes — one per disk — each with ~4000 files (1–2 GB)
Per Disk FS 1 filesystem per drive (assumed)

This is a perfectly parallel design — excellent architecture, by the way. With 132 isolated workloads, your filesystems don’t have to compete for I/O — but filesystem efficiency per disk still matters.


:magnifying_glass_tilted_left: EXT4 vs XFS: Per-Disk, Parallel Storj Node Use

Feature EXT4 XFS
Small # of files per FS (~4K) Adequate Very good
Large files (1–2 GB) Good Excellent
Single-threaded write perf Slightly better in some cases Very close, often better for larger sequential writes
Metadata ops (fsck, inode mgmt) Slower on large disks Way faster, especially fsck-free
Filesystem check (fsck) Very slow on 20–24TB XFS doesn’t need fsck unless corruption is suspected
File fragmentation Moderate Lower – extent-based
Write pattern optimization Average Better with large allocsize, delayed alloc
Filesystem tuning flexibility Less flexible More tuneable (e.g., logdev, logbsize, allocsize)
Performance over time (aging fs) Degrades more Better long-term performance

:brain: With 3TB RAM + 132 Disks

  • Each disk can be cached independently thanks to the RAM — XFS benefits more from delayed allocation + page cache.
  • fsck on 24TB EXT4 volumes - can take hours.
  • XFS recovery - Almost instant due to internal journaling.

:chequered_flag: Final Verdict

:white_check_mark: Choose XFS per disk.

  • Pros for your case:
    • Perfect for large file workloads with low file counts.
    • Avoids fsck delays (huge benefit with 132 disks).
    • Better metadata and extent management.
    • More tunable for sequential workloads and long-term storage efficiency.
    • Proven stable and high-performance in similar archival/cloud storage workloads (like Lustre, Gluster, Veeam, etc.)

:small_orange_diamond: EXT4 would only make sense if:

  • You had smaller disks (under ~4–8TB).
  • Files were small (<100 MB).
  • You had critical compatibility needs (e.g., older kernel/filesystem tools).

Th3Van.dk

8 Likes

EXT4 vs XFS vs ZFS: Per-Disk, Parallel Storj Node Use

Feature EXT4 XFS ZFS
Small # of files per FS (~4K) Adequate :white_check_mark: Very good :white_check_mark: Good (ZFS copes well with low file counts)
Large files (1–2 GB) Good :white_check_mark::white_check_mark: Excellent :white_check_mark::white_check_mark: Excellent (sequential write-friendly)
Single-threaded write perf Slightly better at times Very close, fast Good, but sync writes can bottleneck
Metadata ops (fsck, inode mgmt) Slower on large disks :white_check_mark: fsck-free, fast :white_check_mark: No fsck needed, always consistent
Filesystem check (fsck) :x: Hours on 24TB :white_check_mark: Fast recovery :white_check_mark: Always online, checksummed metadata
File fragmentation Moderate :white_check_mark: Low :white_check_mark: CoW avoids fragmentation but may inflate space
Write pattern optimization Average :white_check_mark: Delayed alloc :white_check_mark: Copy-on-write, adjustable via recordsize
Filesystem tuning flexibility :x: Limited :white_check_mark::white_check_mark: Advanced tuning :white_check_mark::white_check_mark: Lots: compression, recordsize, sync flags
Performance over time (aging fs) Degrades :white_check_mark: Stable :white_check_mark::white_check_mark: Scrubbing & COW keeps it healthy
Snapshots No No :white_check_mark::white_check_mark: Native snapshots, almost free
Data Integrity (bit rot protection) :x: None :x: None :white_check_mark::white_check_mark: End-to-end checksumming + scrubbing
Memory usage :white_check_mark: Light :white_check_mark: Light :x: Heavy (1GB+/node if unoptimized)
Per-disk separation :white_check_mark: Easy :white_check_mark: Easy :warning: Not ideal (ZFS prefers multi-disk pools)
Ideal use case Small disks, legacy High-performance per-disk Enterprise-like long-term archival safety
7 Likes

Th3Van, first congrats on chatgpt congratulating you on your setup :slight_smile:

if any single operator can set up an A-B test set up it’s you. or maybe A-B-C if you throw in ZFS.

5 Likes

This table is outdated, as ext4 got a lot of features that this table claims it does not have. For example, ext4 has delayed allocation for at least a decade now.

It seems we are quite a few danish operators here, any quite big ones :wink:
Cheers from Denmark @Ottetal & @Th3Van

2 Likes

AI slop isn’t reliable. Just for instance the fsck line…xfs doesn’t even have it

3 Likes

I second that observation…A.I. is garbage still.

As a side note… last month’s additional egress or surge pay was almost an additional 20% above Storj’s regular node earnings, which was nice. Sensible considering this new client’s experiments and testing. Did anyone else notice this pay bonus?

I am somewhat disappointed that Storj, once again, started immediately distributing coin for last month’s earnings. Given the markets of late, one would think they would care; however, yet again they are nonchalant regarding distribution cost (ie: market at lows). I think that’s a good thing, but also telling in that this segment is less important to Storj now. Which, I’d agree in terms of sustainable gross margin etc., going forward. Rather wish they would address or offer some of their alternate subsidiarys’ status updates on this platform. Although, not mentioned here, in this forum, the case here seems to be the opposite. In essence, I’m implying this segment, Storj, is now the inferior subsidiary. Ergo, no longer prioritized. There is new 30+ million investment in new business which dwarf this segment of the company (on a cash flow basis at the very least). Good for the company, not so good for SNO’s; nevertheless it is ok in my books. LOVE the current price - thank-you Trump, I’ll watch and wait, but in the interim I took a healthy bite at these levels.

24.68 cents, (actually with STORJ at 24.68 cents today - I accumulated some more. I got a little FLUX today too @ 22 cents.. lol - I might end up running some nodes)
Julio

I don’t get the concern about the price of the token. storj and SNO’s are running a business and should be paid as rapidly and efficiently at the end of each month. It’s an accounts payable process not a crypto speculation process.

and the amount of storj is denominated in fiat USD anyway.

the only thing that might bother storj the company would be the times of high gas fees.

8 Likes

SNOs should be paid at 00:00:01 UTC on the first day of each month automatically.

Raise hands who remembers when payments were delayed because the person calculating the payments was on holiday. raises hand

2 Likes

I get that some SNOs want to hodl, and speculate on the price going up. That’s fine: I’m not their mom :wink:

But I want it to be like getting-paid-in-fiat-with-a-couple-extra-steps. Sell every payout as soon as possible. STORJ coins should exist in your wallet for at most 15 minutes :money_mouth_face:

4 Likes

pay bonus? I earn 50$ less than last month… good pay bonus

I’m not aware of any surge payouts :person_shrugging:
Perhaps you mean that the customers have increased egress. I do not see a problem here and completely do not mind :money_mouth_face:

We do not participate in the tokens market activity, it’s also prohibited, so please, do not invent another conspiracy theory, it’s only your theory, unless you have evidences.
Also, the price of the token doesn’t matter, it’s just an utility token to pay for the service and receive a payout as a supplier. Both sides (suppliers and customers) usages are calculated in USD value, not in tokens. But traders will always being traders. :person_shrugging:

6 Likes

Correct me if I am wrong.

Storj is putting all the $$$ into their pockets/development & paying out public with Storj token which is already pre-minted. (No customer USD is being paid to node operators)

Problem is if they don’t buy back their own token and keep paying out with pre-minted tokens, the circulating supply keeps increasing and value of Storj token keep decreasing.

1 Like

No, the problem is that you think the Storj token has any use other than the following:

  1. Storj (the company) calculates the value in Storj (tokens) according to the current market price. Let’s say that it’s $0.25 = 1 Storj (token).
  2. Storj (the company) calculates the amount an SNO needs to be paid. Let’s go with $10.
  3. Storj (the company) needs to pay to that SNO $10 / 0.25 = 40 Storj (tokens).
  4. Storj (the company) sends the payment for 40 Storj (tokens) to the SNO.
  5. The SNO immediately sells those 40 Storj (tokens).

The SNO has now been paid $10 in Storj (tokens) for his services as a supplier to the company.

This is where your confusion is: You think that next month, if Storj (token) crashes down to $0.01, the SNO is going to be paid less. That’s not the case. Look at steps 1-3. In step 4, the SNO will now receive a payment of 1000 Storj (tokens). Naturally, the next step is step 5.

The cycle repeats for years. At some point, Storj (the company) will run out of pre-minted tokens. Storj (the company) will now be the opposing side of the SNO’s sell trade (in simpler words: the SNO will now sell back to Storj (the company)). The price difference will be virtually zero (paid price - sell price), no more than a rounding error.

TL;DR: Storj’s token price is completely irrelevant. I live for the day that it will never be mentioned again on this forum. One can only hope.

7 Likes

My main question was where is the real USD$$$ going after customer pays? Its definitely not coming towards SNO. Coz we are getting paid in Tokens worth of dollars, not actual dollar.

They have their business expenses too, that they cannot pay in Tokens. Like salary and Taxes. Their own servers and I think they pay for the AWS endpoint too?
In the end it’s not ending all in their own pockets.

1 Like

“Also, the price of the token doesn’t matter”

It does matter.

If they don’t care about token price, project will fall. A small percentage of USD should go to buy back the tokens to keep the value attractive so the investors in crypto space be interested even if they are not SNO’s. If there are no buy orders, who will you sell?

There are thousands of crappy tokens without any usefulness on cryptomarket and they are trading for years. Who sells them? Those who make money from the difference in courses, of course.
The cost of a token primarily depends on the information noise. There is no difference whether Storj buys back its token or not

1 Like