use some healing filesystem (btrfs/zfs is free for example).
I think that, audit failed should be trigger as something need to be investigate instead of waiting global repair - precisely because if something wrong happen, it will happen again.
My primary concern is the repair cost. Backblaze use RS of 17/20 and using ext4, it is fine for them, cause they in the same datacenter, cost is very cheap.
Trigger a global repair should be treat as the last resource, not the default..
zfs is not exactly free. It require much more RAM comparing to ext4 / xfs. I used zfs when there was no choice (piecestore worked good enough only on zfs + ssd for metadata, on ext4 it deleted trash slower than new trash arrived) and I’m happy that I finally can avoid it after migration to hashstore.
I might suspect bit flip or something like that if there were hash mismatch / read error and so on. But in all errors I’ve seen satellite just asking for piece I have no at all.
If you do not have any means to check correctness on your side, then the problem is only detected client-side (through the Reed Solomon encoding, client notices that your piece is bad and ignores it in favor of other node’s). It is not reported back to the node.
In the past I had a case of this, the problem was silent until audits started to fail. The node recovered after I added memtest.
OK, just spun up a node.
You’ve got another 1TB on your tutorial network. I reckon it’s unlikely my nodes will fill up a lot more considering the current circumstances so go nuts
I went ahead with an experiment - I blocked 37.35.96.30 on my firewall and wanted to see how the network will perform without that one. For uploads, I finally got various hosts instead of single, that’s great
For download of files uploaded before, all downloads failed with Error: download failed receiving data: eestream error: failed to download stripe 0: that is indicating major parts of my files were no doubt stored on a single device/SNO and if the device fails users will not be able to receive their files.
It was just for fun and curiosity…
Is it? Ok, no problem. Was just curious how it works… On the other hand, glad to see maybe over 100 different nodes already joined this testing place. Some as exotic as adsl.viettel.vn, virtua.com.br, spectrum.com, suttk.ru, spectrumnet.bg
I am waiting for the stats to be published like https://stats.storjshare.io/ but I think this is last on the list. But grafana is hot and running to get all that stats
ERROR orders failed to settle orders for satellite {“process”: “storagenode”, “satellite_id”: “12kK7Y15fU1b5YADDQiBj2Dd4WsF1LARVWHpW6nXRcusV3wiRZD”, “satellite_id”: “12kK7Y15fU1b5YADDQiBj2Dd4WsF1LARVWHpW6nXRcusV3wiRZD”, “error”: “order: failed to start settlement: rpc: tcp connector failed: rpc: tls peer certificate verification: not signed by any CA in the whitelist: CA cert”, “errorVerbose”: “order: failed to start settlement: rpc: tcp connector failed: rpc: tls peer certificate verification: not signed by any CA in the whitelist: CA cert\n\tstorj.io/storj/storagenode/orders.(*Service).settleWindow:264\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders.func2:219\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93”}
storagenode-dock-TEST-01 | 2026-08-03T22:24:46Z ERROR nodestats:cache Get disk space usage query failed {"process": "storagenode", "error": "nodestats: EOF", "errorVerbose": "nodestats: EOF\n\tstorj.io/storj/storagenode/nodestats.(*Service).GetDailyStorageUsage:73\n\tstorj.io/storj/storagenode/nodestats.(*Cache).CacheSpaceUsage.func1:178\n\tstorj.io/storj/storagenode/nodestats.(*Cache).satelliteLoop:268\n\tstorj.io/storj/storagenode/nodestats.(*Cache).CacheSpaceUsage:177\n\tstorj.io/storj/storagenode/nodestats.(*Cache).Run.func2:120\n\tstorj.io/common/sync2.(*Cycle).Run:102\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93"}
storagenode-dock-TEST-01 | 2026-08-03T22:24:46Z ERROR nodestats:cache payouts err {"process": "storagenode", "satellite": "12kK7Y15fU1b5YADDQiBj2Dd4WsF1LARVWHpW6nXRcusV3wiRZD"}
EDIT: Everything is working after waiting over the night. 9 Additional nodes have been brought online
@kocoten1992 do you have some vague timeline for this project? How long do you intend to run the test sat, what’s next after it and when should we expect a production sat?
Do you have some promises as customers?
Just curious… Thanks!
Also, as I understand, you use this to put together a how to guide about running a sat.
It would be helpful for everyone, operators, clients and future sat admins a wiki with all these parameters we discuss daily, but very few know the true meaning, like RS numbers, audits, etc etc, to help everyone speak the same language.
Top priority right now is finish all ranged-loop services, finish the tutorial (and maybe keep up with it).
After that, a mini site for anonymous registration (example.net email account), figure out how to remove anonymous account and all of it component to clean up database and storage, then I’ll invite real people to test (right now they afraid of giving me email - I’d too actually).
Then an accounting system (not using the default of storj), using tally and rollup into real number, then figure out how to be paid and how to paid storagenode.
Then take feedback what need to be done, after that open a real satellite, then write a site for referral program (integrate with the accounting system too).
Then set up monitoring system and write about it, at the same time keep up with the tutorial, source code change.
No timeline available if you have no idea about what you are doing.. Having said that, any question that deserve a tutorial, keep the question coming, I’ll write about it.