Feedback on Layer 2 Payouts

Node Operators
1

Hello community!

After completing last month’s payouts, we shared some of our thoughts about where things might go next for storage node operators, and one particular piece stood out to many operators here, which was that we were contemplating making zkSync be the default behavior.

We have heard you loud and clear and (unless the community clearly requests otherwise down the road) will not be making zkSync the default.

But we wanted to share more about our thoughts around this. First, our plans with zkSync were just one aspect of a recently published blog post, which we encourage you to read in full. In it, we talk about upcoming changes to the incentive model such as held amount and pricing, in addition to technical aspects like payment layer. Please take a look at it and let us know what other feedback or questions you have.

Over time, many decentralized systems need to re-evaluate their incentive structures (much like Ethereum is in the progress of doing), and we must do so too, as our network grows. We want to make sure we take the time to listen to and get thoughtful feedback from our operator community before making these changes, and so we want to apologize for how this “it’s likely zkSync will become the default” was sold, instead of a more cordial “this is what we’re thinking. What do you think?” approach.

Fees on layer 1 right now over the last few months have sometimes reached over $100 USD per transaction, which is a significant problem. We understand this disproportionately impacts new node operators or small operators who aren’t earning that much. We’ve been investigating many different ways to fix this, but of course the current solution is the minimum payment threshold, in which we simply hang on to your funds for you until your balance is four times larger than the transaction fee we would need to pay to get it to you.

When we originally adopted Ethereum for the STORJ token, we did so because we believe in the future of the Ethereum ecosystem, and believed that it would be the best opportunity to make payments programmatic, decentralized, and keep fees low. The current fee situation is certainly not something we anticipated. That said, we still believe in the Ethereum ecosystem, and believe that a couple of trends make Ethereum the best choice for us.

  1. Layer 2 scaling solutions such as zkRollups and optimistic rollups show great promise at increasing per-block transaction counts by orders of magnitude (see: An Incomplete Guide to Rollups).
  2. Ethereum’s upcoming changes such as the EIP-1559 fee market overhaul appear poised to reduce the fee problem.
  3. Ethereum’s upcoming move to proof of stake will significantly increase performance, significantly reduce resource usage, and overall improve Ethereum all around.
  4. The ERC-20 standard is starting to see great cross-chain support (The Wrap Protocol on Tezos, the EVM chain on Avalanche, upcoming cross-chain support in Polygon, Cardano, Polkadot, etc). ERC-20 has become the flag bearer for tokens such as ours. Perhaps soon you will be able to receive and transfer STORJ token across chains.

We have been on the forefront of pushing this ecosystem to its limits and we want to invest in continuing to adopt new and better technologies to push the ecosystem forward, and we are evaluating many options along these angles.

One useful aspect of zkSync for us is that it works with all wallets, including exchange wallets, via an alternate withdrawal mechanism. A lot of the feedback we received was concern that zkSync wouldn’t work with certain wallets, and if this alternate withdrawal mechanism didn’t exist, we wouldn’t have even suggested it to begin with. So it’s a good point, but one that we didn’t clarify well enough upfront.

The other feedback we received was that using zkSync shifts the cost of withdrawal to the operator, and this is indeed an interesting tradeoff that we believe will lessen as more exchanges and wallet providers add native zkSync support (zkSync backers include Binance and Coinbase, for example). However, given the current cost sensitivity of our storage node operators today, we understand that many storage node operators prefer the current layer 1 minimum payment threshold process and so unless the situation changes significantly and operators want something else, we will be leaving Layer 1 as the default for Storj Labs operated Satellites.

If you’re a node operator who has decided to initiate graceful exit, you may be wondering what happens to funds that haven’t met the minimum payment threshold. When we originally implemented the minimum payment threshold, we did so in a very strict way, which was that even if you gracefully exited and had, for example, $39 accrued, but the minimum payment threshold was $40, we would not pay it out. This seemed undesirable to us, and so we added the concept of wallet addresses becoming “terminal,” in which the wallet is no longer associated with any live node. In cases such as this, for the last few months, we would pay out the balance, regardless of the threshold. We didn’t make a big deal about it, but hopefully the community felt supported.

Recently, we did a sweep and cleaned out many nodes we hadn’t seen in months, and discovered that many of these nodes belonged to operators with a terminal balance of sometimes less than a dollar. These are payments these operators really earned, but we were unable to pay due to the payment threshold before the wallet was marked terminal. We spent somewhere around $15k USD equivalent in Ethereum fees this last month sending storage node operators closing balances that amounted to dollars or cents. We believed that switching the default to zkSync would at least allow us to close out small value accounts more efficiently, with the hope that soon zkSync would have broader support with exchanges and so on, so small balances could more easily be cleaned up when the technology and integrations existed. It was in this context of supporting these smaller operators that we started leaning towards making zkSync be the default.

Instead, given the feedback from our community, we are proposing two replacement changes we think will actually make a lot of sense given current policies:

  1. Instead of calculating the minimum payment threshold average over the last 12 hours upfront and running all payments using that threshold, we want to move to calculate the minimum payment threshold on a per transaction level. We will do our best to run payments at affordable times considering layer 1 fees to get the most payments out, and may even attempt to run payments more than once to catch people we miss the first round. Again, we believe that the Ethereum ecosystem will soon make this problem significantly less of a challenge, if all we do is wait patiently, but we need an interim solution now.

  2. We propose to no longer close out terminal wallet balances. Many active node operators perhaps did not know we were even doing this, but it does affect nodes that are gracefully exiting. So, the proposal is instead, if you are interested in getting your final payment and your balance is less than the payment threshold, then you can either:
    a. let us hang on to your balance for you if and until the payment threshold drops enough or the layer 1 technology advances, or
    b. you can configure your node to opt into a lower-fee option such as zkSync.

It’s worth saying that we are indeed working on adding additional options besides zkSync and are considering other technology stacks. We’ve heard from a lot of you about ways you’d like to see our payment functionality change - please let us know if there is a technology we should be considering. Chances are, we actually are already considering it, but the more we hear from you, the better we can prioritize.

As always, thanks again for being a stellar node operator community, and we commit to do right by you. You are the backbone of this community and network, and we are grateful for all of your contributions. We understand very well that for Storj to succeed, it absolutely has to be economically viable and rewarding to be a storage node operator. Under no circumstances can our business succeed if storage node operators cannot also turn a profit. As we continue to work on refining our payout processes and contemplate possible changes to the incentive model, we voraciously welcome ideas, input, and feedback from our community.

17 Likes
2

JT and I apologize there is so much info and no TLDR, but there’s a lot of information to share and we want to err on the side of transparency. Let’s keep the conversation going.

9 Likes
3

It’s late, I am on my way to bed and digesting this interesting post.
For the time being, let me just chime in to say thank you for being as honest and transparent as possible and for making the effort to listen and act on SNO concerns.
Lots of positive feelings all round.

5 Likes
4

Thanks JT. To simplify the issue of wallets with a “terminal balance” I think it’s perfectly reasonable to introduce a policy which specifies a minimum balance for a payout. Probably something around $10. Assume that catches most of the issues? Not reached payout? - then you forfeit it. (obviously this should only apply to wallets which are no longer in use for several months). Fees on those I agree Storj will have to suck up, but hopefully a minimum payment threshold would alleviate most of the issue.

Re development on other payment technology stacks, I’d personally rather Storj focused on the “day job” with product development. I feel there’s too much focus being placed on appeasing SNOs with super small payments who are also the ones contributing the smallest to the network. I know that sounds harsh, and the whole point is for it to be distributed and inclusive to anyone however I really don’t understand why its so important we are paying out tiny balances each month. I think its perfectly acceptable to introduce a limit (as you have done). No-one is losing out and they have the option of zkSync. The problem will resolve its self with Ethereum network upgrades anyway.

Thanks as always for listening to the community and for your comprehensive update.

5 Likes
5

SNO-initiated withdrawals. Just credit every SNO with n% worth of pending balance and let him decide when to withdraw and what fee to pay.
Let me put it straight, storjlabs sucks at timing the payout. Really. We had a few days of 30 gwei average at the beginning of the month and now we’re back to 60 gwei average, yet you paid >300 gwei to complete payouts.
Let people withdraw when they want and set an arbitrary fee within the previously mentioned credit, then think of some incentive/protective mechanism so people won’t use 100% of the gas allowance by mistake or out of spite.

2 Likes
6

Thanks @jtolio, this was actually really insightful and explains several things we recently discussed here on the forums. I really appreciate that you’re taking our feedback to heart and making some changes. Like you, I believe in the future of zkSync and I mostly think it just needs some wider adoption to be a great option for everyone.

I had read it at the time. Most of the post reiterated how things are already working though. But while those of us paying attention to the pricing model changes already saw a change in node compensation coming, the changes to held amount/bonus structure mentioned are new. However, there’s basically only a one sentence mention and I’m not entirely sure how to interpret what is said. What I can say is that I’m a fan of the idea that further incentives to remain a reliable node operators don’t stop after the first 9 months. I would love to know a little more about your thoughts of what this bonus model could look like though.

Also, it was mentioned elsewhere, but where the article refers to the community earnings estimator to see potential earnings, it actually calls it the earnings calculator and links to the earnings calculator which doesn’t show potential earnings but calculates a nodes earnings based on its DB’s. I think you meant to link to this one instead. Realistic earnings estimator

When I last looked into this, it was still a manual process. It’s great to see that is no longer the case. However, this method still requires a web3 compatible wallet in order to sign the request and an ether balance on that wallet to pay for the transaction fees. The reason some node operators use an exchange address is primarily to avoid those transaction fees. All that said, I think it’s great that this step no longer requires manual forms.

This isn’t even really the case in many scenarios. This really only applies when node operators use an exchange address for payouts. Of you get the payouts in your own wallet, then moving the tokens around is going to cost you a transaction fee anyway. And you can just move them from L2 to any L1 address for practically the same fee as just moving ERC20 tokens on L1. So in most cases zkSync incurs only a negligibly higher fee to transfer tokens out. And the upside is that that fee can be paid in STORJ tokens, so you don’t have to get Ether to access your earnings. That’s a worthy tradeoff for most.

On to the changes proposed!

I’m really happy to see this. I think the old process risked you paying fees that were way too high AND paid out fewer operators than you could have. This change improves it on both ends. I made a suggestion elsewhere already to possibly further improve this by just ordering the payouts in descending payout amounts. Run the process until it hits the threshold and keep monitoring for lower fees and resume when the gas price drops. You can just keep that running until the 15th (if you want to stick to first half of the month, longer is of course possible too). That will further optimize how many payouts can be sent.

This is of course a little more tricky. I think one of the issues is that currently disqualified nodes were treated the same as gracefully exited ones. I think in case of disqualification, what you are proposing is perfectly reasonable. But in case of a node operator who has been reliable in the past and gracefully shut down their node, it is in their and your best interest to make that payout. First of all, they have to have been around and reliable for at least 6 months in order to start the graceful exit. Second, getting the held amount back is exactly the incentive for node operators to run the graceful exit. If that payout is held, that incentive goes away. And lastly, I’m willing to bet that almost all the really small amounts weren’t cases of graceful exit. I would understand if you still went in the direction you suggested, but I think treating graceful exits a little more leniently might be worth considering.

Thanks again for being transparent and having open conversations with us. It makes me feel like it’s absolutely worth taking the time to provide some in depth feedback on your ideas and keeps me a very happy node operator!

Edit: Just adding some more info. For now I’m staying on L1 payouts for a few reasons. I have notifications set up for transactions on my wallet and I’m not aware of something like that for zkSync transactions. Additionally, I would prefer not to split my tokens over 2 different places which would require 2x transaction fees to transfer. Additionally I make enough to usually be over the threshold, though not this last month. These are all just minor reasons of convenience and when exchanges adopt zkSync I will definitely switch. I believe it’s a good option for most, but just slightly inconvenient for my current situation.

6 Likes
7

This is all great feedback. A few comments/thoughts:

@will.topping - I don’t know if it’s any consolation to your point, but in terms of team focus, we basically have no more than 2 people spending a couple of days a month tops on the cryptocurrency transfer pipeline out of an eng team of 42. We’re predominantly focused on the storage platform, but it is still worth considering how to improve our cryptocurrency transfer pipeline some, for all of the reasons in this thread. Perhaps this amount of focus answers more questions than just this one! Haha

@hoarder (and others!) - I of course love the idea for SNO-initiated withdrawals, and I have a question for you. There’s two ways we could implement SNO-initiated withdrawals:

  1. In one way, we extend our software to allow SNOs to trigger our payment pipeline individually, whenever they hit the button. This is a challenge because it requires a complete rewrite of our entire payment pipeline, which is currently designed around batch processing.
  2. In another way, we transfer STORJ token as we currently do to some holding location, and that holding location has a withdrawal button.

So here’s my big question: how is the second scenario different than zkSync? Is the only difference that the transfer costs are paid for by the SNO? If we, say, included some extra STORJ to cover some reasonable USD-valued cost of an expected Ethereum transaction for the withdrawal, would that count as having implemented SNO-initiated withdrawals to layer 1? All of the basic building blocks are there - a SNO’s balance increases monthly, and whenever they want to withdraw that balance to layer 1, they are able to whenever they so choose.

The above is a genuine question for all of you - we saw zkSync as a quick win for SNO-initiated withdrawals, but it wasn’t received that way. We could theoretically fix the fee burden with an incentive, but we wouldn’t be able to fix the pre-withdrawal exposure to token volatility I suppose. Otherwise it’s really pretty dang close. For those of you who wanted SNO-initiated withdrawal, what is the main reason zkSync doesn’t solve that problem?

@BrightSilence - as always, thank you for your incredibly thoughtful and thorough posts. You provide good feedback that we have more to cover in terms of held amount/bonus changes, you’re right. In terms of your suggestion about treating disqualified and graceful exited nodes differently, one thing to know is our system currently doesn’t clearly differentiate between the two cases. Gracefully exited nodes are usually marked as disqualified at the completion of the exit, so our system in some parts of the code struggles to tell the difference. But you’re right, we could have different thresholds based on graceful exit state, and this is a valuable enough suggestion to warrant going and fixing that. We’ll look into it.

One other possibility - perhaps instead of one minimum payment threshold, we consider two? One for steady state, what we have, 4x the tx fee, and one for terminal closing payments, maybe lower like 1x? I’m reluctant to make it a flat fee like @will.topping suggests instead of being relative to transaction costs, simply because the fees seem completely beyond control right now. Maybe this will be more reasonable after EIP-1559?

1 Like
8

I haven’t been mining a lot lately but afaik most pools to have a minimal threshold that is also used for “terminal payments”. If you stay below that threshold, your earnings will eventually be “lost” and belong to the pool operator.
Therefore in my opinion storj can do the same as it is a known concept (at least for the miners). If people “test” storj for a few days/weeks and get a few cents, it’s the same as me “testing” a mining pool (like I did recently, got less than $1, which I’ll never receive, I think. Maybe I’m wrong about it but that’s my impression on mining pools).

3 Likes
9

Short question for now, will read the post more carefully later:

Given that you’ve closed pending terminal wallets already, wouldn’t it be more desirable (and less controversial) to change the policy so that only new nodes’ default is zkSync, without impacting old nodes?

2 Likes
10

Because zkSync withdrawals are too expensive. A random ERC20 transaction in a block that was mined when I was writing this post cost 0.0021 ETH in fees. zkSync L1 withdrawal initiated at the same time would cost 0.012 ETH, or about $42.

2 Likes
11

That really doesn’t line up with when I investigated this a while ago. The difference was about 10% in costs. Can you provide a little more info on how you found these numbers?

Edit: Actually, not even 3%. Found my previous post. New payment method launched (zkSync) - #7 by BrightSilence

12

That would have the impact of creating different classes of SNO’s based on when you joined and might not be an environment storj would want to foster.

13

I wouldn’t say changing the default creates different classes. You could have the installer simply ask and have zkSync selected by default and you could have the docker run command include the zkSync option and explain what it is in the setup steps and how to remove it if you don’t want it. That would be a fairly subtle way to go about it and at least make people really consider the option.

4 Likes
14

Yep, that’s what I was thinking of.

1 Like
15

One could say this is already the case: those who joined the project at different stages may have wildly different experiences with ingress. Some people joined early enough to benefit from surge payments, those who still managed to get a lot of (dormant) test data, those who joined recently and has only observed little traffic…

16

Went to zksync explorer, picked a random address, then pasted it into https://withdraw.zksync.io/
They say on the page that the amount is higher than normal tx, but difference should be negligible.

17

That is the alternative withdrawal. This is an emergency withdrawal system that should only be used if you don’t have access to the private keys of the wallet. This allows you to withdraw to the same address on L1 only and is much more expensive due to the higher complexity of the contract/transaction. It can be used for example if you by accident sent zkSync payouts to an exchange address for which you don’t have the keys.

Normal withdrawal is done through https://wallet.zksync.io/ and is only a few percent higher than a normal L1 ERC20 transaction.

3 Likes
18

So much for the “negligible” difference.

But even then erc20 transfers will be cheaper, people pay 21k gas to transfer STORJ to binance for example, and average tx cost seems to be 36.5k, which is also what I pay when I transfer STORJ to the exchange. It’s not as bad as with the emergency(?) withdrawal, but still a significant difference.

I won’t mind using it as long as extra cost is covered by storjlabs, but then storjlabs will have to figure out a way to enforce one wallet per SNO policy so people don’t try to run every node on a separate wallet to get withdrawal money, consolidate and profit from that.

19

This is incorrect. The amount is different depending on whether the address already holds the tokens or not and since binance moves all deposited balances out of the wallet, you never really pay just 21k. Transactions will cost 51,496 to binance as a result. And yes, I checked that all my transactions to binance cost that much. I checked again that withdrawing to that same address costs 52,700.

You can use this to check.

curl -X POST -H 'Content-type: application/json'   -d '{
     "jsonrpc":"2.0",
     "id":1, "method": "get_tx_fee",
     "params": ["Withdraw", "0xAddressYourWithdrawingToHere", "STORJ"]
     }'   https://api.zksync.io/jsrpc

So 52,700 / 51,496 = 102.3%
You’re literally only paying 2.3% more in transaction fees. At the current gas prices, that is 32 cents. That’s 32 cents on top of 14 USD you’d already be paying anyway.

I’d call the difference between 14.00 USD and 14.32 USD negligible, wouldn’t you agree?

And btw, you can pay that fee in Storj, so it saves you from having to transfer ETH to pay for transactions. If you take this into account, zkSync is actually cheaper.

2 Likes
20

I don’t know why these binance transactions are so cheap, I expected 36508 one be the cheapest, but anyway.
Here’s a couple of transactions, one new, another one fairly dated, both use 21508 gas:
Ethereum Transaction Hash (Txhash) Details | Etherscan
Ethereum Transaction Hash (Txhash) Details | Etherscan

Here’s a couple of 36508 gas transaction:
Ethereum Transaction Hash (Txhash) Details | Etherscan
Ethereum Transaction Hash (Txhash) Details | Etherscan

52700 gas used by zkSync is either 2.45 or 1.44 times more. $4.5 if I go by your numbers, which would be the difference in my case as I don’t think my STORJ transactions ever used more gas.