From d1ab8a8946521a682c525578ec2435a1d3735629 Mon Sep 17 00:00:00 2001 From: Alfredo Garcia Date: Thu, 8 Jul 2021 20:53:14 -0300 Subject: [PATCH] Value pools design (#2430) * add a draft for value pools design * add some `ValueBalance` operators * add `value_balance` methods to modules * fix some minors * finalize the last part of the implementation design * replace wrong AllowNegative with correct NegativeAllowed * add design PR to header * update scope * remove details from transaction `value_balance()` * update definitions * change method name * return Result in operators * fix the TODOs * implement `UpdateWith` for `ValueBalance` * add details to `ValueBalance` serialization * add a panic to block value balance * fix `remaining_transaction_value()` * add the block value balance into `FinalizedState` * populate the `Chain` with the finalized tip value balance * remove redundant text from definition * add docs to `Chain` field * trigger the `remaining_transaction_value()` check * fix mistake * add some tests, remove some not needed sections * add a summary of the implementation * do some minor fixes to consensus rules text * clarify some names * fix `UpdateWith` * move the remaining transaction value consensus rule * fix serialization bug * fix typo * Add a missing test * typo Co-authored-by: Conrado Gouvea Co-authored-by: teor Co-authored-by: Conrado Gouvea --- book/src/dev/rfcs/0012-value-pools.md | 519 ++++++++++++++++++++++++++ 1 file changed, 519 insertions(+) create mode 100644 book/src/dev/rfcs/0012-value-pools.md diff --git a/book/src/dev/rfcs/0012-value-pools.md b/book/src/dev/rfcs/0012-value-pools.md new file mode 100644 index 00000000..8a981378 --- /dev/null +++ b/book/src/dev/rfcs/0012-value-pools.md @@ -0,0 +1,519 @@ +- Feature Name: value_pools +- Start Date: 2021-06-29 +- Design PR: [ZcashFoundation/zebra#2430](https://github.com/ZcashFoundation/zebra/pull/2430) +- Zebra Issue: [ZcashFoundation/zebra#2152](https://github.com/ZcashFoundation/zebra/issues/2152) + +# Summary +[summary]: #summary + +This document describes how to verify the Zcash chain and transaction value pools in Zebra. + +# Motivation +[motivation]: #motivation + +In the Zcash protocol there are consensus rules that: + - prohibit negative chain value pools [ZIP-209], and + - restrict the creation of new money to a specific number of coins in each coinbase transaction. [Spec Section 3.4](https://zips.z.cash/protocol/protocol.pdf#transactions) + +These rules make sure that a fixed amount of Zcash is created by each block, even if there are vulnerabilities in some shielded pools. + +Checking the coins created by coinbase transactions and funding streams is out of scope for this design. + +[ZIP-209]: https://zips.z.cash/zip-0209 + +# Definitions +[definitions]: #definitions + +- `value balance` - The total change in value caused by a subset of the blockchain. +- `transparent value balance` - The change in the value of the transparent pool. The sum of the outputs spent by transparent inputs in `tx_in` fields, minus the sum of newly created outputs in `tx_out` fields. +- `coinbase transparent value balance` - The change in the value of the transparent pool due to a coinbase transaction. The coins newly created by the block, minus the sum of newly created outputs in `tx_out` fields. In this design, we temporarily assume that all coinbase outputs are valid, to avoid checking the created coins. +- `sprout value balance` - The change in the sprout value pool. The sum of all sprout `vpub_old` fields, minus the sum of all `vpub_new` fields. +- `sapling value balance` - The change in the sapling value pool. The negation of the sum of all `valueBalanceSapling` fields. +- `orchard value balance` - The change in the orchard value pool. The negation of the sum of all `valueBalanceOrchard` fields. +- `remaining transaction value` - The leftover value in each transaction, collected by miners as a fee. This value must be non-negative. In Zebra, calculated by subtracting the sprout, sapling, and orchard value balances from the transparent value balance. In the spec, defined as the sum of transparent inputs, minus transparent outputs, plus `v_sprout_new`, minus `v_sprout_old`, plus `vbalanceSapling`, plus `vbalanceOrchard`. +- `transaction value pool balance` - The sum of all the value balances in each transaction. There is a separate value for each transparent and shielded pool. +- `block value pool balance` - The sum of all the value balances in each block. There is a separate value for each transparent and shielded pool. +- `chain value pool balance` - The sum of all the value balances in a valid blockchain. Each of the transparent, sprout, sapling, and orchard chain value pool balances must be non-negative. + +# Guide-level explanation +[guide-level-explanation]: #guide-level-explanation + +There is a value pool for transparent funds, and for each kind of shielded transfer. These value pools exist in each transaction, each block, and each chain. + +We need to check each chain value pool as blocks are added to the chain, to make sure that chain balances never go negative. + +We also need to check that non-coinbase transactions only spend the coins provided by their inputs. + +Each of the chain value pools can change its value with every block added to the chain. This is a state feature and Zebra handle this in the `zebra-state` crate. We propose to store the pool values for the finalized tip height on disk. + +## Summary of the implementation: + +- Create a new type `ValueBalance` that will contain `Amount`s for each pool(transparent, sprout, sapling, orchard). +- Create `value_pool()` methods on each relevant submodule (transparent, joinsplit, sapling and orchard). +- Create a `value_pool()` method in transaction with all the above and in block with all the transaction value balances. +- Pass the value balance of the incoming block into the state. +- Get a previously stored value balance. +- With both values check the consensus rules (constraint violations). +- Update the saved values for the new tip. + +# Reference-level explanation +[reference-level-explanation]: #reference-level-explanation + +## Consensus rules +[consensus-rules]: #consensus-rules + +### Shielded Chain Value Pools + +If any of the "Sprout chain value pool balance", "Sapling chain value pool balance", or "Orchard chain value pool balance" would become negative in the block chain created as a result of accepting a block, then all nodes MUST reject the block as invalid. + +Nodes MAY relay transactions even if one or more of them cannot be mined due to the aforementioned restriction. + +https://zips.z.cash/zip-0209#specification + +### Transparent Transaction Value Pool & Remaining Value + +Transparent inputs to a transaction insert value into a transparent transaction value pool associated with the transaction, and transparent outputs remove value from this pool. As in Bitcoin, the remaining value in the pool is available to miners as a fee. + +Consensus rule: The remaining value in the transparent transaction value pool MUST be nonnegative. + +https://zips.z.cash/protocol/protocol.pdf#transactions + +Note: there is no explicit rule that the remaining balance in the transparent chain value pool must be non-negative. But it follows from the transparent transaction value pool consensus rule, and the definition of value addition. + +### Sprout Chain Value Pool + +Each JoinSplit transfer can be seen, from the perspective of the transparent transaction value pool , as an input and an output simultaneously. + +`vold` takes value from the transparent transaction value pool and `vnew` adds value to the transparent transaction value pool . As a result, `vold` is treated like an output value, whereas `vnew` is treated like an input value. + +As defined in [ZIP-209], the Sprout chain value pool balance for a given block chain is the sum of all `vold` field values for transactions in the block chain, minus the sum of all `vnew` fields values for transactions in the block chain. + +Consensus rule: If the Sprout chain value pool balance would become negative in the block chain created as a result of accepting a block, then all nodes MUST reject the block as invalid. + +https://zips.z.cash/protocol/protocol.pdf#joinsplitbalance + +### Sapling Chain Value Pool + +A positive Sapling balancing value takes value from the Sapling transaction value pool and adds it to the transparent transaction value pool. A negative Sapling balancing value does the reverse. As a result, positive `vbalanceSapling` is treated like an input to the transparent transaction value pool, whereas negative `vbalanceSapling` is treated like an output from that pool. + +As defined in [ZIP-209], the Sapling chain value pool balance for a given block chain is the negation of the sum of all `valueBalanceSapling` field values for transactions in the block chain. + +Consensus rule: If the Sapling chain value pool balance would become negative in the block chain created as a result of accepting a block, then all nodes MUST reject the block as invalid. + +https://zips.z.cash/protocol/protocol.pdf#saplingbalance + +### Orchard Chain Value Pool + +Orchard introduces Action transfers, each of which can optionally perform a spend, and optionally perform an output. Similarly to Sapling, the net value of Orchard spends minus outputs in a transaction is called the Orchard balancing value, measured in zatoshi as a signed integer `vbalanceOrchard`. + +`vbalanceOrchard` is encoded in a transaction as the field `valueBalanceOrchard`. If a transaction has no Action descriptions, `vbalanceOrchard` is implicitly zero. Transaction fields are described in § 7.1 ‘Transaction Encoding and Consensus’ on p. 116. + +A positive Orchard balancing value takes value from the Orchard transaction value pool and adds it to the transparent transaction value pool. A negative Orchard balancing value does the reverse. As a result, positive `vbalanceOrchard` is treated like an input to the transparent transaction value pool, whereas negative `vbalanceOrchard` is treated like an output from that pool. + +Similarly to the Sapling chain value pool balance defined in [ZIP-209], the Orchard chain value pool balance for a given block chain is the negation of the sum of all `valueBalanceOrchard` field values for transactions in the block chain. + +Consensus rule: If the Orchard chain value pool balance would become negative in the block chain created as a result of accepting a block , then all nodes MUST reject the block as invalid. + +https://zips.z.cash/protocol/protocol.pdf#orchardbalance + +## Proposed Implementation + +### Create a new `ValueBalance` type + +- Code will be located in a new file: `zebra-chain/src/value_balance.rs`. +- Supported operators apply to all the `Amount`s inside the type: `+`, `-`, `+=`, `-=`, `sum()`. +- Implementation of the above operators are similar to the ones implemented for `Amount` in `zebra-chain/src/amount.rs`. In particular, we want to return a `Result` on them so we can error when a constraint is violated. +- We will use `Default` to represent a totally empty `ValueBalance`, this is the state of all pools at the genesis block. + +```rust +#[serde(bound = "C: Constraint")] +struct ValueBalance { + transparent: Amount, + sprout: Amount, + sapling: Amount, + orchard: Amount, +} + +impl ValueBalance { + /// [Consensus rule]: The remaining value in the transparent transaction value pool MUST be nonnegative. + /// + /// This rule applies to Block and Mempool transactions. + /// + /// [Consensus rule]: https://zips.z.cash/protocol/protocol.pdf#transactions + fn remaining_transaction_value(&self) -> Result, Err> { + // This rule checks the transparent value balance minus the sum of the sprout, sapling, and orchard + // value balances in a transaction is nonnegative + self.transparent - [self.sprout + self.sapling + self.orchard].sum() + } +} + +impl Add for Result> +where + C: Constraint, +{ + +} + +impl Sub for Result> +where + C: Constraint, +{ + +} + +impl AddAssign for Result> +where + C: Constraint, +{ + +} + +impl SubAssign for Result> +where + C: Constraint, +{ + +} + +impl Sum for Result> +where + C: Constraint, +{ + +} + +impl Default for ValueBalance +where + C: Constraint, +{ + +} +``` + +### Create a method in `Transaction` that returns `ValueBalance` for the transaction + +We first add `value_balance()` methods in all the modules we need and use them to get the value balance for the whole transaction. + +#### Create a method in `Input` that returns `ValueBalance` + +- Method location is at `zebra-chain/src/transparent.rs`. +- Method need `utxos`, this information is available in `verify_transparent_inputs_and_outputs`. +- If the utxos are not available in the block or state, verification will timeout and return an error + +```rust +impl Input { + fn value_balance(&self, utxos: &HashMap) -> ValueBalance { + + } +} +``` + +#### Create a method in `Output` that returns `ValueBalance` + +- Method location is at `zebra-chain/src/transparent.rs`. + +```rust +impl Output { + fn value_balance(&self) -> ValueBalance { + + } +} +``` + +#### Create a method in `JoinSplitData` that returns `ValueBalance` + +- Method location is at `zebra-chain/src/transaction/joinsplit.rs` + +```rust +pub fn value_balance(&self) -> ValueBalance { + +} +``` + +#### Create a method in `sapling::ShieldedData` that returns `ValueBalance` + +- Method location is at `zebra-chain/src/transaction/sapling/shielded_data.rs` + +```rust +pub fn value_balance(&self) -> ValueBalance { + +} +``` + +#### Create a method in `orchard::ShieldedData` that returns `ValueBalance` + +- Method location is at `zebra-chain/src/transaction/orchard/shielded_data.rs` + +```rust +pub fn value_balance(&self) -> ValueBalance { + +} +``` + +#### Create the `Transaction` method + +- Method location: `zebra-chain/src/transaction.rs` +- Method will use all the `value_balances()` we created until now. + +```rust +/// utxos must contain the utxos of every input in the transaction, +/// including UTXOs created by earlier transactions in this block. +pub fn value_balance(&self, utxos: &HashMap) -> ValueBalance { + +} +``` + +### Create a method in `Block` that returns `ValueBalance` for the block + +- Method location is at `zebra-chain/src/block.rs`. +- Method will make use of `Transaction::value_balance` method created before. + +```rust +/// utxos must contain the utxos of every input in the transaction, +/// including UTXOs created by a transaction in this block, +/// then spent by a later transaction that's also in this block. +pub fn value_balance(&self, utxos: &HashMap) -> ValueBalance { + self.transactions() + .map(Transaction::value_balance) + .sum() + .expect("Each block should have at least one coinbase transaction") +} +``` + +### Check the remaining transaction value consensus rule + +- Do the check in `zebra-consensus/src/transaction.rs` +- Make the check part of the [basic checks](https://github.com/ZcashFoundation/zebra/blob/f817df638b1ba8cf8c261c536a30599c805cf04c/zebra-consensus/src/transaction.rs#L168-L177) + +```rust +.. +// Check the remaining transaction value consensus rule: +tx.value_balance().remaining_transaction_value()?; +.. +``` + +### Pass the value balance for this block from the consensus into the state + +- Add a new field into `PreparedBlock` located at `zebra-state/src/request.rs`, this is the `NonFinalized` section of the state. + +```rust +pub struct PreparedBlock { + .. + /// The value balances for each pool for this block. + pub block_value_balance: ValuePool, +} +``` +- In `zebra-consensus/src/block.rs` pass the value balance to the zebra-state: + +```rust +let block_value_balance = block.value_balance(); +let prepared_block = zs::PreparedBlock { + .. + block_value_balance, +}; +``` + +### Add a value pool into the state `Chain` struct + +- This is the value pool for the non finalized part of the blockchain. +- Location of the `Chain` structure where the pool field will be added: `zebra-state/src/service/non_finalized_state/chain.rs` + +```rust +pub struct Chain { + .. + /// The chain value pool balance at the tip of this chain. + value_pool: ValueBalance, +} +``` +- Add a new argument `finalized_tip_value_balance` to the `commit_new_chain()` method located in the same file. +- Pass the new argument to the Chain in: + +```rust +let mut chain = Chain::new(finalized_tip_history_tree, finalized_tip_value_balance); +``` + +Note: We don't need to pass the finalized tip value balance into the `commit_block()` method. + +### Check the consensus rules when the chain is updated or reversed + +- Location: `zebra-state/src/service/non_finalized_state/chain.rs` + +```rust +impl UpdateWith> for Chain { + fn update_chain_state_with(&mut self, value_balance: &ValueBalance) -> Result<(), Err> { + self.value_pool = (self.value_pool + value_balance)?; + Ok(()) + } + fn revert_chain_state_with(&mut self, value_balance: &ValueBalance) -> Result<(), Err> { + self.value_pool = (self.value_pool + value_balance)?; + Ok(()) + } +} +``` + +### Changes to finalized state + +The state service will call `commit_new_chain()`. We need to pass the value pool from the disk into this function. + +```rust +self.mem + .commit_new_chain(prepared, self.disk.history_tree(), self.disk.get_pool())?; +``` + +We now detail what is needed in order to have the `get_pool()` method available. + +#### Serialization of `ValueBalance` + +In order to save `ValueBalance` into the disk database we must implement `IntoDisk` and `FromDisk` for `ValueBalance` and for `Amount`: + +```rust +impl IntoDisk for ValueBalance { + type Bytes = [u8; 32]; + + fn as_bytes(&self) -> Self::Bytes { + [self.transparent.to_bytes(), self.sprout.to_bytes(), + self.sapling.to_bytes(), self.orchard.to_bytes()].concat() + } +} + +impl FromDisk for ValueBalance { + fn from_bytes(bytes: impl AsRef<[u8]>) -> Self { + let array = bytes.as_ref().try_into().unwrap(); + ValueBalance { + transparent: Amount::from_bytes(array[0..8]).try_into().unwrap() + sprout: Amount::from_bytes(array[8..16]).try_into().unwrap() + sapling: Amount::from_bytes(array[16..24]).try_into().unwrap() + orchard: Amount::from_bytes(array[24..32]).try_into().unwrap() + } + } +} + +impl IntoDisk for Amount { + type Bytes = [u8; 8]; + + fn as_bytes(&self) -> Self::Bytes { + self.to_bytes() + } +} + +impl FromDisk for Amount { + fn from_bytes(bytes: impl AsRef<[u8]>) -> Self { + let array = bytes.as_ref().try_into().unwrap(); + Amount::from_bytes(array) + } +} +``` +The above code is going to need a `Amount::from_bytes` new method. + +#### Add a `from_bytes` method in `Amount` + +- Method location is at `zebra-chain/src/amount.rs` +- A `to_bytes()` method already exist, place `from_bytes()` right after it. + +```rust +/// From little endian byte array +pub fn from_bytes(&self, bytes: [u8; 8]) -> Self { + let amount = i64::from_le_bytes(bytes).try_into().unwrap(); + Self(amount, PhantomData) +} +``` + +#### Changes to `zebra-state/src/request.rs` + +Add a new field to `FinalizedState`: + +```rust +pub struct FinalizedBlock { + .. + /// The value balance for transparent, sprout, sapling and orchard + /// inside all the transactions of this block. + pub(crate) block_value_balance: ValueBalance, +} +``` + +Populate it when `PreparedBlock` is converted into `FinalizedBlock`: + +```rust +impl From for FinalizedBlock { + fn from(prepared: PreparedBlock) -> Self { + let PreparedBlock { + .. + block_value_balance, + } = prepared; + Self { + .. + block_value_balance, + } + } +} +``` + +#### Changes to `zebra-state/src/service/finalized_state.rs` + +First we add a column of type `ValueBalance` that will contain `Amount`s for all the pools: transparent, sprout, sapling, orchard: + +```rust +rocksdb::ColumnFamilyDescriptor::new("tip_chain_value_pool", db_options.clone()), +``` + +At block commit(`commit_finalized_direct()`) we create the handle for the new column: + +```rust +let tip_chain_value_pool = self.db.cf_handle("tip_chain_value_pool").unwrap(); +``` + +Next we save each tip value pool into the field for each upcoming block except for the genesis block: + +```rust +// Consensus rule: The block height of the genesis block is 0 +// https://zips.z.cash/protocol/protocol.pdf#blockchain +if height == block::Height(0) { + batch.zs_insert(tip_chain_value_pool, height, ValueBalance::default()); +} else { + let current_pool = self.current_value_pool(); + batch.zs_insert(tip_chain_value_pool, height, (current_pool + finalized.block_value_balance)?); +} +``` + +The `current_value_pool()` function will get the stored value of the pool at the tip as follows: + +```rust +pub fn current_value_pool(&self) -> ValuePool { + self.db.cf_handle("tip_chain_value_pool") +} +``` + +## Test Plan +[test-plan]: #test-plan + +### Unit tests + - Create a transaction that has a negative remaining value. + - Test that the transaction fails the verification in `Transaction::value_balance()` + - To avoid passing the utxo we can have `0` as the amount of the transparent pool and some negative shielded pool. + +### Prop tests + + - Create a chain strategy that ends up with a valid value balance for all the pools (transparent, sprout, sapling, orchard) + - Test that the amounts are all added to disk. + - Add new blocks that will make each pool became negative. + - Test for constraint violations in the value balances for each case. + - Failures should be at `update_chain_state_with()`. + - Test consensus rules success and failures in `revert_chain_state_with()` + - TODO: how? + - serialize and deserialize `ValueBalance` using `IntoDisk` and `FromDisk` + + ### Manual tests + + - Zebra must sync up to tip computing all value balances and never breaking the value pool rules. + +## Future Work +[future-work]: #future-work + +Add an extra state request to verify the speculative chain balance after applying a Mempool transaction. (This is out of scope for our current NU5 and mempool work.) + +Note: The chain value pool balance rules apply to Block transactions, but they are optional for Mempool transactions: +> Nodes MAY relay transactions even if one or more of them cannot be mined due to the aforementioned restriction. + +https://zips.z.cash/zip-0209#specification + +Since Zebra does chain value pool balance validation in the state, we want to skip verifying the speculative chain balance of Mempool transactions.