diff --git a/book/src/dev/rfcs/0004-asynchronous-script-verification.md b/book/src/dev/rfcs/0004-asynchronous-script-verification.md index 4bd37f77..ff807476 100644 --- a/book/src/dev/rfcs/0004-asynchronous-script-verification.md +++ b/book/src/dev/rfcs/0004-asynchronous-script-verification.md @@ -14,20 +14,21 @@ on the ordering of operations in the state layer. As in the rest of Zebra, we want to express our work as a collection of work-items with explicit dependencies, then execute these items concurrently -and in parallel on a thread pool. +and in parallel on a thread pool. # Definitions [definitions]: #definitions -- *UTXO*: unspent transaction output. Transaction outputs are modeled in `zebra-chain` by the [`TransparentOutput`][transout] structure. -- Transaction input: an output of a previous transaction consumed by a later transaction (the one it is an input to). Modeled in `zebra-chain` by the [`TransparentInput`][transin] structure. -- lock script: the script that defines the conditions under which some UTXO can be spent. Stored in the [`TransparentOutput::lock_script`][lock_script] field. -- unlock script: a script satisfying the conditions of the lock script, allowing a UTXO to be spent. Stored in the [`TransparentInput::PrevOut::lock_script`][lock_script] field. +- *UTXO*: unspent transaction output. Transaction outputs are modeled in `zebra-chain` by the [`transparent::Output`][transout] structure. +- Transaction input: an output of a previous transaction consumed by a later transaction (the one it is an input to). Modeled in `zebra-chain` by the [`transparent::Input`][transin] structure. +- lock script: the script that defines the conditions under which some UTXO can be spent. Stored in the [`transparent::Output::lock_script`][lock_script] field. +- unlock script: a script satisfying the conditions of the lock script, allowing a UTXO to be spent. Stored in the [`transparent::Input::PrevOut::lock_script`][lock_script] field. + +[transout]: https://doc.zebra.zfnd.org/zebra_chain/transparent/struct.Output.html +[lock_script]: https://doc.zebra.zfnd.org/zebra_chain/transparent/struct.Output.html#structfield.lock_script +[transin]: https://doc.zebra.zfnd.org/zebra_chain/transparent/enum.Input.html +[unlock_script]: https://doc.zebra.zfnd.org/zebra_chain/transparent/enum.Input.html#variant.PrevOut.field.unlock_script -[transout]: https://doc.zebra.zfnd.org/zebra_chain/transaction/struct.TransparentOutput.html -[lock_script]: https://doc.zebra.zfnd.org/zebra_chain/transaction/struct.TransparentOutput.html#structfield.lock_script -[transin]: https://doc.zebra.zfnd.org/zebra_chain/transaction/enum.TransparentInput.html -[unlock_script]: https://doc.zebra.zfnd.org/zebra_chain/transaction/enum.TransparentInput.html#variant.PrevOut.field.unlock_script # Guide-level explanation [guide-level-explanation]: #guide-level-explanation @@ -55,8 +56,8 @@ done later, at the point that its containing block is committed to the chain. At a high level, this adds a new request/response pair to the state service: -- `Request::AwaitUtxo(OutPoint)` requests a `TransparentOutput` specified by `OutPoint` from the state layer; -- `Response::Utxo(TransparentOutput)` supplies requested the `TransparentOutput`. +- `Request::AwaitUtxo(OutPoint)` requests a `transparent::Output` specified by `OutPoint` from the state layer; +- `Response::Utxo(transparent::Output)` supplies requested the `transparent::Output`. Note that this request is named differently from the other requests, `AwaitUtxo` rather than `GetUtxo` or similar. This is because the request has @@ -72,7 +73,7 @@ is available. For instance, if we begin parallel download and verification of 500 blocks, we should be able to begin script verification of all scripts referencing outputs from existing blocks in parallel, and begin verification of scripts referencing outputs from new blocks as soon as they are committed -to the chain. +to the chain. Because spending outputs from older blocks is more common than spending outputs from recent blocks, this should allow a significant amount of @@ -82,7 +83,7 @@ parallelism. [reference-level-explanation]: #reference-level-explanation We add a `Request::AwaitUtxo(OutPoint)` and -`Response::Utxo(TransparentOutput)` to the state protocol. As described +`Response::Utxo(transparent::Output)` to the state protocol. As described above, the request name is intended to indicate the request's behavior: the request does not resolve until the state layer learns of a UTXO described by the request. @@ -163,7 +164,7 @@ structure described below. ```rust // sketch #[derive(Default, Debug)] -struct PendingUtxos(HashMap>); +struct PendingUtxos(HashMap>); impl PendingUtxos { // adds the outpoint and returns (wrapped) rx end of oneshot @@ -171,7 +172,7 @@ impl PendingUtxos { pub fn queue(&mut self, outpoint: OutPoint) -> impl Future>; // if outpoint is a hashmap key, remove the entry and send output on the channel - pub fn respond(&mut self, outpoint: OutPoint, output: TransparentOutput); + pub fn respond(&mut self, outpoint: OutPoint, output: transparent::Output); // scans the hashmap and removes any entries with closed senders diff --git a/book/src/dev/rfcs/0005-state-updates.md b/book/src/dev/rfcs/0005-state-updates.md index dd9b3a8c..78a78ff7 100644 --- a/book/src/dev/rfcs/0005-state-updates.md +++ b/book/src/dev/rfcs/0005-state-updates.md @@ -264,16 +264,18 @@ is completely empty. The `Chain` type is defined by the following struct and API: ```rust +#[derive(Debug, Default, Clone)] struct Chain { blocks: BTreeMap>, height_by_hash: HashMap, - tx_by_hash: HashMap, + tx_by_hash: HashMap, - utxos: HashSet, - sapling_anchors: HashSet, + created_utxos: HashSet, + spent_utxos: HashSet, sprout_anchors: HashSet, - sapling_nullifiers: HashSet, + sapling_anchors: HashSet, sprout_nullifiers: HashSet, + sapling_nullifiers: HashSet, partial_cumulative_work: PartialCumulativeWork, } ``` @@ -283,14 +285,16 @@ struct Chain { Push a block into a chain as the new tip 1. Update cumulative data members - - Add block to end of `self.blocks` - - Add hash to `height_by_hash` - - for each `transaction` in `block` - - add key: `transaction.hash` and value: `(height, tx_index)` to `tx_by_hash` - - Add new utxos and remove consumed utxos from `self.utxos` - - Add anchors to the appropriate `self._anchors` - - Add nullifiers to the appropriate `self._nullifiers` + - Add the block's hash to `height_by_hash` - Add work to `self.partial_cumulative_work` + - For each `transaction` in `block` + - Add key: `transaction.hash` and value: `(height, tx_index)` to `tx_by_hash` + - Add created utxos to `self.created_utxos` + - Add spent utxos to `self.spent_utxos` + - Add anchors to the appropriate `self._anchors` + - Add nullifiers to the appropriate `self._nullifiers` + +2. Add block to `self.blocks` #### `pub fn pop_root(&mut self) -> Arc` @@ -300,11 +304,13 @@ Remove the lowest height block of the non-finalized portion of a chain. 2. Update cumulative data members - Remove the block's hash from `self.height_by_hash` - - for each `transaction` in `block` - - remove `transaction.hash` from `tx_by_hash` - - Remove new utxos from `self.utxos` - - Remove the anchors from the appropriate `self._anchors` - - Remove the nullifiers from the appropriate `self._nullifiers` + - Subtract work from `self.partial_cumulative_work` + - For each `transaction` in `block` + - Remove `transaction.hash` from `tx_by_hash` + - Remove created utxos from `self.created_utxos` + - Remove spent utxos from `self.spent_utxos` + - Remove the anchors from the appropriate `self._anchors` + - Remove the nullifiers from the appropriate `self._nullifiers` 3. Return the block @@ -321,7 +327,7 @@ Fork a chain at the block with the given hash, if it is part of this chain. 4. Return `forked` -#### `fn pop_tip(&mut self) -> Arc` +#### `fn pop_tip(&mut self)` Remove the highest height block of the non-finalized portion of a chain. @@ -329,14 +335,13 @@ Remove the highest height block of the non-finalized portion of a chain. 2. Update cumulative data members - Remove the corresponding hash from `self.height_by_hash` + - Subtract work from `self.partial_cumulative_work` - for each `transaction` in `block` - remove `transaction.hash` from `tx_by_hash` - - Add consumed utxos and remove new utxos from `self.utxos` - - Remove anchors from the appropriate `self._anchors` - - Remove the nullifiers from the appropriate `self._nullifiers` - - Subtract work from `self.partial_cumulative_work` - -3. Return the block + - Remove created utxos from `self.created_utxos` + - Remove spent utxos from `self.spent_utxos` + - Remove anchors from the appropriate `self._anchors` + - Remove the nullifiers from the appropriate `self._nullifiers` #### `Ord` @@ -358,7 +363,8 @@ handled by `#[derive(Default)]`. 1. initialise cumulative data members - Construct an empty `self.blocks`, `height_by_hash`, `tx_by_hash`, - `self.utxos`, `self._anchors`, `self._nullifiers` + `self.created_utxos`, `self.spent_utxos`, `self._anchors`, + `self._nullifiers` - Zero `self.partial_cumulative_work` **Note:** The chain can be empty if: @@ -367,23 +373,35 @@ handled by `#[derive(Default)]`. all its blocks have been `pop`ped -### `ChainSet` Type -[chainset-type]: #chainset-type +### `NonFinalizedState` Type +[nonfinalizedstate-type]: #nonfinalizedstate-type -The `ChainSet` type represents the set of all non-finalized state. It -consists of a set of non-finalized but verified chains and a set of +The `NonFinalizedState` type represents the set of all non-finalized state. +It consists of a set of non-finalized but verified chains and a set of unverified blocks which are waiting for the full context needed to verify them to become available. -`ChainSet` is defined by the following structure and API: +`NonFinalizedState` is defined by the following structure and API: ```rust -struct ChainSet { - chains: BTreeSet, +/// The state of the chains in memory, incuding queued blocks. +#[derive(Debug, Default)] +pub struct NonFinalizedState { + /// Verified, non-finalized chains. + chain_set: BTreeSet, + /// Blocks awaiting their parent blocks for contextual verification. + contextual_queue: QueuedBlocks, +} - queued_blocks: BTreeMap, - queued_by_parent: BTreeMap>, - queued_by_height: BTreeMap>, +/// A queue of blocks, awaiting the arrival of parent blocks. +#[derive(Debug, Default)] +struct QueuedBlocks { + /// Blocks awaiting their parent blocks for contextual verification. + blocks: HashMap, + /// Hashes from `queued_blocks`, indexed by parent hash. + by_parent: HashMap>, + /// Hashes from `queued_blocks`, indexed by block height. + by_height: BTreeMap>, } ``` @@ -470,10 +488,10 @@ cannot be committed due to missing context. - `Chain` represents the non-finalized portion of a single chain -- `ChainSet` represents the non-finalized portion of all chains and all +- `NonFinalizedState` represents the non-finalized portion of all chains and all unverified blocks that are waiting for context to be available. -- `ChainSet::queue` handles queueing and or commiting blocks and +- `NonFinalizedState::queue` handles queueing and or commiting blocks and reorganizing chains (via `commit_block`) but not finalizing them - Finalized blocks are returned from `finalize` and must still be committed @@ -759,6 +777,20 @@ if the block is not in any non-finalized chain: the `block_by_height` tree (to get the block data). +### `Request::AwaitUtxo(OutPoint)` + +Returns + +- `Response::Utxo(transparent::Output)` + +Implemented by querying: + +- (non-finalized) if any `Chains` contain an `OutPoint` in their `created_utxos` and not their `spent_utxo` get the `transparent::Output` from `OutPoint`'s transaction +- (finalized) else if `OutPoint` is in `utxos_by_outpoint` return the associated `transparent::Output`. +- else wait for `OutPoint` to be created as described in [RFC0004] + +[RFC0004]: https://zebra.zfnd.org/dev/rfcs/0004-asynchronous-script-verification.html + # Drawbacks [drawbacks]: #drawbacks diff --git a/zebra-chain/src/amount.rs b/zebra-chain/src/amount.rs index 17b0633b..d65df02f 100644 --- a/zebra-chain/src/amount.rs +++ b/zebra-chain/src/amount.rs @@ -17,7 +17,7 @@ use byteorder::{ByteOrder, LittleEndian, ReadBytesExt, WriteBytesExt}; type Result = std::result::Result; /// A runtime validated type for representing amounts of zatoshis -#[derive(Debug, Eq, PartialEq, Clone, Copy, Serialize, Deserialize)] +#[derive(Debug, Eq, PartialEq, Clone, Copy, Serialize, Deserialize, Hash)] #[serde(try_from = "i64")] #[serde(bound = "C: Constraint")] pub struct Amount(i64, PhantomData); @@ -234,7 +234,7 @@ impl Constraint for NegativeAllowed { /// 0..=MAX_MONEY, /// ); /// ``` -#[derive(Clone, Copy, Debug, Eq, PartialEq)] +#[derive(Clone, Copy, Debug, Eq, PartialEq, Hash)] pub struct NonNegative {} impl Constraint for NonNegative { diff --git a/zebra-chain/src/sapling/note/nullifiers.rs b/zebra-chain/src/sapling/note/nullifiers.rs index 98a49706..7bd00e99 100644 --- a/zebra-chain/src/sapling/note/nullifiers.rs +++ b/zebra-chain/src/sapling/note/nullifiers.rs @@ -26,7 +26,7 @@ fn prf_nf(nk: [u8; 32], rho: [u8; 32]) -> [u8; 32] { } /// A Nullifier for Sapling transactions -#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize, Deserialize, Hash)] #[cfg_attr( any(test, feature = "proptest-impl"), derive(proptest_derive::Arbitrary) diff --git a/zebra-chain/src/sapling/tree.rs b/zebra-chain/src/sapling/tree.rs index 8fa44cbd..1190f6fe 100644 --- a/zebra-chain/src/sapling/tree.rs +++ b/zebra-chain/src/sapling/tree.rs @@ -62,7 +62,7 @@ struct SaplingNoteCommitmentTree; /// commitment tree corresponding to the final Sapling treestate of /// this block. A root of a note commitment tree is associated with /// each treestate. -#[derive(Clone, Copy, Default, Eq, PartialEq, Serialize, Deserialize)] +#[derive(Clone, Copy, Default, Eq, PartialEq, Serialize, Deserialize, Hash)] #[cfg_attr(any(test, feature = "proptest-impl"), derive(Arbitrary))] pub struct Root(pub [u8; 32]); diff --git a/zebra-chain/src/sprout/note/nullifiers.rs b/zebra-chain/src/sprout/note/nullifiers.rs index 19fa3299..e2365b35 100644 --- a/zebra-chain/src/sprout/note/nullifiers.rs +++ b/zebra-chain/src/sprout/note/nullifiers.rs @@ -62,7 +62,7 @@ impl From for [u8; 32] { } /// A Nullifier for Sprout transactions -#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize, Hash)] #[cfg_attr( any(test, feature = "proptest-impl"), derive(proptest_derive::Arbitrary) diff --git a/zebra-chain/src/sprout/tree.rs b/zebra-chain/src/sprout/tree.rs index 9cccd7e2..220e48ee 100644 --- a/zebra-chain/src/sprout/tree.rs +++ b/zebra-chain/src/sprout/tree.rs @@ -22,7 +22,7 @@ use proptest_derive::Arbitrary; /// commitment tree corresponding to the final Sprout treestate of /// this block. A root of a note commitment tree is associated with /// each treestate. -#[derive(Clone, Copy, Default, Eq, PartialEq, Serialize, Deserialize)] +#[derive(Clone, Copy, Default, Eq, PartialEq, Serialize, Deserialize, Hash)] #[cfg_attr(any(test, feature = "proptest-impl"), derive(Arbitrary))] pub struct Root([u8; 32]); diff --git a/zebra-chain/src/transaction.rs b/zebra-chain/src/transaction.rs index 6e10aedc..22b5d400 100644 --- a/zebra-chain/src/transaction.rs +++ b/zebra-chain/src/transaction.rs @@ -93,10 +93,10 @@ pub enum Transaction { expiry_height: block::Height, /// The net value of Sapling spend transfers minus output transfers. value_balance: Amount, - /// The shielded data for this transaction, if any. - shielded_data: Option, /// The JoinSplit data for this transaction, if any. joinsplit_data: Option>, + /// The shielded data for this transaction, if any. + shielded_data: Option, }, } diff --git a/zebra-chain/src/transparent.rs b/zebra-chain/src/transparent.rs index d2f9e6c6..c764da4f 100644 --- a/zebra-chain/src/transparent.rs +++ b/zebra-chain/src/transparent.rs @@ -40,7 +40,7 @@ impl AsRef<[u8]> for CoinbaseData { /// OutPoint /// /// A particular transaction output reference. -#[derive(Copy, Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[derive(Copy, Clone, Debug, Eq, PartialEq, Serialize, Deserialize, Hash)] #[cfg_attr(any(test, feature = "proptest-impl"), derive(Arbitrary))] pub struct OutPoint { /// References the transaction that contains the UTXO being spent. @@ -86,7 +86,7 @@ pub enum Input { /// I only own one UTXO worth 2 ZEC, I would construct a transaction /// that spends my UTXO and sends 1 ZEC to you and 1 ZEC back to me /// (just like receiving change). -#[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize, Hash)] #[cfg_attr(any(test, feature = "proptest-impl"), derive(Arbitrary))] pub struct Output { /// Transaction value. diff --git a/zebra-chain/src/transparent/script.rs b/zebra-chain/src/transparent/script.rs index 9f9572ef..1e92bda7 100644 --- a/zebra-chain/src/transparent/script.rs +++ b/zebra-chain/src/transparent/script.rs @@ -8,7 +8,7 @@ use std::{ }; /// An encoding of a Bitcoin script. -#[derive(Clone, Eq, PartialEq, Serialize, Deserialize)] +#[derive(Clone, Eq, PartialEq, Serialize, Deserialize, Hash)] #[cfg_attr( any(test, feature = "proptest-impl"), derive(proptest_derive::Arbitrary) diff --git a/zebra-chain/src/work/difficulty.rs b/zebra-chain/src/work/difficulty.rs index 80ecfa11..73589bae 100644 --- a/zebra-chain/src/work/difficulty.rs +++ b/zebra-chain/src/work/difficulty.rs @@ -320,3 +320,44 @@ impl AddAssign for Work { *self = *self + rhs; } } + +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, PartialOrd, Ord)] +/// Partial work used to track relative work in non-finalized chains +pub struct PartialCumulativeWork(u128); + +impl std::ops::Add for PartialCumulativeWork { + type Output = PartialCumulativeWork; + + fn add(self, rhs: Work) -> Self::Output { + let result = self + .0 + .checked_add(rhs.0) + .expect("Work values do not overflow"); + + PartialCumulativeWork(result) + } +} + +impl std::ops::AddAssign for PartialCumulativeWork { + fn add_assign(&mut self, rhs: Work) { + *self = *self + rhs; + } +} + +impl std::ops::Sub for PartialCumulativeWork { + type Output = PartialCumulativeWork; + + fn sub(self, rhs: Work) -> Self::Output { + let result = self.0 + .checked_sub(rhs.0) + .expect("PartialCumulativeWork values do not underflow: all subtracted Work values must have been previously added to the PartialCumulativeWork"); + + PartialCumulativeWork(result) + } +} + +impl std::ops::SubAssign for PartialCumulativeWork { + fn sub_assign(&mut self, rhs: Work) { + *self = *self - rhs; + } +} diff --git a/zebra-state/src/lib.rs b/zebra-state/src/lib.rs index ebcea597..682fdd79 100644 --- a/zebra-state/src/lib.rs +++ b/zebra-state/src/lib.rs @@ -19,9 +19,9 @@ mod util; #[cfg(test)] mod tests; -use memory_state::MemoryState; +use memory_state::NonFinalizedState; use service::QueuedBlock; -use sled_state::SledState; +use sled_state::FinalizedState; pub use config::Config; pub use request::{HashOrHeight, Request}; diff --git a/zebra-state/src/memory_state.rs b/zebra-state/src/memory_state.rs index d95e8d2d..41b75402 100644 --- a/zebra-state/src/memory_state.rs +++ b/zebra-state/src/memory_state.rs @@ -1,3 +1,461 @@ -pub struct MemoryState { - // TODO +//! Non-finalized chain state management as defined by [RFC0005] +//! +//! [RFC0005]: https://zebra.zfnd.org/dev/rfcs/0005-state-updates.html +#![allow(dead_code)] +use std::{ + cmp::Ordering, + collections::BTreeSet, + collections::{BTreeMap, HashMap, HashSet}, + ops::Deref, + sync::Arc, +}; + +use zebra_chain::{ + block::{self, Block}, + primitives::Groth16Proof, + sapling, sprout, transaction, transparent, + work::difficulty::PartialCumulativeWork, +}; + +use crate::service::QueuedBlock; + +/// The state of the chains in memory, incuding queued blocks. +#[derive(Debug, Default)] +pub struct NonFinalizedState { + /// Verified, non-finalized chains. + chain_set: BTreeSet, + /// Blocks awaiting their parent blocks for contextual verification. + contextual_queue: QueuedBlocks, +} + +/// A queue of blocks, awaiting the arrival of parent blocks. +#[derive(Debug, Default)] +struct QueuedBlocks { + /// Blocks awaiting their parent blocks for contextual verification. + blocks: HashMap, + /// Hashes from `queued_blocks`, indexed by parent hash. + by_parent: HashMap>, + /// Hashes from `queued_blocks`, indexed by block height. + by_height: BTreeMap>, +} + +impl NonFinalizedState { + pub fn finalize(&mut self) -> Arc { + todo!() + } + + pub fn queue(&mut self, _block: QueuedBlock) { + todo!() + } + + fn process_queued(&mut self, _new_parent: block::Hash) { + todo!() + } + + fn commit_block(&mut self, _block: QueuedBlock) -> Option { + todo!() + } +} + +#[derive(Debug, Default, Clone)] +struct Chain { + blocks: BTreeMap>, + height_by_hash: HashMap, + tx_by_hash: HashMap, + + created_utxos: HashSet, + spent_utxos: HashSet, + sprout_anchors: HashSet, + sapling_anchors: HashSet, + sprout_nullifiers: HashSet, + sapling_nullifiers: HashSet, + partial_cumulative_work: PartialCumulativeWork, +} + +impl Chain { + /// Push a contextually valid non-finalized block into a chain as the new tip. + pub fn push(&mut self, block: Arc) { + let block_height = block + .coinbase_height() + .expect("valid non-finalized blocks have a coinbase height"); + + // update cumulative data members + self.update_chain_state_with(&block); + self.blocks.insert(block_height, block); + } + + /// Remove the lowest height block of the non-finalized portion of a chain. + pub fn pop_root(&mut self) -> Arc { + let block_height = self.lowest_height(); + + // remove the lowest height block from self.blocks + let block = self + .blocks + .remove(&block_height) + .expect("only called while blocks is populated"); + + // update cumulative data members + self.revert_chain_state_with(&block); + + // return the block + block + } + + fn lowest_height(&self) -> block::Height { + self.blocks + .keys() + .next() + .cloned() + .expect("only called while blocks is populated") + } + + /// Fork a chain at the block with the givin hash, if it is part of this + /// chain. + pub fn fork(&self, fork_tip: block::Hash) -> Option { + if !self.height_by_hash.contains_key(&fork_tip) { + return None; + } + + let mut forked = self.clone(); + + while forked.non_finalized_tip_hash() != fork_tip { + forked.pop_tip(); + } + + Some(forked) + } + + fn non_finalized_tip_hash(&self) -> block::Hash { + self.blocks + .values() + .next_back() + .expect("only called while blocks is populated") + .hash() + } + + /// Remove the highest height block of the non-finalized portion of a chain. + fn pop_tip(&mut self) { + let block_height = self.non_finalized_tip_height(); + + let block = self + .blocks + .remove(&block_height) + .expect("only called while blocks is populated"); + + assert!( + !self.blocks.is_empty(), + "Non-finalized chains must have at least one block to be valid" + ); + + self.revert_chain_state_with(&block); + } + + fn non_finalized_tip_height(&self) -> block::Height { + *self + .blocks + .keys() + .next_back() + .expect("only called while blocks is populated") + } +} + +/// Helper trait to organize inverse operations done on the `Chain` type. Used to +/// overload the `update_chain_state_with` and `revert_chain_state_with` methods +/// based on the type of the argument. +/// +/// This trait was motivated by the length of the `push` and `pop_root` functions +/// and fear that it would be easy to introduce bugs when updating them unless +/// the code was reorganized to keep related operations adjacent to eachother. +trait UpdateWith { + /// Update `Chain` cumulative data members to add data that are derived from + /// `T` + fn update_chain_state_with(&mut self, _: &T); + + /// Update `Chain` cumulative data members to remove data that are derived + /// from `T` + fn revert_chain_state_with(&mut self, _: &T); +} + +impl UpdateWith> for Chain { + fn update_chain_state_with(&mut self, block: &Arc) { + let block_height = block + .coinbase_height() + .expect("valid non-finalized blocks have a coinbase height"); + let block_hash = block.hash(); + + // add hash to height_by_hash + let prior_height = self.height_by_hash.insert(block_hash, block_height); + assert!( + prior_height.is_none(), + "block heights must be unique within a single chain" + ); + + // add work to partial cumulative work + let block_work = block + .header + .difficulty_threshold + .to_work() + .expect("work has already been validated"); + self.partial_cumulative_work += block_work; + + // for each transaction in block + for (transaction_index, transaction) in block.transactions.iter().enumerate() { + let (inputs, outputs, shielded_data, joinsplit_data) = match transaction.deref() { + transaction::Transaction::V4 { + inputs, + outputs, + shielded_data, + joinsplit_data, + .. + } => (inputs, outputs, shielded_data, joinsplit_data), + _ => unreachable!( + "older transaction versions only exist in finalized blocks pre sapling", + ), + }; + + // add key `transaction.hash` and value `(height, tx_index)` to `tx_by_hash` + let transaction_hash = transaction.hash(); + let prior_pair = self + .tx_by_hash + .insert(transaction_hash, (block_height, transaction_index)); + assert!( + prior_pair.is_none(), + "transactions must be unique within a single chain" + ); + + // add the utxos this produced + self.update_chain_state_with(&(transaction_hash, outputs)); + // add the utxos this consumed + self.update_chain_state_with(inputs); + // add sprout anchor and nullifiers + self.update_chain_state_with(joinsplit_data); + // add sapling anchor and nullifier + self.update_chain_state_with(shielded_data); + } + } + + fn revert_chain_state_with(&mut self, block: &Arc) { + let block_hash = block.hash(); + + // remove the blocks hash from `height_by_hash` + assert!( + self.height_by_hash.remove(&block_hash).is_some(), + "hash must be present if block was" + ); + + // remove work from partial_cumulative_work + let block_work = block + .header + .difficulty_threshold + .to_work() + .expect("work has already been validated"); + self.partial_cumulative_work -= block_work; + + // for each transaction in block + for transaction in &block.transactions { + let (inputs, outputs, shielded_data, joinsplit_data) = match transaction.deref() { + transaction::Transaction::V4 { + inputs, + outputs, + shielded_data, + joinsplit_data, + .. + } => (inputs, outputs, shielded_data, joinsplit_data), + _ => unreachable!( + "older transaction versions only exist in finalized blocks pre sapling", + ), + }; + + // remove `transaction.hash` from `tx_by_hash` + let transaction_hash = transaction.hash(); + assert!( + self.tx_by_hash.remove(&transaction_hash).is_some(), + "transactions must be present if block was" + ); + + // remove the utxos this produced + self.revert_chain_state_with(&(transaction_hash, outputs)); + // remove the utxos this consumed + self.revert_chain_state_with(inputs); + // remove sprout anchor and nullifiers + self.revert_chain_state_with(joinsplit_data); + // remove sapling anchor and nullfier + self.revert_chain_state_with(shielded_data); + } + } +} + +impl UpdateWith<(transaction::Hash, &Vec)> for Chain { + fn update_chain_state_with( + &mut self, + (transaction_hash, outputs): &(transaction::Hash, &Vec), + ) { + for (utxo_index, _) in outputs.iter().enumerate() { + self.created_utxos.insert(transparent::OutPoint { + hash: *transaction_hash, + index: utxo_index as u32, + }); + } + } + + fn revert_chain_state_with( + &mut self, + (transaction_hash, outputs): &(transaction::Hash, &Vec), + ) { + for (utxo_index, _) in outputs.iter().enumerate() { + assert!( + self.created_utxos.remove(&transparent::OutPoint { + hash: *transaction_hash, + index: utxo_index as u32, + }), + "created_utxos must be present if block was" + ); + } + } +} + +impl UpdateWith> for Chain { + fn update_chain_state_with(&mut self, inputs: &Vec) { + for consumed_utxo in inputs { + match consumed_utxo { + transparent::Input::PrevOut { outpoint, .. } => { + self.spent_utxos.insert(*outpoint); + } + transparent::Input::Coinbase { .. } => {} + } + } + } + + fn revert_chain_state_with(&mut self, inputs: &Vec) { + for consumed_utxo in inputs { + match consumed_utxo { + transparent::Input::PrevOut { outpoint, .. } => { + assert!( + self.spent_utxos.remove(outpoint), + "spent_utxos must be present if block was" + ); + } + transparent::Input::Coinbase { .. } => {} + } + } + } +} + +impl UpdateWith>> for Chain { + fn update_chain_state_with( + &mut self, + joinsplit_data: &Option>, + ) { + if let Some(joinsplit_data) = joinsplit_data { + for sprout::JoinSplit { + anchor, nullifiers, .. + } in joinsplit_data.joinsplits() + { + self.sprout_anchors.insert(*anchor); + self.sprout_nullifiers.insert(nullifiers[0]); + self.sprout_nullifiers.insert(nullifiers[1]); + } + } + } + + fn revert_chain_state_with( + &mut self, + joinsplit_data: &Option>, + ) { + if let Some(joinsplit_data) = joinsplit_data { + for sprout::JoinSplit { + anchor, nullifiers, .. + } in joinsplit_data.joinsplits() + { + assert!( + self.sprout_anchors.remove(anchor), + "anchor must be present if block was" + ); + assert!( + self.sprout_nullifiers.remove(&nullifiers[0]), + "nullifiers must be present if block was" + ); + assert!( + self.sprout_nullifiers.remove(&nullifiers[1]), + "nullifiers must be present if block was" + ); + } + } + } +} + +impl UpdateWith> for Chain { + fn update_chain_state_with(&mut self, shielded_data: &Option) { + if let Some(shielded_data) = shielded_data { + for sapling::Spend { + anchor, nullifier, .. + } in shielded_data.spends() + { + self.sapling_anchors.insert(*anchor); + self.sapling_nullifiers.insert(*nullifier); + } + } + } + + fn revert_chain_state_with(&mut self, shielded_data: &Option) { + if let Some(shielded_data) = shielded_data { + for sapling::Spend { + anchor, nullifier, .. + } in shielded_data.spends() + { + assert!( + self.sapling_anchors.remove(anchor), + "anchor must be present if block was" + ); + assert!( + self.sapling_nullifiers.remove(nullifier), + "nullifier must be present if block was" + ); + } + } + } +} + +impl PartialEq for Chain { + fn eq(&self, other: &Self) -> bool { + self.partial_cmp(other) == Some(Ordering::Equal) + } +} + +impl Eq for Chain {} + +impl PartialOrd for Chain { + fn partial_cmp(&self, other: &Self) -> Option { + Some(self.cmp(other)) + } +} + +impl Ord for Chain { + fn cmp(&self, other: &Self) -> Ordering { + if self.partial_cumulative_work != other.partial_cumulative_work { + self.partial_cumulative_work + .cmp(&other.partial_cumulative_work) + } else { + let self_hash = self + .blocks + .values() + .last() + .expect("always at least 1 element") + .hash(); + + let other_hash = other + .blocks + .values() + .last() + .expect("always at least 1 element") + .hash(); + + // This comparison is a tie-breaker within the local node, so it does not need to + // be consistent with the ordering on `ExpandedDifficulty` and `block::Hash`. + match self_hash.0.cmp(&other_hash.0) { + Ordering::Equal => unreachable!("Chain tip block hashes are always unique"), + ordering => ordering, + } + } + } } diff --git a/zebra-state/src/service.rs b/zebra-state/src/service.rs index 7c14ad68..2dfd664f 100644 --- a/zebra-state/src/service.rs +++ b/zebra-state/src/service.rs @@ -13,9 +13,10 @@ use zebra_chain::{ parameters::Network, }; -use crate::{BoxError, Config, MemoryState, Request, Response, SledState}; +use crate::{BoxError, Config, FinalizedState, NonFinalizedState, Request, Response}; // todo: put this somewhere +#[derive(Debug)] pub struct QueuedBlock { pub block: Arc, // TODO: add these parameters when we can compute anchors. @@ -26,15 +27,15 @@ pub struct QueuedBlock { struct StateService { /// Holds data relating to finalized chain state. - sled: SledState, + sled: FinalizedState, /// Holds data relating to non-finalized chain state. - _mem: MemoryState, + _mem: NonFinalizedState, } impl StateService { pub fn new(config: Config, network: Network) -> Self { - let sled = SledState::new(&config, network); - let _mem = MemoryState {}; + let sled = FinalizedState::new(&config, network); + let _mem = NonFinalizedState::default(); Self { sled, _mem } } } diff --git a/zebra-state/src/sled_state.rs b/zebra-state/src/sled_state.rs index 5a0fe928..665452dc 100644 --- a/zebra-state/src/sled_state.rs +++ b/zebra-state/src/sled_state.rs @@ -18,16 +18,16 @@ use crate::{BoxError, Config, HashOrHeight, QueuedBlock}; /// - *asynchronous* methods that perform reads. /// /// For more on this distinction, see RFC5. The synchronous methods are -/// implemented as ordinary methods on the [`SledState`]. The asynchronous +/// implemented as ordinary methods on the [`FinalizedState`]. The asynchronous /// methods are not implemented using `async fn`, but using normal methods that /// return `impl Future`. This allows them to move data (e.g., /// clones of handles for [`sled::Tree`]s) into the futures they return. /// /// This means that the returned futures have a `'static` lifetime and don't -/// borrow any resources from the [`SledState`], and the actual database work is +/// borrow any resources from the [`FinalizedState`], and the actual database work is /// performed asynchronously when the returned future is polled, not while it is /// created. This is analogous to the way [`tower::Service::call`] works. -pub struct SledState { +pub struct FinalizedState { /// Queued blocks that arrived out of order, indexed by their parent block hash. queued_by_prev_hash: HashMap, @@ -42,7 +42,7 @@ pub struct SledState { // sapling_anchors: sled::Tree, } -impl SledState { +impl FinalizedState { pub fn new(config: &Config, network: Network) -> Self { let db = config.sled_config(network).open().unwrap(); @@ -86,7 +86,7 @@ impl SledState { /// It's the caller's responsibility to ensure that blocks are committed in /// order. This function is called by [`process_queue`], which ensures order. /// It is intentionally not exposed as part of the public API of the - /// [`SledState`]. + /// [`FinalizedState`]. fn commit_finalized(&mut self, queued_block: QueuedBlock) { let QueuedBlock { block, rsp_tx } = queued_block;