* Add a new block commit task and channels, that don't do anything yet
* Add last_block_hash_sent to the state service, to avoid database accesses
* Update last_block_hash_sent regardless of commit errors
* Rename a field to StateService.max_queued_finalized_height
* Commit finalized blocks to the state in a separate task
* Check for panics in the block write task
* Wait for the block commit task in tests, and check for errors
* Always run a proptest that sleeps once
* Add extra debugging to state shutdowns
* Work around a RocksDB shutdown bug
* Close the finalized block channel when we're finished with it
* Only reset state queue once per error
* Update some TODOs
* Add a module doc comment
* Drop channels and check for closed channels in the block commit task
* Close state channels and tasks on drop
* Remove some duplicate fields across StateService and ReadStateService
* Try tweaking the shutdown steps
* Update and clarify some comments
* Clarify another comment
* Don't try to cancel RocksDB background work on drop
* Fix up some comments
* Remove some duplicate code
* Remove redundant workarounds for shutdown issues
* Remode a redundant channel close in the block commit task
* Remove a mistaken `!force` shutdown condition
* Remove duplicate force-shutdown code and explain it better
* Improve RPC error logging
* Wait for chain tip updates in the RPC tests
* Wait 2 seconds for chain tip updates before skipping them
* Remove an unnecessary block_in_place()
* Fix some test error messages that were changed by earlier fixes
* Expand some comments, fix typos
Co-authored-by: Marek <mail@marek.onl>
* Actually drop children of failed blocks
* Explain why we drop descendants of failed blocks
* Clarify a comment
* Wait for chain tip updates in a failing test on macOS
* Clean duplicate finalized blocks when the non-finalized state activates
* Send an error when receiving a duplicate finalized block
* Update checkpoint block behaviour, document its consensus rule
* Wait for chain tip changes in inbound_block_height_lookahead_limit test
* Wait for the genesis block to commit in the fake peer set mempool tests
* Disable unreliable mempool verification check in the send transaction test
* Appease rustfmt
* Use clear_finalized_block_queue() everywhere that blocks are dropped
* Document how Finalized and NonFinalized clones are different
* Use the same check as commit_finalized() for finalized block heights
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Marek <mail@marek.onl>
* Move the finalized block queue into the StateService
* Move the queued_blocks module to the state service
* Move QueuedFinalized into queued_blocks
* Move the queued_blocks tests into their own module
* Make the FinalizedState cloneable
* Move AwaitUtxos next to the other shared writeable state requests
* Rename ReadResponse::Utxos to ReadResponse::AddressUtxos
```sh
fastmod Utxos AddressUtxos zebra*
```
* Rename an out_point variable to outpoint for consistency
* Rename transparent_utxos to address_utxos
```sh
fastmod transparent_utxos address_utxos zebra*
```
* Run AwaitUtxo without accessing shared mutable chain state
* Fix some incorrect comments
* Explain why some concurrent reads are ok
* Add a TODO
* Stop using self.mem in AwaitUtxo requests
* Update state service module documentation
* Move the QueuedBlock type into the queued_blocks module
* Explain how spent UTXOs are treated by the state
* Clarify how cached Chains impact state read requests
And move repeated comments to the module header.
* fastmod ChainUtxo BestChainUtxo zebra*
* Add an AnyChainUtxo request
* Make AwaitUtxo non-blocking
* Add TODOs for finalized blocks, non-finalized blocks, and reads
* Document how state requests use shared state
* Add a ReadStateService to the StateService
And cleanup service struct fields.
* Redirect Block and Transaction Requests to the ReadStateService
* Put AddressBalance in a consistent enum order
* Turn a repeated comment into documentation
* Tidy doc links
* Run Tip requests concurrently
* Remove some redundant timers
* Run Depth requests concurrently
* Run BlockLocator requests concurrently
* Move BlockLocator tests
* Run FindBlockHashes requests concurrently
* Run FindBlockHeaders requests concurrently
* Use a constant in documentation
Co-authored-by: Marek <mail@marek.onl>
* Link that constant correctly
* Expand block_locator() documentation
* Clarify the difference between tower::Buffers and the state's ordered queues
* Explain block locators better
Co-authored-by: Marek <mail@marek.onl>
* Add history trees for each height in non-fin state
* Refactor formatting
* Pass the treestate to the finalized state
I created a new structure `FinalizedBlockWithTrees` that wraps the
treestate and the finalized block. I did that because the original
`FinalizedBlock` is `Eq`, but `HistoryTree` can't be `Eq`.
This makes Zebra faster because:
1. The finalized state doesn't retrieve the treestate from the disk if
the non-finalized state supplies it.
2.The finalized state doesn't recompute the treestate if the
non-finalized state supplies it.
* Check block commitment before updating hist tree
* Store Sprout commitment trees in non-fin state
* Send trees for the root block to fin-state
When committing a block and sending the treestate from the non-finalized
state to the finalized state, Zebra was sending trees that correspond to
the tip block instead of trees that correspond to the root block of the
best chain. This commit fixes that.
* Refactor doc comments
* Refactor block finalization
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* cargo upgrade --workspace rocksdb
* Add a deny.toml exception for bindgen 0.59
* Move `valid()` and `status()` methods to raw iterators
* Update some outdated comments
* Panic on iterator failures
We might want to change this if there are any common failure modes.
* allow(clippy::unwrap_in_result) in some methods
* Rename a function to prepare_block_header_and_transaction_data_batch()
* Fix formatting of test command timeouts and child process output
* Put some #[cfg()]s in the standard Rust location
* Update some test timings
* Allow code timers to be ignored
* Revert: deserialize stored transactions in a rayon thread
* Add a TODO for the reverted bug fix
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Split disk reads from CPU-heavy Sprout interstitial tree cryptography
* Improve anchor validation debugging and error messages
* Work around a test data bug, and save some CPU
* Remove redundant checks for empty shielded data
* Skip generating unused interstitial treestates
* Do disk fetches and quick checks, then CPU-heavy cryptography
* Wrap HistoryTree in an Arc in the state
* Run CPU-intensive chain validation and updates in parallel rayon threads
* Refactor to prepare for parallel tree root calculations
* Run finalized state note commitment tree root updates in parallel rayon threads
* Update finalized state note commitment trees using parallel rayon threads
* Fix a comment typo and add a TODO
* Split sprout treestate fetch into its own function
* Move parallel note commitment trees to zebra-chain
* Re-calculate the tree roots in the same parallel batches
* Do non-finalized note commitment tree updates in parallel threads
* Update comments about note commitment tree rebuilds
* Do post-fork tree updates in parallel threads
* Add a TODO for parallel tree updates in tests
* Fix broken intra-doc links
* Clarify documentation for sprout treestates
* Spawn large database reads into blocking tokio threads
* Concurrently read all blocks, headers, and transactions from disk
* Run zebra-state transaction deserialization on a rayon thread
* Split disk reads from CPU-heavy Sprout interstitial tree cryptography
* Improve anchor validation debugging and error messages
* Work around a test data bug, and save some CPU
* Remove redundant checks for empty shielded data
* Skip generating unused interstitial treestates
* Do disk fetches and quick checks, then CPU-heavy cryptography
* Wrap HistoryTree in an Arc in the state
* Run CPU-intensive chain validation and updates in parallel rayon threads
* Refactor to prepare for parallel tree root calculations
* Run finalized state note commitment tree root updates in parallel rayon threads
* Update finalized state note commitment trees using parallel rayon threads
* Fix a comment typo and add a TODO
* Split sprout treestate fetch into its own function
* Move parallel note commitment trees to zebra-chain
* Re-calculate the tree roots in the same parallel batches
* Do non-finalized note commitment tree updates in parallel threads
* Update comments about note commitment tree rebuilds
* Do post-fork tree updates in parallel threads
* Add a TODO for parallel tree updates in tests
* Fix broken intra-doc links
* Clarify documentation for sprout treestates
* Sort Cargo.toml dependencies
* Only fetch block headers from the database to answer headers requests
* Move writing to the database to a blocking thread
* Add blocking threads to tests that need them
* Remove mempool downloader requirement for a populated state
And stop populating states that don't need it,
so we can use tokio::time::pause() in those tests.
* Improve debugging for an intermittent test failure
* Try to avoid a race condition populating the mempool in tests
* Wrap Sprout note commitment trees into `Arc`
* Remove a redundant comment
* Rephrase a comment about chain forking
* Remove a redundant comment
The comment is not valid because Zebra uses `bridgetree::Frontier`s from
the `incrementalmerkletree` crate to represent its note commitment
trees. This `struct` does not support popping elements from the tree.
* Wrap Sapling commitment trees into `Arc`
* Remove unnecessary `as_ref`s
* Wrap Orchard commitment trees into `Arc`
* Rename `block_by_height` to `block_header_by_height` in fin state
* Rename `tx_by_hash` to `tx_loc_by_hash` in both (non & fin) states
* Rename `utxo_by_outpoint` to `utxo_by_out_loc` in finalized state
* Reorder the column families so that they match the docs
* Update `struct Chain` in the RFCs
* Increment `DATABASE_FORMAT_VERSION` to 25
* Remove obsolete docs from `0004-asynchronous-script-verification.md`
* Remove an obsolete `TODO` from `disk_db.rs`
* Delete unused snapshots
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Fix significant drop in match scrutinee
https://github.com/rust-lang/rust/issues/93883
* Fix deref immutable value
* Fix explicit 0 index when first() would do
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Fix the syntax of links in comments
* Fix a mistake in the docs
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Remove unnecessary angle brackets from a link
* Revert the changes for links that serve as references
* Revert "Revert the changes for links that serve as references"
This reverts commit 8b091aa9fab453e7d3559a5d474e0879183b9bfb.
* Remove `<` `>` from links that serve as references
This reverts commit 046ef25620ae1a2140760ae7ea379deecb4b583c.
* Don't use `<` `>` in normal comments
* Don't use `<` `>` for normal comments
* Revert changes for comments starting with `//`
* Fix some warnings produced by `cargo doc`
* Fix some rustdoc warnings
* Fix some warnings
* Refactor some changes
* Fix some rustdoc warnings
* Fix some rustdoc warnings
* Resolve various TODOs
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Upgrade tracing and related dependencies
```sh
cargo upgrade --workspace
tracing-error
tracing-subscrber
color-eyre
tracing-flame
tracing-journald
sentry
sentry-tracing
metrics
metrics-exporter-prometheus
reqwest
```
* Update duplicate dependency checks
* Enable the tracing/env-filter feature
* Fix type inference for metrics
Manual changes, plus:
```sh
fastmod "as _" "as f64"
```
* Tidy up some unrelated test code
* Update metrics-exporter-prometheus API
And make unused dependencies optional.
* Adjust test regexes to new tracing format
Also fix some regex bugs, and refactor to simplify.
* Disable color-eyre span traces and track caller in release builds
* Add a feature that enables extra debugging in release builds
* Clean up some redundant features
* Increase a test timeout
* Fix the syntax of links in comments
* Fix a mistake in the docs
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Remove unnecessary angle brackets from a link
* Revert the changes for links that serve as references
* Revert "Revert the changes for links that serve as references"
This reverts commit 8b091aa9fab453e7d3559a5d474e0879183b9bfb.
* Remove `<` `>` from links that serve as references
This reverts commit 046ef25620ae1a2140760ae7ea379deecb4b583c.
* Don't use `<` `>` in normal comments
* Don't use `<` `>` for normal comments
* Revert changes for comments starting with `//`
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Make jobs that use cached state wait for state rebuilds
* Run jobs that need cached state even if the rebuild was skipped
* Fix missing dependencies
And update a TODO
* Split writing transaction indexes into transparent and shielded
* Split writing transparent indexes into created and spent
* Correctly populate spending address transaction ID indexes
* Increment the database format to rebuild address tx ID indexes
* Update non-finalized docs to prevent similar bugs
* Fix a comment
* Make jobs that use cached state wait for state rebuilds
* Run jobs that need cached state even if the rebuild was skipped
* Fix missing dependencies
And update a TODO
* refactor(ci): look for available disks instead of files changed
This ensure that if the constants.rs file was changed, we search for disks available in the whole repository with the same state.
If there's no disk available a rebuild is triggered depending the missing disk. And if there's a disk available, tests are run with this one.
* fix(ci): lwd syncs needs to wait for zebra disk rebuild
* docs(ci): use better comments on integration tests
* fix(ci): we must authenticate to GCP to find disks
* fix(ci): add needed permissions for google auth
* fix(ci): the output needs to be echoed
* imp(ci): reduce diff with main
* fix(ci): remove redundant dependency
Co-authored-by: teor <teor@riseup.net>
* fix(ci): also add `false` to the JSON object output
* fix(ci): hasty copy/paste
* force a push event
* fix(ci): standardize comments
* fix(ci): run disk rebuilds if no disk was found
* fix(ci): do not restrict on push
* fix(ci): build on any event if a cached disk is not found
* fix(ci): sync .patch file with changes on the workflow
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* Impl the elementary structure of the `z_gettreestate` RPC
* Fix merging bugs
* Fix a merge bug
* Fix a merge bug
* Move a derive attribute
Co-authored-by: teor <teor@riseup.net>
* Clarify the support of negative heights
* Add Orchard note commitment trees to the response
* Add the time to the response
* Finalize the `z_gettreestate` RPC
* Add a note that verified blocks have coinbase height
* Refactor `from_str` for `HashOrHeight`
* Fix a mistake in the docs
Co-authored-by: teor <teor@riseup.net>
* Clarify request types
Co-authored-by: teor <teor@riseup.net>
* Simplify `hash_or_height` conversion to height
Co-authored-by: teor <teor@riseup.net>
* Add a TODO about optimization
Co-authored-by: teor <teor@riseup.net>
* Add a doc comment
* Make sure Sapling & Orchard trees don't get mixed up
* Serialize Sapling commitment trees
* Refactor some comments
* Serialize Orchard commitment trees
* Serialize block heights
* Simplify the serialization of commitment trees
* Remove the block time from the RPC response
* Simplify the serialization of block heights
* Put Sapling & Orchard requests together
* Remove a redundant TODO
* Add block times to the RPC response
* Derive `Clone, Debug, Eq, PartialEq` for `GetTreestate`
Co-authored-by: teor <teor@riseup.net>
* Derive `Clone`, `Debug`, `Eq` and `PartialEq` for `SerializedTree`
* Document the fields of `GetTreestate`
* Skip the serialization of empty trees
This ensures compatibility with `zcashd` in the `z_gettreestate` RPC.
* Document the `impl` of `merkle_tree::Hashable` for nodes
* Make the structure of the JSON response consistent with `zcashd`
* Derive `Eq` for nodes
Co-authored-by: teor <teor@riseup.net>
* Convert Sapling commitment trees to a format compatible with zcashd
* Refactor the conversion of Sapling commitment trees
* Refactor some comments
* Refactor comments
* Add a description of the conversion
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Fix comment indenting
* Document the conversion between the dense and sparse formats
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Add a finalized state txids query
* Add an address transaction IDs query, without height filters
* Connect the address transaction ID query to the RPC
* Basic filtering of address transaction IDs by height range
* Add a network and range argument to the getaddresstxids test
* Test all block range combinations for mainnet
* Fix a file descriptor limit error
* Optimise seeking the first transaction for an address
The first transaction's location is part of the address location.
* Filter finalized address transaction IDs by height range
* Filter non-finalized address transaction IDs by the height range
* Fix up snapshot tests for the new height range API
* Add address UTXOs query functions, but without the transaction IDs
* Return transaction IDs along with address UTXOs
* Add a convenience type for address UTXOs
* Add output addresses to the convenience method
* Fix query documentation
* Rename the chain transaction IDs method
* update librustzcash; adapt to new API
* add ticket reference for removing zcash_proofs duplicated dependencies
* update to new zcash_script V5 API
* use zp_tx shorthand
* update to Zcash 4.7.0 dependencies
* update protocol versions
* feat(rpc): Implement `getblockchaininfo` RPC method (#3891)
* Implement `getblockchaininfo` RPC method
* add a test for `get_blockchain_info`
* fix tohex/fromhex
* move comment
* Update lightwalletd acceptance test for getblockchaininfo RPC (#3914)
* change(rpc): Return getblockchaininfo network upgrades in height order (#3915)
* Update lightwalletd acceptance test for getblockchaininfo RPC
* Update some doc comments for network upgrades
* List network upgrades in order in the getblockchaininfo RPC
Also:
- Use a constant for the "missing consensus branch ID" RPC value
- Simplify fetching consensus branch IDs
- Make RPC type derives consistent
- Update RPC type documentation
* Make RPC type derives consistent
* Fix a confusing test comment
* get hashand height at the same time
* fix estimated_height
* fix lint
* add extra check
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* fix typo
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* split test
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* fix(rpc): ignore an expected error in the RPC acceptance tests (#3961)
* Add ignored regexes to test command failure regex methods
* Ignore empty chain error in getblockchaininfo
We expect this error when zebrad starts up with an empty state.
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Make sync error logs more user-friendly (#3944)
- use info level, there is nothing the user needs to do,
particularly for a single error
- explain that the errors are temporary
- hide backtraces, because they look like crashes
* Update test.patch.yml with lightwalletd job (#3970)
* Update test.patch.yml with lightwalletd job
* Remove a workflow condition that will always be false
In general, patch workflows need the
opposite conditions to the original workflow.
But in this case, we know the result of the
condition will always be true, so we can just delete it.
Co-authored-by: teor <teor@riseup.net>
* fix(doc): Fix bugs in the lightwalletd database design (#3964)
* Re-order column families in design in dependency order
* Minor RFC design tweaks and fixes
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Repoint zebra image links to our new zfnd.org site for now (#3949)
* Repoint zebra image links to our new zfnd.org site for now
* Remove images/
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Fix typos (#3956)
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* bump database version to trigger testnet rollback
* reduce minimum protocol version for now (will be changed later)
* update dependencies
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* update versions to match zcash 4.7.0
* deny.toml: update 'darling'
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
Co-authored-by: Dimitris Apostolou <dimitris.apostolou@icloud.com>
* Make address index types consistent
* Simplify non-finalized address index updates
* Update snapshots for address index queries
* Simplify non-finalized UTXO query
* Add a query method for non-finalized address balance changes
* Add a query method for finalized state address balances
* Add a query function for address balances
* Refactor balance queries to make them repeatable
* Retry interrupted finalized balance queries
* Pop chain root blocks until it matches the finalized tip
* Avoid cloning the chain
It has already been cloned by the watch receiver
* Refactor and fix documentation of the balance query code
* Add test-only serialization, and make existing serialization test-only
* Make AddressLocations clearer in the API
* Add UnspentOutputAddressLocation
* Add the AddressLocation to the UTXO database value
* Update the snapshot test code for UnspentOutputAddressLocation
* Update the raw data snapshots
* Update the high-level data snapshots
* Increment the database version
* Make serialization clearer
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Fix code formatting
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Add an empty utxo_by_transparent_addr_loc column family
* Update snapshot data for the new column family
* Add an AddressUnspentOutputs type
* Add round-trip tests for AddressUnspentOutputs
* Move address balances into their own method
* Simplify updating address balances
* Fix utxo_by_out_loc column family name
* Implement reads and writes of address UTXOs
* Update raw data snapshots
* Update the snapshot tests for high-level address UTXOs
* Assert rather than taking empty address snapshots for genesis
* Update high-level address UTXO snapshot data, and delete empty snapshots
* Increment the database version
* Use typed values for all ReadDisk methods
* Implement test-only serialization for transparent::Address
* Implement FromDisk for ()
* Store AddressUnspentOutput as the column family key
* Update round-trip serialization tests for AddressUnspentOutput
* Update snapshot test code, and add a UTXO data snapshot
* Update existing snapshot data
* Add new UTXO snapshot data
* Update column family name
```sh
fastmod utxo_by_transparent_addr_loc utxo_loc_by_transparent_addr_loc zebra*
```
* cargo fmt --all
* cargo insta test --review --delete-unreferenced-snapshots
* Explain why it is ok to use invalid database iterator indexes
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Add explanations of UTXO database updates
* Simplify an assertion
* Remove UnspentOutputAddressLocation and just store transparent::Output
* Update snapshot test data
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Derive Hash for transparent address index types
* Expose some types used by transparent address indexes
* Add an empty transparent transfers type for transparent address indexes
* Update TransparentTransfers with created UTXOs
* Add spent transparent outputs to ContextuallyValidBlock
* Update TransparentTransfers with spent transparent outputs
* Ignore missing spent outputs, so that tests pass
* Remove empty TransparentTransfers after a spend revert
* Update TransparentTransfers with creating and spending transaction IDs
* Ignore duplicate created UTXOs, so that tests pass
* Add some TODO comments
* Remove accidental doctest formatting
* Add address transfers index accessor methods
* Use TransactionLocation in the non-finalized state
* Apply more address index assertions to production code
* Refactor deeply nested code and apply more assertions
* Return UTXOs in chain order
* Return transaction hashes in chain order
* Stop indexing each transparent output multiple times
* Run some more asserts during tests
* Tidy TODO comments
* Fix an incorrect assert condition
* Use OrderedUtxos so that spent UTXOs can be stored in chain order
* Update tests to use OrderedUtxos
* Update the index API for the getaddressutxos query
* Remove redundant arguments in tests
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Add Utxo constructors from output locations
* Store transparent outputs rather than Utxo structs
* Update raw data snapshots
* Increment the state version
* Change OutputLocation to contain a TransactionLocation
* Change OutputLocation reads from the database
* Update some doc comments
* Update some TODOs
* Change deleting spent UTXOs and updating spent balances
* Change adding new UTXOs and adding their values to balances
* Disable dead code warnings
* Update snapshot test code
* Update round-trip tests for OutputLocations
* Update snapshot test data
* Increment the database format version
* Remove a redundant try_into()
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Refactor redundant code
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* ci: attempt at fixing 'Regenerate stateful disks'
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Add an empty balance_by_transparent_addr column family
* Add an AddressBalanceLocation type for balance_by_transparent_addr
* Add serialization for balance_by_transparent_addr types
* Add round-trip tests for the new serialized types
* Add missing round-trip and serialized equality tests
* Add a network field to DiskWriteBatch
* Refactor confusing all_utxos_spent_by_block argument
It was actually just the UTXOs from the state spent by the block,
excluding the UTXOs created and spent within the block.
But now we need it to contain all the spent outputs,
including the ones created by the block.
* Read and update address balances in the finalized state
* Update raw data snapshots for transparent address balances
* Add test-only deserialization for transparent addresses
* Add high-level snapshot test code for address balances
* Add high-level snapshots for address balances
* Increment the state version after NU5 testnet 2 rollback
* Implement disk serialization for block headers and transactions
* Re-order column family initialization to match the design
* Add new empty transaction column families
* Split writing block header and transaction data
* Re-order column families for consistency
* Update write snapshots for transaction split
* Use split block and transaction data when reading
* Update snapshots to include genesis transaction hash location
* Filter all prefix iterators to make sure they return the correct values
* Test that the new transaction indexes are consistent
* Add some cleanup TODOs
* Increment the database format to version 15
* Remove unused fisk format impls for Block
* Add a missing prefix extractor for transaction locations
* Make the database generic over the thread mode
* Replace prefix iteration with iteration from a key, and a filter
Prefix iteration caused database hangs.
* Manually iterate through transaction locations to re-create blocks
Also:
- re-write disk read API to avoid iterator hangs
- move disk read API to ReadDisk
- re-write impl rocksdb::AsColumnFamilyRef to a where clause, for consistency
* Update the database version so it's larger than the NU5 testnet 2 version
* Increment the database format version
* Update IntoDisk and FromDisk docs
* Rename fixed_byte_len to fixed_disk_byte_len
* Add functions that truncate and extend serialized bytes
* Store heights in 3 bytes on disk
* Update database raw data snapshots for 3-byte heights
* Log an error if we ever get close to the maximum disk height
* Store transaction indexes in 2 bytes on disk
* Update database raw data snapshots for 2-byte transaction indexes
* Make doc comment phrasing consistent
* Replace IntoDiskFixed with fixed constants
* Replace u32 byte length literal with a constant calculation
* Fix off-by-one error in MAX_ON_DISK_HEIGHT
* Add proptest seeds for the MAX_ON_DISK_HEIGHT off-by-one error
* Remove redundant module from a Height type
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Add round-trip test for OutputLocation disk serialization
* Make the transaction snapshot tests more accurate
Previously, we were showing the genesis transaction hash at height 0, index 0.
But its hash is actually not stored by location in the database,
because the genesis transaction is skipped due to a consensus rule.
* Update the transaction snapshot data
* Add history tree snapshot tests
At the current test heights, the history trees are all empty.
* Add the history tree snapshot data
* Update comments
* Simplify snapshot test code
* Make some serde::Serialize impls test-only, so we can hex-encode them
This should also speed up release compilation a bit.
* Add snapshot test code for UTXOs
* Add snapshot data for UTXOs
* Add request metrics to ReadStateService
Add a field to distinguish StateService and ReadStateService metrics.
Add missing StateService request metrics.
* Refactor state transaction lookup so it can be shared between services
* Implement ReadState Request::Transaction
* Revert "Drop the Chain watch channel lock before accessing the finalized state"
This reverts commit 8870944d1323fc31e2d009b9938623de6fcbe50f.
* Add a WatchReceiver wrapper that always clones the borrowed watch data
This avoids deadlocks, by holding the read lock for as short a time as possible.
* Drop the shared Arc<Chain>s as quickly as possible
This reduces memory usage.
* Make read::block more flexible, by accepting any AsRef<Chain>
* Make the block method docs consistent
* Avoid livelocks by explicitly dropping the borrow after the clone
* Query Sapling & Orchard trees by height in the finalized state
* Add Sapling & Orchard trees to the non-finalized state
* Add a TODO about concurrent read-only access to Sprout tree
Co-authored-by: teor <teor@riseup.net>
* Update the database format version
* Keep only the most recent Sprout tree in the database
* Check that the database returns empty trees for the genesis block
* Assert that the database returns the highest trees
* Document how to update insta snapshots
* Add note commitment tree insta snapshot tests
* Add comments about cached tree roots in snapshots
* Add snapshot data for sapling and orchard trees
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
* Add state service module docs and cleanup
* Move and add finalized state methods
* Add chain and non-finalized state methods
* Cleanup methods and imports
* Create a ReadStateService type
* Add a stub service implementation
* Add a TODO
* Update ReadStateService request stubs with RPC names and tickets
* Documentation updates
* Make RPC State generic bounds accept a buffered state and a read-only state
* Doc updates
* Add missing proptest-impl feature in RPC dev dependencies
* Move the legacy chain check to the `check` module
And move `populated_state` to the `arbitrary` module.
* Cleanup imports
* Document the state service struct
* Split state block iter into its own module
* Prepare the finalized state for read-only state
* Add a forced shutdown mode, used in test code before forced exits
* Document the small database drop race condition window
* refactor(test/block): rename large single transaction function
```sh
fastmod single_transaction_block single_transaction_block_many_inputs
```
* rustfmt
* test(block): add a test block with many transparent outputs
* doc(db): explain why we can't just get the UTXOs right before they are deleted
* refactor(db): split out a block data write method
* refactor(block): add a height argument to new_outputs
* test(db): add block and transaction round-trip tests
Including large blocks and transactions.
* test(db): fix large block serialization instability in the tests
* doc(block): add TODOs for generating correct blocks
* Make transparent output functions which take a height test-only
* make sure generated blocks are actually over/under-sized
* replace println!() with an error!() log
* refactor(db): simplify block height serialization
* refactor(db): make height serialization length generic
* refactor(db): create a TransactionIndex type
This changes the names of some snapshot types,
but doesn't change any data.
* refactor(db): create transparent OutputIndex and OutputLocation types
This keeps the same serialization, to avoid changing the database version.
* doc(rfc/db): make transparent database type names consistent
* doc(rfc/db): fix a bug in the Utxo.is_coinbase derivation
* fix(db): use the correct serialized size for OutputLocation
* doc(db): fix some comments
* refactor(db): split disk serialization types into their own module
* refactor(db): split the disk format into modules
* doc(db/test): explain the RON serialization format
* fix: check TEST_FAKE_ACTIVATION_HEIGHTS at runtime
* fix(tests): add TEST_FAKE_ACTIVATION_HEIGHTS variable
This variable ensures the test is activated in the `test-fake-activation-heights` step
* fix(docker): do not run specific tests by default in entrypoint.sh
* fix(test): remove extra TEST_FULL_SYNC argument
* imp(timeout): wait for an average build time
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* test(db): snapshot finalized tips
* test(db): snapshot block heights and hashes
* test(db): snapshot block data
* test(db): snapshot transaction hashes and data
* test(db): assert that blocks and transactions are in chain order
* clippy(db/test): cargo clippy --fix
* clippy(db/test): replace &Vec with &[_]
* refactor(state): split database access into modules by Zebra types
Also split the genesis block check from the genesis note commitment trees.
* test(db): snapshot column family names
* fix(db): assert that the default column family is empty on open and close
* feat(test): apply `cargo insta` settings to all tests
* doc(db/test): improve test docs
* test(db): snapshot column family data for the empty state
* refactor(db/test): split out the raw database snapshot
* test(db): snapshot raw database for blocks 0-2
* test(db): initial serialized snapshot data for raw database
* test(db): tweak snapshot file names
* test(db): rename snapshots for consistency
* test(db): store empty column families in a single snapshot
* test(db): simplify snapshot files by combining empty snapshots
* doc(db/test): put comment in a better place
* refactor(db): fastmod assert_default_is_empty assert_default_cf_is_empty
* doc(test): explain when insta settings are needed
* fix(state/test): use the network to initialize the state
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* fix(state): mark DiskWriteBatch as must_use
* doc(state): add TODOs for moving database reads to blocking threads
* doc(state): minor comment tweaks
* refactor(state): split write batch into block, transactions, chain history
* refactor(state): split out a genesis block write method
* refactor(state): just use the empty note commitment trees directly
* refactor(state): split transaction writes into transparent, nullifiers, trees
And change DiskWriteBatch methods to take `&mut self`.
* refactor(state): split chain value pool writes out of history writes
* refactor(state): combine note commitment trees into an agrument struct
* refactor(state): split history and note commitment updates
* refactor(state): calculate current tip height and remove that argument
* refactor(state): move disk_db reads to a new zebra_db module
* refactor(state): make finalized value pool method names consistent
* refactor(state): split database writes into the zebra_db module
* refactor(state): move the block batch method to DiskWriteBatch
* refactor(state): actually add the zebra_db module
Unfortunately, I've lost the interim changes to this file,
so this commit might be the only one that compiles.
* refactor(state): add a newly created file to the cached state CI job
* refactor(state): move RocksDB-specific initialization to a new module
* refactor(state): move RocksDB-specific shutdown to a new module
* refactor(state): temporarily allow RocksDB-specific reads and writes, without a new module
Unlike the last few commits, this one actually compiles.
* refactor(state): add a DiskWriteBatch wrapper for RocksDB writes
* refactor(state): move finalized state test methods to a test module
* refactor(state): split the disk_format module
* refactor(ci): add the new disk_db file to the state CI list
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Add validation of ZIP-221 and ZIP-244 commitments
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Add auth commitment check in the finalized state
* Reset the verifier when comitting to state fails
* Add explanation comment
* Add test with fake activation heights
* Add generate_valid_commitments flag
* Enable fake activation heights using env var instead of feature
* Also update initial_tip_hash; refactor into progress_from_tip()
* Improve comments
* Add fake activation heights test to CI
* Fix bug that caused commitment trees to not match when generating partial arbitrary chains
* Add ChainHistoryBlockTxAuthCommitmentHash::from_commitments to organize and deduplicate code
* Remove stale comment, improve readability
* Allow overriding with PROPTEST_CASES
* partial_chain_strategy(): don't update note commitment trees when not needed; add comment
Co-authored-by: teor <teor@riseup.net>
* Refactor HistoryTree into NonEmptyHistoryTree and HistoryTree
* HistoryTree: use Deref instead of AsRef; remove unneeded PartialEq
* ZIP-221: Validate chain history commitments in the non-finalized state (#2301)
* sketch of implementation
* refined implementation; still incomplete
* update librustzcash, change zcash_history to work with it
* simplified code per review; renamed MMR to HistoryTree
* expand HistoryTree implementation
* handle and propagate errors
* simplify check.rs tracing
* add suggested TODO
* add HistoryTree::prune
* fix bug in pruning
* fix compilation of tests; still need to make them pass
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* improvements from code review
* improve check.rs comments and variable names
* fix HistoryTree which should use BTreeMap and not HashMap; fix non_finalized_state prop tests
* fix finalized_state proptest
* fix non_finalized_state tests by setting the correct commitments
* renamed mmr.rs to history_tree.rs
* Add HistoryTree struct
* expand non_finalized_state protest
* fix typo
* Add HistoryTree struct
* Update zebra-chain/src/primitives/zcash_history.rs
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* fix formatting
* Apply suggestions from code review
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* history_tree.rs: fixes from code review
* fixes to work with updated HistoryTree
* Improvements from code review
* Add Debug implementations to allow comparing Chains with proptest_assert_eq
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Improvements from code review
* Restore blocks returned by PreparedChain since other tests broken; adjust tests with history trees
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* add serialization for value balances
* change test names
* change panic messages
* add a deserialization test
* return the errors from `from_bytes()` methods
* add prop test for serialize/deserialize Amount
* Move amount proptests to amount::tests::prop
Co-authored-by: teor <teor@riseup.net>
* Add ZIP-221 history tree to finalized state
* Improve error / panic handling; improve documentation
* Return error again when preparing batch, fix expect messages
* Fix bug when pushing the Heartwood actiation block to the history tree
* Re-increase database version since it was increased in main
Co-authored-by: teor <teor@riseup.net>
* Tidy chain Cargo.toml
* Organize imports
* Add method to get note commitments from all Actions in Orchard shielded data
* Add method to get note commitments from all JoinSplits in Sprout JoinSplitData
* Add Request and Response variants for awaiting anchors
* Add anchors and note commitment trees to finalized state db
* Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees
* Track anchors and note commitment trees in Chain
Append note commitments to their trees when doing update_chain_state_with,
then use the resulting Sapling and Orchard roots to pass to history_tree, and add
new roots to the anchor sets.
* Handle errors when appending to note commitment trees
* Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with
* Implementing note commitments in finalized state
* Finish serialization of Orchard tree; remove old tree when updating finalize state
* Add serialization and finalized state updates for Sprout and Sapling trees
* Partially handle trees in non-finalized state. Use Option for trees in Chain
* Rebuild trees when forking; change finalized state tree getters to not require height
* Pass empty trees to tests; use empty trees by default in Chain
* Also rebuild anchor sets when forking
* Use empty tree as default in finalized state tree getters (for now)
* Use HashMultiSet for anchors in order to make pop_root() work correctly
* Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS
* Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more
* Apply suggestions from code review
* Add comments about order of note commitments and related methods/fields
* Don't use Option for trees
* Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS
* Remove unneeded anchor set rebuilding in fork()
* Improve proptest formatting
* Add missing comparisons to eq_internal_state
* Renamed sprout::tree::NoteCommitmentTree::hash() to root()
* Improve comments
* Add asserts, add issues to TODOs
* Remove impl Default for Chain since it was only used by tests
* Improve documentation and assertions; add tree serialization tests
* Remove Sprout code, which will be moved to another branch
* Add todo! in Sprout tree append()
* Remove stub request, response *Anchor* handling for now
* Add test for validating Sapling note commitment tree using test blocks
* Increase database version (new columns added for note commitment trees and anchors)
* Update test to make sure the order of sapling_note_commitments() is being tested
* Improve comments and structure of the test
* Improve variable names again
* Rustfmt
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: teor <teor@riseup.net>
* Add an OrderedUtxo type for validation of spends within a block
This change allows us to check that transparent spends use outputs from
earlier in their block. (But we don't actually do that check yet.)
We need to keep the order of UTXOs when we're contextually verifying
each new block that is added to a chain. But the block order is
irrelevant for UTXOs stored in the state.
* Take ownership in utxos_from_ordered_utxos
* Delete a confusing comment
* Reject duplicate sprout nullifiers in the state
* Improve docs and error messages
* Clarify "must be present" assert logs
* Move nullifier checks to their own module
Also:
* make non-finalized nullifier checks and errors generic over
sprout, sapling, and orchard
* create and update module and function documentation
* Fix a block type name in docs
* Move state assertions or skip them during tests
These changes enable state testing, while still asserting in production.
* Add sprout duplicate nullifier tests
* Improve comments
* Set value balance to 0 to pass future chain value pool checks
* Test finalized state in sprout nullifier accept test
* Replace assert with expect
* Improve assertion messages
* Document the new genesis transaction consensus rule
Zebra previously implemented this rule, but we documented it as a bug in
`zcashd`.
* Document the actual behaviour of zs_insert
* Use the block verifier and non-finalized state in the cached state tests
This substantially increases test coverage.
Previously, the cached state tests were configured with
`checkpoint_sync = true`, which only uses the checkpoint
verifier and the finalized state.
* Log the source of blocks in commit_finalized_direct
This lets us check that we're actually testing the non-finalized state
and block verifier in the cached state tests.
It also improves diagnostics for state errors.
* Fail cached state tests if they're using incorrect heights or configs
This makes sure that the cached state tests actually test the transition
from checkpoint to block verification, and the non-finalized state.
* add legacy chain check and tests
* improve has_network_upgrade check
* add docs to legacy_chain_check()
* change arbitrary module structure
* change the panic message
* move legacy chain acceptance into existing tests
* use a reduced_branch_id_strategy()
* add docs to strategy function
* add argument to check for legacy chain into sync_until()
* add nullifier methods to orchard
* store orchard nullifiers
* bump database version
* update `IntoDisk`
* support V5 in `UpdateWith`
* add a test for finalized state
* Use the latest network upgrade in state proptests
This commit changes the state system and database format to track the
provenance of UTXOs, in addition to the outputs themselves.
Specifically, it tracks the following additional metadata:
- the height at which the UTXO was created;
- whether or not the UTXO was created from a coinbase transaction or
not.
This metadata will allow us to:
- check the coinbase maturity consensus rule;
- check the coinbase inputs => no transparent outputs rule;
- implement lookup of transactions by utxo (using the height to find the
block and then scanning the block) for a future RPC mechanism.
Closes#1342
## Motivation
Prior to this PR we've been using `sled` as our database for storing persistent chain data on the disk between boots. We picked sled over rocksdb to minimize our c++ dependencies despite it being a less mature codebase. The theory was if it worked well enough we'd prefer to have a pure rust codebase, but if we ever ran into problems we knew we could easily swap it out with rocksdb.
Well, we ran into problems. Sled's memory usage was particularly high, and it seemed to be leaking memory. On top of all that, the performance for writes was pretty poor, causing us to become bottle-necked on sled instead of the network.
## Solution
This PR replaces `sled` with `rocksdb`. We've seen a 10x improvement in memory usage out of the box, no more leaking, and much better write performance. With this change writing chain data to disk is no longer a limiting factor in how quickly we can sync the chain.
The code in this pull request has:
- [x] Documentation Comments
- [x] Unit Tests and Property Tests
## Review
@hdevalence
teor
Marek
Conrado Gouvea
Alfredo Garcia
Dimitris Apostolou
Deirdre Connolly
Henry de Valence
Jane Lusby
Jane Lusby