* Add tree key format and cached root upgrades
* Document the changes in the upgrades
* Remove unnecessary clippy::unwrap_in_result
* Fix database type
* Bump state version
* Skip some checks if the database is empty
* Fix tests for a short state upgrade
* Disable format checks in some tests
* Document state performance issues
* Clarify upgrade behaviour
* Clarify panic messages
* Delete incorrect genesis trees write code
* Fix metrics handling for genesis
* Remove an unused import
* Explain why genesis anchors are ok
* Update snapshots
* Debug a failing test
* Fix some tests
* Fix missing imports
* Move the state check in a test
* Fix comment and doc typos
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Arya <aryasolhi@gmail.com>
* Clarify what a long upgrade is
* Rename unused function arguments
Co-authored-by: Marek <mail@marek.onl>
* Add all_unordered log regex matching methods
* Fix timing issues with version upgrades and other logs
* Fix argument name in docs
Co-authored-by: Marek <mail@marek.onl>
* Explain match until first for all regexes behaviour better
---------
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Arya <aryasolhi@gmail.com>
* Add and use `FinalizableBlock`
This commit adds `FinalizableBlock`, and uses it instead of
`ContextuallyVerifiedBlockWithTrees` in `commit_finalized_direct()`
* Use `ContextuallyVerifiedBlockWithTrees`
This commit passes `ContextuallyVerifiedBlockWithTrees` instead of
passing separate `finalized`, `history_tree` and `note_commitment_trees`
when storing blocks in the finalized state.
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* add docs to new methods
* fix existing doc
* rename `ContextuallyVerifiedBlockWithTrees` to `SemanticallyVerifiedBlockWithTrees`
* Refactor docs
* Refactor comments
* Add missing docs, fix typo
* Fix rustfmt
---------
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Refactor `CheckpointVerifiedBlock`
This commit turns `CheckpointVerifiedBlock` into a wrapper of
`SemanticallyVerifiedBlock` since both structs have the same fields.
* Refactor `ContextuallyVerifiedBlockWithTrees`
This commit uses `SemanticallyVerifiedBlock` in
`ContextuallyVerifiedBlockWithTrees` instead of
`CheckpointVerifiedBlock`.
* rename verifiers
* rename `PreparedBlock` to `SemanticallyVerifiedBlock`
* rename `CommitBlock` to `SemanticallyVerifiedBlock`
* rename `FinalizedBlock` to `CheckpointVerifiedBlock`
* rename `CommitFinalizedBlock` to `CommitCheckpointVerifiedBlock`
* rename `FinalizedWithTrees` to `ContextuallyVerifiedBlockWithTrees`
* rename `ContextuallyValidBlock` to `ContextuallyVerifiedBlock`
* change some `finalized` variables or function arguments to `checkpoint_verified`
* fix docs
* document the difference between `CheckpointVerifiedBlock` and `ContextuallyVerifiedBlock`
* fix doc links
* apply suggestions to request
Co-authored-by: Marek <mail@marek.onl>
* apply suggestions to service
Co-authored-by: Marek <mail@marek.onl>
* apply suggestions to finalized_state.rs and write.rs
Co-authored-by: Marek <mail@marek.onl>
* fmt
* change some more variable names
* change a few missing generics
* fix checkpoint log issue
* rename more `prepared` vars `semantically_verified`
* fix test regex
* fix test regex 2
---------
Co-authored-by: Marek <mail@marek.onl>
* Add history trees for each height in non-fin state
* Refactor formatting
* Pass the treestate to the finalized state
I created a new structure `FinalizedBlockWithTrees` that wraps the
treestate and the finalized block. I did that because the original
`FinalizedBlock` is `Eq`, but `HistoryTree` can't be `Eq`.
This makes Zebra faster because:
1. The finalized state doesn't retrieve the treestate from the disk if
the non-finalized state supplies it.
2.The finalized state doesn't recompute the treestate if the
non-finalized state supplies it.
* Check block commitment before updating hist tree
* Store Sprout commitment trees in non-fin state
* Send trees for the root block to fin-state
When committing a block and sending the treestate from the non-finalized
state to the finalized state, Zebra was sending trees that correspond to
the tip block instead of trees that correspond to the root block of the
best chain. This commit fixes that.
* Refactor doc comments
* Refactor block finalization
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Split disk reads from CPU-heavy Sprout interstitial tree cryptography
* Improve anchor validation debugging and error messages
* Work around a test data bug, and save some CPU
* Remove redundant checks for empty shielded data
* Skip generating unused interstitial treestates
* Do disk fetches and quick checks, then CPU-heavy cryptography
* Wrap HistoryTree in an Arc in the state
* Run CPU-intensive chain validation and updates in parallel rayon threads
* Refactor to prepare for parallel tree root calculations
* Run finalized state note commitment tree root updates in parallel rayon threads
* Update finalized state note commitment trees using parallel rayon threads
* Fix a comment typo and add a TODO
* Split sprout treestate fetch into its own function
* Move parallel note commitment trees to zebra-chain
* Re-calculate the tree roots in the same parallel batches
* Do non-finalized note commitment tree updates in parallel threads
* Update comments about note commitment tree rebuilds
* Do post-fork tree updates in parallel threads
* Add a TODO for parallel tree updates in tests
* Fix broken intra-doc links
* Clarify documentation for sprout treestates
* Sort Cargo.toml dependencies
* Change OutputLocation to contain a TransactionLocation
* Change OutputLocation reads from the database
* Update some doc comments
* Update some TODOs
* Change deleting spent UTXOs and updating spent balances
* Change adding new UTXOs and adding their values to balances
* Disable dead code warnings
* Update snapshot test code
* Update round-trip tests for OutputLocations
* Update snapshot test data
* Increment the database format version
* Remove a redundant try_into()
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Refactor redundant code
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* ci: attempt at fixing 'Regenerate stateful disks'
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Add an empty balance_by_transparent_addr column family
* Add an AddressBalanceLocation type for balance_by_transparent_addr
* Add serialization for balance_by_transparent_addr types
* Add round-trip tests for the new serialized types
* Add missing round-trip and serialized equality tests
* Add a network field to DiskWriteBatch
* Refactor confusing all_utxos_spent_by_block argument
It was actually just the UTXOs from the state spent by the block,
excluding the UTXOs created and spent within the block.
But now we need it to contain all the spent outputs,
including the ones created by the block.
* Read and update address balances in the finalized state
* Update raw data snapshots for transparent address balances
* Add test-only deserialization for transparent addresses
* Add high-level snapshot test code for address balances
* Add high-level snapshots for address balances
* Increment the state version after NU5 testnet 2 rollback
* Implement disk serialization for block headers and transactions
* Re-order column family initialization to match the design
* Add new empty transaction column families
* Split writing block header and transaction data
* Re-order column families for consistency
* Update write snapshots for transaction split
* Use split block and transaction data when reading
* Update snapshots to include genesis transaction hash location
* Filter all prefix iterators to make sure they return the correct values
* Test that the new transaction indexes are consistent
* Add some cleanup TODOs
* Increment the database format to version 15
* Remove unused fisk format impls for Block
* Add a missing prefix extractor for transaction locations
* Make the database generic over the thread mode
* Replace prefix iteration with iteration from a key, and a filter
Prefix iteration caused database hangs.
* Manually iterate through transaction locations to re-create blocks
Also:
- re-write disk read API to avoid iterator hangs
- move disk read API to ReadDisk
- re-write impl rocksdb::AsColumnFamilyRef to a where clause, for consistency
* Update the database version so it's larger than the NU5 testnet 2 version
teor
Marek
Alfredo Garcia