* Add support for deleting the trees
* Prune the trees
* Remove `Network` from `DiskWriteBatch`
Removing the `Network` from `DiskWriteBatch` makes it easy to
instantiate `DiskWriteBatch`es in `ZebraDb` that remove individual note
commitment trees.
The `Network` from `DiskWriteBatch` was used only for transparent
addresses, so the refactor isn't large. After removing it from
`DiskWriteBatch`, I passed it as a function argument instead. However,
we should simplify the parameter lists because at least two functions
have more than seven parameters now.
* Support individual tree removal in `ZebraDb`
* Refactor the tree removal task
* Prune old comments
* Remove redundant code
* Batch the removals
* delete ranges before relevant network upgrades
* moves prev_tree inits
* add iterator methods for reading note commitment trees
* Sets up skeleton of sapling pipeline
* Replaces .filter with .take_while
Fills in pipeline
Reuses zs_range_iter instead of repeating that code
Updates logic to stop at initial tip height
* uses std threads
* delete_range excludes end key
* fixes off by one bugs
* Log warning when a send fails
* Removes progress logs
* Log join errors instead of panicking
* Revert: Make the `db` field of `ZebraDb` private
* Move `delete_range_sapling_tree`
* Remove a redundant `else if` branch
Rationale:
The condition `n == 1` for the removed branch is true for a subset of
values of `n` in the preceding condition `n >= 1`.
* Use more specific error messages
* Revert: Remove redundant methods for tree removal
* Suggestions for Deduplicate note commitment trees stored in the finalized state (#7330)
* Add TODOs to some `Height` methods
* Add methods for deleting individual trees
* Refactor the tasks for deleting trees
---------
Co-authored-by: arya2 <aryasolhi@gmail.com>
* Allow missing docs directly on derived error types
* Make Request::BestChainBlockHash redirect to the ReadStateService
* Re-write the checkpoint_sync documentation based on the latest consensus rules
* Expose the underlying iterator for CheckpointList
* Validate existing state block hashes at startup, but ignore the result
* Monitor state block hash checkpoint task in the start command
* Fix indentation
* Make logging consistent
* Explain the config needed for full security
* Tidy required checkpoints docs, expand other docs
* Add security and deprecation changelog entries
* Replace task handle vector with a struct
* Add a test that this consensus-critical code actually runs and finishes
* Make some state methods and types available in tests
* Add missing docs
---------
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Add an empty balance_by_transparent_addr column family
* Add an AddressBalanceLocation type for balance_by_transparent_addr
* Add serialization for balance_by_transparent_addr types
* Add round-trip tests for the new serialized types
* Add missing round-trip and serialized equality tests
* Add a network field to DiskWriteBatch
* Refactor confusing all_utxos_spent_by_block argument
It was actually just the UTXOs from the state spent by the block,
excluding the UTXOs created and spent within the block.
But now we need it to contain all the spent outputs,
including the ones created by the block.
* Read and update address balances in the finalized state
* Update raw data snapshots for transparent address balances
* Add test-only deserialization for transparent addresses
* Add high-level snapshot test code for address balances
* Add high-level snapshots for address balances
* Increment the state version after NU5 testnet 2 rollback
* Implement disk serialization for block headers and transactions
* Re-order column family initialization to match the design
* Add new empty transaction column families
* Split writing block header and transaction data
* Re-order column families for consistency
* Update write snapshots for transaction split
* Use split block and transaction data when reading
* Update snapshots to include genesis transaction hash location
* Filter all prefix iterators to make sure they return the correct values
* Test that the new transaction indexes are consistent
* Add some cleanup TODOs
* Increment the database format to version 15
* Remove unused fisk format impls for Block
* Add a missing prefix extractor for transaction locations
* Make the database generic over the thread mode
* Replace prefix iteration with iteration from a key, and a filter
Prefix iteration caused database hangs.
* Manually iterate through transaction locations to re-create blocks
Also:
- re-write disk read API to avoid iterator hangs
- move disk read API to ReadDisk
- re-write impl rocksdb::AsColumnFamilyRef to a where clause, for consistency
* Update the database version so it's larger than the NU5 testnet 2 version
Marek
teor