* Avoid manual handling of previous sapling trees by using iterator windows instead
* Avoid manual sapling subtree index handling by comparing prev and current subtree indexes instead
* Simplify adding notes by using the exact number of remaining notes
* Simplify by skipping the first block, because it can't complete a subtree
* Re-use existing tree update code
* Apply the sapling changes to orchard subtree updates
* add a reverse database column family iterator function
* Make skipping the lowest tree independent of iteration order
* Move new subtree checks into the iterator, rename to end_height
* Split subtree calculation into a new method
* Split the calculate and write methods
* Quickly check the first subtree before running the full upgrade
* Do the quick checks every time Zebra runs, and refactor slow check error handling
* Do quick checks for orchard as well
* Make orchard tree upgrade match sapling upgrade code
* Upgrade subtrees in reverse height order
* Bump the database patch version so the upgrade runs again
* Reset previous subtree upgrade data before doing this one
* Add extra checks to subtree calculation to diagnose errors
* Use correct heights for subtrees completed at the end of a block
* Add even more checks to diagnose issues
* Instrument upgrade methods to improve diagnostics
* Prevent modification of re-used trees
* Debug with subtree positions as well
* Fix an off-by-one error with completed subtrees
* Fix typos and confusing comments
Co-authored-by: Marek <mail@marek.onl>
* Fix mistaken previous tree handling and end tree comments
* Remove unnecessary subtraction in remaining leaves calc
* Log heights when assertions fail
* Fix new subtree detection filter
* Move new subtree check into a method, cleanup unused code
* Remove redundant assertions
* Wait for subtree upgrade before testing RPCs
* Fix subtree search in quick check
* Temporarily upgrade subtrees in forward height order
* Clarify some comments
* Fix missing test imports
* Fix subtree logging
* Add a comment about a potential hang with future upgrades
* Fix zebrad var ownership
* Log more info when add_subtrees.rs fails
* cargo fmt --all
* Fix unrelated clippy::unnecessary_unwrap
* cargo clippy --fix --all-features --all-targets; cargo fmt --all
* Stop the quick check depending on tree de-duplication
* Refactor waiting for the upgrade into functions
* Wait for state upgrades whenever the cached state is updated
* Wait for the testnet upgrade in the right place
* Fix unused variable
* Fix a subtree detection bug and comments
* Remove an early reference to reverse direction
* Stop skipping subtrees completed at the end of blocks
* Actually fix new subtree code
---------
Co-authored-by: Marek <mail@marek.onl>
* Copy the add_subtrees upgrade from the original branch
* Copy the database write changes in shielded.rs from the original branch
* Copy the tree API changes from the original branch
* Simplify subtree APIs to avoid exposing frontiers
* Fix a dead code warning by re-using existing methods
* Use mpsc::Receiver<CancelFormatChange> in the subtree upgrade
* Run the subtree upgrade on startup
* Bump the database format version to 25.2.0
* Fix a confusing 'upgrade complete' log
* Clarify some comments and error messages
* Simplify prev_tree unwrap to avoid an (impossible?) concurrency bug
* Use separate subtree writing functions
* Use common note commitment list code
* Fix subtree completion condition and add asserts
* Simplify subtree API and avoid exposing Address
* Fix API compatibility when Arcs are removed
* Log when each subtree is added
* If a format change is cancelled, don't mark the database as upgraded or do format checks
* Log subtree progress about once every two minutes
* Adds a state validity check for subtrees upgrade
* Orchard is faster, decrease log interval
* Clarify subtree index docs
* Move a log to the correct location
* Refactor subtree upgrade to remove duplicate inverted loop conditions
* updates subtree state validity check
* Add a subtree format check when there is no upgrade
* Fix an off-by-one error with the final subtree check
* Use error-level logs for database format checks
* Skip format checks in tests that create invalid formats
* fix state validity test
* Add a concurrency comment to subtree by height methods
* Add individual subtree state methods: reverts removing these methods in an earlier PR
* fastmod "subtrees_by_index" "subtree_list_by_index_for_rpc"
---------
Co-authored-by: arya2 <aryasolhi@gmail.com>
* Make NoteCommitmentSubtreeIndex compatible with serde-based RPCs
* Add a stub for z_getsubtreesbyindex
* Define a GetSubtrees RPC response type
* Reject invalid shielded pool names
* Make limit optional
* Define state request and response types for subtrees
* Implement FromDisk for NoteCommitmentSubtreeIndex and add a round-trip test
* Make subtrees compatible with round-trip proptests
* Add finalized state subtree list methods and delete unused methods
* Remove Arc from subtrees in zebra-chain
* Remove Arc from subtrees in zebra-state and use BTreeMap
* Implement subtree list lookups in the non-finalized state and delete unused methods
* Implement consistent concurrent subtree read requests
* Implement ToHex for sapling::Node
* Implement ToHex for orchard::Node
* Implement z_get_subtrees_by_index RPC
* Check for the start_index from the non-finalized state
* Remove an unused mut
* Fix missing doc links
* Fix RPC comments
* Temporarily remove the z_get_subtrees_by_index RPC method
* zebra-chain changes from the subtree-boundaries branch
```sh
git checkout -b subtree-boundaries-zebra-chain main
git checkout origin/subtree-boundaries zebra-chain
git commit
```
* Temporarily populate new subtree fields with None - for revert
This temporary commit needs to be reverted in the next PR.
* Applies suggestions from code review
* removes from_repr_unchecked methods
* simplifies loop
* adds subtrees to zebra-state
* uses split_at, from_repr, & updates state-db-upgrades.md
* Update book/src/dev/state-db-upgrades.md
Co-authored-by: teor <teor@riseup.net>
* renames partial_subtree to subtree_data
* tests that subtree serialization format
* adds raw data format serialization round-trip test
* decrements minor version and skips inserting subtrees in db
---------
Co-authored-by: teor <teor@riseup.net>
* zebra-chain changes from the subtree-boundaries branch
```sh
git checkout -b subtree-boundaries-zebra-chain main
git checkout origin/subtree-boundaries zebra-chain
git commit
```
* Temporarily populate new subtree fields with None - for revert
This temporary commit needs to be reverted in the next PR.
* Applies suggestions from code review
* removes from_repr_unchecked methods
* simplifies loop
---------
Co-authored-by: arya2 <aryasolhi@gmail.com>
teor
Arya