* Pass ZebraDB to batch preparation
* Dedup the insertion of Sapling trees into database
* Dedup the insertion of Orchard trees into database
* Update snapshots
* Rename batch preparation of trees
* Simplify the naming of note commitment trees
* Correctly retrieve Sapling trees from fin state
* Correctly retrieve Orchard trees from fin state
* Simplify the naming of methods for Sprout trees
* Simplify the naming of methods for Sapling trees
* Simplify the naming of methods for Orchard trees
* Reduce disk reads by caching trees. (#7276)
* Bump the state minor version
* Reset the state patch version
* Simplify the preparation of genesis trees
* Store the roots of the trees of the genesis block
* Add the genesis roots to snapshots
* fix(test): Don't include shielded data in genesis blocks (#7302)
* fix(state): Fix marking format upgrades (#7304)
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
* Move the state format into a new doc
* Add upgrade instructions
* Link to the format upgrade docs from the upgrade code
* Fix typo
Co-authored-by: Marek <mail@marek.onl>
---------
Co-authored-by: Marek <mail@marek.onl>
* Remove duplicate asserts
There are the same two asserts above the two removed ones.
* Remove workarounds for inserting trees into NFS
NFS = non finalized state
* Use correct height for constructing new chain
We were using the height of the last block instead of the initial block
to construct a new chain.
* Don't push the 0th block into a chain
* Don't commit two blocks at the same height
* Add helpers for heights
* Support the retrieval of deduped Sprout trees
* Dedup Sprout trees
* Refactor docs for adding & removing Sprout trees
* Support the retrieval of deduped Sapling trees
* Dedup Sapling trees
* Refactor docs for adding & removing Sapling trees
* Support the retrieval of deduped Orchard trees
* Dedup Orchard trees
* Refactor docs for adding & removing Orchard trees
* Make the docs for storing trees clearer
* Remove duplicate asserts
There are the same two asserts above the two removed ones.
* Remove workarounds for inserting trees into NFS
NFS = non finalized state
* Use correct height for constructing new chain
We were using the height of the last block instead of the initial block
to construct a new chain.
* Don't push the 0th block into a chain
* Don't commit two blocks at the same height
* Fix typo
* Generate chains with at least two blocks
---------
Co-authored-by: teor <teor@riseup.net>
* Add an async-error feature and an initial module structure
* Implement checking for panics in OS threads and async tasks
* Implement waiting for panics in OS threads and async tasks
* Add a TODO to simplify some state request error handling
* Use the new panic-checking methods in zebra-state
* Use new panic-checking methods in zebra-network
* fixup! Implement waiting for panics in OS threads and async tasks
* Replace existing async code with generic panic-checking methods
* Simplify trait to a single method
* Move thread panic code into generic trait impls
* Simplify option handling
Co-authored-by: Arya <aryasolhi@gmail.com>
* Fix comment
Co-authored-by: Arya <aryasolhi@gmail.com>
* Add missing track_caller
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
* update ecc dependencies no serialization
* use zcash_primitives merke tree functions for serialization/deserialization in orchard and sapling trees
* some more work in sapling/orchard serialization/deserialization
* clippy
* fix doc links
* fix missing doc
* make orchard trees serializa/deserialize as they were before upgrade
* make sapling trees serialize/deserialize as they were before upgrade
* use legacy for sprout
* remove unused code
* readd snapshot tests
* repalce some code
* upgrade zcash_proofs
* remove legacy code for sprout
* fix the count method
* add root to serialize
* fixes in as_bytes and from_bytes
* use legacy code
* add todo about pow2 tests
* remove unused sprout code
* fix doc typos
* Add a recalculate_root() method to trees for tests
* Rename test tree types to make them easier to change
* Add TODOs for tests for old and new serialization formats
* fix doc typos
* add more test to note commitment trees
* fix comment
* fix leaf serializatiuon, reverse to old hashes in pow2 tests
* fix serialization
* put sapling SerializedTree code back
* put orchard SerializedTree code back
* clippy
* add duplicated dependencies until zebra_script updates
* fix a doc link
* minor cleanup
* remove todo comment from tests
* add one more check to tests
* update zebra_script
* update deny.toml
* replace custom function with library
* fix some tests
* update docs
* Remove duplicate dependencies from deny.toml
Based on the list at:
https://github.com/ZcashFoundation/zebra/actions/runs/5557139662/jobs/10150543673?pr=7053#step:5:14
---------
Co-authored-by: teor <teor@riseup.net>
* Move format upgrades to their own module and enum
* Launch a format change thread if needed, and shut it down during shutdown
* Add some TODOs and remove a redundant timer
* Regularly check for panics in the state upgrade task
* Only run example upgrade once, change version field names
* Increment database format to 25.0.2: add format change task
* Log the running and initial disk database format versions on startup
* Add initial disk and running state versions to cached state images in CI
* Fix missing imports
* Fix typo in logs workflow command
* Add a force_save_to_disk argument to the CI workflow
* Move use_internet_connection into zebrad_config()
* fastmod can_spawn_zebrad_for_rpc can_spawn_zebrad_for_test_type zebra*
* Add a spawn_zebrad_without_rpc() function
* Remove unused copy_state() test code
* Assert that upgrades and downgrades happen with the correct versions
* Add a kill_and_return_output() method for tests
* Add a test for new_state_format() versions (no upgrades or downgrades)
* Add use_internet_connection to can_spawn_zebrad_for_test_type()
* Fix workflow parameter passing
* Check that reopening a new database doesn't upgrade (or downgrade) the format
* Allow ephemeral to be set to false even if we don't have a cached state
* Add a test type that will accept any kind of state
* When re-using a directory, configure the state test config with that path
* Actually mark newly created databases with their format versions
* Wait for the state to be opened before testing the format
* Run state format tests on mainnet and testnet configs (no network access)
* run multiple reopens in tests
* Test upgrades run correctly
* Test that version downgrades work as expected (best effort)
* Add a TODO for testing partial updates
* Fix missing test arguments
* clippy if chain
* Fix typo
* another typo
* Pass a database instance to the format upgrade task
* Fix a timing issue in the tests
* Fix version matching in CI
* Use correct env var reference
* Use correct github env file
* Wait for the database to be written before killing Zebra
* Use correct workflow syntax
* Version changes aren't always upgrades
---------
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Add a recalculate_root() method to trees for tests
* Rename test tree types to make them easier to change
* Add TODOs for tests for old and new serialization formats
* fix doc typos
---------
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Add an assert_frontier_eq() method to note commitment trees for tests
* Check round-trip serialization for note commitment trees
* fix typos
---------
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Remove duplicate "running" message, send log file opening logs to stderr
* Add missing block progress bar desc, make them consistent
* Put progress bars in a specific order in each section
* Actually make block progress bar desc consistent
* Make progress bar order even more consistent
* Fix fork blocks plural for 1 block
* Use the correct number of chain fork bars
* Disable confusing partial work display
* Add struct field category comments
* Silence a verbose inventory log
* claridy some checkpoint verifier docs
* update documentation of `Request::CommitSemanticallyVerifiedBlock` and `Request::CommitCheckpointVerifiedBlock`
* replace `prepared` with `semantically_verified` in state service checks code
* replace `non-finalized` where needed in docs of the state service
* fix double space in doc
* replace `finalized` where needed in docs of the state service
* change some docs in state queued_blocks.rs
* Rewrite pending UTXO checkpoint block comment
* Fix trailing space in docs
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
---------
Co-authored-by: teor <teor@riseup.net>
* Activate production features on docs.rs
* Make version requirements more flexible to work around unpublished versions
* Remove a redundant feature requirement that causes publishing issues
* fix release auto-replacement format
* cargo release replace --verbose --execute --package zebrad
* Add a missing 1.0.1 changelog entry
* Fix incorrect commands in the release checklist
* Check the crates.io release script in CI, to avoid release failures
* Fix release date in changelog
* Fix workflow typo
* Add --no-confirm to the workflow
* Fix outdated version in release check command
* After the release, add a "remove do-not-merge" step
* Set git identity before committing
* Don't overwrite tweaked versions on upgrade
* cargo release version doesn't support --dependent-version
* Update the same commands in the release checklist
* Commit to git after each change
* And in the checklist
* Add cargo release config
* In CI, allow any branch and provide a fake previous tag name
* Remove redundant `cargo release commit`
* Use a simpler release version process
* Start simplifying the release dry run in CI
* Fix a cargo dependency lint
* just skip the failing crates for now
* Fix the release checklist to match CI
* Use the latest versions of all Zebra dependencies
* Update license description in README for MIT-only crates
* Draft changelog with trivial issues
* Remove trivial issues
* Update changelog entries as of commit 2a31972 and PR #7103
* Update mainnet and testnet checkpoints as of 2023-06-30
* chore: Release
* Estimate release height for Zebra v1.0.1
Block height 2139118 at 2023-06-30 01:55:38 UTC
Release is likely to be 2023-07-01
2139118 + 1152 * 3 = 2142574
Then round up to the nearest 1000.
* Add methods for loading entire column families from the database
* Add a method that loads all the sprout trees from the database
* Add snapshot tests for sprout note commitment trees
* Add round-trip proptests for tree root database serialization
* Add a manual sprout note commitment tree database serialization snapshot test
* Add tests for 1,2,4,8 note commitments in a tree
* Remove redundant "rand" package rename in dependencies
* Randomly cache roots rather than only caching even roots
---------
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Add and use `FinalizableBlock`
This commit adds `FinalizableBlock`, and uses it instead of
`ContextuallyVerifiedBlockWithTrees` in `commit_finalized_direct()`
* Use `ContextuallyVerifiedBlockWithTrees`
This commit passes `ContextuallyVerifiedBlockWithTrees` instead of
passing separate `finalized`, `history_tree` and `note_commitment_trees`
when storing blocks in the finalized state.
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* add docs to new methods
* fix existing doc
* rename `ContextuallyVerifiedBlockWithTrees` to `SemanticallyVerifiedBlockWithTrees`
* Refactor docs
* Refactor comments
* Add missing docs, fix typo
* Fix rustfmt
---------
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Remove openssl dependency and prevent it coming back
* Put the arguments in the right place
* Put comment in the right place
* Add a default-docker feature to zebrad and use it in workflows and Docker files
* Fix a comment typo
* Make sure that Docker production builds don't use openssl
* Rename feature to default-release-binaries
* Use Arc::into_inner() to avoid potential concurrency issues
* Remove some outdated clippy lint workarounds (fixed in Rust 1.66)
* Update the required Rust version to 1.70
* Refactor `CheckpointVerifiedBlock`
This commit turns `CheckpointVerifiedBlock` into a wrapper of
`SemanticallyVerifiedBlock` since both structs have the same fields.
* Refactor `ContextuallyVerifiedBlockWithTrees`
This commit uses `SemanticallyVerifiedBlock` in
`ContextuallyVerifiedBlockWithTrees` instead of
`CheckpointVerifiedBlock`.
* fix: prevent progress bar from panicking using workaround
* remove more progress bar lengths
* re-add set_len() calls commented out; restore one that shouldn't cause panics
* Update zebra-state/src/service/non_finalized_state.rs
Co-authored-by: teor <teor@riseup.net>
---------
Co-authored-by: teor <teor@riseup.net>
dependabot[bot]
Marek
Arya
Alfredo Garcia
teor
Conrado Gouvea