* refactor(db): simplify block height serialization
* refactor(db): make height serialization length generic
* refactor(db): create a TransactionIndex type
This changes the names of some snapshot types,
but doesn't change any data.
* refactor(db): create transparent OutputIndex and OutputLocation types
This keeps the same serialization, to avoid changing the database version.
* doc(rfc/db): make transparent database type names consistent
* doc(rfc/db): fix a bug in the Utxo.is_coinbase derivation
* fix(db): use the correct serialized size for OutputLocation
* doc(db): fix some comments
* refactor(db): split disk serialization types into their own module
* refactor(db): split the disk format into modules
* doc(db/test): explain the RON serialization format
* test(db): snapshot finalized tips
* test(db): snapshot block heights and hashes
* test(db): snapshot block data
* test(db): snapshot transaction hashes and data
* test(db): assert that blocks and transactions are in chain order
* clippy(db/test): cargo clippy --fix
* clippy(db/test): replace &Vec with &[_]
* refactor(state): move disk_db reads to a new zebra_db module
* refactor(state): make finalized value pool method names consistent
* refactor(state): split database writes into the zebra_db module
* refactor(state): move the block batch method to DiskWriteBatch
* refactor(state): actually add the zebra_db module
Unfortunately, I've lost the interim changes to this file,
so this commit might be the only one that compiles.
* refactor(state): add a newly created file to the cached state CI job
* refactor(state): move RocksDB-specific initialization to a new module
* refactor(state): move RocksDB-specific shutdown to a new module
* refactor(state): temporarily allow RocksDB-specific reads and writes, without a new module
Unlike the last few commits, this one actually compiles.
* refactor(state): add a DiskWriteBatch wrapper for RocksDB writes
* refactor(state): move finalized state test methods to a test module
* refactor(state): split the disk_format module
* refactor(ci): add the new disk_db file to the state CI list
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* add serialization for value balances
* change test names
* change panic messages
* add a deserialization test
* return the errors from `from_bytes()` methods
* add prop test for serialize/deserialize Amount
* Move amount proptests to amount::tests::prop
Co-authored-by: teor <teor@riseup.net>
* Add ZIP-221 history tree to finalized state
* Improve error / panic handling; improve documentation
* Return error again when preparing batch, fix expect messages
* Fix bug when pushing the Heartwood actiation block to the history tree
* Re-increase database version since it was increased in main
Co-authored-by: teor <teor@riseup.net>
* Tidy chain Cargo.toml
* Organize imports
* Add method to get note commitments from all Actions in Orchard shielded data
* Add method to get note commitments from all JoinSplits in Sprout JoinSplitData
* Add Request and Response variants for awaiting anchors
* Add anchors and note commitment trees to finalized state db
* Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees
* Track anchors and note commitment trees in Chain
Append note commitments to their trees when doing update_chain_state_with,
then use the resulting Sapling and Orchard roots to pass to history_tree, and add
new roots to the anchor sets.
* Handle errors when appending to note commitment trees
* Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with
* Implementing note commitments in finalized state
* Finish serialization of Orchard tree; remove old tree when updating finalize state
* Add serialization and finalized state updates for Sprout and Sapling trees
* Partially handle trees in non-finalized state. Use Option for trees in Chain
* Rebuild trees when forking; change finalized state tree getters to not require height
* Pass empty trees to tests; use empty trees by default in Chain
* Also rebuild anchor sets when forking
* Use empty tree as default in finalized state tree getters (for now)
* Use HashMultiSet for anchors in order to make pop_root() work correctly
* Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS
* Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more
* Apply suggestions from code review
* Add comments about order of note commitments and related methods/fields
* Don't use Option for trees
* Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS
* Remove unneeded anchor set rebuilding in fork()
* Improve proptest formatting
* Add missing comparisons to eq_internal_state
* Renamed sprout::tree::NoteCommitmentTree::hash() to root()
* Improve comments
* Add asserts, add issues to TODOs
* Remove impl Default for Chain since it was only used by tests
* Improve documentation and assertions; add tree serialization tests
* Remove Sprout code, which will be moved to another branch
* Add todo! in Sprout tree append()
* Remove stub request, response *Anchor* handling for now
* Add test for validating Sapling note commitment tree using test blocks
* Increase database version (new columns added for note commitment trees and anchors)
* Update test to make sure the order of sapling_note_commitments() is being tested
* Improve comments and structure of the test
* Improve variable names again
* Rustfmt
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: teor <teor@riseup.net>
* Reject duplicate sprout nullifiers in the state
* Improve docs and error messages
* Clarify "must be present" assert logs
* Move nullifier checks to their own module
Also:
* make non-finalized nullifier checks and errors generic over
sprout, sapling, and orchard
* create and update module and function documentation
* Fix a block type name in docs
* Move state assertions or skip them during tests
These changes enable state testing, while still asserting in production.
* Add sprout duplicate nullifier tests
* Improve comments
* Set value balance to 0 to pass future chain value pool checks
* Test finalized state in sprout nullifier accept test
* Replace assert with expect
* Improve assertion messages
* Document the new genesis transaction consensus rule
Zebra previously implemented this rule, but we documented it as a bug in
`zcashd`.
* Document the actual behaviour of zs_insert
* add nullifier methods to orchard
* store orchard nullifiers
* bump database version
* update `IntoDisk`
* support V5 in `UpdateWith`
* add a test for finalized state
* Use the latest network upgrade in state proptests
This commit changes the state system and database format to track the
provenance of UTXOs, in addition to the outputs themselves.
Specifically, it tracks the following additional metadata:
- the height at which the UTXO was created;
- whether or not the UTXO was created from a coinbase transaction or
not.
This metadata will allow us to:
- check the coinbase maturity consensus rule;
- check the coinbase inputs => no transparent outputs rule;
- implement lookup of transactions by utxo (using the height to find the
block and then scanning the block) for a future RPC mechanism.
Closes#1342
## Motivation
Prior to this PR we've been using `sled` as our database for storing persistent chain data on the disk between boots. We picked sled over rocksdb to minimize our c++ dependencies despite it being a less mature codebase. The theory was if it worked well enough we'd prefer to have a pure rust codebase, but if we ever ran into problems we knew we could easily swap it out with rocksdb.
Well, we ran into problems. Sled's memory usage was particularly high, and it seemed to be leaking memory. On top of all that, the performance for writes was pretty poor, causing us to become bottle-necked on sled instead of the network.
## Solution
This PR replaces `sled` with `rocksdb`. We've seen a 10x improvement in memory usage out of the box, no more leaking, and much better write performance. With this change writing chain data to disk is no longer a limiting factor in how quickly we can sync the chain.
The code in this pull request has:
- [x] Documentation Comments
- [x] Unit Tests and Property Tests
## Review
@hdevalence
teor
Marek
Conrado Gouvea
Alfredo Garcia
Deirdre Connolly
Henry de Valence
Jane Lusby
Jane Lusby