* Add methods for loading entire column families from the database
* Add a method that loads all the sprout trees from the database
* Add snapshot tests for sprout note commitment trees
* Add round-trip proptests for tree root database serialization
* Add a manual sprout note commitment tree database serialization snapshot test
* Add tests for 1,2,4,8 note commitments in a tree
* Remove redundant "rand" package rename in dependencies
* Randomly cache roots rather than only caching even roots
---------
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Enable cancel_all_background_work() only on macOS
* Ignore expected "during shutdown" errors, and log other errors
* Disable cancel_all_background_work() but keep the updated docs and error handling
* Add the macOS shutdown crash to the README known issues
---------
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Implement minor and patch database format versions
* Log and update database format versions when opening database
* Refactor the current list of column families into a constant
* Open all available column families, including from future Zebra versions
* Refactor note commitment tree lookups to go through the height methods
* Make Sapling/Orchard note commitment tree lookup forwards compatible
* Ignore errors reading column family lists from disk
* Update format version comments and TODOs
* Correctly log newly created database formats
---------
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Remove verbose continuous_blockchain test logs
* Downgrade verbose zebra-network logs to debug
* Downgrade some state logs to debug during tests
* Mark were we would add always-on log filters, if we needed to
* Reduce the number of mempool property tests, to reduce logging
* Add a new block commit task and channels, that don't do anything yet
* Add last_block_hash_sent to the state service, to avoid database accesses
* Update last_block_hash_sent regardless of commit errors
* Rename a field to StateService.max_queued_finalized_height
* Commit finalized blocks to the state in a separate task
* Check for panics in the block write task
* Wait for the block commit task in tests, and check for errors
* Always run a proptest that sleeps once
* Add extra debugging to state shutdowns
* Work around a RocksDB shutdown bug
* Close the finalized block channel when we're finished with it
* Only reset state queue once per error
* Update some TODOs
* Add a module doc comment
* Drop channels and check for closed channels in the block commit task
* Close state channels and tasks on drop
* Remove some duplicate fields across StateService and ReadStateService
* Try tweaking the shutdown steps
* Update and clarify some comments
* Clarify another comment
* Don't try to cancel RocksDB background work on drop
* Fix up some comments
* Remove some duplicate code
* Remove redundant workarounds for shutdown issues
* Remode a redundant channel close in the block commit task
* Remove a mistaken `!force` shutdown condition
* Remove duplicate force-shutdown code and explain it better
* Improve RPC error logging
* Wait for chain tip updates in the RPC tests
* Wait 2 seconds for chain tip updates before skipping them
* Remove an unnecessary block_in_place()
* Fix some test error messages that were changed by earlier fixes
* Expand some comments, fix typos
Co-authored-by: Marek <mail@marek.onl>
* Actually drop children of failed blocks
* Explain why we drop descendants of failed blocks
* Clarify a comment
* Wait for chain tip updates in a failing test on macOS
* Clean duplicate finalized blocks when the non-finalized state activates
* Send an error when receiving a duplicate finalized block
* Update checkpoint block behaviour, document its consensus rule
* Wait for chain tip changes in inbound_block_height_lookahead_limit test
* Wait for the genesis block to commit in the fake peer set mempool tests
* Disable unreliable mempool verification check in the send transaction test
* Appease rustfmt
* Use clear_finalized_block_queue() everywhere that blocks are dropped
* Document how Finalized and NonFinalized clones are different
* Use the same check as commit_finalized() for finalized block heights
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Marek <mail@marek.onl>
* Move the finalized block queue into the StateService
* Move the queued_blocks module to the state service
* Move QueuedFinalized into queued_blocks
* Move the queued_blocks tests into their own module
* Make the FinalizedState cloneable
* cargo upgrade --workspace rocksdb
* Add a deny.toml exception for bindgen 0.59
* Move `valid()` and `status()` methods to raw iterators
* Update some outdated comments
* Panic on iterator failures
We might want to change this if there are any common failure modes.
* allow(clippy::unwrap_in_result) in some methods
* Only fetch block headers from the database to answer headers requests
* Move writing to the database to a blocking thread
* Add blocking threads to tests that need them
* Remove mempool downloader requirement for a populated state
And stop populating states that don't need it,
so we can use tokio::time::pause() in those tests.
* Improve debugging for an intermittent test failure
* Try to avoid a race condition populating the mempool in tests
* Rename `block_by_height` to `block_header_by_height` in fin state
* Rename `tx_by_hash` to `tx_loc_by_hash` in both (non & fin) states
* Rename `utxo_by_outpoint` to `utxo_by_out_loc` in finalized state
* Reorder the column families so that they match the docs
* Update `struct Chain` in the RFCs
* Increment `DATABASE_FORMAT_VERSION` to 25
* Remove obsolete docs from `0004-asynchronous-script-verification.md`
* Remove an obsolete `TODO` from `disk_db.rs`
* Delete unused snapshots
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Fix the syntax of links in comments
* Fix a mistake in the docs
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Remove unnecessary angle brackets from a link
* Revert the changes for links that serve as references
* Revert "Revert the changes for links that serve as references"
This reverts commit 8b091aa9fab453e7d3559a5d474e0879183b9bfb.
* Remove `<` `>` from links that serve as references
This reverts commit 046ef25620ae1a2140760ae7ea379deecb4b583c.
* Don't use `<` `>` in normal comments
* Don't use `<` `>` for normal comments
* Revert changes for comments starting with `//`
* Fix some warnings produced by `cargo doc`
* Fix some rustdoc warnings
* Fix some warnings
* Refactor some changes
* Fix some rustdoc warnings
* Fix some rustdoc warnings
* Resolve various TODOs
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Fix the syntax of links in comments
* Fix a mistake in the docs
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Remove unnecessary angle brackets from a link
* Revert the changes for links that serve as references
* Revert "Revert the changes for links that serve as references"
This reverts commit 8b091aa9fab453e7d3559a5d474e0879183b9bfb.
* Remove `<` `>` from links that serve as references
This reverts commit 046ef25620ae1a2140760ae7ea379deecb4b583c.
* Don't use `<` `>` in normal comments
* Don't use `<` `>` for normal comments
* Revert changes for comments starting with `//`
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Add test-only serialization, and make existing serialization test-only
* Make AddressLocations clearer in the API
* Add UnspentOutputAddressLocation
* Add the AddressLocation to the UTXO database value
* Update the snapshot test code for UnspentOutputAddressLocation
* Update the raw data snapshots
* Update the high-level data snapshots
* Increment the database version
* Make serialization clearer
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Fix code formatting
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Add an empty utxo_by_transparent_addr_loc column family
* Update snapshot data for the new column family
* Add an AddressUnspentOutputs type
* Add round-trip tests for AddressUnspentOutputs
* Move address balances into their own method
* Simplify updating address balances
* Fix utxo_by_out_loc column family name
* Implement reads and writes of address UTXOs
* Update raw data snapshots
* Update the snapshot tests for high-level address UTXOs
* Assert rather than taking empty address snapshots for genesis
* Update high-level address UTXO snapshot data, and delete empty snapshots
* Increment the database version
* Use typed values for all ReadDisk methods
* Implement test-only serialization for transparent::Address
* Implement FromDisk for ()
* Store AddressUnspentOutput as the column family key
* Update round-trip serialization tests for AddressUnspentOutput
* Update snapshot test code, and add a UTXO data snapshot
* Update existing snapshot data
* Add new UTXO snapshot data
* Update column family name
```sh
fastmod utxo_by_transparent_addr_loc utxo_loc_by_transparent_addr_loc zebra*
```
* cargo fmt --all
* cargo insta test --review --delete-unreferenced-snapshots
* Explain why it is ok to use invalid database iterator indexes
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Add explanations of UTXO database updates
* Simplify an assertion
* Remove UnspentOutputAddressLocation and just store transparent::Output
* Update snapshot test data
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Change OutputLocation to contain a TransactionLocation
* Change OutputLocation reads from the database
* Update some doc comments
* Update some TODOs
* Change deleting spent UTXOs and updating spent balances
* Change adding new UTXOs and adding their values to balances
* Disable dead code warnings
* Update snapshot test code
* Update round-trip tests for OutputLocations
* Update snapshot test data
* Increment the database format version
* Remove a redundant try_into()
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Refactor redundant code
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* ci: attempt at fixing 'Regenerate stateful disks'
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Add an empty balance_by_transparent_addr column family
* Add an AddressBalanceLocation type for balance_by_transparent_addr
* Add serialization for balance_by_transparent_addr types
* Add round-trip tests for the new serialized types
* Add missing round-trip and serialized equality tests
* Add a network field to DiskWriteBatch
* Refactor confusing all_utxos_spent_by_block argument
It was actually just the UTXOs from the state spent by the block,
excluding the UTXOs created and spent within the block.
But now we need it to contain all the spent outputs,
including the ones created by the block.
* Read and update address balances in the finalized state
* Update raw data snapshots for transparent address balances
* Add test-only deserialization for transparent addresses
* Add high-level snapshot test code for address balances
* Add high-level snapshots for address balances
* Increment the state version after NU5 testnet 2 rollback
* Implement disk serialization for block headers and transactions
* Re-order column family initialization to match the design
* Add new empty transaction column families
* Split writing block header and transaction data
* Re-order column families for consistency
* Update write snapshots for transaction split
* Use split block and transaction data when reading
* Update snapshots to include genesis transaction hash location
* Filter all prefix iterators to make sure they return the correct values
* Test that the new transaction indexes are consistent
* Add some cleanup TODOs
* Increment the database format to version 15
* Remove unused fisk format impls for Block
* Add a missing prefix extractor for transaction locations
* Make the database generic over the thread mode
* Replace prefix iteration with iteration from a key, and a filter
Prefix iteration caused database hangs.
* Manually iterate through transaction locations to re-create blocks
Also:
- re-write disk read API to avoid iterator hangs
- move disk read API to ReadDisk
- re-write impl rocksdb::AsColumnFamilyRef to a where clause, for consistency
* Update the database version so it's larger than the NU5 testnet 2 version
* Move the legacy chain check to the `check` module
And move `populated_state` to the `arbitrary` module.
* Cleanup imports
* Document the state service struct
* Split state block iter into its own module
* Prepare the finalized state for read-only state
* Add a forced shutdown mode, used in test code before forced exits
* Document the small database drop race condition window
* refactor(state): split database access into modules by Zebra types
Also split the genesis block check from the genesis note commitment trees.
* test(db): snapshot column family names
* fix(db): assert that the default column family is empty on open and close
* feat(test): apply `cargo insta` settings to all tests
* doc(db/test): improve test docs
* test(db): snapshot column family data for the empty state
* refactor(db/test): split out the raw database snapshot
* test(db): snapshot raw database for blocks 0-2
* test(db): initial serialized snapshot data for raw database
* test(db): tweak snapshot file names
* test(db): rename snapshots for consistency
* test(db): store empty column families in a single snapshot
* test(db): simplify snapshot files by combining empty snapshots
* doc(db/test): put comment in a better place
* refactor(db): fastmod assert_default_is_empty assert_default_cf_is_empty
* doc(test): explain when insta settings are needed
* fix(state/test): use the network to initialize the state
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* fix(state): mark DiskWriteBatch as must_use
* doc(state): add TODOs for moving database reads to blocking threads
* doc(state): minor comment tweaks
* refactor(state): split write batch into block, transactions, chain history
* refactor(state): split out a genesis block write method
* refactor(state): just use the empty note commitment trees directly
* refactor(state): split transaction writes into transparent, nullifiers, trees
And change DiskWriteBatch methods to take `&mut self`.
* refactor(state): split chain value pool writes out of history writes
* refactor(state): combine note commitment trees into an agrument struct
* refactor(state): split history and note commitment updates
* refactor(state): calculate current tip height and remove that argument
* refactor(state): move disk_db reads to a new zebra_db module
* refactor(state): make finalized value pool method names consistent
* refactor(state): split database writes into the zebra_db module
* refactor(state): move the block batch method to DiskWriteBatch
* refactor(state): actually add the zebra_db module
Unfortunately, I've lost the interim changes to this file,
so this commit might be the only one that compiles.
* refactor(state): add a newly created file to the cached state CI job
* refactor(state): move RocksDB-specific initialization to a new module
* refactor(state): move RocksDB-specific shutdown to a new module
* refactor(state): temporarily allow RocksDB-specific reads and writes, without a new module
Unlike the last few commits, this one actually compiles.
* refactor(state): add a DiskWriteBatch wrapper for RocksDB writes
* refactor(state): move finalized state test methods to a test module
* refactor(state): split the disk_format module
* refactor(ci): add the new disk_db file to the state CI list
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
teor
Marek
Alfredo Garcia
Conrado Gouvea