* Implement disk serialization for block headers and transactions
* Re-order column family initialization to match the design
* Add new empty transaction column families
* Split writing block header and transaction data
* Re-order column families for consistency
* Update write snapshots for transaction split
* Use split block and transaction data when reading
* Update snapshots to include genesis transaction hash location
* Filter all prefix iterators to make sure they return the correct values
* Test that the new transaction indexes are consistent
* Add some cleanup TODOs
* Increment the database format to version 15
* Remove unused fisk format impls for Block
* Add a missing prefix extractor for transaction locations
* Make the database generic over the thread mode
* Replace prefix iteration with iteration from a key, and a filter
Prefix iteration caused database hangs.
* Manually iterate through transaction locations to re-create blocks
Also:
- re-write disk read API to avoid iterator hangs
- move disk read API to ReadDisk
- re-write impl rocksdb::AsColumnFamilyRef to a where clause, for consistency
* Update the database version so it's larger than the NU5 testnet 2 version
* Repoint zebra image links to our new zfnd.org site for now
* Remove images/
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Implement `getblockchaininfo` RPC method
* add a test for `get_blockchain_info`
* fix tohex/fromhex
* move comment
* Update lightwalletd acceptance test for getblockchaininfo RPC (#3914)
* change(rpc): Return getblockchaininfo network upgrades in height order (#3915)
* Update lightwalletd acceptance test for getblockchaininfo RPC
* Update some doc comments for network upgrades
* List network upgrades in order in the getblockchaininfo RPC
Also:
- Use a constant for the "missing consensus branch ID" RPC value
- Simplify fetching consensus branch IDs
- Make RPC type derives consistent
- Update RPC type documentation
* Make RPC type derives consistent
* Fix a confusing test comment
* get hashand height at the same time
* fix estimated_height
* fix lint
* add extra check
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* fix typo
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* split test
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* fix(rpc): ignore an expected error in the RPC acceptance tests (#3961)
* Add ignored regexes to test command failure regex methods
* Ignore empty chain error in getblockchaininfo
We expect this error when zebrad starts up with an empty state.
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Increment the database format version
* Update IntoDisk and FromDisk docs
* Rename fixed_byte_len to fixed_disk_byte_len
* Add functions that truncate and extend serialized bytes
* Store heights in 3 bytes on disk
* Update database raw data snapshots for 3-byte heights
* Log an error if we ever get close to the maximum disk height
* Store transaction indexes in 2 bytes on disk
* Update database raw data snapshots for 2-byte transaction indexes
* Make doc comment phrasing consistent
* Replace IntoDiskFixed with fixed constants
* Replace u32 byte length literal with a constant calculation
* Fix off-by-one error in MAX_ON_DISK_HEIGHT
* Add proptest seeds for the MAX_ON_DISK_HEIGHT off-by-one error
* Remove redundant module from a Height type
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Add round-trip test for OutputLocation disk serialization
* Make the transaction snapshot tests more accurate
Previously, we were showing the genesis transaction hash at height 0, index 0.
But its hash is actually not stored by location in the database,
because the genesis transaction is skipped due to a consensus rule.
* Update the transaction snapshot data
* Add history tree snapshot tests
At the current test heights, the history trees are all empty.
* Add the history tree snapshot data
* Update comments
* Simplify snapshot test code
* Make some serde::Serialize impls test-only, so we can hex-encode them
This should also speed up release compilation a bit.
* Add snapshot test code for UTXOs
* Add snapshot data for UTXOs
* Split out ReadRequest and ReadResponse state service enums
* Simplify RPC test vectors
* Split state requests into Request and ReadRequest
* Make zebra-rpc use the new state ReadRequest
* Add request metrics to ReadStateService
Add a field to distinguish StateService and ReadStateService metrics.
Add missing StateService request metrics.
* Refactor state transaction lookup so it can be shared between services
* Implement ReadState Request::Transaction
* Revert "Drop the Chain watch channel lock before accessing the finalized state"
This reverts commit 8870944d1323fc31e2d009b9938623de6fcbe50f.
* Add a WatchReceiver wrapper that always clones the borrowed watch data
This avoids deadlocks, by holding the read lock for as short a time as possible.
* Drop the shared Arc<Chain>s as quickly as possible
This reduces memory usage.
* Make read::block more flexible, by accepting any AsRef<Chain>
* Make the block method docs consistent
* Avoid livelocks by explicitly dropping the borrow after the clone
* Use the read-only state service for RPCs
* Refactor non-finalized block lookup into Chain
* Implement the read-only state block request
* Drop the Chain watch channel lock before accessing the finalized state
* Query Sapling & Orchard trees by height in the finalized state
* Add Sapling & Orchard trees to the non-finalized state
* Add a TODO about concurrent read-only access to Sprout tree
Co-authored-by: teor <teor@riseup.net>
* Update the database format version
* Keep only the most recent Sprout tree in the database
* Check that the database returns empty trees for the genesis block
* Assert that the database returns the highest trees
* Document how to update insta snapshots
* Add note commitment tree insta snapshot tests
* Add comments about cached tree roots in snapshots
* Add snapshot data for sapling and orchard trees
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
* Add state service module docs and cleanup
* Move and add finalized state methods
* Add chain and non-finalized state methods
* Cleanup methods and imports
* Create a ReadStateService type
* Add a stub service implementation
* Add a TODO
* Update ReadStateService request stubs with RPC names and tickets
* Documentation updates
* Make RPC State generic bounds accept a buffered state and a read-only state
* Doc updates
* Add missing proptest-impl feature in RPC dev dependencies
* Move the legacy chain check to the `check` module
And move `populated_state` to the `arbitrary` module.
* Cleanup imports
* Document the state service struct
* Split state block iter into its own module
* Prepare the finalized state for read-only state
* Add a forced shutdown mode, used in test code before forced exits
* Document the small database drop race condition window
* Move the legacy chain check to the `check` module
And move `populated_state` to the `arbitrary` module.
* Cleanup imports
* Document the state service struct
* Split state block iter into its own module
* Drop chain if it becomes invalid
Avoid returning a chain that shouldn't be used again by having the
method take ownership of `self` and dropping it on error.
* Update documentation and comments
Restore them to be closer to how they were on `main`.
* fix(shielded): use RwLock for note commitment tree root caches
* Prepare the non-finalized state for read-only state
* fix(state): performance tweak for chain forks
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* feature(rpc): start adding a `getblock` method
* fix(rpc): replace oneshot
* fix(rpc): replace a panic with error
* fix(rpc): fix test
* feature(rpc): add hex to response
* refactor(rpc): use generic instead of alias
* docs(rpc): improve docs for getblock method
* test(rpc): add a test for getblock method
* deps(rpc): remove non needed tower features
Co-authored-by: teor <teor@riseup.net>
* docs(rpc): add a note to getblock doc
* refactor(rpc): replace alias
* fix(rpc): use `zcash_serialize_to_vec()` instead of logging format
* tests(rpc): add network argument to `populated_state()`
* refactor(rpc): use an error for state service readiness
* fix(rpc): add parameter
* fix(rpc): clippy
* nit(rpc): remove new line from imports
* fix(rpc): remove commented code
* fix(rpc): simplify error
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Use a `SerializedBlock` type to help serializing blocks (#3725)
* Create a `SerializedBlock` helper type
Create a type that can be used as a byte slice, but is guaranteed to
represent a valid block.
* Use `into_iter` instead of `iter`
There's no need to borrow the elements, they can be moved out directly.
This will be necessary because `&Arc<T>` doesn't implement `Borrow<T>`,
so a `SerializedBlock` can't be built directly from an `&Arc<Block>`.
* Use `SerializedBlock` in `GetBlock`
Make the type stricter to avoid storing possibly invalid values. The
bytes are still serialized as a hexadecimal string, through the usage of
`hex`.
The `serde::Deserialize` can't be derived because `hex` requires the
type to also implement `FromHex`.
* feature(rpc): add suggestions from code review
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* tests(rpc): make sure mempool has no requests in get_block test
* fix(rpc): change height argument type in getblock method
* fix(rpc): rustfmt
* fix(rpc): replace panic
* fix(rpc): change getblock response
* fix(rpc): fix lightwalletd test
* tests(rpc): add a getblock error test
* fix(rpc): try another regex
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* refactor(test/block): rename large single transaction function
```sh
fastmod single_transaction_block single_transaction_block_many_inputs
```
* rustfmt
* test(block): add a test block with many transparent outputs
* doc(db): explain why we can't just get the UTXOs right before they are deleted
* refactor(db): split out a block data write method
* refactor(block): add a height argument to new_outputs
* test(db): add block and transaction round-trip tests
Including large blocks and transactions.
* test(db): fix large block serialization instability in the tests
* doc(block): add TODOs for generating correct blocks
* Make transparent output functions which take a height test-only
* make sure generated blocks are actually over/under-sized
* replace println!() with an error!() log
* refactor(db): simplify block height serialization
* refactor(db): make height serialization length generic
* refactor(db): create a TransactionIndex type
This changes the names of some snapshot types,
but doesn't change any data.
* refactor(db): create transparent OutputIndex and OutputLocation types
This keeps the same serialization, to avoid changing the database version.
* doc(rfc/db): make transparent database type names consistent
* doc(rfc/db): fix a bug in the Utxo.is_coinbase derivation
* fix(db): use the correct serialized size for OutputLocation
* doc(db): fix some comments
* refactor(db): split disk serialization types into their own module
* refactor(db): split the disk format into modules
* doc(db/test): explain the RON serialization format
* fix: check TEST_FAKE_ACTIVATION_HEIGHTS at runtime
* fix(tests): add TEST_FAKE_ACTIVATION_HEIGHTS variable
This variable ensures the test is activated in the `test-fake-activation-heights` step
* fix(docker): do not run specific tests by default in entrypoint.sh
* fix(test): remove extra TEST_FULL_SYNC argument
* imp(timeout): wait for an average build time
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* test(db): snapshot finalized tips
* test(db): snapshot block heights and hashes
* test(db): snapshot block data
* test(db): snapshot transaction hashes and data
* test(db): assert that blocks and transactions are in chain order
* clippy(db/test): cargo clippy --fix
* clippy(db/test): replace &Vec with &[_]
* refactor(state): split database access into modules by Zebra types
Also split the genesis block check from the genesis note commitment trees.
* test(db): snapshot column family names
* fix(db): assert that the default column family is empty on open and close
* feat(test): apply `cargo insta` settings to all tests
* doc(db/test): improve test docs
* test(db): snapshot column family data for the empty state
* refactor(db/test): split out the raw database snapshot
* test(db): snapshot raw database for blocks 0-2
* test(db): initial serialized snapshot data for raw database
* test(db): tweak snapshot file names
* test(db): rename snapshots for consistency
* test(db): store empty column families in a single snapshot
* test(db): simplify snapshot files by combining empty snapshots
* doc(db/test): put comment in a better place
* refactor(db): fastmod assert_default_is_empty assert_default_cf_is_empty
* doc(test): explain when insta settings are needed
* fix(state/test): use the network to initialize the state
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* fix(state): mark DiskWriteBatch as must_use
* doc(state): add TODOs for moving database reads to blocking threads
* doc(state): minor comment tweaks
* refactor(state): split write batch into block, transactions, chain history
* refactor(state): split out a genesis block write method
* refactor(state): just use the empty note commitment trees directly
* refactor(state): split transaction writes into transparent, nullifiers, trees
And change DiskWriteBatch methods to take `&mut self`.
* refactor(state): split chain value pool writes out of history writes
* refactor(state): combine note commitment trees into an agrument struct
* refactor(state): split history and note commitment updates
* refactor(state): calculate current tip height and remove that argument
* refactor(state): move disk_db reads to a new zebra_db module
* refactor(state): make finalized value pool method names consistent
* refactor(state): split database writes into the zebra_db module
* refactor(state): move the block batch method to DiskWriteBatch
* refactor(state): actually add the zebra_db module
Unfortunately, I've lost the interim changes to this file,
so this commit might be the only one that compiles.
* refactor(state): add a newly created file to the cached state CI job
* refactor(state): move RocksDB-specific initialization to a new module
* refactor(state): move RocksDB-specific shutdown to a new module
* refactor(state): temporarily allow RocksDB-specific reads and writes, without a new module
Unlike the last few commits, this one actually compiles.
* refactor(state): add a DiskWriteBatch wrapper for RocksDB writes
* refactor(state): move finalized state test methods to a test module
Also:
- upgrades to rlimit 0.7.0
- updates types to match the breaking changes in rlimit
- deletes a manual implementation that was similar to `increase_nofile_limit`,
but not as good on macOS and some BSDs
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* docs: document second part of consensus rules from 7.6 Block Header Encoding and Consensus
* docs: explain that the finalSaplingRoot check is not needed since we checkpoint on Canopy
* refactor(state): split the disk_format module
* refactor(ci): add the new disk_db file to the state CI list
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Remove redundant documentation
The documentation was exactly the same as the documentation from the
trait.
* Calculate a mock time block delta for tests
Simulate a block being added to the chain with a random block time based
on the previous block time and the target spacing time.
* Add a `time` field to `ChainTipBlock`
Store the block time so that it's ready for a future chain that allows
obtaining the chain tip's block time.
* Add `ChainTip::best_tip_block_time` method
Allow obtaining the bes chain tip's block time.
* Add method to obtain both height and block time
Prevent any data races by returning both values so that they refer to
the same chain tip.
* Add `NetworkUpgrade::all_target_spacings` method
Returns all the target spacings defined for a network.
* Create a `NetworkChainTipEstimator` helper type
Isolate the code to calculate the height estimation in a new type, so
that it's easier to understand and doesn't decrease the readability of
the `chain_tip.rs` file.
* Add `ChainTip::estimate_network_chain_tip_height`
This is more of an extension method than a trait method. It uses the
`NetworkChainTipHeightEstimator` to actually perform the estimation, but
obtains the initial information from the current best chain tip.
* Fix typo in documentation
There was an extra closing bracket in the summary line.
* Refactor `MockChainTipSender` into a separate type
Prepare to allow mocking the block time of the best tip as well as the
block height.
* Allow sending mock best tip block times
Add a separate `watch` channel to send the best tip block times from a
`MockChainTipSender` to a `MockChainTip`.
The `best_tip_height_and_block_time` implementation will only return a
value if there's a height and a block time value for the best tip.
* Fix off-by-one height estimation error
Use Euclidean division to force the division result to round down
instead of rounding towards zero. This fixes an off-by-one error when
estimating a height that is lower than the current height, because the
fractionary result was being discarded, and it should have forced the
height to go one block back.
* Fix panics on local times very far in the past
Detect situations that might cause the block height estimate to
underflow, and return the genesis height instead.
* Fix another off-by-one height estimation error
The implementation of `chrono::Duration::num_seconds` adds one to the
number of seconds if it's negative. This breaks the division
calculation, so it has to be compensated for.
* Test network chain tip height estimation
Generate pairs of block heights and check that it's possible to estimate
the larger height from the smaller height and a displaced time
difference.
* Support large block heights
* Document consensus rules referring to expiry heights
* Refactor the docs
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Fix the formatting of an error message
* refactor: Simplify coinbase expiry code so the consensus rule is clear (#3408)
* Fix some outdated TODO comments
* refactor(coinbase expiry): Simplify the code so consensus rule is clear
* Fix the formatting of an error message
* Remove a redundant comment
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Marek <mail@marek.onl>
* Check the max expiry height at parse time
* Test that 2^31 - 1 is the last valid height
* Add tests for nExpiryHeight
* Add tests for expiry heights of V4 transactions
* Add tests for V5 transactions
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* feat(network): send notfound messages to the inventory registry
* refactor(network): move the inventory filter into an async function
* feat(network): avoid routing requests to peers that are missing inventory
* test(network): advertised routing is independent of numeric address value
* test(network): peer set routes requests to peers not missing that inventory
* test(network): peer set fails requests if all ready peers are missing that inventory
* fix(clippy): needless-borrow in the peer set
* fix(lint): remove redundant trailing commas in macro calls
There is no clippy lint for this, maybe because some macros
are sensitive to trailing commas.
(But not the ones changed in this commit.)
* test(network): check the exact number of inventory peers
* doc(network): explain why we ignore inventory send failures
* docs(network): explain why a channel error is ignored
* feat(log): log the state tip height as part of sync progress logs
* fix(log): downgrade some verbose state logs to debug
* feat(log): log successful gossiped block verification at info level
These logs help us diagnose slow progress near the tip.
There won't be very many of these logs,
because they only happen near the tip.
* fix(log): spawn top-level tasks within the global Zebra tracing span
* fix(log): spawn blocking top-level tasks within the global Zebra tracing span
Co-authored-by: teor <teor@riseup.net>
* lint: enable more clippy checks for bug-prone code
* fix(lint): stop denying lints, to avoid being excluded from Crater
Also categorise lints.
* lint: add some lints to the TODO list
* refactor(arithmetic): partial fixes for some integer arithmetic lints
* Document some weird lint behaviour
* cancel background database tasks in `FinalizedState` destructor
* use `shutdown_timeout()`
* Log info-level messages while waiting for background tasks to shut down
* Cancel background tasks during debug_stop_at_height shutdown
This commit moves the database shutdown code into a common function.
* Create a constant for the tokio timeout
* Add a test script for Zebra shutdown errors
* Increase the shutdown timeout to 20 seconds for slower machines
* add title to building zebra
* use imported duration
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Move standard lints into .cargo/config.toml
* Ignore "wrong self convention" in a futures-based trait
This lint might only trigger on beta or nightly at the moment.
* Warn if future incompatibile code is added to Zebra
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
teor
Dimitris Apostolou
Deirdre Connolly
Alfredo Garcia
Conrado Gouvea
Marek
Janito Vaqueiro Ferreira Filho