Commit Graph

99 Commits

Author SHA1 Message Date
Alfredo Garcia eb07bb31d6
rename(state): Rename state verifiers and related code (#6762)
* rename verifiers

* rename `PreparedBlock` to `SemanticallyVerifiedBlock`

* rename `CommitBlock` to `SemanticallyVerifiedBlock`

* rename `FinalizedBlock` to `CheckpointVerifiedBlock`

* rename `CommitFinalizedBlock` to `CommitCheckpointVerifiedBlock`

* rename `FinalizedWithTrees` to `ContextuallyVerifiedBlockWithTrees`

* rename `ContextuallyValidBlock` to `ContextuallyVerifiedBlock`

* change some `finalized` variables or function arguments to `checkpoint_verified`

* fix docs

* document the difference between `CheckpointVerifiedBlock` and `ContextuallyVerifiedBlock`

* fix doc links

* apply suggestions to request

Co-authored-by: Marek <mail@marek.onl>

* apply suggestions to service

Co-authored-by: Marek <mail@marek.onl>

* apply suggestions to finalized_state.rs and write.rs

Co-authored-by: Marek <mail@marek.onl>

* fmt

* change some more variable names

* change a few missing generics

* fix checkpoint log issue

* rename more `prepared` vars `semantically_verified`

* fix test regex

* fix test regex 2

---------

Co-authored-by: Marek <mail@marek.onl>
2023-06-01 12:29:03 +00:00
Arya 8a9113c4e2
Leaves at least 1 block in the partial chain (#6552) 2023-04-20 21:06:23 +00:00
teor 166526a088
feat(ui): Add a terminal-based progress bar to Zebra (#6235)
* Implement Display and to_string() for NetworkUpgrade

* Add a progress-bar feature to zebrad

* Add the progress bar writer to the tracing component

* Add a block progress bar transmitter

* Correctly shut down the progress bar, and shut it down on an interrupt

* Make it clearer that the progress task never exits

* Add a config for writing logs to a file

* Add a progress-bar feature to zebra-network

* Add a progress bar for the address book size

* Add progress bars for never attempted and failed peers

* Add an optional limit and label to connection counters

* Add open connection progress bars

* Improve CheckpointList API and CheckpointVerifier debugging

* Add checkpoint index and checkpoint queue progress bars

* Security: Limit the number of non-finalized chains tracked by Zebra

* Make some NonFinalizedState methods available with proptest-impl

* Add a non-finalized chain count progress bar

* Track the last fork height for newly forked chains

* Add a should_count_metrics to Chain

* Add a display method for PartialCumulativeWork

* Add a progress bar for each chain fork

* Add a NonFinalizedState::disable_metrics() method and switch to using it

* Move metrics out of Chain because we can't update Arc<Chain>

* Fix: consistently use best chain order when searching chains

* Track Chain progress bars in NonFinalizedState

* Display work as bits, not a multiple of the target difficulty

* Handle negative fork lengths by reporting "No fork"

* Correctly disable unused fork bars

* clippy: rewrite using `match _.cmp(_) { ... }`

* Initial mempool progress bar implementation

* Update Cargo.lock

* Add the actual transaction size as a description to the cost bar

* Only show mempool progress bars after first activation

* Add queued and rejected mempool progress bars

* Clarify cost note is actual size

* Add tracing.log_file config and progress-bar feature to zebrad docs

* Derive Clone for Chain

* Upgrade to howudoin 0.1.2 and remove some bug workarounds

* Directly call the debug formatter to Display a Network

Co-authored-by: Arya <aryasolhi@gmail.com>

* Rename the address count metric to num_addresses

Co-authored-by: Arya <aryasolhi@gmail.com>

* Simplify reverse checkpoint lookup

Co-authored-by: Arya <aryasolhi@gmail.com>

* Simplify progress bar shutdown code

Co-authored-by: Arya <aryasolhi@gmail.com>

* Remove unused MIN_TRANSPARENT_TX_MEMPOOL_SIZE

* Document that the progress task runs forever

* Fix progress log formatting

* If progress-bar is on, log to a file by default

* Create missing directories for log files

* Add file security docs for running Zebra with elevated permissions

* Document automatic log file, spell progress-bar correctly

---------

Co-authored-by: Arya <aryasolhi@gmail.com>
2023-04-13 08:42:17 +00:00
Arya 571cbfba7a
change(state): Stop re-downloading blocks that are in non-finalized side chains (#6335)
* Adds 'Contains' request in state, and:

- adds finalized block hashes to sent_blocks

- replaces Depth call with Contains in sync, inbound, and block verifier

* removes unnecessary From impl

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* updates references to Request::Contains

* Renames zs::response::BlockLocation to KnownBlocks

* Updates AlreadyInChain error

* update docs for sent_hashes.add_finalized

* Update zebra-consensus/src/block.rs

Co-authored-by: teor <teor@riseup.net>

* Update comment for `sent_blocks` field in state service

* update KnownBlock request to check the non-finalized state before responding that a block is in the queue

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* update references to renamed method

* Clear sent_blocks when there's a reset

* Move self.finalized_block_write_sender.is_none() to can_fork_chain_at

* revert changes related to checking queue

---------

Co-authored-by: teor <teor@riseup.net>
2023-03-24 07:10:07 +00:00
Alfredo Garcia 4dedffebbc
feat(elasticsearch): Export block data to elasticsearch database (#6274)
* add initial ES support

* hide elasticsearch behind feture, add config

* make the builds and clippy happy

* move ES code to a function

* change database name

* fix database name and panic

* increase close to tip interval

* update deny for elastic

* remove a block clone

* try to fix builds

* fix builds 2

* refactor some imports and unwraps

* typo

* change argument of elasticsearch function

* move elastic call to the end of `commit_finalized_direct`

* get height from block
2023-03-13 21:13:30 +00:00
teor 9452487c61
fix(state): Fix minute-long delays in block verification after a chain fork (#6122)
* Split Chain fields into sections

* Replace Chain.sprout_note_commitment_tree with a lookup method

* Add TODOs

* Show full debug info when tests fail because chains aren't equal

* Print sprout and sapling tree Nodes as hex when debugging

* Correctly revert temporary finalized tip trees and anchors

* Fix tests

* Refactor removal functions

* Replace the Chain.sapling_note_commitment_tree field with a lookup method

* Replace the Chain.orchard_note_commitment_tree field with a lookup method

* Replace the Chain.history_tree field with a lookup method and remove redundant code

* Update comments

* Ignore clippy::unwrap_in_result

* Remove redundant fork() Result

* Put conditional code in blocks

* fastmod history_tree_at_tip history_block_commitment_tree zebra-state
2023-02-13 21:44:31 +00:00
teor 09836d2800
fix(clippy): Put Rust format variables inline (#5783)
* cargo clippy --fix --all-features --all-targets

With rustc 1.67.0-nightly (234151769 2022-12-03)

* cargo fmt --all
2022-12-08 01:05:57 +00:00
Arya eb0a2ef581
change(mempool): Contextually validates mempool transactions in best chain (#5716)
* updates comments

* adds check nullifier no dup fns for transactions

* Adds:
- check::anchors fn for tx iter
- TODO comments for unifying nullifiers and anchors checks
- new state request

Updates unknown anchor errors to accomodate tx-only check

Calls new state fn from transaction verifier

* updates check::anchors fns to use transactions

updates TransactionContextualValidity request to check sprout anchors

adds comment mentioning TransactionContextualValidity ignores UTXOs

* conditions new state req call on is_mempool

updates tests

* fix doc link / lint error

* checks for duplicate nullifiers with closures

* Update zebra-state/src/service/check/nullifier.rs

Co-authored-by: teor <teor@riseup.net>

* documents find_duplicate_nullifier params

moves if let statement into for loop

* renames new state req/res

* asserts correct response variant in tx verifier

* adds CheckBestChainTipShieldedSpends call in tx verifier to async checks

* re-adds tracing instrumentation to check::anchors fn

renames transaction_in_state to transaction_in_chain

* adds block/tx wrapper fns for anchors checks

* uses UnminedTx instead of transaction.hash()

deletes broken test

* updates new state req/res name

* updates tests and uses par_iter for anchors checks

* Updates check::anchors pub fn docs.

* Adds:
- comments / docs
- a TransactionError variant for ValidateContextError

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* moves downcast to From impl

rustfmt

* moves the ValidateContextError into an Arc

updates comments and naming

* leaves par_iter for another PR

* puts io::Error in an Arc

* updates anchors tests to call tx_anchors check

* updates tests to call tx_no_duplicates_in_chain

slightly improves formatting

* Update zebra-consensus/src/error.rs

Co-authored-by: teor <teor@riseup.net>

* moves Arc from HistoryError to ValidateContextError

Co-authored-by: teor <teor@riseup.net>
2022-11-30 04:40:15 +00:00
teor c812f880cf
cleanup(clippy): Use inline format strings (#5489)
* Inline format strings using an automated clippy fix

```sh
cargo clippy --fix --all-features --all-targets -- -A clippy::all -W clippy::uninlined_format_args
cargo fmt --all
```

* Remove unused & and &mut using an automated clippy fix

```sh
cargo clippy --fix --all-features --all-targets -- -A clippy::all -W clippy::uninlined_format_args
```
2022-10-27 13:25:18 +00:00
teor 211dbb437b
1. fix(rpc): Fix slow getblock RPC (verbose=1) using transaction ID index (#5307)
* Add RPC timing to zcash-rpc-diff

* Use transaction hash index for verbose block requests, rather than block data
2022-10-02 23:34:44 +00:00
teor bfdb29b757
3. change(state): Move the finalized queue to the StateService (#5152)
* Move the finalized block queue into the StateService

* Move the queued_blocks module to the state service

* Move QueuedFinalized into queued_blocks

* Move the queued_blocks tests into their own module

* Make the FinalizedState cloneable
2022-09-16 13:53:40 +00:00
teor 36a549ee3c
2. change(state): Run AwaitUtxo read requests without shared mutable chain state (#5107)
* Move AwaitUtxos next to the other shared writeable state requests

* Rename ReadResponse::Utxos to ReadResponse::AddressUtxos

```sh
fastmod Utxos AddressUtxos zebra*
```

* Rename an out_point variable to outpoint for consistency

* Rename transparent_utxos to address_utxos

```sh
fastmod transparent_utxos address_utxos zebra*
```

* Run AwaitUtxo without accessing shared mutable chain state

* Fix some incorrect comments

* Explain why some concurrent reads are ok

* Add a TODO

* Stop using self.mem in AwaitUtxo requests

* Update state service module documentation

* Move the QueuedBlock type into the queued_blocks module

* Explain how spent UTXOs are treated by the state

* Clarify how cached Chains impact state read requests

And move repeated comments to the module header.

* fastmod ChainUtxo BestChainUtxo zebra*

* Add an AnyChainUtxo request

* Make AwaitUtxo non-blocking
2022-09-16 04:13:26 +00:00
teor 00eee8652e
1. change(state): Run most StateService read requests without shared mutable chain state (#5132)
* Add TODOs for finalized blocks, non-finalized blocks, and reads

* Document how state requests use shared state

* Add a ReadStateService to the StateService

And cleanup service struct fields.

* Redirect Block and Transaction Requests to the ReadStateService

* Put AddressBalance in a consistent enum order

* Turn a repeated comment into documentation

* Tidy doc links

* Run Tip requests concurrently

* Remove some redundant timers

* Run Depth requests concurrently

* Run BlockLocator requests concurrently

* Move BlockLocator tests

* Run FindBlockHashes requests concurrently

* Run FindBlockHeaders requests concurrently

* Use a constant in documentation

Co-authored-by: Marek <mail@marek.onl>

* Link that constant correctly

* Expand block_locator() documentation

* Clarify the difference between tower::Buffers and the state's ordered queues

* Explain block locators better

Co-authored-by: Marek <mail@marek.onl>
2022-09-14 00:35:37 +00:00
Marek b8712d9a1e
feat(state): Send treestate from non-finalized state to finalized state (#4721)
* Add history trees for each height in non-fin state

* Refactor formatting

* Pass the treestate to the finalized state

I created a new structure `FinalizedBlockWithTrees` that wraps the
treestate and the finalized block. I did that because the original
`FinalizedBlock` is `Eq`, but `HistoryTree` can't be `Eq`.

This makes Zebra faster because:

1. The finalized state doesn't retrieve the treestate from the disk if
the non-finalized state supplies it.

2.The finalized state doesn't recompute the treestate if the
non-finalized state supplies it.

* Check block commitment before updating hist tree

* Store Sprout commitment trees in non-fin state

* Send trees for the root block to fin-state

When committing a block and sending the treestate from the non-finalized
state to the finalized state, Zebra was sending trees that correspond to
the tip block instead of trees that correspond to the root block of the
best chain. This commit fixes that.

* Refactor doc comments

* Refactor block finalization

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-09-06 09:32:54 +00:00
Marek a87b119a10
feat(state): Store history trees by height in the non-finalized state (#4928)
* Add history trees for each height in non-fin state

* Refactor formatting

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-08-29 06:59:25 +00:00
Conrado Gouvea 6fd750e168
build(deps): bump insta from 1.15.0 to 1.17.1 (#4884)
* build(deps): bump insta from 1.15.0 to 1.17.1

Bumps [insta](https://github.com/mitsuhiko/insta) from 1.15.0 to 1.17.1.
- [Release notes](https://github.com/mitsuhiko/insta/releases)
- [Changelog](https://github.com/mitsuhiko/insta/blob/master/CHANGELOG.md)
- [Commits](https://github.com/mitsuhiko/insta/compare/1.15.0...1.17.1)

---
updated-dependencies:
- dependency-name: insta
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* make zebra_test::init() return the insta drop guard

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-08-04 15:44:44 +00:00
teor 9b2185ad3d
Make FindHeaders and FindHashes run concurrently with state updates (#4826)
And speed up the contains_block_hash state query.
2022-07-27 06:26:17 +10:00
teor 6ad445eb97
1. fix(perf): Run CPU-intensive state updates in parallel rayon threads (#4802)
* Split disk reads from CPU-heavy Sprout interstitial tree cryptography

* Improve anchor validation debugging and error messages

* Work around a test data bug, and save some CPU

* Remove redundant checks for empty shielded data

* Skip generating unused interstitial treestates

* Do disk fetches and quick checks, then CPU-heavy cryptography

* Wrap HistoryTree in an Arc in the state

* Run CPU-intensive chain validation and updates in parallel rayon threads

* Refactor to prepare for parallel tree root calculations

* Run finalized state note commitment tree root updates in parallel rayon threads

* Update finalized state note commitment trees using parallel rayon threads

* Fix a comment typo and add a TODO

* Split sprout treestate fetch into its own function

* Move parallel note commitment trees to zebra-chain

* Re-calculate the tree roots in the same parallel batches

* Do non-finalized note commitment tree updates in parallel threads

* Update comments about note commitment tree rebuilds

* Do post-fork tree updates in parallel threads

* Add a TODO for parallel tree updates in tests

* Fix broken intra-doc links

* Clarify documentation for sprout treestates

* Sort Cargo.toml dependencies
2022-07-22 12:19:11 -04:00
teor 7b1d4527d4
add(log): Log the amount of time it takes to rebuild note commitment trees after a chain fork (#4795)
* Update comments about note commitment tree rebuilds

* Add info-level logs to time note commitment tree rebuilds

* Log the number of rebuilt blocks after a fork, and the time per block

* Move humantime formats to zebra-chain

* Use human-friendly time formatting
2022-07-22 09:17:34 +10:00
teor cbb3232769
Only fetch block headers from the database to answer headers requests (#4792) 2022-07-22 09:15:22 +10:00
Marek 485bac819d
change(state): Wrap commitment trees into `Arc` (#4757)
* Wrap Sprout note commitment trees into `Arc`

* Remove a redundant comment

* Rephrase a comment about chain forking

* Remove a redundant comment

The comment is not valid because Zebra uses `bridgetree::Frontier`s from
the `incrementalmerkletree` crate to represent its note commitment
trees. This `struct` does not support popping elements from the tree.

* Wrap Sapling commitment trees into `Arc`

* Remove unnecessary `as_ref`s

* Wrap Orchard commitment trees into `Arc`
2022-07-15 10:39:41 +10:00
Marek 32faa94fb4
fix(state): Update column family names to match Zebra's database design (#4639)
* Rename `block_by_height` to `block_header_by_height` in fin state

* Rename `tx_by_hash` to `tx_loc_by_hash` in both (non & fin) states

* Rename `utxo_by_outpoint` to `utxo_by_out_loc` in finalized state

* Reorder the column families so that they match the docs

* Update `struct Chain` in the RFCs

* Increment `DATABASE_FORMAT_VERSION` to 25

* Remove obsolete docs from `0004-asynchronous-script-verification.md`

* Remove an obsolete `TODO` from `disk_db.rs`

* Delete unused snapshots

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-06-30 23:00:23 +00:00
Alfredo Garcia 97fb85dca9
lint(clippy): add `unwrap_in_result` lint (#4667)
* `unwrap_in_result` in zebra-chain crate

* `unwrap_in_result` in zebra-script crate

* `unwrap_in_result` in zebra-state crate

* `unwrap_in_result` in zebra-consensus crate

* `unwrap_in_result` in zebra-test crate

* `unwrap_in_result` in zebra-network crate

* `unwrap_in_result` in zebra-rpc crate

* `unwrap_in_result` in zebrad crate

* rustfmt

* revert `?` and add exceptions

* explain some panics better

* move some lint positions

* replace a panic with error

* Fix rustfmt?

Co-authored-by: teor <teor@riseup.net>
2022-06-28 06:22:07 +00:00
Marek 2e50ccc8f3
fix(doc): Fix various doc warnings, part 2 (#4561)
* Fix the syntax of links in comments

* Fix a mistake in the docs

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Remove unnecessary angle brackets from a link

* Revert the changes for links that serve as references

* Revert "Revert the changes for links that serve as references"

This reverts commit 8b091aa9fab453e7d3559a5d474e0879183b9bfb.

* Remove `<` `>` from links that serve as references

This reverts commit 046ef25620ae1a2140760ae7ea379deecb4b583c.

* Don't use `<` `>` in normal comments

* Don't use `<` `>` for normal comments

* Revert changes for comments starting with `//`

* Fix some warnings produced by `cargo doc`

* Fix some rustdoc warnings

* Fix some warnings

* Refactor some changes

* Fix some rustdoc warnings

* Fix some rustdoc warnings

* Resolve various TODOs

Co-authored-by: teor <teor@riseup.net>

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-06-14 01:22:16 +00:00
Marek 6f896ef5a5
fix(doc): Fix the syntax of links in comments (#4494)
* Fix the syntax of links in comments

* Fix a mistake in the docs

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Remove unnecessary angle brackets from a link

* Revert the changes for links that serve as references

* Revert "Revert the changes for links that serve as references"

This reverts commit 8b091aa9fab453e7d3559a5d474e0879183b9bfb.

* Remove `<` `>` from links that serve as references

This reverts commit 046ef25620ae1a2140760ae7ea379deecb4b583c.

* Don't use `<` `>` in normal comments

* Don't use `<` `>` for normal comments

* Revert changes for comments starting with `//`

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2022-05-30 20:12:11 +00:00
teor 2439bed3d2
2. fix(state): index spending transaction IDs for each address (#4355)
* Make jobs that use cached state wait for state rebuilds

* Run jobs that need cached state even if the rebuild was skipped

* Fix missing dependencies

And update a TODO

* Split writing transaction indexes into transparent and shielded

* Split writing transparent indexes into created and spent

* Correctly populate spending address transaction ID indexes

* Increment the database format to rebuild address tx ID indexes

* Update non-finalized docs to prevent similar bugs

* Fix a comment

* Make jobs that use cached state wait for state rebuilds

* Run jobs that need cached state even if the rebuild was skipped

* Fix missing dependencies

And update a TODO

* refactor(ci): look for available disks instead of files changed

This ensure that if the constants.rs file was changed, we search for disks available in the whole repository with the same state.

If there's no disk available a rebuild is triggered depending the missing disk. And if there's a disk available, tests are run with this one.

* fix(ci): lwd syncs needs to wait for zebra disk rebuild

* docs(ci): use better comments on integration tests

* fix(ci): we must authenticate to GCP to find disks

* fix(ci): add needed permissions for google auth

* fix(ci): the output needs to be echoed

* imp(ci): reduce diff with main

* fix(ci): remove redundant dependency

Co-authored-by: teor <teor@riseup.net>

* fix(ci): also add `false` to the JSON object output

* fix(ci): hasty copy/paste

* force a push event

* fix(ci): standardize comments

* fix(ci): run disk rebuilds if no disk was found

* fix(ci): do not restrict on push

* fix(ci): build on any event if a cached disk is not found

* fix(ci): sync .patch file with changes on the workflow

Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
2022-05-20 02:22:01 +00:00
Marek 7c726b246d
feat(rpc): Implement `z_gettreestate` RPC (#3990)
* Impl the elementary structure of the `z_gettreestate` RPC

* Fix merging bugs

* Fix a merge bug

* Fix a merge bug

* Move a derive attribute

Co-authored-by: teor <teor@riseup.net>

* Clarify the support of negative heights

* Add Orchard note commitment trees to the response

* Add the time to the response

* Finalize the `z_gettreestate` RPC

* Add a note that verified blocks have coinbase height

* Refactor `from_str` for `HashOrHeight`

* Fix a mistake in the docs

Co-authored-by: teor <teor@riseup.net>

* Clarify request types

Co-authored-by: teor <teor@riseup.net>

* Simplify `hash_or_height` conversion to height

Co-authored-by: teor <teor@riseup.net>

* Add a TODO about optimization

Co-authored-by: teor <teor@riseup.net>

* Add a doc comment

* Make sure Sapling & Orchard trees don't get mixed up

* Serialize Sapling commitment trees

* Refactor some comments

* Serialize Orchard commitment trees

* Serialize block heights

* Simplify the serialization of commitment trees

* Remove the block time from the RPC response

* Simplify the serialization of block heights

* Put Sapling & Orchard requests together

* Remove a redundant TODO

* Add block times to the RPC response

* Derive `Clone, Debug, Eq, PartialEq` for `GetTreestate`

Co-authored-by: teor <teor@riseup.net>

* Derive `Clone`, `Debug`, `Eq` and `PartialEq` for `SerializedTree`

* Document the fields of `GetTreestate`

* Skip the serialization of empty trees

This ensures compatibility with `zcashd` in the `z_gettreestate` RPC.

* Document the `impl` of `merkle_tree::Hashable` for nodes

* Make the structure of the JSON response consistent with `zcashd`

* Derive `Eq` for nodes

Co-authored-by: teor <teor@riseup.net>

* Convert Sapling commitment trees to a format compatible with zcashd

* Refactor the conversion of Sapling commitment trees

* Refactor some comments

* Refactor comments

* Add a description of the conversion

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>

* Fix comment indenting

* Document the conversion between the dense and sparse formats

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2022-05-12 07:00:12 +00:00
teor c2430c6f45
feat(rpc): implement the get_address_tx_ids RPC method query (#4119)
* Add a finalized state txids query

* Add an address transaction IDs query, without height filters

* Connect the address transaction ID query to the RPC

* Basic filtering of address transaction IDs by height range

* Add a network and range argument to the getaddresstxids test

* Test all block range combinations for mainnet

* Fix a file descriptor limit error

* Optimise seeking the first transaction for an address

The first transaction's location is part of the address location.

* Filter finalized address transaction IDs by height range

* Filter non-finalized address transaction IDs by the height range

* Fix up snapshot tests for the new height range API
2022-04-21 20:19:26 +00:00
Dimitris Apostolou ad971bbdd5
Fix typos (#4112) 2022-04-20 00:26:58 +00:00
teor 65b94f7e50
9. feat(state): add a query function for transparent UTXOs (#4111)
* Add address UTXOs query functions, but without the transaction IDs

* Return transaction IDs along with address UTXOs

* Add a convenience type for address UTXOs

* Add output addresses to the convenience method

* Fix query documentation

* Rename the chain transaction IDs method
2022-04-19 13:34:53 +00:00
teor 8e29219565
8. feat(state): add a query function for transparent address balances (#4097)
* Make address index types consistent

* Simplify non-finalized address index updates

* Update snapshots for address index queries

* Simplify non-finalized UTXO query

* Add a query method for non-finalized address balance changes

* Add a query method for finalized state address balances

* Add a query function for address balances

* Refactor balance queries to make them repeatable

* Retry interrupted finalized balance queries

* Pop chain root blocks until it matches the finalized tip

* Avoid cloning the chain

It has already been cloned by the watch receiver

* Refactor and fix documentation of the balance query code
2022-04-14 13:34:31 +00:00
teor e49c1d7034
feat(state): add transparent address indexes to the non-finalized state (#4022)
* Derive Hash for transparent address index types

* Expose some types used by transparent address indexes

* Add an empty transparent transfers type for transparent address indexes

* Update TransparentTransfers with created UTXOs

* Add spent transparent outputs to ContextuallyValidBlock

* Update TransparentTransfers with spent transparent outputs

* Ignore missing spent outputs, so that tests pass

* Remove empty TransparentTransfers after a spend revert

* Update TransparentTransfers with creating and spending transaction IDs

* Ignore duplicate created UTXOs, so that tests pass

* Add some TODO comments

* Remove accidental doctest formatting

* Add address transfers index accessor methods

* Use TransactionLocation in the non-finalized state

* Apply more address index assertions to production code

* Refactor deeply nested code and apply more assertions

* Return UTXOs in chain order

* Return transaction hashes in chain order

* Stop indexing each transparent output multiple times

* Run some more asserts during tests

* Tidy TODO comments

* Fix an incorrect assert condition

* Use OrderedUtxos so that spent UTXOs can be stored in chain order

* Update tests to use OrderedUtxos

* Update the index API for the getaddressutxos query

* Remove redundant arguments in tests

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-04-12 17:21:46 +00:00
Conrado Gouvea e7c0a78d4d
feat(rpc): add getrawtransaction (#3908)
* feat(rpc): add getrawtransaction

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Apply suggestions from code review

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* address review comments

* move SerializedTransaction to the right module

* Use try_into() instead of 'as'

* add proptests

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-03-24 09:45:37 +00:00
teor 88ab6deeac
4. feat(state): support Request::Transaction in ReadStateService (#3865)
* Add request metrics to ReadStateService

Add a field to distinguish StateService and ReadStateService metrics.
Add missing StateService request metrics.

* Refactor state transaction lookup so it can be shared between services

* Implement ReadState Request::Transaction
2022-03-17 18:48:13 +00:00
teor 67b367929c
3. fix(state): prevent watch channel deadlocks in the state (#3870)
* Revert "Drop the Chain watch channel lock before accessing the finalized state"

This reverts commit 8870944d1323fc31e2d009b9938623de6fcbe50f.

* Add a WatchReceiver wrapper that always clones the borrowed watch data

This avoids deadlocks, by holding the read lock for as short a time as possible.

* Drop the shared Arc<Chain>s as quickly as possible

This reduces memory usage.

* Make read::block more flexible, by accepting any AsRef<Chain>

* Make the block method docs consistent

* Avoid livelocks by explicitly dropping the borrow after the clone
2022-03-17 00:37:44 +00:00
teor 5c62dd62cd
1. feat(state): Use ReadStateService for RPCs (#3847)
* Use the read-only state service for RPCs

* Refactor non-finalized block lookup into Chain

* Implement the read-only state block request

* Drop the Chain watch channel lock before accessing the finalized state
2022-03-15 16:50:28 -03:00
Marek 38a2bcb042
feat(shielded): Store Sapling & Orchard note commitment trees in finalized and non-finalized state (#3818)
* Query Sapling & Orchard trees by height in the finalized state

* Add Sapling & Orchard trees to the non-finalized state

* Add a TODO about concurrent read-only access to Sprout tree

Co-authored-by: teor <teor@riseup.net>

* Update the database format version

* Keep only the most recent Sprout tree in the database

* Check that the database returns empty trees for the genesis block

* Assert that the database returns the highest trees

* Document how to update insta snapshots

* Add note commitment tree insta snapshot tests

* Add comments about cached tree roots in snapshots

* Add snapshot data for sapling and orchard trees

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
2022-03-15 05:18:18 +00:00
teor 9ad47d1081
7. feat(state): add a read-only state service stub (#3835)
* Add state service module docs and cleanup

* Move and add finalized state methods

* Add chain and non-finalized state methods

* Cleanup methods and imports

* Create a ReadStateService type

* Add a stub service implementation

* Add a TODO

* Update ReadStateService request stubs with RPC names and tickets

* Documentation updates

* Make RPC State generic bounds accept a buffered state and a read-only state

* Doc updates

* Add missing proptest-impl feature in RPC dev dependencies
2022-03-11 10:58:22 -03:00
Janito Vaqueiro Ferreira Filho b6a0fcc44c
C. Restore `self` ownership in `Chain::push` method (#3826)
* Drop chain if it becomes invalid

Avoid returning a chain that shouldn't be used again by having the
method take ownership of `self` and dropping it on error.

* Update documentation and comments

Restore them to be closer to how they were on `main`.
2022-03-10 10:02:19 +00:00
teor 1fb646db7e
B. refactor(state): prepare non-finalized chains for shared read-only access (#3811)
* fix(shielded): use RwLock for note commitment tree root caches

* Prepare the non-finalized state for read-only state

* fix(state): performance tweak for chain forks

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-03-10 02:44:39 +00:00
teor 22b8a6003c
3. refactor(state): move database reads and writes to a new zebra_db module (#3579)
* refactor(state): move disk_db reads to a new zebra_db module

* refactor(state): make finalized value pool method names consistent

* refactor(state): split database writes into the zebra_db module

* refactor(state): move the block batch method to DiskWriteBatch

* refactor(state): actually add the zebra_db module

Unfortunately, I've lost the interim changes to this file,
so this commit might be the only one that compiles.

* refactor(state): add a newly created file to the cached state CI job
2022-02-23 00:43:41 +00:00
Conrado Gouvea f6edcc4c92
replace unmantained multiset with mset (#3595)
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-02-22 01:17:21 +00:00
teor 9be13a4fb7
2. Route peer requests based on missing inventory (#3465)
* feat(network): send notfound messages to the inventory registry

* refactor(network): move the inventory filter into an async function

* feat(network): avoid routing requests to peers that are missing inventory

* test(network): advertised routing is independent of numeric address value

* test(network): peer set routes requests to peers not missing that inventory

* test(network): peer set fails requests if all ready peers are missing that inventory

* fix(clippy): needless-borrow in the peer set

* fix(lint): remove redundant trailing commas in macro calls

There is no clippy lint for this, maybe because some macros
are sensitive to trailing commas.
(But not the ones changed in this commit.)

* test(network): check the exact number of inventory peers

* doc(network): explain why we ignore inventory send failures

* docs(network): explain why a channel error is ignored
2022-02-08 01:16:41 +00:00
Conrado Gouvea 4aeabd0b52
Fix interstitial sprout anchors check (#3283)
* Fix interstitial Sprout anchors check

* Update state docs; add sprout_trees_by_anchor to comparisons

* Update book/src/dev/rfcs/0005-state-updates.md

Co-authored-by: Marek <mail@marek.onl>

* Rename `interstitial_roots` to `interstitial_trees`

* Document consensus rules

* Refactor the docs

* Improve the docs for consensus rules

* Update reference to cached state

* Update zebra-state/src/service/check/anchors.rs

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* Fix formatting

Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-01-18 20:18:49 +00:00
Deirdre Connolly b973b7a622
Checking of Sprout anchors in non-finalized state (#3123)
* Do prelim checking of Sprout anchors in non-finalized state

Does not check intra-transaction interstitial states yet

* Populate sprout anchors to allow other state tests to pass

* Preliminary interstitial sprout note commitment tree anchor checks implementation

* Make sure only prior anchors are checked in the same transaction

* Add tests

* Refactor a comment

* Refactor rustdoc

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* Use the first `JoinSplit`s from mainnet

* Print debug messages

* Use correct blocks for the tests

Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2021-12-09 16:50:26 +00:00
Deirdre Connolly e6ffe374d4
Validate sapling, orchard anchors (#3084)
* Add Transaction::sprout_joinsplits()

* Add Anchor variants to ValidateContextError

* Make Chain anchor collections pub(crate)

* tracing::instrument several methods in state

* Add contains_*_anchors methods to FinalizedState

* Add check::anchors module and function

* Verify that anchors_refer_to_earlier_treestates in when updating chains in non-finalized state

* Update zebra-state/src/service/check/anchors.rs

Co-authored-by: teor <teor@riseup.net>

* Add anchors() to sapling::ShieldedData

* Add sapling_anchors() to Transaction

* Use Transaction::sapling_anchors() in the anchors_refer_to_earlier_treestates() check

* Whoops, itertools

* Add a comment for improvement

Co-authored-by: teor <teor@riseup.net>

* Add & use a cfg(test) method on FinalizedState to prep test state with anchors to allow other tests to pass contextual checks

* Allow test nullifier checks to pass by populating anchor sets, allowing test anchor checks to pass

* Add mainnet block 419202 and its sapling note commitment tree root to test vectors

* Test sapling anchor verification using the first few Sapling blocks data

* Correct comment

* assert_eq instead of assert(bool)

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>

* Update zebra-state/src/service/non_finalized_state.rs

Co-authored-by: teor <teor@riseup.net>

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2021-11-30 16:05:35 +00:00
Marek 3c9ad89018
Add Sprout anchors to `zebra-state` (#3100)
* Add Sprout anchors to the state

* Update zebra-state/src/service/non_finalized_state/chain.rs

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* Return new types of note commitments from Sprout transactions

* Refactor the tests

* Refactor some comments

Co-authored-by: teor <teor@riseup.net>

* Increment `DATABASE_FORMAT_VERSION`

* Update `test.yml` with the new image name

* Refactor the `version = 5` transaction description

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* Update comment

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: teor <teor@riseup.net>
2021-11-30 11:05:58 +01:00
teor 1df3bdb089
Check MAX_BLOCK_SIGOPS in the block verifier (#3049)
* Cleanup a function that calls zcash_script

* Remove zebra_test::prelude macros that conflict with the Rust prelude

* Add sigops count support to zebra-script

* Check MAX_BLOCK_SIGOPS in the block verifier

* Test MAX_BLOCK_SIGOPS on generated and historic blocks

* Add SAFETY comments for all unsafe zebra-script code

* Explain where the consensus rule comes from

* Remove unused pretty_assertions dependency

* Allow large test block generation functions with the proptest-impl feature

* Replace `as` with `try_into` for integer conversions in unsafe code

* Expand SAFETY comments
2021-11-15 20:55:32 +00:00
teor 0b82298645 Stop dropping queued block senders without sending a response
This causes a rare panic, because Zebra expects every queued sender
to send a response.
2021-10-07 00:12:45 -04:00
teor c7c1a8a469 Remove known UTXOs when pruning blocks 2021-10-07 00:12:45 -04:00