Commit Graph

80 Commits

Author SHA1 Message Date
Alfredo Garcia eb07bb31d6
rename(state): Rename state verifiers and related code (#6762)
* rename verifiers

* rename `PreparedBlock` to `SemanticallyVerifiedBlock`

* rename `CommitBlock` to `SemanticallyVerifiedBlock`

* rename `FinalizedBlock` to `CheckpointVerifiedBlock`

* rename `CommitFinalizedBlock` to `CommitCheckpointVerifiedBlock`

* rename `FinalizedWithTrees` to `ContextuallyVerifiedBlockWithTrees`

* rename `ContextuallyValidBlock` to `ContextuallyVerifiedBlock`

* change some `finalized` variables or function arguments to `checkpoint_verified`

* fix docs

* document the difference between `CheckpointVerifiedBlock` and `ContextuallyVerifiedBlock`

* fix doc links

* apply suggestions to request

Co-authored-by: Marek <mail@marek.onl>

* apply suggestions to service

Co-authored-by: Marek <mail@marek.onl>

* apply suggestions to finalized_state.rs and write.rs

Co-authored-by: Marek <mail@marek.onl>

* fmt

* change some more variable names

* change a few missing generics

* fix checkpoint log issue

* rename more `prepared` vars `semantically_verified`

* fix test regex

* fix test regex 2

---------

Co-authored-by: Marek <mail@marek.onl>
2023-06-01 12:29:03 +00:00
teor b1ce0e0894
fix(security): Stop panicking on state RPC or block requests with very large heights (#6699)
* fix panic in height

* Update security comments and turn literals into constants

* Add a test-only assertion that the maximum output index can't ever be reached

* Fix a MISSING_BLOCK_ERROR_CODE that was mistakenly 0

* Add production RPC tests with excessive heights

* Add and update snapshots for production RPCs

* Add excessive height tests for most getblocktemplate RPCs

* Use correct snapshot names by running `cargo insta review`

---------

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2023-05-17 15:13:12 +00:00
Arya 3779eb19a6
Tests error conversions (#6665) 2023-05-16 21:06:46 +00:00
Arya 6fdd02220e
fix(rpc): Omit transactions with transparent coinbase spends that are immature at the next block height from block templates (#6510)
* Adds `maturity_height` to VerifiedUnminedTx

Filters out transactions that are invalid at next_block_height in getblocktemplate method

* Adds unit testing

* rustfmt

* rejects txs with immature coinbase spends from mempool

* Condenses fns for transparent coinbase spend check

* Updates calls to VerifiedUnminedTx::new()

* Update zebra-chain/src/transparent/utxo.rs

* Applies suggestions from code review
2023-04-18 03:43:39 +00:00
Arya 8ba3fd921a
change(rpc): Add confirmations to getrawtransaction method response (#6287)
* adds confirmation field to getrawtransaction

* Updates tests

* Adds comment about correctness

* Apply suggestion revisions from review

* fixes overflow bug and adds test for valid confirmations value

* Update test to check that confirmations isn't too high

* Update transaction request to return confirmations

* Applies suggestions from PR review

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* fixes test

* restore derives that were lost in a bad merge

---------

Co-authored-by: teor <teor@riseup.net>
2023-03-26 23:53:44 +00:00
Arya 571cbfba7a
change(state): Stop re-downloading blocks that are in non-finalized side chains (#6335)
* Adds 'Contains' request in state, and:

- adds finalized block hashes to sent_blocks

- replaces Depth call with Contains in sync, inbound, and block verifier

* removes unnecessary From impl

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* updates references to Request::Contains

* Renames zs::response::BlockLocation to KnownBlocks

* Updates AlreadyInChain error

* update docs for sent_hashes.add_finalized

* Update zebra-consensus/src/block.rs

Co-authored-by: teor <teor@riseup.net>

* Update comment for `sent_blocks` field in state service

* update KnownBlock request to check the non-finalized state before responding that a block is in the queue

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* update references to renamed method

* Clear sent_blocks when there's a reset

* Move self.finalized_block_write_sender.is_none() to can_fork_chain_at

* revert changes related to checking queue

---------

Co-authored-by: teor <teor@riseup.net>
2023-03-24 07:10:07 +00:00
teor 4daedbc789
fix(consensus): Check that Zebra's state contains the social consensus chain on startup (#6163)
* Allow missing docs directly on derived error types

* Make Request::BestChainBlockHash redirect to the ReadStateService

* Re-write the checkpoint_sync documentation based on the latest consensus rules

* Expose the underlying iterator for CheckpointList

* Validate existing state block hashes at startup, but ignore the result

* Monitor state block hash checkpoint task in the start command

* Fix indentation

* Make logging consistent

* Explain the config needed for full security

* Tidy required checkpoints docs, expand other docs

* Add security and deprecation changelog entries

* Replace task handle vector with a struct

* Add a test that this consensus-critical code actually runs and finishes

* Make some state methods and types available in tests

* Add missing docs

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2023-02-21 05:30:29 +00:00
Alfredo Garcia eb66f4b1a3
feat(rpc): populate some getblocktemplate RPC block header fields using the state best chain tip (#5659)
* populate block height

* populate cur_time

* populate min_time

* populate capabilities

* populate last_block_hash

* create read state request for getblocktemplate

* refactor to get difficulty fields more properly

* populate bits and target fields

* fix tests

* add target and bits documentation

* docs

* fix docs

* docs

* remove metrixs counter calls

* apply some suggestions from code review

* hide some code behind feature

* simplify the service

* fix error handling

* remove comment

* fox doc

* panic if we dont have enough state

* bring tip data from the state

* make proposal empty

* fix time

* fix docs, consensus rules

* remove non used anymore fn

* remove another non used fn

* remove no needed change

* remove more unused changes

* remove unused anymore change

* apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* fix build and snapshots

* apply testnet consensus rule

* fix test

* rustfmt

* remove time as allowed field to be modified by the miner if mining minimum difficulty block

* move all times to before calculating difficulty

* do some cleanup

* Adjust times so the whole time range is a testnet minimum difficulty block

* Return a GetBlockTemplateChainInfo struct from the difficulty calculation

* Add a Zebra-only max_time field to the getblocktemplate RPC

Co-authored-by: teor <teor@riseup.net>
2022-11-28 09:06:32 +00:00
teor c4fad29824
fix(sync): Pause new downloads when Zebra reaches the lookahead limit (#5561)
* Use correct release for getblocktemplate config

* Include at least 2 full checkpoints in the lookahead limit

* Increase full sync timeout to 36 hours

* Only log "synced block height too far ahead of the tip" once

* Replace AboveLookaheadHeightLimit error with pausing the syncer

* Use AboveLookaheadHeightLimit for blocks a very long way from the tip

* Also add the getblocktemplate config, and fix the test message

* Remove an outdated TODO comment

* Allow syncing again when a small number of blocks are in the queue

* Allow some dead code
2022-11-09 04:42:04 +00:00
teor 343c5e68d4
change(state): Write finalized blocks to the state in a separate thread, to avoid network and RPC hangs (#5134)
* Add a new block commit task and channels, that don't do anything yet

* Add last_block_hash_sent to the state service, to avoid database accesses

* Update last_block_hash_sent regardless of commit errors

* Rename a field to StateService.max_queued_finalized_height

* Commit finalized blocks to the state in a separate task

* Check for panics in the block write task

* Wait for the block commit task in tests, and check for errors

* Always run a proptest that sleeps once

* Add extra debugging to state shutdowns

* Work around a RocksDB shutdown bug

* Close the finalized block channel when we're finished with it

* Only reset state queue once per error

* Update some TODOs

* Add a module doc comment

* Drop channels and check for closed channels in the block commit task

* Close state channels and tasks on drop

* Remove some duplicate fields across StateService and ReadStateService

* Try tweaking the shutdown steps

* Update and clarify some comments

* Clarify another comment

* Don't try to cancel RocksDB background work on drop

* Fix up some comments

* Remove some duplicate code

* Remove redundant workarounds for shutdown issues

* Remode a redundant channel close in the block commit task

* Remove a mistaken `!force` shutdown condition

* Remove duplicate force-shutdown code and explain it better

* Improve RPC error logging

* Wait for chain tip updates in the RPC tests

* Wait 2 seconds for chain tip updates before skipping them

* Remove an unnecessary block_in_place()

* Fix some test error messages that were changed by earlier fixes

* Expand some comments, fix typos

Co-authored-by: Marek <mail@marek.onl>

* Actually drop children of failed blocks

* Explain why we drop descendants of failed blocks

* Clarify a comment

* Wait for chain tip updates in a failing test on macOS

* Clean duplicate finalized blocks when the non-finalized state activates

* Send an error when receiving a duplicate finalized block

* Update checkpoint block behaviour, document its consensus rule

* Wait for chain tip changes in inbound_block_height_lookahead_limit test

* Wait for the genesis block to commit in the fake peer set mempool tests

* Disable unreliable mempool verification check in the send transaction test

* Appease rustfmt

* Use clear_finalized_block_queue() everywhere that blocks are dropped

* Document how Finalized and NonFinalized clones are different

* Use the same check as commit_finalized() for finalized block heights

Co-authored-by: Marek <mail@marek.onl>

Co-authored-by: Marek <mail@marek.onl>
2022-09-28 16:09:56 +00:00
Arya ec115e930f
change(zebrad): opens the database in a blocking tokio thread, which allows tokio to run other tasks (#5228)
* use spawn_blocking to run zebra_state::init from start cmd

* uses zebra_state::spawn_init in copy-state command

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-09-26 15:45:42 +00:00
teor 00eee8652e
1. change(state): Run most StateService read requests without shared mutable chain state (#5132)
* Add TODOs for finalized blocks, non-finalized blocks, and reads

* Document how state requests use shared state

* Add a ReadStateService to the StateService

And cleanup service struct fields.

* Redirect Block and Transaction Requests to the ReadStateService

* Put AddressBalance in a consistent enum order

* Turn a repeated comment into documentation

* Tidy doc links

* Run Tip requests concurrently

* Remove some redundant timers

* Run Depth requests concurrently

* Run BlockLocator requests concurrently

* Move BlockLocator tests

* Run FindBlockHashes requests concurrently

* Run FindBlockHeaders requests concurrently

* Use a constant in documentation

Co-authored-by: Marek <mail@marek.onl>

* Link that constant correctly

* Expand block_locator() documentation

* Clarify the difference between tower::Buffers and the state's ordered queues

* Explain block locators better

Co-authored-by: Marek <mail@marek.onl>
2022-09-14 00:35:37 +00:00
Alfredo Garcia 769d069d0a
feat(state): delete old database directories (#4586)
* delete old database directories

* check if state directory exists

* skip deleting when ephemeral

* split `check_and_delete_old_databases`

* move `check_and_delete_old_databases` to state

* spawn `check_and_delete_old_databases`

* simplity a bit

* fix(state): only delete old database directories inside the cache directory (#4631)

* Add function comments, tweak log

* Simplify version parsing

* Use spawn_blocking to launch the task on a separate thread, do the cleanup last

* Abort the cleanup task when Zebra exits

* Split directory deletion into its own function, handle ownership

* Rename cache_dir to state_dir

* If an outdated state directory is outside the cache directory, don't delete it

* Minimise diffs

* add test

* fix typos

Co-authored-by: teor <teor@riseup.net>

* add `canonicalize` to test regex

* add another match to test

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-06-21 00:59:51 +00:00
Marek b91aaf7863
Use `config.toml` instead of inner attributes for private links (#4627) 2022-06-15 23:21:27 +00:00
Marek cc75c3f5f9
fix(doc): Fix various doc warnings, part 3 (#4611)
* Fix the syntax of links in comments

* Fix a mistake in the docs

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Remove unnecessary angle brackets from a link

* Revert the changes for links that serve as references

* Revert "Revert the changes for links that serve as references"

This reverts commit 8b091aa9fab453e7d3559a5d474e0879183b9bfb.

* Remove `<` `>` from links that serve as references

This reverts commit 046ef25620ae1a2140760ae7ea379deecb4b583c.

* Don't use `<` `>` in normal comments

* Don't use `<` `>` for normal comments

* Revert changes for comments starting with `//`

* Fix some warnings produced by `cargo doc`

* Fix some rustdoc warnings

* Fix some warnings

* Refactor some changes

* Fix some rustdoc warnings

* Fix some rustdoc warnings

* Resolve various TODOs

Co-authored-by: teor <teor@riseup.net>

* Fix some unresolved links

* Allow links to private items

* Fix some unresolved links

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-06-15 03:57:19 +00:00
teor be4e065afb
change(rpc): Simplify RPC types and add documentation (#4218)
* Simplify RPC types and add documentation

* Derive serde traits in production code
2022-05-03 04:10:21 +00:00
teor e49c1d7034
feat(state): add transparent address indexes to the non-finalized state (#4022)
* Derive Hash for transparent address index types

* Expose some types used by transparent address indexes

* Add an empty transparent transfers type for transparent address indexes

* Update TransparentTransfers with created UTXOs

* Add spent transparent outputs to ContextuallyValidBlock

* Update TransparentTransfers with spent transparent outputs

* Ignore missing spent outputs, so that tests pass

* Remove empty TransparentTransfers after a spend revert

* Update TransparentTransfers with creating and spending transaction IDs

* Ignore duplicate created UTXOs, so that tests pass

* Add some TODO comments

* Remove accidental doctest formatting

* Add address transfers index accessor methods

* Use TransactionLocation in the non-finalized state

* Apply more address index assertions to production code

* Refactor deeply nested code and apply more assertions

* Return UTXOs in chain order

* Return transaction hashes in chain order

* Stop indexing each transparent output multiple times

* Run some more asserts during tests

* Tidy TODO comments

* Fix an incorrect assert condition

* Use OrderedUtxos so that spent UTXOs can be stored in chain order

* Update tests to use OrderedUtxos

* Update the index API for the getaddressutxos query

* Remove redundant arguments in tests

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-04-12 17:21:46 +00:00
Deirdre Connolly 0821e30623
Repoint zebra image links to our new zfnd.org site for now (#3949)
* Repoint zebra image links to our new zfnd.org site for now

* Remove images/

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-03-27 23:42:47 +00:00
teor 39dfca8e64
5. change(state): split ReadStateService requests into a ReadRequest enum (#3866)
* Split out ReadRequest and ReadResponse state service enums

* Simplify RPC test vectors

* Split state requests into Request and ReadRequest

* Make zebra-rpc use the new state ReadRequest
2022-03-17 22:59:46 +00:00
teor 86b3315d8a
5. refactor(state): split the state service into modules (#3778)
* Move the legacy chain check to the `check` module

And move `populated_state` to the `arbitrary` module.

* Cleanup imports

* Document the state service struct

* Split state block iter into its own module
2022-03-10 20:40:48 +00:00
Alfredo Garcia 833560411f
feature(rpc): implement getblock api call (#3707)
* feature(rpc): start adding a `getblock` method

* fix(rpc): replace oneshot

* fix(rpc): replace a panic with error

* fix(rpc): fix test

* feature(rpc): add hex to response

* refactor(rpc): use generic instead of alias

* docs(rpc): improve docs for getblock method

* test(rpc): add a test for getblock method

* deps(rpc): remove non needed tower features

Co-authored-by: teor <teor@riseup.net>

* docs(rpc): add a note to getblock doc

* refactor(rpc): replace alias

* fix(rpc): use `zcash_serialize_to_vec()` instead of logging format

* tests(rpc): add network argument to `populated_state()`

* refactor(rpc): use an error for state service readiness

* fix(rpc): add parameter

* fix(rpc): clippy

* nit(rpc): remove new line from imports

* fix(rpc): remove commented code

* fix(rpc): simplify error

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* Use a `SerializedBlock` type to help serializing blocks (#3725)

* Create a `SerializedBlock` helper type

Create a type that can be used as a byte slice, but is guaranteed to
represent a valid block.

* Use `into_iter` instead of `iter`

There's no need to borrow the elements, they can be moved out directly.
This will be necessary because `&Arc<T>` doesn't implement `Borrow<T>`,
so a `SerializedBlock` can't be built directly from an `&Arc<Block>`.

* Use `SerializedBlock` in `GetBlock`

Make the type stricter to avoid storing possibly invalid values. The
bytes are still serialized as a hexadecimal string, through the usage of
`hex`.

The `serde::Deserialize` can't be derived because `hex` requires the
type to also implement `FromHex`.

* feature(rpc): add suggestions from code review

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* tests(rpc): make sure mempool has no requests in get_block test

* fix(rpc): change height argument type in getblock method

* fix(rpc): rustfmt

* fix(rpc): replace panic

* fix(rpc): change getblock response

* fix(rpc): fix lightwalletd test

* tests(rpc): add a getblock error test

* fix(rpc): try another regex

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2022-03-10 01:12:41 +00:00
teor 32017f992b
2. refactor(state): move all RocksDB API calls to the disk_db module (#3578)
* refactor(state): move RocksDB-specific initialization to a new module

* refactor(state): move RocksDB-specific shutdown to a new module

* refactor(state): temporarily allow RocksDB-specific reads and writes, without a new module

Unlike the last few commits, this one actually compiles.

* refactor(state): add a DiskWriteBatch wrapper for RocksDB writes

* refactor(state): move finalized state test methods to a test module
2022-02-22 12:59:44 +00:00
teor 00aa5d96a3
Consolidate standard lints into a cargo config file (#3386)
* Move standard lints into .cargo/config.toml

* Ignore "wrong self convention" in a futures-based trait

This lint might only trigger on beta or nightly at the moment.

* Warn if future incompatibile code is added to Zebra

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-01-24 16:25:06 +00:00
Deirdre Connolly 89b0403582
Enforce Rust edition 2021 (#3332)
* Rust edition 2021: zebra-network, cargo fix --edition and clippy --fix

* Rust edition 2021: zebra-chain, cargo fix --edition

* Rust edition 2021: tower-batch, cargo fix --edition

* Rust edition 2021: tower-fallback, cargo fix --edition

* Rust edition 2021: zebra-client, cargo fix --edition

* Rust edition 2021: zebra-consensus, cargo fix --edition

* Rust edition 2021: zebra-rpc, cargo fix --edition

* Rust edition 2021: zebra-state, cargo fix --edition

* Rust edition 2021: zebra-state, cargo fix --edition

* Rust edition 2021: zebra-test, cargo fix --edition

* Rust edition 2021: zebra-utils, cargo fix --edition

* Rust edition 2021: zebrad, cargo fix --edition

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-01-14 12:10:18 +00:00
Janito Vaqueiro Ferreira Filho 50a5728d0b
Test if the mempool storage is cleared (#2815)
* Move mempool tests into `tests::vector` sub-module

Make it consistent with other test modules and prepare for adding
property tests.

* Reorder imports

Make it consistent with the general guidelines followed on other
modules.

* Export `ChainTipBlock` and `ChainTipSender`

Allow these types to be used in other crates for testing purposes.

* Derive `Arbitrary` for `ChainTipBlock`

Make it easy to generate random `ChainTipBlock`s for usage in property
tests.

* Refactor to move test methods into `tests` module

Reduce the repeated test configuration attributes and make it easier to
see what is test specific and what is part of the general
implementation.

* Add a `Mempool::dummy_call` test helper method

Performs a dummy call just so that `poll_ready` gets called.

* Use `dummy_call` in existing tests

Replace the custom dummy requests with the helper method.

* Test if the mempool is cleared on chain reset

A chain reset should force the mempool storage to be cleared so that
transaction verification can restart using the new chain tip.

* Test if mempool is cleared on syncer restart

If the block synchronizer falls behind and then starts catching up
again, the mempool should be disabled and therefore the storage should
be cleared.
2021-10-01 14:44:25 +00:00
Marek 061ad55144
Sneak chain_tip_change into mempool (#2785)
* Pass ChainTipChange to the mempool

* Fix nits
2021-09-21 17:06:52 +00:00
teor b6fe816473
Add a `ChainTipChange` type to `await` chain tip changes (#2715)
* Rename ChainTipReceiver to CurrentChainTip

`fastmod ChainTipReceiver CurrentChainTip zebra*`

* Update chain tip documentation and variable names

* Basic chain tip change implementation, without resets

Also includes the following name changes:
```
fastmod CurrentChainTip LatestChainTip zebra*
fastmod chain_tip_receiver latest_chain_tip zebra*
```

* Clarify the difference between `LatestChainTip` and `ChainTipChange`
2021-09-01 22:31:16 +00:00
teor d2e14b22f9
Refactor BestTipHeight into a generic ChainTip sender and receiver (#2676)
* Rename BestTipHeight so it can be generalised to ChainTipSender

`fastmod BestTipHeight ChainTipSender zebra*`

For senders:
`fastmod best_tip_height chain_tip_sender zebra*`

For receivers:
`fastmod best_tip_height chain_tip_receiver zebra*`

* Rename best_tip_height module to chain_tip

* Wrap the chain tip watch channel in a ChainTipReceiver type

* Create a ChainTip trait to avoid tricky crate dependencies

And add convenience impls for optional and empty chain tips.

* Use the ChainTip trait in zebra-network

* Replace `Option<ChainTip>` with `NoChainTip`

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-08-27 11:34:33 +10:00
teor 3d792f7195
Validate spends of transparent coinbase outputs (#2525)
* Validate transparent coinbase output maturity and shielding

- Add a CoinbaseSpendRestriction enum and Transaction method
- Validate transparent coinbase spends in non-finalized chains

* Don't use genesis created UTXOs for spends in generated block chains

* Refactor out a new_transaction_ordered_outputs function

* Add Transaction::outputs_mut for tests

* Generate valid transparent spends in arbitrary block chains

* When generating blocks, fixup the block contents, then the block hash

* Test that generated chains contain at least one transparent spend

* Make generated chains long enough for reliable tests

* Add transparent and shielded input and output methods to Transaction

* Split chain generation into 3 functions

* Test that unshielded and immature transparent coinbase spends fail

* Comment punctuation

* Clarify a comment

* Clarify probability calculation

* Test that shielded mature coinbase output spends succeed
2021-07-29 14:23:50 +10:00
Janito Vaqueiro Ferreira Filho 79d41b3aea
Add `zebra_state::init_test` helper function for tests (#2539)
* Create a `zebra_state::init_test` helper function

This function will be used as a replacement for `zebra_state::init`
inside tests. It's a simpler alternative because it can ignore any
details that aren't relevant for tests.

* Use `init_test` inside `zebra-state` tests

Update usages of `init` to use `init_test` instead, which simplifies
most cases.

* Use `zebra_state::init_test` in `zebra-consensus`

Replace usages of `zebra_state::init` with the new helper function. This
simplifies the code a bit.
2021-07-28 20:55:01 -03:00
Janito Vaqueiro Ferreira Filho 76b70fb408
Add `proptest-impl` feature to `zebra-state` (#2529)
* Add `proptest-impl` feature to `zebra-state`

This prepares the `zebra-state` crate to be able to export some
test-specific helper types and functions.

* Add `arbitrary` module to `zebra-state` root

A separate module to contain the `Prepare` trait, since it's required by
some prop-test strategies and therefore can't be in the `tests` module.

* Replace usages of `tests::Prepare`

Use the same trait but placed in a new module that's accessible based on
the feature flag.

* Remove old `Prepare` trait

It was obsoleted by the new copy in the `arbitrary` module.

* Make `StateService` crate-accessible

Prepare for it to be accessible in some test modules.

* Refactor strategy function import

Import the function directly, instead of just its containing module.

* Move some strategy functions to `tests::setup`

Create a new module for the strategy functions that are only used
internally.

Co-authored-by: teor <teor@riseup.net>
2021-07-28 00:01:19 +00:00
teor bfc3e4a46c
Add an OrderedUtxo type for transparent spend validation (#2502)
* Add an OrderedUtxo type for validation of spends within a block

This change allows us to check that transparent spends use outputs from
earlier in their block. (But we don't actually do that check yet.)

We need to keep the order of UTXOs when we're contextually verifying
each new block that is added to a chain. But the block order is
irrelevant for UTXOs stored in the state.

* Take ownership in utxos_from_ordered_utxos

* Delete a confusing comment
2021-07-19 10:52:32 -03:00
Alfredo Garcia f7026d728f
move `Utxo` type to zebra-chain (#2481) 2021-07-12 12:49:33 +10:00
teor d4cc867132
Combine near-duplicate Utxo creation functions (#2467)
* Combine near-duplicate Utxo creation functions

* rustfmt
2021-07-09 12:40:39 +10:00
teor 7d1c1fb84e
Document required request timeouts due to data dependencies (#2337)
* Document required request timeouts due to data dependencies

* Update AwaitUTXO docs
2021-06-18 14:43:05 -03:00
teor 2f0f379a9e
Standardise clippy lints and require docs (#2238)
* Standardise lints across Zebra crates, and add missing docs

The only remaining module with missing docs is `zebra_test::command`

* Todo -> TODO

* Clarify what a transcript ErrorChecker does

Also change `Error` -> `BoxError`

* TransError -> ExpectedTranscriptError

* Output Descriptions -> Output descriptions
2021-06-04 08:48:40 +10:00
Alfredo Garcia 4b34482264
Add hints to port conflict and lock file panics (#1535)
* add hint for port error
* add issue filter for port panic
* add lock file hint
* add metrics endpoint port conflict hint
* add hint for tracing endpoint port conflict
* add acceptance test for resource conflics
* Split out common conflict test code into a function
* Add state, metrics, and tracing conflict tests

* Add a full set of stderr acceptance test functions

This change makes the stdout and stderr acceptance test interfaces
identical.

* move Zcash listener opening
* add todo about hint for disk full
* add constant for lock file
* match path in state cache
* don't match windows cache path

* Use Display for state path logs

Avoids weird escaping on Windows when using Debug

* Add Windows conflict error messages

* Turn PORT_IN_USE_ERROR into a regex

And add another alternative Windows-specific port error

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Jane Lusby <jane@zfnd.org>
2021-01-29 22:36:33 +10:00
teor b1d28b73fd Stop disabling lints that no longer cause warnings on nightly 2021-01-19 11:02:20 -05:00
teor 258789ed9b Use the rustc unknown lints attribute
The clippy unknown lints attribute was deprecated in
nightly in rust-lang/rust#80524. The old lint name now produces a
warning.

Since we're using `allow(unknown_lints)` to suppress warnings, we need to
add the canonical name, so we can continue to build without warnings on
nightly.

But we also need to keep the old name, so we can continue to build
without warnings on stable.

And therefore, we also need to disable the "removed lints" warning,
otherwise we'll get warnings about the old name on nightly.

We'll need to keep this transitional clippy config until rustc 1.51 is
stable.
2021-01-19 11:02:20 -05:00
teor 92eb92d1dd
Disable the nightly clippy unnecessary_wraps lint (#1403)
It seems to be a bit broken - some of our functions return `Result` for
consistency with similar functions. But the lint picks them up anyway.
2020-12-01 12:20:57 +10:00
Henry de Valence 342eb166ff state: track UTXO provenance
This commit changes the state system and database format to track the
provenance of UTXOs, in addition to the outputs themselves.
Specifically, it tracks the following additional metadata:

- the height at which the UTXO was created;
- whether or not the UTXO was created from a coinbase transaction or
  not.

This metadata will allow us to:

- check the coinbase maturity consensus rule;
- check the coinbase inputs => no transparent outputs rule;
- implement lookup of transactions by utxo (using the height to find the
  block and then scanning the block) for a future RPC mechanism.

Closes #1342
2020-11-23 22:18:43 -08:00
Henry de Valence e0817d1747 state: introduce PreparedBlock, FinalizedBlock
This change introduces two new types:

- `PreparedBlock`, representing a block which has undergone semantic
  validation and has been prepared for contextual validation;
- `FinalizedBlock`, representing a block which is ready to be finalized
  immediately;

and changes the `Request::CommitBlock`,`Request::CommitFinalizedBlock`
variants to use these types instead of their previous fields.

This change solves the problem of passing data between semantic
validation and contextual validation, and cleans up the state code by
allowing it to pass around a bundle of data.  Previously, the state code
just passed around an `Arc<Block>`, which forced it to needlessly
recompute block hashes and other data, and was incompatible with the
already-known but not-yet-implemented data transfer requirements, namely
passing in the Sprout and Sapling anchors computed during contextual
validation.

This commit propagates the `PreparedBlock` and `FinalizedBlock` types
through the state code but only uses their data opportunistically, e.g.,
changing .hash() computations to use the precomputed hash.  In the
future, these structures can be extended to pass data through the
verification pipeline for reuse as appropriate.  For instance, these
changes allow the sprout and sapling anchors to be propagated through
the state.
2020-11-23 14:16:39 +10:00
Jane Lusby a122a547be reorganize modules for consistency 2020-11-18 15:09:43 -05:00
teor 54cb9277ef Allow some new clippy nightly lints 2020-11-17 10:07:37 +10:00
teor 672b39a847 Use MAX_BLOCK_REORG_HEIGHT in zebra-checkpoints
MAX_BLOCK_REORG_HEIGHT is 1 less than the constant it replaces. The new
calculation is correct: the 100th block is finalized.
2020-10-26 20:59:40 +10:00
Jane Lusby b3634fa3e7
Make the state service use broadcast channels (#1137)
And refactor error handling
2020-10-09 18:37:24 +10:00
Jane Lusby 855f9b5bcb
Implement MVP of NonFinalizedState and integrate it with the state service (#1101)
* implement most of the chain functions
* implement fork
* fix outpoint handling in Chain struct
* update expect for work
* split utxo into two sets
* update the Chain definition
* remove allow attribute in zebra-state/lib.rs
* merge ChainSet type into MemoryState
* Add error messages to asserts
* export proptest impls for use in downstream crates
* add testjob for disabled feature in zebra-chain
* try to fix github actions syntax
* add module doc comment
* update RFC for utxos
* add missing header
* working proptest for Chain
* propagate back results over channel
* Start updating RFC to match changes
* implement queued block pruning
* and now it syncs wooo!
* remove empty modules
* setup config for proptests
* re-enable missing_docs lint
* update RFC to match changes in impl
* add documentation
* use more explicit variable names
2020-10-08 13:07:32 +10:00
Jane Lusby 352721bd88
Implement RFC5: State updates `Chain` type (#1069)
* Begin work on RFC5 implementation

* I think this is necessary

* holy shit supertrait implemented via subtrait

* implement most of the chain functions

* change to slightly better name

* implement fork

* fix outpoint handling in Chain struct

* update expect for work

* resolve review comment

* split utxo into two sets

* update the Chain definition

* just a little more

* update comment

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* apply changes from code review

* remove allow attribute in zebra-state/lib.rs

* Update zebra-state/src/memory_state.rs

Co-authored-by: teor <teor@riseup.net>

* merge ChainSet type into MemoryState

* rename state impl types

* Add error messages to asserts

* add module doc comment

* update RFC for utxos

* add missing header

Co-authored-by: teor <teor@riseup.net>
2020-09-24 15:46:04 -07:00
Henry de Valence 5d8decd224 state: process CommitFinalizedBlock out of order 2020-09-11 13:37:49 -07:00
Henry de Valence 070013439e state: fill in RFC5-style Request, Response enums
The test transcripts need to be rewritten, so they are removed for now.
2020-09-11 13:37:49 -07:00