Commit Graph

560 Commits

Author SHA1 Message Date
teor 256b1c0008
fix(clippy): Fix new lints in nightly clippy (#5959)
* Derive default using #[default]

* Implement PartialEq manually to satisfy clippy

* Allow a manual derive in test-only code

* Fix some missing docs warnings in the Docker build
2023-01-18 00:27:42 +00:00
Arya 1bb8a9c924
change(rpc): Add getpeerinfo RPC method (#5951)
* adds ValidateBlock request to state

* adds `Request` enum in block verifier

skips solution check for BlockProposal requests

calls CheckBlockValidity instead of Commit block for BlockProposal requests

* uses new Request in references to chain verifier

* adds getblocktemplate proposal mode response type

* makes getblocktemplate-rpcs feature in zebra-consensus select getblocktemplate-rpcs in zebra-state

* Adds PR review revisions

* adds info log in CheckBlockProposalValidity

* Reverts replacement of match statement

* adds `GetBlockTemplate::capabilities` fn

* conditions calling checkpoint verifier on !request.is_proposal

* updates references to validate_and_commit_non_finalized

* adds snapshot test, updates test vectors

* adds `should_count_metrics` to NonFinalizedState

* Returns an error from chain verifier for block proposal requests below checkpoint height

adds feature flags

* adds "proposal" to GET_BLOCK_TEMPLATE_CAPABILITIES_FIELD

* adds back block::Request to zebra-consensus lib

* updates snapshots

* Removes unnecessary network arg

* skips req in tracing intstrument for read state

* Moves out block proposal validation to its own fn

* corrects `difficulty_threshold_is_valid` docs

adds/fixes some comments, adds TODOs

general cleanup from a self-review.

* Update zebra-state/src/service.rs

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Update zebra-rpc/src/methods/get_block_template_rpcs.rs

Co-authored-by: teor <teor@riseup.net>

* check best chain tip

* Update zebra-state/src/service.rs

Co-authored-by: teor <teor@riseup.net>

* Applies cleanup suggestions from code review

* updates gbt acceptance test to make a block proposal

* fixes json parsing mistake

* adds retries

* returns reject reason if there are no retries left

* moves result deserialization to RPCRequestClient method, adds docs, moves jsonrpc_core to dev-dependencies

* moves sleep(EXPECTED_TX_TIME) out of loop

* updates/adds info logs in retry loop

* Revert "moves sleep(EXPECTED_TX_TIME) out of loop"

This reverts commit f7f0926f4050519687a79afc16656c3f345c004b.

* adds `allow(dead_code)`

* tests with curtime, mintime, & maxtime

* Fixes doc comment

* Logs error responses from chain_verifier CheckProposal requests

* Removes retry loop, adds num_txs log

* removes verbose info log

* sorts mempool_txs before generating merkle root

* Make imports conditional on a feature

* Disable new CI tests until bugs are fixed

* adds support for getpeerinfo RPC

* adds vector test

* Always passes address_book to RpcServer

* Update zebra-network/src/address_book.rs

Co-authored-by: teor <teor@riseup.net>

* Renames PeerObserver to AddressBookPeers

* adds getpeerinfo acceptance test

* Asserts that addresses parse into SocketAddr

* adds launches_lightwalletd field to TestType::LaunchWithEmptyState and uses it from the get_peer_info test

* renames peer_observer mod

* uses SocketAddr as `addr` field type in PeerInfo

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2023-01-17 07:09:07 +00:00
Arya 3cbee9465a
change(rpc): Add proposal capability to getblocktemplate (#5870)
* adds ValidateBlock request to state

* adds `Request` enum in block verifier

skips solution check for BlockProposal requests

calls CheckBlockValidity instead of Commit block for BlockProposal requests

* uses new Request in references to chain verifier

* adds getblocktemplate proposal mode response type

* makes getblocktemplate-rpcs feature in zebra-consensus select getblocktemplate-rpcs in zebra-state

* Adds PR review revisions

* adds info log in CheckBlockProposalValidity

* Reverts replacement of match statement

* adds `GetBlockTemplate::capabilities` fn

* conditions calling checkpoint verifier on !request.is_proposal

* updates references to validate_and_commit_non_finalized

* adds snapshot test, updates test vectors

* adds `should_count_metrics` to NonFinalizedState

* Returns an error from chain verifier for block proposal requests below checkpoint height

adds feature flags

* adds "proposal" to GET_BLOCK_TEMPLATE_CAPABILITIES_FIELD

* adds back block::Request to zebra-consensus lib

* updates snapshots

* Removes unnecessary network arg

* skips req in tracing intstrument for read state

* Moves out block proposal validation to its own fn

* corrects `difficulty_threshold_is_valid` docs

adds/fixes some comments, adds TODOs

general cleanup from a self-review.

* Update zebra-state/src/service.rs

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Update zebra-rpc/src/methods/get_block_template_rpcs.rs

Co-authored-by: teor <teor@riseup.net>

* check best chain tip

* Update zebra-state/src/service.rs

Co-authored-by: teor <teor@riseup.net>

* Applies cleanup suggestions from code review

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2023-01-11 23:39:51 +00:00
teor 2f23de80f1
fix(clippy): Fix nightly clippy redundant code lints (#5916)
* Remove needless borrows

* Remove redundant into_iter()

* Remove unnecessary clone()
2023-01-05 15:46:04 +00:00
teor e9d6e975b4
change(rpc): Refactor get block template RPC into stages (#5837)
* Add some TODOs

* Move and rename height_from_signed_int()

* Move get_block_template() support functions to a submodule

* Fix incorrect P2SH comments and logs

* Split initial checks into their own functions

* Split state fetch into its own function, do some cleanup

* Move get_block_template_opts to get_block_template::parameters

* Fix and simplify test imports

* Rename block_height to next_block_height

* Rename to chain_tip_and_local_time to make it clear what it contains

* Split fetching mempool transactions out, include them in long poll id

* Refactor coinbase generation

* Split default root calculation into a separate function

* Use DateTime32 for getblocktemplate times

* Use typed difficulty fields rather than strings

* Split out a generate coinbase and roots function

* Move GetBlockTemplate construction into a method

* Document what happens to unusual difficulty values

* Clean up some TODOs

* fastmod check_address check_miner_address

* cargo fmt --all

* Fix an incorrect panic message

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-12-13 21:25:04 +00:00
teor b722a34102
fix(rpc): Wait for 3 minutes to check Zebra is synced to the tip, rather than 2 (#5840)
* Wait for 3 minutes to check Zebra is synced to the tip, rather than 2

* Increase the MockService request timeout
2022-12-12 23:20:03 +00:00
teor cb1045ae5f
change(mempool): Re-verify mempool transactions after a chain fork, rather than re-downloading them all (#5841)
* Move Drop from mempool::ActiveState to mempool::Downloads, to avoid bugs

* Re-verify mempool transactions after a fork

And add a marker struct for mempool download cancellation.

* Update README based on recent mitigations for some issues, tidy format

* Make mempool proptests easier to debug

* Make UnminedTx Display text much smaller

* Update tests for mempool transaction re-verification after forks

* Retry all stored and pending transactions

* Fix a test to check for mempool reset retries
2022-12-12 23:19:45 +00:00
teor 2041fda7bb
fix(test): Reduce verbose test logs (#5825)
* Remove verbose continuous_blockchain test logs

* Downgrade verbose zebra-network logs to debug

* Downgrade some state logs to debug during tests

* Mark were we would add always-on log filters, if we needed to

* Reduce the number of mempool property tests, to reduce logging
2022-12-08 23:56:01 +00:00
teor 4078e244d3
fix(lint): Box large error types to resolve the clippy large result err variant lint (#5759)
* Box errors to deal with large error warnings, add accessor methods for error properties

* Remove or explain some large enum variant lints

* Turn some tickets into TODOs

* Allow missing docs on single-field error enum variants

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-12-08 06:11:33 +00:00
teor 39bcefb569
fix(test): Move test-only code to test-only modules (#5777)
* Move NoChainTip to a test-only module

* Move test-only ChainSyncStatus code into its own module

* Move RecentSyncLengths mocks to their own module

* Silence an unused test code lint

* Actually, NoChainTip is used in production for isolated connections
2022-12-08 01:06:11 +00:00
teor 09836d2800
fix(clippy): Put Rust format variables inline (#5783)
* cargo clippy --fix --all-features --all-targets

With rustc 1.67.0-nightly (234151769 2022-12-03)

* cargo fmt --all
2022-12-08 01:05:57 +00:00
Arya 1e1222c888
change(rpc): Return an error from getblocktemplate RPC if Zebra is still syncing lots of blocks (#5769)
* updates MAX_ESTIMATED_DISTANCE_TO_NETWORK_TIP

* adds ChainSyncStatus trait

* adds sync_status to GetBlockTemplateRpcs

Returns an error from getblocktemplate rpc if !is_close_to_tip

adds MockSyncStatus for tests

* hide MockSyncStatus behind feature flag

* moves is_close_to_tip call into async block

adds tests

* Update zebra-rpc/src/methods/tests/vectors.rs

Co-authored-by: teor <teor@riseup.net>

* uses SeqCst ordering for atomic bool ops.

uses a constant for not synced error code.

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-12-02 11:21:23 +00:00
teor d778caebb8
change(rpc): Select getblocktemplate RPC transactions according to ZIP-317 (#5724)
* Split the conventional fee check into its own method

This will be used for block production and relaying

* Move getblocktemplate transaction selection into a new zip317 module

* Add a block_production_fee_weight field to VerifiedUnminedTx

* Add a custom Zebra minimum transaction weight for block production

* Implement ZIP-317 transaction selection for block production

* Split weighted index setup into its own function

* Split picking a transaction into its own function
2022-12-01 21:57:22 +00:00
Arya 8f9031880e
fix(ci): update fake_peer_set test to avoid spurious failures (#5758)
* moves mempool.enable below peer_set.expect_request

* adds `sync_close_to_tip` call before starting gossip_best_tip_block_hashes task
2022-12-01 09:43:11 +00:00
teor f573365795
Replace a spurious test failure in mempool_requests_for_transactions with an info message (#5753) 2022-12-01 01:29:16 +00:00
Arya eb0a2ef581
change(mempool): Contextually validates mempool transactions in best chain (#5716)
* updates comments

* adds check nullifier no dup fns for transactions

* Adds:
- check::anchors fn for tx iter
- TODO comments for unifying nullifiers and anchors checks
- new state request

Updates unknown anchor errors to accomodate tx-only check

Calls new state fn from transaction verifier

* updates check::anchors fns to use transactions

updates TransactionContextualValidity request to check sprout anchors

adds comment mentioning TransactionContextualValidity ignores UTXOs

* conditions new state req call on is_mempool

updates tests

* fix doc link / lint error

* checks for duplicate nullifiers with closures

* Update zebra-state/src/service/check/nullifier.rs

Co-authored-by: teor <teor@riseup.net>

* documents find_duplicate_nullifier params

moves if let statement into for loop

* renames new state req/res

* asserts correct response variant in tx verifier

* adds CheckBestChainTipShieldedSpends call in tx verifier to async checks

* re-adds tracing instrumentation to check::anchors fn

renames transaction_in_state to transaction_in_chain

* adds block/tx wrapper fns for anchors checks

* uses UnminedTx instead of transaction.hash()

deletes broken test

* updates new state req/res name

* updates tests and uses par_iter for anchors checks

* Updates check::anchors pub fn docs.

* Adds:
- comments / docs
- a TransactionError variant for ValidateContextError

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* moves downcast to From impl

rustfmt

* moves the ValidateContextError into an Arc

updates comments and naming

* leaves par_iter for another PR

* puts io::Error in an Arc

* updates anchors tests to call tx_anchors check

* updates tests to call tx_no_duplicates_in_chain

slightly improves formatting

* Update zebra-consensus/src/error.rs

Co-authored-by: teor <teor@riseup.net>

* moves Arc from HistoryError to ValidateContextError

Co-authored-by: teor <teor@riseup.net>
2022-11-30 04:40:15 +00:00
teor 0ec502bb85
change(log): Log loaded config path when Zebra starts up (#5733)
* Log zebrad config path as well as config values

* Remove duplicate config log

* Only log config path and data for server commands

* Use config ref instead
2022-11-30 04:40:02 +00:00
Arya a2f2a1402e
fix(zebrad) print usage info for --help flag (#5634)
* print usage info for --help flag

* update known issues and help command's help text

* Update README.md

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* updates README

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2022-11-18 14:24:10 +00:00
teor 074733d183
fix(rpc): Shut down the RPC server properly when Zebra shuts down (#5591)
* Make the queue runner task shut down when the RpcImpl is dropped

* Move RPC server startup into the spawn() tokio future

* Return a shutdown handle from the RPC spawn() method

* Shut down the RPC server properly when Zebra shuts down

* Add a changelog entry for this security fix

* Call RpcServer::shutdown() when it is dropped, and wait

* Block on RPC server shutdown when Zebra's tasks have an error
2022-11-10 14:51:53 +00:00
teor 7e13677197
change(rpc): generate coinbase transactions in the getblocktemplate RPC (#5580)
* Add a getblocktemplate-rpcs feature to zebra-chain, and fix missing feature deps

* Add a coinbase transaction creation stub

* Add coinbase creation to zebra-chain

* Add coinbase creation and miner subsidy to zebra-consensus

* Add the miner config to the GetBlockTemplateRpcImpl

* Generate the coinbase transaction in the getblocktemplate RPC

* Provide fake valid block heights to getblocktemplate RPC tests

* Update getblocktemplate RPC snapshots

* Add a getblocktemplate.coinbase_tx deserialized transaction snapshot test

* Update snapshots

* Return funding stream outputs in the same order every time

* Update snapshots

* Fix a script bytes bug

* Update snapshots
2022-11-10 00:12:27 +00:00
teor c4fad29824
fix(sync): Pause new downloads when Zebra reaches the lookahead limit (#5561)
* Use correct release for getblocktemplate config

* Include at least 2 full checkpoints in the lookahead limit

* Increase full sync timeout to 36 hours

* Only log "synced block height too far ahead of the tip" once

* Replace AboveLookaheadHeightLimit error with pausing the syncer

* Use AboveLookaheadHeightLimit for blocks a very long way from the tip

* Also add the getblocktemplate config, and fix the test message

* Remove an outdated TODO comment

* Allow syncing again when a small number of blocks are in the queue

* Allow some dead code
2022-11-09 04:42:04 +00:00
teor 4a8349fa8a
fix(sync): Make the syncer ignore some new block verification errors (#5537)
* Fix error text for state service for syncer

* Fix error handling in syncer

* cargo fmt --all

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-11-04 06:57:45 +00:00
Arya 2f3b05f8e1
change(rpc): add submitblock RPC method (#5526)
* adds submitblock rpc method

* re-orders imports

* replaces thread::yield_now with async yield_now

* Fix doc warnings and unused variable warnings, add missing docs

* Mark work_id as optional

* Use the same ChainVerifier for downloaded and submitted blocks

* Revert unused changes & minor cleanups

* Document currently-unreachable code

* updates tests and submit_block response for AlreadyVerified error

* Update zebra-rpc/src/methods/get_block_template_rpcs.rs

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* changes names from BlockVerifier to ChainVerifier and block_verifier to chain_verifier to keep it consistent with naming in zebra-consensus

* move how to run the submit_block test example to acceptance.rs

* updates snapshot tests

* moved acceptance test to a separate file

* removes extra tower::ServiceBuilder::new(), updates docs

* updates vectors and snapshot tests, changes hex decoding error in submit_block method from server error to parse error

* hides errors module in zebra-rpc behind a feature flag and adds docs.

* Updates snapshot test, adds mod docs, moves HexData to its own mod, and removes the unrelated make_server_error refactor for now

* update submit block acceptance test mod doc

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-11-04 03:57:08 +00:00
teor 34313b8857
fix(build): Disable new Rust 1.65 lints and fix some test errors (#5549)
* allow(clippy::result_large_err)

* Increase the async executor delay expected by tests

* Split getblocktemplate-rpcs OS tests into a separate matrix job

* Add new patch jobs

* allow(unknown_lints)
2022-11-04 02:00:56 +00:00
teor 71f5e63e64
3. change(rpc): Add fee and sigops fields to getblocktemplate transactions (#5508)
* Add a legacy_sigop_count field to VerifiedUnminedTx

* Add conversions from Vec<VerifiedUnminedTx> to block header roots

* Add fee and sigops field to block template transactions

* Fix up mempool request names

* Increase existing snapshot test coverage

* Document a new method parameter

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2022-11-03 17:03:41 +00:00
teor 142411508b
2. change(rpc): Add some transaction fields to the `getblocktemplate` RPC (#5496)
* Add documentation for the getblocktemplate RPC

* Add a new mempool::Request::Transactions

* Add conversions from Vec<UnminedTx> to merkle::Root and AuthDataRoot

* Fill in the merkle root and auth data root fields

* Delete the Coinbase type, it's the same as Transaction

* Fill in some other existing types

* Add Hex serialization support to some zebra-chain types

* Add TransactionTemplate fields and fill some in

* Fix test hangs by spawning async tasks

* Add temporary workaround for no transactions in the block

* Encode hashes and roots as hex

* Update RPC snapshots

* Add a missing Request::Transactions handler

* Fix doc warnings

* Fix fee serialization

* Update snapshots for serialization changes

* Add a missing Cargo.lock change

* Change depends type

* Remove duplicate feature entry

* Document the new RPC feature

* Fix a comment typo

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Update default roots docs

* Fix comment typo

* Fix a comment typo

Co-authored-by: Arya <aryasolhi@gmail.com>

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: Arya <aryasolhi@gmail.com>
2022-11-03 03:25:01 +00:00
Alfredo Garcia 19cb670614
rpc(config): Add a `mining` section with miner address to config (#5491)
* add mining section to zebra config

* fix features in zebrad `Cargo.toml`

* change field name

* add docs

Co-authored-by: teor <teor@riseup.net>

* add getblocktemplate-rpcs config

* fix commit

Co-authored-by: teor <teor@riseup.net>
2022-11-01 00:43:45 +00:00
teor 7b47aac370
Allow extra lookahead in the verifier, state, and block commit queues (#5490) 2022-10-27 20:16:42 +00:00
teor c812f880cf
cleanup(clippy): Use inline format strings (#5489)
* Inline format strings using an automated clippy fix

```sh
cargo clippy --fix --all-features --all-targets -- -A clippy::all -W clippy::uninlined_format_args
cargo fmt --all
```

* Remove unused & and &mut using an automated clippy fix

```sh
cargo clippy --fix --all-features --all-targets -- -A clippy::all -W clippy::uninlined_format_args
```
2022-10-27 13:25:18 +00:00
Alfredo Garcia 868ba1325e
refactor(config): Move configs to components (#5460)
* move configs to componenets

* remove trailing slash

* remove link from function behind feature

* fix config call in metrics
2022-10-24 23:39:00 +00:00
Arya e9452d9a6f
change(mempool) reject transactions with spent outpoints or nullifiers (#5434)
* adds transactions to ChainTipBlock and rejects transactions from the mempool that have been invalidated by the latest block commit

* merges remove_same_effects in with reject_invalidated_transactions, moves nullifier retrieval to the new Storage method, rejects mined_ids, and updates tests

* updates DuplicateSpend's error message

* fixes tests

* Update zebrad/src/components/mempool/storage.rs

Co-authored-by: teor <teor@riseup.net>

* adds comment

* formatting for the proptest

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
2022-10-24 19:48:45 +00:00
teor 737fbac3fc
Allow extra lookahead blocks in the verifier, state, and block commit task queues (#5465) 2022-10-24 14:55:57 +00:00
teor fd517ee4a7
Make timeouts shorter to reduce full sync times (#5397) 2022-10-14 20:20:24 +00:00
Arya 1cfbe64039
fix(test): resolve a timing issue in fake_peer_set setup (#5398)
* waits for the chain tip update if setup adds mempool transactions and adds match arm for Response::Nil to MempoolTransactionIds request

* updates other match statements checking MempoolTransactionIds requests

* removes wait for chain tip update and waits for AdvertiseBlock earlier instead
2022-10-13 13:34:40 +00:00
Arya a28350e742
change(state): Write non-finalized blocks to the state in a separate thread, to avoid network and RPC hangs (#5257)
* Add a new block commit task and channels, that don't do anything yet

* Add last_block_hash_sent to the state service, to avoid database accesses

* Update last_block_hash_sent regardless of commit errors

* Rename a field to StateService.max_queued_finalized_height

* Commit finalized blocks to the state in a separate task

* Check for panics in the block write task

* Wait for the block commit task in tests, and check for errors

* Always run a proptest that sleeps once

* Add extra debugging to state shutdowns

* Work around a RocksDB shutdown bug

* Close the finalized block channel when we're finished with it

* Only reset state queue once per error

* Update some TODOs

* Add a module doc comment

* Drop channels and check for closed channels in the block commit task

* Close state channels and tasks on drop

* Remove some duplicate fields across StateService and ReadStateService

* Try tweaking the shutdown steps

* Update and clarify some comments

* Clarify another comment

* Don't try to cancel RocksDB background work on drop

* Fix up some comments

* Remove some duplicate code

* Remove redundant workarounds for shutdown issues

* Remode a redundant channel close in the block commit task

* Remove a mistaken `!force` shutdown condition

* Remove duplicate force-shutdown code and explain it better

* Improve RPC error logging

* Wait for chain tip updates in the RPC tests

* Wait 2 seconds for chain tip updates before skipping them

* Remove an unnecessary block_in_place()

* Fix some test error messages that were changed by earlier fixes

* Expand some comments, fix typos

Co-authored-by: Marek <mail@marek.onl>

* Actually drop children of failed blocks

* Explain why we drop descendants of failed blocks

* Clarify a comment

* Wait for chain tip updates in a failing test on macOS

* Clean duplicate finalized blocks when the non-finalized state activates

* Send an error when receiving a duplicate finalized block

* Update checkpoint block behaviour, document its consensus rule

* Wait for chain tip changes in inbound_block_height_lookahead_limit test

* Wait for the genesis block to commit in the fake peer set mempool tests

* Disable unreliable mempool verification check in the send transaction test

* Appease rustfmt

* Use clear_finalized_block_queue() everywhere that blocks are dropped

* Document how Finalized and NonFinalized clones are different

* sends non-finalized blocks to the block write task

* passes ZebraDb to commit_new_chain, commit_block, and no_duplicates_in_finalized_chain instead of FinalizedState

* Update zebra-state/src/service/write.rs

Co-authored-by: teor <teor@riseup.net>

* updates comments, renames send_process_queued, other minor cleanup

* update assert_block_can_be_validated comment

* removes `mem` field from StateService

* removes `disk` field from StateService and updates block_iter to use `ZebraDb` instead of the finalized state

* updates tests that use the disk to use read_service.db instead

* moves best_tip to a read fn and returns finalized & non-finalized states from setup instead of the state service

* changes `contextual_validity` to get the network from the finalized_state instead of another param

* swaps out StateService with FinalizedState and NonFinalizedState in tests

* adds NotReadyToBeCommitted error and returns it from validate_and_commit when a blocks parent hash is not in any chain

* removes NonFinalizedWriteCmd and calls, moves update_latest_channels above rsp_tx.send

* makes parent_errors_map an indexmap

* clears non-finalized block queue when the receiver is dropped and when the StateService is being dropped

* sends non-finalized blocks to the block write task

* passes ZebraDb to commit_new_chain, commit_block, and no_duplicates_in_finalized_chain instead of FinalizedState

* updates comments, renames send_process_queued, other minor cleanup

* Update zebra-state/src/service/write.rs

Co-authored-by: teor <teor@riseup.net>

* update assert_block_can_be_validated comment

* removes `mem` field from StateService

* removes `disk` field from StateService and updates block_iter to use `ZebraDb` instead of the finalized state

* updates tests that use the disk to use read_service.db instead

* moves best_tip to a read fn and returns finalized & non-finalized states from setup instead of the state service

* changes `contextual_validity` to get the network from the finalized_state instead of another param

* swaps out StateService with FinalizedState and NonFinalizedState in tests

* adds NotReadyToBeCommitted error and returns it from validate_and_commit when a blocks parent hash is not in any chain

* removes NonFinalizedWriteCmd and calls, moves update_latest_channels above rsp_tx.send

* makes parent_errors_map an indexmap

* clears non-finalized block queue when the receiver is dropped and when the StateService is being dropped

* removes duplicate field definitions on StateService that were a result of a bad merge

* update NotReadyToBeCommitted error message

* Appear rustfmt

* Fix doc links

* Rename a function to initial_contextual_validity()

* Do error tasks on Err, and success tasks on Ok

* Simplify parent_error_map truncation

* Rewrite best_tip() to use tip()

* Rename latest_mem() to latest_non_finalized_state()

```sh
fastmod latest_mem latest_non_finalized_state zebra*
cargo fmt --all
```

* Simplify latest_non_finalized_state() using a new WatchReceiver API

* Expand some error messages

* Send the result after updating the channels, and document why

* wait for chain_tip_update before cancelling download in mempool_cancel_mined

* adds `sent_non_finalized_block_hashes` field to StateService

* adds batched sent_hash insertions and checks sent hashes in queue_and_commit_non_finalized before adding a block to the queue

* check that the `curr_buf` in SentHashes is not empty before pushing it to the `sent_bufs`

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Fix rustfmt

* Check for finalized block heights using zs_contains()

* adds known_utxos field to SentHashes

* updates comment on SentHashes.add method

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* return early when there's a duplicate hash in QueuedBlocks.queue instead of panicking

* Make finalized UTXOs near the final checkpoint available for full block verification

* Replace a checkpoint height literal with the actual config

* Update mainnet and testnet checkpoints - 7 October 2022

* Fix some state service init arguments

* Allow more lookahead in the downloader, but less lookahead in the syncer

* Add the latest config to the tests, and fix the latest config check

* Increase the number of finalized blocks checked for non-finalized block UTXO spends

* fix(log): reduce verbose logs for block commits (#5348)

* Remove some verbose block write channel logs

* Only warn about tracing endpoint if the address is actually set

* Use CloneError instead of formatting a non-cloneable error

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Increase block verify timeout

* Work around a known block timeout bug by using a shorter timeout

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-10-11 19:25:45 +00:00
Arya 7207f9d732
waits for tip change after each commit call to the state-service in mempool_cancel_mined (#5347) 2022-10-06 10:36:42 +00:00
Arya 59d647cfa1
wait for chain_tip_update before cancelling download in mempool_cancel_mined (#5322) 2022-10-04 02:52:03 +00:00
teor 343c5e68d4
change(state): Write finalized blocks to the state in a separate thread, to avoid network and RPC hangs (#5134)
* Add a new block commit task and channels, that don't do anything yet

* Add last_block_hash_sent to the state service, to avoid database accesses

* Update last_block_hash_sent regardless of commit errors

* Rename a field to StateService.max_queued_finalized_height

* Commit finalized blocks to the state in a separate task

* Check for panics in the block write task

* Wait for the block commit task in tests, and check for errors

* Always run a proptest that sleeps once

* Add extra debugging to state shutdowns

* Work around a RocksDB shutdown bug

* Close the finalized block channel when we're finished with it

* Only reset state queue once per error

* Update some TODOs

* Add a module doc comment

* Drop channels and check for closed channels in the block commit task

* Close state channels and tasks on drop

* Remove some duplicate fields across StateService and ReadStateService

* Try tweaking the shutdown steps

* Update and clarify some comments

* Clarify another comment

* Don't try to cancel RocksDB background work on drop

* Fix up some comments

* Remove some duplicate code

* Remove redundant workarounds for shutdown issues

* Remode a redundant channel close in the block commit task

* Remove a mistaken `!force` shutdown condition

* Remove duplicate force-shutdown code and explain it better

* Improve RPC error logging

* Wait for chain tip updates in the RPC tests

* Wait 2 seconds for chain tip updates before skipping them

* Remove an unnecessary block_in_place()

* Fix some test error messages that were changed by earlier fixes

* Expand some comments, fix typos

Co-authored-by: Marek <mail@marek.onl>

* Actually drop children of failed blocks

* Explain why we drop descendants of failed blocks

* Clarify a comment

* Wait for chain tip updates in a failing test on macOS

* Clean duplicate finalized blocks when the non-finalized state activates

* Send an error when receiving a duplicate finalized block

* Update checkpoint block behaviour, document its consensus rule

* Wait for chain tip changes in inbound_block_height_lookahead_limit test

* Wait for the genesis block to commit in the fake peer set mempool tests

* Disable unreliable mempool verification check in the send transaction test

* Appease rustfmt

* Use clear_finalized_block_queue() everywhere that blocks are dropped

* Document how Finalized and NonFinalized clones are different

* Use the same check as commit_finalized() for finalized block heights

Co-authored-by: Marek <mail@marek.onl>

Co-authored-by: Marek <mail@marek.onl>
2022-09-28 16:09:56 +00:00
Arya ec115e930f
change(zebrad): opens the database in a blocking tokio thread, which allows tokio to run other tasks (#5228)
* use spawn_blocking to run zebra_state::init from start cmd

* uses zebra_state::spawn_init in copy-state command

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-09-26 15:45:42 +00:00
dependabot[bot] d8b0481a4c
build(deps): bump tracing-subscriber from 0.3.11 to 0.3.15 (#5223)
* build(deps): bump tracing-subscriber from 0.3.11 to 0.3.15

Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.3.11 to 0.3.15.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.3.11...tracing-subscriber-0.3.15)

---
updated-dependencies:
- dependency-name: tracing-subscriber
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Note that this dependency update fixes a known Zebra logging bug

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
2022-09-22 04:50:56 +00:00
Arya 6343b83dc1
feat(logs): prints git metadata and platform info when zebrad starts up (#5200)
* adds info log with git & build metadata when zebrad starts up

* Update zebrad/src/application.rs

Co-authored-by: teor <teor@riseup.net>

Co-authored-by: teor <teor@riseup.net>
2022-09-20 21:05:37 +00:00
teor 806dd0f24c
feat(net): return peer metadata from `connect_isolated` functions (#4870)
* Move version into a ConnectionInfo struct

* Add negotiated version to ConnectionInfo

Part of this change was generated using:
```
fastmod --fixed-strings ".version(" ".remote_version(" zebra-network
```

* Add the peer address to ConnectionInfo, add ConnectionInfo to Connection

* Return a Client instance from connect_isolated_* functions

This allows library users to access client ConnectionInfo.

* Add and improve debug formatting

* Add peer services and user agent to ConnectionInfo

* Export the Client type, and fix up a zebrad test

* Export types used by the public API

* Split VersionMessage into its own struct

* Use VersionMessage in ConnectionInfo

* Add a public API test for ConnectionInfo

* Wrap ConnectionInfo in an Arc

* Fix some doc links
2022-09-14 15:00:25 +00:00
teor 43e4a29be2
Fix a new nightly clippy "unwrap_or_else uses default" lint (#5131) 2022-09-12 23:59:07 +00:00
Arya d9fae6e311
fix(test) switch zebrad to a non-blocking tracing logger (#5032)
* adds non-blocking writer for tracing subscriber

* use non_blocking writer for the fmt::Layer with the tokio-console feature as well

* adds doc comment to _guard field

* adds acceptance test

* update filter_handle type to use NonBlocking

* adds more detail on lossy non-blocking writer and sets tracing.filter to "trace" in acceptance test

* drops ZebradApp before process::exit(1) in the event of a FrameworkError

* reduces buffered lines limit to 8000

* adds tracing.buffer_limit config and some comments

* update acceptance.rs

* fix acceptance test

* fixes ambigious phrasing in comment

* updates zebrad/src/application.rs

* Find out what the join error is in the GitHub runner tests

* updates acceptance test to use recv_timeout instead of always waiting 10 seconds, removes unnecessary echo command, and reduces # of rpc requests to 500

* see if sleeping for a few seconds before exiting helps the macOS test pass

* Expand exit sleep docs

Co-authored-by: Arya <aryasolhi@gmail.com>

Co-authored-by: teor <teor@riseup.net>
2022-09-07 07:39:30 +00:00
teor ea34baa8b8
add(test): test disabled `lightwalletd` mempool gRPCs via zebrad logs (#5016)
* add grpc mempool test research

* add a config flag for mempool injection of transactions in test

* Only copy the inner state directory in the send transactions test

* Preload Zcash parameters in some transaction verification tests

* Add a block and transaction Hash method to convert from display order bytes

* Update test coverage docs

* Add debugging output for mempool transaction verification

* Test fetching sent mempool transactions using gRPC

* Add extra log checks to the send transaction test

* Wait for zebrad mempool activation before running gRPC tests

* Update send transaction test for lightwalletd not returning mempool transactions

* Check zebrad logs instead of disabled lightwalletd gRPCs

* Add a debug option that makes RPCs pretend the sync is finished

* Remove an unused debug option

* Remove unused test code and downgrade some logs

* Fix test log checks

* Fix some rustdoc warnings

* Fix a compilation error due to new function arguments

* Make zebrad sync timeouts consistent and remove outdated code

* Document how to increase temporary directory space for tests

* Stop checking for a log that doesn't always happen

* Remove some commented-out code

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Update a comment about run time

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Add new config to new tests from the `main` branch

* Add transactions to the list, rather than replacing the list with each new block

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2022-09-06 13:32:33 +00:00
teor df9540833d
Breaking: fix(rpc): Add a config for multi-threaded RPC server requests (#5013)
* Add a config for increasing the number of RPC threads

* Add unit tests for parallel RPC servers

* Refactor tests to remove duplicate code

* Update the README

* Actually use parallel threads in some RPC tests
2022-09-04 05:03:15 +00:00
Arya 3ff56c22cd
adds start as default subcommand for zebrad (#4957)
* adds start as default subcommand for zebrad

* moves EntryPoint to submodule and adds a test

* moves all start tests to config_test to avoid listener conflicts

* Update zebrad/src/application/entry_point.rs docs

* Revert "moves all start tests to config_test to avoid listener conflicts"

This reverts commit 61ce46f5a13907facc3a11326e7a328d81b2be3d.

* Update based on test API changes from another PR

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-08-30 09:01:33 +00:00
Conrado Gouvea 6fd750e168
build(deps): bump insta from 1.15.0 to 1.17.1 (#4884)
* build(deps): bump insta from 1.15.0 to 1.17.1

Bumps [insta](https://github.com/mitsuhiko/insta) from 1.15.0 to 1.17.1.
- [Release notes](https://github.com/mitsuhiko/insta/releases)
- [Changelog](https://github.com/mitsuhiko/insta/blob/master/CHANGELOG.md)
- [Commits](https://github.com/mitsuhiko/insta/compare/1.15.0...1.17.1)

---
updated-dependencies:
- dependency-name: insta
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* make zebra_test::init() return the insta drop guard

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-08-04 15:44:44 +00:00
Alfredo Garcia a6f15e84a8
fix(test): truncate the number of transactions in send transaction test (#4848)
* truncate the number of transactions in send transaction test

* Limit send transaction test to RPC queue length

Co-authored-by: teor <teor@riseup.net>
2022-08-01 03:48:19 +00:00
teor c811457f46
Allow the mempool_transaction_expiration test to get network requests in any order (#4830) 2022-07-27 12:15:53 +10:00