Commit Graph

321 Commits

Author SHA1 Message Date
Arya 3cbee9465a
change(rpc): Add proposal capability to getblocktemplate (#5870)
* adds ValidateBlock request to state

* adds `Request` enum in block verifier

skips solution check for BlockProposal requests

calls CheckBlockValidity instead of Commit block for BlockProposal requests

* uses new Request in references to chain verifier

* adds getblocktemplate proposal mode response type

* makes getblocktemplate-rpcs feature in zebra-consensus select getblocktemplate-rpcs in zebra-state

* Adds PR review revisions

* adds info log in CheckBlockProposalValidity

* Reverts replacement of match statement

* adds `GetBlockTemplate::capabilities` fn

* conditions calling checkpoint verifier on !request.is_proposal

* updates references to validate_and_commit_non_finalized

* adds snapshot test, updates test vectors

* adds `should_count_metrics` to NonFinalizedState

* Returns an error from chain verifier for block proposal requests below checkpoint height

adds feature flags

* adds "proposal" to GET_BLOCK_TEMPLATE_CAPABILITIES_FIELD

* adds back block::Request to zebra-consensus lib

* updates snapshots

* Removes unnecessary network arg

* skips req in tracing intstrument for read state

* Moves out block proposal validation to its own fn

* corrects `difficulty_threshold_is_valid` docs

adds/fixes some comments, adds TODOs

general cleanup from a self-review.

* Update zebra-state/src/service.rs

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Update zebra-rpc/src/methods/get_block_template_rpcs.rs

Co-authored-by: teor <teor@riseup.net>

* check best chain tip

* Update zebra-state/src/service.rs

Co-authored-by: teor <teor@riseup.net>

* Applies cleanup suggestions from code review

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2023-01-11 23:39:51 +00:00
teor 2f23de80f1
fix(clippy): Fix nightly clippy redundant code lints (#5916)
* Remove needless borrows

* Remove redundant into_iter()

* Remove unnecessary clone()
2023-01-05 15:46:04 +00:00
teor e9d6e975b4
change(rpc): Refactor get block template RPC into stages (#5837)
* Add some TODOs

* Move and rename height_from_signed_int()

* Move get_block_template() support functions to a submodule

* Fix incorrect P2SH comments and logs

* Split initial checks into their own functions

* Split state fetch into its own function, do some cleanup

* Move get_block_template_opts to get_block_template::parameters

* Fix and simplify test imports

* Rename block_height to next_block_height

* Rename to chain_tip_and_local_time to make it clear what it contains

* Split fetching mempool transactions out, include them in long poll id

* Refactor coinbase generation

* Split default root calculation into a separate function

* Use DateTime32 for getblocktemplate times

* Use typed difficulty fields rather than strings

* Split out a generate coinbase and roots function

* Move GetBlockTemplate construction into a method

* Document what happens to unusual difficulty values

* Clean up some TODOs

* fastmod check_address check_miner_address

* cargo fmt --all

* Fix an incorrect panic message

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-12-13 21:25:04 +00:00
teor b722a34102
fix(rpc): Wait for 3 minutes to check Zebra is synced to the tip, rather than 2 (#5840)
* Wait for 3 minutes to check Zebra is synced to the tip, rather than 2

* Increase the MockService request timeout
2022-12-12 23:20:03 +00:00
teor cb1045ae5f
change(mempool): Re-verify mempool transactions after a chain fork, rather than re-downloading them all (#5841)
* Move Drop from mempool::ActiveState to mempool::Downloads, to avoid bugs

* Re-verify mempool transactions after a fork

And add a marker struct for mempool download cancellation.

* Update README based on recent mitigations for some issues, tidy format

* Make mempool proptests easier to debug

* Make UnminedTx Display text much smaller

* Update tests for mempool transaction re-verification after forks

* Retry all stored and pending transactions

* Fix a test to check for mempool reset retries
2022-12-12 23:19:45 +00:00
teor 2041fda7bb
fix(test): Reduce verbose test logs (#5825)
* Remove verbose continuous_blockchain test logs

* Downgrade verbose zebra-network logs to debug

* Downgrade some state logs to debug during tests

* Mark were we would add always-on log filters, if we needed to

* Reduce the number of mempool property tests, to reduce logging
2022-12-08 23:56:01 +00:00
teor 4078e244d3
fix(lint): Box large error types to resolve the clippy large result err variant lint (#5759)
* Box errors to deal with large error warnings, add accessor methods for error properties

* Remove or explain some large enum variant lints

* Turn some tickets into TODOs

* Allow missing docs on single-field error enum variants

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-12-08 06:11:33 +00:00
teor 39bcefb569
fix(test): Move test-only code to test-only modules (#5777)
* Move NoChainTip to a test-only module

* Move test-only ChainSyncStatus code into its own module

* Move RecentSyncLengths mocks to their own module

* Silence an unused test code lint

* Actually, NoChainTip is used in production for isolated connections
2022-12-08 01:06:11 +00:00
teor 09836d2800
fix(clippy): Put Rust format variables inline (#5783)
* cargo clippy --fix --all-features --all-targets

With rustc 1.67.0-nightly (234151769 2022-12-03)

* cargo fmt --all
2022-12-08 01:05:57 +00:00
Arya 1e1222c888
change(rpc): Return an error from getblocktemplate RPC if Zebra is still syncing lots of blocks (#5769)
* updates MAX_ESTIMATED_DISTANCE_TO_NETWORK_TIP

* adds ChainSyncStatus trait

* adds sync_status to GetBlockTemplateRpcs

Returns an error from getblocktemplate rpc if !is_close_to_tip

adds MockSyncStatus for tests

* hide MockSyncStatus behind feature flag

* moves is_close_to_tip call into async block

adds tests

* Update zebra-rpc/src/methods/tests/vectors.rs

Co-authored-by: teor <teor@riseup.net>

* uses SeqCst ordering for atomic bool ops.

uses a constant for not synced error code.

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-12-02 11:21:23 +00:00
teor d778caebb8
change(rpc): Select getblocktemplate RPC transactions according to ZIP-317 (#5724)
* Split the conventional fee check into its own method

This will be used for block production and relaying

* Move getblocktemplate transaction selection into a new zip317 module

* Add a block_production_fee_weight field to VerifiedUnminedTx

* Add a custom Zebra minimum transaction weight for block production

* Implement ZIP-317 transaction selection for block production

* Split weighted index setup into its own function

* Split picking a transaction into its own function
2022-12-01 21:57:22 +00:00
Arya 8f9031880e
fix(ci): update fake_peer_set test to avoid spurious failures (#5758)
* moves mempool.enable below peer_set.expect_request

* adds `sync_close_to_tip` call before starting gossip_best_tip_block_hashes task
2022-12-01 09:43:11 +00:00
teor f573365795
Replace a spurious test failure in mempool_requests_for_transactions with an info message (#5753) 2022-12-01 01:29:16 +00:00
Arya eb0a2ef581
change(mempool): Contextually validates mempool transactions in best chain (#5716)
* updates comments

* adds check nullifier no dup fns for transactions

* Adds:
- check::anchors fn for tx iter
- TODO comments for unifying nullifiers and anchors checks
- new state request

Updates unknown anchor errors to accomodate tx-only check

Calls new state fn from transaction verifier

* updates check::anchors fns to use transactions

updates TransactionContextualValidity request to check sprout anchors

adds comment mentioning TransactionContextualValidity ignores UTXOs

* conditions new state req call on is_mempool

updates tests

* fix doc link / lint error

* checks for duplicate nullifiers with closures

* Update zebra-state/src/service/check/nullifier.rs

Co-authored-by: teor <teor@riseup.net>

* documents find_duplicate_nullifier params

moves if let statement into for loop

* renames new state req/res

* asserts correct response variant in tx verifier

* adds CheckBestChainTipShieldedSpends call in tx verifier to async checks

* re-adds tracing instrumentation to check::anchors fn

renames transaction_in_state to transaction_in_chain

* adds block/tx wrapper fns for anchors checks

* uses UnminedTx instead of transaction.hash()

deletes broken test

* updates new state req/res name

* updates tests and uses par_iter for anchors checks

* Updates check::anchors pub fn docs.

* Adds:
- comments / docs
- a TransactionError variant for ValidateContextError

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* moves downcast to From impl

rustfmt

* moves the ValidateContextError into an Arc

updates comments and naming

* leaves par_iter for another PR

* puts io::Error in an Arc

* updates anchors tests to call tx_anchors check

* updates tests to call tx_no_duplicates_in_chain

slightly improves formatting

* Update zebra-consensus/src/error.rs

Co-authored-by: teor <teor@riseup.net>

* moves Arc from HistoryError to ValidateContextError

Co-authored-by: teor <teor@riseup.net>
2022-11-30 04:40:15 +00:00
teor c4fad29824
fix(sync): Pause new downloads when Zebra reaches the lookahead limit (#5561)
* Use correct release for getblocktemplate config

* Include at least 2 full checkpoints in the lookahead limit

* Increase full sync timeout to 36 hours

* Only log "synced block height too far ahead of the tip" once

* Replace AboveLookaheadHeightLimit error with pausing the syncer

* Use AboveLookaheadHeightLimit for blocks a very long way from the tip

* Also add the getblocktemplate config, and fix the test message

* Remove an outdated TODO comment

* Allow syncing again when a small number of blocks are in the queue

* Allow some dead code
2022-11-09 04:42:04 +00:00
teor 4a8349fa8a
fix(sync): Make the syncer ignore some new block verification errors (#5537)
* Fix error text for state service for syncer

* Fix error handling in syncer

* cargo fmt --all

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-11-04 06:57:45 +00:00
teor 34313b8857
fix(build): Disable new Rust 1.65 lints and fix some test errors (#5549)
* allow(clippy::result_large_err)

* Increase the async executor delay expected by tests

* Split getblocktemplate-rpcs OS tests into a separate matrix job

* Add new patch jobs

* allow(unknown_lints)
2022-11-04 02:00:56 +00:00
teor 71f5e63e64
3. change(rpc): Add fee and sigops fields to getblocktemplate transactions (#5508)
* Add a legacy_sigop_count field to VerifiedUnminedTx

* Add conversions from Vec<VerifiedUnminedTx> to block header roots

* Add fee and sigops field to block template transactions

* Fix up mempool request names

* Increase existing snapshot test coverage

* Document a new method parameter

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2022-11-03 17:03:41 +00:00
teor 142411508b
2. change(rpc): Add some transaction fields to the `getblocktemplate` RPC (#5496)
* Add documentation for the getblocktemplate RPC

* Add a new mempool::Request::Transactions

* Add conversions from Vec<UnminedTx> to merkle::Root and AuthDataRoot

* Fill in the merkle root and auth data root fields

* Delete the Coinbase type, it's the same as Transaction

* Fill in some other existing types

* Add Hex serialization support to some zebra-chain types

* Add TransactionTemplate fields and fill some in

* Fix test hangs by spawning async tasks

* Add temporary workaround for no transactions in the block

* Encode hashes and roots as hex

* Update RPC snapshots

* Add a missing Request::Transactions handler

* Fix doc warnings

* Fix fee serialization

* Update snapshots for serialization changes

* Add a missing Cargo.lock change

* Change depends type

* Remove duplicate feature entry

* Document the new RPC feature

* Fix a comment typo

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Update default roots docs

* Fix comment typo

* Fix a comment typo

Co-authored-by: Arya <aryasolhi@gmail.com>

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: Arya <aryasolhi@gmail.com>
2022-11-03 03:25:01 +00:00
teor 7b47aac370
Allow extra lookahead in the verifier, state, and block commit queues (#5490) 2022-10-27 20:16:42 +00:00
teor c812f880cf
cleanup(clippy): Use inline format strings (#5489)
* Inline format strings using an automated clippy fix

```sh
cargo clippy --fix --all-features --all-targets -- -A clippy::all -W clippy::uninlined_format_args
cargo fmt --all
```

* Remove unused & and &mut using an automated clippy fix

```sh
cargo clippy --fix --all-features --all-targets -- -A clippy::all -W clippy::uninlined_format_args
```
2022-10-27 13:25:18 +00:00
Alfredo Garcia 868ba1325e
refactor(config): Move configs to components (#5460)
* move configs to componenets

* remove trailing slash

* remove link from function behind feature

* fix config call in metrics
2022-10-24 23:39:00 +00:00
Arya e9452d9a6f
change(mempool) reject transactions with spent outpoints or nullifiers (#5434)
* adds transactions to ChainTipBlock and rejects transactions from the mempool that have been invalidated by the latest block commit

* merges remove_same_effects in with reject_invalidated_transactions, moves nullifier retrieval to the new Storage method, rejects mined_ids, and updates tests

* updates DuplicateSpend's error message

* fixes tests

* Update zebrad/src/components/mempool/storage.rs

Co-authored-by: teor <teor@riseup.net>

* adds comment

* formatting for the proptest

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
2022-10-24 19:48:45 +00:00
teor 737fbac3fc
Allow extra lookahead blocks in the verifier, state, and block commit task queues (#5465) 2022-10-24 14:55:57 +00:00
teor fd517ee4a7
Make timeouts shorter to reduce full sync times (#5397) 2022-10-14 20:20:24 +00:00
Arya 1cfbe64039
fix(test): resolve a timing issue in fake_peer_set setup (#5398)
* waits for the chain tip update if setup adds mempool transactions and adds match arm for Response::Nil to MempoolTransactionIds request

* updates other match statements checking MempoolTransactionIds requests

* removes wait for chain tip update and waits for AdvertiseBlock earlier instead
2022-10-13 13:34:40 +00:00
Arya a28350e742
change(state): Write non-finalized blocks to the state in a separate thread, to avoid network and RPC hangs (#5257)
* Add a new block commit task and channels, that don't do anything yet

* Add last_block_hash_sent to the state service, to avoid database accesses

* Update last_block_hash_sent regardless of commit errors

* Rename a field to StateService.max_queued_finalized_height

* Commit finalized blocks to the state in a separate task

* Check for panics in the block write task

* Wait for the block commit task in tests, and check for errors

* Always run a proptest that sleeps once

* Add extra debugging to state shutdowns

* Work around a RocksDB shutdown bug

* Close the finalized block channel when we're finished with it

* Only reset state queue once per error

* Update some TODOs

* Add a module doc comment

* Drop channels and check for closed channels in the block commit task

* Close state channels and tasks on drop

* Remove some duplicate fields across StateService and ReadStateService

* Try tweaking the shutdown steps

* Update and clarify some comments

* Clarify another comment

* Don't try to cancel RocksDB background work on drop

* Fix up some comments

* Remove some duplicate code

* Remove redundant workarounds for shutdown issues

* Remode a redundant channel close in the block commit task

* Remove a mistaken `!force` shutdown condition

* Remove duplicate force-shutdown code and explain it better

* Improve RPC error logging

* Wait for chain tip updates in the RPC tests

* Wait 2 seconds for chain tip updates before skipping them

* Remove an unnecessary block_in_place()

* Fix some test error messages that were changed by earlier fixes

* Expand some comments, fix typos

Co-authored-by: Marek <mail@marek.onl>

* Actually drop children of failed blocks

* Explain why we drop descendants of failed blocks

* Clarify a comment

* Wait for chain tip updates in a failing test on macOS

* Clean duplicate finalized blocks when the non-finalized state activates

* Send an error when receiving a duplicate finalized block

* Update checkpoint block behaviour, document its consensus rule

* Wait for chain tip changes in inbound_block_height_lookahead_limit test

* Wait for the genesis block to commit in the fake peer set mempool tests

* Disable unreliable mempool verification check in the send transaction test

* Appease rustfmt

* Use clear_finalized_block_queue() everywhere that blocks are dropped

* Document how Finalized and NonFinalized clones are different

* sends non-finalized blocks to the block write task

* passes ZebraDb to commit_new_chain, commit_block, and no_duplicates_in_finalized_chain instead of FinalizedState

* Update zebra-state/src/service/write.rs

Co-authored-by: teor <teor@riseup.net>

* updates comments, renames send_process_queued, other minor cleanup

* update assert_block_can_be_validated comment

* removes `mem` field from StateService

* removes `disk` field from StateService and updates block_iter to use `ZebraDb` instead of the finalized state

* updates tests that use the disk to use read_service.db instead

* moves best_tip to a read fn and returns finalized & non-finalized states from setup instead of the state service

* changes `contextual_validity` to get the network from the finalized_state instead of another param

* swaps out StateService with FinalizedState and NonFinalizedState in tests

* adds NotReadyToBeCommitted error and returns it from validate_and_commit when a blocks parent hash is not in any chain

* removes NonFinalizedWriteCmd and calls, moves update_latest_channels above rsp_tx.send

* makes parent_errors_map an indexmap

* clears non-finalized block queue when the receiver is dropped and when the StateService is being dropped

* sends non-finalized blocks to the block write task

* passes ZebraDb to commit_new_chain, commit_block, and no_duplicates_in_finalized_chain instead of FinalizedState

* updates comments, renames send_process_queued, other minor cleanup

* Update zebra-state/src/service/write.rs

Co-authored-by: teor <teor@riseup.net>

* update assert_block_can_be_validated comment

* removes `mem` field from StateService

* removes `disk` field from StateService and updates block_iter to use `ZebraDb` instead of the finalized state

* updates tests that use the disk to use read_service.db instead

* moves best_tip to a read fn and returns finalized & non-finalized states from setup instead of the state service

* changes `contextual_validity` to get the network from the finalized_state instead of another param

* swaps out StateService with FinalizedState and NonFinalizedState in tests

* adds NotReadyToBeCommitted error and returns it from validate_and_commit when a blocks parent hash is not in any chain

* removes NonFinalizedWriteCmd and calls, moves update_latest_channels above rsp_tx.send

* makes parent_errors_map an indexmap

* clears non-finalized block queue when the receiver is dropped and when the StateService is being dropped

* removes duplicate field definitions on StateService that were a result of a bad merge

* update NotReadyToBeCommitted error message

* Appear rustfmt

* Fix doc links

* Rename a function to initial_contextual_validity()

* Do error tasks on Err, and success tasks on Ok

* Simplify parent_error_map truncation

* Rewrite best_tip() to use tip()

* Rename latest_mem() to latest_non_finalized_state()

```sh
fastmod latest_mem latest_non_finalized_state zebra*
cargo fmt --all
```

* Simplify latest_non_finalized_state() using a new WatchReceiver API

* Expand some error messages

* Send the result after updating the channels, and document why

* wait for chain_tip_update before cancelling download in mempool_cancel_mined

* adds `sent_non_finalized_block_hashes` field to StateService

* adds batched sent_hash insertions and checks sent hashes in queue_and_commit_non_finalized before adding a block to the queue

* check that the `curr_buf` in SentHashes is not empty before pushing it to the `sent_bufs`

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Fix rustfmt

* Check for finalized block heights using zs_contains()

* adds known_utxos field to SentHashes

* updates comment on SentHashes.add method

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* return early when there's a duplicate hash in QueuedBlocks.queue instead of panicking

* Make finalized UTXOs near the final checkpoint available for full block verification

* Replace a checkpoint height literal with the actual config

* Update mainnet and testnet checkpoints - 7 October 2022

* Fix some state service init arguments

* Allow more lookahead in the downloader, but less lookahead in the syncer

* Add the latest config to the tests, and fix the latest config check

* Increase the number of finalized blocks checked for non-finalized block UTXO spends

* fix(log): reduce verbose logs for block commits (#5348)

* Remove some verbose block write channel logs

* Only warn about tracing endpoint if the address is actually set

* Use CloneError instead of formatting a non-cloneable error

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Increase block verify timeout

* Work around a known block timeout bug by using a shorter timeout

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-10-11 19:25:45 +00:00
Arya 7207f9d732
waits for tip change after each commit call to the state-service in mempool_cancel_mined (#5347) 2022-10-06 10:36:42 +00:00
Arya 59d647cfa1
wait for chain_tip_update before cancelling download in mempool_cancel_mined (#5322) 2022-10-04 02:52:03 +00:00
teor 343c5e68d4
change(state): Write finalized blocks to the state in a separate thread, to avoid network and RPC hangs (#5134)
* Add a new block commit task and channels, that don't do anything yet

* Add last_block_hash_sent to the state service, to avoid database accesses

* Update last_block_hash_sent regardless of commit errors

* Rename a field to StateService.max_queued_finalized_height

* Commit finalized blocks to the state in a separate task

* Check for panics in the block write task

* Wait for the block commit task in tests, and check for errors

* Always run a proptest that sleeps once

* Add extra debugging to state shutdowns

* Work around a RocksDB shutdown bug

* Close the finalized block channel when we're finished with it

* Only reset state queue once per error

* Update some TODOs

* Add a module doc comment

* Drop channels and check for closed channels in the block commit task

* Close state channels and tasks on drop

* Remove some duplicate fields across StateService and ReadStateService

* Try tweaking the shutdown steps

* Update and clarify some comments

* Clarify another comment

* Don't try to cancel RocksDB background work on drop

* Fix up some comments

* Remove some duplicate code

* Remove redundant workarounds for shutdown issues

* Remode a redundant channel close in the block commit task

* Remove a mistaken `!force` shutdown condition

* Remove duplicate force-shutdown code and explain it better

* Improve RPC error logging

* Wait for chain tip updates in the RPC tests

* Wait 2 seconds for chain tip updates before skipping them

* Remove an unnecessary block_in_place()

* Fix some test error messages that were changed by earlier fixes

* Expand some comments, fix typos

Co-authored-by: Marek <mail@marek.onl>

* Actually drop children of failed blocks

* Explain why we drop descendants of failed blocks

* Clarify a comment

* Wait for chain tip updates in a failing test on macOS

* Clean duplicate finalized blocks when the non-finalized state activates

* Send an error when receiving a duplicate finalized block

* Update checkpoint block behaviour, document its consensus rule

* Wait for chain tip changes in inbound_block_height_lookahead_limit test

* Wait for the genesis block to commit in the fake peer set mempool tests

* Disable unreliable mempool verification check in the send transaction test

* Appease rustfmt

* Use clear_finalized_block_queue() everywhere that blocks are dropped

* Document how Finalized and NonFinalized clones are different

* Use the same check as commit_finalized() for finalized block heights

Co-authored-by: Marek <mail@marek.onl>

Co-authored-by: Marek <mail@marek.onl>
2022-09-28 16:09:56 +00:00
dependabot[bot] d8b0481a4c
build(deps): bump tracing-subscriber from 0.3.11 to 0.3.15 (#5223)
* build(deps): bump tracing-subscriber from 0.3.11 to 0.3.15

Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.3.11 to 0.3.15.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.3.11...tracing-subscriber-0.3.15)

---
updated-dependencies:
- dependency-name: tracing-subscriber
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Note that this dependency update fixes a known Zebra logging bug

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
2022-09-22 04:50:56 +00:00
teor 806dd0f24c
feat(net): return peer metadata from `connect_isolated` functions (#4870)
* Move version into a ConnectionInfo struct

* Add negotiated version to ConnectionInfo

Part of this change was generated using:
```
fastmod --fixed-strings ".version(" ".remote_version(" zebra-network
```

* Add the peer address to ConnectionInfo, add ConnectionInfo to Connection

* Return a Client instance from connect_isolated_* functions

This allows library users to access client ConnectionInfo.

* Add and improve debug formatting

* Add peer services and user agent to ConnectionInfo

* Export the Client type, and fix up a zebrad test

* Export types used by the public API

* Split VersionMessage into its own struct

* Use VersionMessage in ConnectionInfo

* Add a public API test for ConnectionInfo

* Wrap ConnectionInfo in an Arc

* Fix some doc links
2022-09-14 15:00:25 +00:00
teor 43e4a29be2
Fix a new nightly clippy "unwrap_or_else uses default" lint (#5131) 2022-09-12 23:59:07 +00:00
Arya d9fae6e311
fix(test) switch zebrad to a non-blocking tracing logger (#5032)
* adds non-blocking writer for tracing subscriber

* use non_blocking writer for the fmt::Layer with the tokio-console feature as well

* adds doc comment to _guard field

* adds acceptance test

* update filter_handle type to use NonBlocking

* adds more detail on lossy non-blocking writer and sets tracing.filter to "trace" in acceptance test

* drops ZebradApp before process::exit(1) in the event of a FrameworkError

* reduces buffered lines limit to 8000

* adds tracing.buffer_limit config and some comments

* update acceptance.rs

* fix acceptance test

* fixes ambigious phrasing in comment

* updates zebrad/src/application.rs

* Find out what the join error is in the GitHub runner tests

* updates acceptance test to use recv_timeout instead of always waiting 10 seconds, removes unnecessary echo command, and reduces # of rpc requests to 500

* see if sleeping for a few seconds before exiting helps the macOS test pass

* Expand exit sleep docs

Co-authored-by: Arya <aryasolhi@gmail.com>

Co-authored-by: teor <teor@riseup.net>
2022-09-07 07:39:30 +00:00
teor ea34baa8b8
add(test): test disabled `lightwalletd` mempool gRPCs via zebrad logs (#5016)
* add grpc mempool test research

* add a config flag for mempool injection of transactions in test

* Only copy the inner state directory in the send transactions test

* Preload Zcash parameters in some transaction verification tests

* Add a block and transaction Hash method to convert from display order bytes

* Update test coverage docs

* Add debugging output for mempool transaction verification

* Test fetching sent mempool transactions using gRPC

* Add extra log checks to the send transaction test

* Wait for zebrad mempool activation before running gRPC tests

* Update send transaction test for lightwalletd not returning mempool transactions

* Check zebrad logs instead of disabled lightwalletd gRPCs

* Add a debug option that makes RPCs pretend the sync is finished

* Remove an unused debug option

* Remove unused test code and downgrade some logs

* Fix test log checks

* Fix some rustdoc warnings

* Fix a compilation error due to new function arguments

* Make zebrad sync timeouts consistent and remove outdated code

* Document how to increase temporary directory space for tests

* Stop checking for a log that doesn't always happen

* Remove some commented-out code

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Update a comment about run time

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Add new config to new tests from the `main` branch

* Add transactions to the list, rather than replacing the list with each new block

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2022-09-06 13:32:33 +00:00
Conrado Gouvea 6fd750e168
build(deps): bump insta from 1.15.0 to 1.17.1 (#4884)
* build(deps): bump insta from 1.15.0 to 1.17.1

Bumps [insta](https://github.com/mitsuhiko/insta) from 1.15.0 to 1.17.1.
- [Release notes](https://github.com/mitsuhiko/insta/releases)
- [Changelog](https://github.com/mitsuhiko/insta/blob/master/CHANGELOG.md)
- [Commits](https://github.com/mitsuhiko/insta/compare/1.15.0...1.17.1)

---
updated-dependencies:
- dependency-name: insta
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* make zebra_test::init() return the insta drop guard

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-08-04 15:44:44 +00:00
Alfredo Garcia a6f15e84a8
fix(test): truncate the number of transactions in send transaction test (#4848)
* truncate the number of transactions in send transaction test

* Limit send transaction test to RPC queue length

Co-authored-by: teor <teor@riseup.net>
2022-08-01 03:48:19 +00:00
teor c811457f46
Allow the mempool_transaction_expiration test to get network requests in any order (#4830) 2022-07-27 12:15:53 +10:00
teor f81e997090
2. fix(perf): Run CPU-intensive state reads in parallel rayon threads (#4805)
* Split disk reads from CPU-heavy Sprout interstitial tree cryptography

* Improve anchor validation debugging and error messages

* Work around a test data bug, and save some CPU

* Remove redundant checks for empty shielded data

* Skip generating unused interstitial treestates

* Do disk fetches and quick checks, then CPU-heavy cryptography

* Wrap HistoryTree in an Arc in the state

* Run CPU-intensive chain validation and updates in parallel rayon threads

* Refactor to prepare for parallel tree root calculations

* Run finalized state note commitment tree root updates in parallel rayon threads

* Update finalized state note commitment trees using parallel rayon threads

* Fix a comment typo and add a TODO

* Split sprout treestate fetch into its own function

* Move parallel note commitment trees to zebra-chain

* Re-calculate the tree roots in the same parallel batches

* Do non-finalized note commitment tree updates in parallel threads

* Update comments about note commitment tree rebuilds

* Do post-fork tree updates in parallel threads

* Add a TODO for parallel tree updates in tests

* Fix broken intra-doc links

* Clarify documentation for sprout treestates

* Spawn large database reads into blocking tokio threads

* Concurrently read all blocks, headers, and transactions from disk

* Run zebra-state transaction deserialization on a rayon thread
2022-07-22 16:25:32 +00:00
teor 7b1d4527d4
add(log): Log the amount of time it takes to rebuild note commitment trees after a chain fork (#4795)
* Update comments about note commitment tree rebuilds

* Add info-level logs to time note commitment tree rebuilds

* Log the number of rebuilt blocks after a fork, and the time per block

* Move humantime formats to zebra-chain

* Use human-friendly time formatting
2022-07-22 09:17:34 +10:00
teor 394d16a5a5
2. fix(perf): When writing blocks to disk, don't block other async tasks (#4199)
* Only fetch block headers from the database to answer headers requests

* Move writing to the database to a blocking thread

* Add blocking threads to tests that need them

* Remove mempool downloader requirement for a populated state

And stop populating states that don't need it,
so we can use tokio::time::pause() in those tests.

* Improve debugging for an intermittent test failure

* Try to avoid a race condition populating the mempool in tests
2022-07-22 09:16:41 +10:00
Alfredo Garcia 71fe4c4c73
try to do deserialization of transaction in a rayon thread (#4801)
* try to do deserialization of transaction in a rayon thread

* Try tokio::task::block_in_place instead

* fix tests

* add deserialize block into rayon pool

* fill some docs

Co-authored-by: teor <teor@riseup.net>
2022-07-22 09:14:58 +10:00
teor cf4b2f7a67
feat(verify): Concurrently verify proof and signature batches (#4776)
* Initialize the rayon threadpool with a new config for CPU-bound threads

* Verify proofs and signatures on the rayon thread pool

* Only spawn one concurrent batch per verifier, for now

* Allow tower-batch to queue multiple batches

* Fix up a potentially incorrect comment

* Rename some variables for concurrent batches

* Spawn multiple batches concurrently, without any limits

* Simplify batch worker loop using OptionFuture

* Clear pending batches once they finish

* Stop accepting new items when we're at the concurrent batch limit

* Fail queued requests on drop

* Move pending_items and the batch timer into the worker struct

* Add worker fields to batch trace logs

* Run docker tests on PR series

* During full verification, process 20 blocks concurrently

* Remove an outdated comment about yielding to other tasks
2022-07-18 08:43:29 +10:00
teor 87f4308caf
fix(sync): Temporarily set full verification concurrency to 30 blocks (#4726)
* Return the maximum checkpoint height from the chain verifier

* Return the verified block height from the sync downloader

* Track the verified height in the syncer

* Use a lower concurrency limit during full verification

* Get the tip from the state before the first verified block

* Limit the number of submitted download and verify blocks in a batch

* Adjust lookahead limits when transitioning to full verification

* Keep unused extra hashes and submit them to the downloader later

* Remove redundant verified_height and state_tip()

* Split the checkpoint and full verify concurrency configs

* Decrease full verification concurrency to 5 blocks

10 concurrent blocks causes 3 minute stalls on some blocks on my machine.
(And it has about 4x as many cores as a standard machine.)

* cargo +stable fmt --all

* Remove a log that's verbose with smaller lookahead limits

* Apply the full verify concurrency limit to the inbound service

* Add a summary of the config changes to the CHANGELOG

* Increase the default full verify concurrency limit to 30
2022-07-06 10:13:57 -04:00
Alfredo Garcia 97fb85dca9
lint(clippy): add `unwrap_in_result` lint (#4667)
* `unwrap_in_result` in zebra-chain crate

* `unwrap_in_result` in zebra-script crate

* `unwrap_in_result` in zebra-state crate

* `unwrap_in_result` in zebra-consensus crate

* `unwrap_in_result` in zebra-test crate

* `unwrap_in_result` in zebra-network crate

* `unwrap_in_result` in zebra-rpc crate

* `unwrap_in_result` in zebrad crate

* rustfmt

* revert `?` and add exceptions

* explain some panics better

* move some lint positions

* replace a panic with error

* Fix rustfmt?

Co-authored-by: teor <teor@riseup.net>
2022-06-28 06:22:07 +00:00
teor d4b9353d67
feat(log): Show the current network upgrade in progress logs (#4694)
* Improve time logging using humantime

* Only log full seconds, ignore the fractional part

* Move humantime_seconds to tracing::fmt

* Move the progress task to its own module

* Add missing humantime dependency

* Log the network upgrade in progress logs

* Log when Zebra verifies the final checkpoint
2022-06-28 02:51:41 +00:00
Alfredo Garcia 83aa42e649
tests(config): Add tests for old configs (#4676)
* change `initial_mainnet_peers` and `initial_testnet_peers` type to `IndexSet`

* add tests for zebra config files

* add serde feature to indexmap

* remove async

* update config

* fix `stored_config_path()`

* skip tests if config is not found

* improve error

* use CARGO_MANIFEST_DIR

* remove `stored_config_is_newest` test

* move `stored_config_works` test to the end of `valid_generated_config_test`

* space
2022-06-27 00:07:37 +00:00
teor 49cda21f37
Slightly increase some syncer defaults (#4679)
This is a partial revert of PR #4670.
2022-06-24 00:58:49 +00:00
teor c75a68e655
fix(sync): change default sync config to improve reliability (#4670)
* Decrease the default lookahead limit to 400

* Increase the block verification timeout to 10 minutes

* Halve the default concurrent downloads config

* Try to run the spawned download task before queueing the next download

* Allow verification to be cancelled if the verifier is busy
2022-06-22 18:17:21 +00:00
teor 3a7c2c8926
Replace the lookahead limit panic with a warning (#4662)
And decrease the minimum to 400 blocks
2022-06-21 11:07:32 +00:00