Commit Graph

287 Commits

Author SHA1 Message Date
teor ea34baa8b8
add(test): test disabled `lightwalletd` mempool gRPCs via zebrad logs (#5016)
* add grpc mempool test research

* add a config flag for mempool injection of transactions in test

* Only copy the inner state directory in the send transactions test

* Preload Zcash parameters in some transaction verification tests

* Add a block and transaction Hash method to convert from display order bytes

* Update test coverage docs

* Add debugging output for mempool transaction verification

* Test fetching sent mempool transactions using gRPC

* Add extra log checks to the send transaction test

* Wait for zebrad mempool activation before running gRPC tests

* Update send transaction test for lightwalletd not returning mempool transactions

* Check zebrad logs instead of disabled lightwalletd gRPCs

* Add a debug option that makes RPCs pretend the sync is finished

* Remove an unused debug option

* Remove unused test code and downgrade some logs

* Fix test log checks

* Fix some rustdoc warnings

* Fix a compilation error due to new function arguments

* Make zebrad sync timeouts consistent and remove outdated code

* Document how to increase temporary directory space for tests

* Stop checking for a log that doesn't always happen

* Remove some commented-out code

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Update a comment about run time

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Add new config to new tests from the `main` branch

* Add transactions to the list, rather than replacing the list with each new block

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2022-09-06 13:32:33 +00:00
Conrado Gouvea 6fd750e168
build(deps): bump insta from 1.15.0 to 1.17.1 (#4884)
* build(deps): bump insta from 1.15.0 to 1.17.1

Bumps [insta](https://github.com/mitsuhiko/insta) from 1.15.0 to 1.17.1.
- [Release notes](https://github.com/mitsuhiko/insta/releases)
- [Changelog](https://github.com/mitsuhiko/insta/blob/master/CHANGELOG.md)
- [Commits](https://github.com/mitsuhiko/insta/compare/1.15.0...1.17.1)

---
updated-dependencies:
- dependency-name: insta
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* make zebra_test::init() return the insta drop guard

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-08-04 15:44:44 +00:00
Alfredo Garcia a6f15e84a8
fix(test): truncate the number of transactions in send transaction test (#4848)
* truncate the number of transactions in send transaction test

* Limit send transaction test to RPC queue length

Co-authored-by: teor <teor@riseup.net>
2022-08-01 03:48:19 +00:00
teor c811457f46
Allow the mempool_transaction_expiration test to get network requests in any order (#4830) 2022-07-27 12:15:53 +10:00
teor f81e997090
2. fix(perf): Run CPU-intensive state reads in parallel rayon threads (#4805)
* Split disk reads from CPU-heavy Sprout interstitial tree cryptography

* Improve anchor validation debugging and error messages

* Work around a test data bug, and save some CPU

* Remove redundant checks for empty shielded data

* Skip generating unused interstitial treestates

* Do disk fetches and quick checks, then CPU-heavy cryptography

* Wrap HistoryTree in an Arc in the state

* Run CPU-intensive chain validation and updates in parallel rayon threads

* Refactor to prepare for parallel tree root calculations

* Run finalized state note commitment tree root updates in parallel rayon threads

* Update finalized state note commitment trees using parallel rayon threads

* Fix a comment typo and add a TODO

* Split sprout treestate fetch into its own function

* Move parallel note commitment trees to zebra-chain

* Re-calculate the tree roots in the same parallel batches

* Do non-finalized note commitment tree updates in parallel threads

* Update comments about note commitment tree rebuilds

* Do post-fork tree updates in parallel threads

* Add a TODO for parallel tree updates in tests

* Fix broken intra-doc links

* Clarify documentation for sprout treestates

* Spawn large database reads into blocking tokio threads

* Concurrently read all blocks, headers, and transactions from disk

* Run zebra-state transaction deserialization on a rayon thread
2022-07-22 16:25:32 +00:00
teor 7b1d4527d4
add(log): Log the amount of time it takes to rebuild note commitment trees after a chain fork (#4795)
* Update comments about note commitment tree rebuilds

* Add info-level logs to time note commitment tree rebuilds

* Log the number of rebuilt blocks after a fork, and the time per block

* Move humantime formats to zebra-chain

* Use human-friendly time formatting
2022-07-22 09:17:34 +10:00
teor 394d16a5a5
2. fix(perf): When writing blocks to disk, don't block other async tasks (#4199)
* Only fetch block headers from the database to answer headers requests

* Move writing to the database to a blocking thread

* Add blocking threads to tests that need them

* Remove mempool downloader requirement for a populated state

And stop populating states that don't need it,
so we can use tokio::time::pause() in those tests.

* Improve debugging for an intermittent test failure

* Try to avoid a race condition populating the mempool in tests
2022-07-22 09:16:41 +10:00
Alfredo Garcia 71fe4c4c73
try to do deserialization of transaction in a rayon thread (#4801)
* try to do deserialization of transaction in a rayon thread

* Try tokio::task::block_in_place instead

* fix tests

* add deserialize block into rayon pool

* fill some docs

Co-authored-by: teor <teor@riseup.net>
2022-07-22 09:14:58 +10:00
teor cf4b2f7a67
feat(verify): Concurrently verify proof and signature batches (#4776)
* Initialize the rayon threadpool with a new config for CPU-bound threads

* Verify proofs and signatures on the rayon thread pool

* Only spawn one concurrent batch per verifier, for now

* Allow tower-batch to queue multiple batches

* Fix up a potentially incorrect comment

* Rename some variables for concurrent batches

* Spawn multiple batches concurrently, without any limits

* Simplify batch worker loop using OptionFuture

* Clear pending batches once they finish

* Stop accepting new items when we're at the concurrent batch limit

* Fail queued requests on drop

* Move pending_items and the batch timer into the worker struct

* Add worker fields to batch trace logs

* Run docker tests on PR series

* During full verification, process 20 blocks concurrently

* Remove an outdated comment about yielding to other tasks
2022-07-18 08:43:29 +10:00
teor 87f4308caf
fix(sync): Temporarily set full verification concurrency to 30 blocks (#4726)
* Return the maximum checkpoint height from the chain verifier

* Return the verified block height from the sync downloader

* Track the verified height in the syncer

* Use a lower concurrency limit during full verification

* Get the tip from the state before the first verified block

* Limit the number of submitted download and verify blocks in a batch

* Adjust lookahead limits when transitioning to full verification

* Keep unused extra hashes and submit them to the downloader later

* Remove redundant verified_height and state_tip()

* Split the checkpoint and full verify concurrency configs

* Decrease full verification concurrency to 5 blocks

10 concurrent blocks causes 3 minute stalls on some blocks on my machine.
(And it has about 4x as many cores as a standard machine.)

* cargo +stable fmt --all

* Remove a log that's verbose with smaller lookahead limits

* Apply the full verify concurrency limit to the inbound service

* Add a summary of the config changes to the CHANGELOG

* Increase the default full verify concurrency limit to 30
2022-07-06 10:13:57 -04:00
Alfredo Garcia 97fb85dca9
lint(clippy): add `unwrap_in_result` lint (#4667)
* `unwrap_in_result` in zebra-chain crate

* `unwrap_in_result` in zebra-script crate

* `unwrap_in_result` in zebra-state crate

* `unwrap_in_result` in zebra-consensus crate

* `unwrap_in_result` in zebra-test crate

* `unwrap_in_result` in zebra-network crate

* `unwrap_in_result` in zebra-rpc crate

* `unwrap_in_result` in zebrad crate

* rustfmt

* revert `?` and add exceptions

* explain some panics better

* move some lint positions

* replace a panic with error

* Fix rustfmt?

Co-authored-by: teor <teor@riseup.net>
2022-06-28 06:22:07 +00:00
teor d4b9353d67
feat(log): Show the current network upgrade in progress logs (#4694)
* Improve time logging using humantime

* Only log full seconds, ignore the fractional part

* Move humantime_seconds to tracing::fmt

* Move the progress task to its own module

* Add missing humantime dependency

* Log the network upgrade in progress logs

* Log when Zebra verifies the final checkpoint
2022-06-28 02:51:41 +00:00
Alfredo Garcia 83aa42e649
tests(config): Add tests for old configs (#4676)
* change `initial_mainnet_peers` and `initial_testnet_peers` type to `IndexSet`

* add tests for zebra config files

* add serde feature to indexmap

* remove async

* update config

* fix `stored_config_path()`

* skip tests if config is not found

* improve error

* use CARGO_MANIFEST_DIR

* remove `stored_config_is_newest` test

* move `stored_config_works` test to the end of `valid_generated_config_test`

* space
2022-06-27 00:07:37 +00:00
teor 49cda21f37
Slightly increase some syncer defaults (#4679)
This is a partial revert of PR #4670.
2022-06-24 00:58:49 +00:00
teor c75a68e655
fix(sync): change default sync config to improve reliability (#4670)
* Decrease the default lookahead limit to 400

* Increase the block verification timeout to 10 minutes

* Halve the default concurrent downloads config

* Try to run the spawned download task before queueing the next download

* Allow verification to be cancelled if the verifier is busy
2022-06-22 18:17:21 +00:00
teor 3a7c2c8926
Replace the lookahead limit panic with a warning (#4662)
And decrease the minimum to 400 blocks
2022-06-21 11:07:32 +00:00
teor 29e73b3f3e
breaking(diagnostics): make zebrad diagnostics into optional compile-time features (#4539)
* Disable the flamegraph feature by default at compile time

* Disable the journald feature by default at compile time

* Also disable inferno dependency, and rearrange features

* Disable the prometheus feature by default at compile time

* Disable the tracing filter reload feature by default at compile time

* Disable tests when corresponding features are disabled

* Add compile-time tracing features to user docs

* Add compile-time features to the metrics user docs

* Document diagnostics as part of the start command tasks and services

* breaking(diagnostics): rename "enable-sentry" feature to "sentry" (#4623)

* Also skip conflict tests when those ports are disabled

* breaking(diagnostics): rename "enable-sentry" feature to "sentry"

This is mostly:
```sh
fastmod enable-sentry sentry
```

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-06-16 19:56:40 +00:00
teor 83b4e6f975
feat(diagnostics): Add tokio-console support to zebrad (#4519)
* Always activate tokio/tracing feature

And always build tests with all tokio features.

* Refactor tracing-subscriber init to simplify it

* Add the tokio-console feature and dependencies

* Add optional tokio-console support, and log the installed tracing layers at info level

Uses a tracing Registry for tokio-console, and a fmt::Subscriber otherwise.

* Add some TODOs based on tracing-subscriber features

* Fix up some spans

* Add a TODO for fixing a log filter bug in tokio-console mode
2022-06-15 16:43:20 +10:00
Marek cc75c3f5f9
fix(doc): Fix various doc warnings, part 3 (#4611)
* Fix the syntax of links in comments

* Fix a mistake in the docs

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Remove unnecessary angle brackets from a link

* Revert the changes for links that serve as references

* Revert "Revert the changes for links that serve as references"

This reverts commit 8b091aa9fab453e7d3559a5d474e0879183b9bfb.

* Remove `<` `>` from links that serve as references

This reverts commit 046ef25620ae1a2140760ae7ea379deecb4b583c.

* Don't use `<` `>` in normal comments

* Don't use `<` `>` for normal comments

* Revert changes for comments starting with `//`

* Fix some warnings produced by `cargo doc`

* Fix some rustdoc warnings

* Fix some warnings

* Refactor some changes

* Fix some rustdoc warnings

* Fix some rustdoc warnings

* Resolve various TODOs

Co-authored-by: teor <teor@riseup.net>

* Fix some unresolved links

* Allow links to private items

* Fix some unresolved links

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-06-15 03:57:19 +00:00
Alfredo Garcia 6d9bb2226a
fix(config): Duration fields (#4587)
* use `humantime_serde` for config durations

* move debug config option to the bottom

* fix deserialization

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-06-14 06:21:24 +00:00
teor 92fd11f9ad
fix(network): when connecting to peers, ignore invalid ports, and prefer canonical ports (#4564)
* Move peer address validation into its own module

* Add a network parameter to AddressBook and some MetaAddr methods

* Reject invalid initial peers, and connect to them in preferred order

* Reject Flux/ZelCash and misconfigured Zcash peers

* Prefer canonical Zcash ports

* Make peer_preference into a struct method

* Prefer peer addresses with canonical ports for outbound connections

* Also ignore the Zcash regtest port

* Document where field and variant order is required for correctness

* Use the correct peer count

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-06-14 04:58:37 +00:00
Marek b8b35f8da9
fix(doc): Fix various doc warnings, part 1 (#4514)
* Fix the syntax of links in comments

* Fix a mistake in the docs

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Remove unnecessary angle brackets from a link

* Revert the changes for links that serve as references

* Revert "Revert the changes for links that serve as references"

This reverts commit 8b091aa9fab453e7d3559a5d474e0879183b9bfb.

* Remove `<` `>` from links that serve as references

This reverts commit 046ef25620ae1a2140760ae7ea379deecb4b583c.

* Don't use `<` `>` in normal comments

* Don't use `<` `>` for normal comments

* Revert changes for comments starting with `//`

* Fix some warnings produced by `cargo doc`

* Fix some rustdoc warnings

* Fix some warnings

* Refactor some changes

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2022-06-02 15:07:35 +00:00
teor ca0520b2e8
change(deps): Upgrade tracing-subscriber and related dependencies (#4517)
* Upgrade tracing and related dependencies

```sh
cargo upgrade --workspace
tracing-error
tracing-subscrber

color-eyre

tracing-flame
tracing-journald

sentry
sentry-tracing

metrics
metrics-exporter-prometheus
reqwest
```

* Update duplicate dependency checks

* Enable the tracing/env-filter feature

* Fix type inference for metrics

Manual changes, plus:
```sh
fastmod "as _" "as f64"
```

* Tidy up some unrelated test code

* Update metrics-exporter-prometheus API

And make unused dependencies optional.

* Adjust test regexes to new tracing format

Also fix some regex bugs, and refactor to simplify.

* Disable color-eyre span traces and track caller in release builds

* Add a feature that enables extra debugging in release builds

* Clean up some redundant features

* Increase a test timeout
2022-06-01 13:53:51 +10:00
teor 77d9d8f017
feat(log): Log the tracing level when it is set or reloaded (#4515)
* Log the tracing level when it is set or reloaded

* Add the `log` crate as a direct dependency

* Make `log` required
2022-05-27 19:49:51 +00:00
teor c2769dcc50
Activate the mempool after 2 tip syncer runs (#4501)
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-05-26 18:51:52 +00:00
teor fee10ae014
Add height and hash info to syncer errors (#4287) 2022-05-11 06:51:06 +00:00
teor 17c5e9ad6f
Drop sentry dependencies when enable-sentry feature is disabled (#4372) 2022-05-11 05:00:14 +00:00
teor a98e9291b5
cleanup(clippy): Remove redundant bindings, allocations, and generics (#4353)
* Remove let bindings with unit values

* Replace push(format!(_)) with write!(_)

* Remove unused lifetime in test
2022-05-10 03:41:51 +00:00
Janito Vaqueiro Ferreira Filho 79d58285fb
Increase block validation timeouts (#4156)
* Increase UTXO lookup timeout

Avoid block validation failures because UTXOs aren't available on time.

* Increase the block verification timeout

Attempt to reduce the synchronization restarts and consequently improve
performance.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-05-05 00:01:12 +00:00
teor 56f766f9b8
fix(sync): fix testnet syncer loop on large Orchard blocks (#4286)
* Return BlockDownloadVerifyError from download_and_verify

* Check block requests and genesis for temporary errors

* Ignore DuplicateBlockQueuedForDownload as a temporary error

* Propagate error info to the syncer main loop

* Sleep after temporary genesis download and verify errors
2022-05-04 22:04:34 +00:00
teor d476c18339
fix(sync): Add an extra block retry, to speed up the initial sync (#4185)
* Increase the block download retry limit to 3

* Remove an obsolete timing check from the tests

We cancel all block downloads for each sync restart,
so we don't need to wait for the maximum hedged download time.
2022-04-26 16:28:09 +00:00
Conrado Gouvea dff25473aa
refactor: document coinbase rules, refactor to ease understanding (#4056)
* refactor: document coinbase rules, refactor to ease understanding

* Update zebra-consensus/src/block/check.rs

Co-authored-by: teor <teor@riseup.net>

* remove no longer used contains_prevout_input()

* remove unused CoinbaseHasPrevOutInput

* update coinbase description in documentation

Co-authored-by: teor <teor@riseup.net>
2022-04-20 09:31:12 +00:00
Dimitris Apostolou a0c65181cb
Fix typos (#3956)
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-03-28 10:13:37 +10:00
teor 9d1702e93e
Make sync error logs more user-friendly (#3944)
- use info level, there is nothing the user needs to do,
  particularly for a single error
- explain that the errors are temporary
- hide backtraces, because they look like crashes
2022-03-26 02:28:38 +00:00
Conrado Gouvea 41d240feaf
feat(mempool): add TransactionsByMinedId (#3907) 2022-03-18 23:00:03 +00:00
Janito Vaqueiro Ferreira Filho 78080d88d4
fix(sync): prevent synchronizer loop when very close to tip (#3854)
* Refactor to split `ChainSync::sync` method in two

Replace the use of loop labels and `continue` for control flow, and use
early return from a separate method instead. This also allows removing
the `started_once` flag.

* Refactor to create `handle_block_response` helper

Reduce duplicate code and make the main synchronization methods a little
more concise to improve readability.

* Only cancel downloads in case of error

Leave active downloads running if the tips have been exhausted, because
it could have reached the chain tip.
2022-03-18 00:31:12 +00:00
teor 9ad47d1081
7. feat(state): add a read-only state service stub (#3835)
* Add state service module docs and cleanup

* Move and add finalized state methods

* Add chain and non-finalized state methods

* Cleanup methods and imports

* Create a ReadStateService type

* Add a stub service implementation

* Add a TODO

* Update ReadStateService request stubs with RPC names and tickets

* Documentation updates

* Make RPC State generic bounds accept a buffered state and a read-only state

* Doc updates

* Add missing proptest-impl feature in RPC dev dependencies
2022-03-11 10:58:22 -03:00
Janito Vaqueiro Ferreira Filho c24ea1fc3f
Refactor to create a new `zebra-node-services` crate (#3648)
* Create new empty `zebra-node-services` crate

The goal is to store the mempool `Request` and `Response` types so that
the `zebra-rpc` crate can interface with the mempool service without
having to import `zebrad`.

* Move `Gossip` mempool type into new crate

It is required by the `Request` type, which will be moved next.

* Add documentation to `Gossip` variants

Avoid having to add an exception to allow undocumented code.

* Move `mempool::Request` type to new crate

The first part of the service interface definition. Usages have been
changed to refer to the new crate directly, and since this refactor is
still incomplete, some `mp` aliases are used in a few places to refer to
the old module.

* Create an `UnboxMempoolError` helper trait

Centralize some common code to extract and downcast boxed mempool
errors. The `mempool::Response` will need to contain a `BoxError`
instead of a `MempoolError` when it is moved to the
`zebra-node-services` crate, so this prepares the tests to be updated
with less changes.

* Use `UnboxMempoolError` in tests

Make the necessary changes so that the tests are ready to support a
`BoxError` in the `mempool::Response` type.

* Use `BoxError` in `mempool::Response`

Prepare it to be moved to the `zebra-node-services` crate.

* Move `mempool::Response` to `zebra-node-services`

Update usages to import from the new crate directly.

* Remove `mp` aliases for mempool component module

Use any internal types directly instead.

* Replace `tower::BoxService` with custom alias

Remove the dependency of `zebra-node-services` on `tower`.

* Move `Gossip` into a separate `sub-module`

Keep only the main `Request` and `Response` types in the `mempool`
module.

* Use `crate::BoxError` instead of `tower::BoxError`

Follow the existing convention.

* Add missing `gossip.rs` module file

It was missing from a previous refactor commit.
2022-02-25 21:43:21 +00:00
Janito Vaqueiro Ferreira Filho 3ca653120a
Allow forcing `zebrad` to use color output (#3547)
* Allow forcing colored output in `zebrad`

Add a configuration item that allows forcing Zebra to output in color
mode even if the output device is not a terminal.

* Allow enabling colored output from Zebra in tests

Force Zebrad instances to use colored output if the
`ZEBRA_FORCE_USE_COLOR` environment variable is set.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-02-17 00:09:04 +00:00
teor a4dd3b7396
4. Avoid repeated requests to peers after partial responses or errors (#3505)
* fix(network): split synthetic NotFoundRegistry from message NotFoundResponse

* docs(network): Improve `notfound` message documentation

* refactor(network): Rename MustUseOneshotSender to MustUseClientResponseSender

```
fastmod MustUseOneshotSender MustUseClientResponseSender zebra*
```

* docs(network): fix a comment typo

* refactor(network): remove generics from MustUseClientResponseSender

* refactor(network): add an inventory collector to Client, but don't use it yet

* feat(network): register missing peer responses as missing inventory

We register this missing inventory based on peer responses,
or connection errors or timeouts.

Inbound message inventory tracking requires peers to send `notfound` messages.
But `zcashd` skips `notfound` for blocks, so we can't rely on peer messages.
This missing inventory tracking works regardless of peer `notfound` messages.

* refactor(network): rename ResponseStatus to InventoryResponse

```sh
fastmod ResponseStatus InventoryResponse zebra*
```

* refactor(network): rename InventoryStatus::inner() to to_inner()

* fix(network): remove a redundant runtime.enter() in a test

* doc(network): the exact time used to filter outbound peers doesn't matter

* fix(network): handle block requests slightly more efficiently

* doc(network): fix a typo

* fmt(network): `cargo fmt` after rename ResponseStatus to InventoryResponse

* doc(test): clarify some test comments

* test(network): test synthetic notfound from connection errors and peer inventory routing

* test(network): improve inbound test diagnostics

* feat(network): add a proptest-impl feature to zebra-network

* feat(network): add a test-only connect_isolated_with_inbound function

* test(network): allow a response on the isolated peer test connection

* test(network): fix failures in test synthetic notfound

* test(network): Simplify SharedPeerError test assertions

* test(network): test synthetic notfound from partially successful requests

* test(network): MissingInventoryCollector ignores local NotFoundRegistry errors

* fix(network): decrease the inventory rotation interval

This stops us waiting 3-4 sync resets (4 minutes) before we retry a missing block.

Now we wait 1-2 sync resets (2 minutes), which is still a reasonable rate limit.
This should speed up syncing near the tip, and on testnet.

* fmt(network): cargo fmt --all

* cleanup(network): remove unnecessary allow(dead_code)

* cleanup(network): stop importing the whole sync module into tests

* doc(network): clarify syncer inventory retry constraint

* doc(network): add a TODO for a fix to ensure API behaviour remains consistent

* doc(network): fix a function doc typo

* doc(network): clarify how we handle peers that don't send `notfound`

* docs(network): clarify a test comment

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-02-15 01:44:33 +00:00
teor 9f2028feff
3. Send notfound when Zebra doesn't have a block or transaction (#3466)
* refactor(network): rename Advertised to Available

```sh
fastmod Advertised Available zebra*
fastmod advertised available zebra*
```

* refactor(network): allow different available and missing types inside an InventoryStatus

And rename it to ResponseStatus.

Split the methods between ResponseStatus and an InventoryStatus alias.

* refactor(network): add a block_hash convenience method to InventoryHash

* test(network): improve failure logs for connection tests

* fix(inbound): move address sanitization into the response future

* feat(network): send notfound when Zebra doesn't have a block or transaction

* doc(network): move module docs to the top of each module

This makes them more likely to get updated when the module changes.

* fix(network): stop sending unsupported missing inventory types to the registry

* test(network): inbound messages are forwarded to the registry

* test(inbound): test Peers requests to the inbound service, directly and via TCP

* test(network): notfound block responses are sent by the inbound service

* test(network): notfound tx responses are sent by the inbound service

* test(network): increase sync test mock service timeout

The code that these tests use hasn't actually changed much,
and they are only failing on some platforms (coverage, macOS).

So it seems like the extra concurrent inbound tests have pushed them
past their time limit.
(Perhaps due to TCP system calls, or extra serialization work.)

* doc(network): fix typo

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* test(network): remove unnecessary multi-threaded runtime from tests

This prevents `MockService<zebra_state>` timeouts
in the `sync_block_too_high_extend_tips` test,
at the cost of reducing coverage of different execution orders.

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2022-02-14 01:51:34 +00:00
Janito Vaqueiro Ferreira Filho eb98b7a4b2
Estimate network chain tip height based on local node time and current best tip (#3492)
* Remove redundant documentation

The documentation was exactly the same as the documentation from the
trait.

* Calculate a mock time block delta for tests

Simulate a block being added to the chain with a random block time based
on the previous block time and the target spacing time.

* Add a `time` field to `ChainTipBlock`

Store the block time so that it's ready for a future chain that allows
obtaining the chain tip's block time.

* Add `ChainTip::best_tip_block_time` method

Allow obtaining the bes chain tip's block time.

* Add method to obtain both height and block time

Prevent any data races by returning both values so that they refer to
the same chain tip.

* Add `NetworkUpgrade::all_target_spacings` method

Returns all the target spacings defined for a network.

* Create a `NetworkChainTipEstimator` helper type

Isolate the code to calculate the height estimation in a new type, so
that it's easier to understand and doesn't decrease the readability of
the `chain_tip.rs` file.

* Add `ChainTip::estimate_network_chain_tip_height`

This is more of an extension method than a trait method. It uses the
`NetworkChainTipHeightEstimator` to actually perform the estimation, but
obtains the initial information from the current best chain tip.

* Fix typo in documentation

There was an extra closing bracket in the summary line.

* Refactor `MockChainTipSender` into a separate type

Prepare to allow mocking the block time of the best tip as well as the
block height.

* Allow sending mock best tip block times

Add a separate `watch` channel to send the best tip block times from a
`MockChainTipSender` to a `MockChainTip`.

The `best_tip_height_and_block_time` implementation will only return a
value if there's a height and a block time value for the best tip.

* Fix off-by-one height estimation error

Use Euclidean division to force the division result to round down
instead of rounding towards zero. This fixes an off-by-one error when
estimating a height that is lower than the current height, because the
fractionary result was being discarded, and it should have forced the
height to go one block back.

* Fix panics on local times very far in the past

Detect situations that might cause the block height estimate to
underflow, and return the genesis height instead.

* Fix another off-by-one height estimation error

The implementation of `chrono::Duration::num_seconds` adds one to the
number of seconds if it's negative. This breaks the division
calculation, so it has to be compensated for.

* Test network chain tip height estimation

Generate pairs of block heights and check that it's possible to estimate
the larger height from the smaller height and a displaced time
difference.
2022-02-11 01:27:02 +00:00
Deirdre Connolly 2f33300c08
Use string literal to format unreachable message in test (#3483)
* Use string literal to format unreachable message

* Try removing the redundant format

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-02-08 08:34:39 +00:00
teor fa071562fd
fix(network): increase state concurrency and syncer lookahead (#3455)
* fix(state): set state concurrency based on other services' concurrency

* fix(sync): increase the sync downloader lookahead limit

It seems like the recent tokio upgrade made this code even more efficient,
so on testnet we can have around 6000 blocks in flight.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-02-02 22:44:15 +00:00
Alfredo Garcia e5b5ea5889
feat(log): log the state tip height as part of sync progress logs (#3437)
* feat(log): log the state tip height as part of sync progress logs

* fix(log): downgrade some verbose state logs to debug

* feat(log): log successful gossiped block verification at info level

These logs help us diagnose slow progress near the tip.

There won't be very many of these logs,
because they only happen near the tip.

* fix(log): spawn top-level tasks within the global Zebra tracing span

* fix(log): spawn blocking top-level tasks within the global Zebra tracing span

Co-authored-by: teor <teor@riseup.net>
2022-01-28 19:12:19 -03:00
Alfredo Garcia 3c1ba59001
Reduce log level of components (#3418)
* reduce log level of components

* revert some log downgrades

* dedupe log
2022-01-28 14:24:53 -03:00
Alfredo Garcia cc594f1c58
Security: Stop RocksDB or tokio calling unexpected code when zebrad exits (#3392)
* cancel background database tasks in `FinalizedState` destructor

* use `shutdown_timeout()`

* Log info-level messages while waiting for background tasks to shut down

* Cancel background tasks during debug_stop_at_height shutdown

This commit moves the database shutdown code into a common function.

* Create a constant for the tokio timeout

* Add a test script for Zebra shutdown errors

* Increase the shutdown timeout to 20 seconds for slower machines

* add title to building zebra

* use imported duration

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-01-26 13:16:44 +00:00
teor d076b999f3
Fix syncer download order and add sync tests (#3168)
* Refactor so that RetryLimit::Future is std::marker::Sync

* Make the syncer future std::marker::Send by spawning tips futures

* Download synced blocks in chain order, not HashSet order

* Improve MockService failure messages

* Add closure-based responses to the MockService API

* Move MockChainTip to zebra-chain

* Add a MockChainTipSender type alias

* Support MockChainTip in ChainSync and its downloader

* Add syncer tests for obtain tips, extend tips, and wrong block hashes

* Add block too high tests for obtain tips and extend tips

* Add syncer tests for duplicate FindBlocks response hashes

* Allow longer request delays for mocked services in syncer tests
2022-01-11 14:11:35 -03:00
teor 3b75e912d1
Add a copy-state zebrad command, which copies blocks between two state services (#3175)
* Add a copy-state command, which copies blocks between two state services

* Check blocks were written correctly

* Add extra logging to debug shutdown

* Add a block height limit argument

* Let the target state start from any height

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-12-22 02:07:52 +00:00
Janito Vaqueiro Ferreira Filho b71833292d
Use `MockedClientHandle` in other tests (#3241)
* Move `MockedClientHandle` to `peer` module

It's more closely related to a `Client` than the `PeerSet`, and this
prepares it to be used by other tests.

* Rename `MockedClientHandle` to `ClientTestHarness`

Reduce confusion, and clarify that the client is not mocked.

Co-authored-by: teor <teor@riseup.net>

* Add clarification to `mock_peers` documentation

Explicitly say how the generated data is returned.

* Rename method to `wants_connection_heartbeats`

The `Client` service only represents one direction of a connection, so
`is_connected` is not the exact term.

Co-authored-by: teor <teor@riseup.net>

* Mock `Client` instead of `LoadTrackedClient`

Move where the conversion from mocked `Client` to mocked
`LoadTrackedClient` in order to make the test helper more easily used by
other tests.

* Use `ClientTestHarness` in `initialize` tests

Replace the boilerplate code to create a fake `Client` instance with
usages of the `ClientTestHarness` constructor.

* Allow receiving requests from `Client` instance

Create a helper type to wrap the result, to make it easier to assert on
specific events after trying to receive a request.

* Allow inspecting the current error in the slot

Share the `ErrorSlot` between the `Client` and the handle, so that the
handle can be used to inspect the contents of the `ErrorSlot`.

* Allow placing an error into the `ErrorSlot`

Assuming it is initially empty. If it already has an error, the code
will panic.

* Allow gracefully closing the request receiver

Close the endpoint with the appropriate call to the `close()` method.

* Allow dropping the request receiver endpoint

Forcefully closes the endpoint.

* Rename field to `client_request_receiver`

Also rename the related methods to include
`outbound_client_request_receiver` to make it more precise.

Co-authored-by: teor <teor@riseup.net>

* Allow dropping the heartbeat shutdown receiver

Allows the `Client` to detect that the channel has been closed.

* Rename fn. to `drop_heartbeat_shutdown_receiver`

Make it clear that it affects the heartbeat task.

Co-authored-by: teor <teor@riseup.net>

* Move `NowOrLater` into a new `now-or-later` crate

Make it easily accessible to other crates.

* Add `IsReady` extension trait for `Service`

Simplifies checking if a service is immediately ready to be called.

* Add extension method to check for readiness error

Checks if the `Service` isn't immediately ready because a call to
`ready` immediately returns an error.

* Rename method to `is_failed`

Avoid negated method names.

Co-authored-by: teor <teor@riseup.net>

* Add a `IsReady::is_pending` extension method

Checks if a `Service` is not ready to be called.

* Use `ClientTestHarness` in `Client` test vectors

Reduce repeated code and try to improve readability.

* Create a new `ClientTestHarnessBuilder` type

A builder to create test `Client` instances using mock data which can be
tracked and manipulated through a `ClientTestHarness`.

* Allow configuring the `Client`'s mocked version

Add a `with_version` builder method.

* Use `ClientTestHarnessBuilder` in `PeerVersions`

Use the builder to set the peer version, so that the `version` parameter
can be removed from the constructor later.

* Use a default mock version where possible

Reduce noise when setting up the harness for tests that don't really
care about the remote peer version.

* Remove `Version` parameter from the `build` method

The `with_version` builder method should be used instead.

* Fix some typos and outdated info in the release checklist

* Add extra client tests for zero and multiple readiness checks (#3273)

And document existing tests.

* Replace `NowOrLater` with `futures::poll!` (#3272)

* Replace NowOrLater with the futures::poll! macro in zebrad

* Replace NowOrLater with the futures::poll! macro in zebra-test

* Remove the now-or-later crate

* remove unused imports

* rustfmt

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-12-22 06:13:26 +10:00