* Join similar imports
Avoid the confusion that might cause one to think that they come from
different modules or crates.
* Create an `Arguments` helper type
A type to keep track of a list of arguments for a sub-process. It makes
it easier for overriding parameters with new values.
* Create an `args!` helper macro
Make it simpler to create `Arguments` instances with known values.
* Require `Arguments` for `spawn_child` method
Change the method to have an `Arguments` parameter, and merge it with
some default values before passing them forward.
* Use `Arguments` in `spawn_lightwalletd_child`
Change the method to use an `Arguments` instance, and merge it with some
default options.
* Use `Arguments` in `spawn_child_with_command`
Require an `Arguments` instance in the `spawn_child_with_command`
extension method. Makes it simpler to call from `spawn_child` and
`spawn_lightwalletd_child` extension methods.
* Test if argument order is preserved
Check that when building an `Arguments` instance, the order that the
arguments are set is preserved in the generated list of strings.
* Refactor test to improve readability
Also separates some common code to be reused by later tests.
* Test overriding arguments
Check to see if overriding arguments behaves as expected, by keeping the
argument order when overriding and not introducing duplicates.
* Refactor test to improve readability
Move out a chunk of code so that the test itself is easier to read and
to make that code reusable by a later test.
* Test that `Arguments` instances can be merged
Merge two `Arguments` instances built from two lists of arguments, and
check that the expanded strings preserve order and override rules.
* Add Eq derives on Arguments
Co-authored-by: teor <teor@riseup.net>
* Return error code -8 for missing blocks, like zcashd does
* Test that getblock returns error code -8 for missing blocks
* Make RegexSet failures easier to read in logs
* Update lightwalletd acceptance tests for log message changes
* Add extra failure strings for lightwalletd logs
* Improve link formatting
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* Implement `getblockchaininfo` RPC method
* add a test for `get_blockchain_info`
* fix tohex/fromhex
* move comment
* Update lightwalletd acceptance test for getblockchaininfo RPC (#3914)
* change(rpc): Return getblockchaininfo network upgrades in height order (#3915)
* Update lightwalletd acceptance test for getblockchaininfo RPC
* Update some doc comments for network upgrades
* List network upgrades in order in the getblockchaininfo RPC
Also:
- Use a constant for the "missing consensus branch ID" RPC value
- Simplify fetching consensus branch IDs
- Make RPC type derives consistent
- Update RPC type documentation
* Make RPC type derives consistent
* Fix a confusing test comment
* get hashand height at the same time
* fix estimated_height
* fix lint
* add extra check
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* fix typo
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* split test
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* fix(rpc): ignore an expected error in the RPC acceptance tests (#3961)
* Add ignored regexes to test command failure regex methods
* Ignore empty chain error in getblockchaininfo
We expect this error when zebrad starts up with an empty state.
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Make command test matching code accept generic regexes
And add generic conversions to regexes.
* Document test command structs
* Support matching multiple regexes internally in the test command
* Make it easier to call the generic regex methods
* Add a missing API usage comment
* Fix a potential hang in test child error reports
* Revert Option<Child> process handling
* Revert "Revert Option<Child> process handling"
This reverts commit 2af30086858d104dcb0ec87383996c36bcaa7371.
* Add a set of failure regexes to test command output
* Allow debug-printing TestChild again
* When the child is dropped, check any remaining output
* Document a wait_with_output edge case
* Improve failure regex panic output
* Improve builder ergonomics
* Add internal tests for failure regex panics
It would be easy to disable these panics, and never realise.
* Add some module structure TODOs
* Stop panicking if the child process has already been taken
* Add test APIs for consuming child output lines
* Fix a hang on child process drop
* Handle output being already taken in wait_with_output
And document some edge cases we don't handle yet
* Use bash's read command in the TestChild stderr test
And check the actual command we're using to see if it errors.
* Pretty print full failure regex list
* Add the test child command line to the failure regex logs
* Simplify the sync_until test code
* Check logs for failure messages in lightwalletd integration tests
* Add more TODOs and improve comments
* Allow missing branch ID 00000000 provided by zcashd
* Revert "Revert Option<Child> process handling"
This reverts commit 2af30086858d104dcb0ec87383996c36bcaa7371.
* Add a set of failure regexes to test command output
* Allow debug-printing TestChild again
* When the child is dropped, check any remaining output
* Document a wait_with_output edge case
* Improve failure regex panic output
* Improve builder ergonomics
* Add internal tests for failure regex panics
It would be easy to disable these panics, and never realise.
* Add some module structure TODOs
* Stop panicking if the child process has already been taken
* Add test APIs for consuming child output lines
* Fix a hang on child process drop
* Handle output being already taken in wait_with_output
And document some edge cases we don't handle yet
* Use bash's read command in the TestChild stderr test
And check the actual command we're using to see if it errors.
* Pretty print full failure regex list
* Add the test child command line to the failure regex logs
* feat(rpc): implement `getrawmempool` rpc method
* tests(rpc): start implementing a test
* Return a `Vec<String>` from `get_raw_mempool`
Ensure the returned type is serialized as an array of hexadecimal
strings.
* Make `getrawmempool` test a property test
Test it with random sets of transaction IDs.
* Fix code formatting inside `proptest!`
Run `rustfmt` inside the `proptest!` block.
* tests(rpc): fix lightwalletd test
* fix(rpc): remove non used return type
* tests(rpc): remove assert
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Allow extra time for lightwalletd integration tests
* Wait for a successful block ingestion in lightwalletd integration
* Check for any ingestor block, not just block 1
* feature(rpc): start adding a `getblock` method
* fix(rpc): replace oneshot
* fix(rpc): replace a panic with error
* fix(rpc): fix test
* feature(rpc): add hex to response
* refactor(rpc): use generic instead of alias
* docs(rpc): improve docs for getblock method
* test(rpc): add a test for getblock method
* deps(rpc): remove non needed tower features
Co-authored-by: teor <teor@riseup.net>
* docs(rpc): add a note to getblock doc
* refactor(rpc): replace alias
* fix(rpc): use `zcash_serialize_to_vec()` instead of logging format
* tests(rpc): add network argument to `populated_state()`
* refactor(rpc): use an error for state service readiness
* fix(rpc): add parameter
* fix(rpc): clippy
* nit(rpc): remove new line from imports
* fix(rpc): remove commented code
* fix(rpc): simplify error
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Use a `SerializedBlock` type to help serializing blocks (#3725)
* Create a `SerializedBlock` helper type
Create a type that can be used as a byte slice, but is guaranteed to
represent a valid block.
* Use `into_iter` instead of `iter`
There's no need to borrow the elements, they can be moved out directly.
This will be necessary because `&Arc<T>` doesn't implement `Borrow<T>`,
so a `SerializedBlock` can't be built directly from an `&Arc<Block>`.
* Use `SerializedBlock` in `GetBlock`
Make the type stricter to avoid storing possibly invalid values. The
bytes are still serialized as a hexadecimal string, through the usage of
`hex`.
The `serde::Deserialize` can't be derived because `hex` requires the
type to also implement `FromHex`.
* feature(rpc): add suggestions from code review
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* tests(rpc): make sure mempool has no requests in get_block test
* fix(rpc): change height argument type in getblock method
* fix(rpc): rustfmt
* fix(rpc): replace panic
* fix(rpc): change getblock response
* fix(rpc): fix lightwalletd test
* tests(rpc): add a getblock error test
* fix(rpc): try another regex
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Sometimes lightwalletd goes straight to "Another refresh in progress",
without logging "Mempool refresh error" first.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* fix(ci): remove an unused trigger path
* doc(ci): explain lightwalletd trigger paths
* fix(test): check for adityapk00/lightwalletd behaviour in test harness
* fix(ci): work around buildx command error
* fix(ci): revert the workaround
* tests(rpc): add an rpc endpoint test
* tests(rpc): add an rpc port conflict test
* tests(rpc): refactor some imports
* tests(rpc): fix failures, make test more complete
* tests(rpc): parse json response for better coverage
* tests(rpc): change request
* tests(rpc): wait until port is open in rpc_endpoint test
* tests(rpc): add a delay between launching 2 nodes
* tests(rpc): try 5 seconds
* refactor(rpc): open rpc server faster
* tests(rpc): extend `LAUNCH_DELAY` to 15 seconds
* fix(rpc): disable rpc_conflict test for windows
* fix(ci): skip the RPC tests if the network is disabled
* rustfmt
* fix(zebrad/test): test function return type
* tests(rpc): print server output in assert
* fix(rpc): fix acceptance test looking for string in `build` field
* fix(rpc): reduce the number of acceptable characters in version output
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* fix(test): only run lightwalletd test when the ZEBRA_TEST_LIGHTWALLETD env var is set
* fix(test): actually skip the test
* doc(zebrad): add some test TODOs
* doc(test): document zebrad-specific process launch methods
* refactor(zebrad): split lightwalletd launch into its own testing methods
* fix(zebrad/test): simplify file writing
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* refactor(zebrad/test): rename argument variables
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* lint(zebrad/tests): remove unused import
* fix(zebrad/test): restore SocketAddr import that was removed on main
* rustfmt
* fix(zebrad/test): update integration test to match adityapk00/lightwalletd behaviour
* rustfmt
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
The mempool is only activated once, so we must check for that log first.
After mempool activation, the stop regex is logged at least once.
(It might be logged before as well, but we can't rely on that.)
When checking that the mempool didn't activate,
wait for the `zebrad` command to exit,
then check the entire log.
* fix(test): make full sync tests more efficient
Increasing the lookahead limit should increase sync speed,
at the cost of increasing memory usage.
* doc(test): remove a redundant TODO
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* feat(log): log current height when logging sync progress
* fix(test): log the specific error when full sync tests fail
* doc(start): remove an obsolete TODO
We can't decrease this log level, because the tests rely on it.
* fix(test): wait until mempool activates in full sync tests
Changes the log message and log test so that the test only finishes
when the mempool has activated.
There is still a race condition here, between the log timer and
mempool activation. But it should be very rare, because the mempool
is activated immediately when `is_close_to_tip()` becomes true.
* fix(test): warn when Zebra stalls below the maximum checkpoint height
This also improves the full sync tests,
because the warning is checked before logging a successful sync.
* feat(log): warn when sync stalls downloading the genesis block
* fix(test): warn when the state hasn't committed a block for a long time
This also improves the full sync tests,
because the warning is checked before logging a successful sync.
* doc(test): update some sync acceptance test comments
* fix(log): use Display formatting to log chrono::Duration
Debug formatting is complicated and hard to read.
* fix(log): stop saying that we've activated the mempool without checking it
We're not checking if the mempool is active, so we can't say that.
* fix(log): minor tidying and TODOs
* fix(doc): fix a typo in the tests
* fix(log): explain the post-checkpoint blocks in progress warning calculations
* fix(doc): explain what could happen if we don't wait for extra blocks
* fix(log): add a percent symbol to a percent log
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Allow forcing colored output in `zebrad`
Add a configuration item that allows forcing Zebra to output in color
mode even if the output device is not a terminal.
* Allow enabling colored output from Zebra in tests
Force Zebrad instances to use colored output if the
`ZEBRA_FORCE_USE_COLOR` environment variable is set.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Create a `MempoolBehavior` helper type
Prepare to replace the `Option<Height>` with a custom type that is more
flexible and provides more meaning when used.
* Use `MempoolBehavior` instead of `Option<Height>`
Clarify what the argument is, and prepare the code so that a new variant
can be added to the `MempoolBehavior` later.
* Allow the mempool to be automatically activated
Add a new variant to `MempoolBehavior` that indicates that the mempool
should become active during the test without needing a forced activation
height.
* Test full synchronization
Run `zebrad` and wait until full synchronization completes. Check if the
mempool is automatically activated when it reaches the tip.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Previously, the test would wait 10 seconds for the process to launch.
Now it waits until the process has used the conflicting resource.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Move standard lints into .cargo/config.toml
* Ignore "wrong self convention" in a futures-based trait
This lint might only trigger on beta or nightly at the moment.
* Warn if future incompatibile code is added to Zebra
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Use `testdir()` instead of `TempDir::new()`
Reduce repeated code and make it easier to change from using `tempdir`
to use `tempfile` instead.
* Replace `tempdir` with `tempfile` in `zebrad`
Use `tempfile`'s `TempDir` instead.
* Use `tempdir()` instead of `TempDir::new()`
Reduce repeated code and make it easier to upgrade to `tempfile`.
* Use `tempfile` instead of `tempdir`
Replace obsoleted `tempdir` dependency with `tempfile`.
* Use `tempfile` instead of `tempdir`
Replace obsoleted `tempdir` dependency with `tempfile`.
* Update `Cargo.lock`
Update it now that `tempdir` has been replaced with `tempfile`.
* Remove `tempdir` from `deny.toml` exceptions
Ban duplicate versions of the `tempdir` dependency.
* Remove `inferno` from `deny.toml` exceptions
It apparently isn't needed anymore.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Implement graceful shutdown for the peer set
* Use the minimum lookahead limit in acceptance tests
* Enable a doctest that compiles with newly public modules
* Update `tower` to version `0.4.9`
Update to latest version to add support for Tokio version 1.
* Replace usage of `ServiceExt::ready_and`
It was deprecated in favor of `ServiceExt::ready`.
* Update Tokio dependency to version `1.13.0`
This will break the build because the code isn't ready for the update,
but future commits will fix the issues.
* Replace import of `tokio::stream::StreamExt`
Use `futures::stream::StreamExt` instead, because newer versions of
Tokio don't have the `stream` feature.
* Use `IntervalStream` in `zebra-network`
In newer versions of Tokio `Interval` doesn't implement `Stream`, so the
wrapper types from `tokio-stream` have to be used instead.
* Use `IntervalStream` in `inventory_registry`
In newer versions of Tokio the `Interval` type doesn't implement
`Stream`, so `tokio_stream::wrappers::IntervalStream` has to be used
instead.
* Use `BroadcastStream` in `inventory_registry`
In newer versions of Tokio `broadcast::Receiver` doesn't implement
`Stream`, so `tokio_stream::wrappers::BroadcastStream` instead. This
also requires changing the error type that is used.
* Handle `Semaphore::acquire` error in `tower-batch`
Newer versions of Tokio can return an error if the semaphore is closed.
This shouldn't happen in `tower-batch` because the semaphore is never
closed.
* Handle `Semaphore::acquire` error in `zebrad` test
On newer versions of Tokio `Semaphore::acquire` can return an error if
the semaphore is closed. This shouldn't happen in the test because the
semaphore is never closed.
* Update some `zebra-network` dependencies
Use versions compatible with Tokio version 1.
* Upgrade Hyper to version 0.14
Use a version that supports Tokio version 1.
* Update `metrics` dependency to version 0.17
And also update the `metrics-exporter-prometheus` to version 0.6.1.
These updates are to make sure Tokio 1 is supported.
* Use `f64` as the histogram data type
`u64` isn't supported as the histogram data type in newer versions of
`metrics`.
* Update the initialization of the metrics component
Make it compatible with the new version of `metrics`.
* Simplify build version counter
Remove all constants and use the new `metrics::incement_counter!` macro.
* Change metrics output line to match on
The snapshot string isn't included in the newer version of
`metrics-exporter-prometheus`.
* Update `sentry` to version 0.23.0
Use a version compatible with Tokio version 1.
* Remove usage of `TracingIntegration`
This seems to not be available from `sentry-tracing` anymore, so it
needs to be replaced.
* Add sentry layer to tracing initialization
This seems like the replacement for `TracingIntegration`.
* Remove unnecessary conversion
Suggested by a Clippy lint.
* Update Cargo lock file
Apply all of the updates to dependencies.
* Ban duplicate tokio dependencies
Also ban git sources for tokio dependencies.
* Stop allowing sentry-tracing git repository in `deny.toml`
* Allow remaining duplicates after the tokio upgrade
* Use C: drive for CI build output on Windows
GitHub Actions uses a Windows image with two disk drives, and the
default D: drive is smaller than the C: drive. Zebra currently uses a
lot of space to build, so it has to use the C: drive to avoid CI build
failures because of insufficient space.
Co-authored-by: teor <teor@riseup.net>
This matches the settings for `sync_large_checkpoints_mainnet`.
Also reduce the number of blocks synced to reduce network load.
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Increase the restart test timeout to 10 seconds
It shouldn't take this long.
But maybe the CI VMs are under a lot of load?
* Add extensive logging to diagnose CI state reload failures
* add some additional checks to the acceptance mempool test
* add an additional mempool test
* do proposed fixes to `sync_until`
* Ignore "can't kill an exited process" errors
Co-authored-by: teor <teor@riseup.net>
* Update some comments
* Add a mempool debug_enable_at_height config
* Rename a field in the mempool crawler
* Propagate syncer channel errors through the crawler
We don't want to ignore these errors, because they might indicate a shutdown.
(Or a bug that we should fix.)
* Use debug_enable_at_height in the mempool crawler
* Log when the mempool is activated or deactivated
* Deny unknown fields and apply defaults for all configs
* Move Duration last, as required for TOML tables
* Add a basic mempool acceptance test
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Use the block verifier and non-finalized state in the cached state tests
This substantially increases test coverage.
Previously, the cached state tests were configured with
`checkpoint_sync = true`, which only uses the checkpoint
verifier and the finalized state.
* Log the source of blocks in commit_finalized_direct
This lets us check that we're actually testing the non-finalized state
and block verifier in the cached state tests.
It also improves diagnostics for state errors.
* Fail cached state tests if they're using incorrect heights or configs
This makes sure that the cached state tests actually test the transition
from checkpoint to block verification, and the non-finalized state.
* add legacy chain check and tests
* improve has_network_upgrade check
* add docs to legacy_chain_check()
* change arbitrary module structure
* change the panic message
* move legacy chain acceptance into existing tests
* use a reduced_branch_id_strategy()
* add docs to strategy function
* add argument to check for legacy chain into sync_until()
* Disable IPv6 tests when $ZEBRA_SKIP_IPV6_TESTS is set
This allows users to disable IPv6 tests in environments where IPv6 is not
configured.
* Add network test env var constants
* Replace env strings with constants
fastmod '"ZEBRA_SKIP_NETWORK_TESTS"' zebra_test::net::ZEBRA_SKIP_NETWORK_TESTS
fastmod '"ZEBRA_SKIP_IPV6_TESTS"' zebra_test::net::ZEBRA_SKIP_IPV6_TESTS
* Add functions to skip network tests
* Replace test network env var checks with test function
fastmod --fixed-strings 'env::var_os(zebra_test::net::ZEBRA_SKIP_NETWORK_TESTS).is_some()' 'zebra_test::net::zebra_skip_network_tests()'
fastmod --fixed-strings 'env::var_os(zebra_test::net::ZEBRA_SKIP_IPV6_TESTS).is_some()' 'zebra_test::net::zebra_skip_ipv6_tests()'
* Remove redundant logging and use statements
* Gossip dynamically allocated listener ports to peers
Previously, Zebra would either gossip port `0`, which is invalid, or skip
gossiping its own dynamically allocated listener port.
* Improve "no configured peers" warning
And downgrade from error to warning, because inbound-only nodes are a
valid use case.
* Move random_known_port to zebra-test
* Add tests for dynamic local listener ports and the AddressBook
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
- Add a custom semver match for `zebrad` versions
- Prefer "line contains string" matches, so tests ignore minor changes
- Escape regex meta-characters when a literal string match is intended
- Rename test functions so they are more precise
- Rewrite match internals to remove duplicate code and enable custom matches
- Document match functions
* Standardise lints across Zebra crates, and add missing docs
The only remaining module with missing docs is `zebra_test::command`
* Todo -> TODO
* Clarify what a transcript ErrorChecker does
Also change `Error` -> `BoxError`
* TransError -> ExpectedTranscriptError
* Output Descriptions -> Output descriptions
Janito Vaqueiro Ferreira Filho
teor
Alfredo Garcia
Dimitris Apostolou
Conrado P. L. Gouvea
Deirdre Connolly