* Patch equihash to use the solver branch
* Add an internal-miner feature and set up its dependencies
* Remove 'Experimental' from mining RPC docs
* Fix a nightly clippy::question_mark lint
* Move a byte array utility function to zebra-chain
* fixup! Add an internal-miner feature and set up its dependencies
* Add an equihash::Solution::solve() method with difficulty checks
* Check solution is valid before returning it
* Add a TODO to check for peers before mining
* Move config validation into GetBlockTemplateRpcImpl::new()
* fixup! fixup! Add an internal-miner feature and set up its dependencies
* Use the same generic constraints for GetBlockTemplateRpcImpl struct and impls
* Start adding an internal miner component
* Add the miner task to the start command
* Add basic miner code
* Split out a method to mine one block
* Spawn to a blocking thread
* Wait until a valid template is available
* Handle shutdown
* Run mining on low priority threads
* Ignore some invalid solutions
* Use a difference nonce for each solver thread
* Update TODOs
* Change the patch into a renamed dependency to simplify crate releases
* Clean up instrumentation and TODOs
* Make RPC instances cloneable and clean up generics
* Make LongPollId Copy so it's easier to use
* Add API to restart mining if there's a new block template
* Actually restart mining if there's a new block template
* Tidy instrumentation
* fixup! Move config validation into GetBlockTemplateRpcImpl::new()
* fixup! Make RPC instances cloneable and clean up generics
* Run the template generator and one miner concurrently
* Reduce logging
* Fix a bug in getblocktemplate RPC tip change detection
* Work around some watch channel change bugs
* Rate-limit template changes in the receiver
* Run one mining solver per available core
* Use updated C code with double-free protection
* Update to the latest solver branch
* Return and submit all valid solutions
* Document what INPUT_LENGTH means
* Fix watch channel change detection
* Don't return early when a mining task fails
* Spawn async miner tasks to avoid cooperative blocking, deadlocks, and improve shutdown responsiveness
* Make existing parallelism docs and configs consistent
* Add a mining parallelism config
* Use the minimum of the configured or available threads for mining
* Ignore optional feature fields in tests
* Downgrade some frequent logs to debug
* Document new zebrad features and tasks
* Describe the internal-miner feature in the CHANGELOG
* Update dependency to de-duplicate equihash solutions
* Use futures::StreamExt instead of TryStreamExt
* Fix a panic message typo
* Provide access to DiskDb and DiskWriteBatch outside the state using a feature
* Actually let's export ZebraDb for the format upgrade code
* Pass column families to ZebraDb as an argument
* Allow the database kind to be changed in config.rs
* Use the state kind in finalized_state.rs
* Allow different database kinds in ZebraDb, but don't move the upgrade code yet
* Allow different database kinds in DiskDb
* Allow different database kinds in upgrade.rs, but don't split the upgrade code out yet
* Add new arguments to raw database tests
* Fix doc links
* Fix internal imports
* Fix unused code
* Update zebrad version metadata
* Create a specific state database delete function
* Fix state exports
* Fix zebrad tests
* Fix zebrad state write tests
* Make CI run again
* Fix dead code warnings for test methods
* Remove unnecessary async on some tests
* Fix logging required by tests
* Fix logging required in test itself
* Fix variable names
* Try to copy the message and add regexes
* Skip external branches in ci-unit-tests-docker.yml
* Patch external repositories in ci-unit-tests-docker.patch.yml
* Revert ci-unit-tests-docker.patch.yml
* Add a separate external PR patch in ci-unit-tests-docker.patch-external.yml
* Document ci-unit-tests-docker.patch.yml
* Document ci-unit-tests-docker.yml
* Skip dependabot config validation for external PRs
* Trivial Rust code modification to make Rust workflows run
* Use correct method for checking source branch refs
* Fix the check condition
* Simplify job structure
* Use correct skip condition
* Make the start docs visible so we can link to them
* Skip integration tests on external PRs
* Allow external PRs to pass CD Docker tests
* Allow external PRs to pass Docs build and deploy
* Add a progress bar config that is disabled unless the feature is on
* Simplify the default config
* Enable the progress bar feature by default, but require the config
* Rename progress bars config to avoid merge conflicts
* Use a log file when the progress bar is activated
* Document how to configure progress bars
* Handle log files in config_tests and check config path
* Fix doc link
* Fix path check
* Fix config log matching
* Fix clippy warning
* Add tracing to config tests
* It's zebrad not zebra
* cargo fmt --all
* Update release for config file changes
* Fix config test failures
* Allow printing to stdout in a method
* Move format upgrades to their own module and enum
* Launch a format change thread if needed, and shut it down during shutdown
* Add some TODOs and remove a redundant timer
* Regularly check for panics in the state upgrade task
* Only run example upgrade once, change version field names
* Increment database format to 25.0.2: add format change task
* Log the running and initial disk database format versions on startup
* Add initial disk and running state versions to cached state images in CI
* Fix missing imports
* Fix typo in logs workflow command
* Add a force_save_to_disk argument to the CI workflow
* Move use_internet_connection into zebrad_config()
* fastmod can_spawn_zebrad_for_rpc can_spawn_zebrad_for_test_type zebra*
* Add a spawn_zebrad_without_rpc() function
* Remove unused copy_state() test code
* Assert that upgrades and downgrades happen with the correct versions
* Add a kill_and_return_output() method for tests
* Add a test for new_state_format() versions (no upgrades or downgrades)
* Add use_internet_connection to can_spawn_zebrad_for_test_type()
* Fix workflow parameter passing
* Check that reopening a new database doesn't upgrade (or downgrade) the format
* Allow ephemeral to be set to false even if we don't have a cached state
* Add a test type that will accept any kind of state
* When re-using a directory, configure the state test config with that path
* Actually mark newly created databases with their format versions
* Wait for the state to be opened before testing the format
* Run state format tests on mainnet and testnet configs (no network access)
* run multiple reopens in tests
* Test upgrades run correctly
* Test that version downgrades work as expected (best effort)
* Add a TODO for testing partial updates
* Fix missing test arguments
* clippy if chain
* Fix typo
* another typo
* Pass a database instance to the format upgrade task
* Fix a timing issue in the tests
* Fix version matching in CI
* Use correct env var reference
* Use correct github env file
* Wait for the database to be written before killing Zebra
* Use correct workflow syntax
* Version changes aren't always upgrades
---------
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* print a Zebra logo and some text in progress bar mode
* add network to printed line, add heart to logo
* print logo and message regardless of progress-bar; document how logo was generated
* De-duplicate app_version and user_agent code, rename to build_version
* Make RPC testnet flag forward-compatible with additional testnets
* Fix RPC tests with new argument
* Use "modified" rather than "dirty" for uncommitted changes in build metadata
* Split the vergen version into its own function
* updates Cargo.toml
* Migrate to abscissa 0.7.0
* Avoid panic from calling color_eyre::install twice
* Uses 'start' as the default subcommand
* updates default cmd logic
* Fixes minor cli issues
* removes outdated check in acceptance test
* Adds a test for process_cli_args, fixes version_args test.
Adds -V to process_cli_args match case
* Revert "fix(clippy): Silence future-incompat warnings until we upgrade Abscissa (#6024)"
This reverts commit dd90f79b48.
* Drops the worker guard to flush logs when zebra shuts down
* Adds cargo feature to clap
* restores process_cli_args
* updates deny.toml
* Updates EntryPoint help template
* Updates subcommand help msgs
* removes trailing whitespace, capitalizes sentences
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* revert parts of revert "Revert fix(clippy): Silence future-incompat warnings until we upgrade Abscissa"
* Applies suggestions from code review
* Moves EntryPoint to its own module
* fixes version_args test
* Updates changelog
* Prunes redundant test cases
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Revert "Prunes redundant test cases"
This reverts commit 3f7397918489144805c17d0594775aa699e87b9d.
* Update zebrad/src/commands/entry_point.rs
Co-authored-by: teor <teor@riseup.net>
* Add missing import
* Updates `process_cli_args` to return a result
---------
Co-authored-by: teor <teor@riseup.net>
* Add extra vergen metadata to startup and panic reports
* Upgrade to vergen 8.1.3
* Update build script docs
* Upgrade env var names for vergen 8.1.3
* Switch to `git2` dependency feature
* Use renamed environmental variable for `git describe`
* Ignore vergen errors if `.git` doesn't exist
* Update Cargo.lock
* Remove unused deny.toml exceptions with --all-features
Some of these might be from this PR, or the missed dependency updates.
* add user agent as argument, use git to auto build zebra user agent
* try to fix test
* fix typo
* change expect text
* remove newline
* fix some docs
Co-authored-by: Marek <mail@marek.onl>
---------
Co-authored-by: Marek <mail@marek.onl>
* refuse to run Zebra if it is too old
* update the release checklist to consider the constants
* bring newline back
* apply new end of support code
* attempt to add tests (not working yet)
* move eos to progress task
* move tests
* add acceptance test (not working)
* fix tests
* change to block height checks (ugly code)
* change warn days
* refactor estimated blocks per day, etc
* move end of support code to its own task
* change test
* fix some docs
* move constants
* remove uneeded conversions
* downgrade tracing
* reduce end of support time, fix ci changing debugs to info again
* update instructions
* add failure messages
* cargo lock update
* unify releaase name constant
* change info msg
* clippy fixes
* add a block explorer
* ignore testnet in end of support task
* change panic to 16 weeks
* add some documentation about end of support
* Tweak docs wording
---------
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Log zebrad config path as well as config values
* Remove duplicate config log
* Only log config path and data for server commands
* Use config ref instead
* print usage info for --help flag
* update known issues and help command's help text
* Update README.md
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* updates README
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* adds non-blocking writer for tracing subscriber
* use non_blocking writer for the fmt::Layer with the tokio-console feature as well
* adds doc comment to _guard field
* adds acceptance test
* update filter_handle type to use NonBlocking
* adds more detail on lossy non-blocking writer and sets tracing.filter to "trace" in acceptance test
* drops ZebradApp before process::exit(1) in the event of a FrameworkError
* reduces buffered lines limit to 8000
* adds tracing.buffer_limit config and some comments
* update acceptance.rs
* fix acceptance test
* fixes ambigious phrasing in comment
* updates zebrad/src/application.rs
* Find out what the join error is in the GitHub runner tests
* updates acceptance test to use recv_timeout instead of always waiting 10 seconds, removes unnecessary echo command, and reduces # of rpc requests to 500
* see if sleeping for a few seconds before exiting helps the macOS test pass
* Expand exit sleep docs
Co-authored-by: Arya <aryasolhi@gmail.com>
Co-authored-by: teor <teor@riseup.net>
* adds start as default subcommand for zebrad
* moves EntryPoint to submodule and adds a test
* moves all start tests to config_test to avoid listener conflicts
* Update zebrad/src/application/entry_point.rs docs
* Revert "moves all start tests to config_test to avoid listener conflicts"
This reverts commit 61ce46f5a13907facc3a11326e7a328d81b2be3d.
* Update based on test API changes from another PR
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Initialize the rayon threadpool with a new config for CPU-bound threads
* Verify proofs and signatures on the rayon thread pool
* Only spawn one concurrent batch per verifier, for now
* Allow tower-batch to queue multiple batches
* Fix up a potentially incorrect comment
* Rename some variables for concurrent batches
* Spawn multiple batches concurrently, without any limits
* Simplify batch worker loop using OptionFuture
* Clear pending batches once they finish
* Stop accepting new items when we're at the concurrent batch limit
* Fail queued requests on drop
* Move pending_items and the batch timer into the worker struct
* Add worker fields to batch trace logs
* Run docker tests on PR series
* During full verification, process 20 blocks concurrently
* Remove an outdated comment about yielding to other tasks
* Warn the user when Zebra cannot parse `zebrad.toml`
* Test that Zebra warns the user when it cannot parse `zebrad.toml`
* Fix up a mistaken merge change
* Suggest how to fix `zebrad.toml` when Zebra cannot parse it
Co-authored-by: teor <teor@riseup.net>
* Disable the flamegraph feature by default at compile time
* Disable the journald feature by default at compile time
* Also disable inferno dependency, and rearrange features
* Disable the prometheus feature by default at compile time
* Disable the tracing filter reload feature by default at compile time
* Disable tests when corresponding features are disabled
* Add compile-time tracing features to user docs
* Add compile-time features to the metrics user docs
* Document diagnostics as part of the start command tasks and services
* breaking(diagnostics): rename "enable-sentry" feature to "sentry" (#4623)
* Also skip conflict tests when those ports are disabled
* breaking(diagnostics): rename "enable-sentry" feature to "sentry"
This is mostly:
```sh
fastmod enable-sentry sentry
```
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Fix the syntax of links in comments
* Fix a mistake in the docs
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Remove unnecessary angle brackets from a link
* Revert the changes for links that serve as references
* Revert "Revert the changes for links that serve as references"
This reverts commit 8b091aa9fab453e7d3559a5d474e0879183b9bfb.
* Remove `<` `>` from links that serve as references
This reverts commit 046ef25620ae1a2140760ae7ea379deecb4b583c.
* Don't use `<` `>` in normal comments
* Don't use `<` `>` for normal comments
* Revert changes for comments starting with `//`
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Move `init_tracing` to `lib.rs`
Allow it to be reused by other binaries.
* Fix `hex` dependency inclusion in `zebra-chain`
It would previously fail because `hex` was being included without the
`serde` feature, even though we required that for `transparent::Script`.
* Implement `FromStr` for `Network`
Make it easy to receive it as a command-line parameter.
* Add `zebra-tip-height` utility
Obtains the chain tip height of a directory containing Zebra state.
* Remove Tokio dependency from `zebra-utils`
It wasn't actually used by the `zebra-tip-height` utility.
* Remove `BoxStateService` type alias
It's not needed if we don't return the unused state service. This also
allows removing the `tower` dependency.
* Remove unnecessary attribute
Leftover from copied code.
* Make `cache_dir` argument optional
Fallback to the default Zebra state cache directory.
* Remove added newline
Minor formatting fix, to avoid adding an unnecessary newline.
* Move `tip-height` command into `zebrad`
Make it available as a sub-command in `zebrad`.
* Make some zebrad sub-commands only log warnings
Co-authored-by: teor <teor@riseup.net>
* Stop useless crawler attempts when there are no peers and no crawl responses
* Disable GitHub bug report URLs when the disk is full
* Add help text for the `zebrad start` tracing filter option
* Update `tower` to version `0.4.9`
Update to latest version to add support for Tokio version 1.
* Replace usage of `ServiceExt::ready_and`
It was deprecated in favor of `ServiceExt::ready`.
* Update Tokio dependency to version `1.13.0`
This will break the build because the code isn't ready for the update,
but future commits will fix the issues.
* Replace import of `tokio::stream::StreamExt`
Use `futures::stream::StreamExt` instead, because newer versions of
Tokio don't have the `stream` feature.
* Use `IntervalStream` in `zebra-network`
In newer versions of Tokio `Interval` doesn't implement `Stream`, so the
wrapper types from `tokio-stream` have to be used instead.
* Use `IntervalStream` in `inventory_registry`
In newer versions of Tokio the `Interval` type doesn't implement
`Stream`, so `tokio_stream::wrappers::IntervalStream` has to be used
instead.
* Use `BroadcastStream` in `inventory_registry`
In newer versions of Tokio `broadcast::Receiver` doesn't implement
`Stream`, so `tokio_stream::wrappers::BroadcastStream` instead. This
also requires changing the error type that is used.
* Handle `Semaphore::acquire` error in `tower-batch`
Newer versions of Tokio can return an error if the semaphore is closed.
This shouldn't happen in `tower-batch` because the semaphore is never
closed.
* Handle `Semaphore::acquire` error in `zebrad` test
On newer versions of Tokio `Semaphore::acquire` can return an error if
the semaphore is closed. This shouldn't happen in the test because the
semaphore is never closed.
* Update some `zebra-network` dependencies
Use versions compatible with Tokio version 1.
* Upgrade Hyper to version 0.14
Use a version that supports Tokio version 1.
* Update `metrics` dependency to version 0.17
And also update the `metrics-exporter-prometheus` to version 0.6.1.
These updates are to make sure Tokio 1 is supported.
* Use `f64` as the histogram data type
`u64` isn't supported as the histogram data type in newer versions of
`metrics`.
* Update the initialization of the metrics component
Make it compatible with the new version of `metrics`.
* Simplify build version counter
Remove all constants and use the new `metrics::incement_counter!` macro.
* Change metrics output line to match on
The snapshot string isn't included in the newer version of
`metrics-exporter-prometheus`.
* Update `sentry` to version 0.23.0
Use a version compatible with Tokio version 1.
* Remove usage of `TracingIntegration`
This seems to not be available from `sentry-tracing` anymore, so it
needs to be replaced.
* Add sentry layer to tracing initialization
This seems like the replacement for `TracingIntegration`.
* Remove unnecessary conversion
Suggested by a Clippy lint.
* Update Cargo lock file
Apply all of the updates to dependencies.
* Ban duplicate tokio dependencies
Also ban git sources for tokio dependencies.
* Stop allowing sentry-tracing git repository in `deny.toml`
* Allow remaining duplicates after the tokio upgrade
* Use C: drive for CI build output on Windows
GitHub Actions uses a Windows image with two disk drives, and the
default D: drive is smaller than the C: drive. Zebra currently uses a
lot of space to build, so it has to use the C: drive to avoid CI build
failures because of insufficient space.
Co-authored-by: teor <teor@riseup.net>
* Allow deliberate instances of the new nightly clippy::derivable_impls lint
We want our config defaults to be explicit.
Not so sure about the application defaults, but they also contain a config.
* Also allow unknown lint names
Stable doesn't know about this lint, but nightly does.
* Use the git version + new commit count + hash for the app version
This helps diagnose bugs in versions of Zebra built from git branches,
rather than git version tags.
* Fill in assert
* Also log semver string
* Fix syntax
* Handle vergen using the cargo package version or raw git tag
* s/Semver/SemVer/
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
Enable builds where:
* there is no google cloud git commit env var, and
* there is no `.git` directory.
By making all `vergen` env vars optional, and skipping any env vars that
don't exist.
* build(deps): bump vergen from 3.2.0 to 5.1.1
* fix hardcoded version for Tracing struct
* add additional metadata
* remove extra allocations for metadata
* Remove zebrad code version from release checklist
The zebrad code automatically uses the crate version now.
* Sort panic metadata into rough categories
Co-authored-by: teor <teor@riseup.net>
* add hint for port error
* add issue filter for port panic
* add lock file hint
* add metrics endpoint port conflict hint
* add hint for tracing endpoint port conflict
* add acceptance test for resource conflics
* Split out common conflict test code into a function
* Add state, metrics, and tracing conflict tests
* Add a full set of stderr acceptance test functions
This change makes the stdout and stderr acceptance test interfaces
identical.
* move Zcash listener opening
* add todo about hint for disk full
* add constant for lock file
* match path in state cache
* don't match windows cache path
* Use Display for state path logs
Avoids weird escaping on Windows when using Debug
* Add Windows conflict error messages
* Turn PORT_IN_USE_ERROR into a regex
And add another alternative Windows-specific port error
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Jane Lusby <jane@zfnd.org>
* Add the configured network to error reports
* Log the configured network at error level
* Create the global span immediately after activating tracing
And leak the span guard, so the span is always active.
* Include panic metadata in the report and URL
* Use `Main` and `Test` in the global span
`net=Mainnet` is a bit redundant
In our README, we tell users to ignore these errors, so we should also
disable the issue URL.
Also include the hash in the error. (We don't want the span active for
all messages, we just want the hash in the error.)
teor
Conrado Gouvea
Arya
Alfredo Garcia
Marek
Janito Vaqueiro Ferreira Filho
Kirill Fomichev
Deirdre Connolly