* fix(security): Randomly drop connections when inbound service is overloaded
* Uses progressively higher drop probabilities
* Replaces Error::Overloaded with Fatal when internal services shutdown
* Applies suggestions from code review.
* Quickens initial drop probability decay and updates comment
* Applies suggestions from code review.
* Fixes drop connection probablity calc
* Update connection state metrics for different overload/error outcomes
* Split overload handler into separate methods
* Add unit test for drop probability function properties
* Add respond_error methods to zebra-test to help with type resolution
* Initial test that Overloaded errors cause some continues and some closes
* Tune the number of test runs and test timing
* Fix doctests and replace some confusing example requests
---------
Co-authored-by: arya2 <aryasolhi@gmail.com>
* Ignore out of order Address Book changes, and restructure the function
* Handle concurrent changes using the connection state machine order
* Handle out of order changes correctly
* Pass times through the call stack so they are consistent in tests
* Add time arguments to tests
* Fix tests that were broken by the address order checks
* fastmod wall_ local_ zebra*
* cargo fmt --all
* Fix a bug in the concurrent change check
* Test all the new apply and skip checks for address changes
* Document more edge cases and increase the concurrency time slightly
* Simplify enum ordering matches
* Fix comment typos
Co-authored-by: Arya <aryasolhi@gmail.com>
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
* ZIPs were updated to remove ambiguity, this was tracked in #1267.
* #2105 was fixed by #3039 and #2379 was closed by #3069
* #2230 was a duplicate of #2231 which was closed by #2511
* #3235 was obsoleted by #2156 which was fixed by #3505
* #1850 was fixed by #2944, #1851 was fixed by #2961 and #2902 was fixed by #2969
* We migrated to Rust 2021 edition in Jan 2022 with #3332
* #1631 was closed as not needed
* #338 was fixed by #3040 and #1162 was fixed by #3067
* #2079 was fixed by #2445
* #4794 was fixed by #6122
* #1678 stopped being an issue
* #3151 was fixed by #3934
* #3204 was closed as not needed
* #1213 was fixed by #4586
* #1774 was closed as not needed
* #4633 was closed as not needed
* Clarify behaviour of difficulty spacing
Co-authored-by: teor <teor@riseup.net>
* Update comment to reflect implemented behaviour
Co-authored-by: teor <teor@riseup.net>
* Update comment to reflect implemented behaviour when retrying block downloads
Co-authored-by: teor <teor@riseup.net>
* Update `TODO` to remove closed issue and clarify when we might want to fix
Co-authored-by: teor <teor@riseup.net>
* Update `TODO` to remove closed issue and clarify what we might want to change in future
Co-authored-by: teor <teor@riseup.net>
* Clarify benefits of how we do block verification
Co-authored-by: teor <teor@riseup.net>
* Fix rustfmt errors
---------
Co-authored-by: teor <teor@riseup.net>
* Update MAX_TX_INV_IN_MESSAGE for ZIP-239 WTX IDs
* Combine multiple transaction updates into a single gossip & rate-limit gossips
* Rate-limit block gossips
* Fix mempool_transaction_expiration gossip test timings
* Enforce MAX_TX_INV_IN_MESSAGE in the network layer, rather than each service
* Fix documentation for `Message::Tx`
* Split MAX_INV_IN_RECEIVED_MESSAGE and MAX_TX_INV_IN_SENT_MESSAGE
* Fix log message typo
* Move some docs to/from another PR
---------
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Add a PeerSocketAddr type which hides its IP address, but shows the port
* Manually replace SocketAddr with PeerSocketAddr where needed
```sh
fastmod SocketAddr PeerSocketAddr zebra-network
```
* Add missing imports
* Make converting into PeerSocketAddr easier
* Fix some unused imports
* Add a canonical_peer_addr() function
* Fix connection handling for PeerSocketAddr
* Fix serialization for PeerSocketAddr
* Fix tests for PeerSocketAddr
* Remove some unused imports
* Fix address book listener handling
* Remove redundant imports and conversions
* Update outdated IPv4-mapped IPv6 address code
* Make addresses canonical when deserializing
* Stop logging peer addresses in RPC code
* Update zebrad tests with new PeerSocketAddr type
* Update zebra-rpc tests with new PeerSocketAddr type
---------
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* add user agent as argument, use git to auto build zebra user agent
* try to fix test
* fix typo
* change expect text
* remove newline
* fix some docs
Co-authored-by: Marek <mail@marek.onl>
---------
Co-authored-by: Marek <mail@marek.onl>
* refuse to run Zebra if it is too old
* update the release checklist to consider the constants
* bring newline back
* apply new end of support code
* attempt to add tests (not working yet)
* move eos to progress task
* move tests
* add acceptance test (not working)
* fix tests
* change to block height checks (ugly code)
* change warn days
* refactor estimated blocks per day, etc
* move end of support code to its own task
* change test
* fix some docs
* move constants
* remove uneeded conversions
* downgrade tracing
* reduce end of support time, fix ci changing debugs to info again
* update instructions
* add failure messages
* cargo lock update
* unify releaase name constant
* change info msg
* clippy fixes
* add a block explorer
* ignore testnet in end of support task
* change panic to 16 weeks
* add some documentation about end of support
* Tweak docs wording
---------
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Reject nodes using the ZClassic default ports
* Always check regtest and other coin ports, even if no network is supplied
* Warn if Zebra is configured with ports from other coins
* Allow unspecified addresses and ports for inbound listeners
---------
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Close the new connection if Zebra unexpectedly generates a duplicate random nonce
* Add a missing test module comment
* Avoid peer attacks that replay self-connection nonces to manipulate the nonce set (needs tests)
* Add a test that makes sure network self-connections fail
* Log an info level when self-connections fail (this should be rare)
* Just use plain blocks for mutex critical sections
* Add a missing space
* Silence an extremely verbose error in zebra-consensus tests
This disables around 10,000 logs like:
2023-04-18T02:46:28.441662Z WARN init{config=Config { checkpoint_sync: true, debug_skip_parameter_preload: false } network=Mainnet debug_skip_parameter_preload=true}: unexpected error: Closed in state request while verifying previous state checkpoints
* Increase the outbound connection interval to 100ms
* Start the inbound service as soon as possible, and the syncer last
* Increase acceptance test time limits to get more debug info
* Add more debug info to inbound service overload tracing messages
* Use a stricter connection rate limit for successful inbound peer connections
* Limit the number of nonces in the self-connection nonce set
* Rate-limit failed inbound connections as well
* Justify the sleep and the yield_now
* Use the configured connection limit rather than a constant
* Tests that the number of nonces is limited (#37)
* Tests that the number of nonces is limited
* removes unused constant
* test that it reaches the nonce limit
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
* Implement Display and to_string() for NetworkUpgrade
* Add a progress-bar feature to zebrad
* Add the progress bar writer to the tracing component
* Add a block progress bar transmitter
* Correctly shut down the progress bar, and shut it down on an interrupt
* Make it clearer that the progress task never exits
* Add a config for writing logs to a file
* Add a progress-bar feature to zebra-network
* Add a progress bar for the address book size
* Add progress bars for never attempted and failed peers
* Add an optional limit and label to connection counters
* Add open connection progress bars
* Improve CheckpointList API and CheckpointVerifier debugging
* Add checkpoint index and checkpoint queue progress bars
* Security: Limit the number of non-finalized chains tracked by Zebra
* Make some NonFinalizedState methods available with proptest-impl
* Add a non-finalized chain count progress bar
* Track the last fork height for newly forked chains
* Add a should_count_metrics to Chain
* Add a display method for PartialCumulativeWork
* Add a progress bar for each chain fork
* Add a NonFinalizedState::disable_metrics() method and switch to using it
* Move metrics out of Chain because we can't update Arc<Chain>
* Fix: consistently use best chain order when searching chains
* Track Chain progress bars in NonFinalizedState
* Display work as bits, not a multiple of the target difficulty
* Handle negative fork lengths by reporting "No fork"
* Correctly disable unused fork bars
* clippy: rewrite using `match _.cmp(_) { ... }`
* Initial mempool progress bar implementation
* Update Cargo.lock
* Add the actual transaction size as a description to the cost bar
* Only show mempool progress bars after first activation
* Add queued and rejected mempool progress bars
* Clarify cost note is actual size
* Add tracing.log_file config and progress-bar feature to zebrad docs
* Derive Clone for Chain
* Upgrade to howudoin 0.1.2 and remove some bug workarounds
* Directly call the debug formatter to Display a Network
Co-authored-by: Arya <aryasolhi@gmail.com>
* Rename the address count metric to num_addresses
Co-authored-by: Arya <aryasolhi@gmail.com>
* Simplify reverse checkpoint lookup
Co-authored-by: Arya <aryasolhi@gmail.com>
* Simplify progress bar shutdown code
Co-authored-by: Arya <aryasolhi@gmail.com>
* Remove unused MIN_TRANSPARENT_TX_MEMPOOL_SIZE
* Document that the progress task runs forever
* Fix progress log formatting
* If progress-bar is on, log to a file by default
* Create missing directories for log files
* Add file security docs for running Zebra with elevated permissions
* Document automatic log file, spell progress-bar correctly
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
* Limit version user agents to 256 bytes, rather than 2MB, needs failure tests
* Limit all inv messages to 50,000 entries, existing tests cover this
* Limit reject message strings based on network protocol, needs success and failure tests
* Catch up as fast as possible if inventory rotation is delayed, existing tests cover this
* Truncate inv channel hashes to 1000, needs success and failure tests
* Limit inv registry size to 4 MB, needs over-limit tests for inv and addr
* Test size constraints on version user agent, reject command, and reject reason (#13)
* Test inventory registry memory limits for hashes and peers (#14)
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
---------
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Arya <aryasolhi@gmail.com>
* Combine transaction IDs that arrive close together rather than overwriting them
* Reduce the size of transaction ID gossip logs
* Limit the maximum number of times we wait for new changes before sending
* Make logs even shorter
* Expand correctness comment
* Remove trailing space
---------
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* bump v1.0.0-rc.4
* Add new mainnet and testnet checkpoints generated using zcashd
* make changes to changelog from review
* move item to new section
* change section name
---------
Co-authored-by: teor <teor@riseup.net>
* Derive default using #[default]
* Implement PartialEq manually to satisfy clippy
* Allow a manual derive in test-only code
* Fix some missing docs warnings in the Docker build
* adds ValidateBlock request to state
* adds `Request` enum in block verifier
skips solution check for BlockProposal requests
calls CheckBlockValidity instead of Commit block for BlockProposal requests
* uses new Request in references to chain verifier
* adds getblocktemplate proposal mode response type
* makes getblocktemplate-rpcs feature in zebra-consensus select getblocktemplate-rpcs in zebra-state
* Adds PR review revisions
* adds info log in CheckBlockProposalValidity
* Reverts replacement of match statement
* adds `GetBlockTemplate::capabilities` fn
* conditions calling checkpoint verifier on !request.is_proposal
* updates references to validate_and_commit_non_finalized
* adds snapshot test, updates test vectors
* adds `should_count_metrics` to NonFinalizedState
* Returns an error from chain verifier for block proposal requests below checkpoint height
adds feature flags
* adds "proposal" to GET_BLOCK_TEMPLATE_CAPABILITIES_FIELD
* adds back block::Request to zebra-consensus lib
* updates snapshots
* Removes unnecessary network arg
* skips req in tracing intstrument for read state
* Moves out block proposal validation to its own fn
* corrects `difficulty_threshold_is_valid` docs
adds/fixes some comments, adds TODOs
general cleanup from a self-review.
* Update zebra-state/src/service.rs
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Update zebra-rpc/src/methods/get_block_template_rpcs.rs
Co-authored-by: teor <teor@riseup.net>
* check best chain tip
* Update zebra-state/src/service.rs
Co-authored-by: teor <teor@riseup.net>
* Applies cleanup suggestions from code review
* updates gbt acceptance test to make a block proposal
* fixes json parsing mistake
* adds retries
* returns reject reason if there are no retries left
* moves result deserialization to RPCRequestClient method, adds docs, moves jsonrpc_core to dev-dependencies
* moves sleep(EXPECTED_TX_TIME) out of loop
* updates/adds info logs in retry loop
* Revert "moves sleep(EXPECTED_TX_TIME) out of loop"
This reverts commit f7f0926f4050519687a79afc16656c3f345c004b.
* adds `allow(dead_code)`
* tests with curtime, mintime, & maxtime
* Fixes doc comment
* Logs error responses from chain_verifier CheckProposal requests
* Removes retry loop, adds num_txs log
* removes verbose info log
* sorts mempool_txs before generating merkle root
* Make imports conditional on a feature
* Disable new CI tests until bugs are fixed
* adds support for getpeerinfo RPC
* adds vector test
* Always passes address_book to RpcServer
* Update zebra-network/src/address_book.rs
Co-authored-by: teor <teor@riseup.net>
* Renames PeerObserver to AddressBookPeers
* adds getpeerinfo acceptance test
* Asserts that addresses parse into SocketAddr
* adds launches_lightwalletd field to TestType::LaunchWithEmptyState and uses it from the get_peer_info test
* renames peer_observer mod
* uses SocketAddr as `addr` field type in PeerInfo
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Changelog with trivial entries
* Delete trivial entries
* Summarise known issues in README, but don't change the list yet
* Add block timeouts to known issues
* Update the release template to add missing version files
* Bump crate versions
* Add the required Rust version to the release checklist
* Update the Rust version requirement to 1.65, Zebra now uses `let ... else ...`
* Update checkpoints
* Add checkpoints to the CHANGELOG
* Breaking Rust compiler version change
* Clarify the latest stable supported rust version
* sets relay = true when there's no relay byte
* adds test for missing relay field
moves codec tests to their own module
* simplifies version_message_omitted_relay test
* adds `version_message_with_relay` test
adds doc comment to `read_version`
renames codec/tests.rs to codec/tests/vectors.rs
* updates doc comment
* Remove verbose continuous_blockchain test logs
* Downgrade verbose zebra-network logs to debug
* Downgrade some state logs to debug during tests
* Mark were we would add always-on log filters, if we needed to
* Reduce the number of mempool property tests, to reduce logging
teor
dependabot[bot]
Pili Guerra
Alfredo Garcia
Arya
Deirdre Connolly
Marek