* Revert "Remove commented-out code"
This reverts commit 9e69777925f103ee11e5940bba95b896c828839b.
* Implement deserialization for `addrv2` messages
* Limit addr and addrv2 messages to MAX_ADDRS_IN_MESSAGE
* Clarify address version comments
* Minor cleanups and fixes
* Add preallocation tests for AddrV2
* Add serialization tests for AddrV2
* Use prop_assert in AddrV2 proptests
* Use a generic utility method for deserializing IP addresses in `addrv2`
* Document the purpose of a conversion to MetaAddr
* Fix a comment typo, and clarify that comment
* Clarify the unsupported AddrV2 network ID error and enum variant names
```sh
fastmod AddrV2UnimplementedError UnsupportedAddrV2NetworkIdError zebra-network
fastmod Unimplemented Unsupported zebra-network
```
* Fix and clarify unsupported AddrV2 comments
* Replace `panic!` with `unreachable!`
* Clarify a comment about skipping a length check in a test
* Remove a redundant test
* Basic addr (v1) and addrv2 deserialization tests
* Test deserialized IPv4 and IPv6 values in addr messages
* Remove redundant io::Cursor
* Add comments with expected values of address test vectors
* Add a `Duration32::from_days` constructor
Make it simpler to construct a `Duration32` representing a certain
number of days.
* Add `MetaAddr::was_not_recently_seen` method
A helper method to check if a peer was never seen before or if it was
last seen a long time ago. This will be one of the conditions to
consider a peer as unreachable.
* Add `MetaAddr::is_probably_unreachable` method
A helper method to check if a peer should be considered unreachable. It
is considered unreachable if recent connection attempts have failed and
it was not recently seen.
If a peer is considered unreachable, Zebra shouldn't attempt to connect
to it again.
* Do not keep trying to connect to unreachable peer
A peer is probably unreachable if it was last seen a long time ago and
if it's last connection attempt failed.
* Test `was_not_recently_seen`
Redo the calculation on arbitrary `MetaAddr`s.
* Test `is_probably_unreachable`
Redo the calculation on arbitrary `MetaAddr`s.
* Test if probably unreachable peers are ignored
Given an `AddressBook` with a list of arbitrary `MetaAddr`s, check that
none of the peers listed for a reconnection is probably unreachable.
* Rename unit test to improve clarity
Remove the double negative from the name.
Co-authored-by: teor <teor@riseup.net>
* Rename constant to `MAX_RECENT_PEER_AGE`
Make the purpose of the constant clearer.
Co-authored-by: teor <teor@riseup.net>
* Rename method to `last_seen_is_recent`
Remove the double negative from the name.
* Rename method to `is_probably_reachable`
Avoid having to negate the result of the method in security critical
filter.
* Move check into `is_ready_for_connection_attempt`
Make sure the check is used in any place that requires a peer that's
ready for a connection attempt.
* Improve test documention
Describe the goal of the test better.
Co-authored-by: teor <teor@riseup.net>
* Improve `is_probably_reachable` documentation
List the conditions as bullet points.
Co-authored-by: teor <teor@riseup.net>
* Document what happens when peers have no last seen time
Co-authored-by: teor <teor@riseup.net>
* Limit the number of outbound connections in the crawler
* Make zebra-network channel bounds depend on config.peerset_initial_target_size
* Bias Zebra towards outbound connections
And turn connection limits into `Config` methods.
* Downgrade some connection logs to debug
* Remove verbose or outdated fields in tracing logs
* Clarify connection limits
Includes:
- `fastmod OUTBOUND_PEER_BIAS_FRACTION OUTBOUND_PEER_BIAS_DENOMINATOR zebra*`
- clarify connection limit documentation
* Clarify inventory channel capacity
* Add zebra_network::initialize tests with limited numbers of peers
* Avoid cooperative async task starvation in the peer crawler and listener
If we don't yield in these loops, they can run for a long time before
tokio forces them to yield.
* Test the crawler with small connection limits
And use the multi-threaded runtime to avoid long hangs.
* Stop using the multi-threaded executor in tests where it's not needed
* Avoid starvation for every connection
Adds yields after inbound successes and initial peer connections.
* Add a crawler peer connection success test
* Add outbound connection limit tests
* Improve outbound tests
* Increment the crates that have new commits since the last version
* Increment the crates that depend on crates that have changed
* Increment the version of `zebra-script`
* Use the `zebrad` version in the `zebra-network` user agent string
* Use the `v1.0.0-alpha.19` git tag in `README.md`
* Copy the draft changelog into `CHANGELOG.md`
* Delete bumps
* Update CHANGELOG.md
Co-authored-by: teor <teor@riseup.net>
* Add newly merged PRs
Co-authored-by: teor <teor@riseup.net>
* Update versions for zebra v1.0.0-alpha.18 release
* WIP: Initial PR list
* Remove uninteresting version bumps from CHANGELOG
* Categorise and group PRs in CHANGELOG, removing uninteresting PRs
* Further refine and categorise changelog entries
* Fix tag url
* Final changes to CHANGELOG
* Add a changelog description
* Spacing
* Clarify and fix changelog PR descriptions
* Add PRs that are about to be merged
* More slight clarifications
* Spacing
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Check return value of zcash_script_new_precomputed_tx
* Set the NU5 testnet activation height to 1_590_000
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Update Nu5 constants to new values
* Update ZIP-244 test vectors for new branch ID
* Squashed commit of the following:
commit bdb120a249e3e889a913114a712505defdade1d4
Author: Deirdre Connolly <durumcrustulum@gmail.com>
Date: Tue Oct 5 11:54:01 2021 -0400
Use pallas::Base::from_str_vartime() in sinsemilla tests
commit e99fa4925857840fa65ccfb4a076ec412e416576
Author: Deirdre Connolly <durumcrustulum@gmail.com>
Date: Tue Oct 5 11:45:24 2021 -0400
Compiles
commit a5200181146bfd2aa1e09abea2caaa7a7ceb006e
Author: Deirdre Connolly <durumcrustulum@gmail.com>
Date: Tue Oct 5 10:15:17 2021 -0400
Incomplete upgrade of deps
* Squashed commit of the following:
commit 8d1b76ec5626517817c3a4d9f3950acc90a359df
Author: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Date: Tue Oct 5 04:02:26 2021 +0000
Update `zcash_script` to support V5 transactions
Use a newer version of `zcash_script` that has been updated to support
V5 transactions.
commit 371233628ae61e0c25d6ba8f31d9dba42823becb
Author: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Date: Tue Oct 5 03:06:20 2021 +0000
Update Zcash dependencies
Update some Zcash crates:
- `halo2`
- `incrementalmerkletree' (patch version)
- `orchard` (patch version)
- `zcash_history` (patch version)
- `zcash_note_encryption` (patch version)
- `zcash_primitives` (patch version)
And also update the `group` dependency so that the code remains
compatible.
commit de5cf1ec40c3fc08670fc971cdf3e65e13d9f4c7
Author: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Date: Tue Oct 5 03:04:13 2021 +0000
Update error message assertion
Use the updated message for the expected error variant.
* Update `zcash_script` to support V5 transactions
Use a newer version of `zcash_script` that has been updated to support
V5 transactions.
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: teor <teor@riseup.net>
* Update versions for zebra v1.0.0-alpha.12 release
* Update Cargo.lock
* Update release checklist with latest version changes to help keep track for future releases
* Remove reference to the fact that tower-fallback was not updated
* Rename some methods and constants for clarity
Using the following commands:
```
fastmod '\bis_ready_for_attempt\b' is_ready_for_connection_attempt
# One instance required a tweak, because of the ASCII diagram.
fastmod '\bwas_recently_live\b' has_connection_recently_responded
fastmod '\bwas_recently_attempted\b' was_connection_recently_attempted
fastmod '\bwas_recently_failed\b' has_connection_recently_failed
fastmod '\bLIVE_PEER_DURATION\b' MIN_PEER_RECONNECTION_DELAY
```
* Use `Instant::elapsed` for conciseness
Instead of `Instant::now().saturating_duration_since`. They're both
equivalent, and `elapsed` only panics if the `Instant` is somehow
synthetically generated.
* Allow `Duration32` to be created in other crates
Export the `Duration32` from the `zebra_chain::serialization` module.
* Add some new `Duration32` constructors
Create some helper `const` constructors to make it easy to create
constant durations. Add methods to create a `Duration32` from seconds,
minutes and hours.
* Avoid gossiping unreachable peers
When sanitizing the list of peers to gossip, remove those that we
haven't seen in more than three hours.
* Test if unreachable addresses aren't gossiped
Create a property test with random addreses inserted into an
`AddressBook`, and verify that the sanitized list of addresses does not
contain any addresses considered unreachable.
* Test if new alternate address isn't gossipable
Create a new alternate peer, because that type of `MetaAddr` does not
have `last_response` or `untrusted_last_seen` times. Verify that the
peer is not considered gossipable.
* Test if local listener is gossipable
The `MetaAddr` representing the local peer's listening address should
always be considered gossipable.
* Test if gossiped peer recently seen is gossipable
Create a `MetaAddr` representing a gossiped peer that was reported to be
seen recently. Check that the peer is considered gossipable.
* Test peer reportedly last seen in the future
Create a `MetaAddr` representing a peer gossiped and reported to have
been last seen in a time that's in the future. Check that the peer is
considered gossipable, to check that the fallback calculation is working
as intended.
* Test gossiped peer reportedly seen long ago
Create a `MetaAddr` representing a gossiped peer that was reported to
last have been seen a long time ago. Check that the peer is not
considered gossipable.
* Test if just responded peer is gossipable
Create a `MetaAddr` representing a peer that has just responded and
check that it is considered gossipable.
* Test if recently responded peer is gossipable
Create a `MetaAddr` representing a peer that last responded within the
duration a peer is considered reachable. Verify that the peer is
considered gossipable.
* Test peer that responded long ago isn't gossipable
Create a `MetaAddr` representing a peer that last responded outside the
duration a peer is considered reachable. Verify that the peer is not
considered gossipable.
* Support a min protocol version during initial block download
But don't actually use the state height yet.
Also rename some functions and constants.
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
- Add a custom semver match for `zebrad` versions
- Prefer "line contains string" matches, so tests ignore minor changes
- Escape regex meta-characters when a literal string match is intended
- Rename test functions so they are more precise
- Rewrite match internals to remove duplicate code and enable custom matches
- Document match functions
* Rename field to `wait_next_handshake`
Make the name a bit more clear regarding to the field's purpose.
* Move `MIN_PEER_CONNECTION_INTERVAL` to `constants`
Move it to the `constants` module so that it is placed closer to other
constants for consistency and to make it easier to see any relationships
when changing them.
* Rate limit calls to `CandidateSet::update()`
This effectively rate limits requests asking for more peer addresses
sent to the same peer. A new `min_next_crawl` field was added to
`CandidateSet`, and `update` only sends requests for more peer addresses
if the call happens after the instant specified by that field. After
sending the requests, the field value is updated so that there is a
`MIN_PEER_GET_ADDR_INTERVAL` wait time until the next `update` call
sends requests again.
* Include `update_initial` in rate limiting
Move the rate limiting code from `update` to `update_timeout`, so that
both `update` and `update_initial` get rate limited.
* Test `CandidateSet::update` rate limiting
Create a `CandidateSet` that uses a mocked `PeerService`. The mocked
service always returns an empty list of peers, but it also checks that
the requests only happen after expected instants, determined by the
fanout amount and the rate limiting interval.
* Refactor to create a `mock_peer_service` helper
Move the code from the test to a utility function so that another test
will be able to use it as well.
* Check number of times service was called
Use an `AtomicUsize` shared between the service and the test body that
the service increments on every call. The test can then verify if the
service was called the number of times it expected.
* Test calling `update` after `update_initial`
The call to `update` should be skipped because the call to
`update_initial` should also be considered in the rate limiting.
* Mention that call to `update` may be skipped
Make it clearer that in this case the rate limiting causes calls to be
skipped, and not that there's an internal sleep that happens.
Also remove "to the same peers", because it's more general than that.
Co-authored-by: teor <teor@riseup.net>
Zebra avoids having a majority of addresses from a single peer by asking
3 peers for new addresses.
Also update a bunch of security comments and related documentation.
* Stop ignoring inbound message errors and handshake timeouts
To avoid hangs, Zebra needs to maintain the following invariants in the
handshake and heartbeat code:
- each handshake should run in a separate spawned task
(not yet implemented)
- every message, error, timeout, and shutdown must update the peer address state
- every await that depends on the network must have a timeout
Once the Connection is created, it should handle timeouts.
But we need to handle timeouts during handshake setup.
* Avoid hangs by adding a timeout to the candidate set update
Also increase the fanout from 1 to 2, to increase address diversity.
But only return permanent errors from `CandidateSet::update`, because
the crawler task exits if `update` returns an error.
Also log Peers response errors in the CandidateSet.
* Use the select macro in the crawler to reduce hangs
The `select` function is biased towards its first argument, risking
starvation.
As a side-benefit, this change also makes the code a lot easier to read
and maintain.
* Split CrawlerAction::Demand into separate actions
This refactor makes the code a bit easier to read, at the cost of
sometimes blocking the crawler on `candidates.next()`.
That's ok, because `next` only has a short (< 100 ms) delay. And we're
just about to spawn a separate task for each handshake.
* Spawn a separate task for each handshake
This change avoids deadlocks by letting each handshake make progress
independently.
* Move the dial task into a separate function
This refactor improves readability.
* Fix buggy future::select function usage
And document the correctness of the new code.
Zebra's latest alpha checkpoints on Canopy activation, continues our work on NU5, and fixes a security issue.
Some notable changes include:
## Added
- Log address book metrics when PeerSet or CandidateSet don't have many peers (#1906)
- Document test coverage workflow (#1919)
- Add a final job to CI, so we can easily require all the CI jobs to pass (#1927)
## Changed
- Zebra has moved its mandatory checkpoint from Sapling to Canopy (#1898, #1926)
- This is a breaking change for users that depend on the exact height of the mandatory checkpoint.
## Fixed
- tower-batch: wake waiting workers on close to avoid hangs (#1908)
- Assert that pre-Canopy blocks use checkpointing (#1909)
- Fix CI disk space usage by disabling incremental compilation in coverage builds (#1923)
## Security
- Stop relying on unchecked length fields when preallocating vectors (#1925)
* replace to_socket_addrs
* refactor `resolve()` into `resolve_host()`
* use `resolve_host()` to resolve config peers
* add DNS_LOOKUP_TIMEOUT constant
* don't block the main thread in initialize
* add hint for port error
* add issue filter for port panic
* add lock file hint
* add metrics endpoint port conflict hint
* add hint for tracing endpoint port conflict
* add acceptance test for resource conflics
* Split out common conflict test code into a function
* Add state, metrics, and tracing conflict tests
* Add a full set of stderr acceptance test functions
This change makes the stdout and stderr acceptance test interfaces
identical.
* move Zcash listener opening
* add todo about hint for disk full
* add constant for lock file
* match path in state cache
* don't match windows cache path
* Use Display for state path logs
Avoids weird escaping on Windows when using Debug
* Add Windows conflict error messages
* Turn PORT_IN_USE_ERROR into a regex
And add another alternative Windows-specific port error
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Jane Lusby <jane@zfnd.org>
* Bump versions where appropriate
Tested with cargo install --locked --path etc
* Remove fixed panics from 'Known Issues'
* Change to alpha release series in the README
Co-authored-by: teor <teor@riseup.net>
Previously we set the crate versions to 3.x, so that the major version was
aligned with the NU version. But we want to be able to make API changes
independently of the NU schedule.
Per https://zips.z.cash/zip-0251, nodes compatible with Canopy
activation on mainnet MUST advertise protocol version 170013 or later.
Once Canopy activates on testnet or mainnet, Canopy nodes SHOULD reject
new connections from pre-Canopy nodes, so this also increases the
minimum version.
* increase the EWMA default and decay
* increase the block download retries
* increase the request and block download timeouts
* increase the sync timeout
Closes#536.
This removes:
- the user-agent (we can add a mechanism to specify extra BIP14 components later, if any users ask us for that feature);
- the EWMA parameters (these were put in the config just to avoid making a choice);
- the peer connection timeout (we can change the default value if anyone ever has a problem with it);
- the peer set request buffer size (setting this too low can make the application deadlock);
The new peer interval is left in.
We had a brief discussion on discord and it seemed like we had consensus on the
following versioning policy:
* zebrad: match major version to NU version, so we will start by releasing
zebrad 3.0.0;
* zebra-* libraries: start by matching zebrad's version, then increment major
versions of each library as we need to make breaking changes (potentially
faster than the zebrad version, always respecting semver but making no
guarantees about the longevity of major releases).
This commit sets all of the crate versions to 3.0.0-alpha.0 -- the -alpha.0
marks it as a prerelease not subject to perfect adherence to compatibility
guarantees.
When we perform a handshake with a remote peer, we need to encode the
version messages with a particular network version before we find out
what the remote peer's version preference is. So in addition to having
a CURRENT_VERSION constant (which represents our preference), we need to
have a MIN_VERSION during the handshake (and later to determine whether
we'll talk to the peer at all).
* Replace Version MetaAddr by (Services, SocketAddr).
The version handshake message doesn't include last-seen timestamps for
the address fields, unlike other messages, so instead of modeling the
message data with a `MetaAddr` (which includes a timestamp), we should
just use a tuple.
* Simplify try_read_version implementation.
Because we no longer need to construct fake timestamps for the
`MetaAddr` fields, we don't need to use any of the parsed fields while
parsing later fields, and we can neatly wrap up the entire parsing logic
into a single expression.
* fmt
I didn't have the toolchain-specified `rustfmt` because I was mostly
offline and couldn't download it.
teor
Janito Vaqueiro Ferreira Filho
Conrado Gouvea
Marek
Alfredo Garcia
Pili Guerra
Deirdre Connolly
Henry de Valence
Jack Grigg
Dimitris Apostolou