* Remove redundant documentation
The documentation was exactly the same as the documentation from the
trait.
* Calculate a mock time block delta for tests
Simulate a block being added to the chain with a random block time based
on the previous block time and the target spacing time.
* Add a `time` field to `ChainTipBlock`
Store the block time so that it's ready for a future chain that allows
obtaining the chain tip's block time.
* Add `ChainTip::best_tip_block_time` method
Allow obtaining the bes chain tip's block time.
* Add method to obtain both height and block time
Prevent any data races by returning both values so that they refer to
the same chain tip.
* Add `NetworkUpgrade::all_target_spacings` method
Returns all the target spacings defined for a network.
* Create a `NetworkChainTipEstimator` helper type
Isolate the code to calculate the height estimation in a new type, so
that it's easier to understand and doesn't decrease the readability of
the `chain_tip.rs` file.
* Add `ChainTip::estimate_network_chain_tip_height`
This is more of an extension method than a trait method. It uses the
`NetworkChainTipHeightEstimator` to actually perform the estimation, but
obtains the initial information from the current best chain tip.
* Fix typo in documentation
There was an extra closing bracket in the summary line.
* Refactor `MockChainTipSender` into a separate type
Prepare to allow mocking the block time of the best tip as well as the
block height.
* Allow sending mock best tip block times
Add a separate `watch` channel to send the best tip block times from a
`MockChainTipSender` to a `MockChainTip`.
The `best_tip_height_and_block_time` implementation will only return a
value if there's a height and a block time value for the best tip.
* Fix off-by-one height estimation error
Use Euclidean division to force the division result to round down
instead of rounding towards zero. This fixes an off-by-one error when
estimating a height that is lower than the current height, because the
fractionary result was being discarded, and it should have forced the
height to go one block back.
* Fix panics on local times very far in the past
Detect situations that might cause the block height estimate to
underflow, and return the genesis height instead.
* Fix another off-by-one height estimation error
The implementation of `chrono::Duration::num_seconds` adds one to the
number of seconds if it's negative. This breaks the division
calculation, so it has to be compensated for.
* Test network chain tip height estimation
Generate pairs of block heights and check that it's possible to estimate
the larger height from the smaller height and a displaced time
difference.
* feat(log): log the state tip height as part of sync progress logs
* fix(log): downgrade some verbose state logs to debug
* feat(log): log successful gossiped block verification at info level
These logs help us diagnose slow progress near the tip.
There won't be very many of these logs,
because they only happen near the tip.
* fix(log): spawn top-level tasks within the global Zebra tracing span
* fix(log): spawn blocking top-level tasks within the global Zebra tracing span
Co-authored-by: teor <teor@riseup.net>
* Tweak crawler timings so peers are more likely to be available
* Tweak min peer connection interval so we try all peers
* Let other tasks run between fanouts, so we're more likely to choose different peers
* Let other tasks run between retries, so we're more likely to choose different peers
* Let other tasks run after peer crawler DemandDrop
This makes it more likely that peers will become ready.
* Use a named CancelHeartbeatTask unit struct for the channel type
* Prefer cancel handles in selects, if both are ready
* Fix message metrics to just show the command name
* Add metrics for internal requests and responses
* Add internal requests and responses to the messages dashboard
* Add a canceled metric, and peer addresses to request and response metrics
* Add a canceled messages graph
* Add connection state metrics for currently open connections
* Fix the connection state graph with new metrics
* Always send an error before dropping pending responses
* Move error detail logging into `fail_with`
* Delete an unused timer future
* Make error strings in metrics less verbose
* Downgrade some error logs to info
* Remove a redundant expect
* Avoid unnecessary allocations for connection state metrics
* Fix missed updates to mempool and block gossip metrics
* Use a single-thread shared Tokio runtime
This allows it to pause the time and more closely resembles the
environment that's set by default for asynchronous tests.
* Add a `zebra_test::init_async` helper function
Calls `zebra_test::init` but also constructs a single-thread Tokio
runtime and returns it. This makes it simpler to initialize asynchronous
tests that can't use the `#[tokio::test]` attribute.
* Replace usages of `Runtime::new` in tests
Use the new `zebra_test::init_async()` helper function instead.
* Replace `runtime::Builder::new_current_thread()`
Use the new `zebra_test::init_async()` helper function instead.
* Replace `runtime::Builder::new_multi_thread()`
Use the new `zebra_test::init_async()` helper function instead. The test
with the change doesn't necessarily have to use a multi-thread runtime.
* Tweak a log message
* Only retry failed DNS once, then use the other DNS responses
* Limit broadcasts to half the peers
* Use a longer minimum interval for GetAddr requests
* Reduce the syncer and mempool crawler fanouts
* Stop resetting the mempool twice when it starts up
This spawns two crawlers, which send two fanouts,
so it can use up a lot of peers.
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Implement graceful shutdown for the peer set
* Use the minimum lookahead limit in acceptance tests
* Enable a doctest that compiles with newly public modules
* Update `tower` to version `0.4.9`
Update to latest version to add support for Tokio version 1.
* Replace usage of `ServiceExt::ready_and`
It was deprecated in favor of `ServiceExt::ready`.
* Update Tokio dependency to version `1.13.0`
This will break the build because the code isn't ready for the update,
but future commits will fix the issues.
* Replace import of `tokio::stream::StreamExt`
Use `futures::stream::StreamExt` instead, because newer versions of
Tokio don't have the `stream` feature.
* Use `IntervalStream` in `zebra-network`
In newer versions of Tokio `Interval` doesn't implement `Stream`, so the
wrapper types from `tokio-stream` have to be used instead.
* Use `IntervalStream` in `inventory_registry`
In newer versions of Tokio the `Interval` type doesn't implement
`Stream`, so `tokio_stream::wrappers::IntervalStream` has to be used
instead.
* Use `BroadcastStream` in `inventory_registry`
In newer versions of Tokio `broadcast::Receiver` doesn't implement
`Stream`, so `tokio_stream::wrappers::BroadcastStream` instead. This
also requires changing the error type that is used.
* Handle `Semaphore::acquire` error in `tower-batch`
Newer versions of Tokio can return an error if the semaphore is closed.
This shouldn't happen in `tower-batch` because the semaphore is never
closed.
* Handle `Semaphore::acquire` error in `zebrad` test
On newer versions of Tokio `Semaphore::acquire` can return an error if
the semaphore is closed. This shouldn't happen in the test because the
semaphore is never closed.
* Update some `zebra-network` dependencies
Use versions compatible with Tokio version 1.
* Upgrade Hyper to version 0.14
Use a version that supports Tokio version 1.
* Update `metrics` dependency to version 0.17
And also update the `metrics-exporter-prometheus` to version 0.6.1.
These updates are to make sure Tokio 1 is supported.
* Use `f64` as the histogram data type
`u64` isn't supported as the histogram data type in newer versions of
`metrics`.
* Update the initialization of the metrics component
Make it compatible with the new version of `metrics`.
* Simplify build version counter
Remove all constants and use the new `metrics::incement_counter!` macro.
* Change metrics output line to match on
The snapshot string isn't included in the newer version of
`metrics-exporter-prometheus`.
* Update `sentry` to version 0.23.0
Use a version compatible with Tokio version 1.
* Remove usage of `TracingIntegration`
This seems to not be available from `sentry-tracing` anymore, so it
needs to be replaced.
* Add sentry layer to tracing initialization
This seems like the replacement for `TracingIntegration`.
* Remove unnecessary conversion
Suggested by a Clippy lint.
* Update Cargo lock file
Apply all of the updates to dependencies.
* Ban duplicate tokio dependencies
Also ban git sources for tokio dependencies.
* Stop allowing sentry-tracing git repository in `deny.toml`
* Allow remaining duplicates after the tokio upgrade
* Use C: drive for CI build output on Windows
GitHub Actions uses a Windows image with two disk drives, and the
default D: drive is smaller than the C: drive. Zebra currently uses a
lot of space to build, so it has to use the C: drive to avoid CI build
failures because of insufficient space.
Co-authored-by: teor <teor@riseup.net>
* Fix some links and linewraps
* Flesh out module-level rustdoc for mempool component
* Links for Downloads and Storage
* Tidy rustdoc links
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Tidy links and notes for some mempool rustdoc
* Fix link
* Fix some links and linewraps
* minor spacing
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Limit the size and age of the ZIP-401 rejected transaction ID list
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Fix bug in EvictionList; improve documentation
* Separate public and non-public parts of the documentation
* Fix tests
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Fix bug in EvictionList::len()
* Make EvictionList::len() mutable; prune the list inside it
* Limit the size of EvictedList::ordered_entries
* Increase eviction_list_time_mixed time constants to try to make it pass on MacOS
* Simplify logic by assuming refreshes will never happen
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Compiling fixes
* Remove MEMPOOL_SIZE and just rely on the ZIP-401 cost limit
Co-authored-by: teor <teor@riseup.net>
* ZIP-401 weighted random mempool eviction
* rename zcash.mempool.total_cost.bytes to zcash.mempool.cost.bytes
Co-authored-by: teor <teor@riseup.net>
* Remove duplicated lines
* Add cost() method to UnminedTx
Update serialization failure messages
* More docs quoting ZIP-401 rules
* Change mempool::Storage::new() to handle Copy-less HashMap, HashSet
* mempool: tidy cost types and evict_one()
* More consensus rule docs
* Refactor calculating mempool costs for Unmined transactions
* Add a note on asympotic performance of calculating weights of txs in mempool
* Bump test mempool / storage config to avoid weighted random cost limits
* Use mempool tx_cost_limit = u64::MAX for some tests
* Remove failing tests for now
* Allow(clippy::field-reassign-with-default) because of a move on a type that doesn't impl Copy
* Fix mistaken doctest formatting
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Increase test timeout for Windows builds
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Try simulating a chain growth
* Adjust the transaction expiry height
The mempool evicts expired transactions. When working with mocked data,
appending a new block typically clears the mempool because transactions become
expired. For this reason, the expiry height of each transactions is adjusted so
that it is greater than the new chain tip's height.
* Refactor the code so that it works with `VerifiedUnminedTx`
* Fix a typo
* Fix clippy warnings
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Rename tx downloader & verifier metrics
* Add version to mempool metrics
* Add new metrics
* Make sure mempool gauges are zeroed when instances are dropped
* Updated mempool grafana dashboard
* Removed transaction verification dashboard; moved to mempool
* Update mempool dashboard
* Add reason to error labels in mempool dashboard
* Rename some metrics per review
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Guarantee unique IDs in mempool service responses
* Guarantee unique IDs in crawler task mempool Queue requests
Also update the tests to use unique IDs.
* Add a CheckForVerifiedTransactions mempool request
Also document the mempool request and response variants.
* Spawn a QueueChecker task to check for newly verified transactions
This task makes sure that transactions reliably propagate,
rather than relying on peer requests or responses to trigger propagation.
* Update the start command documentation
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Impl Drop, Default and take() for ActiveState
* Refactor Mempool::poll_ready to check disabled and reset first
Also remove some levels of nesting.
* Use the same code for dropping and resetting the mempool
* Document where the tasks are dropped when switching states
* Log mempool resets at info level
And add heights to mempool enable/disable/reset logs
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Guarantee unique IDs in mempool service responses
* Guarantee unique IDs in crawler task mempool Queue requests
Also update the tests to use unique IDs.
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Create a new VerifiedUnminedTx containing the miner fee
* Use VerifiedUnminedTx in mempool verification responses
And do a bunch of other cleanups.
* Use VerifiedUnminedTx in mempool download and verifier
* Use VerifiedUnminedTx in mempool storage and verified set
* Impl Display for VerifiedUnminedTx, and some convenience methods
* Use VerifiedUnminedTx in existing tests
* Get the transaction fee from utxos
* Return the transaction fee from the verifier
* Avoid calculating the fee for coinbase transactions
Coinbase transactions don't have fees. In case of a coinbase transaction, the
verifier returns a zero fee.
* Update the result obtained by `Downloads`
* Update some comments
* Add a mempool debug_enable_at_height config
* Rename a field in the mempool crawler
* Propagate syncer channel errors through the crawler
We don't want to ignore these errors, because they might indicate a shutdown.
(Or a bug that we should fix.)
* Use debug_enable_at_height in the mempool crawler
* Log when the mempool is activated or deactivated
* Deny unknown fields and apply defaults for all configs
* Move Duration last, as required for TOML tables
* Add a basic mempool acceptance test
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* do not advertise rejected transactions
* do not broadcast transaction that are expired
* change dummy var name
* simplify code, performance
* clippy
* add some test coverage
* clippy
Co-authored-by: teor <teor@riseup.net>
* Split mempool config into its own module
Also:
- expand config docs
- clean up mempool imports
* Pass the mempool config to the mempool
* Create the transaction sender channel inside the mempool 1/2
This simplifies all the code that calls the mempool.
Also:
- update the mempool enabled state before returning the new mempool
- add some test module doc comments
* Refactor a setup function out of the mempool unit tests 2/2
Also:
- update the setup function to handle the latest mempool changes
* Clarify a comment
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Limit the size of rejection lists when there is a spend conflict
Previously, `insert` would return early with an error,
and skip limiting the rejection list sizes.
* Use prop_assert macros in proptests, rather than assert
* Add `HashSet`s to help spend conflict detection
Keep track of the spent transparent outpoints and the revealed
nullifiers.
Clippy complained that the `ActiveState` had variants with large size
differences, but that was expected, so I disabled that lint on that
`enum`.
* Clear the `HashSet`s when clearing the mempool
Clear them so that they remain consistent with the set of verified
transactions.
* Use `HashSet`s to check for spend conflicts
Store new outputs into its respective `HashSet`, and abort if a
duplicate output is found.
* Remove inserted outputs when aborting
Restore the `HashSet` to its previous state.
* Remove tracked outputs when removing a transaction
Keep the mempool storage in a consistent state when a transaction is
removed.
* Remove tracked outputs when evicting from mempool
Ensure eviction also keeps the tracked outputs consistent with the
verified transactions.
* Refactor to create a `VerifiedSet` helper type
Move the code to handle the output caches into the new type. Also move
the eviction code to make things a little simpler.
* Refactor to have a single `remove` method
Centralize the code that handles the removal of a transaction to avoid
mistakes.
* Move mempool size limiting back to `Storage`
Because the evicted transactions must be added to the rejected list.
* Remove leftover `dbg!` statement
Leftover from some temporary testing code.
Co-authored-by: teor <teor@riseup.net>
* Remove unnecessary `TODO`
It is more speculation than planning, so it doesn't add much value.
Co-authored-by: teor <teor@riseup.net>
* Fix typo in documentation
The verb should match the subject "transactions" which is plural.
Co-authored-by: teor <teor@riseup.net>
* Add a comment to warn about correctness
There's a subtle but important detail in the implementation that should
be made more visible to avoid mistakes in the future.
Co-authored-by: teor <teor@riseup.net>
* Remove outdated comment
Left-over from the attempt to move the eviction into the `VerifiedSet`.
* Improve comment explaining lint removal
Rewrite the comment explaining why the Clippy lint was ignored.
* Check for spend conflicts in `VerifiedSet`
Refactor to avoid API misuse.
* Test rejected transaction rollback
Using two transactions, perform the same test adding a conflict to both
of them to check if the second inserted transaction is properly
rejected. Then remove any conflicts from the second transaction and add
it again. That should work, because if it doesn't it means that when the
second transaction was rejected it left things it shouldn't in the
cache.
* Test removal of multiple transactions
When removing multiple transactions from the mempool storage, all of the
ones requested should be removed and any other transaction should be
still be there afterwards.
* Increase mempool size to 4, so that spend conflict tests work
If the mempool size is smaller than 4,
these tests don't fail on a trivial removal bug.
Because we need a minimum number of transactions in the mempool
to trigger the bug.
Also commit a proptest seed that fails on a trivial removal bug.
(This seed fails if we remove indexes in order,
because every index past the first removes the wrong transaction.)
* Summarise transaction data in proptest error output
* Summarise spend conflict field data in proptest error output
* Summarise multiple removal field data in proptest error output
And replace the very large proptest debug output with the new summary.
Co-authored-by: teor <teor@riseup.net>
* bradcast transactions to peers after they get inserted into mempool
* remove network argument from mempool init
* remove dbg left
* remove return value in mempool enable call
* rename channel sender and receiver vars
* change unwrap() to expect()
* change the channel to a hashset
* fix build
* fix tests
* rustfmt
* fix tiny space issue inside macro
Co-authored-by: teor <teor@riseup.net>
* check errors/panics in transaction gossip tests
* fix build of newly added tests
* Stop dropping the inbound service and mempool in a test
Keeping the mempool around avoids a transaction broadcast task error,
so we can test that there are no other errors in the task.
* Tweak variable names and add comments
* Avoid unexpected drops by returning a mempool guard in tests
* Use BoxError to simplify service types in tests
* Make all returned service types consistent in tests
We want to be able to change the setup without changing the tests.
Co-authored-by: teor <teor@riseup.net>
* Split mempool storage errors into tip-based and chain-based
* Expire tip rejections every time we get a new block
FailedVerification and SpendConflict rejections only apply to the current tip.
The next tip can provide missing inputs, or evict conflicting transactions.
* Enforce MAX_EVICTION_MEMORY_ENTRIES for mempool reject lists
* Implement a task that gossips verified block hashes
* Log an info message for block broadcasts
* Simplify the gossip task
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Re-use the old tip change if there is no new tip change
Also improve the comments.
* Add an assertion message
* Rename task join handles and futures in start method
* Add a dedicated BlockGossipError type
This type helps distinguish between syncer and state errors.
* Test that committed blocks are gossiped to peers
Also do a minor type cleanup on the existing test code,
replacing `Option<Vec<_>>` with `Vec<_>`.
* Formatting
* Remove excess newlines
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Clear the initial gossiped blocks during test setup
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Remove unused mempool storage errors
Preparation for ticket #2819.
Removing these errors means that we don't have to decide
which type of transaction ID match we want for them.
* Remove unused mempool errors, and deduplicate storage errors
* rustfmt
* Add a mempool transaction removal method for mined IDs
And use this method to remove expired transactions,
because all transactions with the same mined ID expire at the same height.
* Remove mined transaction IDs from the mempool
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Rename type parameter to be more explicit
Replace the single letter with a proper name.
* Remove imports for `Request` and `Response`
The type names will conflict with the ones for the mempool service.
* Attach `Mempool` service to the `Crawler`
Add a field to the `Crawler` type to store a way to access the `Mempool`
service.
* Forward crawled transactions to downloader
The crawled transactions are now sent to the transaction downloader and
verifier, to be included in the mempool.
* Derive `Eq` and `PartialEq` for `mempool::Request`
Make it simpler to use the `MockService::expect_request` method.
* Test if crawled transactions are downloaded
Create some dummy crawled transactions, and let the crawler discover
them. Then check if they are forwarded to the mempool to be downloaded
and verified.
* Don't send empty transaction ID list to downloader
Ignore response from peers that don't provide any crawled transactions.
* Log errors when forwarding crawled transaction IDs
Calling the Mempool service should not fail, so if an error happens it
should be visible. However, errors when downloading individual
transactions can happen from time to time, so there's no need for them
to be very visible.
* Document existing `mempool::Crawler` test
Provide some depth as to what the test expect from the crawler's
behavior.
* Refactor to create `setup_crawler` helper function
Make it easier to reuse the common test setup code.
* Simplify code to expect requests
Now that `zebra_network::Request` implement `Eq`, the call can be
simplified into `expect_request`.
* Refactor to create `respond_with_transaction_ids`
A helper function that checks for a network crawl request and responds
with the given list of crawled transaction IDs.
* Refactor to create `crawler_iterator` helper
A function to intercept and respond to the fanned-out requests sent
during a single crawl iteration.
* Refactor to create `respond_to_queue_request`
Reduce the repeated code necessary to intercept and reply to a request
for queuing transactions to be downloaded.
* Add `respond_to_queue_request_with_error` helper
Intercepts a mempool request to queue transactions to be downloaded, and
responds with an error, simulating an internal problem in the mempool
service implementation.
* Derive `Arbitrary` for `NetworkUpgrade`
This is required for deriving `Arbitrary` for some error types.
* Derive `Arbitrary` for `TransactionError`
Allow random transaction errors to be generated for property tests.
* Derive `Arbitrary` for `MempoolError`
Allow random Mempool errors to be generated for property tests.
* Test if errors don't stop the mempool crawler
The crawler should be robust enough to continue operating even if the
mempool service fails to download transactions or even fails to handle
requests to enqueue transactions.
* Reduce the log level for download errors
They should happen regularly, so there's no need to have them with a
high visibility level.
Co-authored-by: teor <teor@riseup.net>
* Stop crawler if service stops
If `Mempool::poll_ready` returns an error, it's because the mempool
service has stopped and can't handle any requests, so the crawler should
stop as well.
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Move mempool tests into `tests::vector` sub-module
Make it consistent with other test modules and prepare for adding
property tests.
* Reorder imports
Make it consistent with the general guidelines followed on other
modules.
* Export `ChainTipBlock` and `ChainTipSender`
Allow these types to be used in other crates for testing purposes.
* Derive `Arbitrary` for `ChainTipBlock`
Make it easy to generate random `ChainTipBlock`s for usage in property
tests.
* Refactor to move test methods into `tests` module
Reduce the repeated test configuration attributes and make it easier to
see what is test specific and what is part of the general
implementation.
* Add a `Mempool::dummy_call` test helper method
Performs a dummy call just so that `poll_ready` gets called.
* Use `dummy_call` in existing tests
Replace the custom dummy requests with the helper method.
* Test if the mempool is cleared on chain reset
A chain reset should force the mempool storage to be cleared so that
transaction verification can restart using the new chain tip.
* Test if mempool is cleared on syncer restart
If the block synchronizer falls behind and then starts catching up
again, the mempool should be disabled and therefore the storage should
be cleared.
* Send mined transaction IDs to the download and verify task for cancellation
* Pass a HashSet of transaction hashes to be cancelled
* Add mempool_cancel_mined() test
* Fix starvation in test
* Fix typo in comment
* mempool - support transaction expiration
* use `LatestChainTip` instead of state call
* clippy
* remove spawn task
* remove non needed async from function
* remove return value
* add a `expiry_height_mut()` method to `Transaction` for testing purposes
* fix `remove_expired_transactions()`
* add a `mempool_transaction_expiration()` test
* tidy cleanup to `expiry_height()`
* improve docs
* fix the build
* try fix macos build
* extend tests
* add doc to function
* clippy
* fix build
* start tests at block two
* Cancel download and verify tasks when the mempool is deactivated
* Refactor enable/disable logic to use a state enum
* Add helper test functions to enable/disable the mempool
* Add documentation about errors on service calls
* Improvements from review
* Improve documentation
* Fix bug in test
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: teor <teor@riseup.net>
* Add `Transaction::spent_outpoints` getter method
Returns an iterator over the UTXO `OutPoint`s spent by the transaction.
* Add `mempool::Error::Conflict` variant
An error representing that a transaction was rejected because it
conflicts with another transaction that's already in the mempool.
* Reject conflicting mempool transactions
Reject including a transaction in the mempool if it spends outputs
already spent by, or reveals nullifiers already revealed by another
transaction in the mempool.
* Fix typo in documentation
Remove the `r` that was incorrectly added.
Co-authored-by: teor <teor@riseup.net>
* Specify that the conflict is a spend conflict
Make the situation clearer, because there are other types of conflict.
Co-authored-by: teor <teor@riseup.net>
* Clarify that the outpoints are from inputs
Because otherwise it could lead to confusion because it could also mean
the outputs of the transaction represented as `OutPoint` references.
Co-authored-by: teor <teor@riseup.net>
* Create `storage::tests::vectors` module
Refactor to follow the convention used for other tests.
* Add an `AtLeastOne::first_mut` method
A getter to allow changing the first element.
* Add an `AtLeastOne::push` method
Allow appending elements to the collection.
* Derive `Arbitrary` for `FieldNotPresent`
This is just to make the code that generates arbitrary anchors a bit
simpler.
* Test if conflicting transactions are rejected
Generate two transactions (either V4 or V5) and insert a conflicting
spend, which can be either a transparent UTXO, or a nullifier for one of
the shielded pools. Check that any attempt to insert both transactions
causes one to be accepted and the other to be rejected.
* Delete a TODO comment that we decided not to do
Co-authored-by: teor <teor@riseup.net>
* Add tests for mempool Request::Queue
* Update test to work after refactoring
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Update the expiry TODO
* Clear the mempool at a chain tip reset
* Clear the mempool by using a sync method (#2777)
* Clear the mempool by using a sync method
* Update docs
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Refactor last_tip_change()
* Apply suggestions from code review
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Fix brackets
* Use best_tip_block instead of manual borrowing
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Remove return of redundant vector length
An attempt to improve readability a bit by not returning a tuple with a
value that can be obtained from a single return type.
* Refactor `unmined_transactions_in_blocks`
Use a more functional style to try to make it a bit clearer.
* Use ranges in `unmined_transactions_in_blocks`
Allow a finer control over the block range to extract the transactions
from.
* Refactor mempool test code to improve clarity
It was previously not clear that only the first genesis transaction was
being used. The remaining transactions in the genesis block were
discarded and then readded later.
* Replace `oneshot` with `call`
Remove a redundant internal `ready_and` call.
* Return an `impl Iterator` instead of a `Vec<_>`
Remove unnecessary deserializations and heap allocations.
* Refactor `mempool_storage_basic_for_network` test
Make the separation between the transactions expected to be in the
mempool and those expected to be rejected clearer.
* Replace `Iterator` with `DoubleEndedIterator`
Allow getting the last transaction more easily.
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Use `MockService` in inbound test
Refactor the `mempool_requsets_for_transactions` test so that it uses a
`MockService` instead of the `mock_peer_set` function.
* Use `MockService` in the basic mempool test
Refactor the `mempool_service_basic` test so that it uses a
`MockService` instead of the `mock_peer_set` helper function.
* Remove the `mock_peer_set` helper function
It is not used anymore, since the usages were replaced with
`MockService`s.
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Implement initial service mocking helpers
Adds a [`MockService`] type, which can be configured and built for usage
in unit tests or proptests. The mocked service can then be used to
intercept requests and respond indivdiually to them.
* Use `MockService in the `mempool::Crawler` test
Refactor it to remove the helper mock function, and use the new
`MockService` helper type.
* Use `MockService` in `CandidateSet` test vectors
Refactor to remove the manual mocking of the peer set service.
* Panic if a response is not sent by `MockService`
Change the current semantics to require all `MockService` usages to
respond to every intercepted request.
A `must_use` attribute was added to the `ResponseSender` so that the
compiler can warn when this doesn't happen.
* Allow generic error types in `MockService`
Replace the hard-coded `BoxError` as the `Service`'s error type with a
generic type parameter. This allows mocking services in locations that
require specific error types.
* Add a `ResponseSender::request` getter
Allow inspecting the request again before responding, and using
information from the request in the response.
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Pass sync_status to mempool
* Update zebrad/src/components/mempool.rs
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Remove enabled flag for now; will be handled in #2723
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Check if tx already exists in mempool or state before downloading
* Reorder checks
* Add rejected test; refactor into separate function
* Wrap mempool in buffered service
* Rename RejectedTransactionsById -> RejectedTransactionsIds
* Add RejectedTransactionIds response; fix request name
* Organize imports
* add a test for Storage::rejected_transactions
* add test for mempool `Request::RejectedTransactionIds`
* change buffer size to 1 in the test
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Using `&mut self` as the receiver in the method signatures allows Rust
to infer that the type is properly `Sync`, and therefore `Send`. This
allows removing the `Mutex` work-around.
* Store a `SyncStatus` handle in the `Crawler`
The helper type will make it easier to determine if the crawler is
enabled or not.
* Pause crawler if mempool is disabled
Implement waiting until the mempool becomes enabled, so that the crawler
does not run while the mempool is disabled.
If the `MempoolStatus` helper is unable to determine if the mempool is
enabled, stop the crawler task entirely.
* Update test to consider when crawler is paused
Change the mempool crawler test so that it's a proptest that tests
different chain sync. lengths. This leads to different scenarios with
the crawler pausing and resuming.
Co-authored-by: teor <teor@riseup.net>
This avoids peer set contention when most peers are busy.
Also exit the task if the peer service returns a readiness error,
because that means it's permanently unusable.
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* First pass at a Mempool Service, incl. a storage layer underneath
* Fixed up Mempool service and storage
* allow dead code where needed
* clippy
* typo
* only drain if the mempool is full
* add a basic storage test
* remove space
* fix test for when MEMPOOL_SIZE change
* group some imports
* add a basic mempool service test
* add clippy suggestions
* remove not needed allow dead code
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: teor <teor@riseup.net>
* Create initial `mempool::Crawler` type
The mempool crawler is responsible for periodically asking peers for
transactions to insert into the local mempool. This initial
implementation will periodically ask for transactions, but won't do
anything with them yet.
Also, the crawler is currently configured to be always enabled, but this
should be fixed to avoid crawling while Zebra is still syncing the
chain.
* Add a timeout to peer responses
Prevent the crawler from getting stuck if there's communication with a
peer that takes too long to respond.
* Run the mempool crawler in Zebra
Spawn a task for the crawler when Zebra starts.
* Test if the crawler is sending requests
Create a mock for the `PeerSet` service to intercept requests and verify
that the transaction requests are sent periodically.
* Use `full` Tokio features when testing
Make it simpler to select the features for test builds.
Co-authored-by: teor <teor@riseup.net>
* Link to the issue for crawler activation
Make it easy to navigate from the `TODO` comment to the current project
planning.
Co-authored-by: teor <teor@riseup.net>
* Link to the issue for downloading transactions
Make it easy to navigate from the `TODO` comment to the current project
planning.
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: teor <teor@riseup.net>
Janito Vaqueiro Ferreira Filho
Alfredo Garcia
teor
Deirdre Connolly
Conrado Gouvea
Marek
Deirdre Connolly