* Upgrade tracing and related dependencies
```sh
cargo upgrade --workspace
tracing-error
tracing-subscrber
color-eyre
tracing-flame
tracing-journald
sentry
sentry-tracing
metrics
metrics-exporter-prometheus
reqwest
```
* Update duplicate dependency checks
* Enable the tracing/env-filter feature
* Fix type inference for metrics
Manual changes, plus:
```sh
fastmod "as _" "as f64"
```
* Tidy up some unrelated test code
* Update metrics-exporter-prometheus API
And make unused dependencies optional.
* Adjust test regexes to new tracing format
Also fix some regex bugs, and refactor to simplify.
* Disable color-eyre span traces and track caller in release builds
* Add a feature that enables extra debugging in release builds
* Clean up some redundant features
* Increase a test timeout
* Tidy links and notes for some mempool rustdoc
* Fix link
* Fix some links and linewraps
* minor spacing
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Limit the size and age of the ZIP-401 rejected transaction ID list
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Fix bug in EvictionList; improve documentation
* Separate public and non-public parts of the documentation
* Fix tests
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Fix bug in EvictionList::len()
* Make EvictionList::len() mutable; prune the list inside it
* Limit the size of EvictedList::ordered_entries
* Increase eviction_list_time_mixed time constants to try to make it pass on MacOS
* Simplify logic by assuming refreshes will never happen
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Compiling fixes
* Remove MEMPOOL_SIZE and just rely on the ZIP-401 cost limit
Co-authored-by: teor <teor@riseup.net>
* ZIP-401 weighted random mempool eviction
* rename zcash.mempool.total_cost.bytes to zcash.mempool.cost.bytes
Co-authored-by: teor <teor@riseup.net>
* Remove duplicated lines
* Add cost() method to UnminedTx
Update serialization failure messages
* More docs quoting ZIP-401 rules
* Change mempool::Storage::new() to handle Copy-less HashMap, HashSet
* mempool: tidy cost types and evict_one()
* More consensus rule docs
* Refactor calculating mempool costs for Unmined transactions
* Add a note on asympotic performance of calculating weights of txs in mempool
* Bump test mempool / storage config to avoid weighted random cost limits
* Use mempool tx_cost_limit = u64::MAX for some tests
* Remove failing tests for now
* Allow(clippy::field-reassign-with-default) because of a move on a type that doesn't impl Copy
* Fix mistaken doctest formatting
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Increase test timeout for Windows builds
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Rename tx downloader & verifier metrics
* Add version to mempool metrics
* Add new metrics
* Make sure mempool gauges are zeroed when instances are dropped
* Updated mempool grafana dashboard
* Removed transaction verification dashboard; moved to mempool
* Update mempool dashboard
* Add reason to error labels in mempool dashboard
* Rename some metrics per review
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Impl Drop, Default and take() for ActiveState
* Refactor Mempool::poll_ready to check disabled and reset first
Also remove some levels of nesting.
* Use the same code for dropping and resetting the mempool
* Document where the tasks are dropped when switching states
* Log mempool resets at info level
And add heights to mempool enable/disable/reset logs
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Create a new VerifiedUnminedTx containing the miner fee
* Use VerifiedUnminedTx in mempool verification responses
And do a bunch of other cleanups.
* Use VerifiedUnminedTx in mempool download and verifier
* Use VerifiedUnminedTx in mempool storage and verified set
* Impl Display for VerifiedUnminedTx, and some convenience methods
* Use VerifiedUnminedTx in existing tests
* Limit the size of rejection lists when there is a spend conflict
Previously, `insert` would return early with an error,
and skip limiting the rejection list sizes.
* Use prop_assert macros in proptests, rather than assert
* Add `HashSet`s to help spend conflict detection
Keep track of the spent transparent outpoints and the revealed
nullifiers.
Clippy complained that the `ActiveState` had variants with large size
differences, but that was expected, so I disabled that lint on that
`enum`.
* Clear the `HashSet`s when clearing the mempool
Clear them so that they remain consistent with the set of verified
transactions.
* Use `HashSet`s to check for spend conflicts
Store new outputs into its respective `HashSet`, and abort if a
duplicate output is found.
* Remove inserted outputs when aborting
Restore the `HashSet` to its previous state.
* Remove tracked outputs when removing a transaction
Keep the mempool storage in a consistent state when a transaction is
removed.
* Remove tracked outputs when evicting from mempool
Ensure eviction also keeps the tracked outputs consistent with the
verified transactions.
* Refactor to create a `VerifiedSet` helper type
Move the code to handle the output caches into the new type. Also move
the eviction code to make things a little simpler.
* Refactor to have a single `remove` method
Centralize the code that handles the removal of a transaction to avoid
mistakes.
* Move mempool size limiting back to `Storage`
Because the evicted transactions must be added to the rejected list.
* Remove leftover `dbg!` statement
Leftover from some temporary testing code.
Co-authored-by: teor <teor@riseup.net>
* Remove unnecessary `TODO`
It is more speculation than planning, so it doesn't add much value.
Co-authored-by: teor <teor@riseup.net>
* Fix typo in documentation
The verb should match the subject "transactions" which is plural.
Co-authored-by: teor <teor@riseup.net>
* Add a comment to warn about correctness
There's a subtle but important detail in the implementation that should
be made more visible to avoid mistakes in the future.
Co-authored-by: teor <teor@riseup.net>
* Remove outdated comment
Left-over from the attempt to move the eviction into the `VerifiedSet`.
* Improve comment explaining lint removal
Rewrite the comment explaining why the Clippy lint was ignored.
* Check for spend conflicts in `VerifiedSet`
Refactor to avoid API misuse.
* Test rejected transaction rollback
Using two transactions, perform the same test adding a conflict to both
of them to check if the second inserted transaction is properly
rejected. Then remove any conflicts from the second transaction and add
it again. That should work, because if it doesn't it means that when the
second transaction was rejected it left things it shouldn't in the
cache.
* Test removal of multiple transactions
When removing multiple transactions from the mempool storage, all of the
ones requested should be removed and any other transaction should be
still be there afterwards.
* Increase mempool size to 4, so that spend conflict tests work
If the mempool size is smaller than 4,
these tests don't fail on a trivial removal bug.
Because we need a minimum number of transactions in the mempool
to trigger the bug.
Also commit a proptest seed that fails on a trivial removal bug.
(This seed fails if we remove indexes in order,
because every index past the first removes the wrong transaction.)
* Summarise transaction data in proptest error output
* Summarise spend conflict field data in proptest error output
* Summarise multiple removal field data in proptest error output
And replace the very large proptest debug output with the new summary.
Co-authored-by: teor <teor@riseup.net>
* Split mempool storage errors into tip-based and chain-based
* Expire tip rejections every time we get a new block
FailedVerification and SpendConflict rejections only apply to the current tip.
The next tip can provide missing inputs, or evict conflicting transactions.
* Enforce MAX_EVICTION_MEMORY_ENTRIES for mempool reject lists
* Remove unused mempool storage errors
Preparation for ticket #2819.
Removing these errors means that we don't have to decide
which type of transaction ID match we want for them.
* Remove unused mempool errors, and deduplicate storage errors
* rustfmt
* Add a mempool transaction removal method for mined IDs
And use this method to remove expired transactions,
because all transactions with the same mined ID expire at the same height.
* Remove mined transaction IDs from the mempool
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Add `Transaction::spent_outpoints` getter method
Returns an iterator over the UTXO `OutPoint`s spent by the transaction.
* Add `mempool::Error::Conflict` variant
An error representing that a transaction was rejected because it
conflicts with another transaction that's already in the mempool.
* Reject conflicting mempool transactions
Reject including a transaction in the mempool if it spends outputs
already spent by, or reveals nullifiers already revealed by another
transaction in the mempool.
* Fix typo in documentation
Remove the `r` that was incorrectly added.
Co-authored-by: teor <teor@riseup.net>
* Specify that the conflict is a spend conflict
Make the situation clearer, because there are other types of conflict.
Co-authored-by: teor <teor@riseup.net>
* Clarify that the outpoints are from inputs
Because otherwise it could lead to confusion because it could also mean
the outputs of the transaction represented as `OutPoint` references.
Co-authored-by: teor <teor@riseup.net>
* Create `storage::tests::vectors` module
Refactor to follow the convention used for other tests.
* Add an `AtLeastOne::first_mut` method
A getter to allow changing the first element.
* Add an `AtLeastOne::push` method
Allow appending elements to the collection.
* Derive `Arbitrary` for `FieldNotPresent`
This is just to make the code that generates arbitrary anchors a bit
simpler.
* Test if conflicting transactions are rejected
Generate two transactions (either V4 or V5) and insert a conflicting
spend, which can be either a transparent UTXO, or a nullifier for one of
the shielded pools. Check that any attempt to insert both transactions
causes one to be accepted and the other to be rejected.
* Delete a TODO comment that we decided not to do
Co-authored-by: teor <teor@riseup.net>
* Update the expiry TODO
* Clear the mempool at a chain tip reset
* Clear the mempool by using a sync method (#2777)
* Clear the mempool by using a sync method
* Update docs
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Refactor last_tip_change()
* Apply suggestions from code review
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Fix brackets
* Use best_tip_block instead of manual borrowing
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Check if tx already exists in mempool or state before downloading
* Reorder checks
* Add rejected test; refactor into separate function
* Wrap mempool in buffered service
* Rename RejectedTransactionsById -> RejectedTransactionsIds
* Add RejectedTransactionIds response; fix request name
* Organize imports
* add a test for Storage::rejected_transactions
* add test for mempool `Request::RejectedTransactionIds`
* change buffer size to 1 in the test
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* First pass at a Mempool Service, incl. a storage layer underneath
* Fixed up Mempool service and storage
* allow dead code where needed
* clippy
* typo
* only drain if the mempool is full
* add a basic storage test
* remove space
* fix test for when MEMPOOL_SIZE change
* group some imports
* add a basic mempool service test
* add clippy suggestions
* remove not needed allow dead code
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: teor <teor@riseup.net>
teor
Conrado Gouvea
Deirdre Connolly
Alfredo Garcia
Janito Vaqueiro Ferreira Filho
Marek
Deirdre Connolly