Commit Graph

67 Commits

Author SHA1 Message Date
teor f2a2a403a8
fix(log): Remove redundant startup logs, fix progress bar number, order, and wording (#7087)
* Remove duplicate "running" message, send log file opening logs to stderr

* Add missing block progress bar desc, make them consistent

* Put progress bars in a specific order in each section

* Actually make block progress bar desc consistent

* Make progress bar order even more consistent

* Fix fork blocks plural for 1 block

* Use the correct number of chain fork bars

* Disable confusing partial work display

* Add struct field category comments

* Silence a verbose inventory log
2023-07-05 07:08:59 +00:00
Conrado Gouvea fd78556000
fix: prevent progress bar from panicking using workaround (#6940)
* fix: prevent progress bar from panicking using workaround

* remove more progress bar lengths

* re-add set_len() calls commented out; restore one that shouldn't cause panics

* Update zebra-state/src/service/non_finalized_state.rs

Co-authored-by: teor <teor@riseup.net>

---------

Co-authored-by: teor <teor@riseup.net>
2023-06-14 19:07:02 +00:00
teor dc5198959e
fix(security): Rate-limit and size-limit peer transaction ID messages (#6625)
* Update MAX_TX_INV_IN_MESSAGE for ZIP-239 WTX IDs

* Combine multiple transaction updates into a single gossip & rate-limit gossips

* Rate-limit block gossips

* Fix mempool_transaction_expiration gossip test timings

* Enforce MAX_TX_INV_IN_MESSAGE in the network layer, rather than each service

* Fix documentation for `Message::Tx`

* Split MAX_INV_IN_RECEIVED_MESSAGE and MAX_TX_INV_IN_SENT_MESSAGE

* Fix log message typo

* Move some docs to/from another PR

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2023-05-15 21:10:28 +00:00
teor 166526a088
feat(ui): Add a terminal-based progress bar to Zebra (#6235)
* Implement Display and to_string() for NetworkUpgrade

* Add a progress-bar feature to zebrad

* Add the progress bar writer to the tracing component

* Add a block progress bar transmitter

* Correctly shut down the progress bar, and shut it down on an interrupt

* Make it clearer that the progress task never exits

* Add a config for writing logs to a file

* Add a progress-bar feature to zebra-network

* Add a progress bar for the address book size

* Add progress bars for never attempted and failed peers

* Add an optional limit and label to connection counters

* Add open connection progress bars

* Improve CheckpointList API and CheckpointVerifier debugging

* Add checkpoint index and checkpoint queue progress bars

* Security: Limit the number of non-finalized chains tracked by Zebra

* Make some NonFinalizedState methods available with proptest-impl

* Add a non-finalized chain count progress bar

* Track the last fork height for newly forked chains

* Add a should_count_metrics to Chain

* Add a display method for PartialCumulativeWork

* Add a progress bar for each chain fork

* Add a NonFinalizedState::disable_metrics() method and switch to using it

* Move metrics out of Chain because we can't update Arc<Chain>

* Fix: consistently use best chain order when searching chains

* Track Chain progress bars in NonFinalizedState

* Display work as bits, not a multiple of the target difficulty

* Handle negative fork lengths by reporting "No fork"

* Correctly disable unused fork bars

* clippy: rewrite using `match _.cmp(_) { ... }`

* Initial mempool progress bar implementation

* Update Cargo.lock

* Add the actual transaction size as a description to the cost bar

* Only show mempool progress bars after first activation

* Add queued and rejected mempool progress bars

* Clarify cost note is actual size

* Add tracing.log_file config and progress-bar feature to zebrad docs

* Derive Clone for Chain

* Upgrade to howudoin 0.1.2 and remove some bug workarounds

* Directly call the debug formatter to Display a Network

Co-authored-by: Arya <aryasolhi@gmail.com>

* Rename the address count metric to num_addresses

Co-authored-by: Arya <aryasolhi@gmail.com>

* Simplify reverse checkpoint lookup

Co-authored-by: Arya <aryasolhi@gmail.com>

* Simplify progress bar shutdown code

Co-authored-by: Arya <aryasolhi@gmail.com>

* Remove unused MIN_TRANSPARENT_TX_MEMPOOL_SIZE

* Document that the progress task runs forever

* Fix progress log formatting

* If progress-bar is on, log to a file by default

* Create missing directories for log files

* Add file security docs for running Zebra with elevated permissions

* Document automatic log file, spell progress-bar correctly

---------

Co-authored-by: Arya <aryasolhi@gmail.com>
2023-04-13 08:42:17 +00:00
Arya d7842bd467
fix(rpc): Check that mempool transactions are valid for the state's chain info in getblocktemplate (#6416)
* check last seen tip hash from mempool in fetch_mempool_transactions()

* Moves last_seen_tip_hash to ActiveState

* fixes tests and tests fixes

* continues to the next iteration of the loop to make fresh state and mempool requests if called with a long poll id

* Update zebra-rpc/src/methods/get_block_template_rpcs.rs

Co-authored-by: teor <teor@riseup.net>

* adds allow[unused_variable) for linter

* expects a chain tip when not(test)

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Addresses comments in code review

* - Removes second call to `last_tip_change()`

- Fixes tests using enabled mempool

* Adds note about chain tip action requirement to test method `enable()`

* updates doc comment

* Update zebrad/src/components/mempool.rs

Co-authored-by: teor <teor@riseup.net>

* fixes test

---------

Co-authored-by: teor <teor@riseup.net>
2023-04-03 23:22:07 +00:00
teor f92db25f72
Fix: stop ignoring updates to newly initialized mempools (#6448) 2023-04-03 17:57:56 +00:00
teor 60ebefc988
fix(mempool): Stop ignoring some transaction broadcasts (#6230)
* Combine transaction IDs that arrive close together rather than overwriting them

* Reduce the size of transaction ID gossip logs

* Limit the maximum number of times we wait for new changes before sending

* Make logs even shorter

* Expand correctness comment

* Remove trailing space

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2023-03-02 04:56:24 +00:00
Arya f253213e34
fix(mempool): Re-verify transactions that were verified at a different tip height (#6154)
* checks tip height before mempool insertions

* adds unit test for reverifying txs

* Adds TODO

* Adds correctness note

* dedup best_tip_height() calls

* Update zebrad/src/components/mempool.rs

Co-authored-by: teor <teor@riseup.net>

* uses Option for expected tip height

* removes misplaced dummy_call()

* calls wait_for_chain_tip without a timeout where it doesn't matter and skips instead of panicking where it doesn't

* Update zebrad/src/components/mempool/tests/vector.rs

* removes whitespace for rustfmt

---------

Co-authored-by: teor <teor@riseup.net>
2023-02-16 20:06:42 +00:00
teor e20cf957e3
fix(consensus): Verify the lock times of mempool transactions (#6027)
* Implement the BestChainNextMedianTimePast state request

* Verify the lock times of mempool transactions

* Document that the mempool already handles lock time rejections correctly

* Fix existing tests

* Add new mempool lock time success and failure tests
2023-01-27 21:46:51 +00:00
teor 256b1c0008
fix(clippy): Fix new lints in nightly clippy (#5959)
* Derive default using #[default]

* Implement PartialEq manually to satisfy clippy

* Allow a manual derive in test-only code

* Fix some missing docs warnings in the Docker build
2023-01-18 00:27:42 +00:00
teor cb1045ae5f
change(mempool): Re-verify mempool transactions after a chain fork, rather than re-downloading them all (#5841)
* Move Drop from mempool::ActiveState to mempool::Downloads, to avoid bugs

* Re-verify mempool transactions after a fork

And add a marker struct for mempool download cancellation.

* Update README based on recent mitigations for some issues, tidy format

* Make mempool proptests easier to debug

* Make UnminedTx Display text much smaller

* Update tests for mempool transaction re-verification after forks

* Retry all stored and pending transactions

* Fix a test to check for mempool reset retries
2022-12-12 23:19:45 +00:00
teor 4078e244d3
fix(lint): Box large error types to resolve the clippy large result err variant lint (#5759)
* Box errors to deal with large error warnings, add accessor methods for error properties

* Remove or explain some large enum variant lints

* Turn some tickets into TODOs

* Allow missing docs on single-field error enum variants

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-12-08 06:11:33 +00:00
Arya 1e1222c888
change(rpc): Return an error from getblocktemplate RPC if Zebra is still syncing lots of blocks (#5769)
* updates MAX_ESTIMATED_DISTANCE_TO_NETWORK_TIP

* adds ChainSyncStatus trait

* adds sync_status to GetBlockTemplateRpcs

Returns an error from getblocktemplate rpc if !is_close_to_tip

adds MockSyncStatus for tests

* hide MockSyncStatus behind feature flag

* moves is_close_to_tip call into async block

adds tests

* Update zebra-rpc/src/methods/tests/vectors.rs

Co-authored-by: teor <teor@riseup.net>

* uses SeqCst ordering for atomic bool ops.

uses a constant for not synced error code.

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-12-02 11:21:23 +00:00
teor 71f5e63e64
3. change(rpc): Add fee and sigops fields to getblocktemplate transactions (#5508)
* Add a legacy_sigop_count field to VerifiedUnminedTx

* Add conversions from Vec<VerifiedUnminedTx> to block header roots

* Add fee and sigops field to block template transactions

* Fix up mempool request names

* Increase existing snapshot test coverage

* Document a new method parameter

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2022-11-03 17:03:41 +00:00
teor 142411508b
2. change(rpc): Add some transaction fields to the `getblocktemplate` RPC (#5496)
* Add documentation for the getblocktemplate RPC

* Add a new mempool::Request::Transactions

* Add conversions from Vec<UnminedTx> to merkle::Root and AuthDataRoot

* Fill in the merkle root and auth data root fields

* Delete the Coinbase type, it's the same as Transaction

* Fill in some other existing types

* Add Hex serialization support to some zebra-chain types

* Add TransactionTemplate fields and fill some in

* Fix test hangs by spawning async tasks

* Add temporary workaround for no transactions in the block

* Encode hashes and roots as hex

* Update RPC snapshots

* Add a missing Request::Transactions handler

* Fix doc warnings

* Fix fee serialization

* Update snapshots for serialization changes

* Add a missing Cargo.lock change

* Change depends type

* Remove duplicate feature entry

* Document the new RPC feature

* Fix a comment typo

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Update default roots docs

* Fix comment typo

* Fix a comment typo

Co-authored-by: Arya <aryasolhi@gmail.com>

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: Arya <aryasolhi@gmail.com>
2022-11-03 03:25:01 +00:00
Arya e9452d9a6f
change(mempool) reject transactions with spent outpoints or nullifiers (#5434)
* adds transactions to ChainTipBlock and rejects transactions from the mempool that have been invalidated by the latest block commit

* merges remove_same_effects in with reject_invalidated_transactions, moves nullifier retrieval to the new Storage method, rejects mined_ids, and updates tests

* updates DuplicateSpend's error message

* fixes tests

* Update zebrad/src/components/mempool/storage.rs

Co-authored-by: teor <teor@riseup.net>

* adds comment

* formatting for the proptest

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
2022-10-24 19:48:45 +00:00
teor ea34baa8b8
add(test): test disabled `lightwalletd` mempool gRPCs via zebrad logs (#5016)
* add grpc mempool test research

* add a config flag for mempool injection of transactions in test

* Only copy the inner state directory in the send transactions test

* Preload Zcash parameters in some transaction verification tests

* Add a block and transaction Hash method to convert from display order bytes

* Update test coverage docs

* Add debugging output for mempool transaction verification

* Test fetching sent mempool transactions using gRPC

* Add extra log checks to the send transaction test

* Wait for zebrad mempool activation before running gRPC tests

* Update send transaction test for lightwalletd not returning mempool transactions

* Check zebrad logs instead of disabled lightwalletd gRPCs

* Add a debug option that makes RPCs pretend the sync is finished

* Remove an unused debug option

* Remove unused test code and downgrade some logs

* Fix test log checks

* Fix some rustdoc warnings

* Fix a compilation error due to new function arguments

* Make zebrad sync timeouts consistent and remove outdated code

* Document how to increase temporary directory space for tests

* Stop checking for a log that doesn't always happen

* Remove some commented-out code

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Update a comment about run time

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Add new config to new tests from the `main` branch

* Add transactions to the list, rather than replacing the list with each new block

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2022-09-06 13:32:33 +00:00
Marek cc75c3f5f9
fix(doc): Fix various doc warnings, part 3 (#4611)
* Fix the syntax of links in comments

* Fix a mistake in the docs

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Remove unnecessary angle brackets from a link

* Revert the changes for links that serve as references

* Revert "Revert the changes for links that serve as references"

This reverts commit 8b091aa9fab453e7d3559a5d474e0879183b9bfb.

* Remove `<` `>` from links that serve as references

This reverts commit 046ef25620ae1a2140760ae7ea379deecb4b583c.

* Don't use `<` `>` in normal comments

* Don't use `<` `>` for normal comments

* Revert changes for comments starting with `//`

* Fix some warnings produced by `cargo doc`

* Fix some rustdoc warnings

* Fix some warnings

* Refactor some changes

* Fix some rustdoc warnings

* Fix some rustdoc warnings

* Resolve various TODOs

Co-authored-by: teor <teor@riseup.net>

* Fix some unresolved links

* Allow links to private items

* Fix some unresolved links

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-06-15 03:57:19 +00:00
teor a98e9291b5
cleanup(clippy): Remove redundant bindings, allocations, and generics (#4353)
* Remove let bindings with unit values

* Replace push(format!(_)) with write!(_)

* Remove unused lifetime in test
2022-05-10 03:41:51 +00:00
Conrado Gouvea 41d240feaf
feat(mempool): add TransactionsByMinedId (#3907) 2022-03-18 23:00:03 +00:00
Janito Vaqueiro Ferreira Filho c24ea1fc3f
Refactor to create a new `zebra-node-services` crate (#3648)
* Create new empty `zebra-node-services` crate

The goal is to store the mempool `Request` and `Response` types so that
the `zebra-rpc` crate can interface with the mempool service without
having to import `zebrad`.

* Move `Gossip` mempool type into new crate

It is required by the `Request` type, which will be moved next.

* Add documentation to `Gossip` variants

Avoid having to add an exception to allow undocumented code.

* Move `mempool::Request` type to new crate

The first part of the service interface definition. Usages have been
changed to refer to the new crate directly, and since this refactor is
still incomplete, some `mp` aliases are used in a few places to refer to
the old module.

* Create an `UnboxMempoolError` helper trait

Centralize some common code to extract and downcast boxed mempool
errors. The `mempool::Response` will need to contain a `BoxError`
instead of a `MempoolError` when it is moved to the
`zebra-node-services` crate, so this prepares the tests to be updated
with less changes.

* Use `UnboxMempoolError` in tests

Make the necessary changes so that the tests are ready to support a
`BoxError` in the `mempool::Response` type.

* Use `BoxError` in `mempool::Response`

Prepare it to be moved to the `zebra-node-services` crate.

* Move `mempool::Response` to `zebra-node-services`

Update usages to import from the new crate directly.

* Remove `mp` aliases for mempool component module

Use any internal types directly instead.

* Replace `tower::BoxService` with custom alias

Remove the dependency of `zebra-node-services` on `tower`.

* Move `Gossip` into a separate `sub-module`

Keep only the main `Request` and `Response` types in the `mempool`
module.

* Use `crate::BoxError` instead of `tower::BoxError`

Follow the existing convention.

* Add missing `gossip.rs` module file

It was missing from a previous refactor commit.
2022-02-25 21:43:21 +00:00
teor a4d1a1801c
Security: Drop blocks that are a long way ahead of the tip (#3167)
* Document the chain verifier

* Drop gossiped blocks that are too far ahead of the tip

* Add extra gossiped block metrics

* Allow extra gossiped blocks, now we have a stricter limit

* Fix a comment

* Check the exact number of blocks in a downloaded block response

* Drop synced blocks that are too far ahead of the tip

* Add extra synced block metrics

* Test dropping gossiped blocks that are too far ahead of the tip

* Allow an extra checkpoint's worth of blocks in the verifier queues

* Actually let's try two extra checkpoints

* Scale extra height limit with lookahead limit

* Also drop blocks that are behind the finalized tip

* Downgrade a noisy log

* Use a debug log for already verified gossiped blocks

* Use debug logs for already verified synced blocks
2021-12-17 13:31:51 -03:00
teor c85ea18b43
Fix slow Zebra startup times, to reduce CI failures (#3104)
* Tweak a log message

* Only retry failed DNS once, then use the other DNS responses

* Limit broadcasts to half the peers

* Use a longer minimum interval for GetAddr requests

* Reduce the syncer and mempool crawler fanouts

* Stop resetting the mempool twice when it starts up

This spawns two crawlers, which send two fanouts,
so it can use up a lot of peers.

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-11-30 21:04:32 +00:00
teor 375a997d2f
Stop downloading unnecessary blocks in Zebra acceptance tests (#3072)
* Implement graceful shutdown for the peer set

* Use the minimum lookahead limit in acceptance tests

* Enable a doctest that compiles with newly public modules
2021-11-19 01:55:38 +00:00
Dimitris Apostolou afb8b3d477
Fix typos (#3055)
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-11-12 19:30:22 +00:00
Deirdre Connolly da8be766dd
Mempool doc (#2978)
* Fix some links and linewraps

* Flesh out module-level rustdoc for mempool component

* Links for Downloads and Storage

* Tidy rustdoc links

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2021-11-01 14:32:48 +00:00
Deirdre Connolly 0381c2347b
ZIP-401: weighted random mempool eviction (#2889)
* ZIP-401 weighted random mempool eviction

* rename zcash.mempool.total_cost.bytes to zcash.mempool.cost.bytes

Co-authored-by: teor <teor@riseup.net>

* Remove duplicated lines

* Add cost() method to UnminedTx

Update serialization failure messages

* More docs quoting ZIP-401 rules

* Change mempool::Storage::new() to handle Copy-less HashMap, HashSet

* mempool: tidy cost types and evict_one()

* More consensus rule docs

* Refactor calculating mempool costs for Unmined transactions

* Add a note on asympotic performance of calculating weights of txs in mempool

* Bump test mempool / storage config to avoid weighted random cost limits

* Use mempool tx_cost_limit = u64::MAX for some tests

* Remove failing tests for now

* Allow(clippy::field-reassign-with-default) because of a move on a type that doesn't impl Copy

* Fix mistaken doctest formatting

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>

* Increase test timeout for Windows builds

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2021-10-27 00:21:19 +00:00
teor a166964a34
Document why CheckForVerifiedTransactions is required (#2955) 2021-10-26 10:19:12 -03:00
Conrado Gouvea a5d1467624
Additional mempool metrics (#2878)
* Rename tx downloader & verifier metrics

* Add version to mempool metrics

* Add new metrics

* Make sure mempool gauges are zeroed when instances are dropped

* Updated mempool grafana dashboard

* Removed transaction verification dashboard; moved to mempool

* Update mempool dashboard

* Add reason to error labels in mempool dashboard

* Rename some metrics per review

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-10-19 17:10:25 +00:00
teor 303c8cf5ef
Add a queue checker task, to make sure mempool transactions propagate (#2888)
* Guarantee unique IDs in mempool service responses

* Guarantee unique IDs in crawler task mempool Queue requests

Also update the tests to use unique IDs.

* Add a CheckForVerifiedTransactions mempool request

Also document the mempool request and response variants.

* Spawn a QueueChecker task to check for newly verified transactions

This task makes sure that transactions reliably propagate,
rather than relying on peer requests or responses to trigger propagation.

* Update the start command documentation

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-10-18 19:23:21 +00:00
teor 42ce79aad9
Cancel pending download tasks when the mempool is disabled (#2886)
* Impl Drop, Default and take() for ActiveState

* Refactor Mempool::poll_ready to check disabled and reset first

Also remove some levels of nesting.

* Use the same code for dropping and resetting the mempool

* Document where the tasks are dropped when switching states

* Log mempool resets at info level

And add heights to mempool enable/disable/reset logs

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2021-10-18 17:39:56 +00:00
teor 40c907dd09
Remove duplicate IDs in mempool requests and responses (#2887)
* Guarantee unique IDs in mempool service responses

* Guarantee unique IDs in crawler task mempool Queue requests

Also update the tests to use unique IDs.

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2021-10-18 15:31:11 +00:00
Deirdre Connolly 4648f8fc70
Make some mempool functions associated with the `mempool::Storage` type (#2872)
* Make some mempool functions associated with the Mempool type

* Move some functions to methods on mempool::Storage
2021-10-15 15:03:13 -03:00
teor a21dd26a5e
Insert new mempool transactions, then check for rejections (#2874)
Previously, we checked some rejections before inserting,
so we could accept some new transactions that should be rejected.
2021-10-14 10:19:21 -03:00
teor b64ed62777
Add a debug config that enables the mempool (#2862)
* Update some comments

* Add a mempool debug_enable_at_height config

* Rename a field in the mempool crawler

* Propagate syncer channel errors through the crawler

We don't want to ignore these errors, because they might indicate a shutdown.
(Or a bug that we should fix.)

* Use debug_enable_at_height in the mempool crawler

* Log when the mempool is activated or deactivated

* Deny unknown fields and apply defaults for all configs

* Move Duration last, as required for TOML tables

* Add a basic mempool acceptance test

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2021-10-13 15:04:49 +00:00
Alfredo Garcia 8120e8abac
Avoid broadcasting mempool rejected or expired transactions to peers (#2858)
* do not advertise rejected transactions

* do not broadcast transaction that are expired

* change dummy var name

* simplify code, performance

* clippy

* add some test coverage

* clippy

Co-authored-by: teor <teor@riseup.net>
2021-10-13 00:50:35 +00:00
teor b274ee4066
Pass the mempool config to the mempool (#2861)
* Split mempool config into its own module

Also:
- expand config docs
- clean up mempool imports

* Pass the mempool config to the mempool

* Create the transaction sender channel inside the mempool 1/2

This simplifies all the code that calls the mempool.

Also:
- update the mempool enabled state before returning the new mempool
- add some test module doc comments

* Refactor a setup function out of the mempool unit tests 2/2

Also:
- update the setup function to handle the latest mempool changes

* Clarify a comment

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2021-10-12 17:31:54 +00:00
Conrado Gouvea 31e7a21721
Add expired transactions to the mempool rejected list (#2852)
* Add expired transactions to the mempool rejected list

* Apply suggestions from code review

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Refactor contains_rejected() by just calling rejection_error()

* Improve rejection_error() documentation

* Improve rejected_transaction_count() documentation

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: teor <teor@riseup.net>
2021-10-11 21:23:43 +00:00
Janito Vaqueiro Ferreira Filho 9e78a8af40
Refactor mempool spend conflict checks to increase performance (#2826)
* Add `HashSet`s to help spend conflict detection

Keep track of the spent transparent outpoints and the revealed
nullifiers.

Clippy complained that the `ActiveState` had variants with large size
differences, but that was expected, so I disabled that lint on that
`enum`.

* Clear the `HashSet`s when clearing the mempool

Clear them so that they remain consistent with the set of verified
transactions.

* Use `HashSet`s to check for spend conflicts

Store new outputs into its respective `HashSet`, and abort if a
duplicate output is found.

* Remove inserted outputs when aborting

Restore the `HashSet` to its previous state.

* Remove tracked outputs when removing a transaction

Keep the mempool storage in a consistent state when a transaction is
removed.

* Remove tracked outputs when evicting from mempool

Ensure eviction also keeps the tracked outputs consistent with the
verified transactions.

* Refactor to create a `VerifiedSet` helper type

Move the code to handle the output caches into the new type. Also move
the eviction code to make things a little simpler.

* Refactor to have a single `remove` method

Centralize the code that handles the removal of a transaction to avoid
mistakes.

* Move mempool size limiting back to `Storage`

Because the evicted transactions must be added to the rejected list.

* Remove leftover `dbg!` statement

Leftover from some temporary testing code.

Co-authored-by: teor <teor@riseup.net>

* Remove unnecessary `TODO`

It is more speculation than planning, so it doesn't add much value.

Co-authored-by: teor <teor@riseup.net>

* Fix typo in documentation

The verb should match the subject "transactions" which is plural.

Co-authored-by: teor <teor@riseup.net>

* Add a comment to warn about correctness

There's a subtle but important detail in the implementation that should
be made more visible to avoid mistakes in the future.

Co-authored-by: teor <teor@riseup.net>

* Remove outdated comment

Left-over from the attempt to move the eviction into the `VerifiedSet`.

* Improve comment explaining lint removal

Rewrite the comment explaining why the Clippy lint was ignored.

* Check for spend conflicts in `VerifiedSet`

Refactor to avoid API misuse.

* Test rejected transaction rollback

Using two transactions, perform the same test adding a conflict to both
of them to check if the second inserted transaction is properly
rejected. Then remove any conflicts from the second transaction and add
it again. That should work, because if it doesn't it means that when the
second transaction was rejected it left things it shouldn't in the
cache.

* Test removal of multiple transactions

When removing multiple transactions from the mempool storage, all of the
ones requested should be removed and any other transaction should be
still be there afterwards.

* Increase mempool size to 4, so that spend conflict tests work

If the mempool size is smaller than 4,
these tests don't fail on a trivial removal bug.
Because we need a minimum number of transactions in the mempool
to trigger the bug.

Also commit a proptest seed that fails on a trivial removal bug.
(This seed fails if we remove indexes in order,
because every index past the first removes the wrong transaction.)

* Summarise transaction data in proptest error output

* Summarise spend conflict field data in proptest error output

* Summarise multiple removal field data in proptest error output

And replace the very large proptest debug output with the new summary.

Co-authored-by: teor <teor@riseup.net>
2021-10-10 23:54:46 +00:00
Alfredo Garcia 724967d488
Send `AdvertiseTransactionIds` to peers (#2823)
* bradcast transactions to peers after they get inserted into mempool

* remove network argument from mempool init

* remove dbg left

* remove return value in mempool enable call

* rename channel sender and receiver vars

* change unwrap() to expect()

* change the channel to a hashset

* fix build

* fix tests

* rustfmt

* fix tiny space issue inside macro

Co-authored-by: teor <teor@riseup.net>

* check errors/panics in transaction gossip tests

* fix build of newly added tests

* Stop dropping the inbound service and mempool in a test

Keeping the mempool around avoids a transaction broadcast task error,
so we can test that there are no other errors in the task.

* Tweak variable names and add comments

* Avoid unexpected drops by returning a mempool guard in tests

* Use BoxError to simplify service types in tests

* Make all returned service types consistent in tests

We want to be able to change the setup without changing the tests.

Co-authored-by: teor <teor@riseup.net>
2021-10-08 08:59:46 -03:00
Alfredo Garcia 0683e0b40b
Add a mempool config section (#2845)
* add mempool config section

* change to Duration

* fix typo in var name

* document consensus rules

* tweak var name
2021-10-07 22:47:37 +00:00
Conrado Gouvea dd1f0a6dcc
Add transactions that failed verification to the mempool rejected list (#2821)
* Add transactions that failed verification to the mempool rejected list

* Add tests

* Work with recent changes
2021-10-07 21:34:01 +00:00
teor 664d4384d4
Un-reject mempool transactions if the rejection depends on the current tip (#2844)
* Split mempool storage errors into tip-based and chain-based

* Expire tip rejections every time we get a new block

FailedVerification and SpendConflict rejections only apply to the current tip.
The next tip can provide missing inputs, or evict conflicting transactions.

* Enforce MAX_EVICTION_MEMORY_ENTRIES for mempool reject lists
2021-10-07 16:58:42 -03:00
teor b6a60c6c17 Fix names that are exact or effects depending on the list 2021-10-07 13:43:53 -04:00
teor e470ed00e6 Return rejected transaction ID iterators from mempool storage
Also rename methods for consistency.
2021-10-07 13:43:53 -04:00
teor 3357c58c41 Return transaction iterators from mempool storage
Also rename transaction methods for consistency.
2021-10-07 13:43:53 -04:00
teor 964c819c80
Split storage errors into same effects and exact matches (#2833)
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2021-10-07 15:04:23 +00:00
teor a3a4773047
Remove unused mempool errors (#2831)
* Remove unused mempool storage errors

Preparation for ticket #2819.

Removing these errors means that we don't have to decide
which type of transaction ID match we want for them.

* Remove unused mempool errors, and deduplicate storage errors

* rustfmt
2021-10-07 11:20:38 +10:00
teor d1ce8e3e6d
Remove transactions in newly committed blocks from the mempool (#2827)
* Add a mempool transaction removal method for mined IDs

And use this method to remove expired transactions,
because all transactions with the same mined ID expire at the same height.

* Remove mined transaction IDs from the mempool

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2021-10-07 09:45:14 +10:00
Janito Vaqueiro Ferreira Filho 5d9893cf31
Send crawled transaction IDs to downloader (#2801)
* Rename type parameter to be more explicit

Replace the single letter with a proper name.

* Remove imports for `Request` and `Response`

The type names will conflict with the ones for the mempool service.

* Attach `Mempool` service to the `Crawler`

Add a field to the `Crawler` type to store a way to access the `Mempool`
service.

* Forward crawled transactions to downloader

The crawled transactions are now sent to the transaction downloader and
verifier, to be included in the mempool.

* Derive `Eq` and `PartialEq` for `mempool::Request`

Make it simpler to use the `MockService::expect_request` method.

* Test if crawled transactions are downloaded

Create some dummy crawled transactions, and let the crawler discover
them. Then check if they are forwarded to the mempool to be downloaded
and verified.

* Don't send empty transaction ID list to downloader

Ignore response from peers that don't provide any crawled transactions.

* Log errors when forwarding crawled transaction IDs

Calling the Mempool service should not fail, so if an error happens it
should be visible. However, errors when downloading individual
transactions can happen from time to time, so there's no need for them
to be very visible.

* Document existing `mempool::Crawler` test

Provide some depth as to what the test expect from the crawler's
behavior.

* Refactor to create `setup_crawler` helper function

Make it easier to reuse the common test setup code.

* Simplify code to expect requests

Now that `zebra_network::Request` implement `Eq`, the call can be
simplified into `expect_request`.

* Refactor to create `respond_with_transaction_ids`

A helper function that checks for a network crawl request and responds
with the given list of crawled transaction IDs.

* Refactor to create `crawler_iterator` helper

A function to intercept and respond to the fanned-out requests sent
during a single crawl iteration.

* Refactor to create `respond_to_queue_request`

Reduce the repeated code necessary to intercept and reply to a request
for queuing transactions to be downloaded.

* Add `respond_to_queue_request_with_error` helper

Intercepts a mempool request to queue transactions to be downloaded, and
responds with an error, simulating an internal problem in the mempool
service implementation.

* Derive `Arbitrary` for `NetworkUpgrade`

This is required for deriving `Arbitrary` for some error types.

* Derive `Arbitrary` for `TransactionError`

Allow random transaction errors to be generated for property tests.

* Derive `Arbitrary` for `MempoolError`

Allow random Mempool errors to be generated for property tests.

* Test if errors don't stop the mempool crawler

The crawler should be robust enough to continue operating even if the
mempool service fails to download transactions or even fails to handle
requests to enqueue transactions.

* Reduce the log level for download errors

They should happen regularly, so there's no need to have them with a
high visibility level.

Co-authored-by: teor <teor@riseup.net>

* Stop crawler if service stops

If `Mempool::poll_ready` returns an error, it's because the mempool
service has stopped and can't handle any requests, so the crawler should
stop as well.

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2021-10-05 10:55:42 +10:00