Zebra

Commit Graph

Author	SHA1	Message	Date
Marek	abcabd1931	Use `OrderedUtxo` in `CheckpointVerifiedBlock` (#6971 )	2023-06-19 22:48:59 +00:00
Alfredo Garcia	8a7c871480	rename(state): do additional renaming for clarification purposes (#6967 ) * do renames by script ``` git ls-tree --full-tree -r --name-only HEAD \| \ grep -v 'book' \| grep -v 'CHANGELOG.md' \| \ xargs sed -i 's/CommitFinalized/CommitCheckpointVerified/g' git ls-tree --full-tree -r --name-only HEAD \| \ grep -v 'book' \| grep -v 'CHANGELOG.md' \| \ xargs sed -i 's/commit_finalized_block/commit_checkpoint_verified/g' git ls-tree --full-tree -r --name-only HEAD \| \ grep -v 'book' \| grep -v 'CHANGELOG.md' \| \ xargs sed -i 's/drain_queue_and_commit_finalized/drain_finalized_queue_and_commit/g' git ls-tree --full-tree -r --name-only HEAD \| \ grep -v 'book' \| grep -v 'CHANGELOG.md' \| \ xargs sed -i 's/queue_and_commit_finalized/queue_and_commit_to_finalized_state/g' git ls-tree --full-tree -r --name-only HEAD \| \ grep -v 'book' \| grep -v 'CHANGELOG.md' \| \ xargs sed -i 's/queued_finalized_blocks/finalized_state_queued_blocks/g' git ls-tree --full-tree -r --name-only HEAD \| \ grep -v 'book' \| grep -v 'CHANGELOG.md' \| \ xargs sed -i 's/max_queued_finalized_height/max_finalized_queue_height/g' git ls-tree --full-tree -r --name-only HEAD \| \ grep -v 'book' \| grep -v 'CHANGELOG.md' \| \ xargs sed -i 's/send_finalized_block_error/send_checkpoint_verified_block_error/g' git ls-tree --full-tree -r --name-only HEAD \| \ grep -v 'book' \| grep -v 'CHANGELOG.md' \| \ xargs sed -i 's/QueuedFinalized/QueuedCheckpointVerified/g' git ls-tree --full-tree -r --name-only HEAD \| \ grep -v 'book' \| grep -v 'CHANGELOG.md' \| \ xargs sed -i 's/queue_and_commit_non_finalized/queue_and_commit_to_non_finalized_state/g' git ls-tree --full-tree -r --name-only HEAD \| \ grep -v 'book' \| grep -v 'CHANGELOG.md' \| \ xargs sed -i 's/queued_non_finalized_blocks/non_finalized_state_queued_blocks/g' git ls-tree --full-tree -r --name-only HEAD \| \ grep -v 'book' \| grep -v 'CHANGELOG.md' \| \ xargs sed -i 's/send_non_finalized_block_error/send_semantically_verified_block_error/g' git ls-tree --full-tree -r --name-only HEAD \| \ grep -v 'book' \| grep -v 'CHANGELOG.md' \| \ xargs sed -i 's/QueuedNonFinalized/QueuedSemanticallyVerified/g' git ls-tree --full-tree -r --name-only HEAD \| \ grep -v 'book' \| grep -v 'CHANGELOG.md' \| \ xargs sed -i 's/last_sent_finalized_block_hash/finalized_block_write_last_sent_hash/g' git ls-tree --full-tree -r --name-only HEAD \| \ grep -v 'book' \| grep -v 'CHANGELOG.md' \| \ xargs sed -i 's/sent_non_finalized_block_hashes/non_finalized_block_write_sent_hashes/g' git ls-tree --full-tree -r --name-only HEAD \| \ grep -v 'book' \| grep -v 'CHANGELOG.md' \| \ xargs sed -i 's/invalid_block_reset_receiver/invalid_block_write_reset_receiver/g' cargo fmt --all ``` * add missing log renames by script ``` git ls-tree --full-tree -r --name-only HEAD \| \ grep -v 'book' \| grep -v 'CHANGELOG.md' \| \ xargs sed -i 's/queued finalized block/finalized state queue block/g' git ls-tree --full-tree -r --name-only HEAD \| \ grep -v 'book' \| grep -v 'CHANGELOG.md' \| \ xargs sed -i 's/queued non-finalized block/non-finalized state queue block/g' cargo fmt --all ```	2023-06-15 05:12:45 +00:00
Alfredo Garcia	eb07bb31d6	rename(state): Rename state verifiers and related code (#6762 ) * rename verifiers * rename `PreparedBlock` to `SemanticallyVerifiedBlock` * rename `CommitBlock` to `SemanticallyVerifiedBlock` * rename `FinalizedBlock` to `CheckpointVerifiedBlock` * rename `CommitFinalizedBlock` to `CommitCheckpointVerifiedBlock` * rename `FinalizedWithTrees` to `ContextuallyVerifiedBlockWithTrees` * rename `ContextuallyValidBlock` to `ContextuallyVerifiedBlock` * change some `finalized` variables or function arguments to `checkpoint_verified` * fix docs * document the difference between `CheckpointVerifiedBlock` and `ContextuallyVerifiedBlock` * fix doc links * apply suggestions to request Co-authored-by: Marek <mail@marek.onl> * apply suggestions to service Co-authored-by: Marek <mail@marek.onl> * apply suggestions to finalized_state.rs and write.rs Co-authored-by: Marek <mail@marek.onl> * fmt * change some more variable names * change a few missing generics * fix checkpoint log issue * rename more `prepared` vars `semantically_verified` * fix test regex * fix test regex 2 --------- Co-authored-by: Marek <mail@marek.onl>	2023-06-01 12:29:03 +00:00
Arya	571cbfba7a	change(state): Stop re-downloading blocks that are in non-finalized side chains (#6335 ) * Adds 'Contains' request in state, and: - adds finalized block hashes to sent_blocks - replaces Depth call with Contains in sync, inbound, and block verifier * removes unnecessary From impl * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * updates references to Request::Contains * Renames zs::response::BlockLocation to KnownBlocks * Updates AlreadyInChain error * update docs for sent_hashes.add_finalized * Update zebra-consensus/src/block.rs Co-authored-by: teor <teor@riseup.net> * Update comment for `sent_blocks` field in state service * update KnownBlock request to check the non-finalized state before responding that a block is in the queue * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * update references to renamed method * Clear sent_blocks when there's a reset * Move self.finalized_block_write_sender.is_none() to can_fork_chain_at * revert changes related to checking queue --------- Co-authored-by: teor <teor@riseup.net>	2023-03-24 07:10:07 +00:00
teor	f31609e411	change(state): Add block channel metrics, in preparation for block fork metrics (#5327 ) * Add minimum queued block height to metrics * Add metrics to sent block cache * Add metrics for the last sent finalized height * Refactor sent metrics * Add new block channel metrics to a grafana dashboard	2022-10-20 20:02:58 +00:00
Arya	a28350e742	change(state): Write non-finalized blocks to the state in a separate thread, to avoid network and RPC hangs (#5257 ) * Add a new block commit task and channels, that don't do anything yet * Add last_block_hash_sent to the state service, to avoid database accesses * Update last_block_hash_sent regardless of commit errors * Rename a field to StateService.max_queued_finalized_height * Commit finalized blocks to the state in a separate task * Check for panics in the block write task * Wait for the block commit task in tests, and check for errors * Always run a proptest that sleeps once * Add extra debugging to state shutdowns * Work around a RocksDB shutdown bug * Close the finalized block channel when we're finished with it * Only reset state queue once per error * Update some TODOs * Add a module doc comment * Drop channels and check for closed channels in the block commit task * Close state channels and tasks on drop * Remove some duplicate fields across StateService and ReadStateService * Try tweaking the shutdown steps * Update and clarify some comments * Clarify another comment * Don't try to cancel RocksDB background work on drop * Fix up some comments * Remove some duplicate code * Remove redundant workarounds for shutdown issues * Remode a redundant channel close in the block commit task * Remove a mistaken `!force` shutdown condition * Remove duplicate force-shutdown code and explain it better * Improve RPC error logging * Wait for chain tip updates in the RPC tests * Wait 2 seconds for chain tip updates before skipping them * Remove an unnecessary block_in_place() * Fix some test error messages that were changed by earlier fixes * Expand some comments, fix typos Co-authored-by: Marek <mail@marek.onl> * Actually drop children of failed blocks * Explain why we drop descendants of failed blocks * Clarify a comment * Wait for chain tip updates in a failing test on macOS * Clean duplicate finalized blocks when the non-finalized state activates * Send an error when receiving a duplicate finalized block * Update checkpoint block behaviour, document its consensus rule * Wait for chain tip changes in inbound_block_height_lookahead_limit test * Wait for the genesis block to commit in the fake peer set mempool tests * Disable unreliable mempool verification check in the send transaction test * Appease rustfmt * Use clear_finalized_block_queue() everywhere that blocks are dropped * Document how Finalized and NonFinalized clones are different * sends non-finalized blocks to the block write task * passes ZebraDb to commit_new_chain, commit_block, and no_duplicates_in_finalized_chain instead of FinalizedState * Update zebra-state/src/service/write.rs Co-authored-by: teor <teor@riseup.net> * updates comments, renames send_process_queued, other minor cleanup * update assert_block_can_be_validated comment * removes `mem` field from StateService * removes `disk` field from StateService and updates block_iter to use `ZebraDb` instead of the finalized state * updates tests that use the disk to use read_service.db instead * moves best_tip to a read fn and returns finalized & non-finalized states from setup instead of the state service * changes `contextual_validity` to get the network from the finalized_state instead of another param * swaps out StateService with FinalizedState and NonFinalizedState in tests * adds NotReadyToBeCommitted error and returns it from validate_and_commit when a blocks parent hash is not in any chain * removes NonFinalizedWriteCmd and calls, moves update_latest_channels above rsp_tx.send * makes parent_errors_map an indexmap * clears non-finalized block queue when the receiver is dropped and when the StateService is being dropped * sends non-finalized blocks to the block write task * passes ZebraDb to commit_new_chain, commit_block, and no_duplicates_in_finalized_chain instead of FinalizedState * updates comments, renames send_process_queued, other minor cleanup * Update zebra-state/src/service/write.rs Co-authored-by: teor <teor@riseup.net> * update assert_block_can_be_validated comment * removes `mem` field from StateService * removes `disk` field from StateService and updates block_iter to use `ZebraDb` instead of the finalized state * updates tests that use the disk to use read_service.db instead * moves best_tip to a read fn and returns finalized & non-finalized states from setup instead of the state service * changes `contextual_validity` to get the network from the finalized_state instead of another param * swaps out StateService with FinalizedState and NonFinalizedState in tests * adds NotReadyToBeCommitted error and returns it from validate_and_commit when a blocks parent hash is not in any chain * removes NonFinalizedWriteCmd and calls, moves update_latest_channels above rsp_tx.send * makes parent_errors_map an indexmap * clears non-finalized block queue when the receiver is dropped and when the StateService is being dropped * removes duplicate field definitions on StateService that were a result of a bad merge * update NotReadyToBeCommitted error message * Appear rustfmt * Fix doc links * Rename a function to initial_contextual_validity() * Do error tasks on Err, and success tasks on Ok * Simplify parent_error_map truncation * Rewrite best_tip() to use tip() * Rename latest_mem() to latest_non_finalized_state() ```sh fastmod latest_mem latest_non_finalized_state zebra* cargo fmt --all ``` * Simplify latest_non_finalized_state() using a new WatchReceiver API * Expand some error messages * Send the result after updating the channels, and document why * wait for chain_tip_update before cancelling download in mempool_cancel_mined * adds `sent_non_finalized_block_hashes` field to StateService * adds batched sent_hash insertions and checks sent hashes in queue_and_commit_non_finalized before adding a block to the queue * check that the `curr_buf` in SentHashes is not empty before pushing it to the `sent_bufs` * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Fix rustfmt * Check for finalized block heights using zs_contains() * adds known_utxos field to SentHashes * updates comment on SentHashes.add method * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * return early when there's a duplicate hash in QueuedBlocks.queue instead of panicking * Make finalized UTXOs near the final checkpoint available for full block verification * Replace a checkpoint height literal with the actual config * Update mainnet and testnet checkpoints - 7 October 2022 * Fix some state service init arguments * Allow more lookahead in the downloader, but less lookahead in the syncer * Add the latest config to the tests, and fix the latest config check * Increase the number of finalized blocks checked for non-finalized block UTXO spends * fix(log): reduce verbose logs for block commits (#5348) * Remove some verbose block write channel logs * Only warn about tracing endpoint if the address is actually set * Use CloneError instead of formatting a non-cloneable error Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Increase block verify timeout * Work around a known block timeout bug by using a shorter timeout Co-authored-by: teor <teor@riseup.net> Co-authored-by: Marek <mail@marek.onl> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>	2022-10-11 19:25:45 +00:00
teor	343c5e68d4	change(state): Write finalized blocks to the state in a separate thread, to avoid network and RPC hangs (#5134 ) * Add a new block commit task and channels, that don't do anything yet * Add last_block_hash_sent to the state service, to avoid database accesses * Update last_block_hash_sent regardless of commit errors * Rename a field to StateService.max_queued_finalized_height * Commit finalized blocks to the state in a separate task * Check for panics in the block write task * Wait for the block commit task in tests, and check for errors * Always run a proptest that sleeps once * Add extra debugging to state shutdowns * Work around a RocksDB shutdown bug * Close the finalized block channel when we're finished with it * Only reset state queue once per error * Update some TODOs * Add a module doc comment * Drop channels and check for closed channels in the block commit task * Close state channels and tasks on drop * Remove some duplicate fields across StateService and ReadStateService * Try tweaking the shutdown steps * Update and clarify some comments * Clarify another comment * Don't try to cancel RocksDB background work on drop * Fix up some comments * Remove some duplicate code * Remove redundant workarounds for shutdown issues * Remode a redundant channel close in the block commit task * Remove a mistaken `!force` shutdown condition * Remove duplicate force-shutdown code and explain it better * Improve RPC error logging * Wait for chain tip updates in the RPC tests * Wait 2 seconds for chain tip updates before skipping them * Remove an unnecessary block_in_place() * Fix some test error messages that were changed by earlier fixes * Expand some comments, fix typos Co-authored-by: Marek <mail@marek.onl> * Actually drop children of failed blocks * Explain why we drop descendants of failed blocks * Clarify a comment * Wait for chain tip updates in a failing test on macOS * Clean duplicate finalized blocks when the non-finalized state activates * Send an error when receiving a duplicate finalized block * Update checkpoint block behaviour, document its consensus rule * Wait for chain tip changes in inbound_block_height_lookahead_limit test * Wait for the genesis block to commit in the fake peer set mempool tests * Disable unreliable mempool verification check in the send transaction test * Appease rustfmt * Use clear_finalized_block_queue() everywhere that blocks are dropped * Document how Finalized and NonFinalized clones are different * Use the same check as commit_finalized() for finalized block heights Co-authored-by: Marek <mail@marek.onl> Co-authored-by: Marek <mail@marek.onl>	2022-09-28 16:09:56 +00:00
teor	bfdb29b757	3. change(state): Move the finalized queue to the StateService (#5152 ) * Move the finalized block queue into the StateService * Move the queued_blocks module to the state service * Move QueuedFinalized into queued_blocks * Move the queued_blocks tests into their own module * Make the FinalizedState cloneable	2022-09-16 13:53:40 +00:00

8 Commits