* Stub `sendrawtransaction` RPC method
Register the RPC method, and stub an implementation that currently just
panics. The method has a single `String` parameter with the hexadecimal
string of the raw transaction's bytes and returns a
`SentTransactionHash` wrapper type that's just a hexadecimal `String` of
the sent transaction's hash.
* Add mempool service instance to `RpcImpl`
Use a type parameter to represent the mempool service using the
interface defined by `zebra-node-services`.
* Update test vector to use a mock mempool service
Update the test to be compatible with the changes to `RpcImpl`. The mock
mempool service is expected to not be used during the test.
* Use a `tower::Buffer` for the mempool service
Make it simpler to send requests to the service in a concurrent manner.
* Return a `Future` from `send_raw_transaction`
Make the call asynchronous.
* Implement `sendrawtransaction` RPC
Deserialize the transaction and send it to be queued for verification
and subsequent inclusion in the mempool.
* Test if mempool receives sent raw transaction
Use a mock service as the mempool service and check that it receives a
sent raw transaction.
* Test using non-hexadecimal string parameter
The method should return an error.
* Test with bytes that fail deserialization
Check that the method returns an invalid parameters error if the input
can't be deserialized as a `Transaction`.
* Test if mempool errors are forwarded to caller
Mempool service errors should be sent back to the remote caller as
server errors.
* Test transactions rejected by the mempool service
Transactions that are rejected by the mempool service should result in
a server error being sent to the caller.
* Improve error message
Add the word "structurally" to make it clear that the issue is in the
transaction's deserialization.
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* Add note regarding missing `allowhighfees` param.
The parameter isn't supported yet because `lightwalletd` doesn't use it.
* Update the documentation to be consistent
Follow the convention adopted by the `get_info` RPC method.
* Implement `ToHex` and `FromHex` for `Hash`
Make it easier to generate hexadecimal strings from `transaction::Hash`
instances.
* Use `ToHex` in `Debug` and `Display`
Reduce repeated code.
* Refactor to add `bytes_in_display_order` method
Use it to remove repeated code and improve clarity a bit.
* Use `hex::serialize` to serialize transaction hash
Make the type stricter in its contents, while still serializing the
transaction has as a hexadecimal string.
* Simplify serialization attribute
Deserialization should also use `hex::deserialize`, so using the shorter
attribute makes things easier to read and more future proof.
* Update zebra-chain/src/transaction/hash.rs
* Remove unnecessary lifetime
The anonymous lifetime is automatically inferred by the compiler.
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* docs: document consensus rules from 4.6 Action Descriptions
* Apply suggestions from code review
Co-authored-by: Marek <mail@marek.onl>
* Apply suggestions from code review
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Marek <mail@marek.onl>
* change `anchorSapling` type
* implement PartialEq manually for clippy
* use `unique_by` in place of `sorted`
* replace panic with new error
* improve some serialize/deserialize calls for sapling anchors
* fix arbitrary for sapling::tree::Root
* remove dedup()
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Support large block heights
* Document consensus rules referring to expiry heights
* Refactor the docs
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Fix the formatting of an error message
* refactor: Simplify coinbase expiry code so the consensus rule is clear (#3408)
* Fix some outdated TODO comments
* refactor(coinbase expiry): Simplify the code so consensus rule is clear
* Fix the formatting of an error message
* Remove a redundant comment
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Marek <mail@marek.onl>
* Check the max expiry height at parse time
* Test that 2^31 - 1 is the last valid height
* Add tests for nExpiryHeight
* Add tests for expiry heights of V4 transactions
* Add tests for V5 transactions
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* document the `header` field
* document the `nVersionGroupId` field
* document the `nConsensusBranchId` field
* document the `lock_time` field
* document the `nExpiryHeight` field (and some missing `lock_time`)
* add missing note to `header` field in serialization
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Add all_previous_outputs; load UTXOs in transaction verifier
* Remove UTXO loading and returning from script.rs
* Don't pass state service to script verifier
* Remove output from is_valid()
* Refactor loading UTXOs to separate function
* Pass all_previous_output to sighash
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Create AwaitUtxo only when needed; formatting
* Add comments about output vectors in tests
* Change sighash() to receive reference and avoid cloning
* Expand comments
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Ensure that sapling::keys::TransmissionKey jubjub point is always in the prime order group
* Adjust TransmissionKey check; add AuthorizingKey check
* Move ValueCommitment small order check to deserialization
* Apply suggestions from code review
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Use is_torsion_free() instead of is_identity() and is_prime_order()
* Add EphemeralPublicKey small order check on instantiation; remove old checks
* Use VerificationKey instead of VerificationKeyBytes; fix tests
* Use ValidatingKey instead of VerificationKeyBytes for rk
* Reject identity when creating an Orchard EphemeralPublicKey
* Make documentation more consistent, use generator in tests
* s/JubJub/Jubjub/
* Fix zebra-consensus tests (insert_fake_orchard_shielded_data)
* Create NotSmallOrderValueCommitment, since intermediate values can be the identity
* Clarify documentation
* rustdoc
* Tidy rustdoc
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: Pili Guerra <mpguerra@users.noreply.github.com>
* Create a `LockTime::unlocked` helper constructor
Returns a `LockTime` that is unlocked at the genesis block.
* Return `Option<LockTime>` from `lock_time` method
Prepare to return `None` for when a transaction has its lock time
disabled.
* Return `None` instead of zero `LockTime`
Because a zero lock time means that the transaction was unlocked at the
genesis block, so it was never actually locked.
* Rephrase zero lock time check comment
Clarify that the check is not redundant, and is necessary for the
genesis transaction.
Co-authored-by: teor <teor@riseup.net>
* Add a `transparent::Input::sequence` getter method
Retrieve a transparent input's sequence number.
* Check if lock time is enabled by a sequence number
Validate the consensus rule that the lock time is only enabled if at
least one transparent input has a value different from `u32::MAX` as its
sequence number.
* Add more Zcash specific details to comment
Explain the Zcash specific lock time behaviors.
Co-authored-by: teor <teor@riseup.net>
* Add `time` field to `Request::Block` variant
The block time to use to check if the transaction was unlocked and
allowed to be included in the block.
* Add `Request::block_time` getter
Returns the block time for the block that owns the transaction being
validated or the current time plus a tolerance for mempool transactions.
* Validate transaction lock times
If they are enabled by a transaction's transparent input sequence
numbers, make sure that they are in the past.
* Add comments with consensus rule parts
Make it easier to map what part of the consensus rule each match arm is
responsible for.
Co-authored-by: teor <teor@riseup.net>
* add testnet test blocks around nu5
* validate coinbase expiration height
* change const name and doc
Co-authored-by: teor <teor@riseup.net>
* change commit location
Co-authored-by: teor <teor@riseup.net>
* use pre Nu5 rules when there is no activation height
* add sapling final root to nu5 test vectors
* fix tests
Co-authored-by: teor <teor@riseup.net>
* Implement incremental note commitment Merkle tree for Sprout
* Add tests for Sprout note commitment tree
* Remove the `Arbitrary` attribute
* Reverse the vector of empty roots
* Add more tests
* Refactor rustdoc
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Refactor rustdoc
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* rustdoc
* Rustdoc
* rustdoc links
* Oops, need the trait in scope to use it
* Avoid accessing the wrapped hash directly
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* rustfmt
* Add typing
* Avoid accessing the wrapped hash directly
* Implement incremental note commitment Merkle tree for Sprout
* Add tests for Sprout note commitment tree
* Remove the `Arbitrary` attribute
* Reverse the vector of empty roots
* Add more tests
* Refactor rustdoc
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Refactor rustdoc
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* rustdoc
* Rustdoc
* rustdoc links
* Oops, need the trait in scope to use it
* Avoid accessing the wrapped hash directly
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* rustfmt
* Add typing
* Avoid accessing the wrapped hash directly
* Add Overwinter final roots (test vectors)
* Test sprout note commitments trees on Overwinter blocks
* Add new test vectors
* Finish the tests for the note commitment trees
* Make the wrapped hash in `Root` private
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* Revert "Remove commented-out code"
This reverts commit 9e69777925f103ee11e5940bba95b896c828839b.
* Implement deserialization for `addrv2` messages
* Limit addr and addrv2 messages to MAX_ADDRS_IN_MESSAGE
* Clarify address version comments
* Minor cleanups and fixes
* Add preallocation tests for AddrV2
* Add serialization tests for AddrV2
* Use prop_assert in AddrV2 proptests
* Use a generic utility method for deserializing IP addresses in `addrv2`
* Document the purpose of a conversion to MetaAddr
* Fix a comment typo, and clarify that comment
* Clarify the unsupported AddrV2 network ID error and enum variant names
```sh
fastmod AddrV2UnimplementedError UnsupportedAddrV2NetworkIdError zebra-network
fastmod Unimplemented Unsupported zebra-network
```
* Fix and clarify unsupported AddrV2 comments
* Replace `panic!` with `unreachable!`
* Clarify a comment about skipping a length check in a test
* Remove a redundant test
* Basic addr (v1) and addrv2 deserialization tests
* Test deserialized IPv4 and IPv6 values in addr messages
* Remove redundant io::Cursor
* Add comments with expected values of address test vectors
* ZIP-401 weighted random mempool eviction
* rename zcash.mempool.total_cost.bytes to zcash.mempool.cost.bytes
Co-authored-by: teor <teor@riseup.net>
* Remove duplicated lines
* Add cost() method to UnminedTx
Update serialization failure messages
* More docs quoting ZIP-401 rules
* Change mempool::Storage::new() to handle Copy-less HashMap, HashSet
* mempool: tidy cost types and evict_one()
* More consensus rule docs
* Refactor calculating mempool costs for Unmined transactions
* Add a note on asympotic performance of calculating weights of txs in mempool
* Bump test mempool / storage config to avoid weighted random cost limits
* Use mempool tx_cost_limit = u64::MAX for some tests
* Remove failing tests for now
* Allow(clippy::field-reassign-with-default) because of a move on a type that doesn't impl Copy
* Fix mistaken doctest formatting
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Increase test timeout for Windows builds
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Limit tx size
Zebra now limits the transaction size in the `zcash_deserialize()` method for
`Transaction`.
* Remove unused error variants (#2941)
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Limit tx size
Zebra now limits the transaction size in the `zcash_deserialize()` method for
`Transaction`.
* Test the tx deserialization limit
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Create a new VerifiedUnminedTx containing the miner fee
* Use VerifiedUnminedTx in mempool verification responses
And do a bunch of other cleanups.
* Use VerifiedUnminedTx in mempool download and verifier
* Use VerifiedUnminedTx in mempool storage and verified set
* Impl Display for VerifiedUnminedTx, and some convenience methods
* Use VerifiedUnminedTx in existing tests
* Limit the size of rejection lists when there is a spend conflict
Previously, `insert` would return early with an error,
and skip limiting the rejection list sizes.
* Use prop_assert macros in proptests, rather than assert
* Add `HashSet`s to help spend conflict detection
Keep track of the spent transparent outpoints and the revealed
nullifiers.
Clippy complained that the `ActiveState` had variants with large size
differences, but that was expected, so I disabled that lint on that
`enum`.
* Clear the `HashSet`s when clearing the mempool
Clear them so that they remain consistent with the set of verified
transactions.
* Use `HashSet`s to check for spend conflicts
Store new outputs into its respective `HashSet`, and abort if a
duplicate output is found.
* Remove inserted outputs when aborting
Restore the `HashSet` to its previous state.
* Remove tracked outputs when removing a transaction
Keep the mempool storage in a consistent state when a transaction is
removed.
* Remove tracked outputs when evicting from mempool
Ensure eviction also keeps the tracked outputs consistent with the
verified transactions.
* Refactor to create a `VerifiedSet` helper type
Move the code to handle the output caches into the new type. Also move
the eviction code to make things a little simpler.
* Refactor to have a single `remove` method
Centralize the code that handles the removal of a transaction to avoid
mistakes.
* Move mempool size limiting back to `Storage`
Because the evicted transactions must be added to the rejected list.
* Remove leftover `dbg!` statement
Leftover from some temporary testing code.
Co-authored-by: teor <teor@riseup.net>
* Remove unnecessary `TODO`
It is more speculation than planning, so it doesn't add much value.
Co-authored-by: teor <teor@riseup.net>
* Fix typo in documentation
The verb should match the subject "transactions" which is plural.
Co-authored-by: teor <teor@riseup.net>
* Add a comment to warn about correctness
There's a subtle but important detail in the implementation that should
be made more visible to avoid mistakes in the future.
Co-authored-by: teor <teor@riseup.net>
* Remove outdated comment
Left-over from the attempt to move the eviction into the `VerifiedSet`.
* Improve comment explaining lint removal
Rewrite the comment explaining why the Clippy lint was ignored.
* Check for spend conflicts in `VerifiedSet`
Refactor to avoid API misuse.
* Test rejected transaction rollback
Using two transactions, perform the same test adding a conflict to both
of them to check if the second inserted transaction is properly
rejected. Then remove any conflicts from the second transaction and add
it again. That should work, because if it doesn't it means that when the
second transaction was rejected it left things it shouldn't in the
cache.
* Test removal of multiple transactions
When removing multiple transactions from the mempool storage, all of the
ones requested should be removed and any other transaction should be
still be there afterwards.
* Increase mempool size to 4, so that spend conflict tests work
If the mempool size is smaller than 4,
these tests don't fail on a trivial removal bug.
Because we need a minimum number of transactions in the mempool
to trigger the bug.
Also commit a proptest seed that fails on a trivial removal bug.
(This seed fails if we remove indexes in order,
because every index past the first removes the wrong transaction.)
* Summarise transaction data in proptest error output
* Summarise spend conflict field data in proptest error output
* Summarise multiple removal field data in proptest error output
And replace the very large proptest debug output with the new summary.
Co-authored-by: teor <teor@riseup.net>
* Add `Arc<Transaction>` conversions for Transaction IDs
* Use UnminedTxId as the transaction verifier response type
* Use UnminedTx in transaction verifier mempool requests
* Refactor is_mempool into a transaction verifier request method
* Order derives in alphabetical order
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* add value balances to non finalized state
* fix 2 tests
* fix remaining constrain issues in tests
* extend value pool test to non finalized
* WIP: fix tests after adding value pools to non-finalized state (#2647)
* Update Chain::eq_internal_state with Chain.value_balance
Also increase the number of cases in its tests,
because they didn't detect this bug.
* Calculate the chain value pool change before `Chain::push`
Code
- store the chain value pool change in `ContextuallyValidBlock`
- convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos`
(rather than `from` or `into`)
- replace `block_utxos` with `new_outputs` in `PreparedBlock`
- replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock`
Tests
- create test methods for `PreparedBlock` and `ContextuallyValidBlock`
- use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos`
to make tests pass
* fix conflicts
* build `set_current_value_pool()` only for tests
* remove redundant cfgs
* change cfg of set_current_value_pool()
* Clarify some chain field documentation
* Fix bugs in the non-finalized chain value pool calculations
1. Only revert the chain value pool balances when the tip is popped.
Don't modify them when the root is finalized.
2. Only update or revert the chain value pool balances once per block.
(Previously, the block changes were multiplied by the number of *transactions*.)
And make corresponding changes to method names and documentation.
* Add extra proptests to try to identify value balance failures
* Simplify some transaction generation code
* Add extra debugging info to value balance errors
* Actually update non-finalized chain value pools in `UpdateWith`
Previously, we were dropping the updated value pools in the `Ok` result.
So the initial (finalized) chain value pool balances were never modified.
* Rename and document value balance add methods
The names and documentation of these methods were confusing.
* Create genesis-based proptests that check chain value pools
* Increase coverage for some test vectors
* Test each chain value balance calculation for blocks 0-10
* Make continuous blockchain test errors easier to debug
* Test the exact transparent pool values for the first few blocks
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Rename internal network requests for wide transaction IDs
fastmod TransactionsByHash TransactionsById zebra*
fastmod AdvertiseTransactions AdvertiseTransactionIds zebra*
fastmod MempoolTransactions MempoolTransactionIds zebra*
fastmod TransactionHashes TransactionIds zebra*
* Update network transaction request/response comments
* Rename a transaction hash method for wide transaction IDs
fastmod transaction_hashes transaction_ids zebra-network
* Add UnminedTxId methods and conversions for InventoryHash
* Map WtxIds to unmined transaction network messages
Also, use UnminedTxId and UnminedTx in:
* Zebra's internal request and response format, and
* external Zcash network protocol messages.
* Enable WtxId mempool inventory tracking for peers
* Further clarify transaction IDs
* Use Witnessed rather than Wide for transaction IDs
And rename narrow to legacy when it only applies to v1-v4 transactions.
Otherwise, rename it to mined ID.
* Rename a missed binding
* Remove an incorrectly named binding
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Make the `AuthDigest` display order match transaction IDs
And derive `Hash`, just like transaction IDs.
Don't derive `serde` for now, because it's not needed.
* Move transaction::Hash test to tests module
* Add a simple AuthDigest display order test
* Add a WtxId type for wide transaction IDs
* Add conversions between transaction IDs and bytes
* Use the WtxId type in external network protocol messages
* Generate chains with valid chain value pool balances
* Move MAX_PARTIAL_CHAIN_BLOCKS to zebra-chain
* Fix generated value overflow based on the maximum number of values
And split it into its own method.
* Split fix_remaining_value into smaller methods
* Remove unused methods
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Make Amount arithmetic more generic
To modify generated amounts, we need some extra operations on `Amount`.
We also need to extend existing operations to both `NonNegative` and
`NegativeAllowed` amounts.
* Add a constrain method for ValueBalance
* Derive Eq for ValueBalance
* impl Neg for ValueBalance
* Make some Amount arithmetic expectations explicit
* Explain why we use i128 for multiplication
And expand the overflow error details.
* Expand Amount::sum error details
* Make amount::Error field order consistent
* Rename an amount::Error variant to Constraint, so it's clearer
* Add specific pool variants to ValueBalanceError
* Update coinbase remaining value consensus rule comment
This consensus rule was updated recently to include coinbase transactions,
but Zebra doesn't check block subsidy or miner fees yet.
* Add test methods for modifying transparent values and shielded value balances
* Temporarily set values and value balances to zero in proptests
In both generated chains and proptests that construct their own transactions.
Using zero values reduces value calculation and value check test coverage.
A future change will use non-zero values, and fix them so the check passes.
* Add extra fields to remaining transaction value errors
* Swap the transparent value balance sign to match shielded value balances
This makes the signs of all the chain value pools consistent.
* Use a NonNegative constraint for transparent values
This fix:
* makes the type signature match the consensus rules
* avoids having to write code to handle negative values
* Allocate total generated transaction input value to outputs
If there isn't enough input value for an output, set it to zero.
Temporarily reduce all generated values to avoid overflow.
(We'll remove this workaround when we calculate chain value balances.)
* Consistently use ValueBalanceError for ValueBalances
* Make the value balance signs match the spec
And rename and document methods so their signs are clearer.
* Convert amount::Errors to specific pool ValueBalanceErrors
* Move some error changes to the next PR
* Add extra info to remaining transaction value errors (#2585)
* Distinguish between overflow and negative remaining transaction value errors
And make some error types cloneable.
* Add methods for updating chain value pools (#2586)
* Move amount::test to amount::tests:vectors
* Make ValueBalance traits more consistent with Amount
- implement Add and Sub variants with Result and Assign
- derive Hash
* Clarify some comments and expects
* Create ValueBalance update methods for blocks and transactions
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
This improves proptest results in CI and locally.
Proptests should be faster, because they are not discarding 1/16 results.
Failures should be minimised more often, improving failure logs,
and generating proptest seeds locally and in CI.
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Make legacy chain limit clearer
That way, it doesn't get confused with the coinbase maturity limit.
* Allow 1-5 transactions in each generated block, not always 5
* rustfmt
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Tidy chain Cargo.toml
* Organize imports
* Add method to get note commitments from all Actions in Orchard shielded data
* Add method to get note commitments from all JoinSplits in Sprout JoinSplitData
* Add Request and Response variants for awaiting anchors
* Add anchors and note commitment trees to finalized state db
* Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees
* Track anchors and note commitment trees in Chain
Append note commitments to their trees when doing update_chain_state_with,
then use the resulting Sapling and Orchard roots to pass to history_tree, and add
new roots to the anchor sets.
* Handle errors when appending to note commitment trees
* Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with
* Implementing note commitments in finalized state
* Finish serialization of Orchard tree; remove old tree when updating finalize state
* Add serialization and finalized state updates for Sprout and Sapling trees
* Partially handle trees in non-finalized state. Use Option for trees in Chain
* Rebuild trees when forking; change finalized state tree getters to not require height
* Pass empty trees to tests; use empty trees by default in Chain
* Also rebuild anchor sets when forking
* Use empty tree as default in finalized state tree getters (for now)
* Use HashMultiSet for anchors in order to make pop_root() work correctly
* Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS
* Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more
* Apply suggestions from code review
* Add comments about order of note commitments and related methods/fields
* Don't use Option for trees
* Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS
* Remove unneeded anchor set rebuilding in fork()
* Improve proptest formatting
* Add missing comparisons to eq_internal_state
* Renamed sprout::tree::NoteCommitmentTree::hash() to root()
* Improve comments
* Add asserts, add issues to TODOs
* Remove impl Default for Chain since it was only used by tests
* Improve documentation and assertions; add tree serialization tests
* Remove Sprout code, which will be moved to another branch
* Add todo! in Sprout tree append()
* Remove stub request, response *Anchor* handling for now
* Add test for validating Sapling note commitment tree using test blocks
* Increase database version (new columns added for note commitment trees and anchors)
* Update test to make sure the order of sapling_note_commitments() is being tested
* Improve comments and structure of the test
* Improve variable names again
* Rustfmt
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: teor <teor@riseup.net>
* add value_balance methods to transparent and shielded
* add value_balance() to transaction
* check the remaining value consensus rule
* change error name
* fix doc and nitpick
* refactor value_balance() method for joinsplit
* changes to value_balance() of Inputs
* implement joinsplits() method(not working)
* remove created methods
* remove special case
* change return error in utilities
* move utils functions to transaction methods
* fix the docs
* simplify some code
* add constrains explicitly
* remove turbofish
* refactor some transaction methods
* fix value balance signs, add docs
* simplify some code
* avoid panic in consensus check
* add missing doc
* move remaining value balance check to the state
* make changes from the last review
Co-authored-by: teor <teor@riseup.net>
* Always use librustzcash for sighash and remove old sighash code
Also added ZIP-143 test vectors
* Remove librustzcash_sighash test that is no longer needed
* Refactor to create `verify_sapling_shielded_data`
Move the code to verify Sapling shielded data into a new helper method
that returns `AsyncChecks`.
* Test verifying a Sapling transaction with spends
Use the test vectors to find a transaction that has Sapling spends and
test if it the verifier considers it valid.
* Create a helper method to list test transactions
Transforms the block test vectors into a list of transactions and block
heights for each transaction.
* Use new helper function in V4 Sapling spend test
Also use the block height for that transaction as specified in the test
vector.
* Test V4 tx. with Sapling outputs but no spends
Find a transaction V4 vector that has Sapling outputs but no spends, and
check that the verifier accepts it.
* move network_upgrade check into zebra-chain
* fix the errors
* rename function
* typo fix
* rename the check function
* make changes from last code review
* stop panicking on invalid orchard nullifiers
* add context to error
* use `from_bytes_wide` for nullifiers in arbitrary
* orchard::Nullifier vec to array conversion is a bit clearer and simpler
Co-authored-by: teor <teor@riseup.net>
* Add a `Transaction::version` getter
Returns the version of the transaction as a `u32`.
* Add `Transaction::is_overwintered` helper method
Returns if the `fOverwintered` flag should be set for the transaction's
version.
* Use new helpers to serialize transaction version
Reduce the repeated code and make it less error-prone with future
changes.
* Add getter methods to `transaction::Request` type
Refactor to move the type deconstruction code into the `Request` type.
The main objective is to make it easier to split the call handler into
methods that receive the request directly.
* Refactor to create `verify_v4_transaction` helper
Split the code specific to V4 transactions into a separate helper
method.
* Create `verify_v5_transaction` helper method
Prepare a separate method to have the validation code.
* Add `UnsupportedByNetworkUpgrade` error variant
An error for when a transaction's version isn't supported by the network
upgrade of the block it's included or for the current network upgrade if
the transaction is for the mempool.
* Verify a V5 transaction's network upgrade
For now, only NU5 supports V5 transactions.
* Test that V5 transaction is rejected on Canopy
Create a fake V5 transaction and try to verify it using a block height
from Canopy's activation. The verifier should reject the transaction
with an error saying that the network upgrade does not support that
transaction version.
* Test if V5 tx. is accepted after NU5 activation
Create a fake V5 transaction and pretend it is placed in a block that
has a height after the NU5 activation. The test should succeed, but
since the NU5 activation height has not been specified yet (neither for
the testnet nor the mainnet), for now this test is marked as
`should_panic`.
* Add `TODO` comment to the code
Add more detail to what's left to do, and link to the appropriate PRs.
* Use `u32` to store transaction version
Use a type consistent with how the version is specified.
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: teor <teor@riseup.net>
* Add a `at_least_one!` macro for testing
Similar to the `vec!` macro, but doesn't allow creating an empty list.
* Test if `has_inputs_and_outputs` considers actions
Create a dummy transaction with no inputs and no outputs, and add a
dummy Orchard action to it. The `check::has_inputs_and_outputs`
should succeed, because the consensus rule considers having Orchard
actions as having inputs and/or outputs.
* Refactor to create helper function
Move the code to create a fake Orchard shielded data instance to a
helper function in `zebra_chain::transaction::arbitrary`, so that other
tests can also use it.
* Test coinbase V5 transaction with enable spends
A V5 coinbase transaction that has Orchard shielded data MUST NOT have
the enable spends flag set.
* Test if coinbase without enable spends is valid
A coinbase transaction with Orchard shielded data and without the enable
spends flag set should be valid.
* Add a security comment about the `at_least_one!` macro
This macro must not be used outside tests, because it allows memory denial
of service.
Co-authored-by: teor <teor@riseup.net>
* Implement `PartialEq<i64>` for `Amount`
Allows to compare an `Amount` instance directly to an integer.
* Add `SerializationError::BadTransactionBalance`
Error variant representing deserialization of a transaction that doesn't
conform to the Sapling consensus rule where the balance MUST be zero if
there aren't any shielded spends and outputs.
* Validate consensus rule when deserializing
Return an error if the deserialized V4 transaction has a non-zero value
balance but doesn't have any Sapling shielded spends nor outputs.
* Add consensus rule link to field documentation
Describe how the consensus rule is validated structurally by
`ShieldedData`.
* Clarify that `value_balance` is zero
Make the description more concise and objective.
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Update field documentation
Include information about how the consensus rule is guaranteed during
serialization.
Co-authored-by: teor <teor@riseup.net>
* Remove `check::sapling_balances_match` function
The check is redundant because the respective consensus rule is
validated structurally by `ShieldedData`.
* Test deserialization of invalid V4 transaction
A transaction with no Sapling shielded spends and no outputs but with a
non-zero balance value should fail to deserialize.
* Change least-significant byte of the value balance
State how the byte index is calculated, and change the least
significant-byte to be non-zero.
Co-authored-by: teor <teor@riseup.net>
* Standardise lints across Zebra crates, and add missing docs
The only remaining module with missing docs is `zebra_test::command`
* Todo -> TODO
* Clarify what a transcript ErrorChecker does
Also change `Error` -> `BoxError`
* TransError -> ExpectedTranscriptError
* Output Descriptions -> Output descriptions
* Restore SummaryDebug on arbitrary chains
And also add it to some more proptest vectors.
* Reduce most arbitrary vectors from 10 to 4
This makes debugging easier
* Make SummaryDebug generic over collections and exact size iterators
* Document DisplayToDebug
* Fix documentation comment
Was missing a slash to become documentation.
* Add documentation link to type reference
Just to help navigation a bit.
* Implement `Transaction::orchard_actions()` getter
Returns an iterator to iterator over the actions in the Orchard shielded
data (if there is one, otherwise it returns an empty iterator).
* Add V5 support for `has_inputs_and_outputs`
Checks if the transaction has Orchard actions. If it does, it is
considered to have inputs and outputs.
* Refactor transaction test vectors
Make it easier to reuse the fake V5 transaction converter in other test
vectors.
* Move helper function to `zebra-chain` crate
Place it together with some other helper functions, including the one
that actually creates the fake V5 transaction.
* Test transaction with no inputs
`check::has_inputs_and_outputs` should return an error indicating that
the transaction has no inputs.
* Test transaction with no outputs
`check::has_inputs_and_outputs` should return an error indicating that
the transaction has no outputs.
* Note that transaction is fake in `expect` message
Should make the message easier to find, and also gives emphasis to the
fact that the transaction is a fake conversion to V5.
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: teor <teor@riseup.net>
* Fix scriptCode serialization and sighash test vectors
The scriptCode was being serialized without the compact size prefix, and the test vectors included the prefix in the script, which cancelled each other
* initialize the work on parsing orchard data in V5
* add the rest of orchard serialization
* fix serialization according to spec
* fix arbitrary for Signature<SpendAuth>
* move deserialization of AuthorizedAction to shielded_data module
* use `from_bits_truncate` to generate valid arbitrary flags
* change panic message
* fix serialization/deserialization when nActionsOrchard is empty
* fix Halo2Proof deserialization
* implement ZcashSerialize and ZcashDeserialize for flags
* implement ZcashSerialize and ZcashDeserialize for orchard::tree::Root
* use ZcashSerialize and ZcashDeserialize for binding_sig
* implement from_parts()
* implement Arbitrary for Signature<Binding>
* add trusted preallocate with tests
* fix Arbitrary for orchard Nullifier
* Use zcash_serialize_bytes instead of write_compactsize
Co-authored-by: teor <teor@riseup.net>
* Security: panic if an internally generated time is out of range
If Zebra has a bug where it generates blocks, transactions, or meta
addresses with bad times, panic. This avoids sending bad data onto the
network.
(Previously, Zebra would truncate some of these times, silently
corrupting the underlying data.)
Make it clear that deserialization of these objects is infalliable.
* add consensus_branch_id field to transaction v5
* clippy
* rustfmt
* replace consensus_branch_id with network_upgrade
* remove unintended test files
* change method name
* some clanups
* add network_upgrade as a constant in tests
* use std in created function
* add comment to manual arbitrary impl
* create custom strategy to deal with NetworkUpgrade
* Add a missing TODO comment
Co-authored-by: teor <teor@riseup.net>
* validate sapling v5 tx
* Make itertools dependency optional
We only need itertools when the `proptest-impl` feature is enabled.
* Check if V4 and V5 coinbase transactions contain PrevOut transparent inputs
This is a bugfix on V4 transaction validation. The PrevOut consensus
rule was not explicitly stated in the Zcash spec until April 2021.
(But it was implied by Bitcoin, and partially implemented by Zebra.)
Also do the shielded sapling input check for V5 transactions.
* Add spec and orchard TODOs to has_inputs_and_outputs
Also make the variable names match the spec.
* Sort transaction functions to match v5 data order
* Simplify transaction input and output checks
Move counts or iterators into `Transaction` methods, so we can remove
duplicate code, and make the consensus rule logic clearer.
* Update sapling_balances_match for Transaction v5
- Quote from the spec
- Explain why the function is redunant for v5
- Rename the function so it's clear that it is sapling-specific
Co-authored-by: teor <teor@riseup.net>
* Redesign Sapling data model for V5 shared anchor and spends
The shared anchor is only present if there are any spends.
As part of this change, delete the manual PartialEq impl and its tests,
because we can derive PartialEq now.
* Stop creating a temporary Vec for the spend and output iterators
* Rename TransferData variants
Interactive rename using the following commands:
```sh
fastmod Spends SpendsAndMaybeOutputs
fastmod NoSpends JustOutputs
```
* Refactor out common sprout nullifier code
* Implement the AtLeastOne constrained vector type
This vector wrapper ensures that it always contains at least one element.
* Simplify Sapling TransferData using AtLeastOne
Also update the RFC to use AtLeastOne for Orchard.
* serialize/deserialize spaling shielded data in v5 transaction
* fix serialize/deserialize fields order according to spec
* remove extra clone calls
* more serialize fixes
* clippy: fix empty array
* tidy comments
* Add v4 and v5 transaction tests
Also make sure that serialized bytes match if structs match.
* Test fake v5 blocks made out of pre-NU5 block test vectors
* Add outputs-only tests for v5 shared anchor serialization
* Refactor sapling::ShieldedData V5 serialization into its own impl
* Fix spec name typos
* Simplify sapling shielded data parsing
* Delete redundant V5 transaction wrappers in tests
And split out sapling ShieldedData serialization.
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* add sapling shielded data to transaction V5
* implement nullifiers
* test v5 in shielded_data_roundtrip
* Explicitly design serialization for Transaction V5
Implement serialization for V4 and V5 spends and outputs, to make sure
that the design works.
* Test serialization for v5 spends and outputs
Also add a few missing v4 tests.
* Delete a disabled proptest
* Make v5 transactions a top-level heading
And add a missing serialized type.
* Fix a comment typo
* v5 transaction RFC: split array serialization
Based on #2017
* RFC: explicitly describe serialized field order
And link to the spec
* RFC: add the shared anchor serialization rule test
Co-authored-by: teor <teor@riseup.net>
* Move the preallocate tests into their own files
And move the MetaAddr proptest into its own file.
Also do some minor formatting and cleanups.
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* Implement SafePreallocate. Resolves#1880
* Add proptests for SafePreallocate
* Apply suggestions from code review
Comments which did not include replacement code will be addressed in a follow-up commit.
Co-authored-by: teor <teor@riseup.net>
* Rename [Safe-> Trusted]Allocate. Add doc and tests
Add tests to show that the largest allowed vec under TrustedPreallocate
is small enough to fit in a Zcash block/message (depending on type).
Add doc comments to all TrustedPreallocate test cases.
Tighten bounds on max_trusted_alloc for some types.
Note - this commit does NOT include TrustedPreallocate
impls for JoinSplitData, String, and Script.
These impls will be added in a follow up commit
* Implement SafePreallocate. Resolves#1880
* Add proptests for SafePreallocate
* Apply suggestions from code review
Comments which did not include replacement code will be addressed in a follow-up commit.
Co-authored-by: teor <teor@riseup.net>
* Rename [Safe-> Trusted]Allocate. Add doc and tests
Add tests to show that the largest allowed vec under TrustedPreallocate
is small enough to fit in a Zcash block/message (depending on type).
Add doc comments to all TrustedPreallocate test cases.
Tighten bounds on max_trusted_alloc for some types.
Note - this commit does NOT include TrustedPreallocate
impls for JoinSplitData, String, and Script.
These impls will be added in a follow up commit
* Impl TrustedPreallocate for Joinsplit
* Impl ZcashDeserialize for Vec<u8>
* Arbitrary, TrustedPreallocate, Serialize, and tests for Spend<SharedAnchor>
Co-authored-by: teor <teor@riseup.net>
* start refactoring transaction v4 for transaction v5
- move ShieldedData to sapling
- add AnchorVariant
- rename shielded_data to sapling_shielded data in V4
- move value_balance into ShieldedData
- update prop tests for new structure
* add AnchorVariant to Spend
- make anchor types available from sapling crate
- update serialize
* change shielded_balances_match() arguments
* change variable name anchor to shared_anchor in ShieldedData
* fix empty value balance serialization
* use AnchorV in shielded spends
* Rename anchor to per_spend_anchor
* Use nullifiers function directly in non-finalized state
* Use self.value_balance instead of passing it as an argument
* Add missing fields to ShieldedData PartialEq
* Derive Copy for tag types
* Add doc comments for ShieldedData refactor
* Implement a per-spend anchor compatibility iterator
Co-authored-by: teor <teor@riseup.net>
Zebra believes the untrusted `JoinSplit` list size field when
deserializing `JoinSplit`s, and preallocates a `Vec` based on that size.
This is trivial a memory exhaustion attack.
Instead, use the current auto-growing implementation, which is limited
by the size of the message data.
* add transaction V5 stub
* add v5_strategy
* deduplicate version group ids
* Update comment for V5 transactions
* Add V5 transactions to non_finalized_state
Currently these are all `unimplemented!(...)`
* Fix struct matches
* Apply trivial panic message changes
* add zcash_deserialize for V5
* make all tx versions explicit in sprout and sapling nullifier functions
* match exhaustively in sprout and sapling nullifier functions
* fix matches in zebra-consensus
* fix NU5 strategy
* We're still deciding if v5 transactions support Sprout
Co-authored-by: teor <teor@riseup.net>
* Add NU5 variant to NetworkUpgrade
* Add consensus branch ID for NU5
* Add network protocol versions for NU5
* Add NU5 to the protocol::version_consistent test
* Make unimplemented panic messages more specific
* Block target spacing doesn't change in NU5
* add comments for future updates for NU5
Co-authored-by: teor <teor@riseup.net>
* Reverse displayed endianness of transaction and block hashes
* fix zebra-checkpoints utility for new hash order
* Stop using "zebrad revhex" in zebrad-hash-lookup
* Rebuild checkpoint lists in new hash order
This change also adds additional checkpoints to the end of each list.
* Replace TransactionHash with transaction::Hash
This change should have been made in #905, but we missed Debug impls
and some docs.
Co-authored-by: Ramana Venkata <vramana@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
Janito Vaqueiro Ferreira Filho
Conrado Gouvea
Alfredo Garcia
Marek
Deirdre Connolly
Deirdre Connolly
teor
Preston Evans
Jane Lusby
Jane Lusby