The derived Debug impl just shows u8s as numbers, which isn't what we
want. There are basically two reasonable options here:
1. Hex-encoded bytes
2. Escaped ASCII
I picked (2) because a lot of coinbase data has ascii text in it.
`zcashd` converts the PoWLimit into a compact representation before
using it to perform difficulty filter checks.
The Zcash specification converts to compact for the default difficulty
filter, but not for testnet minimum difficulty blocks. (ZIP 205 and
ZIP 208 don't specify this conversion either.) See #1277.
* Implement Expanded to Compact Difficulty
* Implement Arbitrary for CompactDifficulty
Remove the derive, and generate values from random block
hashes.
* Implement Arbitrary for ExpandedDifficulty and Work
* Use Arbitrary for CompactDifficulty in Arbitrary for Block
* Test difficulty on all block test vectors
And cleanup some duplicate test code
* Round-trip tests for compact test cases
* Round-trip tests for compact difficulty in block test vectors
* Make Add for Work return PartialCumulativeWork
Remove AddAssign for Work
Rewrite a proptest using Sub for PartialCumulativeWork
Use Arbitrary for Work
* Add roundtrip work sum tests
* Add roundtrip comparison difficulty tests
* Add failing proptest cases due to test bugs
* Use Some(_) rather than _.into()
* Reduce visibility of difficulty type inner values
* Split work and other difficulty proptests
This change makes sure that rejected work values don't disable property
tests on other types.
* Reverse displayed endianness of transaction and block hashes
* fix zebra-checkpoints utility for new hash order
* Stop using "zebrad revhex" in zebrad-hash-lookup
* Rebuild checkpoint lists in new hash order
This change also adds additional checkpoints to the end of each list.
* Replace TransactionHash with transaction::Hash
This change should have been made in #905, but we missed Debug impls
and some docs.
Co-authored-by: Ramana Venkata <vramana@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
Reduce the default number of cases run by the equihash proptests,
so they run in under 10 seconds on my machine.
Set the PROPTEST_CASES env var to override the default.
* add general and founders reward subsidy modules
* validate founders reward
* Use funding streams after Canopy on testnet
ZIP-1014 only applies to mainnet, where Canopy is at the first halving.
On testnet, Canopy is before the first halving, and the dev fund rules
apply from Canopy. (See ZIP-214.)
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Jane Lusby <jlusby42@gmail.com>
* pass all test vectors through current subsidy validation
* Add testnet and halving subsidy tests
* add subsidy validation error tests
* rename block validation methods
* add network to block verifier
* add amount operators
* Implement Ord, Eq, and Hash for Amount
* Implement Add<Height> for Height
And make the existing Height operators do range checks.
* Apply operator suggestions
Co-authored-by: Jane Lusby <jlusby42@gmail.com>
* implement most of the chain functions
* implement fork
* fix outpoint handling in Chain struct
* update expect for work
* split utxo into two sets
* update the Chain definition
* remove allow attribute in zebra-state/lib.rs
* merge ChainSet type into MemoryState
* Add error messages to asserts
* export proptest impls for use in downstream crates
* add testjob for disabled feature in zebra-chain
* try to fix github actions syntax
* add module doc comment
* update RFC for utxos
* add missing header
* working proptest for Chain
* propagate back results over channel
* Start updating RFC to match changes
* implement queued block pruning
* and now it syncs wooo!
* remove empty modules
* setup config for proptests
* re-enable missing_docs lint
* update RFC to match changes in impl
* add documentation
* use more explicit variable names
* Begin work on RFC5 implementation
* I think this is necessary
* holy shit supertrait implemented via subtrait
* implement most of the chain functions
* change to slightly better name
* implement fork
* fix outpoint handling in Chain struct
* update expect for work
* resolve review comment
* split utxo into two sets
* update the Chain definition
* just a little more
* update comment
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* apply changes from code review
* remove allow attribute in zebra-state/lib.rs
* Update zebra-state/src/memory_state.rs
Co-authored-by: teor <teor@riseup.net>
* merge ChainSet type into MemoryState
* rename state impl types
* Add error messages to asserts
* checkpoint so I can split off arbitrary changes into a PR
* export proptest impls for use in downstream crates
* add testjob for disabled feature in zebra-chain
* run rustfmt
* try to fix github actions syntax
* differentiate name
* prove that github action tests zebra-chain build without features
* revert change from last commit now that test is running
* remove accidentally introduced newline
* checkpoint
* add module doc comment
* update RFC for utxos
* add missing header
* working proptest for Chain
* apply change from chain impl PR
* setup config for proptests
* Update zebra-chain/src/block/arbitrary.rs
Co-authored-by: teor <teor@riseup.net>
* run rustfmt
Co-authored-by: teor <teor@riseup.net>
To avoid confusion with TESTNET_BLOCKS.
Automated search and replace, made using this script:
sed -i 's/TEST_BLOCKS/BLOCKS/' \
$(grep -r TEST_BLOCKS zebra* | cut -d: -f1 | sort -u)
* Move block test vectors into separate files
* Refactor block test vectors into their own module
* Remove some duplicate test vector data
* Add mainnet block test vectors for each network upgrade
* Add testnet block test vectors for each network upgrade
* Update the TEST_BLOCKS list
* Add basic tests using TEST_BLOCKS
* Ignore shell.nix
* Begin work on RFC5 implementation
* I think this is necessary
* holy shit supertrait implemented via subtrait
* implement most of the chain functions
* change to slightly better name
* implement fork
* fix outpoint handling in Chain struct
* update expect for work
* resolve review comment
* split utxo into two sets
* update the Chain definition
* just a little more
* update comment
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* apply changes from code review
* remove allow attribute in zebra-state/lib.rs
* Update zebra-state/src/memory_state.rs
Co-authored-by: teor <teor@riseup.net>
* merge ChainSet type into MemoryState
* rename state impl types
* Add error messages to asserts
* add module doc comment
* update RFC for utxos
* add missing header
Co-authored-by: teor <teor@riseup.net>
* export proptest impls for use in downstream crates
* add testjob for disabled feature in zebra-chain
* run rustfmt
* try to fix github actions syntax
* differentiate name
* prove that github action tests zebra-chain build without features
* revert change from last commit now that test is running
* remove accidentally introduced newline
* Update .github/workflows/ci.yml
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Implement sighash
* move sighash logic to a separate module
* start filling in more of the alg
* start setting up a test case
* make the test useful
* Iter transaction inputs
* better error message for expect
* add support for zip243 sighash
* ohey first testvector is passing, yayyy
* pass the second testvector
* add last testvector
* move a use statement
* use common deserialization code for amount everywhere
* cleanup attributes
* bring in fixed preimage
* fix discrepancy with spec
* always deserialize as a signed value
* Update zebra-chain/src/transaction/sighash.rs
* update unreachable statements
* add serialization impls for nonnegative amounts
* Apply suggestions from code review
* document sighash fn
* tweek docs
* fix mistake in translation for zip243
* consistent error messages
* reorder because i like it more that way
* document more panics
* Update zebra-chain/src/amount.rs
* Add comment regarding the serialization of spend descriptions in sighash
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* attempt to use zcashconsensus crate in zebra-script
* boop
* update verify fn to use zebra types
* a bit more cleanup
* cleanup
* more
* beep boop
* fix renamed member
* cleaning
* get a real branch id
* remove as of yet unneeded api
* Update zebra-chain/src/transaction.rs
* Update zebra-chain/src/transaction.rs
* more cleanup
* oops wrong dep section
* use a tuple to communicate arg association
* update to use published version of zcash_script
* fix new compiler error
* install llvm on windows
* fix bindgen bug????
* try to get docker file to win
* okay try everything
* fix windows build maybe
* always download choco
* fix paths for moved types
* try a different error message
* try convenience script
* try installing just llvm
* add back one more
* try installing some headers
* try a diff package
* try everything
* remove the minimum
* try newer docker builder image
* cleanup docker image
* cleanup extra ci step
The ShieldedData container for the spend and output descriptions of a
transaction does *not* move, because it's encoding transaction
structure. A good sign that this is the right factoring is that the
transaction module now only needs to pull in sapling::{Spend, Output}
and not all the internals.
This is the first in a sequence of changes that change the block:: items
to not include Block as a prefix in their name, in accordance with the
Rust API guidelines.
This Merkle tree is the SHA256d one used only for including transactions
in a block, so it should be kept there in order to not be confused with
other Merkle trees (like the note commitment trees).
It's not accurate to call it a LightClientRootHash, because it's not
always a root has for a light client -- sometimes it's a different kind
of root hash.
This moves the tests::generate module into the block tests. Because
this whole set of changes is just focused on reorganization, the
generate code is unchanged, but in the future, the code should be
rewritten as a collection of proptest strategies.
This extracts the `difficulty` module from `block` and the
`equihash_solution` module from the crate root. The PoW calculations
are significantly more complicated than the other block code and pretty
dissimilar from it, so it makes more sense to create a common proof of
work module.
The `EquihashSolution` and `EQUIHASH_SOLUTION_SIZE` are renamed to
`equihash::Solution` and `equihash::SOLUTION_SIZE` and imported that
way, except in `block/header.rs`, to avoid a conflict with the
`equihash` crate. In the future it would be better to encapsulate the
equihash solution check into the `equihash::Solution` type so that
callers only need to import our `work::equihash`.
The test organization leaves a little to be desired but I think that
this can be improved as we fill out the proof of work implementation.
This moves the transaction strategies to `arbitrary`, to live with the
other strategy impls (`Arbitrary` is just a default type-associated
strategy), splits the proptests into a new `prop` module, and splits the
test vector checks into a `vector`s module.
This ensures that we keep code in leaf modules and only have
organization (use statements) in non-leaf modules.
This extracts the SHA256d code from being split across two modules and puts it
in one module, under serialization.
The code is unchanged except for three deleted tests:
* `sha256d_flush` in `sha256d_writer` (not a meaningful test);
* `transactionhash_debug` (constructs an invalid transaction hash, and the
behavior is tested in the next test);
* `decode_state_debug` (we do not need to test the Debug output of
DecodeState);
* feature: Implement CompactDifficulty to Work
* fix: Test Work on all difficulty test cases
* fix: Add Bitcoin test vectors for difficulty
* feature: Cumulative Work
* impl Add, AddAssign for Work
* derive Default for Work
* test cumulative work
* fix: comments
* fix: More comments
* Fix comment links
* Remove unused import
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* chain: add Transaction::is_coinbase()
This matches the check in zcashd/src/primitives/transaction.h:682
(CTransaction::IsCoinBase).
* chain: correct Block::is_coinbase_first
This matches zcashd/src/main.cpp:3968-3974 in CheckBlock.
Previously, the check allowed the first transaction to have multiple coinbase inputs.
* chain: return slices from Transaction::inputs()/outputs()
They're slices internally so we might as well just expose them that way.
This changes the `light_client_root_hash` field to `light_client_root_bytes` to
hint that it's unparsed, and makes it public to match the rest of the
`BlockHeader` fields. The `LightClientRootHash` serialization methods are
hidden from the public API, so that the `LightClientRootHash` has to be
constructed by the method on the `Block`.
* use the right variant in LightClientRootHash::from_bytes()
* make block.header.light_client_root_hash pub(super)
* add tests for LightClientRootHash and block.light_client_root_hash
The Heartwood upgrade changes the meaning of the hashFinalSaplingRoot to
hashLightClientRoot. Since we don't know the network upgrade heights in
zebra-chain, we just use [u8; 32] for now.
This uses serde's try_from attribute to run deserialized values through the
TryFrom impl. Also adds a test to make sure that validation actually does
happen.
We had a brief discussion on discord and it seemed like we had consensus on the
following versioning policy:
* zebrad: match major version to NU version, so we will start by releasing
zebrad 3.0.0;
* zebra-* libraries: start by matching zebrad's version, then increment major
versions of each library as we need to make breaking changes (potentially
faster than the zebrad version, always respecting semver but making no
guarantees about the longevity of major releases).
This commit sets all of the crate versions to 3.0.0-alpha.0 -- the -alpha.0
marks it as a prerelease not subject to perfect adherence to compatibility
guarantees.
* Add checkpoint list generation scripts
* Limit the checkpoint block data size
* Limit the checkpoint height gap
* Add Mainnet and Testnet checkpoint lists
* Parse hard-coded checkpoint lists
The lists were generated using the following limits:
- 256 MB spacing, based on block byte size, and
- 2000 blocks.
* Add MIN and MAX for BlockHeight and LockTime
* Remove duplicate test cases
* fix a comment about the minimum lock time
The minimum LockTime::Time is 5 November 1985 00:53:20 UTC, so the first
day that only contains valid times is 6 November 1985 (in all timezones).
Similarly, the maximum LockTime::Time is 7 February 2106 06::28::15 UTC,
so the last day that only contains valid times in all time zones is
5 February 2106.
* fix: Reject checkpoint lists with bad hashes or heights
Reject the all-zeroes hash, because it is the parent hash of the genesis
block, and should never appear in a checkpoint list.
Reject checkpoint heights that are greater than the maximum block
height.
* add proptests for invalid forms of input
* apply proptests to each valid block we have
* start moving equihash logic to BlockHeader
* provide an error based interface
* cleanup logging in dependent crate
* use is_equihash_solution_valid everywhere
* switch to proper equihash crate branch
* add block size limit test
* calculate max_transactions_in_block and max_inputs_in_tx
* abstract block generation
* move all test vectors to zebra-test
This change brings in the `equihash` crate from librustzcash and uses it to add a basic `is_valid` test for `EquihashSolutions`.
Co-authored-by: Jane Lusby <jane@zfnd.org>
* create hash submodule for block
* create header submodule for block
* create serialize submodule for block
* add newline to hash.rs (fmt)
* Update zebra-chain/src/block/tests.rs
Co-authored-by: Jane Lusby <jlusby42@gmail.com>
Co-authored-by: Henry de Valence <hdevalence@hdevalence.ca>
Co-authored-by: Jane Lusby <jlusby42@gmail.com>
Use MAX constants for the block header version and time arbitrary test
ranges. Reduces the block header time arbitrary test range from 2**32 to
2**32-1 (u32::MAX). (2**32 is an invalid time value, which gets
truncated to 0 during serialization.)
Also add some comments about DateTime conversions.
Part of #477.
consensus: Add a checkpoint verifier stub
This stub only verifies blocks whose hashes are in the checkpoint
list.
It doesn't have tests, chain child verifies to their ancestors, or
support checkpoint maximum height queries.
Part of #429.
The maximum block size is 2,000,000 bytes. This commit also limits the
maximum transaction size in parsed blocks. (See #484 for the
corresponding limit on mempool transactions.)
The proptests might test the maximum block size, but they are
randomised. So we also want to explicitly test large block sizes.
(See #482 for these test cases and tests.)
Part of #477.
Verify the value of the equihash solution size field in block headers.
This field isn't stored in the BlockHeader struct, so we need to verify
it at parse time.
Part of #477.
* rename zebra-storage to zebra-state
* Setup initial skeleton for zebra-state
* add test
* Apply suggestions from code review
Co-authored-by: Henry de Valence <hdevalence@hdevalence.ca>
* move shared test vectors to a common crate
Co-authored-by: Jane Lusby <jane@zfnd.org>
Co-authored-by: Henry de Valence <hdevalence@hdevalence.ca>
Until it can be imported into the jubjub crate and upstreamed.
It has been ported from the types used in zcash_primitives and cleaned up
but still needs testing.
Currently fills/receives 32 random bytes and forces the top 4 bits to
zero, ala clamping. If there is a nicer way to csprng'ly fill 252 bits
without clamping, that would be nicer, less bias.
And test that we can still derive TransmissionKey's from PayingKey's,
as they are aliases for x25519_dalek PublicKey/StaticSecret types that already
impl From<_> for each other.
The CoinbaseData field can only be constructed by the transaction parser, so we
can ensure that a coinbase input is always serializable, as CoinbaseData
instances can't be constructed outside of the parser that maintains the data
size invariant.
Because we represent each transaction version as a different variant of the
Transaction enum, we end up in a situation where fields that are common to
different transaction versions are awkward to access, requiring a match
statement with identical match arms.
To fix this, this commit adds the following convenience methods:
* `Transaction::inputs() -> impl Iterator<Item=&TransparentInput>`;
* `Transaction::outputs() -> impl Iterator<Item=&TransparentOutput>`;
* `Transaction::lock_time() -> LockTime`;
* `Transaction::expiry_height() -> Option<ExpiryHeight>`;
The last returns an `Option` because the field is only present in V3 and V4
transactions.
There are some remaining fields that do not get common accessors, because it
probably doesn't make sense to access independently of knowing the transaction
version: `joinsplit_data`, `shielded_data`, `value_balance`.
BIP34, which is included in Zcash, encodes the block height into each
block by adding it into the unused BitcoinScript field of the block's
coinbase transaction. However, this is done just by requiring that the
script pushes the block height onto the stack when it executes, and
there are multiple different ways to push data onto the stack in
BitcoinScript. Also, the genesis block does not include the block
height, by accident.
Because we want to *parse* transactions into an algebraic data type that
encodes their structural properties, rather than allow possibly-invalid
data to float through the internals of our node, we want to extract the
block height upfront and store it separately from the rest of the
coinbase data, which is inert. So the serialization code now contains
just enough logic to parse BitcoinScript-encoded block heights, and
special-case the encoding of the genesis block.
Elsewhere in the source code, the `LockTime` struct requires that we
must use block heights less than 500,000,000 (above which the number is
interpreted as a unix timestamp, not a height). To unify invariants, we
ensure that the parsing logic works with block heights up to
500,000,000, even though these are unlikely to ever be used for Zcash.
Coinbase inputs are handled differently from other inputs and have
different consensus rules, so they should be represented differently in
the source code. This lets us discard extraneous details (for instance,
it's not necessary to maintain the all-zero hash) and specialize logic.
Closes#158.
As discussed on the issue, this makes it possible to safely serialize
data into hashes, and encourages serializable data to make illegal
states unrepresentable.
ShieldedData objects must have at least one spend or output; using Either
ensures that at least one must be present. This is similar to the
JoinSplitData case, but slightly more complicated: rather than enforcing that
one list has at least one element (which can be done as `(first, rest)`), here
we need to use Either. This has the downside that it is possible to construct
multiple equivalent internal representations (choosing whether a spend or
output goes in the `first` slot), but this easily fixed with a custom PartialEq
implementation.
This replaces the read_list function and makes the code significantly cleaner.
The only downside is that it loses exact preallocation, but this is probably not a big deal.
In the previous transaction modeling I defined the structs so that the number
of old and new commitments for a JoinSplit were variable, when in fact the
Sprout design fixes both to be 2. So now they are hardcoded as 2 in the source
code as well. This commit also fixes some missing `pub` fields on the
`JoinSplit` struct.
* Added a few more top-level fields for the Transaction struct
* Add a placeholder Script type.
This could alternately use bytes::Bytes to save some allocations
but I don't think this is important to get perfectly now. In the future, we
will want to have all of the script handling code in the zebra-script crate,
but we will need to have a container type for an encoded script in zebra-chain,
because otherwise zebra-chain would depend on zebra-script and not the other
way around.
* Rename Transaction{Input,Output} -> Transparent{Input,Output}.
These are only *transparent* inputs and outputs, so by putting Transparent in
the name (instead of Transaction) it's more clear that a transaction's inputs
or outputs can also be shielded.
* Add a LockTime enum.
* First attempt at a Transaction enum.
This attempts to map the versioning and field presence rules into an ADT, so
that structurally invalid transactions (e.g., a BCTV14 proof in a Sapling
transaction) are unrepresentable.
* Update zebra-chain/src/transaction.rs
Co-Authored-By: Daira Hopwood <daira@jacaranda.org>
* Add fixme note on type refinement.
* Rename Transaction variants according to version.
* Split transaction.rs into submodules.
* Start filling in spend, output descriptions.
* Progress on JoinSplit data structures.
This has a lot of duplication and should really use generics to abstract over
Sprout-on-BCTV14 or Sprout-on-Groth16.
* Add data types for Bctv14 and Groth16 proofs.
This also adds a trait to abstract over them.
* Make JoinSplit descriptions generic over the proof system.
* Update zebra-chain/src/transaction/joinsplit.rs
Failure uses a distinct Fail trait rather than the standard library's
Error trait, which causes a lot of interoperability problems with tower
and other Error-using crates. Since failure was created, the standard
library's Error trait was improved, and its conveniences are now
available without the custom Fail trait using `thiserror` (for easy
error derives) and `anyhow` (for a better boxed Error).
* Fix authorship, license information.
I *thought* I had done a sed pass over the Cargo defaults when doing
repository initialization, but I guess I missed it or something.
Anyways, fixed now.
This removes the inventory vector structs from `zebra-chain` (as they
are really part of the network protocol) and refactors them into a
single `InventoryHash` type. This corresponds to Bitcoin's "inventory
vector" but with a different, better name (it's not a vector, it's just
a typed hash of some other item).
This adds convenience methods to `ReadZcashExt` that read 4 and 12 byte
fixed size arrays from the `Reader`, making the actual parsing code more
legible.
Closes#10.
The core serialization logic is now in zebra-chain and consists of two
pairs of traits:
These are analogues of the Serde `Serialize` and `Deserialize` traits,
but explicitly intended for consensus-critical serialization formats.
Thus some struct `Foo` may have derived `Serialize` and `Deserialize`
implementations for (internal) use with Serde, and explicitly-written
`ZcashSerialize` and `ZcashDeserialize` implementations for use in
consensus-critical contexts. The consensus-critical implementations
provide `zcash`-prefixed `zcash_serialize` and `zcash_deserialize`
methods to make it clear in client contexts that the serialization is
consensus-critical.
These are utility traits, analogous to the `ReadBytesExt` and
`WriteBytesExt` traits provided by `byteorder`. A generic
implementation is provided for any `io::Read` or `io::Write`, so that
bringing the traits into scope adds additional Zcash-specific traits to
generic readers and writers -- for instance, writing a `u64` in the
Bitcoin "CompactSize" format.
Jane Lusby
dependabot[bot]
Henry de Valence
teor
Alfredo Garcia
Deirdre Connolly
Jane Lusby
Ramana Venkata
Dimitris Apostolou
dependabot-preview[bot]
Kiminuo