Commit Graph

196 Commits

Author SHA1 Message Date
teor 758eb6e0ea
fix(rpc): Fix bugs and performance of `getnetworksolps` & `getnetworkhashps` RPCs (#7647)
* Handle negative and zero getnetworksolsps arguments correctly

* Simplify the solsps loop structure

* Simplify loop by avoiding manual iteration and peeking

* Avoid division by zero

* Use min and max times rather than first and last times

* Refactor block iterators so they are more efficient

* Make finding chains easier

* Simplify block iteration code

* Remove implemented TODO comments

* Simplify internal iterator state

* Implement iteration by any chain item

* Iterate block headers rather than full blocks

* Ignore code that is (sometimes) dead

* Fix a dead code warning

* Add a no blocks in state error constant

* Check result values in the RPC test

* Fix invalid calculation handling
2023-10-11 02:02:51 +00:00
Marek daee5e5fcd
fix(chain): Return errors instead of panicking in methods for `Height`s (#7591)
* Return errors instead of panicking

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Turn `unwrap`s into `expect`s

* Refactor the error messages

---------

Co-authored-by: teor <teor@riseup.net>
2023-09-21 05:58:04 +00:00
teor 7a7d79dfaf
fix(state): Use correct end heights for end of block subtrees during the full sync (#7566)
* Avoid manual handling of previous sapling trees by using iterator windows instead

* Avoid manual sapling subtree index handling by comparing prev and current subtree indexes instead

* Simplify adding notes by using the exact number of remaining notes

* Simplify by skipping the first block, because it can't complete a subtree

* Re-use existing tree update code

* Apply the sapling changes to orchard subtree updates

* add a reverse database column family iterator function

* Make skipping the lowest tree independent of iteration order

* Move new subtree checks into the iterator, rename to end_height

* Split subtree calculation into a new method

* Split the calculate and write methods

* Quickly check the first subtree before running the full upgrade

* Do the quick checks every time Zebra runs, and refactor slow check error handling

* Do quick checks for orchard as well

* Make orchard tree upgrade match sapling upgrade code

* Upgrade subtrees in reverse height order

* Bump the database patch version so the upgrade runs again

* Reset previous subtree upgrade data before doing this one

* Add extra checks to subtree calculation to diagnose errors

* Use correct heights for subtrees completed at the end of a block

* Add even more checks to diagnose issues

* Instrument upgrade methods to improve diagnostics

* Prevent modification of re-used trees

* Debug with subtree positions as well

* Fix an off-by-one error with completed subtrees

* Fix typos and confusing comments

Co-authored-by: Marek <mail@marek.onl>

* Fix mistaken previous tree handling and end tree comments

* Remove unnecessary subtraction in remaining leaves calc

* Log heights when assertions fail

* Fix new subtree detection filter

* Move new subtree check into a method, cleanup unused code

* Remove redundant assertions

* Wait for subtree upgrade before testing RPCs

* Fix subtree search in quick check

* Temporarily upgrade subtrees in forward height order

* Clarify some comments

* Fix missing test imports

* Fix subtree logging

* Add a comment about a potential hang with future upgrades

* Fix zebrad var ownership

* Log more info when add_subtrees.rs fails

* cargo fmt --all

* Fix unrelated clippy::unnecessary_unwrap

* cargo clippy --fix --all-features --all-targets; cargo fmt --all

* Stop the quick check depending on tree de-duplication

* Refactor waiting for the upgrade into functions

* Wait for state upgrades whenever the cached state is updated

* Wait for the testnet upgrade in the right place

* Fix unused variable

* Fix a subtree detection bug and comments

* Remove an early reference to reverse direction

* Stop skipping subtrees completed at the end of blocks

* Actually fix new subtree code

---------

Co-authored-by: Marek <mail@marek.onl>
2023-09-19 14:49:36 +00:00
Alfredo Garcia e567ab812d
change(deps): Run `cargo update` (#7429)
* cargo update

* update rustls duplicated dependency version

* replace deprecated function
2023-08-31 00:49:01 +00:00
Marek d8f5d6b6f1
change(state): Deduplicate note commitment trees stored in the finalized state (#7312)
* Add support for deleting the trees

* Prune the trees

* Remove `Network` from `DiskWriteBatch`

Removing the `Network` from `DiskWriteBatch` makes it easy to
instantiate `DiskWriteBatch`es in `ZebraDb` that remove individual note
commitment trees.

The `Network` from `DiskWriteBatch` was used only for transparent
addresses, so the refactor isn't large. After removing it from
`DiskWriteBatch`, I passed it as a function argument instead. However,
we should simplify the parameter lists because at least two functions
have more than seven parameters now.

* Support individual tree removal in `ZebraDb`

* Refactor the tree removal task

* Prune old comments

* Remove redundant code

* Batch the removals

* delete ranges before relevant network upgrades

* moves prev_tree inits

* add iterator methods for reading note commitment trees

* Sets up skeleton of sapling pipeline

* Replaces .filter with .take_while

Fills in pipeline

Reuses zs_range_iter instead of repeating that code

Updates logic to stop at initial tip height

* uses std threads

* delete_range excludes end key

* fixes off by one bugs

* Log warning when a send fails

* Removes progress logs

* Log join errors instead of panicking

* Revert: Make the `db` field of `ZebraDb` private

* Move `delete_range_sapling_tree`

* Remove a redundant `else if` branch

Rationale:

The condition `n == 1` for the removed branch is true for a subset of
values of `n` in the preceding condition `n >= 1`.

* Use more specific error messages

* Revert: Remove redundant methods for tree removal

* Suggestions for Deduplicate note commitment trees stored in the finalized state (#7330)

* Add TODOs to some `Height` methods

* Add methods for deleting individual trees

* Refactor the tasks for deleting trees

---------

Co-authored-by: arya2 <aryasolhi@gmail.com>
2023-08-17 00:41:11 +00:00
Marek 9ebd56092b
change(test): Refactor the tests of non-finalized state (#7262)
* Allow generating blocks with only coinbase tx

* Fix the `forked_equals_pushed_genesis` test
2023-07-19 21:20:37 +00:00
Marek 1db4f567f7
change(state): Deduplicate note commitment trees in non-finalized state (#7239)
* Remove duplicate asserts

There are the same two asserts above the two removed ones.

* Remove workarounds for inserting trees into NFS

NFS = non finalized state

* Use correct height for constructing new chain

We were using the height of the last block instead of the initial block
to construct a new chain.

* Don't push the 0th block into a chain

* Don't commit two blocks at the same height

* Add helpers for heights

* Support the retrieval of deduped Sprout trees

* Dedup Sprout trees

* Refactor docs for adding & removing Sprout trees

* Support the retrieval of deduped Sapling trees

* Dedup Sapling trees

* Refactor docs for adding & removing Sapling trees

* Support the retrieval of deduped Orchard trees

* Dedup Orchard trees

* Refactor docs for adding & removing Orchard trees

* Make the docs for storing trees clearer
2023-07-18 23:04:11 +00:00
Marek f9a5635105
fix(state): Remove workarounds for storing trees (#7218)
* Remove duplicate asserts

There are the same two asserts above the two removed ones.

* Remove workarounds for inserting trees into NFS

NFS = non finalized state

* Use correct height for constructing new chain

We were using the height of the last block instead of the initial block
to construct a new chain.

* Don't push the 0th block into a chain

* Don't commit two blocks at the same height

* Fix typo

* Generate chains with at least two blocks

---------

Co-authored-by: teor <teor@riseup.net>
2023-07-18 04:54:22 +00:00
teor 147b8fa3a8
cleanup(rust): Fix new nightly clippy warnings (#7135)
* Fix "comparison is always true" warning

* Add missing Send bound

* cargo clippy --fix --all-features --all-targets

* incorrect implementation of clone on a Copy type

* cargo fmt --all
2023-07-05 07:11:27 +00:00
Alfredo Garcia 4323097794
remove redundant configuration feature (#6929) 2023-06-14 19:06:08 +00:00
teor d3ce022ecc
change(utils): Add a direct connection mode to zebra-checkpoints (#6516)
* Rename variables so it's clearer what they do

* Fully document zebra-checkpoints arguments

* Remove some outdated references to zcashd

* Add a json-conversion feature for converting JSON to valid Heights

* Simplify zebra-checkpoints code using conversion methods

* Track the last checkpoint height rather than the height gap

* Move all the CLI-specific code into a single function

* Remove cfg(feature) from the test RPC client API

* Move the RpcRequestClient into zebra-node-services

* Fix the case of RpcRequestClient

* Add transport and addr arguments to zebra-checkpoints

* Make zebra-checkpoints support both CLI and direct JSON-RPC connections

* Fix RpcRequestClient compilation

* Add a suggestion for zcashd authentication failures

* Document required features

* Handle differences in CLI & direct parameter and response formats

* Replace a custom function with an existing dependency function

* Add a checkpoint list test for minimum height gaps
2023-04-26 23:35:53 +00:00
Arya 6fdd02220e
fix(rpc): Omit transactions with transparent coinbase spends that are immature at the next block height from block templates (#6510)
* Adds `maturity_height` to VerifiedUnminedTx

Filters out transactions that are invalid at next_block_height in getblocktemplate method

* Adds unit testing

* rustfmt

* rejects txs with immature coinbase spends from mempool

* Condenses fns for transparent coinbase spend check

* Updates calls to VerifiedUnminedTx::new()

* Update zebra-chain/src/transparent/utxo.rs

* Applies suggestions from code review
2023-04-18 03:43:39 +00:00
Arya 403c074502
fix(chain): Validate header versions when serializing blocks (#6475)
* checks block header version when serializing

* fixes test panic

* adds docs

* test block header (de)serialization
2023-04-12 04:54:02 +00:00
Marek 2a48d4cf25
change(chain): Refactor the handling of height differences (#6330)
* Unify the `impl`s of `Sub` and `Add` for `Height`

* Adjust tests for `Height` subtraction

* Use `Height` instead of `i32`

* Use `block:Height` in RPC tests

* Use `let .. else` statement

Co-authored-by: Arya <aryasolhi@gmail.com>

* Update zebra-consensus/src/block/subsidy/general.rs

* Refactor the handling of height differences

* Remove a redundant comment

* Update zebrad/src/components/sync/progress.rs

Co-authored-by: Arya <aryasolhi@gmail.com>

* Update progress.rs

* impl TryFrom<u32> for Height

* Make some test assertions clearer

* Refactor estimate_up_to()

* Restore a comment that was accidentally removed

* Document when estimate_distance_to_network_chain_tip() returns None

* Change HeightDiff to i64 and make Height.sub(Height) return HeightDiff (no Option)

* Update chain tip estimates for HeightDiff i64

* Update subsidy for HeightDiff i64

* Fix some height calculation test edge cases

* Fix the funding stream interval calculation

---------

Co-authored-by: Arya <aryasolhi@gmail.com>
Co-authored-by: teor <teor@riseup.net>
2023-03-29 23:06:31 +00:00
Arya 083ddea58a
change(docs): Replaces XXX with TODO (#6417)
* Replaces XXX with TODO:

* Updates block/tests
2023-03-28 04:13:04 +00:00
Pili Guerra 87f9e02dbf
Fix tipo. Credit: @rex4539 (#6328) 2023-03-15 19:40:54 +00:00
teor a8370a8eda
Remove unused impl ZcashDeserialize for Height (#6139) 2023-02-10 20:24:46 +00:00
teor 8c451968ee
fix(diagnostics): Hex-encode debug format of commitments, nonces, and nullifiers (#5960)
* Format commitments and nonces as hex

* Format Sprout and Sapling nullifiers as hex when debugging

* Format Sprout commitments as hex when debugging

* Format redpallas keys as hex when debugging

* Update code that we're going to delete in the next PR anyway
2023-01-17 13:57:22 +00:00
teor e9d6e975b4
change(rpc): Refactor get block template RPC into stages (#5837)
* Add some TODOs

* Move and rename height_from_signed_int()

* Move get_block_template() support functions to a submodule

* Fix incorrect P2SH comments and logs

* Split initial checks into their own functions

* Split state fetch into its own function, do some cleanup

* Move get_block_template_opts to get_block_template::parameters

* Fix and simplify test imports

* Rename block_height to next_block_height

* Rename to chain_tip_and_local_time to make it clear what it contains

* Split fetching mempool transactions out, include them in long poll id

* Refactor coinbase generation

* Split default root calculation into a separate function

* Use DateTime32 for getblocktemplate times

* Use typed difficulty fields rather than strings

* Split out a generate coinbase and roots function

* Move GetBlockTemplate construction into a method

* Document what happens to unusual difficulty values

* Clean up some TODOs

* fastmod check_address check_miner_address

* cargo fmt --all

* Fix an incorrect panic message

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-12-13 21:25:04 +00:00
teor 2041fda7bb
fix(test): Reduce verbose test logs (#5825)
* Remove verbose continuous_blockchain test logs

* Downgrade verbose zebra-network logs to debug

* Downgrade some state logs to debug during tests

* Mark were we would add always-on log filters, if we needed to

* Reduce the number of mempool property tests, to reduce logging
2022-12-08 23:56:01 +00:00
teor 09836d2800
fix(clippy): Put Rust format variables inline (#5783)
* cargo clippy --fix --all-features --all-targets

With rustc 1.67.0-nightly (234151769 2022-12-03)

* cargo fmt --all
2022-12-08 01:05:57 +00:00
Arya eb0a2ef581
change(mempool): Contextually validates mempool transactions in best chain (#5716)
* updates comments

* adds check nullifier no dup fns for transactions

* Adds:
- check::anchors fn for tx iter
- TODO comments for unifying nullifiers and anchors checks
- new state request

Updates unknown anchor errors to accomodate tx-only check

Calls new state fn from transaction verifier

* updates check::anchors fns to use transactions

updates TransactionContextualValidity request to check sprout anchors

adds comment mentioning TransactionContextualValidity ignores UTXOs

* conditions new state req call on is_mempool

updates tests

* fix doc link / lint error

* checks for duplicate nullifiers with closures

* Update zebra-state/src/service/check/nullifier.rs

Co-authored-by: teor <teor@riseup.net>

* documents find_duplicate_nullifier params

moves if let statement into for loop

* renames new state req/res

* asserts correct response variant in tx verifier

* adds CheckBestChainTipShieldedSpends call in tx verifier to async checks

* re-adds tracing instrumentation to check::anchors fn

renames transaction_in_state to transaction_in_chain

* adds block/tx wrapper fns for anchors checks

* uses UnminedTx instead of transaction.hash()

deletes broken test

* updates new state req/res name

* updates tests and uses par_iter for anchors checks

* Updates check::anchors pub fn docs.

* Adds:
- comments / docs
- a TransactionError variant for ValidateContextError

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* moves downcast to From impl

rustfmt

* moves the ValidateContextError into an Arc

updates comments and naming

* leaves par_iter for another PR

* puts io::Error in an Arc

* updates anchors tests to call tx_anchors check

* updates tests to call tx_no_duplicates_in_chain

slightly improves formatting

* Update zebra-consensus/src/error.rs

Co-authored-by: teor <teor@riseup.net>

* moves Arc from HistoryError to ValidateContextError

Co-authored-by: teor <teor@riseup.net>
2022-11-30 04:40:15 +00:00
dependabot[bot] 2680e3c6b2
build(deps): bump chrono from 0.4.22 to 0.4.23 (#5629)
* build(deps): bump chrono from 0.4.22 to 0.4.23

Bumps [chrono](https://github.com/chronotope/chrono) from 0.4.22 to 0.4.23.
- [Release notes](https://github.com/chronotope/chrono/releases)
- [Changelog](https://github.com/chronotope/chrono/blob/main/CHANGELOG.md)
- [Commits](https://github.com/chronotope/chrono/compare/v0.4.22...v0.4.23)

---
updated-dependencies:
- dependency-name: chrono
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* uses Utx::timestamp_opt instead of timestamp

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: arya2 <aryasolhi@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-11-16 16:47:45 +00:00
Alfredo Garcia 4ccd0741b6
feature(rpc): add fixed values to getblocktemplate response (#5558)
* add fixed values to getblocktemplate rpc call response

* suggestion: Avoid new uses of lazy_static (#5559)

* Avoid using lazy_static

* Add some missing documentation

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* minor fixes

* move docs to struct

* add fixed values to coinbase tx

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
2022-11-08 22:55:11 +00:00
teor 71f5e63e64
3. change(rpc): Add fee and sigops fields to getblocktemplate transactions (#5508)
* Add a legacy_sigop_count field to VerifiedUnminedTx

* Add conversions from Vec<VerifiedUnminedTx> to block header roots

* Add fee and sigops field to block template transactions

* Fix up mempool request names

* Increase existing snapshot test coverage

* Document a new method parameter

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2022-11-03 17:03:41 +00:00
teor 142411508b
2. change(rpc): Add some transaction fields to the `getblocktemplate` RPC (#5496)
* Add documentation for the getblocktemplate RPC

* Add a new mempool::Request::Transactions

* Add conversions from Vec<UnminedTx> to merkle::Root and AuthDataRoot

* Fill in the merkle root and auth data root fields

* Delete the Coinbase type, it's the same as Transaction

* Fill in some other existing types

* Add Hex serialization support to some zebra-chain types

* Add TransactionTemplate fields and fill some in

* Fix test hangs by spawning async tasks

* Add temporary workaround for no transactions in the block

* Encode hashes and roots as hex

* Update RPC snapshots

* Add a missing Request::Transactions handler

* Fix doc warnings

* Fix fee serialization

* Update snapshots for serialization changes

* Add a missing Cargo.lock change

* Change depends type

* Remove duplicate feature entry

* Document the new RPC feature

* Fix a comment typo

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Update default roots docs

* Fix comment typo

* Fix a comment typo

Co-authored-by: Arya <aryasolhi@gmail.com>

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: Arya <aryasolhi@gmail.com>
2022-11-03 03:25:01 +00:00
teor c812f880cf
cleanup(clippy): Use inline format strings (#5489)
* Inline format strings using an automated clippy fix

```sh
cargo clippy --fix --all-features --all-targets -- -A clippy::all -W clippy::uninlined_format_args
cargo fmt --all
```

* Remove unused & and &mut using an automated clippy fix

```sh
cargo clippy --fix --all-features --all-targets -- -A clippy::all -W clippy::uninlined_format_args
```
2022-10-27 13:25:18 +00:00
Alfredo Garcia 233220ace0
remove dead code from zebra-chain (#5464)
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-10-25 03:23:50 +00:00
teor ea34baa8b8
add(test): test disabled `lightwalletd` mempool gRPCs via zebrad logs (#5016)
* add grpc mempool test research

* add a config flag for mempool injection of transactions in test

* Only copy the inner state directory in the send transactions test

* Preload Zcash parameters in some transaction verification tests

* Add a block and transaction Hash method to convert from display order bytes

* Update test coverage docs

* Add debugging output for mempool transaction verification

* Test fetching sent mempool transactions using gRPC

* Add extra log checks to the send transaction test

* Wait for zebrad mempool activation before running gRPC tests

* Update send transaction test for lightwalletd not returning mempool transactions

* Check zebrad logs instead of disabled lightwalletd gRPCs

* Add a debug option that makes RPCs pretend the sync is finished

* Remove an unused debug option

* Remove unused test code and downgrade some logs

* Fix test log checks

* Fix some rustdoc warnings

* Fix a compilation error due to new function arguments

* Make zebrad sync timeouts consistent and remove outdated code

* Document how to increase temporary directory space for tests

* Stop checking for a log that doesn't always happen

* Remove some commented-out code

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Update a comment about run time

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Add new config to new tests from the `main` branch

* Add transactions to the list, rather than replacing the list with each new block

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2022-09-06 13:32:33 +00:00
teor 66ea40f215
Resolve unnecessary borrow lint and remove redundant imports (#5045)
```sh
cargo clippy --fix --all-features --all-targets
```
2022-09-02 18:20:32 +00:00
Conrado Gouvea 6fd750e168
build(deps): bump insta from 1.15.0 to 1.17.1 (#4884)
* build(deps): bump insta from 1.15.0 to 1.17.1

Bumps [insta](https://github.com/mitsuhiko/insta) from 1.15.0 to 1.17.1.
- [Release notes](https://github.com/mitsuhiko/insta/releases)
- [Changelog](https://github.com/mitsuhiko/insta/blob/master/CHANGELOG.md)
- [Commits](https://github.com/mitsuhiko/insta/compare/1.15.0...1.17.1)

---
updated-dependencies:
- dependency-name: insta
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* make zebra_test::init() return the insta drop guard

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-08-04 15:44:44 +00:00
teor 6ad445eb97
1. fix(perf): Run CPU-intensive state updates in parallel rayon threads (#4802)
* Split disk reads from CPU-heavy Sprout interstitial tree cryptography

* Improve anchor validation debugging and error messages

* Work around a test data bug, and save some CPU

* Remove redundant checks for empty shielded data

* Skip generating unused interstitial treestates

* Do disk fetches and quick checks, then CPU-heavy cryptography

* Wrap HistoryTree in an Arc in the state

* Run CPU-intensive chain validation and updates in parallel rayon threads

* Refactor to prepare for parallel tree root calculations

* Run finalized state note commitment tree root updates in parallel rayon threads

* Update finalized state note commitment trees using parallel rayon threads

* Fix a comment typo and add a TODO

* Split sprout treestate fetch into its own function

* Move parallel note commitment trees to zebra-chain

* Re-calculate the tree roots in the same parallel batches

* Do non-finalized note commitment tree updates in parallel threads

* Update comments about note commitment tree rebuilds

* Do post-fork tree updates in parallel threads

* Add a TODO for parallel tree updates in tests

* Fix broken intra-doc links

* Clarify documentation for sprout treestates

* Sort Cargo.toml dependencies
2022-07-22 12:19:11 -04:00
teor cbb3232769
Only fetch block headers from the database to answer headers requests (#4792) 2022-07-22 09:15:22 +10:00
teor 42ef8846b3
fix(clippy): resolve or disable new nightly clippy lints (#4739)
* Fix new dead_code lints in generated pin-project code

* Fix and ignore new needless_borrow lints
2022-07-05 11:04:47 +10:00
Alfredo Garcia 97fb85dca9
lint(clippy): add `unwrap_in_result` lint (#4667)
* `unwrap_in_result` in zebra-chain crate

* `unwrap_in_result` in zebra-script crate

* `unwrap_in_result` in zebra-state crate

* `unwrap_in_result` in zebra-consensus crate

* `unwrap_in_result` in zebra-test crate

* `unwrap_in_result` in zebra-network crate

* `unwrap_in_result` in zebra-rpc crate

* `unwrap_in_result` in zebrad crate

* rustfmt

* revert `?` and add exceptions

* explain some panics better

* move some lint positions

* replace a panic with error

* Fix rustfmt?

Co-authored-by: teor <teor@riseup.net>
2022-06-28 06:22:07 +00:00
Marek b8b35f8da9
fix(doc): Fix various doc warnings, part 1 (#4514)
* Fix the syntax of links in comments

* Fix a mistake in the docs

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Remove unnecessary angle brackets from a link

* Revert the changes for links that serve as references

* Revert "Revert the changes for links that serve as references"

This reverts commit 8b091aa9fab453e7d3559a5d474e0879183b9bfb.

* Remove `<` `>` from links that serve as references

This reverts commit 046ef25620ae1a2140760ae7ea379deecb4b583c.

* Don't use `<` `>` in normal comments

* Don't use `<` `>` for normal comments

* Revert changes for comments starting with `//`

* Fix some warnings produced by `cargo doc`

* Fix some rustdoc warnings

* Fix some warnings

* Refactor some changes

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2022-06-02 15:07:35 +00:00
Marek 6f896ef5a5
fix(doc): Fix the syntax of links in comments (#4494)
* Fix the syntax of links in comments

* Fix a mistake in the docs

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Remove unnecessary angle brackets from a link

* Revert the changes for links that serve as references

* Revert "Revert the changes for links that serve as references"

This reverts commit 8b091aa9fab453e7d3559a5d474e0879183b9bfb.

* Remove `<` `>` from links that serve as references

This reverts commit 046ef25620ae1a2140760ae7ea379deecb4b583c.

* Don't use `<` `>` in normal comments

* Don't use `<` `>` for normal comments

* Revert changes for comments starting with `//`

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2022-05-30 20:12:11 +00:00
Conrado Gouvea 95f14ffdcd
add support for getblock with verbosity=1 (#4511) 2022-05-27 09:41:11 +00:00
teor be91ab29ee
fix(clippy): resolve various clippy warnings (#4473)
* clippy: unused import on non-linux platforms

* Fix some instances of clippy::derive_partial_eq_without_eq

* Move a deref to fix clippy::significant_drop_in_scrutinee

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-05-25 20:56:32 +00:00
Marek 7c726b246d
feat(rpc): Implement `z_gettreestate` RPC (#3990)
* Impl the elementary structure of the `z_gettreestate` RPC

* Fix merging bugs

* Fix a merge bug

* Fix a merge bug

* Move a derive attribute

Co-authored-by: teor <teor@riseup.net>

* Clarify the support of negative heights

* Add Orchard note commitment trees to the response

* Add the time to the response

* Finalize the `z_gettreestate` RPC

* Add a note that verified blocks have coinbase height

* Refactor `from_str` for `HashOrHeight`

* Fix a mistake in the docs

Co-authored-by: teor <teor@riseup.net>

* Clarify request types

Co-authored-by: teor <teor@riseup.net>

* Simplify `hash_or_height` conversion to height

Co-authored-by: teor <teor@riseup.net>

* Add a TODO about optimization

Co-authored-by: teor <teor@riseup.net>

* Add a doc comment

* Make sure Sapling & Orchard trees don't get mixed up

* Serialize Sapling commitment trees

* Refactor some comments

* Serialize Orchard commitment trees

* Serialize block heights

* Simplify the serialization of commitment trees

* Remove the block time from the RPC response

* Simplify the serialization of block heights

* Put Sapling & Orchard requests together

* Remove a redundant TODO

* Add block times to the RPC response

* Derive `Clone, Debug, Eq, PartialEq` for `GetTreestate`

Co-authored-by: teor <teor@riseup.net>

* Derive `Clone`, `Debug`, `Eq` and `PartialEq` for `SerializedTree`

* Document the fields of `GetTreestate`

* Skip the serialization of empty trees

This ensures compatibility with `zcashd` in the `z_gettreestate` RPC.

* Document the `impl` of `merkle_tree::Hashable` for nodes

* Make the structure of the JSON response consistent with `zcashd`

* Derive `Eq` for nodes

Co-authored-by: teor <teor@riseup.net>

* Convert Sapling commitment trees to a format compatible with zcashd

* Refactor the conversion of Sapling commitment trees

* Refactor some comments

* Refactor comments

* Add a description of the conversion

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>

* Fix comment indenting

* Document the conversion between the dense and sparse formats

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2022-05-12 07:00:12 +00:00
teor e49c1d7034
feat(state): add transparent address indexes to the non-finalized state (#4022)
* Derive Hash for transparent address index types

* Expose some types used by transparent address indexes

* Add an empty transparent transfers type for transparent address indexes

* Update TransparentTransfers with created UTXOs

* Add spent transparent outputs to ContextuallyValidBlock

* Update TransparentTransfers with spent transparent outputs

* Ignore missing spent outputs, so that tests pass

* Remove empty TransparentTransfers after a spend revert

* Update TransparentTransfers with creating and spending transaction IDs

* Ignore duplicate created UTXOs, so that tests pass

* Add some TODO comments

* Remove accidental doctest formatting

* Add address transfers index accessor methods

* Use TransactionLocation in the non-finalized state

* Apply more address index assertions to production code

* Refactor deeply nested code and apply more assertions

* Return UTXOs in chain order

* Return transaction hashes in chain order

* Stop indexing each transparent output multiple times

* Run some more asserts during tests

* Tidy TODO comments

* Fix an incorrect assert condition

* Use OrderedUtxos so that spent UTXOs can be stored in chain order

* Update tests to use OrderedUtxos

* Update the index API for the getaddressutxos query

* Remove redundant arguments in tests

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-04-12 17:21:46 +00:00
teor b5efeb4ab3
Encode CommitmentErrors as hex (#4021) 2022-04-01 22:31:03 +00:00
Conrado Gouvea b9640fbb55
fix(chain): make FromHex consistent with ToHex for tx/block hashes (#3893) 2022-03-17 20:24:48 +00:00
Alfredo Garcia f39ac48c59
feature(rpc): Implement `getbestblockhash` method (#3754)
* feature(rpc): start adding a `getblock` method

* fix(rpc): replace oneshot

* fix(rpc): replace a panic with error

* fix(rpc): fix test

* feature(rpc): add hex to response

* refactor(rpc): use generic instead of alias

* docs(rpc): improve docs for getblock method

* test(rpc): add a test for getblock method

* deps(rpc): remove non needed tower features

Co-authored-by: teor <teor@riseup.net>

* docs(rpc): add a note to getblock doc

* refactor(rpc): replace alias

* fix(rpc): use `zcash_serialize_to_vec()` instead of logging format

* tests(rpc): add network argument to `populated_state()`

* refactor(rpc): use an error for state service readiness

* fix(rpc): add parameter

* fix(rpc): clippy

* nit(rpc): remove new line from imports

* fix(rpc): remove commented code

* fix(rpc): simplify error

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* Use a `SerializedBlock` type to help serializing blocks (#3725)

* Create a `SerializedBlock` helper type

Create a type that can be used as a byte slice, but is guaranteed to
represent a valid block.

* Use `into_iter` instead of `iter`

There's no need to borrow the elements, they can be moved out directly.
This will be necessary because `&Arc<T>` doesn't implement `Borrow<T>`,
so a `SerializedBlock` can't be built directly from an `&Arc<Block>`.

* Use `SerializedBlock` in `GetBlock`

Make the type stricter to avoid storing possibly invalid values. The
bytes are still serialized as a hexadecimal string, through the usage of
`hex`.

The `serde::Deserialize` can't be derived because `hex` requires the
type to also implement `FromHex`.

* feature(rpc): add suggestions from code review

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* tests(rpc): make sure mempool has no requests in get_block test

* fix(rpc): change height argument type in getblock method

* fix(rpc): rustfmt

* fix(rpc): replace panic

* fix(rpc): change getblock response

* fix(rpc): fix lightwalletd test

* tests(rpc): add a getblock error test

* fix(rpc): try another regex

* feature(rpc): add `getbestblockhash` RPC method

* feature(rpc): Add a `pub struct SerializedBlockHash` type

* tests(rpc): add a unit test for `getbestblockhash` method

* tests(rpc): make sure no requests are sent to mempool in getbestblockhash test

* tests(rpc): refactor check

Co-authored-by: teor <teor@riseup.net>

* fix(rpc): fixes after rebase

* refactor(rpc): refactor `GetBestBlockHash`

* fix(rpc): unused variables

Co-authored-by: teor <teor@riseup.net>

* docs(rpc): update

* fix(rpc): add panic

* fix(rpc): fix panic

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2022-03-11 05:13:08 +00:00
Alfredo Garcia 833560411f
feature(rpc): implement getblock api call (#3707)
* feature(rpc): start adding a `getblock` method

* fix(rpc): replace oneshot

* fix(rpc): replace a panic with error

* fix(rpc): fix test

* feature(rpc): add hex to response

* refactor(rpc): use generic instead of alias

* docs(rpc): improve docs for getblock method

* test(rpc): add a test for getblock method

* deps(rpc): remove non needed tower features

Co-authored-by: teor <teor@riseup.net>

* docs(rpc): add a note to getblock doc

* refactor(rpc): replace alias

* fix(rpc): use `zcash_serialize_to_vec()` instead of logging format

* tests(rpc): add network argument to `populated_state()`

* refactor(rpc): use an error for state service readiness

* fix(rpc): add parameter

* fix(rpc): clippy

* nit(rpc): remove new line from imports

* fix(rpc): remove commented code

* fix(rpc): simplify error

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* Use a `SerializedBlock` type to help serializing blocks (#3725)

* Create a `SerializedBlock` helper type

Create a type that can be used as a byte slice, but is guaranteed to
represent a valid block.

* Use `into_iter` instead of `iter`

There's no need to borrow the elements, they can be moved out directly.
This will be necessary because `&Arc<T>` doesn't implement `Borrow<T>`,
so a `SerializedBlock` can't be built directly from an `&Arc<Block>`.

* Use `SerializedBlock` in `GetBlock`

Make the type stricter to avoid storing possibly invalid values. The
bytes are still serialized as a hexadecimal string, through the usage of
`hex`.

The `serde::Deserialize` can't be derived because `hex` requires the
type to also implement `FromHex`.

* feature(rpc): add suggestions from code review

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* tests(rpc): make sure mempool has no requests in get_block test

* fix(rpc): change height argument type in getblock method

* fix(rpc): rustfmt

* fix(rpc): replace panic

* fix(rpc): change getblock response

* fix(rpc): fix lightwalletd test

* tests(rpc): add a getblock error test

* fix(rpc): try another regex

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2022-03-10 01:12:41 +00:00
teor 7283b4bfd0
4. test(db): add large transaction tests (#3759)
* refactor(test/block): rename large single transaction function

```sh
fastmod single_transaction_block single_transaction_block_many_inputs
```

* rustfmt

* test(block): add a test block with many transparent outputs

* doc(db): explain why we can't just get the UTXOs right before they are deleted

* refactor(db): split out a block data write method

* refactor(block): add a height argument to new_outputs

* test(db): add block and transaction round-trip tests

Including large blocks and transactions.

* test(db): fix large block serialization instability in the tests

* doc(block): add TODOs for generating correct blocks

* Make transparent output functions which take a height test-only

* make sure generated blocks are actually over/under-sized

* replace println!() with an error!() log
2022-03-09 23:34:50 +00:00
Conrado Gouvea bb232ccb74
fix: generate well-formed finalSaplingRoot in Arbitrary implementation (#3573)
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-02-17 20:10:13 +00:00
Alfredo Garcia 137ae4e041
refactor(anchorSapling): Change type to force consensus rule validation (#3544)
* change `anchorSapling` type

* implement PartialEq manually for clippy

* use `unique_by` in place of `sorted`

* replace panic with new error

* improve some serialize/deserialize calls for sapling anchors

* fix arbitrary for sapling::tree::Root

* remove dedup()

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-02-17 03:20:22 +00:00
Marek 683b88c819
Support large block heights (#3401)
* Support large block heights

* Document consensus rules referring to expiry heights

* Refactor the docs

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Fix the formatting of an error message

* refactor: Simplify coinbase expiry code so the consensus rule is clear (#3408)

* Fix some outdated TODO comments

* refactor(coinbase expiry): Simplify the code so consensus rule is clear

* Fix the formatting of an error message

* Remove a redundant comment

Co-authored-by: Marek <mail@marek.onl>

Co-authored-by: Marek <mail@marek.onl>

* Check the max expiry height at parse time

* Test that 2^31 - 1 is the last valid height

* Add tests for nExpiryHeight

* Add tests for expiry heights of V4 transactions

* Add tests for V5 transactions

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-02-11 00:32:57 +00:00
teor 4f0d7bd737
lint: add extra integer lints, and partially fix some code (#3409)
* lint: enable more clippy checks for bug-prone code

* fix(lint): stop denying lints, to avoid being excluded from Crater

Also categorise lints.

* lint: add some lints to the TODO list

* refactor(arithmetic): partial fixes for some integer arithmetic lints

* Document some weird lint behaviour
2022-01-27 11:34:15 -03:00