Commit Graph

530 Commits

Author SHA1 Message Date
Marek 32faa94fb4
fix(state): Update column family names to match Zebra's database design (#4639)
* Rename `block_by_height` to `block_header_by_height` in fin state

* Rename `tx_by_hash` to `tx_loc_by_hash` in both (non & fin) states

* Rename `utxo_by_outpoint` to `utxo_by_out_loc` in finalized state

* Reorder the column families so that they match the docs

* Update `struct Chain` in the RFCs

* Increment `DATABASE_FORMAT_VERSION` to 25

* Remove obsolete docs from `0004-asynchronous-script-verification.md`

* Remove an obsolete `TODO` from `disk_db.rs`

* Delete unused snapshots

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-06-30 23:00:23 +00:00
teor 9e277ed91a
Release Zebra 1.0.0-beta.12 (#4714)
* Update Zebra to 1.0.0-beta.12

* Simplify tower-* versioning by updating both to 0.2.27

* Simplify version search and replace by removing an old version from the docs

* Update Cargo.lock

* Add draft changelog as of PR #4693

* Update CHANGELOG to PR #4680
2022-06-29 17:14:16 +00:00
Alfredo Garcia 97fb85dca9
lint(clippy): add `unwrap_in_result` lint (#4667)
* `unwrap_in_result` in zebra-chain crate

* `unwrap_in_result` in zebra-script crate

* `unwrap_in_result` in zebra-state crate

* `unwrap_in_result` in zebra-consensus crate

* `unwrap_in_result` in zebra-test crate

* `unwrap_in_result` in zebra-network crate

* `unwrap_in_result` in zebra-rpc crate

* `unwrap_in_result` in zebrad crate

* rustfmt

* revert `?` and add exceptions

* explain some panics better

* move some lint positions

* replace a panic with error

* Fix rustfmt?

Co-authored-by: teor <teor@riseup.net>
2022-06-28 06:22:07 +00:00
Conrado Gouvea 6b22794396
deps: upgrade ECC dependencies (#4699) 2022-06-27 23:12:56 +00:00
dependabot[bot] 82b8d7f95d
build(deps): bump tower from 0.4.12 to 0.4.13 (#4644)
Bumps [tower](https://github.com/tower-rs/tower) from 0.4.12 to 0.4.13.
- [Release notes](https://github.com/tower-rs/tower/releases)
- [Commits](https://github.com/tower-rs/tower/compare/tower-0.4.12...tower-0.4.13)

---
updated-dependencies:
- dependency-name: tower
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-06-21 17:12:19 +00:00
dependabot[bot] 25396bc6e9
build(deps): bump insta from 1.14.1 to 1.15.0 (#4645)
Bumps [insta](https://github.com/mitsuhiko/insta) from 1.14.1 to 1.15.0.
- [Release notes](https://github.com/mitsuhiko/insta/releases)
- [Changelog](https://github.com/mitsuhiko/insta/blob/master/CHANGELOG.md)
- [Commits](https://github.com/mitsuhiko/insta/compare/1.14.1...1.15.0)

---
updated-dependencies:
- dependency-name: insta
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-06-21 03:38:29 +00:00
Alfredo Garcia 769d069d0a
feat(state): delete old database directories (#4586)
* delete old database directories

* check if state directory exists

* skip deleting when ephemeral

* split `check_and_delete_old_databases`

* move `check_and_delete_old_databases` to state

* spawn `check_and_delete_old_databases`

* simplity a bit

* fix(state): only delete old database directories inside the cache directory (#4631)

* Add function comments, tweak log

* Simplify version parsing

* Use spawn_blocking to launch the task on a separate thread, do the cleanup last

* Abort the cleanup task when Zebra exits

* Split directory deletion into its own function, handle ownership

* Rename cache_dir to state_dir

* If an outdated state directory is outside the cache directory, don't delete it

* Minimise diffs

* add test

* fix typos

Co-authored-by: teor <teor@riseup.net>

* add `canonicalize` to test regex

* add another match to test

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-06-21 00:59:51 +00:00
Marek b91aaf7863
Use `config.toml` instead of inner attributes for private links (#4627) 2022-06-15 23:21:27 +00:00
teor 83b4e6f975
feat(diagnostics): Add tokio-console support to zebrad (#4519)
* Always activate tokio/tracing feature

And always build tests with all tokio features.

* Refactor tracing-subscriber init to simplify it

* Add the tokio-console feature and dependencies

* Add optional tokio-console support, and log the installed tracing layers at info level

Uses a tracing Registry for tokio-console, and a fmt::Subscriber otherwise.

* Add some TODOs based on tracing-subscriber features

* Fix up some spans

* Add a TODO for fixing a log filter bug in tokio-console mode
2022-06-15 16:43:20 +10:00
Marek cc75c3f5f9
fix(doc): Fix various doc warnings, part 3 (#4611)
* Fix the syntax of links in comments

* Fix a mistake in the docs

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Remove unnecessary angle brackets from a link

* Revert the changes for links that serve as references

* Revert "Revert the changes for links that serve as references"

This reverts commit 8b091aa9fab453e7d3559a5d474e0879183b9bfb.

* Remove `<` `>` from links that serve as references

This reverts commit 046ef25620ae1a2140760ae7ea379deecb4b583c.

* Don't use `<` `>` in normal comments

* Don't use `<` `>` for normal comments

* Revert changes for comments starting with `//`

* Fix some warnings produced by `cargo doc`

* Fix some rustdoc warnings

* Fix some warnings

* Refactor some changes

* Fix some rustdoc warnings

* Fix some rustdoc warnings

* Resolve various TODOs

Co-authored-by: teor <teor@riseup.net>

* Fix some unresolved links

* Allow links to private items

* Fix some unresolved links

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-06-15 03:57:19 +00:00
teor 3825caae03
fix(clippy): Resolve some lifetime and reference lints (#4578)
* Fix significant drop in match scrutinee

https://github.com/rust-lang/rust/issues/93883

* Fix deref immutable value

* Fix explicit 0 index when first() would do

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-06-14 06:21:07 +00:00
dependabot[bot] 8d0a1b011c
build(deps): bump tokio from 1.18.2 to 1.19.2 (#4579)
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.18.2 to 1.19.2.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.18.2...tokio-1.19.2)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-06-14 06:20:48 +00:00
Marek 2e50ccc8f3
fix(doc): Fix various doc warnings, part 2 (#4561)
* Fix the syntax of links in comments

* Fix a mistake in the docs

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Remove unnecessary angle brackets from a link

* Revert the changes for links that serve as references

* Revert "Revert the changes for links that serve as references"

This reverts commit 8b091aa9fab453e7d3559a5d474e0879183b9bfb.

* Remove `<` `>` from links that serve as references

This reverts commit 046ef25620ae1a2140760ae7ea379deecb4b583c.

* Don't use `<` `>` in normal comments

* Don't use `<` `>` for normal comments

* Revert changes for comments starting with `//`

* Fix some warnings produced by `cargo doc`

* Fix some rustdoc warnings

* Fix some warnings

* Refactor some changes

* Fix some rustdoc warnings

* Fix some rustdoc warnings

* Resolve various TODOs

Co-authored-by: teor <teor@riseup.net>

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-06-14 01:22:16 +00:00
Marek 0e4c51c431
V1.0.0 beta.11 release (#4567)
* Bump crate versions

* Increment the protocol user agent

* Increment the version in `README.md`

* Increment the version in `install.md`

* Update `README.md`

* Update the changelog

* Increment the versions of crates that depend on other crates

* Update `Cargo.lock`
2022-06-03 22:55:05 +00:00
dependabot[bot] 6c0e74924f
build(deps): bump insta from 1.14.0 to 1.14.1 (#4542)
Bumps [insta](https://github.com/mitsuhiko/insta) from 1.14.0 to 1.14.1.
- [Release notes](https://github.com/mitsuhiko/insta/releases)
- [Changelog](https://github.com/mitsuhiko/insta/blob/master/CHANGELOG.md)
- [Commits](https://github.com/mitsuhiko/insta/compare/1.14.0...1.14.1)

---
updated-dependencies:
- dependency-name: insta
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-06-03 15:01:05 +10:00
Marek b8b35f8da9
fix(doc): Fix various doc warnings, part 1 (#4514)
* Fix the syntax of links in comments

* Fix a mistake in the docs

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Remove unnecessary angle brackets from a link

* Revert the changes for links that serve as references

* Revert "Revert the changes for links that serve as references"

This reverts commit 8b091aa9fab453e7d3559a5d474e0879183b9bfb.

* Remove `<` `>` from links that serve as references

This reverts commit 046ef25620ae1a2140760ae7ea379deecb4b583c.

* Don't use `<` `>` in normal comments

* Don't use `<` `>` for normal comments

* Revert changes for comments starting with `//`

* Fix some warnings produced by `cargo doc`

* Fix some rustdoc warnings

* Fix some warnings

* Refactor some changes

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2022-06-02 15:07:35 +00:00
teor ca0520b2e8
change(deps): Upgrade tracing-subscriber and related dependencies (#4517)
* Upgrade tracing and related dependencies

```sh
cargo upgrade --workspace
tracing-error
tracing-subscrber

color-eyre

tracing-flame
tracing-journald

sentry
sentry-tracing

metrics
metrics-exporter-prometheus
reqwest
```

* Update duplicate dependency checks

* Enable the tracing/env-filter feature

* Fix type inference for metrics

Manual changes, plus:
```sh
fastmod "as _" "as f64"
```

* Tidy up some unrelated test code

* Update metrics-exporter-prometheus API

And make unused dependencies optional.

* Adjust test regexes to new tracing format

Also fix some regex bugs, and refactor to simplify.

* Disable color-eyre span traces and track caller in release builds

* Add a feature that enables extra debugging in release builds

* Clean up some redundant features

* Increase a test timeout
2022-06-01 13:53:51 +10:00
Marek 6f896ef5a5
fix(doc): Fix the syntax of links in comments (#4494)
* Fix the syntax of links in comments

* Fix a mistake in the docs

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Remove unnecessary angle brackets from a link

* Revert the changes for links that serve as references

* Revert "Revert the changes for links that serve as references"

This reverts commit 8b091aa9fab453e7d3559a5d474e0879183b9bfb.

* Remove `<` `>` from links that serve as references

This reverts commit 046ef25620ae1a2140760ae7ea379deecb4b583c.

* Don't use `<` `>` in normal comments

* Don't use `<` `>` for normal comments

* Revert changes for comments starting with `//`

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2022-05-30 20:12:11 +00:00
dependabot[bot] 715d204939
build(deps): bump regex from 1.5.5 to 1.5.6 (#4463)
Bumps [regex](https://github.com/rust-lang/regex) from 1.5.5 to 1.5.6.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/1.5.5...1.5.6)

---
updated-dependencies:
- dependency-name: regex
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-05-26 08:12:55 +00:00
dependabot[bot] 2ee52f808c
build(deps): bump once_cell from 1.11.0 to 1.12.0 (#4462)
Bumps [once_cell](https://github.com/matklad/once_cell) from 1.11.0 to 1.12.0.
- [Release notes](https://github.com/matklad/once_cell/releases)
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md)
- [Commits](https://github.com/matklad/once_cell/compare/v1.11.0...v1.12.0)

---
updated-dependencies:
- dependency-name: once_cell
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-05-25 20:56:12 +00:00
dependabot[bot] 9425cb3f1c
build(deps): bump once_cell from 1.10.0 to 1.11.0 (#4447)
Bumps [once_cell](https://github.com/matklad/once_cell) from 1.10.0 to 1.11.0.
- [Release notes](https://github.com/matklad/once_cell/releases)
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md)
- [Commits](https://github.com/matklad/once_cell/compare/v1.10.0...v1.11.0)

---
updated-dependencies:
- dependency-name: once_cell
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-05-21 23:49:36 +00:00
teor 2439bed3d2
2. fix(state): index spending transaction IDs for each address (#4355)
* Make jobs that use cached state wait for state rebuilds

* Run jobs that need cached state even if the rebuild was skipped

* Fix missing dependencies

And update a TODO

* Split writing transaction indexes into transparent and shielded

* Split writing transparent indexes into created and spent

* Correctly populate spending address transaction ID indexes

* Increment the database format to rebuild address tx ID indexes

* Update non-finalized docs to prevent similar bugs

* Fix a comment

* Make jobs that use cached state wait for state rebuilds

* Run jobs that need cached state even if the rebuild was skipped

* Fix missing dependencies

And update a TODO

* refactor(ci): look for available disks instead of files changed

This ensure that if the constants.rs file was changed, we search for disks available in the whole repository with the same state.

If there's no disk available a rebuild is triggered depending the missing disk. And if there's a disk available, tests are run with this one.

* fix(ci): lwd syncs needs to wait for zebra disk rebuild

* docs(ci): use better comments on integration tests

* fix(ci): we must authenticate to GCP to find disks

* fix(ci): add needed permissions for google auth

* fix(ci): the output needs to be echoed

* imp(ci): reduce diff with main

* fix(ci): remove redundant dependency

Co-authored-by: teor <teor@riseup.net>

* fix(ci): also add `false` to the JSON object output

* fix(ci): hasty copy/paste

* force a push event

* fix(ci): standardize comments

* fix(ci): run disk rebuilds if no disk was found

* fix(ci): do not restrict on push

* fix(ci): build on any event if a cached disk is not found

* fix(ci): sync .patch file with changes on the workflow

Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
2022-05-20 02:22:01 +00:00
Alfredo Garcia 8150a720ca
release: bump versions and create changelog entry for `v1.0.0-beta.10` (#4430)
* bump versions

* add the changelog for beta 10

* Update CHANGELOG.md

Co-authored-by: Marek <mail@marek.onl>

* Distinguish CI and Rust dependency updates

* Include all the dep bumps in PR #4405

* join 2 dependencies

* Include incrementalmerkletree dep bump in changelog

Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
Co-authored-by: Marek <mail@marek.onl>
2022-05-19 19:23:30 +00:00
Conrado Gouvea 55822dda41
change: NU5 mainnet dependency upgrades (#4405)
* upgrade dependencies to NU5 mainnet versions

* migrate to bitvec 1.0

* update deny.toml

* point to published redjubjub 0.5

* Point at zcash-script@0.1.6 on crates.io

* update Cargo.lock

* specify patch versions of dependencies

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
2022-05-19 01:03:55 +00:00
Marek 7c726b246d
feat(rpc): Implement `z_gettreestate` RPC (#3990)
* Impl the elementary structure of the `z_gettreestate` RPC

* Fix merging bugs

* Fix a merge bug

* Fix a merge bug

* Move a derive attribute

Co-authored-by: teor <teor@riseup.net>

* Clarify the support of negative heights

* Add Orchard note commitment trees to the response

* Add the time to the response

* Finalize the `z_gettreestate` RPC

* Add a note that verified blocks have coinbase height

* Refactor `from_str` for `HashOrHeight`

* Fix a mistake in the docs

Co-authored-by: teor <teor@riseup.net>

* Clarify request types

Co-authored-by: teor <teor@riseup.net>

* Simplify `hash_or_height` conversion to height

Co-authored-by: teor <teor@riseup.net>

* Add a TODO about optimization

Co-authored-by: teor <teor@riseup.net>

* Add a doc comment

* Make sure Sapling & Orchard trees don't get mixed up

* Serialize Sapling commitment trees

* Refactor some comments

* Serialize Orchard commitment trees

* Serialize block heights

* Simplify the serialization of commitment trees

* Remove the block time from the RPC response

* Simplify the serialization of block heights

* Put Sapling & Orchard requests together

* Remove a redundant TODO

* Add block times to the RPC response

* Derive `Clone, Debug, Eq, PartialEq` for `GetTreestate`

Co-authored-by: teor <teor@riseup.net>

* Derive `Clone`, `Debug`, `Eq` and `PartialEq` for `SerializedTree`

* Document the fields of `GetTreestate`

* Skip the serialization of empty trees

This ensures compatibility with `zcashd` in the `z_gettreestate` RPC.

* Document the `impl` of `merkle_tree::Hashable` for nodes

* Make the structure of the JSON response consistent with `zcashd`

* Derive `Eq` for nodes

Co-authored-by: teor <teor@riseup.net>

* Convert Sapling commitment trees to a format compatible with zcashd

* Refactor the conversion of Sapling commitment trees

* Refactor some comments

* Refactor comments

* Add a description of the conversion

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>

* Fix comment indenting

* Document the conversion between the dense and sparse formats

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2022-05-12 07:00:12 +00:00
teor 12e8130941
fix(state): return non-finalized UTXOs and tx IDs in address queries (#4356)
* Assert that address TxIDs and UTXOs are in chain order

* Always output colour by default in zcash-rpc-diff

* Cross-check getaddressutxos and getaddressbalance in zcash-rpc-diff

* Make balances with no UTXOs match in zcash-rpc-diff

* Add some TODOs

* Display the actual connected node software in zcash-rpc-diff

* Log address UTXOs request summaries

* Log address count for address UTXO requests

* Simplify zcash-rpc-diff node names

* Log chain address UTXOs request processing

* Stop ignoring all non-finalized UTXOs in address queries

* Make zcash-rpc-diff node names more consistent

* Downgrade logs to debug level

* Stop ignoring all non-finalized tx IDs in address queries
2022-05-11 21:43:17 +00:00
dependabot[bot] cb0487cec9
build(deps): bump tokio from 1.18.1 to 1.18.2 (#4358)
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.18.1 to 1.18.2.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.18.1...tokio-1.18.2)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-05-10 03:42:08 +00:00
Conrado Gouvea c190e3027c
v1.0.0-beta.9 release (#4349) 2022-05-09 13:32:53 -03:00
dependabot[bot] 32056e0e5c
build(deps): bump thiserror from 1.0.30 to 1.0.31 (#4259)
Bumps [thiserror](https://github.com/dtolnay/thiserror) from 1.0.30 to 1.0.31.
- [Release notes](https://github.com/dtolnay/thiserror/releases)
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.30...1.0.31)

---
updated-dependencies:
- dependency-name: thiserror
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-05-05 01:08:06 +00:00
dependabot[bot] 2e643c8631
build(deps): bump serde from 1.0.136 to 1.0.137 (#4261)
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.136 to 1.0.137.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.136...v1.0.137)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-05-04 04:00:30 +00:00
dependabot[bot] 47f144205c
build(deps): bump tokio from 1.18.0 to 1.18.1 (#4280)
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.18.0 to 1.18.1.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.18.0...tokio-1.18.1)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-05-03 20:55:43 +00:00
teor be4e065afb
change(rpc): Simplify RPC types and add documentation (#4218)
* Simplify RPC types and add documentation

* Derive serde traits in production code
2022-05-03 04:10:21 +00:00
teor 86b252aad7
fix(state): Tweak ephemeral state directory path and improve docs (#4248)
* Fix state directory and docs

* Explain state versions better, and fix a docs bug

* fix docs

Co-authored-by: Marek <mail@marek.onl>

* add extra info to doc

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: Marek <mail@marek.onl>
2022-04-29 22:02:09 +00:00
dependabot[bot] 1691d9e855
build(deps): bump tokio from 1.17.0 to 1.18.0 (#4242)
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.17.0 to 1.18.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.17.0...tokio-1.18.0)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-28 17:54:33 +00:00
dependabot[bot] 597f55319d
build(deps): bump spandoc from 0.2.1 to 0.2.2 (#4221)
Bumps [spandoc](https://github.com/yaahc/spandoc) from 0.2.1 to 0.2.2.
- [Release notes](https://github.com/yaahc/spandoc/releases)
- [Changelog](https://github.com/yaahc/spandoc/blob/master/CHANGELOG.md)
- [Commits](https://github.com/yaahc/spandoc/compare/v0.2.1...v0.2.2)

---
updated-dependencies:
- dependency-name: spandoc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-28 07:08:30 +00:00
Janito Vaqueiro Ferreira Filho 5a94a09292
add(test): Integration test to send transactions using lightwalletd (#4068)
* Export the `zebra_state::Config::db_path` method

Make it easier for tests to discover the sub-directory used to store
the chain state data.

* Generate code for interfacing with lightwalletd

Use the `tonic-build` crate to generate Rust code for communicating with
lightwalletd using gRPC.

The `*.proto` files were obtained from the Zcash lightwalletd
repository.

* Use `block::Height` instead of `Height`

Import the `block` instead to make it slightly clearer.

* Add helper function to remove a file if it exists

Try to remove it and ignore an error if it says that the file doesn't
exist. This will be used later to remove the lock file from a copied
chain state directory.

* Add helper function to copy chain state dirs

Copy an existing chain state directory into a new temporary directory.

* Add a `BoxStateService` type alias

Make it easier to write and read a boxed version of a state service.

* Add a helper function to start the state service

Make it easier to specify the state service to use an existing state
cache directory.

* Import `eyre!` macro at the module level

Allow it to be used in different places without having to repeat the
imports.

* Add `load_tip_height_from_state_directory` helper

A function to discover the current chain tip height stored in a state
cache.

* Add helper function to prepare partial sync. state

Loads a partially synchronized cached state directory into a temporary
directory that can be used by a zebrad instance, and also returns the
chain tip block height of that state.

* Add `perform_full_sync_starting_from` helper

Runs a zebrad with an existing partially synchronized state, and
finishes synchronizing it to the network chain tip.

* Add function to load transactions from a block

Use a provided state service to load all transactions from a block at a
specified height.

The state service is a generic type parameter, because
`zebra_state::service::ReadStateService` is not exported publicly. Using
a generic type parameter also allows the service to be wrapped in layers
if needed in the future.

* Add `load_transactions_from_block_after` helper

A function to load transactions from a block stored in a cached state
directory. The cached state must be synchronized to a chain tip higher
than the requested height.

* Add helper function to load some test transactions

Given a partially synchronized chain state, it will extend that chain by
performing a full synchronization, and obtain some transactions from one
of the newly added blocks.

* Update `spawn_zebrad_for_rpc_without_initial_peers`

Wait until the mempool is activated.

* Add method to start lightwalletd with RPC server

Returns the lightwalletd instance and the port that it's listening for
RPC connections.

The instance can reuse an existing cached lightwalletd state if the
`LIGHTWALLETD_DATA_DIR` environment variable is set.

* Add a `LightwalletdRpcClient` type alias

To make it easier to identify the type generated from the Protobuf
files.

* Add helper function to connect to lightwalletd

Prepare an RPC client to send requests to a lightwalletd instance.

* Add a `prepare_send_transaction_request` helper

Creates a request message for lightwalletd to send a transaction.

* Add test to send transactions using lightwalletd

Obtain some valid transactions from future blocks and try to send them
to a lightwalletd instance connected to a zebrad instance that hasn't
seen those transactions yet. The transactions should be successfully
queued in Zebra's mempool.

* Make `zebra_directory` parameter generic

Allow using a `TempDir` or a `PathBuf`.

* Move lightwalletd protobuf files

Place them closer to the module directory, so that it's clearer that
they specify the RPC protocol for lightwalletd, and not Zebra itself.

* Don't use coinbase transactions in the test

Coinbase transactions are rejected by the mempool.

* Don't remove state lock file

It is removed automatically by Zebra when it shuts down, so if it exists
it should be reported as a bug.

* Force mempool to be enabled in Zebrad instance

Speed up the initialization of the Zebrad instance used for lightwalletd
to connect to.

* Refactor to create `LIGHTWALLETD_DATA_DIR_VAR`

Document how the environment variable can be used to speed up the test.

* Check for process errors in spawned Zebra instance

Enable checking for known process failure messages.

* Add `FINISH_PARTIAL_SYNC_TIMEOUT` constant

Document why it exists and how the choice of the value affects the test.

* Add `LIGHTWALLETD_TEST_TIMEOUT` constant

And use it for the Zebrad and the Lightwalletd instances used in the
send transaction integration test.

* Check `lightwalletd` process for errors

Enable checking the lightwalletd process for known failure messages.

* Update `tonic` and `prost` dependencies

Use the latest version and fix CI failures because `rustfmt` isn't
installed in the build environment.

* Create `send_transaction_test` module

Move the send transaction using lightwalletd test and its helper
functions into a new module.

* Move `LIGHTWALLETD_TEST_TIMEOUT` constant

Place it in the parent `lightwalletd` module.

* Move gRPC helper functions and types to `rpc` mod.

Make them more accessible so that they can be used by other tests.

* Create a `cached_state` module

Move the test utility functions related to using a cached Zebra state
into the module.

* Move `perform_full_sync_starting_from` to `sync`

Keep to closer to the synchronization utility functions.

* Move Zebra cached state path variable constant

Place it in the `cached_state` module.

* Skip test if `ZEBRA_TEST_LIGHTWALLETD` is not set

Make it part of the set of tests ignored as a whole if no lightwalletd
tests should be executed.

* Move `spawn_zebrad_for_rpc_without_initial_peers`

Place it in the `launch` sub-module.

* Rename `rpc` module into `wallet_grpc`

Avoid any potential misunderstandings when the name is seen out of
context.

* Allow duplicate `heck` dependency

At least until `structopt` is updated or `zebra-utils` is updated to use
`clap` 3.

* Fix a deny.toml typo

* fix(build): CMake is required by `prost` crate

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
2022-04-27 23:06:11 +00:00
Janito Vaqueiro Ferreira Filho b356a67391
change(state): Remove `active_value` field from `ChainTipSender` (#4175)
* Refactor instrumented records in `ChainTipSender`

Record the fields using a helper method. This reduces duplicate code and
prepares for removing the `active_value` field from the type, because
the replacement for it has to be atomically read to avoid a deadlock
when reading from the `watch::Sender` endpoint.

* Replace `active_value` field with sender borrow

Now that Tokio has been updated to a version in which `watch::Sender`
has a `borrow` method, we can remove the `active_value` field.

To prevent a deadlock like the one that happened with the synchronizer
some time ago, the method instrumentation was slightly refactored to
have the fields recorded in helper methods that avoid obtaining a
read-lock twice.

* Remove some `TODO` comments

They are resolved with the current changes.

* Refactor to avoid obtaining current `Span` twice

This is a minor optimization to remove an unnecessary second call to
`tracing::Span::current()`.

* Add a comment about preventing dead-locks

Explain why a binding can't be created for the old tip reference.

* Retry after log ssh failures in full sync test

Co-authored-by: teor <teor@riseup.net>
2022-04-27 04:13:02 +00:00
Dimitris Apostolou b15bc8a2e1
Fix typos (#4186) 2022-04-25 19:45:28 +00:00
Alfredo Garcia 56aabb1db1
feat(rpc): Implement `getaddressutxos` RPC method. (#4087)
* implement display for `Script`

* implement `getaddressutxos`

* fix space

* normalize list of addresses as argument to rpc methods

* implement `AddressStrings`

* make a doc clearer

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-04-25 03:00:52 +00:00
Dimitris Apostolou 04b0bb9358
Fix typos (#4164) 2022-04-22 01:10:33 +00:00
teor c2430c6f45
feat(rpc): implement the get_address_tx_ids RPC method query (#4119)
* Add a finalized state txids query

* Add an address transaction IDs query, without height filters

* Connect the address transaction ID query to the RPC

* Basic filtering of address transaction IDs by height range

* Add a network and range argument to the getaddresstxids test

* Test all block range combinations for mainnet

* Fix a file descriptor limit error

* Optimise seeking the first transaction for an address

The first transaction's location is part of the address location.

* Filter finalized address transaction IDs by height range

* Filter non-finalized address transaction IDs by the height range

* Fix up snapshot tests for the new height range API
2022-04-21 20:19:26 +00:00
Janito Vaqueiro Ferreira Filho e5f00c5902
feat(rpc): Implement `getaddressbalance` RPC (#4138)
* Add `Amount::serialize_as_string` helper method

A helper method that makes it easier to serialize an `Amount` as a
string. This is needed for the response type of the `getaccountbalance`
RPC.

* Implement state service call for address balance

Add `Read{Request,Response}::AddressBalance` variants and implement the
handler that calls the query function.

* Create an `AddressBalance` response type

Only contains the `balance` field which is needed by `lightwalletd`.
That field is serialized as a string, following the RPC specification.

* Implement `get_address_balance` RPC

Query the read-only state service for the information, and wrap it in an
`AddressBalance` response type so that it is serialized correctly.

* Run `rustfmt` inside `proptest!` block

Fix some minor formatting details.

* Test `get_address_balance` with valid addresses

Check that the RPC leads to a query to the mocked state service for a
balance amount.

* Test `get_address_balance` with invalid addresses

An error message should be returned by the RPC.

* Rename metric to `address_balance`

Keep it consistent with how it's named in other places.

Co-authored-by: teor <teor@riseup.net>

* Revert "Add `Amount::serialize_as_string` helper method"

This reverts commit 01b432e3d2ac2313a90d55d06b3fa855c0b71330.

* Serialize amount as an integer

This is different from what the documentation says, but it's what
lightwalletd expects.

* Add reference to RPC documentation

Make sure it is linked to for easy access.

* Create an `AddressStrings` type

To be used as the input for the `get_address_balance` RPC method.

* Use `AddressStrings` in `get_address_balance` RPC

Fix the input parameter so that the list of address strings is placed
inside a JSON map.

* Update property tests to use `AddressStrings`

Make sure the proper input type is created.

Co-authored-by: teor <teor@riseup.net>
2022-04-20 18:27:00 +00:00
Conrado Gouvea dff25473aa
refactor: document coinbase rules, refactor to ease understanding (#4056)
* refactor: document coinbase rules, refactor to ease understanding

* Update zebra-consensus/src/block/check.rs

Co-authored-by: teor <teor@riseup.net>

* remove no longer used contains_prevout_input()

* remove unused CoinbaseHasPrevOutInput

* update coinbase description in documentation

Co-authored-by: teor <teor@riseup.net>
2022-04-20 09:31:12 +00:00
Dimitris Apostolou ad971bbdd5
Fix typos (#4112) 2022-04-20 00:26:58 +00:00
teor 65b94f7e50
9. feat(state): add a query function for transparent UTXOs (#4111)
* Add address UTXOs query functions, but without the transaction IDs

* Return transaction IDs along with address UTXOs

* Add a convenience type for address UTXOs

* Add output addresses to the convenience method

* Fix query documentation

* Rename the chain transaction IDs method
2022-04-19 13:34:53 +00:00
teor 75671c53a2
1.0.0-beta.8 Release (#4135)
* Update changelog for 1.0.0-beta.8

* Bump Zebra crate versions

The tower crates and their dependencies haven't changed,
so their versions don't need to be bumped.

Some Zebra crates haven't changed,
but I bumped all the versions to keep them consistent.

* Explicitly say that we support NU5 testnet 2

* Expand PR #3799 into changelog entries

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-04-19 13:02:17 +10:00
Conrado Gouvea d79e71e969
change(nu5): use new V5 transaction script verification API (#3799)
* update librustzcash; adapt to new API

* add ticket reference for removing zcash_proofs duplicated dependencies

* update to new zcash_script V5 API

* use zp_tx shorthand

* update to Zcash 4.7.0 dependencies

* update protocol versions

* feat(rpc): Implement `getblockchaininfo` RPC method (#3891)

* Implement `getblockchaininfo` RPC method

* add a test for `get_blockchain_info`

* fix tohex/fromhex

* move comment

* Update lightwalletd acceptance test for getblockchaininfo RPC (#3914)

* change(rpc): Return getblockchaininfo network upgrades in height order (#3915)

* Update lightwalletd acceptance test for getblockchaininfo RPC

* Update some doc comments for network upgrades

* List network upgrades in order in the getblockchaininfo RPC

Also:
- Use a constant for the "missing consensus branch ID" RPC value
- Simplify fetching consensus branch IDs
- Make RPC type derives consistent
- Update RPC type documentation

* Make RPC type derives consistent

* Fix a confusing test comment

* get hashand height at the same time

* fix estimated_height

* fix lint

* add extra check

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* fix typo

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* split test

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* fix(rpc): ignore an expected error in the RPC acceptance tests (#3961)

* Add ignored regexes to test command failure regex methods

* Ignore empty chain error in getblockchaininfo

We expect this error when zebrad starts up with an empty state.

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Make sync error logs more user-friendly (#3944)

- use info level, there is nothing the user needs to do,
  particularly for a single error
- explain that the errors are temporary
- hide backtraces, because they look like crashes

* Update test.patch.yml with lightwalletd job (#3970)

* Update test.patch.yml with lightwalletd job

* Remove a workflow condition that will always be false

In general, patch workflows need the
opposite conditions to the original workflow.

But in this case, we know the result of the
condition will always be true, so we can just delete it.

Co-authored-by: teor <teor@riseup.net>

* fix(doc): Fix bugs in the lightwalletd database design (#3964)

* Re-order column families in design in dependency order

* Minor RFC design tweaks and fixes

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Repoint zebra image links to our new zfnd.org site for now (#3949)

* Repoint zebra image links to our new zfnd.org site for now

* Remove images/

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Fix typos (#3956)

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* bump database version to trigger testnet rollback

* reduce minimum protocol version for now (will be changed later)

* update dependencies

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* update versions to match zcash 4.7.0

* deny.toml: update 'darling'

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
Co-authored-by: Dimitris Apostolou <dimitris.apostolou@icloud.com>
2022-04-19 10:14:16 +10:00
teor 8e29219565
8. feat(state): add a query function for transparent address balances (#4097)
* Make address index types consistent

* Simplify non-finalized address index updates

* Update snapshots for address index queries

* Simplify non-finalized UTXO query

* Add a query method for non-finalized address balance changes

* Add a query method for finalized state address balances

* Add a query function for address balances

* Refactor balance queries to make them repeatable

* Retry interrupted finalized balance queries

* Pop chain root blocks until it matches the finalized tip

* Avoid cloning the chain

It has already been cloned by the watch receiver

* Refactor and fix documentation of the balance query code
2022-04-14 13:34:31 +00:00
Conrado Gouvea 53a42999ef
7. feat(db): Add a transparent address transaction index (#4038)
* feat(db): add transaction location index

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* add address_tx_ids(); also index spends from addresses

Co-authored-by: teor <teor@riseup.net>
2022-04-13 23:48:35 +00:00
Alfredo Garcia 7b7d22aabc
feat(rpc): Implement what we can of `getaddresstxids` RPC method. (#4062)
* implement `getaddresstxids` rpc method with dummy empty response

* use already public function

* fix some docs

* pass a list of addresses to the state request

* sync range errors with zcashd

* refactor a loop

* fix grammar

* fix tests

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
2022-04-13 08:48:13 +00:00