Commit Graph

19 Commits

Author SHA1 Message Date
Jane Lusby 4c9bb87df2
zebra-state: replace sled with rocksdb (#1325)
## Motivation

Prior to this PR we've been using `sled` as our database for storing persistent chain data on the disk between boots. We picked sled over rocksdb to minimize our c++ dependencies despite it being a less mature codebase. The theory was if it worked well enough we'd prefer to have a pure rust codebase, but if we ever ran into problems we knew we could easily swap it out with rocksdb.

Well, we ran into problems. Sled's memory usage was particularly high, and it seemed to be leaking memory. On top of all that, the performance for writes was pretty poor, causing us to become bottle-necked on sled instead of the network.

## Solution

This PR replaces `sled` with `rocksdb`. We've seen a 10x improvement in memory usage out of the box, no more leaking, and much better write performance. With this change writing chain data to disk is no longer a limiting factor in how quickly we can sync the chain.

The code in this pull request has:
  - [x] Documentation Comments
  - [x] Unit Tests and Property Tests

## Review

@hdevalence
2020-11-18 18:05:06 -08:00
Deirdre Connolly 87d749ee4f I love it when capitalization matters, contrary to the docs 2020-11-14 22:31:38 -05:00
Deirdre Connolly 477eac7f19 Properly use Dockerfile ARG values 2020-11-14 22:31:38 -05:00
Deirdre Connolly 7023465d91 Pass in workflow inputs for network and checkpoint_sync (with defaults) all the way down 2020-11-14 22:31:38 -05:00
Deirdre Connolly 8b5b1b49cc
Run acceptance tests post-merge with large cached state (#1282)
* Create and mount persistent disk to store zebrad state, update runner container config to use

* Enable checkpoint sync in zebrad image config

* Lower state memory cache from 500MB to 50MB

* Upgrade host to n2-standard-4

* Bump zebrad-cache disk size to 100GB

* Copy zebrad as the tests are compiled with a hardcoded path to it

* Rename all debug binaries for easy invocation

* Name state cache disk, use the correct path to binaries

* Create volume and all that jazz on instance creation

Otherwise there's a lot of on-instance commands to do that is just handled by this shortcut.

* Explicitly mount the state cache and cleanup test instance

* Wait for zebra-test container to start then attach

* Always clean up even if the tests step fails

* Keep fast sleep but only print 'waiting' once
2020-11-12 15:18:35 -05:00
Deirdre Connolly a9029beb87 Explicitly pass our config to the start command
Resolves #1005
2020-09-04 01:02:32 -04:00
Jane Lusby 1d6183ef84
Make zebra-script wrap the C++ script implementation in zcashconsensus (#708)
* attempt to use zcashconsensus crate in zebra-script

* boop

* update verify fn to use zebra types

* a bit more cleanup

* cleanup

* more

* beep boop

* fix renamed member

* cleaning

* get a real branch id

* remove as of yet unneeded api

* Update zebra-chain/src/transaction.rs

* Update zebra-chain/src/transaction.rs

* more cleanup

* oops wrong dep section

* use a tuple to communicate arg association

* update to use published version of zcash_script

* fix new compiler error

* install llvm on windows

* fix bindgen bug????

* try to get docker file to win

* okay try everything

* fix windows build maybe

* always download choco

* fix paths for moved types

* try a different error message

* try convenience script

* try installing just llvm

* add back one more

* try installing some headers

* try a diff package

* try everything

* remove the minimum

* try newer docker builder image

* cleanup docker image

* cleanup extra ci step
2020-08-18 11:08:53 -07:00
Deirdre Connolly b4245f4d01
Fix healthcheck (tracing) endpoint (#895)
* Switch to n1-highcpu-2

* Add tracing.endpoint_addr config to release docker image
2020-08-13 03:31:09 -04:00
Deirdre Connolly e6d986a057 Put all ports EXPOSE'd on the same line 2020-06-19 03:46:09 -04:00
Deirdre Connolly f95e7d5595 Add health check 2020-06-19 03:46:09 -04:00
Deirdre Connolly 30f01c6ff0 Use updated 'zebrad seed' command, move binary to root, no workdir 2020-06-19 03:46:09 -04:00
Deirdre Connolly 29117f7c73 Explicit WORKDIR for runner stage 2020-06-19 03:46:09 -04:00
Deirdre Connolly 84c34b2e48 Use ENTRYPOINT plus CMD for command 2020-06-19 03:46:09 -04:00
Deirdre Connolly e86bbf372b Run 'zebrad seed' instead of 'connect' for now 2020-06-19 03:46:09 -04:00
Deirdre Connolly 0ccf167125 gcloud workflow to deploy containers via managed instance group
Also default command to 'zebrad connect' until 'start' is fleshed out.
2020-06-19 03:46:09 -04:00
Deirdre Connolly fb810f0850 Try debian:buster-slim instead of alpine 2020-01-30 13:39:13 -05:00
Deirdre Connolly 3c1f468c39 Fix CMD arguments 2020-01-30 13:39:13 -05:00
Deirdre Connolly 7f42fd8799 Add multiple stages to Dockerfile to run 'seed' 2020-01-30 13:39:13 -05:00
Deirdre Connolly 8fe3403de0
Continuous integration (#2)
* Add CI workflow similar to other zebra

* Bump cache TTL to 24hours

* Expand image name to include full repo owner/repo-name/branch-name

* Force to lowercase because google container registry demands it

This may not be universally shell compatible

* Use bash as gcloud action container entrypoint
2019-09-05 13:08:48 -04:00