fix(ci): Increase the full sync timeout (#5129)

* Increase the full sync timeout to 28 hours

And put the timeout constant with the rest of them.

* Update developer documentation for the new constant location

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

book/src/dev/continuous-integration.md

@@ -47,7 +47,7 @@ Please shut down large instances when they are not being used.
 ### Automated Deletion
 
 The [Delete GCP Resources](https://github.com/ZcashFoundation/zebra/blob/main/.github/workflows/delete-gcp-resources.yml)
-workflow automatically deletes instance templates, disks, and images older than 1 week.
+workflow automatically deletes instance templates, disks, and images older than a few days.
 
 Running instances and their disks are protected from deletion.
 
@@ -91,7 +91,7 @@ https://github.com/ZcashFoundation/zebra/runs/8181760421?check_suite_focus=true#
 The earlier failure can also be in another job, check out the whole workflow run for details.
 (Use the "Summary" button on the top left of the job details, and zoom in.)
 
-### Resolving CI Sync Timeouts
+### Fixing CI Sync Timeouts
 
 CI sync jobs near the tip will take different amounts of time as:
 - the blockchain grows, and
@@ -111,11 +111,11 @@ To fix a CI sync timeout, follow these steps until the timeouts are fixed:
 
 5. If a Rust test fails with "command did not log any matches for the given regex, within the ... timeout":
 
-    a. If it's the full sync test, [increase the full sync timeout](https://github.com/ZcashFoundation/zebra/commit/9fb87425b76ba3747985ea2f22043ff0276a03bd#diff-8fbc73b0a92a4f48656ffe7d85d55c612c755202dcb7284d8f6742a38a6e9614R367)
+    a. If it's the full sync test, [increase the full sync timeout](https://github.com/ZcashFoundation/zebra/pull/5129/files)
 
     b. If it's an update sync test, [increase the update sync timeouts](https://github.com/ZcashFoundation/zebra/commit/9fb87425b76ba3747985ea2f22043ff0276a03bd#diff-92f93c26e696014d82c3dc1dbf385c669aa61aa292f44848f52167ab747cb6f6R51)
 
-### Resolving Duplicate Dependencies in `Check deny.toml bans`
+### Fixing Duplicate Dependencies in `Check deny.toml bans`
 
 Zebra's CI checks for duplicate crate dependencies: multiple dependencies on different versions of the same crate.
 If a developer or dependabot adds a duplicate dependency, the `Check deny.toml bans` CI job will fail.
@@ -149,7 +149,7 @@ To fix duplicate dependencies, follow these steps until the duplicate dependenci
 
 4. Repeat step 3 until the dependency warnings are fixed. Adding a single exception can resolve multiple warnings.
 
-### Resolving Disk Full Errors
+### Fixing Disk Full Errors
 
 If the Docker cached state disks are full, increase the disk sizes in:
 - [deploy-gcp-tests.yml](https://github.com/ZcashFoundation/zebra/blob/main/.github/workflows/deploy-gcp-tests.yml)

zebrad/tests/common/sync.rs

@@ -74,6 +74,10 @@ pub const LARGE_CHECKPOINT_TIMEOUT: Duration = Duration::from_secs(180);
 /// cause the test to fail.
 pub const FINISH_PARTIAL_SYNC_TIMEOUT: Duration = Duration::from_secs(11 * 60 * 60);
 
+/// The maximum time to wait for Zebrad to synchronize up to the chain tip starting from the
+/// genesis block.
+pub const FINISH_FULL_SYNC_TIMEOUT: Duration = Duration::from_secs(28 * 60 * 60);
+
 /// The test sync height where we switch to using the default lookahead limit.
 ///
 /// Most tests only download a few blocks. So tests default to the minimum lookahead limit,
@@ -366,9 +370,6 @@ pub fn create_cached_database_height(
 ) -> Result<()> {
     eprintln!("creating cached database");
 
-    // 24 hours
-    let timeout = Duration::from_secs(24 * 60 * 60);
-
     // Use a persistent state, so we can handle large syncs
     let mut config = cached_mandatory_checkpoint_test_config()?;
     // TODO: add convenience methods?
@@ -381,7 +382,7 @@ pub fn create_cached_database_height(
     let mut child = dir
         .with_exact_config(&config)?
         .spawn_child(args!["start"])?
-        .with_timeout(timeout)
+        .with_timeout(FINISH_FULL_SYNC_TIMEOUT)
         .bypass_test_capture(true);
 
     let network = format!("network: {},", network);