consensus: add timeout to UTXO queries (#1391)

The state service API says explicitly that AwaitUTXO requests should be coupled
with a timeout layer. I didn't add this when I was testing and fixing the UTXO
lookup code (#1348, #1358) because causing zebrad to hang on a failed
dependency was useful for identifying cases where the code wasn't useful (and
then inspecting execution traces).

As a side effect, this ticket resolves most of the hangs in #1389, because
far-future gossiped blocks will have their UTXO lookups time out, though we
may wish to do other work as part of debugging the combined sync+gossip logic.

zebra-consensus/src/script.rs

@@ -1,5 +1,6 @@
 use std::{collections::HashMap, future::Future, pin::Pin, sync::Arc};
 
+use tower::timeout::Timeout;
 use tracing::Instrument;
 
 use zebra_chain::{parameters::NetworkUpgrade, transaction::Transaction, transparent};
@@ -7,6 +8,15 @@ use zebra_state::Utxo;
 
 use crate::BoxError;
 
+/// A timeout applied to UTXO lookup requests.
+///
+/// The exact value is non-essential, but this should be long enough to allow
+/// out-of-order verification of blocks (UTXOs are not required to be ready
+/// immediately) while being short enough to prune blocks that are too far in the
+/// future to be worth keeping in the queue, and to fail blocks that reference
+/// invalid UTXOs.
+const UTXO_LOOKUP_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(10 * 60);
+
 /// Asynchronous script verification.
 ///
 /// The verifier asynchronously requests the UTXO a transaction attempts
@@ -20,12 +30,14 @@ use crate::BoxError;
 /// [RFC4]: https://zebra.zfnd.org/dev/rfcs/0004-asynchronous-script-verification.html
 #[derive(Debug, Clone)]
 pub struct Verifier<ZS> {
-    state: ZS,
+    state: Timeout<ZS>,
 }
 
 impl<ZS> Verifier<ZS> {
     pub fn new(state: ZS) -> Self {
-        Self { state }
+        Self {
+            state: Timeout::new(state, UTXO_LOOKUP_TIMEOUT),
+        }
     }
 }