Skip to content
twostacktwostacktwostack

Overview

Matching a good architecture with any particular use-case is never an easy task. When we first wrote this section in 2020, the question we were trying to answer was “How does my application talk to the Bitcoin network?”. Every answer we gave took the same form: find a copy of the blockchain that somebody has indexed, and ask it questions.

That is no longer how we build applications. In this section we will explore a different model, one in which our applications don’t need an index of the blockchain at all.

How we used to build

Section titled “How we used to build”

In 2020 an application had a handful of ways to interact with the Bitcoin network:

  • connect directly to a Bitcoin Node over JSON-RPC, and listen for events on its ZeroMQ interface.
  • send Transactions to a miner through the Merchant API (mAPI).
  • query a Generic API Provider such as WhatsOnChain for addresses, Transactions and UTXOs.
  • run an Application-Specific API server (TXT, TXQ or TXBox at the time) that stored only the Transactions our application cared about.

Figure 1 shows the most refined version of that picture.

An Application-Specific API in front of a miner's Merchant API
Figure 1: How we built in 2020: an Application-Specific API

Every one of these approaches rests on the same assumption: that somewhere, someone keeps an index of every Transaction, and our application asks that index about the ones it cares about. That assumption has not aged well.

  • Indexing the whole blockchain is an expensive business, and few providers are willing to do it. An application that depends on an indexer inherits its outages, its rate limits, and its view of the truth.
  • Teranode, the node software the network is moving to, is built to process Transactions at scale, not to answer an application’s questions. It does not serve gettxoutproof, for example, which our applications once used to fetch merkle proofs.
  • The Merchant API was replaced by ARC, and ARC in turn by Arcade.

The Application-Specific API was the right instinct. Our applications should only keep the Transactions that concern their own domain. What has changed is where those Transactions are kept, and how we check them.

How we build now

Section titled “How we build now”

Nobody needs the whole blockchain. Each party needs the Transactions that concern it, and the means to check them.

This is the idea behind Simplified Payment Verification (SPV), as described in section 8 of the Bitcoin whitepaper. Applications built this way keep their Transactions in three places (Figure 2):

  1. The user’s wallet. Each wallet holds the full history of its own Transactions, each with the merkle proof that shows it was mined.
  2. The overlay network. A federation of domain-specific nodes holds the Transactions of one domain (a token, a message box service, a marketplace) at the edge of the network, close to the applications that use them.
  3. The Bitcoin network, only for what the network alone can provide: broadcasting a Transaction and getting back its merkle proof (through Arcade), and the chain of block headers that proofs are checked against.
Users’ wallets exchange BEEF with each other and submit to and look up from the overlay network of domain nodes. Below them, the Bitcoin network provides Arcade for broadcasting and merkle paths, and peers or a CDN for block headers.Users’ walletsOverlay networkBitcoin networkAliceBobdomain nodedomain nodedomain nodeArcadepeers / CDNfullhistoryfullhistoryBEEFsubmit,look upbroadcastmerkle pathsblock headers
Figure 2: The three places an application’s Transactions live

What is missing from Figure 2 is as important as what is in it. There is no Bitcoin Node of our own, and there is no index of the whole blockchain. Transactions travel from party to party as BEEF, a format that carries a Transaction together with the proof that its inputs are real, and every party checks what it receives against block headers it holds itself.

The applications we describe in this section are not hypothetical. SpiffyVault and OverNode, our own Flutter applications, are built this way on top of libspiffy, our SPV wallet library.

In this section

Section titled “In this section”