Overlay services

An overlay service is a server that watches Transactions for some topic, and answers queries about the Outputs it has admitted. BSV overlays speak two protocols:

• BRC-22 submit: we send a Transaction (as BEEF) along with the topics it is meant for, and each host answers with the Outputs it admitted (a STEAK).
• BRC-24 lookup: we ask a lookup service a query, and the hosts answer with the matching Outputs, each with its Transaction in BEEF.

Hosts advertise what they run by means of PushDrop tokens. A SHIP advertisement says that a host takes Transactions for a topic (tm_...), and a SLAP advertisement says that it runs a lookup service (ls_...). OverlayClient finds hosts in the same way that @bsv/sdk’s LookupResolver and TopicBroadcaster do. It asks a set of SLAP trackers which hosts run a service, and it asks ls_ship which hosts take a topic.

Overlays, wallets and the network

Figure 1 shows where an overlay sits: between the wallets that use it, and the Teranode network underneath.

Figure 1: An overlay between the wallets and the Teranode network

Referring to Figure 1, let’s follow one Transaction through it:

1. Alice’s wallet broadcasts the Transaction through Arcade, and gets back its status and, once it is mined, its merkle path. In our applications the wallet does its own broadcasting, as described in Broadcasting and merkle proofs.
2. To share the Transaction with everyone else in its domain, Alice’s wallet asks the SHIP / SLAP trackers which hosts take the topic.
3. It submits the Transaction to those hosts as BEEF (BRC-22). Each host’s topic manager admits the Outputs that follow its rules, and the hosts that share a topic keep each other in sync.
4. Later, Bob’s wallet looks the Output up at any host that runs the lookup service (BRC-24). The answer comes back with the Transaction in BEEF.

It should be noted that an overlay host is an index, not a blockchain scanner. It holds only the Transactions that its users submit to it and manage through it, and it never needs to read the rest of the chain. Like a wallet, it reaches the network through Arcade, to broadcast and to get merkle paths, and it needs no connection of its own to a Teranode node.

Nothing in Figure 1 has to be taken on trust either. A host checks the BEEF it is given before it admits anything, and Bob’s wallet checks the BEEF it gets back against its own headers. The trackers hold nothing but advertisements, which the hosts publish as PushDrop tokens on the overlay itself.

Do note that the overlay is for state that many parties share, e.g. a token’s current holder, or which host keeps a message box. A payment between two wallets doesn’t need it: Alice simply hands Bob the BEEF, along the arrow at the top of Figure 1. We look at this split more closely in Overlay networks.

The client

OverlayClient.forNetwork(OverlayNetwork.mainnet) starts from the trackers that the SDK ships (OverlayNetwork.testnet for testnet). We can also give it our own trackers, a timeout (5 seconds by default) and an http.Client. The examples pass in a MockClient from package:http/testing.dart, which answers with recorded responses from real hosts, so that the examples run offline:

// In an app: OverlayClient.forNetwork(OverlayNetwork.mainnet), which
// starts from the SLAP trackers @bsv/sdk ships. Here the trackers and
// the HTTP client are stand-ins, so that we run offline.
final overlay = OverlayClient(
  slapTrackers: const ['https://tracker-a.example', 'https://tracker-b.example'],
  httpClient: MockClient(overlayHosts),
);

Call close() when you are done with the client. It closes the HTTP client.

Finding hosts

hostsFor returns the hosts that run a lookup service, taken from the SLAP advertisements that the trackers hold. Only HTTPS hosts are used, and the answer is kept for five minutes.

// The hosts that run ls_messagebox, from the trackers' SLAP
// advertisements (kept for five minutes).
expect(await overlay.hostsFor('ls_messagebox'), ['https://overlay-us-1.bsvb.tech', 'https://overlay-ap-1.bsvb.tech']);

hostsInterestedIn(topic) does the same for a topic, through ls_ship.

If we would rather skip discovery altogether, for a local overlay or a host we trust, we give hostOverrides:

final overlay = OverlayClient(
  slapTrackers: const ['https://tracker-a.example'],
  hostOverrides: const {
    'ls_messagebox': ['https://my-overlay.example'],
  },
  httpClient: MockClient((request) async {
    asked.add(request.url.host);
    return json({'type': 'output-list', 'outputs': []});
  }),
);
await overlay.lookup('ls_messagebox', {});
expect(asked, ['my-overlay.example']); // the trackers were not asked

Looking up outputs

lookup(service, query) asks every host that runs the service, and merges their answers into one entry per outpoint. One host answering is enough, and hosts that fail are ignored.

// A lookup asks every host, and merges the answers by outpoint. One host
// answering is enough.
final List<LookupOutput> outputs = await overlay.lookup('ls_messagebox', {'identityKey': '02' * 33});
expect(outputs, isEmpty);

Each LookupOutput has the BEEF it came in, the Transaction, the Output index, and shortcuts for the txid, the Output, its Locking Script and the outpoint (txid.index). If we got a lookup answer some other way, OverlayClient.parseOutputList reads it for us. Here it reads SHIP advertisements, which OverlayAdminToken.decode then turns into their fields:

final outputs = OverlayClient.parseOutputList(fixture('ls_ship_tm_messagebox'));
final adverts = [for (final o in outputs) OverlayAdminToken.decode(o.lockingScript)!];
expect([for (final a in adverts) (a.protocol, a.topicOrService, a.domain)], [
  ('SHIP', 'tm_messagebox', 'https://overlay-us-1.bsvb.tech'),
  ('SHIP', 'tm_messagebox', 'https://overlay-ap-1.bsvb.tech'),
]);
// Each LookupOutput carries its Transaction and BEEF.
expect(outputs.first.outpoint, '${outputs.first.txid}.${outputs.first.outputIndex}');

The shape of the query belongs to the lookup service itself. ls_messagebox, for example, takes an identityKey.

Submitting a Transaction

submit(beef, topics) sends the BEEF to every host whose SHIP advertisement names one of the topics. It collects each host’s answer, or the reason it gave none:

// Sent to every host whose SHIP advertisement names the topic.
final SubmitResult result = await overlay.submit(beef, ['tm_messagebox']);
expect(result.acknowledgedIn('tm_messagebox'), isTrue); // at least one host admitted it
expect(result.answers.keys, ['https://overlay-us-1.bsvb.tech']);
expect(result.failures.keys, ['https://overlay-ap-1.bsvb.tech']); // and why: HTTP 503

acknowledgedIn(topic) is true when at least one host admitted an Output, or retained or removed a coin, in that topic. Do note that a host which answers but admits nothing does not count as an acknowledgement.

Caution

The BEEF must hold the Transaction’s ancestry with merkle proofs, since overlays validate it. The BEEF of a libspiffy plugin payment may not. See Host security rules, gap K1.

Errors

lookup and submit throw an OverlayException when they cannot do anything at all: no host runs the service or takes the topic, or no host answered a lookup. A submit where only some of the hosts fail still returns, with those hosts listed in failures.

final overlay = OverlayClient(
  slapTrackers: const ['https://tracker-a.example'],
  httpClient: MockClient((request) async => http.Response('down', 503)),
);
await expectLater(overlay.lookup('ls_messagebox', {}), throwsA(isA<OverlayException>()));