Architecture and request flow

Clear Exchange is a private exchange where publishers compete to answer AI agents' questions with licensed content. One Node.js process serves the API, the web UI and (optionally) the demo publishers. It has no runtime dependencies.

Request flow#

agent ──POST /v1/query──▶ Exchange.query
                           ├─ cache (normalised query) ── hit: reuse the candidate pool
                           ├─ routeQuery(): profiles + telemetry prior → top FANOUT_CAP sources,
                           │                 one exploration slot for new/under-served sources
                           ├─ fanOut(): SupplyAdapter.search() under FANOUT_BUDGET_MS, shortened
                           │            tmax per source, skip if too little left, ignore late replies
                           ├─ rankCandidates(): constraints → re-score (one scale) ± capped self-score,
                           │    overclaim penalty, prior → price (card is a ceiling) + fee on top
                           │    → drop over max_price → sort by relevance → select cheapest in band
                           └─ offers (stored with a TTL in ExchangeState: memory + VAR_DIR/state.jsonl)
agent ──POST /v1/acquire──▶ Exchange.acquire: the same clearing, then accept + retrieve the selected offer
agent ──accept──▶ issueEntitlement() ── Signer (Ed25519 today; attested key later)
agent ──GET content + Bearer──▶ verifyEntitlement() → SupplyAdapter.retrieve()
                                 ├─ content_hash check, Ledger.record()  publisher price + fee = all-in
                                 └─ telemetry: content_retrieved (source_role "index")
agent ──POST /v1/telemetry + Bearer──▶ validateAgentBatch() → grounded / cited / presented / engaged

Step by step:

1. Query. parseQueryRequest() validates the body. The exchange creates or continues a session.
2. Route. If the normalised query is in the cache, the cached candidate pool is reused. Otherwise routeQuery() picks at most FANOUT_CAP sources (see Routing).
3. Fan out. fanOut() calls each routed source's search() in parallel under one time budget.
4. Rank. rankCandidates() applies constraints, re-scores every candidate, prices it from its rate card, adds the fee, drops what is over budget, sorts by relevance and selects the cheapest offer in the relevance band (see Ranking and selection).
5. Offer. Offers are stored for OFFER_TTL_SECONDS, bound to the querying agent, in ExchangeState (journalled to VAR_DIR/state.jsonl, so they survive a restart). Every response reports its stage timings against the routing budget. POST /v1/acquire stops here: it takes the selected offer, stores only that one, and runs steps 6 and 7 itself, returning the entitlement and the content in one response.
6. Accept. issueEntitlement() signs a JWS with the resolved terms, price and fee split (Entitlements).
7. Retrieve. The exchange verifies the entitlement, calls the source's retrieve(), checks the content hash, records the ledger entry once per entitlement and emits content_retrieved.
8. Report. The agent posts SPUR events; the exchange validates them against the entitlement and stores them (Telemetry). Stage counts feed each source's telemetry prior.

Code map#

src/
  domain/money.ts          integer micros; no floats in any price or ledger value
  ratecard/                rate card schema v1.x (validation) and pure pricing
  supply/                  SupplyAdapter interface, source registry, v1 offer contract, http-offer adapter,
                           local catalogue and demo verticals, coverage report, Wikimedia (live),
                           TollBit and Prebid/Trusted Server stubs
  routing/                 source profiles, router, fan-out under a time budget, cache, per-source stats
  demo/mock-publishers.ts  mock publishers speaking the offer contract (honest, discounting, inflating, slow)
  relevance/               RelevanceScorer interface, BM25 default
  exchange/                query parsing, ranking (pure), Exchange service, errors
  entitlement/             Signer and key resolver interfaces, JWS issue and verify
  clearing/ledger.ts       fee arithmetic and the append-only ledger
  telemetry/spur.ts        SPUR v1.0 event building and agent batch validation
  telemetry/timings.ts     recent query timings, percentiles per source (GET /v1/admin/timings)
  storage/jsonl.ts         append-only log (memory, or a JSONL file reloaded at startup)
  storage/state.ts         sessions, offers and issued entitlements: memory + compacted JSONL journal
  http/server.ts           node:http routing, JSON in and out
  http/static.ts           web UI assets: fixed allowlist, served from memory with a strict CSP
  docs/                    docs site: Markdown renderer and pages built from docs/ at startup; see docs-site.md
  app.ts                   composition root: chooses signer, storage, scorer and adapters
  config.ts, server.ts     env config and production checks; entry point
data/                      seeded fictional publishers and articles, rate cards (incl. demo Wikimedia cards),
                           sources/ (sources files) and demo-publishers/ (mock publisher content)
web/                       marketing page (index.html) and app (app.html, app.js, lib.js); fonts/ the headline serif; see web-ui.md
docs/                      this documentation, also served as the docs site
harness/                   synthetic agent-user test harness; see agent-harness.md
                           survey/ the LLM customer survey; see customer-survey.md
scripts/                   e2e-agent.ts (end-to-end walkthrough), keygen.ts, mock-publishers.ts,
                           check-live-sources.ts (CI probe), record-wikimedia-fixtures.ts
test/                      node:test suites; fixtures/wikimedia/ holds recorded API responses

Exchange (src/exchange/exchange.ts) is transport-agnostic; src/http/server.ts wraps it. Tests build exchanges directly with buildExchange(config, overrides) to swap adapters, scorer, signer, clock or fetch.

Routing#

Route, don't index. The exchange holds a thin profile per source: declared topics plus up to 100 sampled titles and abstracts, never full text. Samples come from the source's declared profile, from probe_queries the exchange sends at startup, and from what the source returns to live queries.

• Each query is scored against the profiles with BM25, plus PRIOR_WEIGHT × (telemetry prior − 0.5). profile.min_score keeps a broad source (an encyclopedia) in contention for any query.
• At most FANOUT_CAP (5) sources are routed. EXPLORE_SLOTS (1) of them go to sources routed fewer than EXPLORE_MIN_QUERIES (20) times, so new sources get traffic and build a history. When there are no more sources than the cap, all are routed (reason: "all").
• Stub adapters are never routed to.

Fan-out follows Prebid Server's timeout handling: one FANOUT_BUDGET_MS (900 ms) budget; each source gets a tmax equal to what is left, minus FANOUT_RESERVE_MS for scoring and the response, minus its network_allowance_ms. A source whose tmax would fall below MIN_SOURCE_TMAX_MS is skipped. Late replies are aborted and ignored. SUPPLY_TIMEOUT_MS is a backstop per source. A failing or late source is listed in stats.supply_errors and timings.dropped_sources, and the query clears from the others. GET /v1/admin/timings shows latency percentiles and drop rates per source.

Complete fan-outs are cached per normalised query for ROUTE_CACHE_TTL_SECONDS (60). The query response's routing block shows every decision.

Ranking and selection#

Sources' self-scores are not trusted. Every candidate is re-scored by the exchange's RelevanceScorer on the same text (title and snippet for remote sources), so all sources compete on one scale. The default scorer is BM25 over the query's candidate pool, mapped with s / (s + 6) so scores stay absolute rather than "best = 1.0", then scaled by query coverage: x (0.5 + 0.5 x coverage), where coverage is the IDF-weighted share of the query's terms the candidate contains (terms no candidate contains are ignored). IDF grows with the pool, so without coverage one rare word shared with an off-topic query could lift a candidate over the relevance floor once the pool holds a few hundred items; a candidate that matches the whole query is unaffected. Routing scores profiles with plain saturated BM25. Then:

• a source's self_score is a weak feature, weighted by SELF_SCORE_WEIGHT and capped at ±SELF_SCORE_CAP (0.05), and discounted by the source's overclaim;
• a source whose claims exceed the exchange's score by more than DIVERGENCE_TOLERANCE (now, or as a moving average) loses DIVERGENCE_PENALTY × overclaim;
• the telemetry prior adds PRIOR_WEIGHT × (prior − 0.5);
• when the query asks for a period ("today", "this month", "latest", "2009"), recency boosts items inside it and demotes stale or undated ones (src/exchange/recency.ts, see API: Recency and freshness).

Each offer's scoring shows the parts.

Relevance first; price competes only inside the band.

1. Candidates failing a constraint, or below the floor (BAND_FLOOR, default 0.5, or the query's higher min_relevance), are dropped however cheap.
2. Each is priced from its rate card (Rate cards); the fee goes on top.
3. Only candidates with relevance ≥ BAND_RATIO (0.9) × the top candidate form the band; the rest are not offered. The top is taken before the price cap, so a cheap second-best item cannot become "the best" because the best one is over budget.
4. In-band offers whose all-in price exceeds max_price are excluded, not down-ranked, and summarised in price_hint.
5. The rest are the offers, sorted by relevance (price breaks exact ties only). The cheapest is selected, and is always returned even if limit would cut it.
6. If nothing is left, the response says "result": "no_good_match" with a no_match reason instead of offering low-quality content (API: No good match).

Worked example (test/selection.test.ts): cap $0.010; A 0.92 at $0.008, B 0.88 at $0.004, C 0.60 at $0.001. The band threshold is 0.828, so C is out; B wins on price: publisher $0.004, fee $0.0004, buyer pays $0.0044. Had B cost $0.011 it would break the cap and A would win.

When relevant offers were excluded only on price, price_hint gives the price and relevance of the cheapest and the most relevant of them. Offers another constraint would still exclude are left out of the hint.

Relevance is replaceable. RelevanceScorer (src/relevance/scorer.ts) is async and returns scores in [0, 1]. An embedding or re-ranker scorer can be passed through buildExchange(config, { scorer }).

Pricing and clearing#

• A rate card is a ceiling: sources may discount below it, never charge above it.
• The clearing fee sits on top of the publisher price: fee = floor(publisher × CLEARING_FEE_BPS / 10000) (default 1000 bps, 10%, a placeholder to test). The buyer sees one all-in price.
• The split is fixed at offer time, signed into the entitlement (cx.clearing) and recorded in the ledger on first retrieval (gross = all-in, net = publisher, fee).
• The ledger is idempotent per entitlement. Zero-price content creates no entry. A failed retrieval records nothing.
• No money moves; the ledger is a record.

Three separate formats#

Each can evolve without breaking the others.

Persistence#

VAR_DIR defaults to ./var (git-ignored); empty means memory only. Production sets /data, a Fly volume, so state survives restarts and deploys (Durable state). A volume attaches to one machine, so the demo runs exactly one. A shared database behind AppendLog and ExchangeState would replace the files without changing callers, and allow more machines. See Roadmap.

Key decisions#

• TypeScript on Node 22, no runtime dependencies. Node runs .ts directly (type stripping), so there is no build step. node:crypto provides Ed25519, node:http serves the API and node:test runs the tests. A small dependency surface also matters for the planned confidential VM, where everything in the image is part of the attested measurement.
• One interface for all supply. Every source plugs in through SupplyAdapter, so adding a source never changes routing, ranking, pricing or entitlement code (Adding a source).
• Integer money. All prices and ledger values are integer micros.
• Keys behind interfaces. Token code depends only on Signer and VerificationKeyResolver, so an attested signer can be added later.
• One composition root. src/app.ts is the single place where the signer, storage, scorer and adapters are chosen.