Customer survey

npm run survey runs a generated population of AI customer agents against the exchange and turns what they say into one report: themes ranked by how many customers raised them, representative quotes, adoption and willingness-to-pay signals, and the top recommended changes. It is the closest thing to customer feedback at scale before there are customers.

It sits next to the agent-user harness. The harness checks that the exchange keeps its contract with four fixed personas. The survey asks a crowd of different customers whether they would actually buy, and why not.

npm run survey                       # in-process exchange, rule-based, no model calls, free
npm run survey -- --dry-run          # print the population, cost estimate and load limits; call nothing
npm run survey -- --history          # compare earlier runs

# Real role-played feedback against the live site (costs money; see "Cost")
export ANTHROPIC_API_KEY=sk-ant-...
HARNESS_LLM=anthropic ADMIN_TOKEN=... npm run survey -- https://clear-exchange.fly.dev

What happens in a run#

1. A population is generated from a seed (--seed, default clear), so the same seed gives the same customers and runs compare over time.
   • Buyers (default 24, --buyers, up to 200) are AI products across twelve types: research assistants, news copilots, finance copilots, coding agents, shopping assistants, legal research tools, health information assistants, education tutors, enterprise RAG platforms, travel planners, marketing writers and climate/ESG analysts. Each has a company stage (solo developer to large enterprise) with a matching monthly budget and volume, a technical depth (novice, intermediate, expert), a temperament (all critical, in different ways: the cost-obsessed founder, the compliance-first lead, the sceptical CTO), the content source it uses today, a realistic task, the usage tier it needs and the most it would pay per item.
   • Publishers (default 6, --publishers) vary in size (solo newsletter to multi-title group, plus trade and academic), vertical, revenue model and stance on AI (blocking crawlers, regretting a cheap flat-fee deal, losing search traffic, negotiating collectively).
2. Each buyer reads the docs the way an integrator would: the docs index, API reference, licence terms, entitlements and telemetry pages, fetched from the target's /docs/ (or the repository's docs/ if the target serves none).
3. It plans up to three queries in its own words, with the usage tier, the price it is really willing to pay (held to its ceiling) and any licence filters it needs (for example commercial use, no share-alike).
4. It runs them over HTTP: query, judge the offers (accept at most one per query, or walk away), accept, retrieve with the entitlement and report SPUR telemetry at the depth its sophistication suggests. Hard limits apply whatever the model says: it can only accept offers that exist, within its budget.
5. It gives candid feedback: would it adopt (yes, trial, no) and how likely, what it would pay per item and per month, what is missing, what confused it, what would make it switch from its current source, and issues with a theme label, severity, detail and suggestion, plus a quote in its own voice.
6. Publishers review what they can see: the rate card guide and licence terms page, current rate cards (from /v1/admin/rate-cards, or the repository's seeded cards), the run's pricing outcomes (offers shown, accepted, rejection reasons, price range, demand priced out), the ledger and telemetry for this run's agents (with an admin token), the website and demo app text, and the buyers' verdicts. Each says whether it would join, on what terms and at what prices.
7. Aggregation: issues are clustered into themes (by the model in LLM mode, by label otherwise), ranked by the number of distinct agents who raised them and then by severity (high 3, medium 2, low 1), with up to four quotes per theme (one per agent, most severe first). The model then recommends the top five changes from the ranked themes and signals.

The report#

Each run writes reports/survey/<run_id>.md and <run_id>.json (everything, including every transcript) and appends one line to reports/survey/history.jsonl. The markdown has headline signals (adoption by buyers and publishers, willingness to pay against what was actually paid, the query funnel, including how many queries were answered "no good match"), the top recommended changes, the ranked themes with quotes, changes since the previous run of the same mode against the same target (in percentage points of agents, so runs of different sizes compare), adoption by product type and company stage, every buyer and publisher verdict, what confused integrators, what is missing, what would make them switch, publisher views on price, and an appendix of each session. Transcripts show agents each offer's published and updated dates, the period the exchange read from the query, and the no-good-match reason and message, so their feedback can speak to freshness and honest empty answers.

npm run survey -- --history prints a table of all recorded runs. Reports are not committed (reports/ is git-ignored); keep the directory, or point --out somewhere durable, to build up history.

Modes#

Rule-based (default, CI and tests). Without HARNESS_LLM, plans come from each profile's example questions, decisions from fixed offer rules and feedback is templated from what happened (empty results, prices over budget, failures). It exercises the whole pipeline offline and deterministically, and the report says plainly that its opinions are templated. Tests never call a real model.

LLM. With HARNESS_LLM=anthropic and ANTHROPIC_API_KEY, every plan, decision, piece of feedback, publisher review, the clustering and the recommendations come from the model role-playing each customer. Prompts tell agents they are customers evaluating a vendor, not assistants: be candid and critical, say no when the answer is no, never praise what did not demonstrably work, ground every claim in the docs or their own transcript, and compare with what they use today. A reply that cannot be used falls back to the rule for that step, and the report lists where.

Treat LLM results as directional evidence from simulated customers, not as real customer data. They are best at surfacing friction, gaps in the docs and objections to pricing and terms; they are weakest at predicting real willingness to pay.

Another vendor is one entry in PROVIDERS in harness/brain.ts, implementing LlmProvider and reporting token usage through onUsage, plus its prices in MODEL_PRICES in harness/survey/cost.ts.

Cost and safety#

• Estimate first. Every run prints a typical and worst-case model cost before it starts; --dry-run stops there. The typical figure assumes the shared docs prefix is cached (Anthropic prompt caching) and replies of usual length; the worst case assumes no caching and every reply at its token limit.
• Hard cap. --max-cost (default $5) is enforced on every call: before a call, the worst case for that call is reserved, and a call that could take spend past the cap is refused. Agents that have not started when the cap is reached are skipped and listed in the report as "not started: spending cap reached"; aggregation falls back to rule-based grouping if needed. Spend is metered from the token usage the API reports, at the list prices in harness/survey/cost.ts. An unknown model is refused unless you give its price, because the cap cannot be enforced without one.
• Concurrency. --concurrency (default 4) limits agents running at once and model calls in flight. Rate-limited or overloaded API replies are retried with backoff (failed calls are not billed).
• Load on the exchange. --rps caps requests per second to the target across all agents: default 2 against a URL, 50 in-process. A default 24-buyer run makes about 180 to 250 requests (more when agents buy more), so one to two minutes of gentle traffic on the live site.
• Records. Purchases on a deployed target create real ledger entries and telemetry (no money moves). Agent ids start with survey- so they can be filtered out.

Rough cost per run at list prices (October 2026), from the built-in estimator:

With the default $5 cap a default run finishes comfortably; for 40 or more Sonnet buyers raise the cap (for example --max-cost 8). Prices change: check https://platform.claude.com/docs/en/about-claude/pricing and update MODEL_PRICES.

Options#

Exit code 0 when the run completed (whatever the customers said), 2 when it could not run.

Code#

harness/survey/
  cli.ts          entry point: arguments, mode selection, estimate, reports and history
  run.ts          one run: docs, buyers (pooled), publisher view, publishers, aggregation
  population.ts   seeded generator of buyer and publisher profiles
  agents.ts       the buyer flow over HTTP; what a publisher is shown
  mind.ts         RuleMind (deterministic) and LlmMind (prompts, parsing, fallbacks)
  cost.ts         prices, CostMeter (hard cap), MeteredProvider, Semaphore, RateLimiter, estimate
  aggregate.ts    themes, signals, history entries and run-to-run diffs
  report.ts       markdown report and history table
  docs.ts         fetch and flatten the docs, website and rate cards
  types.ts        shared types

test/survey.test.ts covers the population, cost controls, the LLM path with a scripted fake provider (including the cap stopping a run), aggregation and full rule-based runs against an in-process exchange. CI runs a small rule-based survey and uploads the report as an artifact.