Integrate storyboard-driven testing into @adcp/client

[bokelley]

Summary

Storyboards should be the primary testing surface in @adcp/client, replacing the current scenario-based comply system. Today, storyboard runs call comply() which runs ALL scenarios, then retrofits results onto storyboard steps via comply_scenario string matching. Only 26 of 63 steps have mappings — five media buy storyboards are completely dark.

Problem

Three layers do overlapping work:

1. @adcp/client — testAllScenarios(), individual scenario functions, SCENARIO_REQUIREMENTS
2. Server compliance layer — wraps testAllScenarios() with track grouping
3. Storyboard run endpoint — runs comply(), then matches results back to steps

Storyboards define the exact tasks and expected behavior. The client library has SingleAgentClient with typed methods for every AdCP task. These should be connected directly.

Proposal

Add src/lib/testing/storyboard/ module to @adcp/client:

runStoryboard(agentUrl, storyboard, options)

• Parse storyboard YAML into executable step sequence
• Iterate phases/steps, calling SingleAgentClient methods directly (client.getProducts(), client.buildCreative(), etc.)
• Run per-step validations (schema check, field_present, field_value) defined in the YAML
• Propagate stateful context between steps (e.g., account_id from sync_accounts feeds into get_products)
• Return per-step pass/fail results with timing and error details

runStoryboardStep(client, step, context)

• Execute a single step for interactive/progressive testing
• Returns result + updated context for next step
• Enables step-by-step debugging from the dashboard UI

Unify with platform types

• Platform type → recommended storyboard(s) mapping replaces SCENARIO_REQUIREMENTS + DEFAULT_SCENARIOS
• comply() calls runStoryboard() instead of testAllScenarios()

Storyboard YAML enrichment

Add context_outputs / context_inputs to steps for stateful data flow:

steps:
  - id: sync_accounts
    task: sync_accounts
    context_outputs:
      - path: "accounts[0].account_id"
        key: "account_id"
  - id: get_products_brief
    task: get_products
    context_inputs:
      - key: "account_id"
        inject_at: "account.account_id"

Ship storyboard YAMLs with the package

The docs/storyboards/*.yaml files from the spec repo become the executable test definitions, published alongside @adcp/client.

Task-to-method mapping

Storyboard task	Client method
get_products	client.getProducts()
create_media_buy	client.createMediaBuy()
build_creative	client.buildCreative()
sync_accounts	client.syncAccounts()
list_creative_formats	client.listCreativeFormats()
preview_creative	client.previewCreative()
sync_creatives	client.syncCreatives()
get_media_buys	client.getMediaBuys()
get_media_buy_delivery	client.getMediaBuyDelivery()
get_signals	client.getSignals()

Migration path

1. Build runStoryboard() / runStoryboardStep() in @adcp/client
2. Server's comply() switches from testAllScenarios() to runStoryboard()
3. Dashboard gets step-by-step execution via runStoryboardStep()
4. Deprecate SCENARIO_REQUIREMENTS, DEFAULT_SCENARIOS, individual scenario functions
5. Platform types map to storyboards instead of scenario lists

Related

• Storyboard YAML definitions: adcontextprotocol/adcp docs/storyboards/*.yaml (12 storyboards)
• Current testing module: src/lib/testing/
• Server comply wrapper: adcontextprotocol/adcp server/src/addie/services/compliance-testing.ts

[bokelley]

Dependency: adcontextprotocol/adcp#1962

PR #1962 adds structured accountability terms to the media buy protocol:

• performance_standards — viewability, IVT, completion rate, brand safety, attention score (array of metric + threshold + vendor)
• measurement_terms — billing measurement vendor + makegood policy (additional_delivery, credit, invoice_adjustment)
• cancellation_policy — notice period + cancellation fee
• required_performance_standards filter on get_products
• TERMS_REJECTED error code

When building media buy storyboards for this issue, guaranteed buy workflows should include:

1. Discovery with required_performance_standards filter (buyer says "I need DV for viewability at 70% MRC")
2. Product response showing seller's default performance_standards and measurement_terms
3. create_media_buy with buyer-proposed performance_standards on package request
4. Confirmed package echoing agreed terms
5. Creative assignment validation (tracker tags from agreed vendors)
6. Step validations checking that TERMS_REJECTED is handled correctly when seller can't meet proposed terms

Schemas: performance-standard.json, measurement-terms.json, cancellation-policy.json, makegood-remedy.json, performance-standard-metric.json

This should be merged before storyboard work begins so the schemas are available in @adcp/client.