eTLD+1 binding does not IDNA-normalize, so a U-label agent URL fails to bind to an A-label brand domain

[KonstantinMirin]

Summary

adcp.signing.etld.host_from applies no IDNA conversion, so same_registrable_domain — the eTLD+1 gate the brand-authorization binding runs on — returns False when the two sides spell the same IDN host differently. A brand that publishes its brand.json under a U-label host and lists its agent under the A-label form (or the reverse) fails the binding.

Separately, the same function is asymmetric with itself on the trailing FQDN-root dot, and its docstring describes behaviour that only one of its two branches has.

Cause

src/adcp/signing/etld.py:73-82:

if "://" in value:
    parts = urlsplit(value)
    host = parts.hostname
    if not host:
        raise ValueError(f"URL has no host: {value!r}")
    return host.lower()                              # :78 — no dot strip, no IDNA
stripped = value.strip().rstrip(".").lower()         # :79 — dot stripped, still no IDNA

Neither branch performs UTS-46 / IDNA-2008 encoding, and tldextract does not do it either — it accepts a U-label and returns a U-label eTLD+1. So the two spellings never reduce to a common form.

The docstring at :63-64 claims the function "trims a single trailing dot (the FQDN root separator) so Example.COM. and example.com compare equal". The bare-host branch does that; the URL branch does not. The bare-host branch also uses rstrip("."), which strips all trailing dots, not one.

Reproduce

from adcp.signing.etld import host_from, registrable_domain, same_registrable_domain

# IDNA gap — the load-bearing one
registrable_domain("straße.de")            # 'straße.de'
registrable_domain("xn--strae-oqa.de")     # 'xn--strae-oqa.de'
same_registrable_domain("https://shop.straße.de/", "xn--strae-oqa.de")
# False   <- same host, two spellings, binding fails

# Trailing-dot asymmetry
host_from("https://Example.COM./")   # 'example.com.'
host_from("Example.COM.")            # 'example.com'
host_from("https://x.example.com..") # 'x.example.com..'
host_from("x.example.com..")         # 'x.example.com'

Impact

The IDNA gap is a real binding failure, fail-closed. src/adcp/signing/brand_authz.py:255 calls same_registrable_domain(agent_url, brand_host) as step 2a of the eTLD+1 binding. For brand_domain="bücher.example" and a brand.json listing https://ads.xn--bcher-kva.example/, the agent is found by _find_listed_agents (byte-equal URL match) but step 2a returns False, the authorized_operators[] fallback does not match either, and the result is BrandAuthorizationResult(False, reason="binding_failed") (brand_authz.py:278-283). A legitimate agent is refused; nothing is wrongly accepted. Scope is limited to brands on IDN hosts, which is a small population — but the failure is silent from the publisher's side and binding_failed points at the wrong thing.

The trailing-dot asymmetry does not currently affect the binding. tldextract strips trailing dots itself, so registrable_domain and same_registrable_domain return identical results for both spellings — verified above (registrable_domain("https://example.com.") and registrable_domain("example.com") both give example.com). The defect is observable only through host_from, which is a public export (src/adcp/signing/__init__.py:195,430), and through the docstring, which is wrong. Worth fixing as correctness and as a precondition for the IDNA change; not worth describing as a security issue.

The workaround already in the tree

src/adcp/adagents.py:378-383 does exactly the conversion host_from is missing, by hand, at its own call site:

def _idna_ascii_host(hostname: str, context: str) -> str:
    try:
        return idna.encode(hostname.rstrip("."), uts46=True).decode("ascii").lower()
    except idna.IDNAError as e:
        raise AdagentsValidationError(...) from e

and applies it to both arguments before calling same_registrable_domain (adagents.py:402-404). That is the redirect path working around this gap locally. It is also a hand-retyped copy of _idna_canonicalize.canonicalize_host steps 1/2/4 and diverges from it in three ways: no leading/trailing .strip(), transitional= left to the idna>=3 default rather than pinned the way the helper pins it, and no IP-literal short-circuit (so a public IPv6-literal redirect target raises "invalid IDNA hostname" rather than passing through). None of the three is reachable today — _normalize_domain and _check_safe_host run upstream of both call sites — so this is a maintenance duplicate rather than a live bug. It disappears entirely if host_from does the conversion.

Suggested direction

Route both branches of host_from through _idna_canonicalize.canonicalize_host, which already strips exactly one trailing dot, lowercases, short-circuits IP literals, and encodes with uts46=True, transitional_processing=False. canonical_formats/references.py:38 establishes that a non-signing module may import that helper, so adagents.py can drop _idna_ascii_host in the same change.

Two decisions the change has to make explicitly rather than incidentally:

1. registrable_domain() return values change for IDN inputs. It is public API; today straße.de returns straße.de, after the change xn--strae-oqa.de. Both sides of same_registrable_domain move together so the predicate stays correct, but adopters logging or persisting the value see it change.
2. canonicalize_host raises on underscore hosts and labels over 63 bytes, where host_from returns them unchanged today. The existing six callers are split between fail-open (ip_pinned_transport.py:119, key_origins.py) and fail-closed (jwks.py, revocation_fetcher.py), so "match the convention" is not available as an answer. Given etld is a comparison path and registrable_domain already documents a fail-closed None contract, mapping the raise onto None is probably right — but it should be stated in the diff.

Related

• @target-uri canonicalization applies no IDNA A-label conversion, and the verifier does not reject raw U-labels #977 / fix(signing): reject malformed authorities, convert IDN hosts to A-labels #985 — the same missing A-label conversion on canonical.py's signature-base path. Different module, different consumer (@authority vs the brand binding); fixing one does not fix the other.
• Canonicalization now strips a trailing FQDN-root dot; the spec defines no rule and ships no vector #986 — trailing-root-dot handling in canonical.py. That issue asks upstream to define the rule for the signature base, where the byte value is on the wire. etld's dot handling is internal to eTLD+1 comparison and is not spec-visible, so it does not need the same upstream decision.
• Migrate IDNA canonicalization from stdlib (IDNA 2003) to PyPI idna (IDNA 2008) across signing/ #777 (closed) — migrated IDNA canonicalization from the stdlib (IDNA 2003) to the PyPI idna package (IDNA 2008) "across signing/". etld.py was not converted in that pass; host_from still does a bare .lower(). This is residue of that migration rather than a new regression — which is also why the fix here is to call the helper Migrate IDNA canonicalization from stdlib (IDNA 2003) to PyPI idna (IDNA 2008) across signing/ #777 established rather than to introduce another one.

[bokelley]

Triage

Classification: Bug
Bucket(s): signing (label not in repo — flagged)
Status: ready-for-human

What the experts said:

• ad-tech-protocol-expert: Non-breaking bug fix; ADCP spec #3690 mandates A-label normalization in the eTLD+1 path; routing host_from through canonicalize_host is correct; old U-label output was never a valid contract value, so the return-value change for IDN inputs is not a SemVer break; IDNAError → None in registrable_domain() is consistent with its documented fail-closed contract.
• code-reviewer: Approach is sound; canonicalize_host is the right consolidation (already 4 consumers in signing); host_from() must not leak idna.IDNAError through its public contract — re-raise as ValueError in host_from or catch and map to None in registrable_domain(); removing _idna_ascii_host is safe but degrades error message at adagents.py:405; tag as semver-minor, not patch.
• security-reviewer: No fail-open vector in the normalization itself (UTS-46 confusables produce distinct punycode, not ASCII collisions). Two exception-path mitigations required before shipping: (1) same_registrable_domain at brand_authz.py:255 is unguarded — an IDNA-invalid agent_url currently propagates an exception out of is_authorized() rather than returning BrandAuthorizationResult(False, ...); (2) if _idna_ascii_host is removed from adagents.py, the IDNAError → AdagentsValidationError wrapping at lines 402–403 must be preserved, because the caller at line 1051 catches only (AdagentsValidationError, AdagentsTimeoutError) and a raw IDNAError would escape uncaught.

My take: Fix direction is clear and spec-mandated; all three experts agree on the approach. Flagging for human review because the two exception-path changes touch the live brand-authorization binding and the adagents redirect guard and warrant explicit sign-off before the change ships to IDN-facing deployments. @bokelley

---

Triaged by Claude Code. Session: https://claude.ai/code/${CLAUDE_CODE_REMOTE_SESSION_ID}

---

Generated by Claude Code