Add verified browser action plans - #65
Open
rgarcia wants to merge 1 commit into
Open
Conversation
This was referenced Jul 14, 2026
Contributor
Author
|
bugbot run |
![cursor[bot]](https://cdn.github.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 2d43984. Configure here.
rgarcia
force-pushed
the
hypeship/browser-semantic-waits
branch
from
July 14, 2026 03:08
092e9a6 to
f187fc1
Compare
rgarcia
force-pushed
the
hypeship/browser-action-plans
branch
from
July 14, 2026 03:08
2d43984 to
32495d9
Compare
Contributor
Author
|
bugbot run |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.
Autofix Details
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Batch summary ignores failed waits
- The batch status now always surfaces failed browser waits before action-plan outcomes, so mixed batches no longer hide wait failures.
- ✅ Fixed: Missing TSDoc on exported formatter
- Added TSDoc to
formatBrowserActResultdescribing its purpose, inputs, and why diff output is truncated versus structured details.
- Added TSDoc to
Or push these changes by commenting:
@cursor push 33c635dbb5
Preview (33c635dbb5)
diff --git a/packages/agent/src/tools.ts b/packages/agent/src/tools.ts
--- a/packages/agent/src/tools.ts
+++ b/packages/agent/src/tools.ts
@@ -199,17 +199,15 @@
const acts = readResults.flatMap((read) => read.type === "browser_act" ? [read.result] : []);
const waits = readResults.flatMap((read) => read.type === "browser_wait_for" ? [read.result] : []);
const failedWait = ["interrupted", "timed_out", "unverifiable"].find((status) => waits.some((wait) => wait.status === status));
- const statusText = acts.length > 0
+ const actStatusText = acts.length > 0
? acts.some((act) => act.outcome === "didnt")
? "Browser action plan did not satisfy its expectations."
: acts.every((act) => act.outcome === "worked")
? "Browser action plan worked."
: "Browser action plan outcome is unknown."
- : waits.length === 0
- ? "Actions executed successfully."
- : failedWait
- ? `Browser condition ${failedWait}.`
- : "Browser condition satisfied.";
+ : undefined;
+ const waitStatusText = waits.length === 0 ? undefined : failedWait ? `Browser condition ${failedWait}.` : "Browser condition satisfied.";
+ const statusText = failedWait ? `Browser condition ${failedWait}.` : actStatusText ?? waitStatusText ?? "Actions executed successfully.";
return { content, details: { statusText, readResults } };
}
@@ -287,6 +285,11 @@
return [`wait_for: ${result.status}/${result.evidence}${reason} after ${result.elapsed_ms}ms`, ...result.details].join("\n");
}
+/**
+ * Formats a browser action plan result for model-facing tool text output.
+ * Includes per-step outcomes and successor details while capping successor
+ * diff lines so large diffs stay in structured `readResults`, not tool text.
+ */
export function formatBrowserActResult(result: BrowserActResult): string {
const lines = [`browser_act: ${result.outcome}`];
if (result.stopped_at !== undefined) lines.push(`stopped_at: ${result.stopped_at} (${result.stop_reason ?? "unknown"})`);You can send follow-ups to the cloud agent here.
rgarcia
force-pushed
the
hypeship/browser-semantic-waits
branch
from
July 14, 2026 03:17
f187fc1 to
1ad2322
Compare
rgarcia
force-pushed
the
hypeship/browser-action-plans
branch
2 times, most recently
from
July 14, 2026 03:27
abf99f3 to
1bf73c9
Compare
Contributor
Author
|
bugbot run |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.
Autofix Details
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Baseline wait aborts navigation expects
- I allowed location expectations with an explicit pre-action baseline to continue past initial navigation drift so the poll loop can mark successful navigations as newly verified.
- ✅ Fixed: Mixed batch ignores wait failures
- I updated mixed-batch execution to stop after browser_wait_for results with interrupted, timed_out, or unverifiable statuses, matching the existing browser_act stop behavior.
Or push these changes by commenting:
@cursor push fbc43fc737
Preview (fbc43fc737)
diff --git a/packages/agent/src/translator/browser-wait.ts b/packages/agent/src/translator/browser-wait.ts
--- a/packages/agent/src/translator/browser-wait.ts
+++ b/packages/agent/src/translator/browser-wait.ts
@@ -109,7 +109,9 @@
}
const initial = evaluateBrowserExpectation(options.expect, baseline, baseline, runtime.resolveRef);
if (!baseline.complete) return terminal("unverifiable", "unverifiable", initial, initial, started, now(), "incomplete_observation");
- if (runtime.liveNavigationEpoch(targetId) !== baseline.navigationEpoch || [...baseline.generations].some(([key, generation]) => runtime.liveGeneration(key) !== generation)) return terminal("interrupted", "unverifiable", initial, initial, started, now(), "navigation");
+ const baselineNavigated = runtime.liveNavigationEpoch(targetId) !== baseline.navigationEpoch || [...baseline.generations].some(([key, generation]) => runtime.liveGeneration(key) !== generation);
+ const allowLocationRecovery = !!options.baseline && isLocationExpectation(options.expect);
+ if (baselineNavigated && !allowLocationRecovery) return terminal("interrupted", "unverifiable", initial, initial, started, now(), "navigation");
if (expired()) return timedOut(started, now());
if (initial.reason === "stale_ref") return terminal("interrupted", "unverifiable", initial, initial, started, now(), initial.reason);
if (initial.truth === true && !options.baseline) return terminal("satisfied", "preexisting", initial, initial, started, now());
diff --git a/packages/agent/src/translator/translator.ts b/packages/agent/src/translator/translator.ts
--- a/packages/agent/src/translator/translator.ts
+++ b/packages/agent/src/translator/translator.ts
@@ -136,7 +136,10 @@
await flush();
const reads = await this.browser().execute(action);
result.readResults.push(...reads);
- if (action.type === "browser_act" && reads.some((read) => read.type === "browser_act" && read.result.stop_reason)) break;
+ if (
+ (action.type === "browser_act" && reads.some((read) => read.type === "browser_act" && read.result.stop_reason)) ||
+ (action.type === "browser_wait_for" && reads.some((read) => read.type === "browser_wait_for" && (read.result.status === "interrupted" || read.result.status === "timed_out" || read.result.status === "unverifiable")))
+ ) break;
continue;
}
switch (action.type) {
diff --git a/packages/agent/test/browser-wait.test.ts b/packages/agent/test/browser-wait.test.ts
--- a/packages/agent/test/browser-wait.test.ts
+++ b/packages/agent/test/browser-wait.test.ts
@@ -131,6 +131,23 @@
expect(result).toMatchObject({ status: "satisfied", evidence: "newly_verified" });
});
+ it("allows location expectations to verify when a pre-action baseline is already stale", async () => {
+ let time = 0;
+ const baseline = observation([], true, { url: "https://a.test/start", generations: new Map([["page", 0]]) });
+ const result = await waitForBrowserExpectation({
+ selectTarget: async () => "page",
+ observeTarget: async () => observation([], true, { url: "https://a.test/done", navigationEpoch: 1, generations: new Map([["page", 1]]) }),
+ dialogCount: () => 0,
+ targetExists: async () => true,
+ liveGeneration: () => 1,
+ liveNavigationEpoch: () => 1,
+ resolveRef: missingRef,
+ now: () => time,
+ delay: async (ms) => { time += ms; },
+ }, { expect: { type: "url", changed: true }, baseline, targetId: "page", timeoutMs: 20, pollMs: 10 });
+ expect(result).toMatchObject({ status: "satisfied", evidence: "newly_verified" });
+ });
+
it("reports navigation instead of timeout when a location change lands at the deadline", async () => {
let time = 0, reads = 0;
const result = await waitForBrowserExpectation({
diff --git a/packages/agent/test/translator-browser.test.ts b/packages/agent/test/translator-browser.test.ts
--- a/packages/agent/test/translator-browser.test.ts
+++ b/packages/agent/test/translator-browser.test.ts
@@ -300,6 +300,30 @@
await translator.executeBatch([{ type: "browser_act", steps: [{ type: "wait" }] }, { type: "browser_text" }]);
expect(executed).toEqual(["browser_act"]);
});
+
+ it.each(["interrupted", "timed_out", "unverifiable"] as const)("stops a mixed batch after a browser_wait_for %s result", async (status) => {
+ const { client } = createClient();
+ const executed: string[] = [];
+ const result: BrowserWaitForResult = {
+ status,
+ evidence: status === "timed_out" ? "failed" : "unverifiable",
+ initial: { truth: false, details: ["before"] },
+ final: { truth: false, details: ["after"] },
+ elapsed_ms: 1,
+ ...(status === "interrupted" ? { reason: "navigation" } : {}),
+ details: [],
+ };
+ const executor = { execute: async (action: CuaBrowserAction) => {
+ executed.push(action.type);
+ return action.type === "browser_wait_for" ? [{ type: "browser_wait_for", result } as BatchReadResult] : [];
+ } } as unknown as BrowserExecutor;
+ const translator = new InternalComputerTranslator({ browser, client, createBrowserExecutor: () => executor });
+ await translator.executeBatch([
+ { type: "browser_wait_for", expect: { type: "text", text: "Ready" } },
+ { type: "browser_text" },
+ ]);
+ expect(executed).toEqual(["browser_wait_for"]);
+ });
});
describe("InternalComputerTranslator computer additions", () => {You can send follow-ups to the cloud agent here.
rgarcia
force-pushed
the
hypeship/browser-semantic-waits
branch
from
July 14, 2026 03:39
1ad2322 to
07370f2
Compare
rgarcia
force-pushed
the
hypeship/browser-action-plans
branch
from
July 14, 2026 03:39
1bf73c9 to
56c86e1
Compare
Contributor
Author
|
bugbot run |
rgarcia
force-pushed
the
hypeship/browser-action-plans
branch
from
July 14, 2026 03:47
56c86e1 to
626c63b
Compare
Contributor
Author
|
bugbot run |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.
Autofix Details
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Stop reason masks failed expectation
- I reordered stop-reason resolution so failed semantic waits are classified before post-action observation completeness fallbacks.
- ✅ Fixed: Misleading stopped_at on successor race
- I removed the late-boundary fallback that assigned
stopped_atduring successor collection when no step loop interruption occurred.
- I removed the late-boundary fallback that assigned
Or push these changes by commenting:
@cursor push a17aa0b911
Preview (a17aa0b911)
diff --git a/packages/agent/src/translator/browser-act.ts b/packages/agent/src/translator/browser-act.ts
--- a/packages/agent/src/translator/browser-act.ts
+++ b/packages/agent/src/translator/browser-act.ts
@@ -88,13 +88,14 @@
const postBoundary = after && afterTargets ? boundary(before, after, targets, afterTargets, dialogs, runtime) : undefined;
if (after && afterTargets) { current = after; targets = afterTargets; dialogs = runtime.dialogCount(); }
- stopReason = (!after?.complete ? "control_flow" : undefined)
- ?? postBoundary
+ stopReason = postBoundary
?? waitStopReason(waitResult)
?? (stale ? "stale_ref"
: actionError ? "action_failed"
: expectation?.status === "failed" ? "expectation_failed"
- : expectation?.status === "preexisting" || expectation?.status === "unverifiable" || !after ? "control_flow" : undefined);
+ : undefined)
+ ?? (!after?.complete ? "control_flow" : undefined)
+ ?? (expectation?.status === "preexisting" || expectation?.status === "unverifiable" || !after ? "control_flow" : undefined);
if (stopReason) { stoppedAt = index; break; }
}
@@ -123,7 +124,6 @@
current = observed; targets = successorTargets; dialogs = runtime.dialogCount();
if (lateBoundary) {
stopReason ??= lateBoundary;
- stoppedAt ??= finalStepIndex;
successorError = new Error(`${lateBoundary} changed successor observation`);
continue;
}
diff --git a/packages/agent/test/translator-browser.test.ts b/packages/agent/test/translator-browser.test.ts
--- a/packages/agent/test/translator-browser.test.ts
+++ b/packages/agent/test/translator-browser.test.ts
@@ -156,6 +156,13 @@
expect(rt.dispatched).toEqual(["click"]);
});
+ it("keeps expectation_failed when post-action observation fails", async () => {
+ const result = await runBrowserAct({ type: "browser_act", steps: [{ type: "click", ref: "e1", expect: { type: "text", text: "Missing" } }] }, runtime([
+ observation("before"), observation("before"), observation("after"),
+ ], [waitResult("failed")], { failObservationAt: 2 }));
+ expect(result).toMatchObject({ outcome: "didnt", stopped_at: 0, stop_reason: "expectation_failed", steps: [{ outcome: "didnt", expectation: { status: "failed" } }] });
+ });
+
it("stops on a stale ref without dispatching later steps", async () => {
const rt = runtime([observation("before"), observation("before"), observation("successor")], [], { dispatchError: new Error("ref e1 is stale") });
const result = await runBrowserAct({ type: "browser_act", steps: [{ type: "click", ref: "e1" }, { type: "type", text: "no" }] }, rt);
@@ -190,6 +197,7 @@
observation("before"), observation("before"), observation("after"), observation("stale"), observation("stable"),
], [waitResult("newly_verified")], { targets: [["target-1"], ["target-1"], ["target-1"], ["target-1", "target-2"], ["target-1", "target-2"]] }));
expect(result).toMatchObject({ stop_reason: "control_flow", successor: { status: "observed", title: "stable", text: "stable" } });
+ expect(result).not.toHaveProperty("stopped_at");
});
it("preserves a timed-out final expectation when the successor satisfies it", async () => {You can send follow-ups to the cloud agent here.
rgarcia
force-pushed
the
hypeship/browser-semantic-waits
branch
from
July 14, 2026 03:56
07370f2 to
cf78571
Compare
rgarcia
force-pushed
the
hypeship/browser-action-plans
branch
from
July 14, 2026 03:56
626c63b to
dc039c6
Compare
Contributor
Author
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
1 issue from previous review remains unresolved.
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit dc039c6. Configure here.
rgarcia
force-pushed
the
hypeship/browser-semantic-waits
branch
from
July 14, 2026 04:05
cf78571 to
eac9a33
Compare
rgarcia
force-pushed
the
hypeship/browser-action-plans
branch
from
July 14, 2026 04:05
dc039c6 to
6f97b80
Compare
Contributor
Author
|
bugbot run |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Final expect upgrade leaves stop reason
- When successor evaluation upgrades an unverifiable final expectation to true, runBrowserAct now clears the inherited stop_reason/stopped_at so the verified result reports worked consistently.
Or push these changes by commenting:
@cursor push 8158041927
Preview (8158041927)
diff --git a/packages/agent/src/translator/browser-act.ts b/packages/agent/src/translator/browser-act.ts
--- a/packages/agent/src/translator/browser-act.ts
+++ b/packages/agent/src/translator/browser-act.ts
@@ -131,11 +131,16 @@
if (!observed.complete) throw new Error("successor observation incomplete");
if (action.expect && finalExpectation && finalExpectation.status !== "failed") {
const evaluation = runtime.evaluate(action.expect, observed, baseline);
+ const priorFinalStatus = finalExpectation.status;
finalExpectation = {
status: evaluation.truth === true ? finalExpectation.before === true ? "preexisting" : "newly_verified" : evaluation.truth === false ? "failed" : "unverifiable",
before: finalExpectation.before, after: evaluation.truth,
details: [...finalExpectation.details, ...evaluation.details.map((detail) => `successor: ${detail}`)],
};
+ if (evaluation.truth === true && priorFinalStatus === "unverifiable") {
+ stopReason = undefined;
+ stoppedAt = undefined;
+ }
if (evaluation.truth !== true) {
stopReason = evaluation.truth === false ? "expectation_failed" : "control_flow";
stoppedAt = finalStepIndex;
diff --git a/packages/agent/test/translator-browser.test.ts b/packages/agent/test/translator-browser.test.ts
--- a/packages/agent/test/translator-browser.test.ts
+++ b/packages/agent/test/translator-browser.test.ts
@@ -199,6 +199,15 @@
expect(result).toMatchObject({ outcome: "didnt", stop_reason: "expectation_failed", final_expectation: { status: "failed", after: false }, successor: { status: "observed", title: "after" } });
});
+ it("clears an unverifiable final stop reason when successor verification succeeds", async () => {
+ const result = await runBrowserAct({ type: "browser_act", steps: [{ type: "wait" }], expect: { type: "url", contains: "after" } }, runtime([
+ observation("before"), observation("before"), observation("after"), observation("after"),
+ ], [waitResult("unverifiable")]));
+ expect(result).toMatchObject({ outcome: "worked", final_expectation: { status: "newly_verified", after: true }, successor: { status: "observed", title: "after" } });
+ expect(result).not.toHaveProperty("stop_reason");
+ expect(result).not.toHaveProperty("stopped_at");
+ });
+
it("does not report incomplete successor observations as authoritative", async () => {
const incomplete = { ...observation("after"), complete: false };
const result = await runBrowserAct({ type: "browser_act", steps: [{ type: "click", ref: "e1", expect: { type: "text", text: "Done" } }] }, runtime([You can send follow-ups to the cloud agent here.
rgarcia
force-pushed
the
hypeship/browser-semantic-waits
branch
from
July 14, 2026 04:13
eac9a33 to
8fd179f
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Plan expect skipped on navigation
- Terminal navigation now still runs and revalidates the plan-level expectation at the last step even when the step has no per-step expect, and verified plan expectations now count as verified navigation outcomes.
Or push these changes by commenting:
@cursor push f044da8cbe
Preview (f044da8cbe)
diff --git a/packages/agent/src/translator/browser-act.ts b/packages/agent/src/translator/browser-act.ts
--- a/packages/agent/src/translator/browser-act.ts
+++ b/packages/agent/src/translator/browser-act.ts
@@ -101,7 +101,7 @@
}
let finalExpectation: BrowserActExpectationEvidence | undefined;
- const terminalNavigation = stopReason === "navigation" && stoppedAt === finalStepIndex && steps.at(-1)?.outcome === "worked";
+ const terminalNavigation = stopReason === "navigation" && stoppedAt === finalStepIndex;
if (action.expect && (!stopReason || terminalNavigation)) {
try {
const result = await runtime.wait(action.expect!, baseline, baseline.targetId, action.tab_id, action.timeout_ms, action.poll_ms);
@@ -163,7 +163,8 @@
const verified = action.expect
? finalExpectation?.status === "newly_verified"
: steps.length === action.steps.length && steps.every((step) => step.outcome === "worked");
- const verifiedNavigation = stopReason === "navigation" && steps.length === action.steps.length && steps.at(-1)?.outcome === "worked";
+ const verifiedNavigation = stopReason === "navigation" && steps.length === action.steps.length
+ && (steps.at(-1)?.outcome === "worked" || finalExpectation?.status === "newly_verified");
return {
outcome: failed ? "didnt" : !timedOut && verified && (!stopReason || verifiedNavigation) ? "worked" : "unknown",
steps,
diff --git a/packages/agent/test/translator-browser.test.ts b/packages/agent/test/translator-browser.test.ts
--- a/packages/agent/test/translator-browser.test.ts
+++ b/packages/agent/test/translator-browser.test.ts
@@ -185,6 +185,17 @@
expect(result).toMatchObject({ outcome: "worked", stopped_at: 0, stop_reason: "navigation", final_expectation: { status: "newly_verified" } });
});
+ it("evaluates plan expectations after terminal navigation without step expectations", async () => {
+ const before = observation("before");
+ const after = observation("after", 1);
+ const result = await runBrowserAct({
+ type: "browser_act",
+ steps: [{ type: "click", ref: "e1" }],
+ expect: { type: "url", contains: "after" },
+ }, runtime([before, before, after, after], [waitResult("newly_verified")]));
+ expect(result).toMatchObject({ outcome: "worked", stopped_at: 0, stop_reason: "navigation", final_expectation: { status: "newly_verified" } });
+ });
+
it("recollects a successor after a raced target boundary", async () => {
const result = await runBrowserAct({ type: "browser_act", steps: [{ type: "click", ref: "e1", expect: { type: "text", text: "Done" } }] }, runtime([
observation("before"), observation("before"), observation("after"), observation("stale"), observation("stable"),You can send follow-ups to the cloud agent here.
rgarcia
force-pushed
the
hypeship/browser-semantic-waits
branch
from
July 14, 2026 04:22
8fd179f to
7b76f06
Compare
rgarcia
force-pushed
the
hypeship/browser-action-plans
branch
from
July 14, 2026 04:22
9c821b9 to
8056413
Compare
rgarcia
force-pushed
the
hypeship/browser-semantic-waits
branch
from
July 24, 2026 16:27
7b76f06 to
3892983
Compare
rgarcia
force-pushed
the
hypeship/browser-action-plans
branch
from
July 24, 2026 18:54
8056413 to
4058969
Compare
rgarcia
force-pushed
the
hypeship/browser-action-plans
branch
3 times, most recently
from
July 27, 2026 15:24
625ade5 to
e77e923
Compare
Contributor
Author
|
bugbot run |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.
Autofix Details
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Failed step expects misreported global timeout
- I only treat timed-out semantic waits as timeouts when evidence is not failed, so failed waits now report expectation_failed and still attempt successor observation.
- ✅ Fixed: Worked act still skips batch
- Batch stopping now ignores browser_act stop_reason when that act outcome is worked, so later independent actions in mixed batches continue executing.
Or push these changes by commenting:
@cursor push 2a4d485fb9
Preview (2a4d485fb9)
diff --git a/packages/agent/src/index.ts b/packages/agent/src/index.ts
--- a/packages/agent/src/index.ts
+++ b/packages/agent/src/index.ts
@@ -10,7 +10,19 @@
export type {
BatchExecutionResult,
BatchReadResult,
+ BrowserActExpectationEvidence,
+ BrowserActExpectationStatus,
+ BrowserActObservedSuccessor,
+ BrowserActOutcome,
+ BrowserActResult,
+ BrowserActStepResult,
+ BrowserActStopReason,
+ BrowserActSuccessor,
+ BrowserActUnavailableSuccessor,
BrowserExpectationEvidence,
+ BrowserExpectationState,
+ BrowserObservationDiff,
+ BrowserObservationDiffEntry,
BrowserWaitForResult,
BrowserWaitReason,
} from "./translator/types";
diff --git a/packages/agent/src/tools.ts b/packages/agent/src/tools.ts
--- a/packages/agent/src/tools.ts
+++ b/packages/agent/src/tools.ts
@@ -16,8 +16,12 @@
} from "@onkernel/cua-ai";
import { InternalComputerTranslator, type KernelBrowser } from "./translator/translator";
import type { AgentTool, AgentToolResult } from "@earendil-works/pi-agent-core";
-import type { BrowserWaitForResult } from "./translator/types";
+import type { BrowserActResult, BrowserWaitForResult } from "./translator/types";
+const BROWSER_ACT_DIFF_ENTRY_LIMIT = 200;
+const BROWSER_ACT_DIFF_CHAR_LIMIT = 20_000;
+const BROWSER_ACT_RESULT_CHAR_LIMIT = 50_000;
+
export interface ComputerToolOptions {
browser: KernelBrowser;
client: Kernel;
@@ -41,6 +45,7 @@
| { type: "cursor_position"; x: number; y: number }
| { type: "browser_text"; label: string; bytes: number }
| { type: "browser_wait_for"; result: BrowserWaitForResult }
+ | { type: "browser_act"; result: BrowserActResult }
>;
}
@@ -179,6 +184,9 @@
} else if (read.type === "browser_wait_for") {
readResults.push(read);
content.push({ type: "text", text: formatBrowserWaitResult(read.result) });
+ } else if (read.type === "browser_act") {
+ readResults.push(read);
+ content.push({ type: "text", text: formatBrowserActResult(read.result) });
} else {
readResults.push({ type: "screenshot", bytes: read.data.length });
content.push({ type: "image", data: read.data.toString("base64"), mimeType: read.mimeType });
@@ -194,13 +202,18 @@
} catch (err) {
throw new Error(`Actions failed: ${errorMessage(err)}`, { cause: err });
}
+ const acts = readResults.flatMap((read) => read.type === "browser_act" ? [read.result] : []);
const waits = readResults.flatMap((read) => read.type === "browser_wait_for" ? [read.result] : []);
const failedWait = ["interrupted", "timed_out", "unverifiable"].find((status) => waits.some((wait) => wait.status === status));
- let statusText = waits.length === 0
- ? "Actions executed successfully."
+ let statusText = acts.some((act) => act.outcome === "didnt")
+ ? "Browser action plan did not satisfy its expectations."
: failedWait
? `Browser condition ${failedWait}.`
- : "Browser condition satisfied.";
+ : acts.some((act) => act.outcome === "unknown")
+ ? "Browser action plan outcome is unknown."
+ : acts.length > 0
+ ? "Browser action plan worked."
+ : waits.length > 0 ? "Browser condition satisfied." : "Actions executed successfully.";
if (skippedActions) {
const skipped = `${skippedActions} subsequent action${skippedActions === 1 ? " was" : "s were"} skipped.`;
statusText = `${statusText} ${skipped}`;
@@ -290,6 +303,52 @@
return [`wait_for: ${result.status}/${result.evidence}${reason} after ${result.elapsed_ms}ms`, ...result.details].join("\n");
}
+/** Format model-facing plan feedback with bounded diff entries, diff characters, and total characters. */
+export function formatBrowserActResult(result: BrowserActResult): string {
+ const lines = [`browser_act: ${result.outcome}`];
+ if (result.stopped_at !== undefined) lines.push(`stopped_at: ${result.stopped_at} (${result.stop_reason ?? "unknown"})`);
+ for (const step of result.steps) {
+ lines.push(`step ${step.index} ${step.type}: ${step.outcome} — ${step.diagnostics.join("; ")}`);
+ for (const diagnostic of step.expectation?.diagnostics ?? []) lines.push(` ${diagnostic}`);
+ }
+ if (result.final_expectation) {
+ lines.push(`final expectation: ${result.final_expectation.status}`);
+ for (const diagnostic of result.final_expectation.diagnostics) lines.push(` ${diagnostic}`);
+ }
+ if (result.successor.status === "unavailable") {
+ lines.push(`successor unavailable: ${result.successor.error}`);
+ return boundedBrowserActOutput(lines);
+ }
+ const { diff } = result.successor;
+ const addedCount = diff.added.reduce((total, entry) => total + entry.count, 0);
+ const removedCount = diff.removed.reduce((total, entry) => total + entry.count, 0);
+ lines.push(`successor: ${result.successor.title} (${result.successor.url})`);
+ lines.push(`diff: ${diff.changed ? `+${addedCount} -${removedCount}` : "unchanged"}`);
+ if (diff.url) lines.push(` url: ${diff.url.before} -> ${diff.url.after}`);
+ if (diff.title) lines.push(` title: ${diff.title.before} -> ${diff.title.after}`);
+ const changes = [
+ ...diff.added.map((entry) => ` + ${entry.line}${entry.count === 1 ? "" : ` ×${entry.count}`}`),
+ ...diff.removed.map((entry) => ` - ${entry.line}${entry.count === 1 ? "" : ` ×${entry.count}`}`),
+ ];
+ let emittedChars = 0;
+ let emittedEntries = 0;
+ for (const change of changes) {
+ if (emittedEntries >= BROWSER_ACT_DIFF_ENTRY_LIMIT || emittedChars + change.length > BROWSER_ACT_DIFF_CHAR_LIMIT) break;
+ lines.push(change);
+ emittedEntries += 1;
+ emittedChars += change.length;
+ }
+ if (emittedEntries < changes.length) lines.push(` … ${changes.length - emittedEntries} more diff entries omitted`);
+ lines.push(result.successor.text);
+ return boundedBrowserActOutput(lines);
+}
+
+function boundedBrowserActOutput(lines: readonly string[]): string {
+ const output = lines.join("\n");
+ if (output.length <= BROWSER_ACT_RESULT_CHAR_LIMIT) return output;
+ return `${output.slice(0, BROWSER_ACT_RESULT_CHAR_LIMIT)}\n… browser_act output truncated at ${BROWSER_ACT_RESULT_CHAR_LIMIT} characters`;
+}
+
function formatPlaywrightResult(result: unknown): string {
return typeof result === "string" ? result : JSON.stringify(result);
}
diff --git a/packages/agent/src/translator/browser-act.ts b/packages/agent/src/translator/browser-act.ts
new file mode 100644
--- /dev/null
+++ b/packages/agent/src/translator/browser-act.ts
@@ -1,0 +1,358 @@
+import type { CuaActionBrowserAct, CuaActionBrowserSnapshot, CuaBrowserActStep, CuaBrowserExpectation } from "@onkernel/cua-ai";
+import { diffObservations, type BrowserObservation, type BrowserPresentation } from "./browser-observation";
+import type { BrowserExpectationEvaluation } from "./browser-wait";
+import type { BrowserActExpectationEvidence, BrowserActResult, BrowserActStepResult, BrowserWaitForResult } from "./types";
+
+const DEFAULT_ACT_TIMEOUT_MS = 30_000;
+type ActTimeoutReason = "global_timeout" | "step_timeout";
+type ActDeadline = { at: number; reason: ActTimeoutReason };
+
+class BrowserActDeadlineError extends Error {
+ constructor(readonly reason: ActTimeoutReason) {
+ super(reason === "global_timeout" ? "browser action plan timed out" : "browser action step timed out");
+ }
+}
+
+/**
+ * Narrow adapter between plan policy and browser mechanics. Implementations own live
+ * target/ref/CDP state; the orchestrator owns sequencing, deadlines, attribution, and
+ * stop decisions. `observe` is intentionally a full fenced AX observation: one baseline
+ * plus pre/post observations around steps make causal claims safer, but are not cheap.
+ */
+export interface BrowserActRuntime {
+ observe(tabId?: string): Promise<BrowserObservation>;
+ targetIds(): Promise<string[]>;
+ dialogCount(): number;
+ liveGeneration(frameId: string): number;
+ liveNavigationEpoch(targetId: string): number;
+ executeStep(step: CuaBrowserActStep, tabId?: string): Promise<void>;
+ wait(expect: CuaBrowserExpectation, baseline: BrowserObservation, targetId: string, tabId?: string, timeoutMs?: number, pollMs?: number): Promise<BrowserWaitForResult>;
+ evaluate(expect: CuaBrowserExpectation, observation: BrowserObservation, baseline: BrowserObservation): BrowserExpectationEvaluation;
+ present(observation: BrowserObservation, action: CuaActionBrowserSnapshot): BrowserPresentation;
+ render(presentation: BrowserPresentation): string;
+}
+
+/**
+ * Execute a dependent action plan and derive causal outcomes, stop reasons, and a stable
+ * successor. Expectations are evaluated against observations captured before input; a
+ * condition already matched before input is `preexisting`, never proof that input worked.
+ */
+export async function runBrowserAct(action: CuaActionBrowserAct, runtime: BrowserActRuntime): Promise<BrowserActResult> {
+ const finalStepIndex = action.steps.length - 1;
+ const globalDeadline: ActDeadline = { at: Date.now() + (action.timeout_ms ?? DEFAULT_ACT_TIMEOUT_MS), reason: "global_timeout" };
+ let baseline: BrowserObservation;
+ let current: BrowserObservation;
+ let targets: string[];
+ try {
+ baseline = current = await beforeDeadline(() => runtime.observe(action.tab_id), globalDeadline);
+ if (!isComplete(baseline)) throw new Error("baseline observation incomplete");
+ targets = await beforeDeadline(() => runtime.targetIds(), globalDeadline);
+ } catch (error) {
+ return unavailable(error, 0);
+ }
+ let dialogs = runtime.dialogCount();
+ const steps: BrowserActStepResult[] = [];
+ let stoppedAt: number | undefined;
+ let stopReason: BrowserActResult["stop_reason"];
+ let timedOut = false;
+
+ for (let index = 0; index < action.steps.length; index += 1) {
+ const step = action.steps[index]!;
+ const deadline = stepDeadline(step, globalDeadline);
+ let before: BrowserObservation;
+ let nextTargets: string[];
+ try {
+ before = await beforeDeadline(() => runtime.observe(action.tab_id), deadline);
+ nextTargets = await beforeDeadline(() => runtime.targetIds(), deadline);
+ } catch (error) {
+ const timeout = timeoutReason(error);
+ steps.push(stepResult(index, step, "unknown", [timeout ? message(error) : `pre-action observation failed: ${message(error)}`]));
+ timedOut ||= timeout !== undefined;
+ stoppedAt = index; stopReason = timeout ?? "control_flow"; break;
+ }
+ const preBoundary = boundary(current, before, targets, nextTargets, dialogs, runtime);
+ current = before; targets = nextTargets; dialogs = runtime.dialogCount();
+ if (!isComplete(before) || preBoundary) {
+ const reason = isComplete(before) ? `${preBoundary} detected before action` : "pre-action observation incomplete";
+ steps.push(stepResult(index, step, "unknown", [reason]));
+ stoppedAt = index; stopReason = preBoundary ?? "control_flow"; break;
+ }
+
+ const diagnostics: string[] = [];
+ let actionError: unknown;
+ let timeout: ActTimeoutReason | undefined;
+ try {
+ await beforeDeadline(() => runtime.executeStep(step, action.tab_id), deadline);
+ diagnostics.push("action dispatched");
+ } catch (error) {
+ actionError = error;
+ timeout = timeoutReason(error);
+ diagnostics.push(message(error));
+ }
+
+ let expectation: BrowserActExpectationEvidence | undefined;
+ let waitResult: BrowserWaitForResult | undefined;
+ let after: BrowserObservation | undefined;
+ let afterTargets: string[] | undefined;
+ if (!timeout) {
+ try {
+ if (step.expect) {
+ waitResult = await beforeDeadline(
+ () => runtime.wait(step.expect!, before, before.targetId, action.tab_id, remaining(deadline), action.poll_ms),
+ deadline,
+ );
+ expectation = expectationEvidence(waitResult);
+ if (waitResult.status === "timed_out" && expectation.status !== "failed") timeout = deadline.reason;
+ }
+ if (!timeout) {
+ after = await beforeDeadline(() => runtime.observe(action.tab_id), deadline);
+ afterTargets = await beforeDeadline(() => runtime.targetIds(), deadline);
+ }
+ } catch (error) {
+ timeout = timeoutReason(error);
+ diagnostics.push(timeout ? message(error) : `post-action observation failed: ${message(error)}`);
+ }
+ }
+
+ timedOut ||= timeout !== undefined;
+ const stale = isStale(actionError) || waitResult?.reason === "stale_ref" || expectation?.status === "unverifiable" && expectation.reason === "stale_ref";
+ const outcome = stepOutcome(expectation, actionError, stale);
+ if (expectation) diagnostics.push(`expectation ${expectation.status}`);
+ steps.push(stepResult(index, step, outcome, diagnostics, expectation));
+
+ const postBoundary = after && afterTargets ? boundary(before, after, targets, afterTargets, dialogs, runtime) : undefined;
+ if (after && afterTargets) { current = after; targets = afterTargets; dialogs = runtime.dialogCount(); }
+ stopReason = timeout
+ ?? (stale
+ ? "stale_ref"
+ : expectation?.status === "failed"
+ ? "expectation_failed"
+ : actionError
+ ? "action_failed"
+ : waitStopReason(waitResult)
+ ?? postBoundary
+ ?? (!after || !isComplete(after) || expectation?.status === "preexisting" || expectation?.status === "unverifiable" ? "control_flow" : undefined));
+ if (stopReason) { stoppedAt = index; break; }
+ }
+
+ let finalExpectation: BrowserActExpectationEvidence | undefined;
+ const terminalNavigation = stopReason === "navigation" && stoppedAt === finalStepIndex;
+ if (action.expect && (!stopReason || terminalNavigation)) {
+ try {
+ const result = await beforeDeadline(
+ () => runtime.wait(action.expect!, baseline, baseline.targetId, action.tab_id, remaining(globalDeadline), action.poll_ms),
+ globalDeadline,
+ );
+ finalExpectation = expectationEvidence(result);
+ timedOut ||= result.status === "timed_out";
+ if (result.status === "timed_out") { stopReason = "global_timeout"; stoppedAt = finalStepIndex; }
+ else if (finalExpectation.status === "failed") { stopReason = "expectation_failed"; stoppedAt = finalStepIndex; }
+ else if (finalExpectation.status === "unverifiable") { stopReason = waitStopReason(result) ?? "control_flow"; stoppedAt = finalStepIndex; }
+ } catch (error) {
+ const timeout = timeoutReason(error);
+ finalExpectation = unverifiableEvidence(runtime.evaluate(action.expect, baseline, baseline), message(error));
+ timedOut ||= timeout !== undefined;
+ stopReason = timeout ?? "control_flow"; stoppedAt = finalStepIndex;
+ }
+ }
+
+ let successor: BrowserActResult["successor"] | undefined;
+ let successorError: unknown;
+ if (stopReason === "global_timeout") {
+ successor = { status: "unavailable", error: "browser action plan timed out" };
+ }
+ for (let attempt = 0; attempt < 3 && !successor; attempt += 1) {
+ try {
+ const observed = await beforeDeadline(() => runtime.observe(action.tab_id), globalDeadline);
+ const successorTargets = await beforeDeadline(() => runtime.targetIds(), globalDeadline);
+ const lateBoundary = boundary(current, observed, targets, successorTargets, dialogs, runtime);
+ current = observed; targets = successorTargets; dialogs = runtime.dialogCount();
+ if (lateBoundary) {
+ stopReason ??= lateBoundary;
+ successorError = new Error(`${lateBoundary} changed successor observation`);
+ continue;
+ }
+ if (!isComplete(observed)) throw new Error("successor observation incomplete");
+ if (action.expect && finalExpectation && finalExpectation.status !== "failed") {
+ const evaluation = runtime.evaluate(action.expect, observed, baseline);
+ finalExpectation = evidenceFromEvaluation(
+ finalExpectation.before,
+ evaluation,
+ [...finalExpectation.diagnostics, ...evaluation.details.map((detail) => `successor: ${detail}`)],
+ );
+ if (evaluation.truth !== true) {
+ stopReason = evaluation.truth === false ? "expectation_failed" : "control_flow";
+ stoppedAt = finalStepIndex;
+ } else if (stopReason === "control_flow" && stoppedAt === finalStepIndex) {
+ stopReason = undefined;
+ stoppedAt = undefined;
+ }
+ }
+ const complete: CuaActionBrowserSnapshot = { type: "browser_snapshot", tab_id: action.tab_id, depth: Number.MAX_SAFE_INTEGER };
+ const presentation = runtime.present(observed, { type: "browser_snapshot", tab_id: action.tab_id, ...action.successor });
+ successor = {
+ status: "observed",
+ text: runtime.render(presentation),
+ url: observed.url,
+ title: observed.title,
+ diff: diffObservations(runtime.present(baseline, complete), runtime.present(observed, complete)),
+ };
+ } catch (error) {
+ successorError = error;
+ if (timeoutReason(error)) break;
+ }
+ }
+ if (!successor) successor = { status: "unavailable", error: message(successorError ?? new Error("successor observation unavailable")) };
+
+ return {
+ outcome: planOutcome(action, steps, finalExpectation, timedOut, stopReason),
+ steps,
+ ...(stoppedAt === undefined ? {} : { stopped_at: stoppedAt }),
+ ...(stopReason ? { stop_reason: stopReason } : {}),
+ ...(finalExpectation ? { final_expectation: finalExpectation } : {}),
+ successor,
+ };
+}
+
+/** A step works only when acknowledged input has newly established its postcondition. */
+function stepOutcome(expectation: BrowserActExpectationEvidence | undefined, actionError: unknown, stale: boolean): BrowserActStepResult["outcome"] {
+ if (expectation?.status === "failed" || stale) return "didnt";
+ if (expectation?.status === "newly_verified" && !actionError) return "worked";
+ return "unknown";
+}
+
+/**
+ * A plan works when its final condition was newly established, or—without a final
+ * condition—every step independently worked. Definitive failure wins; all uncertainty,
+ * preexisting evidence, missing expectations, timeouts, and unsafe stops remain unknown.
+ */
+function planOutcome(
+ action: CuaActionBrowserAct,
+ steps: readonly BrowserActStepResult[],
+ finalExpectation: BrowserActExpectationEvidence | undefined,
+ timedOut: boolean,
+ stopReason: BrowserActResult["stop_reason"],
+): BrowserActResult["outcome"] {
+ if (steps.some((step) => step.outcome === "didnt") || finalExpectation?.status === "failed") return "didnt";
+ if (timedOut) return "unknown";
+ const verified = action.expect
+ ? finalExpectation?.status === "newly_verified"
+ : steps.length === action.steps.length && steps.every((step) => step.outcome === "worked");
+ const verifiedNavigation = stopReason === "navigation" && steps.length === action.steps.length
+ && (steps.at(-1)?.outcome === "worked" || finalExpectation?.status === "newly_verified");
+ return verified && (!stopReason || verifiedNavigation) ? "worked" : "unknown";
+}
+
+function stepDeadline(step: CuaBrowserActStep, globalDeadline: ActDeadline): ActDeadline {
+ if (step.timeout_ms === undefined) return globalDeadline;
+ const at = Date.now() + step.timeout_ms;
+ return at < globalDeadline.at ? { at, reason: "step_timeout" } : globalDeadline;
+}
+
+function remaining(deadline: ActDeadline): number {
+ return Math.max(0, deadline.at - Date.now());
+}
+
+function beforeDeadline<T>(operation: () => Promise<T>, deadline: ActDeadline): Promise<T> {
+ const timeoutMs = remaining(deadline);
+ if (timeoutMs <= 0) return Promise.reject(new BrowserActDeadlineError(deadline.reason));
+ return new Promise<T>((resolve, reject) => {
+ const timer = setTimeout(() => reject(new BrowserActDeadlineError(deadline.reason)), timeoutMs);
+ void Promise.resolve()
+ .then(operation)
+ .then(
+ (value) => { clearTimeout(timer); resolve(value); },
+ (error: unknown) => { clearTimeout(timer); reject(error); },
+ );
+ });
+}
+
+function timeoutReason(error: unknown): ActTimeoutReason | undefined {
+ return error instanceof BrowserActDeadlineError ? error.reason : undefined;
+}
+
+function isComplete(observation: BrowserObservation): boolean {
+ return observation.incompleteFrames.length === 0;
+}
+
+function expectationEvidence(result: BrowserWaitForResult): BrowserActExpectationEvidence {
+ const before = expectationState(result.initial.truth);
+ const diagnostics = result.details;
+ if (result.evidence === "newly_verified" && before !== "matched" && result.final.truth === true) {
+ return { status: "newly_verified", before, after: "matched", diagnostics };
+ }
+ if (result.evidence === "preexisting" && before === "matched" && result.final.truth === true) {
+ return { status: "preexisting", before, after: "matched", diagnostics };
+ }
+ if (result.evidence === "failed" && result.final.truth === false) {
+ return { status: "failed", before, after: "not_matched", diagnostics };
+ }
+ return {
+ status: "unverifiable",
+ before,
+ after: "unknown",
+ diagnostics,
+ ...(result.reason ? { reason: result.reason } : {}),
+ };
+}
+
+function evidenceFromEvaluation(
+ before: BrowserActExpectationEvidence["before"],
+ evaluation: BrowserExpectationEvaluation,
+ diagnostics: string[],
+): BrowserActExpectationEvidence {
+ if (evaluation.truth === true) {
+ return before === "matched"
+ ? { status: "preexisting", before, after: "matched", diagnostics }
+ : { status: "newly_verified", before, after: "matched", diagnostics };
+ }
+ if (evaluation.truth === false) return { status: "failed", before, after: "not_matched", diagnostics };
+ return { status: "unverifiable", before, after: "unknown", diagnostics, ...(evaluation.reason ? { reason: evaluation.reason } : {}) };
+}
+
+function unverifiableEvidence(before: BrowserExpectationEvaluation, diagnostic: string): BrowserActExpectationEvidence {
+ return {
+ status: "unverifiable",
+ before: expectationState(before.truth),
+ after: "unknown",
+ diagnostics: [...before.details, diagnostic],
+ ...(before.reason ? { reason: before.reason } : {}),
+ };
+}
+
+function expectationState(truth: boolean | undefined): BrowserActExpectationEvidence["before"] {
+ return truth === true ? "matched" : truth === false ? "not_matched" : "unknown";
+}
+
+function boundary(before: BrowserObservation, after: BrowserObservation, oldTargets: readonly string[], newTargets: readonly string[], dialogs: number, runtime: BrowserActRuntime): BrowserActResult["stop_reason"] {
+ if (runtime.dialogCount() > dialogs) return "dialog";
+ if (oldTargets.length !== newTargets.length || oldTargets.some((id, index) => id !== newTargets[index])) return "control_flow";
+ if (before.targetId !== after.targetId || before.navigationEpoch !== after.navigationEpoch || runtime.liveNavigationEpoch(after.targetId) !== after.navigationEpoch) return "navigation";
+ if (before.generations.size !== after.generations.size) return "navigation";
+ for (const [frame, generation] of after.generations) {
+ if (runtime.liveGeneration(frame) !== generation || before.generations.get(frame) !== generation) return "navigation";
+ }
+ return undefined;
+}
+
+function waitStopReason(result?: BrowserWaitForResult): BrowserActResult["stop_reason"] {
+ if (!result?.reason) return undefined;
+ if (result.reason === "navigation" || result.reason === "dialog" || result.reason === "stale_ref") return result.reason;
+ return "control_flow";
+}
+
+function stepResult(index: number, step: CuaBrowserActStep, outcome: BrowserActStepResult["outcome"], diagnostics: string[], expectation?: BrowserActExpectationEvidence): BrowserActStepResult {
+ return { index, type: step.type, outcome, diagnostics, ...(expectation ? { expectation } : {}) };
+}
+
+function unavailable(error: unknown, stoppedAt: number): BrowserActResult {
+ return { outcome: "unknown", steps: [], stopped_at: stoppedAt, stop_reason: timeoutReason(error) ?? "control_flow", successor: { status: "unavailable", error: message(error) } };
+}
+
+function isStale(error: unknown): boolean {
+ return !!error && /(?:stale.*ref|ref.*stale)/i.test(message(error));
+}
+
+function message(error: unknown): string {
+ return error instanceof Error ? error.message : String(error);
+}
diff --git a/packages/agent/src/translator/browser-observation.ts b/packages/agent/src/translator/browser-observation.ts
--- a/packages/agent/src/translator/browser-observation.ts
+++ b/packages/agent/src/translator/browser-observation.ts
@@ -1,3 +1,5 @@
+import type { BrowserObservationDiff, BrowserObservationDiffEntry } from "./types";
+
/** Minimal CDP accessibility node used by browser observations. */
export interface AXNode {
readonly nodeId: string;
@@ -78,6 +80,31 @@
readonly shape: string;
}
+/** Compare complete presentations while normalizing away snapshot-scoped refs. */
+export function diffObservations(before: BrowserPresentation, after: BrowserPresentation): BrowserObservationDiff {
+ const beforeCounts = lineCounts(before);
+ const afterCounts = lineCounts(after);
+ const difference = (source: ReadonlyMap<string, number>, other: ReadonlyMap<string, number>): BrowserObservationDiffEntry[] =>
+ [...source].flatMap(([line, count]) => {
+ const delta = count - (other.get(line) ?? 0);
+ return delta > 0 ? [{ line, count: delta }] : [];
+ });
+ const added = difference(afterCounts, beforeCounts);
+ const removed = difference(beforeCounts, afterCounts);
+ const url = before.observation.url === after.observation.url ? undefined : { before: before.observation.url, after: after.observation.url };
+ const title = before.observation.title === after.observation.title ? undefined : { before: before.observation.title, after: after.observation.title };
+ return { changed: added.length > 0 || removed.length > 0 || !!url || !!title, added, removed, ...(url ? { url } : {}), ...(title ? { title } : {}) };
+}
+
+function lineCounts(presentation: BrowserPresentation): ReadonlyMap<string, number> {
+ const counts = new Map<string, number>();
+ for (const { text } of presentation.lines) {
+ const line = text.replace(/\u0000/g, "ref");
+ counts.set(line, (counts.get(line) ?? 0) + 1);
+ }
+ return counts;
+}
+
/** Signals that browser state changed while an observation was collected. */
export class ObservationChangedError extends Error {
constructor(message = "Browser observation changed during collection") {
diff --git a/packages/agent/src/translator/browser-ref-lifecycle.ts b/packages/agent/src/translator/browser-ref-lifecycle.ts
--- a/packages/agent/src/translator/browser-ref-lifecycle.ts
+++ b/packages/agent/src/translator/browser-ref-lifecycle.ts
@@ -193,6 +193,11 @@
this.deleteUnusedFrame(frame);
}
+ /** Current generation for an observation frame key, defaulting to zero before registration. */
+ currentGeneration(frameKey: string): number {
+ return this.frames.get(frameKey)?.generation ?? this.targets.get(frameKey)?.generation ?? 0;
+ }
+
isRefCurrent(entry: RefEntry): boolean {
if (entry.frameId === entry.targetId) return this.targets.get(entry.targetId)?.generation === entry.generation;
const frame = this.frames.get(entry.frameId);
diff --git a/packages/agent/src/translator/browser-wait.ts b/packages/agent/src/translator/browser-wait.ts
--- a/packages/agent/src/translator/browser-wait.ts
+++ b/packages/agent/src/translator/browser-wait.ts
@@ -17,8 +17,9 @@
observeTarget(targetId: string): Promise<BrowserObservation>;
dialogCount(): number;
targetExists(targetId: string): Promise<boolean>;
- /** Reserved for runtimes that expose generation checks outside observations. */
+ /** Optional lifecycle counters for runtimes that expose out-of-band stability checks. */
liveGeneration?(frameId: string): number;
+ liveNavigationEpoch?(targetId: string): number;
resolveRef: BrowserRefResolver;
now?(): number;
delay?(ms: number): Promise<void>;
@@ -30,6 +31,9 @@
timeoutMs?: number;
pollMs?: number;
tabId?: string;
+ /** Supply the pre-action state when verifying an action postcondition. */
+ baseline?: BrowserObservation;
+ targetId?: string;
}
function expectationNodes(observation: BrowserObservation): AXNode[] {
@@ -98,9 +102,9 @@
let targetId: string;
let baseline: BrowserObservation;
try {
- targetId = await beforeDeadline(() => runtime.selectTarget(options.tabId), remaining(), now);
+ targetId = options.targetId ?? await beforeDeadline(() => runtime.selectTarget(options.tabId), remaining(), now);
if (expired()) return timedOut(started, now());
- baseline = await runtime.observeTarget(targetId);
+ baseline = options.baseline ?? await runtime.observeTarget(targetId);
} catch (error) {
if (error instanceof WaitDeadlineError) return timedOut(started, now());
return failedObservation(started, now(), error);
@@ -109,12 +113,16 @@
if (initial.reason === "stale_ref" && !("any" in options.expect)) {
return terminal("interrupted", "unverifiable", initial, initial, started, now(), initial.reason);
}
+ if (options.baseline && baseline.incompleteFrames.length > 0) {
+ return terminal("unverifiable", "unverifiable", initial, initial, started, now(), "incomplete_observation");
+ }
if (expired()) return timedOut(started, now(), initial, initial);
- if (initial.truth === true) return terminal("satisfied", "preexisting", initial, initial, started, now());
+ if (initial.truth === true && !options.baseline) return terminal("satisfied", "preexisting", initial, initial, started, now());
const dialogs = runtime.dialogCount();
let final = initial;
while (!expired()) {
- await sleep(Math.min(poll, remaining()));
+ try { await beforeDeadline(() => sleep(Math.min(poll, remaining())), remaining(), now); }
+ catch (error) { if (error instanceof WaitDeadlineError) break; throw error; }
if (expired()) break;
let exists: boolean;
try { exists = await beforeDeadline(() => runtime.targetExists(targetId), remaining(), now); }
@@ -137,11 +145,11 @@
return terminal("interrupted", "unverifiable", initial, final, started, now(), final.reason);
}
if ((crossDocumentNavigation || sameDocumentNavigation) && locationExpectation) {
- if (final.truth === true && !expired()) return terminal("satisfied", "newly_verified", initial, final, started, now());
+ if (final.truth === true && !expired()) return terminal("satisfied", satisfiedEvidence(initial), initial, final, started, now());
continue;
}
if (expired()) break;
- if (final.truth === true) return terminal("satisfied", "newly_verified", initial, final, started, now());
+ if (final.truth === true) return terminal("satisfied", satisfiedEvidence(initial), initial, final, started, now());
if (observation.incompleteFrames.length > 0) {
final = { ...final, truth: undefined, reason: final.reason ?? "incomplete_observation" };
continue;
@@ -166,6 +174,10 @@
}
}
+function satisfiedEvidence(initial: BrowserExpectationEvidence): BrowserWaitForResult["evidence"] {
+ return initial.truth === true ? "preexisting" : "newly_verified";
+}
+
function containsLocationExpectation(expectation: CuaBrowserExpectation): boolean {
if ("all" in expectation) return expectation.all.some(containsLocationExpectation);
if ("any" in expectation) return expectation.any.some(containsLocationExpectation);
diff --git a/packages/agent/src/translator/browser.ts b/packages/agent/src/translator/browser.ts
--- a/packages/agent/src/translator/browser.ts
+++ b/packages/agent/src/translator/browser.ts
@@ -1,5 +1,6 @@
import {
normalizeGotoUrl,
+ type CuaActionBrowserAct,
type CuaActionBrowserClick,
type CuaActionBrowserDrag,
type CuaActionBrowserFill,
@@ -12,6 +13,7 @@
type CuaActionBrowserSnapshot,
type CuaActionBrowserWaitFor,
type CuaBrowserAction,
+ type CuaBrowserActStep,
type CuaBrowserExpectation,
} from "@onkernel/cua-ai";
import { CdpConnection, type CdpEventMessage } from "./cdp";
@@ -40,6 +42,7 @@
type ObservationLine,
type RenderContext,
} from "./browser-observation";
+import { runBrowserAct } from "./browser-act";
import {
REF_STATE_VERSION,
RefGenerationLifecycle,
@@ -48,8 +51,8 @@
type GenerationCapture,
type RefEntry,
} from "./browser-ref-lifecycle";
-import { waitForBrowserExpectation, type BrowserExpectationEvaluation } from "./browser-wait";
-import type { BatchReadResult, BrowserWaitForResult } from "./types";
+import { evaluateBrowserExpectation, waitForBrowserExpectation, type BrowserExpectationEvaluation } from "./browser-wait";
+import type { BatchReadResult, BrowserActResult, BrowserWaitForResult } from "./types";
const SNAPSHOT_CHAR_LIMIT = 50_000;
const DEFAULT_SNAPSHOT_DEPTH = 15;
@@ -74,6 +77,15 @@
/**
* Executes browser-plane canonical actions over CDP.
*
+ * Ownership boundary: this class owns connection-coupled mutable state and primitive
+ * browser mechanics—CDP sessions/targets, ref resolution, document generations,
+ * observation collection/rendering, and individual input operations. Multi-operation
+ * policy belongs outside this class behind narrow runtime interfaces: semantic polling
+ * lives in `browser-wait.ts`, and dependent plan control flow lives in `browser-act.ts`.
+ * A future feature should be added here only when it must directly coordinate the live
+ * CDP/ref state; orchestration expressible as observe/evaluate/execute primitives should
+ * be a separate module adapted by this executor.
+ *
* Element refs are snapshot-scoped: each snapshot/find mints `e<N>` ids
* mapped to CDP backend node ids for the target's current generation. Any
* main-frame navigation — our own navigate() or a page-initiated one seen
@@ -282,6 +294,8 @@
switch (action.type) {
case "browser_snapshot":
return [{ type: "browser_text", label: "snapshot", text: await this.snapshot(action) }];
+ case "browser_act":
+ return [{ type: "browser_act", result: await this.act(action) }];
case "browser_wait_for":
return [{ type: "browser_wait_for", result: await this.waitFor(action) }];
case "browser_text":
@@ -355,18 +369,50 @@
}
private waitFor(action: CuaActionBrowserWaitFor): Promise<BrowserWaitForResult> {
- return waitForBrowserExpectation(
- {
- selectTarget: (tabId) => this.resolveTarget(tabId),
- observeTarget: (targetId) => this.observe(targetId),
- dialogCount: () => this.dialogNotes.length,
- targetExists: async (targetId) => (await this.cdp.pageTargets()).some((target) => target.targetId === targetId),
- resolveRef: (expectation, observation) => this.evaluateRefExpectation(expectation, observation),
- },
- { expect: action.expect, timeoutMs: action.timeout_ms, pollMs: action.poll_ms, tabId: action.tab_id },
- );
+ return this.waitForExpectation(action.expect, { timeoutMs: action.timeout_ms, pollMs: action.poll_ms, tabId: action.tab_id });
}
+ private waitForExpectation(expect: CuaBrowserExpectation, options: { timeoutMs?: number; pollMs?: number; tabId?: string; baseline?: BrowserObservation; targetId?: string }): Promise<BrowserWaitForResult> {
+ return waitForBrowserExpectation({
+ selectTarget: (tabId) => this.resolveTarget(tabId),
+ observeTarget: (targetId) => this.observe(targetId, false),
+ dialogCount: () => this.dialogNotes.length,
+ targetExists: async (targetId) => (await this.cdp.pageTargets()).some((target) => target.targetId === targetId),
+ resolveRef: (expectation, observation) => this.evaluateRefExpectation(expectation, observation),
+ }, { expect, ...options });
+ }
+
+ private act(action: CuaActionBrowserAct): Promise<BrowserActResult> {
+ return runBrowserAct(action, {
+ observe: (tabId) => this.observe(tabId, false),
+ targetIds: async () => (await this.cdp.pageTargets()).map((target) => target.targetId).sort(),
+ dialogCount: () => this.dialogNotes.length,
+ liveGeneration: (frameId) => this.lifecycle.currentGeneration(frameId),
+ liveNavigationEpoch: (targetId) => this.navigationEpochs.get(targetId) ?? 0,
+ executeStep: (step, tabId) => this.executeActStep(step, tabId),
+ wait: (condition, baseline, targetId, tabId, timeoutMs, pollMs) => this.waitForExpectation(condition, { baseline, targetId, tabId, timeoutMs, pollMs }),
+ evaluate: (condition, observation, baseline) => evaluateBrowserExpectation(condition, observation, baseline, (ref, state) => this.evaluateRefExpectation(ref, state)),
+ present: (observation, snapshot) => this.presentObservation(observation, snapshot),
+ render: (presentation) => this.renderObservation(presentation, false),
+ });
+ }
+
+ private async executeActStep(step: CuaBrowserActStep, tabId?: string): Promise<void> {
+ switch (step.type) {
+ case "click": return this.click({ type: "browser_click", ref: step.ref, button: step.button, num_clicks: step.num_clicks, modifiers: step.modifiers, tab_id: tabId });
+ case "hover": return this.hover({ type: "browser_hover", ref: step.ref, tab_id: tabId });
+ case "fill": return this.fill({ type: "browser_fill", ref: step.ref, value: step.value, tab_id: tabId });
+ case "scroll_to": return this.scrollTo({ type: "browser_scroll_to", ref: step.ref, tab_id: tabId });
+ case "key": return this.key({ type: "browser_key", text: step.text, repeat: step.repeat, tab_id: tabId });
+ case "type": {
+ const session = await this.session(tabId);
+ await this.cdp.send("Input.insertText", { text: step.text }, session);
+ return;
+ }
+ case "wait": await new Promise((resolve) => setTimeout(resolve, step.ms ?? 0)); return;
+ }
+ }
+
private evaluateRefExpectation(
expectation: Extract<CuaBrowserExpectation, { type: "ref" }>,
observation: BrowserObservation,
@@ -547,11 +593,11 @@
};
}
- private renderObservation(presentation: BrowserPresentation): string {
+ private renderObservation(presentation: BrowserPresentation, comparePrevious = true): string {
const { observation, cacheKey, lines, shape } = presentation;
const cached = this.lastSnapshots.get(observation.targetId);
this.lastSnapshots.set(observation.targetId, { key: cacheKey, shape });
- if (cached?.key === cacheKey && cached.shape === shape) return UNCHANGED_SNAPSHOT;
+ if (comparePrevious && cached?.key === cacheKey && cached.shape === shape) return UNCHANGED_SNAPSHOT;
let text = "";
for (const line of lines) {
if (text.length > SNAPSHOT_CHAR_LIMIT) break;
@@ -737,6 +783,7 @@
const modifiers = modifierBits(action.modifiers);
const button = action.button ?? "left";
const clicks = action.num_clicks ?? 1;
+ if (!Number.isInteger(clicks) || clicks < 1 || clicks > 3) throw new Error("num_clicks must be an integer between 1 and 3");
await this.cdp.send("Input.dispatchMouseEvent", { type: "mouseMoved", x: point.x, y: point.y, modifiers }, point.session);
// Native multi-clicks are separate press/release cycles with an
// incrementing clickCount; a single pair with the final count is not how
diff --git a/packages/agent/src/translator/translator.ts b/packages/agent/src/translator/translator.ts
--- a/packages/agent/src/translator/translator.ts
+++ b/packages/agent/src/translator/translator.ts
@@ -137,8 +137,12 @@
await flush();
const reads = await this.browser().execute(action);
result.readResults.push(...reads);
- const failedWait = reads.some((read) => read.type === "browser_wait_for" && read.result.status !== "satisfied");
- if (failedWait) {
+ const stopped = reads.some((read) =>
+ read.type === "browser_wait_for"
+ ? read.result.status !== "satisfied"
+ : read.type === "browser_act" && read.result.stop_reason !== undefined && read.result.outcome !== "worked",
+ );
+ if (stopped) {
const skippedActions = actions.length - index - 1;
... diff truncated: showing 800 of 1911 linesYou can send follow-ups to the cloud agent here.
rgarcia
force-pushed
the
hypeship/browser-action-plans
branch
2 times, most recently
from
July 27, 2026 15:37
1e3a2ee to
c5aba5e
Compare
Contributor
Author
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit c5aba5e. Configure here.
rgarcia
force-pushed
the
hypeship/browser-action-plans
branch
2 times, most recently
from
July 27, 2026 19:28
1852e18 to
4e1d290
Compare
rgarcia
marked this pull request as ready for review
July 27, 2026 19:28
Contributor
Author
|
bugbot run |
rgarcia
force-pushed
the
hypeship/browser-action-plans
branch
from
July 27, 2026 19:30
4e1d290 to
22d9d77
Compare
Contributor
Author
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 22d9d77. Configure here.
rgarcia
force-pushed
the
hypeship/browser-action-plans
branch
from
July 27, 2026 19:41
22d9d77 to
ed09120
Compare
Contributor
Author
|
bugbot run |
rgarcia
force-pushed
the
hypeship/browser-action-plans
branch
from
July 27, 2026 19:46
ed09120 to
8a1a41b
Compare
Contributor
Author
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 8a1a41b. Configure here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
summary
actin browser mode (browser_actin hybrid mode) for dependent browser action plans of up to 20 stepsworked,didnt, orunknownoutcomes with documented stop reasons and typed before/after expectation evidencebehavior
browser_actis for dependent sequences where later input is only safe if earlier input produced the expected state. A step's optionaltimeout_mscovers performing that step and verifying its expectation. The parenttimeout_mscovers the complete sequence plus its final expectation; step deadlines are capped by the remaining global deadline. Verification reuses the semantic wait engine added in #64.The action plan does not claim success merely because input was dispatched. A step is
workedonly when its postcondition is newly verified; preexisting conditions and uncertain delivery remainunknown. Expectation evidence is a discriminated union of validbefore/afterstates, while human-readable diagnostics are explicitly non-control-flow data.BrowserExecutorremains the adapter for connection-coupled CDP/ref mechanics. Sequencing, deadline, attribution, and stop policy live inbrowser-act.tsbehind the narrowBrowserActRuntimeinterface; this ownership boundary is documented on both abstractions.base
main(includes #63 observation/ref lifecycle hardening and #64 semantic waits)size
1,394 changed lines: 1,348 additions, 46 deletions.
testing
npm run typechecknpm test --workspace @onkernel/cua-ai— 182 passednpm test --workspace @onkernel/cua-agent— 325 passed, 19 skipped live testsnpm test --workspace @onkernel/cua-cli— 94 passedgit diff --check origin/main...HEADNote
Medium Risk
Large new browser automation path with nuanced timeout, navigation, and expectation semantics; behavior changes for mixed batches and semantic wait evidence when baselines are supplied.
Overview
Adds
browser_act(exposed asactin browser mode) so models can run dependent multi-step browser plans with optional semantic expectations, shared deadlines, and honestworked/didnt/unknownoutcomes.Execution is split into
browser-act.tsorchestration (sequencing, timeouts, stop reasons, successor collection) andBrowserExecutorCDP mechanics. Plans stop on navigation, dialogs, stale refs, failed expectations, or incomplete observations; mixed canonical batches now skip trailing actions when a plan hits a control-flow boundary. Model-facing feedback includes a successor AX snapshot, a ref-normalized page diff, and capped formatting. Semantic waits gain baseline/target options so post-action verification can distinguish preexisting vs newly verified conditions.Also adds a draft
docs/agent-tool-configuration-spec.mddescribing a future explicittools[]API (not implemented here).browser_clicknum_clicksis tightened to integers 1–3 in schemas and at dispatch.Reviewed by Cursor Bugbot for commit 8a1a41b. Bugbot is set up for automated code reviews on this repo. Configure here.