Surface tool calls in the SystemLanguageModel transcript - #12
Merged
Conversation
ActuallyTaylor
force-pushed
the
feature/central-tool-resolution
branch
from
August 3, 2026 19:35
3f72c2c to
6925e25
Compare
ActuallyTaylor
force-pushed
the
feature/system-streaming-tool-calling
branch
from
August 3, 2026 19:36
3a89e24 to
a4d3879
Compare
ActuallyTaylor
force-pushed
the
feature/central-tool-resolution
branch
from
August 3, 2026 19:39
6925e25 to
28e5c10
Compare
ActuallyTaylor
force-pushed
the
feature/system-streaming-tool-calling
branch
2 times, most recently
from
August 3, 2026 19:47
58b72cd to
a64cfdb
Compare
ActuallyTaylor
changed the base branch from
feature/central-tool-resolution
to
feature/coreml-streaming-tool-calling
August 3, 2026 19:49
ActuallyTaylor
force-pushed
the
feature/coreml-streaming-tool-calling
branch
from
August 3, 2026 20:03
fef1463 to
92eebd2
Compare
ActuallyTaylor
force-pushed
the
feature/system-streaming-tool-calling
branch
from
August 3, 2026 20:03
a64cfdb to
4973735
Compare
ActuallyTaylor
force-pushed
the
feature/coreml-streaming-tool-calling
branch
from
August 3, 2026 20:07
92eebd2 to
a72e53f
Compare
ActuallyTaylor
force-pushed
the
feature/system-streaming-tool-calling
branch
from
August 3, 2026 20:07
4973735 to
66ea665
Compare
ActuallyTaylor
force-pushed
the
feature/coreml-streaming-tool-calling
branch
from
August 3, 2026 20:09
a72e53f to
78cbcab
Compare
ActuallyTaylor
force-pushed
the
feature/system-streaming-tool-calling
branch
2 times, most recently
from
August 3, 2026 20:17
ce492c4 to
ef5944d
Compare
ActuallyTaylor
force-pushed
the
feature/coreml-streaming-tool-calling
branch
2 times, most recently
from
August 3, 2026 20:18
ad2e937 to
859a64d
Compare
ActuallyTaylor
force-pushed
the
feature/system-streaming-tool-calling
branch
2 times, most recently
from
August 3, 2026 20:40
95f9bde to
d968cf9
Compare
ActuallyTaylor
force-pushed
the
feature/coreml-streaming-tool-calling
branch
from
August 3, 2026 20:40
859a64d to
c61dff5
Compare
ActuallyTaylor
force-pushed
the
feature/system-streaming-tool-calling
branch
from
August 3, 2026 20:44
d968cf9 to
6d79d02
Compare
ActuallyTaylor
force-pushed
the
feature/coreml-streaming-tool-calling
branch
from
August 3, 2026 22:23
360efcb to
97c09d6
Compare
ActuallyTaylor
force-pushed
the
feature/system-streaming-tool-calling
branch
from
August 3, 2026 22:23
6d79d02 to
4e487b4
Compare
SystemLanguageModel.respond always returned an empty transcriptEntries slice, so tool calls and tool outputs that FoundationModels recorded were dropped instead of reaching session.transcript. Every other provider returns the tool-related entries it produced, and LanguageModelSession.respond appends them ahead of the response entry. FoundationModels does expose this: LanguageModelSession.Response carries transcriptEntries, and its Transcript.ToolCall/ToolOutput values hold the ids FoundationModels assigned. Convert those into their AnyLanguageModel equivalents and return them. Only .toolCalls and .toolOutput are converted; instructions, prompts, and responses are owned by LanguageModelSession and mirroring them would duplicate entries. This makes the previously #if false'd assertion in withTools real. Its old expectation was also wrong: it compared ToolOutput.id against "getWeather", but the tool name lives in toolName while id is the call identifier. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…Model streams Callers of streamResponse expect tool calls and their outputs to land in session.transcript while the stream is still running, with .toolCalls ahead of the matching .toolOutput. SystemLanguageModel surfaced neither: it never touched session.transcript during a stream, so a Transcript-driven UI saw nothing until LanguageModelSession appended the final response. FoundationModels gives us no tool channel on the stream itself — ResponseStream.Snapshot carries only content and rawContent. Its LanguageModelSession.transcript is the only place tool activity shows up, and it grows as the turn progresses, so mirror new tool entries out of it into session.transcript. Mirroring in the order FoundationModels recorded them is what gives the .toolCalls before .toolOutput ordering; the ids are the ones FoundationModels assigned rather than anything synthesized here. Response text now also grows the transcript as it streams, mirroring after tool entries so the turn reads .toolCalls, .toolOutput, .response. Mirroring happens on snapshot boundaries instead of the instant a call is recorded, because polling concurrently would share a non-Sendable FoundationModels.LanguageModelSession across tasks. Tool entries therefore appear with the first snapshot after the tool ran, still mid-stream, plus a final sweep before the stream finishes. session.toolExecutionDelegate remains unconsulted, and AnyToolWrapper documents why. FoundationModels owns the tool loop and hands a tool only its arguments — no call id, no session, no view of sibling calls — so didGenerateToolCalls has no batch to report, the callbacks taking Transcript.ToolCall have no real id to pass, and .stop has no expression at all. Honoring it partially would silently execute a tool the caller asked to stop. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ActuallyTaylor
changed the base branch from
feature/coreml-streaming-tool-calling
to
main
August 4, 2026 00:10
ActuallyTaylor
force-pushed
the
feature/system-streaming-tool-calling
branch
from
August 4, 2026 00:10
4e487b4 to
2825b8d
Compare
ActuallyTaylor
commented
Aug 4, 2026
ActuallyTaylor
left a comment
ActuallyTaylor
left a comment
Collaborator
Author
There was a problem hiding this comment.
Looks almost good, just some comments in the tests
Two things a reader shouldn't have to infer. `toolTranscriptEntry` returned `nil` both for entries that aren't tool activity and for entries whose contents fail to convert, and a bare `compactMap` at the call sites made the first case look like an accidental over-filter — as if prompts and responses were being dropped by mistake. They are dropped deliberately: `LanguageModelSession` appends the prompt before calling the model and the response after, so carrying FoundationModels' own copies back would duplicate them. `isToolActivity(_:)` now states that at each call site, leaving `compactMap` to mean only "conversion failed". The tool tests also asserted with bare `#expect`s, so a failure reported a boolean with no indication of what was expected. Each now carries the reason, matching the ones that already did.
Collaborator
Author
|
Both comments addressed in Failure reasons. Every bare The The problem was that Verified on-device: 17/17 pass, including 🤖 Generated with Claude Code |
`toolTranscriptEntry` read as though it converted any transcript entry, which made the `compactMap` at its call sites look like it was discarding prompts and responses by accident. It only ever yields tool activity, so `toolActivityEntry(from:)` says that, and the `nil` result reads as "not tool activity" rather than a dropped conversion. The doc comment keeps the reason those entries are skipped: `LanguageModelSession` appends the prompt before calling the model and the response after, so mirroring FoundationModels' copies would duplicate them.
ActuallyTaylor
commented
Aug 4, 2026
ActuallyTaylor
left a comment
ActuallyTaylor
left a comment
Collaborator
Author
There was a problem hiding this comment.
Looks good, not the same as the other models but it seems that the conversions between the Apple Foundations Model API and this API are tricky
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This provider bridges FoundationModels, which owns the tool loop itself, so this is narrower than the other provider PRs.
responddropped every transcript entry. It returnedtranscriptEntries: []unconditionally in three places, so tool calls FoundationModels had already executed were silently discarded. It now converts FM'sResponse.transcriptEntries. This also made a#if false'd block inwithToolsreal — its old assertion was wrong too, comparingToolOutput.idagainst"getWeather"when that's thetoolName.Streaming now mirrors tool activity into
session.transcriptas the turn progresses, using the ids FM assigned rather than synthesized ones, preserving.toolCallsbefore.toolOutput. The provider previously never touched the transcript mid-stream at all. Mirroring happens on snapshot boundaries rather than the instant FM records a call, because polling concurrently would share a non-SendableFM session across tasks — entries still land mid-stream, plus a final sweep before the stream finishes.Note
toolExecutionDelegateis not consulted, and this is documented rather than faked. FM invokes tools throughTool.call(arguments:), which receives the decoded arguments and nothing else — no call id, no session, no view of sibling calls.didGenerateToolCallsis defined over a batch FM never surfaces; the delegate'sTranscript.ToolCall.idcould never match the entry that later appears; and.stophas no expression at all, since FM offers no way to halt a turn from inside a tool. Partial support would be worse than none — a caller returning.stopwould get its tool executed anyway, silently.Verified on-device with Apple Intelligence available: 17/17 pass including
withToolsandstreamWithTools.Based on #4; merge that first.
🤖 Generated with Claude Code