Add streaming tool calling to the MLX provider - #9
Merged
Conversation
ActuallyTaylor
force-pushed
the
feature/mlx-streaming-tool-calling
branch
from
August 3, 2026 19:36
709c428 to
c34c831
Compare
ActuallyTaylor
force-pushed
the
feature/mlx-streaming-tool-calling
branch
2 times, most recently
from
August 3, 2026 19:47
2d0d080 to
cfc98fb
Compare
ActuallyTaylor
changed the base branch from
feature/central-tool-resolution
to
feature/openresponses-streaming-tool-calling
August 3, 2026 19:48
ActuallyTaylor
force-pushed
the
feature/mlx-streaming-tool-calling
branch
from
August 3, 2026 20:03
cfc98fb to
a5a0ea7
Compare
ActuallyTaylor
force-pushed
the
feature/mlx-streaming-tool-calling
branch
from
August 3, 2026 20:07
a5a0ea7 to
043c0ee
Compare
ActuallyTaylor
force-pushed
the
feature/mlx-streaming-tool-calling
branch
2 times, most recently
from
August 3, 2026 20:17
66176ee to
f2fb47c
Compare
ActuallyTaylor
force-pushed
the
feature/mlx-streaming-tool-calling
branch
2 times, most recently
from
August 3, 2026 20:40
9500897 to
8f4d598
Compare
MLXLanguageModel.streamResponse passed `tools: nil` to makeUserInput and its generation loop ignored `.toolCall` items, so tool calls were never requested from the model and never handled. Streaming with tools silently behaved as if the session had no tools, while the non-streaming `respond` path supported them fully. streamResponse now passes the session's tool specs via the existing mlxToolSpecs(for:) helper and wraps generation in a turn loop that mirrors the semantics of AnthropicLanguageModel.streamResponse. Each turn accumulates `.chunk` text and collects `.toolCall` items; when a turn ends with tool calls pending they are executed and the assistant turn plus tool results are appended to the chat before generating again. The loop exits when a turn completes without tool calls. Tool execution reuses the same makeTranscriptToolCalls and resolveToolCalls functions the non-streaming path uses, including its max-iteration ceiling and repeated-tool-call-signature loop detection, so both paths abort identically on a runaway model. A `.stop` resolution appends the tool calls and finishes the stream without executing them. The transcript is updated live: growStreamingTranscript as text arrives, and appendTranscriptEntry for tool activity with `.toolCalls` always appended before the `.toolOutput` entries of the same turn. Accumulated text resets per turn so the trailing transcript response entry tracks only the turn currently being generated, matching the Anthropic provider. Resource management is unchanged in shape: the GPU memory scope and generation slot are still released through the idempotent finishScope/finishGenerationSlot pair guarded by didEndScope/didReleaseGenerationSlot, now covering the loop's normal exit, the `.stop` early return, the thrown-error path, and cancellation. The KV cache is resolved and stored per turn, as in respond.
ActuallyTaylor
changed the base branch from
feature/openresponses-streaming-tool-calling
to
main
August 3, 2026 20:44
ActuallyTaylor
force-pushed
the
feature/mlx-streaming-tool-calling
branch
from
August 3, 2026 20:44
8f4d598 to
befc05d
Compare
ActuallyTaylor
commented
Aug 3, 2026
ActuallyTaylor
left a comment
ActuallyTaylor
left a comment
Collaborator
Author
There was a problem hiding this comment.
Same style as the other prs, all looks good
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
streamResponsepassedtools: niltomakeUserInputand folded.toolCallitems into a no-op, so tools were never requested and never handled.respondalready had a complete loop.Streaming now passes the session's tool specs and wraps generation in a turn loop: it accumulates text and collects tool calls, then stores the KV cache, appends the assistant text to the chat, applies the same max-iteration ceiling and repeated-signature guards
responduses, and runs the sharedresolveToolCalls. Tool results feed back as.toolchat messages before the next turn.Resource handling is preserved on every exit path. The GPU memory scope and generation slot are acquired once outside the loop and released through the idempotent
finishScope()/finishGenerationSlot()pair on all four exits — normalbreak, the.stopearly return (which releases before returning), thecatch, and consumer-sideonTermination.The
type == String.selfrestriction is unchanged.Verified live via
xcodebuildper the README's MLX instructions, againstmlx-community/Qwen3-0.6B-4bit: 19/19 tests pass includingwithToolsandstreamWithTools. (The suite gates itself to Xcode becauseswift testhits a Metal library loading failure that also breaks the pre-existing tests.)Based on #4; merge that first.
🤖 Generated with Claude Code