feat: add multimodal embed() activity and provider adapters - #926
Open
jherr wants to merge 17 commits into
Open
Conversation
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds the shared ChangesEmbedding platform
Estimated code review effort: 5 (Critical) | ~90 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://cdn.github.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 9
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.agentsroom/agents.json:
- Around line 7-9: Remove the tracked generated files under .agentsroom,
including agents.json and prompts.json, from version control while leaving the
existing ignore rule intact so future state files remain untracked.
In `@docs/config.json`:
- Around line 309-313: Add an updatedAt field with value "2026-07-10" to both
new documentation entries identified by the "Embeddings" and "Cohere" labels in
docs/config.json, alongside their existing addedAt fields.
In `@docs/embeddings.md`:
- Around line 113-116: Add the missing trailing comma after the text object in
the multimodal example’s content array, before the following image object, so
the array is valid TypeScript.
In `@packages/ai-cohere/src/adapters/embedding.ts`:
- Around line 140-151: Add configurable AbortController-based timeouts with a
sensible default to both the Cohere API fetch in the embedding request method
and the image URL fetch. Create and wire an abort signal for each request,
ensure the timer is cleared after completion, and preserve any existing
caller-provided cancellation behavior if supported by the adapter configuration.
- Around line 228-245: Validate image URLs before fetching when allowUrlFetch is
enabled: parse the value in the URL-fetch branch and reject loopback, private,
link-local, and other non-public IP addresses (including equivalent IPv6 ranges)
before calling fetch in the image URL handling logic. Preserve the existing
error behavior for disallowed fetching and fetch failures, and ensure malformed
or unsupported URLs are rejected clearly.
In `@packages/ai-gemini/src/adapters/embedding.ts`:
- Around line 48-51: Update the Gemini embedding adapter constructor to call
super(model, {}) instead of passing config, so BaseEmbeddingAdapter does not
retain the Gemini apiKey in this.config while createGeminiClient(config) still
receives the credentials.
In `@packages/ai/src/activities/embed/adapter.ts`:
- Around line 108-110: Resolve the generateId signature mismatch by either
updating generateId to accept and use a prefix parameter consistent with
Cohere’s generateId(this.name) call and embed/index.ts createId(prefix), or
changing the downstream call to match the no-argument API. Also replace
substring(7) with the consistent slice(2, 9) random suffix logic.
In `@packages/ai/src/activities/embed/index.ts`:
- Around line 216-226: Add the optional modelOptions field to the event payload
interfaces in the ai-event-client index, specifically
EmbeddingRequestStartedEvent, EmbeddingRequestCompletedEvent, and
EmbeddingRequestErrorEvent, using the appropriate existing model-options type.
Keep the modelOptions fields in all corresponding emissions in the embedding
activity aligned with these interface definitions.
In `@testing/e2e/src/routes/api.embedding.ts`:
- Around line 47-68: Change the factories declaration to a type that allows
missing provider keys, such as an object index signature returning (() => any) |
undefined, while retaining the optional call on factories[provider] in the
embedding factory lookup. This preserves runtime safety for unsupported
providers and resolves the unnecessary optional-chain lint error.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: e692e98a-9142-4e54-b846-abe1a11d5aa4
⛔ Files ignored due to path filters (1)
pnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (71)
.agentsroom/.gitignore.agentsroom/agents.json.agentsroom/prompts.json.changeset/embed-activity-multimodal.mddocs/adapters/bedrock.mddocs/adapters/cohere.mddocs/adapters/gemini.mddocs/adapters/mistral.mddocs/adapters/ollama.mddocs/adapters/openai.mddocs/config.jsondocs/embeddings.mddocs/migration/migration-from-vercel-ai.mddocs/migration/migration.mdpackages/ai-bedrock/src/adapters/embedding.tspackages/ai-bedrock/src/embedding/embedding-provider-options.tspackages/ai-bedrock/src/index.tspackages/ai-bedrock/src/model-meta.tspackages/ai-bedrock/tests/embedding-adapter.test.tspackages/ai-cohere/README.mdpackages/ai-cohere/package.jsonpackages/ai-cohere/src/adapters/embedding.tspackages/ai-cohere/src/embedding/embedding-provider-options.tspackages/ai-cohere/src/index.tspackages/ai-cohere/src/model-meta.tspackages/ai-cohere/src/utils/client.tspackages/ai-cohere/tests/embedding-adapter.test.tspackages/ai-cohere/tsconfig.jsonpackages/ai-cohere/vite.config.tspackages/ai-event-client/src/index.tspackages/ai-gemini/src/adapters/embedding.tspackages/ai-gemini/src/embedding/embedding-provider-options.tspackages/ai-gemini/src/index.tspackages/ai-gemini/src/model-meta.tspackages/ai-gemini/tests/embedding-adapter.test.tspackages/ai-mistral/package.jsonpackages/ai-mistral/src/adapters/embedding.tspackages/ai-mistral/src/embedding/embedding-provider-options.tspackages/ai-mistral/src/index.tspackages/ai-mistral/src/model-meta.tspackages/ai-mistral/tests/embedding-adapter.test.tspackages/ai-ollama/src/adapters/embedding.tspackages/ai-ollama/src/embedding/embedding-provider-options.tspackages/ai-ollama/src/index.tspackages/ai-ollama/src/model-meta.tspackages/ai-ollama/tests/embedding-adapter.test.tspackages/ai-openai/src/adapters/embedding.tspackages/ai-openai/src/embedding/embedding-provider-options.tspackages/ai-openai/src/index.tspackages/ai-openai/src/model-meta.tspackages/ai-openai/tests/embedding-adapter.test.tspackages/ai/src/activities/embed/adapter.tspackages/ai/src/activities/embed/index.tspackages/ai/src/activities/index.tspackages/ai/src/activities/middleware/types.tspackages/ai/src/index.tspackages/ai/src/middlewares/otel.tspackages/ai/src/types.tspackages/ai/src/utilities/embedding-input.tspackages/ai/tests/embed-per-model-type-safety.test.tspackages/ai/tests/embed.test.tstesting/e2e/fixtures/embedding/basic.jsontesting/e2e/global-setup.tstesting/e2e/src/components/EmbeddingUI.tsxtesting/e2e/src/lib/feature-support.tstesting/e2e/src/lib/features.tstesting/e2e/src/lib/types.tstesting/e2e/src/routeTree.gen.tstesting/e2e/src/routes/$provider/$feature.tsxtesting/e2e/src/routes/api.embedding.tstesting/e2e/tests/embedding.spec.ts
Comment on lines
+7
to
+9
| "id": "agent-1776361243376-3sekdc", | ||
| "claudeSessionId": "96773a93-be2a-45a9-a732-ceb224d3d0e5" | ||
| } |
Contributor
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Verify whether .agentsroom files are tracked despite the root .gitignore rule
git ls-files .agentsroom/Repository: TanStack/ai
Length of output: 221
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== root .gitignore ==\n'
if [ -f .gitignore ]; then
cat -n .gitignore
else
echo 'No root .gitignore'
fi
printf '\n== .agentsroom/.gitignore ==\n'
if [ -f .agentsroom/.gitignore ]; then
cat -n .agentsroom/.gitignore
else
echo 'No .agentsroom/.gitignore'
fi
printf '\n== .agentsroom/agents.json ==\n'
cat -n .agentsroom/agents.jsonRepository: TanStack/ai
Length of output: 2498
Remove the tracked .agentsroom state — .agentsroom/agents.json:7-9 is still committed even though .agentsroom/ is ignored, and it includes a claudeSessionId. Untrack the generated .agentsroom files (including agents.json and prompts.json) so personal/session data stays out of git.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.agentsroom/agents.json around lines 7 - 9, Remove the tracked generated
files under .agentsroom, including agents.json and prompts.json, from version
control while leaving the existing ignore rule intact so future state files
remain untracked.
AlemTuzlak
reviewed
Jul 14, 2026
Comment on lines
+234
to
+247
| input: { | ||
| type: "content", | ||
| content: [ | ||
| { type: "text", content: "a red guitar" }, | ||
| { | ||
| type: "image", | ||
| source: { | ||
| type: "data", | ||
| value: productPhoto, | ||
| mimeType: "image/png", | ||
| }, | ||
| }, | ||
| ], | ||
| }, |
Contributor
There was a problem hiding this comment.
why is this an array above and then an object with type: "content" here with content being an array of parts? why do we not use the same convention we use across all our other generations
AlemTuzlak
reviewed
Jul 14, 2026
AlemTuzlak
reviewed
Jul 14, 2026
Comment on lines
+223
to
+240
| ## Provider Support Matrix | ||
|
|
||
| | Provider | Models | Modalities | `dimensions` | Usage reported | | ||
| | --- | --- | --- | --- | --- | | ||
| | OpenAI | text-embedding-3-small, text-embedding-3-large | text | ✅ | ✅ | | ||
| | Cohere | embed-v4.0 | text + image | ✅ | ✅ | | ||
| | Gemini | gemini-embedding-001 | text | ✅ | ❌ | | ||
| | Mistral | mistral-embed | text | ❌ | ✅ | | ||
| | Mistral | codestral-embed | text | ✅ | ✅ | | ||
| | Bedrock | amazon.titan-embed-text-v2:0 | text | ✅ (256/512/1024) | ✅ | | ||
| | Bedrock | amazon.titan-embed-image-v1 | text + image | ✅ (256/384/1024) | ✅ | | ||
| | Bedrock | cohere.embed-english-v3, cohere.embed-multilingual-v3 | text | ❌ | ❌ | | ||
| | Ollama | nomic-embed-text, mxbai-embed-large, … | text | ❌ | ✅ | | ||
|
|
||
| Notes: | ||
|
|
||
| - Bedrock Titan models have no batch API — a batch of N items runs as N requests under a small concurrency cap. | ||
| - Gemini's Vertex-only multimodal embedding model (`multimodalembedding@001`) is not supported; `@tanstack/ai-gemini` targets the Gemini API. |
Contributor
There was a problem hiding this comment.
i would remove this complete section, will be outdated in a day probably
Contributor
🚀 Changeset Version Preview19 package(s) bumped directly, 31 bumped as dependents. 🟥 Major bumps
🟨 Minor bumps
🟩 Patch bumps
|
|
View your CI Pipeline Execution ↗ for commit ab41cd7
☁️ Nx Cloud last updated this comment at |
@tanstack/ai
@tanstack/ai-acp
@tanstack/ai-angular
@tanstack/ai-anthropic
@tanstack/ai-bedrock
@tanstack/ai-claude-code
@tanstack/ai-client
@tanstack/ai-code-mode
@tanstack/ai-code-mode-skills
@tanstack/ai-codex
@tanstack/ai-cohere
@tanstack/ai-devtools-core
@tanstack/ai-durable-stream
@tanstack/ai-elevenlabs
@tanstack/ai-event-client
@tanstack/ai-fal
@tanstack/ai-gemini
@tanstack/ai-grok
@tanstack/ai-grok-build
@tanstack/ai-groq
@tanstack/ai-isolate-cloudflare
@tanstack/ai-isolate-node
@tanstack/ai-isolate-quickjs
@tanstack/ai-mcp
@tanstack/ai-memory
@tanstack/ai-mistral
@tanstack/ai-ollama
@tanstack/ai-openai
@tanstack/ai-opencode
@tanstack/ai-openrouter
@tanstack/ai-preact
@tanstack/ai-react
@tanstack/ai-react-ui
@tanstack/ai-sandbox
@tanstack/ai-sandbox-cloudflare
@tanstack/ai-sandbox-daytona
@tanstack/ai-sandbox-docker
@tanstack/ai-sandbox-local-process
@tanstack/ai-sandbox-sprites
@tanstack/ai-sandbox-vercel
@tanstack/ai-solid
@tanstack/ai-solid-ui
@tanstack/ai-svelte
@tanstack/ai-utils
@tanstack/ai-vue
@tanstack/ai-vue-ui
@tanstack/openai-base
@tanstack/preact-ai-devtools
@tanstack/react-ai-devtools
@tanstack/solid-ai-devtools
commit: |
Introduce a typed embed() primitive with per-model input modalities, Matryoshka dimensions, and adapters for OpenAI, Gemini, Mistral, Ollama, Bedrock, and a new @tanstack/ai-cohere package. Co-authored-by: Cursor <cursoragent@cursor.com>
These personal agent config files should not ship in the embed PR. Co-authored-by: Cursor <cursoragent@cursor.com>
Align embedding event types with emitted modelOptions, harden generateId, fix e2e factory typing, and add Cohere fetch timeouts plus SSRF guards. Co-authored-by: Cursor <cursoragent@cursor.com>
# Conflicts: # testing/e2e/src/routeTree.gen.ts
Address review feedback on PR #926: the multimodal `embed()` input used a `{ type: 'content', content: [...] }` wrapper that existed only in embeddings and broke the `Array<ContentPart>` convention used everywhere else (e.g. chat message content). Replace `EmbeddingContentItem` with `EmbeddingContentParts = Array<TextPart | ImagePart>`. A fused multimodal item is now a nested array of parts; the top-level `input` array is uniformly the item list (one vector per element), so you fuse by nesting: `[[textPart, imagePart]]`. The resolver disambiguates on `Array.isArray`, and per-model narrowing still rejects images/fused arrays on text-only models at compile time. Also remove the Provider Support Matrix from docs/embeddings.md (goes stale quickly), and propagate the input-shape change across docs, adapter READMEs, tests, and the changeset. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
docs/config.json (2)
603-605: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winAdd the missing
addedAtto the Mistral entry.The Mistral documentation entry has
updatedAtbut noaddedAt, violating the requirement to maintain both timestamps. Preserve the page’s original publication date.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/config.json` around lines 603 - 605, Add the missing addedAt field to the Mistral entry identified by label "Mistral", preserving the page’s original publication date, while keeping the existing updatedAt value unchanged.Source: Coding guidelines
489-495: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winRefresh stale
updatedAtvalues to July 22, 2026.These entries correspond to documentation changes in this PR but still use
2026-07-10. Per the docs metadata guideline, substantive content changes should refreshupdatedAtto today’s date.Also applies to: 567-570, 579-588, 603-605
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/config.json` around lines 489 - 495, Update every affected documentation entry in docs/config.json, including the “From Vercel AI SDK” entry and the additional referenced ranges, changing stale updatedAt values from 2026-07-10 to 2026-07-22 while leaving labels, links, and addedAt values unchanged.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@docs/config.json`:
- Around line 603-605: Add the missing addedAt field to the Mistral entry
identified by label "Mistral", preserving the page’s original publication date,
while keeping the existing updatedAt value unchanged.
- Around line 489-495: Update every affected documentation entry in
docs/config.json, including the “From Vercel AI SDK” entry and the additional
referenced ranges, changing stale updatedAt values from 2026-07-10 to 2026-07-22
while leaving labels, links, and addedAt values unchanged.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 49682a70-eeef-47b9-8c97-93d9b17a8a60
📒 Files selected for processing (13)
.changeset/embed-activity-multimodal.mddocs/adapters/bedrock.mddocs/adapters/cohere.mddocs/config.jsondocs/embeddings.mdpackages/ai-bedrock/tests/embedding-adapter.test.tspackages/ai-cohere/README.mdpackages/ai-cohere/tests/embedding-adapter.test.tspackages/ai/src/activities/embed/index.tspackages/ai/src/types.tspackages/ai/src/utilities/embedding-input.tspackages/ai/tests/embed-per-model-type-safety.test.tspackages/ai/tests/embed.test.ts
🚧 Files skipped from review as they are similar to previous changes (11)
- docs/adapters/bedrock.md
- packages/ai-cohere/README.md
- packages/ai/tests/embed-per-model-type-safety.test.ts
- packages/ai/src/utilities/embedding-input.ts
- packages/ai/src/activities/embed/index.ts
- docs/adapters/cohere.md
- packages/ai/src/types.ts
- packages/ai-bedrock/tests/embedding-adapter.test.ts
- packages/ai-cohere/tests/embedding-adapter.test.ts
- packages/ai/tests/embed.test.ts
- .changeset/embed-activity-multimodal.md
…on guide Drop the unnecessary `ignore` tag on the embeddings "Before (Vercel AI SDK)" block. The Vercel packages aren't installed, so their imports resolve to `any` (same as every other non-ignored "Before" block), meaning the snippet already passes `kiira check` — no reason to skip it. Addresses PR #926 review feedback. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
# Conflicts: # testing/e2e/src/routeTree.gen.ts
…s metadata
- Pass `{}` to `super()` in GeminiEmbeddingAdapter so BaseEmbeddingAdapter no
longer stores the Gemini apiKey on `this.config` (createGeminiClient still
receives the credentials). Matches every other embedding adapter.
- docs/config.json: add missing `addedAt` to the Mistral entry and refresh
stale `updatedAt` to 2026-07-22 on the docs this PR substantively changed
(migration guides, openai/gemini/ollama/mistral adapters).
Addresses PR #926 review feedback.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
embed()activity with per-model input modalities, Matryoshkadimensions, middleware/OTel/devtools integration, andembeddings: [{ vector, index }]results@tanstack/ai-coherepackage for embed-v4.0Test plan
pnpm test:pr(sherif, knip, docs, eslint, lib, types, build)@tanstack/aiand provider packagespnpm --filter @tanstack/ai-e2e test:e2e -- tests/embedding.spec.ts(4/4 passed)Made with Cursor
Summary by CodeRabbit
New Features
embed()activity for single/batch text and multimodal embeddings, including fused inputs and per-item indexed results.Documentation
Tests