feat: add Vertex AI support for Anthropic and Google providers#535
feat: add Vertex AI support for Anthropic and Google providers#535lucharo wants to merge 6 commits intoplastic-labs:mainfrom
Conversation
Adds fallback client initialization for Google Cloud Vertex AI when no direct API keys are configured: - AsyncAnthropicVertex client when ANTHROPIC_VERTEX_PROJECT_ID is set (reads region from ANTHROPIC_VERTEX_REGION, defaults to "global") - genai.Client(vertexai=True) when GOOGLE_GENAI_USE_VERTEXAI=true (for both generation and embeddings) - Configurable embedding model via LLM_EMBEDDING_MODEL setting This lets self-hosted Honcho instances use Vertex AI service account credentials instead of requiring direct API keys from each provider.
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
WalkthroughAdds an optional Changes
Sequence Diagram(s)sequenceDiagram
participant App as App/Settings
participant Env as Environment
participant Gemini as genai.Client
participant Anthropic as AsyncAnthropic / AsyncAnthropicVertex
App->>Env: read GOOGLE_GENAI_USE_VERTEXAI, GEMINI_API_KEY, ANTHROPIC_VERTEX_*
App->>Gemini: if GEMINI_API_KEY -> genai.Client(api_key=...)
App->>Gemini: else if GOOGLE_GENAI_USE_VERTEXAI=="true" -> genai.Client(vertexai=True)
App->>Gemini: choose embedding model from settings.LLM.EMBEDDING_MODEL or fallback
App->>Anthropic: if ANTHROPIC_VERTEX_* set -> try AsyncAnthropicVertex()
App->>Anthropic: else -> AsyncAnthropic()
Note over Anthropic, Gemini: Anthropic Vertex handled same as Anthropic in streaming/system-message flows
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
The region varies by GCP project and deployment. Hardcoding a default would silently fail for users in other regions.
AsyncAnthropicVertex is not a subclass of AsyncAnthropic, so 'case AsyncAnthropic():' never matches Vertex clients. The function returns None implicitly, causing NoneType crashes throughout the deriver and dream services.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@src/utils/clients.py`:
- Around line 264-268: The fallback path currently re-imports
AsyncAnthropicVertex unguarded, which can raise ImportError and bypass the
earlier guarded import; replace the local "from anthropic import
AsyncAnthropicVertex" with the guarded AsyncAnthropicVertex symbol already
imported at the top of the module, and before instantiating it ensure the symbol
is present/usable (e.g., if AsyncAnthropicVertex is None or not callable raise
an explicit ImportError or configuration error with a clear message), then
instantiate AsyncAnthropicVertex(timeout=600.0) and assign it to
CLIENTS["anthropic"].
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 9afe777f-79ab-4831-ae5d-de5f09465fc6
📒 Files selected for processing (1)
src/utils/clients.py
There was a problem hiding this comment.
♻️ Duplicate comments (1)
src/utils/clients.py (1)
264-268:⚠️ Potential issue | 🟠 MajorRemove the unguarded
AsyncAnthropicVertexre-import in the Vertex fallback.Line 265 bypasses the guarded import at Line 10-13. If Vertex env vars are set but the installed
anthropicSDK does not exposeAsyncAnthropicVertex, this path fails at import time instead of raising a controlled config error.Proposed fix
try: from anthropic import AsyncAnthropicVertex + _HAS_ANTHROPIC_VERTEX = True except ImportError: AsyncAnthropicVertex = type("_Unavailable", (), {}) + _HAS_ANTHROPIC_VERTEX = False @@ elif os.environ.get("ANTHROPIC_VERTEX_PROJECT_ID") and os.environ.get("CLOUD_ML_REGION"): - from anthropic import AsyncAnthropicVertex + if not _HAS_ANTHROPIC_VERTEX: + raise ValueError( + "Anthropic Vertex is enabled, but AsyncAnthropicVertex is unavailable in " + "the installed anthropic SDK." + ) # SDK reads ANTHROPIC_VERTEX_PROJECT_ID and CLOUD_ML_REGION from env anthropic = AsyncAnthropicVertex(timeout=600.0) CLIENTS["anthropic"] = anthropicVerification (read-only): after applying the fix, there should be no nested
from anthropic import AsyncAnthropicVerteximport in the fallback branch.#!/bin/bash set -euo pipefail echo "Guarded import block:" rg -n -C2 'try:|from anthropic import AsyncAnthropicVertex|except ImportError|_HAS_ANTHROPIC_VERTEX|_Unavailable' src/utils/clients.py echo echo "Fallback block (should NOT contain a nested re-import after fix):" rg -n -C3 'elif os\.environ\.get\("ANTHROPIC_VERTEX_PROJECT_ID"\).*CLOUD_ML_REGION|from anthropic import AsyncAnthropicVertex' src/utils/clients.pyAs per coding guidelines, “Use explicit error handling with appropriate exception types”.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/utils/clients.py` around lines 264 - 268, The fallback branch currently re-imports AsyncAnthropicVertex unguarded; instead, remove the nested "from anthropic import AsyncAnthropicVertex" and use the module-level guard (_HAS_ANTHROPIC_VERTEX) and the already-available AsyncAnthropicVertex symbol to instantiate the client (AsyncAnthropicVertex(timeout=600.0)) and assign to CLIENTS["anthropic"]; if _HAS_ANTHROPIC_VERTEX is False, raise a clear config error (e.g., RuntimeError or ImportError) that the installed anthropic SDK doesn't provide AsyncAnthropicVertex and instruct to install/upgrade the SDK or disable the Vertex env vars, referencing AsyncAnthropicVertex, _HAS_ANTHROPIC_VERTEX, CLIENTS["anthropic"], and the ANTHROPIC_VERTEX_PROJECT_ID/CLOUD_ML_REGION env checks.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Duplicate comments:
In `@src/utils/clients.py`:
- Around line 264-268: The fallback branch currently re-imports
AsyncAnthropicVertex unguarded; instead, remove the nested "from anthropic
import AsyncAnthropicVertex" and use the module-level guard
(_HAS_ANTHROPIC_VERTEX) and the already-available AsyncAnthropicVertex symbol to
instantiate the client (AsyncAnthropicVertex(timeout=600.0)) and assign to
CLIENTS["anthropic"]; if _HAS_ANTHROPIC_VERTEX is False, raise a clear config
error (e.g., RuntimeError or ImportError) that the installed anthropic SDK
doesn't provide AsyncAnthropicVertex and instruct to install/upgrade the SDK or
disable the Vertex env vars, referencing AsyncAnthropicVertex,
_HAS_ANTHROPIC_VERTEX, CLIENTS["anthropic"], and the
ANTHROPIC_VERTEX_PROJECT_ID/CLOUD_ML_REGION env checks.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: fef716d1-14c8-49b9-8377-258a6d74ab9b
📒 Files selected for processing (1)
src/utils/clients.py
Remove unguarded re-import that bypassed the try/except at module level. Use _HAS_ANTHROPIC_VERTEX flag to raise a clear error when the SDK doesn't export the class.
|
@lucharo thanks for the contribution! the vertex support is worth keeping but please rebase onto main / merge main and re-apply your changes, as we've heavily refactored the src/config, embedding client, and clients.py. happy to review once that's done |
3 participants
Vertex AI support for self-hosted Honcho -- use GCP service accounts instead of direct API keys.
What changed
src/utils/clients.py-- whenANTHROPIC_VERTEX_PROJECT_IDandANTHROPIC_VERTEX_REGIONare both set, initializes anAsyncAnthropicVertexclient. WhenGOOGLE_GENAI_USE_VERTEXAI=trueis set, initializesgenai.Client(vertexai=True). Both only kick in when no direct API key is configured.src/embedding_client.py-- same Vertex AI fallback for the Gemini embedding client. Also makes the embedding model name configurable viaLLM_EMBEDDING_MODEL.src/config.py-- addsEMBEDDING_MODELfield (optional, defaults togemini-embedding-001).Why
If you're self-hosting on GCP and already have Vertex AI access through a service account, you shouldn't need to go get separate API keys from Anthropic and Google. This wires up the existing Vertex AI auth path.
Tested with
AnthropicVertexgemini-embedding-2-preview) via Vertex AIExisting behavior is unaffected -- the Vertex paths only activate when env vars are set and no API key exists.
Summary by CodeRabbit