Skip to content

docs: add RKLLM Rockchip NPU integration - #11442

Open
localai-org-maint-bot wants to merge 1 commit into
masterfrom
bot/issue-3192-rkllm
Open

docs: add RKLLM Rockchip NPU integration#11442
localai-org-maint-bot wants to merge 1 commit into
masterfrom
bot/issue-3192-rkllm

Conversation

@localai-org-maint-bot

Copy link
Copy Markdown
Collaborator

Description

This PR addresses #3192 by documenting the supported path for serving RKLLM models through LocalAI. It connects LocalAI's cloud-proxy backend to the OpenAI-compatible server shipped by upstream RKLLM, without redistributing Rockchip runtime binaries.

The guide covers:

  • deploying the upstream RKLLM Flask server and a converted .rkllm model
  • a complete LocalAI passthrough model configuration
  • current upstream targets: RK3588, RK3576, RV1126B, and RK3562
  • the current RK3566/Quartz64 limitation
  • streaming, concurrency, API-surface, and tool-call limitations

It also adds RKLLM to the backend overview and compatibility table.

Notes for Reviewers

This is intentionally an integration through the existing wire-faithful proxy. Upstream already owns the SoC-specific runtime and OpenAI-compatible server; LocalAI adds its API surface, auth, routing, usage tracking, and UI in front of it.

Verification:

  • git diff --check FETCH_HEAD...HEAD
  • extracted YAML example parsed strictly and asserted for cloud-proxy/RKLLM fields
  • go test ./backend/go/cloud-proxy
  • independent review: no remaining findings

The full core/config suite reached 499/500 specs; its existing remote-fixture test was blocked by this environment's HTTP proxy fetching raw.githubusercontent.com.

Signed commits

  • Yes, I signed my commits.
  • Documentation updated (docs/content/) for user-facing changes, or not applicable

Document how to connect LocalAI to the upstream RKLLM OpenAI-compatible server through cloud-proxy, including supported SoCs and current limitations.

Assisted-by: Codex:gpt-5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant