Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -686,7 +686,7 @@ This benchmark is a big step up from our previous 8k1k/1k1k benchmark, as it tru
<Figure
src="/images/kimi-k3-the-manos-the-mythos-the/inferencex-agentic-trace-distributions.png"
alt="Distribution histograms of the agentic trace dataset: input tokens per turn, output tokens per turn, uncached input tokens per request, and turns per conversation"
caption="Source: [InferenceX](https://inferencex.semianalysis.com/datasets)"
caption="Source: [InferenceX](https://inferencex.semianalysis.com/agentx)"
/>

For Kimi K3, Day 0 bringup was easier than DSv4 due to better documentation and preparation ahead of weights release. Appropriate images and a speculative decoder model were released at the same time as the weights.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ title: 'Ultra-High Interactivity on NVIDIA GPUs? TileRT on InferenceX'
subtitle: 'Can TileRT software on NVIDIA GPUs compete with Cerebras, Groq LPU, and SambaNova? Batch size 1, disaggregated engine, high-throughput prefill engine, high-interactivity decode engine'
date: '2026-08-10'
publishDate: '2026-08-10'
modifiedDate: '2026-08-18'
tags:
- benchmark
- gpu
Expand All @@ -13,6 +14,8 @@ tags:
- tilert
- vllm
- glm5
- agentx
- agentic
---

_Originally published on the [SemiAnalysis newsletter](https://newsletter.semianalysis.com/p/ultra-high-interactivity-on-nvidia) on August 10, 2026._
Expand Down Expand Up @@ -235,9 +238,9 @@ Work is being done to simplify this, especially as software development can be a

AI coding agents accelerate tuning within known templates, but novel transformations still require expert judgment. A monolithic persistent kernel also reduces the usefulness of conventional per-kernel profiler timelines, making automated feedback loops more difficult.

## Next steps with TileRT and InferenceX
## Next steps with TileRT

We are actively working on moving TileRT benchmarking from InferenceX's single-turn 8k/1k to our new agentic coding benchmark, which we call AgentX. This scenario replays real Claude Code and Codex traces with long-context, multi-turn requests, realistic subagent activity, and dynamic tool-use delays. Its median input length is 140k tokens, while the theoretical median cache-hit rate roofline reaches 99.2%.
AgentX, InferenceX's long-context, multi-turn coding scenario, is now live for supported models and serving stacks. It replays privacy-preserving workload shapes derived from real Claude Code and Codex sessions, with long-context, multi-turn requests, realistic subagent activity, and dynamic tool-use delays. Its median input length is 140k tokens, while the theoretical median cache-hit rate roofline reaches 99.2%. Extending TileRT benchmarking from its current single-turn 8k/1k workload to AgentX remains the next step for this serving stack.

<Figure
src="/images/ultra-high-interactivity-on-nvidia/agentx-benchmark-characteristics.png"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -223,7 +223,7 @@ The second nuance in CoreWeave’s performance results is that they are using De

Oddly enough, CoreWeave’s choice of using DeepSeek R1 671B is theoretically more favourable towards the Blackwell baseline, and not Rubin. Rubin’s main advantages lie in a higher HBM capacity, higher CPU DRAM capacity, and greater HBM bandwidth, meaning that Rubin is more optimized for multi-trillion parameter models like Fable 5, Gemini Pro, Kimi K3, and Qwen3.8 2.4T.

The third noteworthy item is that CoreWeave uses only single turn 8k/1k input/output tokens. Theoretically, multi-turn long context workloads like Agentic Coding should do better on Rubin, due to Rubin’s higher HBM capacity and bandwidth, but this would not be captured on a simple single-turn benchmark. [Our upcoming AgentX benchmark scenario created in collaboration with Weka, LMCache, the vLLM/SGLang community, Nvidia, AMD, and many others in the community will provide a realistic agentic workload to benchmark inference performance.](https://inferencex.semianalysis.com/datasets/cc-traces-weka-062126) We encourage everyone to adopt this inference benchmark!
The third noteworthy item is that CoreWeave uses only single turn 8k/1k input/output tokens. Theoretically, multi-turn long context workloads like Agentic Coding should do better on Rubin, due to Rubin’s higher HBM capacity and bandwidth, but this would not be captured on a simple single-turn benchmark. [Our upcoming AgentX benchmark scenario created in collaboration with Weka, LMCache, the vLLM/SGLang community, Nvidia, AMD, and many others in the community will provide a realistic agentic workload to benchmark inference performance.](https://inferencex.semianalysis.com/agentx/cc-traces-weka-062126) We encourage everyone to adopt this inference benchmark!

Finally, we note that CoreWeave’s testing was done on a pre-production rack without a scale-out fabric. Specifically, CoreWeave used a Dell Engineering Sample (ES) rack. We do believe these results are valuable as they use wide EP and PD disagg, which uses the NVL72 scale-up backplane and proves that it is working well. This backplane faced many reliability challenges during the ramp of GB200 NVL72 Oberon, [as we have noted in our Accelerator model.](https://semianalysis.com/accelerator-hbm-model/)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -686,7 +686,7 @@ QB 的做法是:在当前批次的截断阈值和路由结果下,求出一
<Figure
src="/images/kimi-k3-the-manos-the-mythos-the/inferencex-agentic-trace-distributions.png"
alt="agentic 轨迹数据集的分布直方图:每轮输入 token、每轮输出 token、每次请求未命中缓存的输入 token,以及每次会话的轮数"
caption="来源:[InferenceX](https://inferencex.semianalysis.com/datasets)"
caption="来源:[InferenceX](https://inferencex.semianalysis.com/zh/agentx)"
/>

对 Kimi K3 来说,Day 0 的 bringup 比 DSv4 更轻松,因为权重发布前的文档和准备工作更充分。合适的镜像和投机解码模型与权重同时发布。
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ title: '在 NVIDIA GPU 上实现超高交互性?TileRT 登陆 InferenceX'
subtitle: '运行在 NVIDIA GPU 上的 TileRT 软件能否与 Cerebras、Groq LPU、SambaNova 竞争?批大小为 1、分离式引擎、高吞吐量预填充引擎、高交互性解码引擎'
date: '2026-08-10'
publishDate: '2026-08-10'
modifiedDate: '2026-08-18'
tags:
- benchmark
- gpu
Expand All @@ -13,6 +14,8 @@ tags:
- tilert
- vllm
- glm5
- agentx
- agentic
---

_本文最初于 2026 年 8 月 10 日发布在 [SemiAnalysis 通讯](https://newsletter.semianalysis.com/p/ultra-high-interactivity-on-nvidia)。_
Expand Down Expand Up @@ -235,9 +238,9 @@ TileRT 继承了 ASIC 厂商最大的弱点。静态提前编译意味着模型

AI 编码智能体能在已知模板内加速调优,但新颖的变换仍需专家判断。此外,单体式持久化 kernel 也削弱了传统逐 kernel 性能剖析时间线的作用,使自动化反馈闭环更加困难。

## TileRT 与 InferenceX 的下一步
## TileRT 的下一步

我们正在积极推进,将 TileRT 的基准测试从 InferenceX 的单轮 8k/1k 场景扩展到我们新的智能体编码基准——AgentX。该场景回放真实的 Claude Code 与 Codex 轨迹,包含长上下文、多轮请求、真实的子智能体活动与动态工具调用延迟。其输入长度中位数为 140k tokens,理论缓存命中率中位数屋顶线可达 99.2%。
InferenceX 的长上下文多轮编码场景 AgentX 现已在具备对应数据的模型与服务栈上正式上线。它回放从真实 Claude Code 与 Codex 会话衍生出的隐私保护工作负载形态,包含长上下文、多轮请求、真实的 subagent 活动与动态工具调用延迟。其输入长度中位数为 140k tokens,理论缓存命中率中位数屋顶线可达 99.2%。对 TileRT 而言,下一步仍是将目前的单轮 8k/1k 基准测试扩展到 AgentX

<Figure
src="/images/ultra-high-interactivity-on-nvidia/agentx-benchmark-characteristics.png"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -223,7 +223,7 @@ CoreWeave 性能结果的第二个细节是他们使用的是 DeepSeek R1 671B

有趣的是,CoreWeave 选择 DeepSeek R1 671B 在理论上反而更有利于 Blackwell 基线,而非 Rubin。Rubin 的主要优势在于更大的 HBM 容量、更大的 CPU DRAM 容量和更高的 HBM 带宽,这意味着 Rubin 更适合 Fable 5、Gemini Pro、Kimi K3、Qwen3.8 2.4T 这类多万亿参数模型。

第三点值得注意的是,CoreWeave 只使用了单轮 8k/1k 输入/输出 token。理论上,Agentic Coding 这类多轮长上下文工作负载在 Rubin 上应有更好表现(得益于其更大的 HBM 容量和带宽),但简单的单轮基准测试无法体现这一点。[我们即将推出的 AgentX 基准测试场景由 Weka、LMCache、vLLM/SGLang 社区、Nvidia、AMD 及社区众多伙伴合作打造,将提供真实的 agentic 工作负载来评测推理性能。](https://inferencex.semianalysis.com/datasets/cc-traces-weka-062126)我们鼓励所有人采用这一推理基准测试!
第三点值得注意的是,CoreWeave 只使用了单轮 8k/1k 输入/输出 token。理论上,Agentic Coding 这类多轮长上下文工作负载在 Rubin 上应有更好表现(得益于其更大的 HBM 容量和带宽),但简单的单轮基准测试无法体现这一点。[我们即将推出的 AgentX 基准测试场景由 Weka、LMCache、vLLM/SGLang 社区、Nvidia、AMD 及社区众多伙伴合作打造,将提供真实的 agentic 工作负载来评测推理性能。](https://inferencex.semianalysis.com/zh/agentx/cc-traces-weka-062126)我们鼓励所有人采用这一推理基准测试!

最后,我们注意到 CoreWeave 的测试是在一台没有 scale-out 网络的预量产机架上完成的。具体来说,CoreWeave 使用的是 Dell 工程样机(Engineering Sample,ES)机架。我们仍然认为这些结果有价值:它们启用了宽 EP 和 PD 分离,用到了 NVL72 的 scale-up 背板,证明其工作良好。这块背板在 GB200 NVL72 Oberon 爬坡期间曾面临诸多可靠性挑战,[我们在加速器模型中有过记录。](https://semianalysis.com/accelerator-hbm-model/)

Expand Down
4 changes: 2 additions & 2 deletions packages/app/cypress/component/chart-selectors.cy.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@ describe('Chart Selectors', () => {
.should('not.contain.text', 'Agentic');
});

it('explains the agentic workload in a tooltip that links to /datasets', () => {
it('explains the agentic workload in a tooltip that links to /agentx', () => {
cy.mount(<ScenarioSelectorHarness />);
cy.get('[data-testid="scenario-agentic-info"]').trigger('pointermove', {
pointerType: 'mouse',
Expand All @@ -151,7 +151,7 @@ describe('Chart Selectors', () => {
);
cy.get('[data-testid="scenario-agentic-info-link"]')
.should('be.visible')
.and('have.attr', 'href', '/datasets');
.and('have.attr', 'href', '/agentx');
});

it('hides the agentic explainer on fixed-sequence scenarios', () => {
Expand Down
2 changes: 1 addition & 1 deletion packages/app/cypress/component/dataset-list.cy.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ describe('DatasetList', () => {
cy.contains('cc-traces-weka (256k)').should('be.visible');
cy.contains('1,234').should('be.visible'); // conversation_count, localized
cy.contains('82%').should('be.visible'); // cachedPct
cy.get('a[href="/datasets/cc-traces-weka-full"]').should('exist');
cy.get('a[href="/agentx/cc-traces-weka-full"]').should('exist');
});

it('shows the empty state when no datasets are ingested', () => {
Expand Down
6 changes: 3 additions & 3 deletions packages/app/cypress/component/footer.cy.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -58,9 +58,9 @@ describe('Footer', () => {
cy.get('[data-testid="footer-link-supporters"]')
.should('contain.text', 'Supporters')
.and('have.attr', 'href', '/quotes');
cy.get('[data-testid="footer-link-datasets"]')
.should('contain.text', 'Datasets')
.and('have.attr', 'href', '/datasets');
cy.get('[data-testid="footer-link-agentx"]')
.should('contain.text', 'AgentX')
.and('have.attr', 'href', '/agentx');
cy.get('[data-testid="footer-link-articles"]')
.should('contain.text', 'Articles')
.and('have.attr', 'href', '/blog');
Expand Down
56 changes: 42 additions & 14 deletions packages/app/cypress/component/header.cy.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -105,22 +105,27 @@ describe('Header', () => {
cy.get('[data-testid="nav-link-compare"]').should('have.attr', 'href', '/compare');
});

it('shows Datasets as a top-level nav link and highlights dataset child pages', () => {
cy.get('[data-testid="nav-link-datasets"]')
it('shows AgentX as a top-level nav link and highlights AgentX child pages', () => {
cy.get('[data-testid="nav-link-agentx"]')
.should('be.visible')
.and('have.attr', 'href', '/datasets');
.and('have.attr', 'href', '/agentx')
.find('[data-nav-badge="agentx"]')
.should('have.text', 'NEW');

mountHeader('/datasets/claude-code-traces');
cy.get('[data-testid="nav-link-datasets"]').should('have.class', 'text-brand');
mountHeader('/agentx/claude-code-traces');
cy.get('[data-testid="nav-link-agentx"]').should('have.class', 'text-brand');
});

it('keeps Datasets in the Chinese navigation tree', () => {
mountHeader('/zh/datasets');
cy.get('[data-testid="nav-link-datasets"]')
it('keeps AgentX in the Chinese navigation tree', () => {
mountHeader('/zh/agentx');
cy.get('[data-testid="nav-link-agentx"]')
.should('be.visible')
.and('contain.text', '数据集')
.and('have.attr', 'href', '/zh/datasets')
.and('contain.text', 'AgentX')
.and('have.attr', 'href', '/zh/agentx')
.and('have.class', 'text-brand');
cy.get('[data-testid="nav-link-agentx"]')
.find('[data-nav-badge="agentx"]')
.should('have.text', '新');
});

it('keeps footer destinations out of the primary nav', () => {
Expand Down Expand Up @@ -151,7 +156,11 @@ describe('Header', () => {
cy.contains('a', 'Overview').should('be.visible').and('have.attr', 'href', '/overview');
cy.contains('a', 'Dashboard').should('be.visible').and('have.attr', 'href', '/inference');
cy.contains('a', 'Comparisons').should('be.visible').and('have.attr', 'href', '/compare');
cy.contains('a', 'Datasets').should('be.visible').and('have.attr', 'href', '/datasets');
cy.contains('a', 'AgentX')
.should('be.visible')
.and('have.attr', 'href', '/agentx')
.find('[data-nav-badge="agentx"]')
.should('have.text', 'NEW');
cy.contains('a', 'Supporters').should('not.exist');
cy.contains('a', 'Articles').should('not.exist');
});
Expand All @@ -167,8 +176,27 @@ describe('Header', () => {
cy.wrap(mockRouter.push).should('have.been.calledTwice');
});

it('keeps every primary link inside the header at the desktop breakpoint', () => {
cy.viewport(1024, 720);
it('uses the hamburger without horizontal overflow from 1009 through 1024 CSS pixels', () => {
[1009, 1012, 1020, 1024].forEach((width) => {
cy.viewport(width, 720);
cy.get('[data-testid="nav-link-dashboard"]').should('not.be.visible');
cy.get('[data-testid="mobile-menu-toggle"]').should('be.visible');
cy.document().then((doc) => {
expect(doc.documentElement.scrollWidth, `${width}px document scrollWidth`).to.be.at.most(
doc.documentElement.clientWidth,
);
});
cy.get('[data-testid="header"]').then(($header) => {
const header = $header[0];
expect(header.scrollWidth, `${width}px header scrollWidth`).to.be.at.most(
header.clientWidth,
);
});
});
});

it('keeps every primary link inside the header at the xl desktop breakpoint', () => {
cy.viewport(1280, 720);
cy.get('[data-testid="header"]').then(($header) => {
const header = $header[0];
const bounds = header.getBoundingClientRect();
Expand Down Expand Up @@ -237,7 +265,7 @@ describe('Header', () => {
cy.get('[data-testid="mobile-menu-toggle"]').click();
cy.get('[data-testid="mobile-menu"]').should('be.visible');
cy.get('[data-testid="mobile-menu"]').within(() => {
['Home', 'Overview', 'Dashboard', 'Comparisons', 'Datasets', 'About'].forEach((label) => {
['Home', 'Overview', 'Dashboard', 'Comparisons', 'AgentX', 'About'].forEach((label) => {
cy.contains('a', label).should('be.visible');
});
['Supporters', 'Articles'].forEach((label) => {
Expand Down
2 changes: 1 addition & 1 deletion packages/app/cypress/e2e/agentic-point-time-series.cy.ts
Original file line number Diff line number Diff line change
Expand Up @@ -235,7 +235,7 @@ describe('Agentic point request metric time series', () => {

it('restores the request timeline view after browser Back from a dataset route', () => {
cy.window().then((win) => {
win.history.pushState({}, '', '/datasets/test-dataset/conversations/conversation-1');
win.history.pushState({}, '', '/agentx/test-dataset/conversations/conversation-1');
});
cy.go('back');
cy.location('pathname').should('eq', '/inference/agentic/206885');
Expand Down
2 changes: 1 addition & 1 deletion packages/app/cypress/e2e/datasets-distributions.cy.ts
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ describe('Dataset distribution percentiles', () => {
cy.intercept('GET', '/api/v1/datasets/test-dataset/conversations*', {
body: { total: 0, items: [] },
});
cy.visit('/datasets/test-dataset', { onBeforeLoad: unlockAgenticGate });
cy.visit('/agentx/test-dataset', { onBeforeLoad: unlockAgenticGate });
});

it('shows P50/P75/P90/P95 for ISL, OSL, and uncached input', () => {
Expand Down
2 changes: 1 addition & 1 deletion packages/app/cypress/e2e/datasets-flamegraph-time.cy.ts
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ describe('Dataset conversation flamegraph timing', () => {
},
},
});
cy.visit('/datasets/test-dataset/conversations/conversation-1', {
cy.visit('/agentx/test-dataset/conversations/conversation-1', {
onBeforeLoad: unlockAgenticGate,
});
});
Expand Down
Loading