[harrymove-ctrl]

Summary

Fixes #1037

• Adds MAX_EMBED_INPUT_BYTES (16_384) and an isomorphic truncateUtf8Bytes helper (TextEncoder / TextDecoder, zero Node builtins) in packages/sdk/src/ai/middleware.ts that bounds UTF-8 strings to at most 16,384 bytes without splitting multi-byte codepoints (2-, 3-, or 4-byte UTF-8 sequences) across the boundary.
• Bounds both the memwal.recall(...) query in transformParams and the background memwal.analyze(...) payload in saveInBackground (wrapGenerate and wrapStream) to 16_384 UTF-8 bytes while preserving the full untruncated user turn in the prompt sent to the language model.
• Updates packages/sdk/CHANGELOG.md and docs/sdk/changelog.mdx under 0.1.11.

Verification

• pnpm --filter @mysten-incubation/memwal build
• pnpm --filter @mysten-incubation/memwal test (373/373 passing, including new test/withMemWal-long-turn.test.mjs)
• node scripts/verify-manual-sdk-release.mjs --base origin/dev --head HEAD
• node scripts/check-docs-code-sync.mjs
• node scripts/check-docs-freshness.mjs

[harrymove-ctrl]

@ducnmm Ready for review — bounds withMemWal recall queries and auto-save analyze() inputs to 16,384 UTF-8 bytes without splitting multi-byte UTF-8 codepoints across the boundary (while keeping the full user turn in the LLM prompt), with regression coverage for wrapGenerate, wrapStream, and multi-byte UTF-8 boundaries (#1037).

[HoangDucBach]

Issue counts by severity

bugs: 1
suggestions: 2
nits: 0

The recall fix meets #1037, but the auto-save change goes beyond the ticket and drops facts that /api/analyze accepts today. Fix the analyze() bound first.

Verified: Built the PR head, ran test/withMemWal-long-turn.test.mjs (4/4 pass) and probed truncateUtf8Bytes with 2-, 3- and 4-byte codepoints and leading whitespace (all within 16,384 bytes, no U+FFFD). Not verified: behavior against a live relayer, and the full 373-test suite from the PR description.

[HoangDucBach]

[bug]: saveInBackground truncates the auto-save payload to 16,384 bytes, but /api/analyze accepts up to MAX_ANALYZE_TEXT_BYTES = 64 KiB (services/server/src/routes/analyze.rs L100) and already handles 16-64 KiB input by skipping only the pre-extract embed (L133-142). #1037 is about recall() only (the 400 comes from the embedder). A 20 KiB turn that analyze handled before now loses every fact after byte 16,384, silently.

Fix: remove the truncation here, or bound at 64 KiB with a separate constant, and keep MAX_EMBED_INPUT_BYTES for the recall query. Reasoned from the server code, not run against a relayer. The #1037 comment citing MAX_ANALYZE_TEXT_BYTES as 16,384 is also wrong.

[HoangDucBach]

[suggestion]: truncateUtf8Bytes keeps only the head of the turn. For "paste 20 KB, then ask a question" the question is cut and the recall query (L201) is the start of the paste (probed: question kept = false for a 20,000-byte paste followed by a short question).

Keep the tail, or head + tail, for the recall query. Reasoned from the probe, not run with a live recall.

[HoangDucBach]

[suggestion]: This entry is under 0.1.11, which is already published on npm (2026-10-28), so the changelog describes a release that does not contain this fix. The same applies to docs/sdk/changelog.mdx L47. Move it to the next version section if the release verifier allows it; I did not check how verify-manual-sdk-release.mjs treats the version header.

[harrymove-ctrl]

@HoangDucBach @ducnmm Updated to address review feedback (commit ff4b135):

1. Auto-save analyze limit: Defined MAX_ANALYZE_INPUT_BYTES = 65_536 (64 KiB) matching the server MAX_ANALYZE_TEXT_BYTES ceiling. Auto-save in saveInBackground now bounds at 64 KiB instead of 16 KiB, so turns between 16 KiB and 64 KiB are handed to analyze() untruncated with zero fact loss.
2. Recall query head-tail preservation: Added strategy: "head-tail" to truncateUtf8Bytes and applied it to recall queries. When a long turn contains a large paste followed by questions or instructions at the end, both the start context and the trailing question are preserved within the 16 KiB embedder limit.
3. Changelog and version alignment: Moved the changelog entries to a new 0.1.12 section in packages/sdk/CHANGELOG.md and docs/sdk/changelog.mdx, bumping packages/sdk/package.json to 0.1.12. All manifest and docs synchronization checks pass cleanly.

[HoangDucBach]

Approved!