fix(rag): tokenize CJK text so BM25 retrieval can score non-Latin queries - #146

kuroudo-ai wants to merge 1 commit into

fix(rag): tokenize CJK text so BM25 retrieval can score non-Latin queries#146
kuroudo-ai wants to merge 1 commit into
LLMQuant:masterfrom
kuroudo-ai:fix/bm25-cjk-tokenization

fix(rag): tokenize CJK text for BM25 retrieval