Repository navigation
Malay filenames in the question come back translated (lama.txt -> old.txt) #51
Description
Shipped model, v0.8.0 (run 7):
$ camne nak delete lama.txt
rm old.txt
$ camne nak rename fail baru.txt jadi lama.txt
mv new.txt old.txt
$ camne nak tengok isi file nota.txt
cat note.txt
$ camne nak delete file 'lama.txt'
rm lama.txt
Quoting the name protects it; a bare Malay-word filename is translated to its English equivalent. Likely cause is the pool: NL was translated from English rows whose command names the file (rm old.txt), the translation API turned old.txt into lama.txt in the NL while the command stayed byte-identical, so the model learned lama.txt -> old.txt as a mapping. Filenames need to be on the stoplist the same way technical nouns are (CLAUDE.md, dataset work), or the NL side needs the original filename restored after translation.
Check: grep -c 'lama.txt' dataset/out/pool_v5.jsonl against rows whose cmd has old.txt.
README notes the quoting workaround for now. Same family as #47 (data, not code); measure with a handful of exact-output prompts, then a run.
Activity
opariffazman commentedon Aug 17, 2026
Measured (2026-08-18), details in RESULTS.md on #67.
The hypothesis here is falsified:
grep lama.txt | grep old.txtis 0 in every pool, and every NL filename absent from its command (219 rows in pool_v9) is a legitimate mention. The mapping is the base model's translation prior; what beats it is head rows carrying Malay filenames byte-identical on both sides.Eight exact-output prompts, now
EXACTintraining/probe.py(#51 exact n/8):rm old.txt,cat note.txt,cp report.pdf work/, …)nak rename fail baru.txt jadi lama.txt→mv new.txt old.txtmv {t} {t2}block): 7/8 — rename right on every name pair exceptbaru.txt, a name in zero pool rowsbaru.txtin the{t}list, 220 rows): 8/8 on SFT and DPONot shipped: run 13's BM ALFA is 0.423 vs 0.497, CI [−0.125, −0.022] — the first head-row-family model where the CI clears zero. Data fix is in #67 and rides with the next model that clears ALFA. README quoting workaround stays until then.
Merge pull request #67 from officialdad/fix/51-malay-filenames