Allow overriding past_present_share_buffer in LMEvaluator ortgenai backend

[jiafatom]

Describe your changes

The ortgenai lm-eval backend (LMEvalORTGenAIEvaluator) always derived past_present_share_buffer from the exported genai_config.json, with no way to override it from a recipe/evaluator config. Some models (e.g. Gemma 4) export with shared past/present buffers enabled but require it disabled for correct KV-cache handling during evaluation. Additionally, LMEvaluator gave recipe authors no way to enable a model's chat template or few-shot prompting, so instruction-tuned models were evaluated 0-shot on raw prompts and scored near-random. Today these gaps force recipe authors to bypass LMEvaluator and write a standalone eval.py that calls lm_eval.simple_evaluate directly — the declarative Olive config path simply produces wrong results for these models.

This PR makes these settings overridable end-to-end and adds optional sample logging:

• LMEvalORTGenAIEvaluator now accepts an explicit past_present_share_buffer argument that takes precedence over the exported value. The resolution logic is extracted into a small _resolve_past_present_share_buffer(override, genai_config) helper (explicit override wins; otherwise fall back to the exported value, defaulting to False).
• LMEvaluator gains a model_args dict that is forwarded verbatim to the lm-eval model backend constructor, so genai-specific options can be set from the evaluator config. User-provided model_args override the Olive-derived defaults (batch_size, max_length, ep, ...). Default is {}, so existing configs are unaffected.
• LMEvaluator prompting controls: new apply_chat_template, system_instruction, fewshot_as_multiturn and num_fewshot options, forwarded to lm_eval.simple_evaluate. Instruction-tuned models score near-random on multiple-choice tasks when evaluated 0-shot on raw prompts; wrapping requests in the model's chat template and matching the card's few-shot protocol fixes this. Empirically, on a Gemma-4 E2B export a German MMMLU subset went from ~0.16–0.24 (0-shot, no template) to ~0.52–0.68 (5-shot + chat template). Defaults preserve lm-eval's legacy behavior (no chat template, no system prompt, task-default few-shot), so existing configs are unaffected.
• LMEvaluator sample logging: a new sample_log_num (and optional sample_log_dir) option. When sample_log_num > 0, lm-eval's log_samples is enabled and the first N per-task predictions — question, options, resolved prediction (argmax of choice loglikelihoods to letter + text, or the raw generation), target, and acc — are written to <task>_samples.jsonl. Rather than duplicating the writer, OliveEvaluator.save_sample_log is decoupled to take a (name, sample_log_dir) pair instead of a Metric, so LMEvaluator reuses the exact same JSONL writer as the accuracy evaluators; a small _lmeval_samples_to_output helper maps lm-eval sample records into an OliveModelOutput (carrying question/options/prediction_index/acc via extras). This makes it possible to inspect why a task scores low (e.g. distinguishing a genuinely weak/quantized model from a broken eval). Default is 0, so existing configs are unaffected.

With this, such models can be evaluated with a standard LMEvaluator config instead of a bespoke script:

{
    "evaluators": {
        "mmlu": {
            "type": "LMEvaluator",
            "tasks": ["mmlu"],
            "model_class": "ortgenai",
            "num_fewshot": 5,
            "apply_chat_template": true,
            "fewshot_as_multiturn": true,
            "sample_log_num": 100,
            "model_args": { "past_present_share_buffer": false }
        }
    }
}

Checklist before requesting a review

• Add unit tests for this change.
• Make sure all tests can pass.
• Update documents if necessary.
• Lint and apply fixes to your code by running lintrunner -a
• Is this a user-facing change? If yes, give a description of this change to be included in the release notes.

Release note: LMEvaluator now accepts a model_args dict forwarded to the lm-eval model backend; apply_chat_template, system_instruction, fewshot_as_multiturn and num_fewshot prompting options forwarded to lm_eval.simple_evaluate (so instruction-tuned models can be evaluated with their chat template and the card's few-shot protocol); and a sample_log_num/sample_log_dir option to log the first N per-task predictions vs. targets to a JSONL file for debugging (reusing the accuracy evaluators' save_sample_log writer). The ortgenai backend accepts an explicit past_present_share_buffer override (previously only read from genai_config.json).

(Optional) Issue link

[Copilot]

Pull request overview

This PR enhances Olive’s LMEvaluator integration with lm-eval by allowing recipe configs to pass backend-specific constructor kwargs via a new model_args dict, and by enabling an explicit override of past_present_share_buffer for the ortgenai backend to address KV-cache correctness issues for certain exported models (e.g., Gemma 4).

Changes:

• Add model_args support to LMEvaluator, forwarding kwargs to the selected lm-eval backend constructor with user values overriding Olive-derived defaults.
• Add a past_present_share_buffer override path to LMEvalORTGenAIEvaluator, with precedence over the exported genai_config.json value (and defaulting to False when absent).
• Add unit tests covering the new resolution helper and the forwarding behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
olive/evaluator/olive_evaluator.py	Introduces model_args forwarding into lm-eval backend construction.
olive/evaluator/lmeval_ort.py	Adds past_present_share_buffer override resolution for the ortgenai backend.
test/evaluator/test_olive_evaluator.py	Adds a test ensuring model_args reaches the backend constructor.
test/evaluator/test_lmeval_ort.py	Adds tests for _resolve_past_present_share_buffer precedence/default behavior.

[jiafatom]

Thanks for the review! Addressed all three in 02d7818:

1. simple_evaluate batch size inconsistency — now uses effective_batch_size = model_init_args["batch_size"] so lm-eval batching matches the backend after any model_args override.
2. Non-dict model_args — validated in __init__ with an early ValueError("model_args must be a dict, ...").
3. Test coverage — the forwarding test now also asserts simple_evaluate receives the overridden batch size, plus a new test that a non-dict model_args is rejected.