Python: ONNX connector puts Phi-3-vision / Phi-4-multimodal media tags outside the user turn, and rejects repetition_penalty > 1 #14735

Describe the bug

Two bugs in the Python ONNX connector (semantic_kernel.connectors.ai.onnx):

1. ONNXTemplate.PHI3V and ONNXTemplate.PHI4MM build the wrong prompt for multimodal messages. phi3v_template and phi4mm_template in utils.py write <|image_N|> / <|audio_N|> outside the <|user|> ... <|end|> turn, and they open a separate <|user|> turn for every TextContent item. Where the tag lands depends on the item order, so [text, image] puts the image after <|end|> and [image, text] puts it before <|user|>.
2. OnnxGenAIPromptExecutionSettings limits repetition_penalty and length_penalty to [0, 1] (source). In onnxruntime-genai, repetition_penalty = 1.0 means no penalty and values above 1.0 penalize repetition. The validator therefore rejects every value that reduces repetition, and it accepts values below 1.0, which make repetition more likely. length_penalty is an exponent and can be above 1 or below 0.

To Reproduce

No model is needed. Tested with semantic-kernel 1.47.0 and onnxruntime-genai 0.18.0 (the version pinned by the onnx extra), and on main at df1cb33.

from pydantic import ValidationError

from semantic_kernel.connectors.ai.onnx import OnnxGenAIPromptExecutionSettings
from semantic_kernel.connectors.ai.onnx.utils import ONNXTemplate, apply_template
from semantic_kernel.contents import AudioContent, AuthorRole, ChatHistory, ChatMessageContent, ImageContent, TextContent


def user(*items):
    h = ChatHistory()
    h.add_message(ChatMessageContent(role=AuthorRole.USER, items=list(items)))
    return h


img, wav = ImageContent(uri="file:///tmp/cat.png"), AudioContent(uri="file:///tmp/a.wav")
print(repr(apply_template(user(TextContent(text="Describe this image."), img), ONNXTemplate.PHI3V)))
print(repr(apply_template(user(img, TextContent(text="Describe this image.")), ONNXTemplate.PHI3V)))
print(repr(apply_template(user(TextContent(text="Transcribe the audio."), wav), ONNXTemplate.PHI4MM)))

for kw in ({"repetition_penalty": 1.1}, {"repetition_penalty": 0.9}, {"length_penalty": 1.5}, {"length_penalty": -0.5}):
    try:
        OnnxGenAIPromptExecutionSettings(**kw)
        print("ACCEPTED", kw)
    except ValidationError as e:
        print("REJECTED", kw, "->", e.errors()[0]["msg"])

Actual output:

'<|user|>\nDescribe this image.<|end|>\n<|image_1|>\n<|assistant|>\n'
'<|image_1|>\n<|user|>\nDescribe this image.<|end|>\n<|assistant|>\n'
'<|user|>\nTranscribe the audio.<|end|>\n<|audio_1|>\n<|assistant|>\n'
REJECTED {'repetition_penalty': 1.1} -> Input should be less than or equal to 1
ACCEPTED {'repetition_penalty': 0.9}
REJECTED {'length_penalty': 1.5} -> Input should be less than or equal to 1
REJECTED {'length_penalty': -0.5} -> Input should be greater than or equal to 0

The existing unit tests in tests/unit/connectors/ai/onnx/services/test_onnx_utils.py assert the first format, which is why they pass.

Expected behavior

1. Each user message becomes one user turn. The image placeholders come first, then the audio placeholders, then the text:

   '<|user|>\n<|image_1|>\nDescribe this image.<|end|>\n<|assistant|>\n'   # [text, image] and [image, text]
   '<|user|>\n<|audio_1|>\nTranscribe the audio.<|end|>\n<|assistant|>\n'
   

   Sources:

   • Phi-3-vision-128k-instruct model card: <|user|>\n<|image_1|>\n{prompt}<|end|>\n<|assistant|>\n. In multi-turn chats the image goes in the first user turn only.
   • Phi-3.5-vision-instruct model card: "For multi-image usage, add multiple image placeholders in the front of the prompts. <|image_{}|> index should start from 1", e.g. <|user|>\n<|image_1|>\n<|image_2|>\n<|image_3|>\n<|image_4|>\n{prompt}<|end|>\n<|assistant|>\n. Its processing_phi3_v.py asserts that the image ids start at 1 and are consecutive.
   • Phi-4-multimodal-instruct model card ("Input Formats"): <|user|><|image_1|>Describe the image in detail.<|end|><|assistant|>, <|user|><|image_1|><|image_2|><|image_3|>Summarize the content of the images.<|end|><|assistant|>, <|user|><|audio_1|>{task prompt}<|end|><|assistant|> and, for vision + speech, <|user|><|image_1|><|image_2|><|image_3|><|audio_1|><|end|><|assistant|>.
   • onnxruntime-genai multimodal example, examples/python/common.py get_user_content, used by model-mm.py: "Combine all image tags, audio tags, and text into one user content". For phi3v this is "".join(f"<|image_{i + 1}|>\n" ...) + prompt, and for phi4mm it is image_tags + audio_tags + prompt.

2. repetition_penalty accepts any value > 0, and length_penalty accepts any float. From the onnxruntime-genai source:

   • src/config.h says float repetition_penalty{1.0f}; // 1.0 means no penalty. and float length_penalty{1.0f}; // ... length_penalty > 0.0 promotes longer sequences, while length_penalty < 0.0 encourages shorter sequences. These lines are the same at v0.18.0, the version the onnx extra pins.
   • src/sampling_distribution.h ApplyRepetitionPenaltyToLogits, called from Search_Cpu::ApplyRepetitionPenalty in src/search.cpp, computes score < 0 ? score * penalty : score / penalty. A value of 1.0 is a no-op, values above 1.0 lower the scores of tokens already generated, 0 divides by zero, and negative values flip the sign.
   • src/beam_search_scorer.cpp computes sum_logprobs / std::pow(length, length_penalty_).
   • onnxruntime-genai checks top_p and top_k when it creates a generator (src/generator/generators.cpp). It does not range-check either penalty.
   • The config reference says "repetition_penalty: Penalty for repeating tokens. 1.0 means no penalty."

Proposed fix

I have both fixes ready, with unit tests, as two small PRs:

1. In utils.py, add a small helper that builds the content of one user turn: the image tags (numbered across the whole history, as today), then the audio tags (Phi-4-mm only), then the text items joined with \n. Both templates would emit <|user|>\n{content}<|end|>\n once per user message. Then update the two existing tests and add cases for [text, image], [image, text], multiple images, image numbering across turns, several text items, text + audio, audio only, and image + audio. The newline style (<|role|>\n...<|end|>\n) stays as it is today.
2. In OnnxGenAIPromptExecutionSettings, change repetition_penalty to Field(gt=0.0) and drop the range on length_penalty. Then add tests that 1.1, 1.5 and 2.0 are accepted, that 0 and negative values are rejected for repetition_penalty, and that length_penalty accepts -1.0, -0.5, 1.5 and 2.0. Defaults stay None, so nothing is sent to onnxruntime-genai unless the user sets a value.

Neither change touches a public API signature.

Questions / notes

• Placeholder order. The fix follows the model cards and the onnxruntime-genai example: all media placeholders go before the text, images before audio. It does not keep the order of the content items, so [text, image] gives the same prompt as [image, text]. Would you rather keep the item order inside the single user turn? Both processors would accept that (Phi-3-vision splits on <|image_N|>, and Phi-4-mm replaces the tags wherever they are), but the documented format puts the placeholders first.
• Newlines for Phi-4-multimodal. The Phi-4-multimodal model card and its chat_template use no newline after the role and placeholder tokens (<|user|><|image_1|>...<|end|><|assistant|>). The onnxruntime-genai example adds \n after each placeholder. The SK template keeps its current \n style. Tell me if you want it to match the model card exactly.
• ONNXTemplate.PHI4. It produces the Phi-3 format (<|user|>\n...<|end|>\n<|assistant|>\n), which is correct for Phi-4-mini-instruct. microsoft/phi-4 (14B), however, uses <|im_start|>user<|im_sep|>...<|im_end|>. I have not changed this. Is PHI4 meant only for Phi-4-mini, so a docstring note is enough, or would a separate template for Phi-4 (14B) be welcome?
• diversity_penalty is marked "Unused param" / "Not supported" in onnxruntime-genai. I left it alone.

Platform

• Language: Python
• Source: semantic-kernel 1.47.0 from pip, and main at df1cb33; onnxruntime-genai 0.18.0
• AI model: no model needed; the bug is in prompt construction and settings validation (Phi-3-vision, Phi-3.5-vision and Phi-4-multimodal ONNX models are affected)
• OS: Linux (x86_64)

Additional context

I'd like to work on this, could you assign me?