Skip to content

docs: clarify custom predictor protocol settings #709

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
euisuh wants to merge 2 commits into
base: main
Choose a base branch
from
+16 −1
Open

docs: clarify custom predictor protocol settings #709

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a34e89c
docs: clarify custom predictor protocol settings
euisuh Jul 24, 2026
94666ae
docs: refine predictor protocol wording
euisuh Jul 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 16 additions & 1 deletion ...el-serving/predictive-inference/frameworks/custom-predictor/custom-predictor.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,19 @@ if __name__ == "__main__":
`return_response_headers=True` can be added to return response headers for v1 and v2 endpoints
:::

:::note[REST protocol version]
The Python `ModelServer` registers both REST v1 and REST v2 routes. The request
path and handler payload type determine which protocol is used: v1 requests call
paths such as `/v1/models/{model_name}:predict` with a `dict` payload, while v2
requests call paths such as `/v2/models/{model_name}/infer` and should use
`InferRequest`/`InferResponse` as shown in the Open(v2) example below.

The `--predictor_protocol` argument, and its deprecated alias `--protocol`,
primarily configure the protocol used for outbound calls to a separate
predictor, for example from a transformer. They do not select which REST routes
the custom predictor exposes.
:::

### Build Custom Serving Image with BuildPacks
[Buildpacks](https://buildpacks.io/) allows you to transform your inference code into images that can be deployed on KServe without
needing to define the `Dockerfile`. Buildpacks automatically determines the python application and then install the dependencies from the
Expand Down Expand Up @@ -186,6 +199,9 @@ You can supply additional command arguments on the container spec to configure t
- `--grpc_max_send_message_length`: The max message length for gRPC send message. Default is 8388608 bytes (8 MB).
- `--grpc_max_receive_message_length`: The max message length for gRPC receive message. Default is 8388608 bytes (8 MB).
- `--model_name`: The model name deployed in the model server, the default name the same as the service name.
- `--predictor_protocol`: The inference protocol used when this container calls a separate predictor, for example from a transformer.
Valid values are `v1`, `v2`, and `grpc-v2`. It does not control the protocol routes exposed by the custom predictor.
- `--protocol`: Deprecated alias for `--predictor_protocol`.
- `--max_asyncio_workers`: Max number of workers to spawn for python async io loop, by default it is `min(32,cpu.limit + 4)`.
- `--enable_latency_logging`: Whether to log latency metrics per request, the default is True.
- `--configure_logging`: Whether to configure KServe and Uvicorn logging, the default is True.
Expand All @@ -202,7 +218,6 @@ You can supply additional environment variables on the container spec.

- `STORAGE_URI`: load a model from a storage system supported by KServe e.g. `pvc://` `s3://`. This acts the same as `storageUri` when using a built-in predictor.
The data will be available at `/mnt/models` in the container. For example, the following `STORAGE_URI: "pvc://my_model/model.onnx"` will be accessible at `/mnt/models/model.onnx`
- `PROTOCOL`: specify the protocol version supported by the model e.g `V1`. This acts the same as `protocolVersion` when using a built-in predictor.

@spolti spolti Aug 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello,

please, don't fully remove the PROTOCOL env — instead, rewrite it to accurately describe what it does: it signals the Go controller which protocol version the predictor uses (for routing purposes), but does not change which routes the Python ModelServer exposes.

We could do it in the --protocol option added.

- `KSERVE_LOGLEVEL`: sets the `kserve` and `kserve_trace`'s logger verbosity. Default is `INFO`.

Apply the YAML to deploy the InferenceService on KServe
Expand Down