Skip to content

Add token classification offset mapping documentation #659

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Yuuuuuu0319 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add token classification offset mapping documentation #659

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions docs/model-serving/generative-inference/overview.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -194,9 +194,20 @@ Below is an explanation of command line arguments supported by the Hugging Face
- `--trust_remote_code`: Allow loading of models and tokenizers with custom code.
- `--tensor_input_names`: The tensor input names passed to the model for triton inference server backend.
- `--return_token_type_ids`: Return token type ids.
- `--return_offsets_mapping`: Return tokenizer offset mappings in the response outputs. This is applicable to `token_classification` tasks and can be used to align predicted tokens with character spans in the original input text.
- `--return_probabilities`: Return probabilities of predicted indexes. This is only applicable for tasks 'sequence_classification', 'token_classification' and 'fill_mask'.
- `--disable_log_requests`: Disable logging of requests.

#### Token Classification Notes

For `token_classification` tasks, the runtime can use tokenizer-provided offset mappings to align predicted tokens with the original input text.

- Offset mappings provide the start and end character positions of each token in the original string.
- This is useful for named entity recognition (NER) and other span-based token classification tasks.
- If the tokenizer does not support offset mappings, accurate character-level span alignment may not be available.

Enable this behavior using the `--return_offsets_mapping` runtime argument. It is recommended to use tokenizers that support `return_offsets_mapping=True` for accurate results.

### vLLM Specific Configuration

The Hugging Face runtime supports **all** vLLM engine arguments, allowing for complete customization of the vLLM backend. These arguments can be passed directly when deploying the runtime. For a comprehensive list of available parameters, refer to [vLLM's engine arguments documentation](https://docs.vllm.ai/en/latest/serving/engine_args.html).
Expand Down