diff --git a/11-embeddings-reranker-classification-tensorrt/BISV2-meta-llama-llama-3.2-3b-instruct-calib-dataset-fp8/README.md b/11-embeddings-reranker-classification-tensorrt/BISV2-meta-llama-llama-3.2-3b-instruct-calib-dataset-fp8/README.md new file mode 100644 index 00000000..a42ec168 --- /dev/null +++ b/11-embeddings-reranker-classification-tensorrt/BISV2-meta-llama-llama-3.2-3b-instruct-calib-dataset-fp8/README.md @@ -0,0 +1,171 @@ +# Baseten Inference Stack with meta-llama/Llama-3.2-3B-Instruct-calib-dataset + +This is a Deployment for Baseten Inference Stack with meta-llama/Llama-3.2-3B-Instruct-calib-dataset. Baseten Inference Stack is Baseten's solution for production-grade deployments via TensorRT-LLM for Causal Language Models models. (e.g. LLama, Qwen, Mistral) + +With Baseten Inference Stack you get the following benefits by default: +- *Lowest-latency* latency, beating frameworks such as vllm +- *Highest-throughput* inference, automatically using XQA kernels, paged kv caching and inflight batching. +- *distributed inference* run large models (such as LLama-405B) tensor-parallel +- *json-schema based structured output for any model* +- *chunked prefilling* for long generation tasks + +Optionally, you can also enable: +- *speculative decoding* using an external draft model or self-speculative decoding +- *fp8 quantization* deployments on H100, H200 and L4 GPUs +- *fp4 quantization* deployments on B200 GPUs to get even more speed + + +# Examples: +This deployment is specifically designed for the Hugging Face model [meta-llama/Llama-3.2-3B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct). +Suitable models can be identified by the `ForCausalLM` suffix in the model name. Currently we support e.g. LLama, Qwen, Mistral models. + +meta-llama/Llama-3.2-3B-Instruct is a text-generation model, used to generate text given a prompt. \nIt is frequently used in chatbots, text completion, structured output and more. + +This model is quantized to FP8 for deployment, which is supported by Nvidia's newest GPUs e.g. H100, H100_40GB or L4. Quantization is optional, but leads to higher efficiency. + +## Deployment with Truss + +Before deployment: + +1. Make sure you have a [Baseten account](https://app.baseten.co/signup) and [API key](https://app.baseten.co/settings/account/api_keys). +2. Install the latest version of Truss: `pip install --upgrade truss` +Note: [This is a gated/private model] Retrieve your Hugging Face token from the [settings](https://huggingface.co/settings/tokens). Set your Hugging Face token as a Baseten secret [here](https://app.baseten.co/settings/secrets) with the key `hf_access_token`. Do not set the actual value of key in the config.yaml. `hf_access_token: null` is fine - the true value will be fetched from the secret store. + +First, clone this repository: +```sh +git clone https://github.com/basetenlabs/truss-examples.git +cd 11-embeddings-reranker-classification-tensorrt/BISV2-meta-llama-llama-3.2-3b-instruct-calib-dataset-fp8 +``` + +With `11-embeddings-reranker-classification-tensorrt/BISV2-meta-llama-llama-3.2-3b-instruct-calib-dataset-fp8` as your working directory, you can deploy the model with the following command. Paste your Baseten API key if prompted. + +```sh +truss push --publish +# prints: +# ✨ Model BISV2-meta-llama-llama-3.2-3b-instruct-calib-dataset-fp8-truss-example was successfully pushed ✨ +# 🪵 View logs for your deployment at https://app.baseten.co/models/yyyyyy/logs/xxxxxx +``` + +## Call your model + +### OpenAI compatible inference +This solution is OpenAI compatible, which means you can use the OpenAI client library to interact with the model. + +```python +from openai import OpenAI +import os + +client = OpenAI( + api_key=os.environ['BASETEN_API_KEY'], + base_url="https://model-xxxxxx.api.baseten.co/environments/production/sync/v1" +) + +# Default completion +response_completion = client.completions.create( + model="not_required", + prompt="Q: Tell me everything about Baseten.co! A:", + temperature=0.3, + max_tokens=100, +) + +# Chat completion +response_chat = client.chat.completions.create( + model="", + messages=[ + {"role": "user", "content": "Tell me everything about Baseten.co!"} + ], + temperature=0.3, + max_tokens=100, +) + +# Structured output +from pydantic import BaseModel + +class CalendarEvent(BaseModel): + name: str + date: str + participants: list[str] + +completion = client.beta.chat.completions.parse( + model="not_required", + messages=[ + {"role": "system", "content": "Extract the event information."}, + {"role": "user", "content": "Alice and Bob are going to a science fair on Friday."}, + ], + response_format=CalendarEvent, +) + +event = completion.choices[0].message.parsed + +# If you model supports tool-calling, you can use the following example: +tools = [{ + "type": "function", + "function": { + "name": "get_weather", + "description": "Get current temperature for a given location.", + "parameters": { + "type": "object", + "properties": { + "location": { + "type": "string", + "description": "City and country e.g. Bogotá, Colombia" + } + }, + "required": [ + "location" + ], + "additionalProperties": False + }, + "strict": True + } +}] + +completion = client.chat.completions.create( + model="not_required", + messages=[{"role": "user", "content": "What is the weather like in Paris today?"}], + tools=tools +) + +print(completion.choices[0].message.tool_calls) +``` + + +## Config.yaml +By default, the following configuration is used for this deployment. This config uses `quantization_type=fp8_kv`. This is optional, remove the `quantization_type` field or set it to `no_quant` for float16/bfloat16. +Note: [This is a gated/private model] Retrieve your Hugging Face token from the [settings](https://huggingface.co/settings/tokens). Set your Hugging Face token as a Baseten secret [here](https://app.baseten.co/settings/secrets) with the key `hf_access_token`. Do not set the actual value of key in the config.yaml. `hf_access_token: null` is fine - the true value will be fetched from the secret store. +```yaml +model_metadata: + example_model_input: + max_tokens: 512 + messages: + - content: Tell me everything you know about optimized inference. + role: user + stream: true + temperature: 0.5 + tags: + - openai-compatible +model_name: BISV2-meta-llama-llama-3.2-3b-instruct-calib-dataset-fp8-truss-example +python_version: py39 +resources: + accelerator: H100_40GB + cpu: '1' + memory: 10Gi + use_gpu: true +trt_llm: + build: + checkpoint_repository: + repo: meta-llama/Llama-3.2-3B-Instruct + revision: main + source: HF + quantization_config: + calib_dataset: baseten/quant_calibration_dataset_v1 + quantization_type: fp8_kv + runtime: + max_batch_size: 32 + max_num_tokens: 32768 + max_seq_len: 32768 + +``` + +## Support +If you have any questions or need assistance, please open an issue in this repository or contact our support team. diff --git a/11-embeddings-reranker-classification-tensorrt/BISV2-meta-llama-llama-3.2-3b-instruct-calib-dataset-fp8/config.yaml b/11-embeddings-reranker-classification-tensorrt/BISV2-meta-llama-llama-3.2-3b-instruct-calib-dataset-fp8/config.yaml new file mode 100644 index 00000000..73737182 --- /dev/null +++ b/11-embeddings-reranker-classification-tensorrt/BISV2-meta-llama-llama-3.2-3b-instruct-calib-dataset-fp8/config.yaml @@ -0,0 +1,31 @@ +model_metadata: + example_model_input: + max_tokens: 512 + messages: + - content: Tell me everything you know about optimized inference. + role: user + stream: true + temperature: 0.5 + tags: + - openai-compatible +model_name: BISV2-meta-llama-llama-3.2-3b-instruct-calib-dataset-fp8-truss-example +python_version: py39 +resources: + accelerator: H100_40GB + cpu: '1' + memory: 10Gi + use_gpu: true +trt_llm: + inference_stack: v2 + build: + checkpoint_repository: + repo: meta-llama/Llama-3.2-3B-Instruct + revision: main + source: HF + quantization_config: + calib_dataset: baseten/quant_calibration_dataset_v1 + quantization_type: fp8_kv + runtime: + max_batch_size: 32 + max_num_tokens: 32768 + max_seq_len: 32768 diff --git a/11-embeddings-reranker-classification-tensorrt/BISV2-meta-llama-llama-3.3-70b-instruct-calib-dataset-fp4/README.md b/11-embeddings-reranker-classification-tensorrt/BISV2-meta-llama-llama-3.3-70b-instruct-calib-dataset-fp4/README.md new file mode 100644 index 00000000..7e764acc --- /dev/null +++ b/11-embeddings-reranker-classification-tensorrt/BISV2-meta-llama-llama-3.3-70b-instruct-calib-dataset-fp4/README.md @@ -0,0 +1,170 @@ +# Baseten Inference Stack with meta-llama/Llama-3.3-70B-Instruct-calib-dataset + +This is a Deployment for Baseten Inference Stack with meta-llama/Llama-3.3-70B-Instruct-calib-dataset. Baseten Inference Stack is Baseten's solution for production-grade deployments via TensorRT-LLM for Causal Language Models models. (e.g. LLama, Qwen, Mistral) + +With Baseten Inference Stack you get the following benefits by default: +- *Lowest-latency* latency, beating frameworks such as vllm +- *Highest-throughput* inference, automatically using XQA kernels, paged kv caching and inflight batching. +- *distributed inference* run large models (such as LLama-405B) tensor-parallel +- *json-schema based structured output for any model* +- *chunked prefilling* for long generation tasks + +Optionally, you can also enable: +- *speculative decoding* using an external draft model or self-speculative decoding +- *fp8 quantization* deployments on H100, H200 and L4 GPUs +- *fp4 quantization* deployments on B200 GPUs to get even more speed + + +# Examples: +This deployment is specifically designed for the Hugging Face model [meta-llama/Llama-3.3-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct). +Suitable models can be identified by the `ForCausalLM` suffix in the model name. Currently we support e.g. LLama, Qwen, Mistral models. + +meta-llama/Llama-3.3-70B-Instruct is a text-generation model, used to generate text given a prompt. \nIt is frequently used in chatbots, text completion, structured output and more. + + +## Deployment with Truss + +Before deployment: + +1. Make sure you have a [Baseten account](https://app.baseten.co/signup) and [API key](https://app.baseten.co/settings/account/api_keys). +2. Install the latest version of Truss: `pip install --upgrade truss` +Note: [This is a gated/private model] Retrieve your Hugging Face token from the [settings](https://huggingface.co/settings/tokens). Set your Hugging Face token as a Baseten secret [here](https://app.baseten.co/settings/secrets) with the key `hf_access_token`. Do not set the actual value of key in the config.yaml. `hf_access_token: null` is fine - the true value will be fetched from the secret store. + +First, clone this repository: +```sh +git clone https://github.com/basetenlabs/truss-examples.git +cd 11-embeddings-reranker-classification-tensorrt/BISV2-meta-llama-llama-3.3-70b-instruct-calib-dataset-fp4 +``` + +With `11-embeddings-reranker-classification-tensorrt/BISV2-meta-llama-llama-3.3-70b-instruct-calib-dataset-fp4` as your working directory, you can deploy the model with the following command. Paste your Baseten API key if prompted. + +```sh +truss push --publish +# prints: +# ✨ Model BISV2-meta-llama-llama-3.3-70b-instruct-calib-dataset-fp4-truss-example was successfully pushed ✨ +# 🪵 View logs for your deployment at https://app.baseten.co/models/yyyyyy/logs/xxxxxx +``` + +## Call your model + +### OpenAI compatible inference +This solution is OpenAI compatible, which means you can use the OpenAI client library to interact with the model. + +```python +from openai import OpenAI +import os + +client = OpenAI( + api_key=os.environ['BASETEN_API_KEY'], + base_url="https://model-xxxxxx.api.baseten.co/environments/production/sync/v1" +) + +# Default completion +response_completion = client.completions.create( + model="not_required", + prompt="Q: Tell me everything about Baseten.co! A:", + temperature=0.3, + max_tokens=100, +) + +# Chat completion +response_chat = client.chat.completions.create( + model="", + messages=[ + {"role": "user", "content": "Tell me everything about Baseten.co!"} + ], + temperature=0.3, + max_tokens=100, +) + +# Structured output +from pydantic import BaseModel + +class CalendarEvent(BaseModel): + name: str + date: str + participants: list[str] + +completion = client.beta.chat.completions.parse( + model="not_required", + messages=[ + {"role": "system", "content": "Extract the event information."}, + {"role": "user", "content": "Alice and Bob are going to a science fair on Friday."}, + ], + response_format=CalendarEvent, +) + +event = completion.choices[0].message.parsed + +# If you model supports tool-calling, you can use the following example: +tools = [{ + "type": "function", + "function": { + "name": "get_weather", + "description": "Get current temperature for a given location.", + "parameters": { + "type": "object", + "properties": { + "location": { + "type": "string", + "description": "City and country e.g. Bogotá, Colombia" + } + }, + "required": [ + "location" + ], + "additionalProperties": False + }, + "strict": True + } +}] + +completion = client.chat.completions.create( + model="not_required", + messages=[{"role": "user", "content": "What is the weather like in Paris today?"}], + tools=tools +) + +print(completion.choices[0].message.tool_calls) +``` + + +## Config.yaml +By default, the following configuration is used for this deployment. This config uses `quantization_type=fp4`. This is optional, remove the `quantization_type` field or set it to `no_quant` for float16/bfloat16. +Note: [This is a gated/private model] Retrieve your Hugging Face token from the [settings](https://huggingface.co/settings/tokens). Set your Hugging Face token as a Baseten secret [here](https://app.baseten.co/settings/secrets) with the key `hf_access_token`. Do not set the actual value of key in the config.yaml. `hf_access_token: null` is fine - the true value will be fetched from the secret store. +```yaml +model_metadata: + example_model_input: + max_tokens: 512 + messages: + - content: Tell me everything you know about optimized inference. + role: user + stream: true + temperature: 0.5 + tags: + - openai-compatible +model_name: BISV2-meta-llama-llama-3.3-70b-instruct-calib-dataset-fp4-truss-example +python_version: py39 +resources: + accelerator: B200 + cpu: '1' + memory: 10Gi + use_gpu: true +trt_llm: + build: + checkpoint_repository: + repo: meta-llama/Llama-3.3-70B-Instruct + revision: main + source: HF + quantization_config: + calib_dataset: baseten/quant_calibration_dataset_v1 + quantization_type: fp4 + runtime: + max_batch_size: 32 + max_num_tokens: 32768 + max_seq_len: 32768 + +``` + +## Support +If you have any questions or need assistance, please open an issue in this repository or contact our support team. diff --git a/11-embeddings-reranker-classification-tensorrt/BISV2-meta-llama-llama-3.3-70b-instruct-calib-dataset-fp4/config.yaml b/11-embeddings-reranker-classification-tensorrt/BISV2-meta-llama-llama-3.3-70b-instruct-calib-dataset-fp4/config.yaml new file mode 100644 index 00000000..09466d41 --- /dev/null +++ b/11-embeddings-reranker-classification-tensorrt/BISV2-meta-llama-llama-3.3-70b-instruct-calib-dataset-fp4/config.yaml @@ -0,0 +1,31 @@ +model_metadata: + example_model_input: + max_tokens: 512 + messages: + - content: Tell me everything you know about optimized inference. + role: user + stream: true + temperature: 0.5 + tags: + - openai-compatible +model_name: BISV2-meta-llama-llama-3.3-70b-instruct-calib-dataset-fp4-truss-example +python_version: py39 +resources: + accelerator: B200 + cpu: '1' + memory: 10Gi + use_gpu: true +trt_llm: + inference_stack: v2 + build: + checkpoint_repository: + repo: meta-llama/Llama-3.3-70B-Instruct + revision: main + source: HF + quantization_config: + calib_dataset: baseten/quant_calibration_dataset_v1 + quantization_type: fp4 + runtime: + max_batch_size: 32 + max_num_tokens: 32768 + max_seq_len: 32768 diff --git a/11-embeddings-reranker-classification-tensorrt/BISV2-qwen-qwen3-32b-fp4-mlp-only/README.md b/11-embeddings-reranker-classification-tensorrt/BISV2-qwen-qwen3-32b-fp4-mlp-only/README.md new file mode 100644 index 00000000..23fc859c --- /dev/null +++ b/11-embeddings-reranker-classification-tensorrt/BISV2-qwen-qwen3-32b-fp4-mlp-only/README.md @@ -0,0 +1,170 @@ +# Baseten Inference Stack with Qwen/Qwen3-32B + +This is a Deployment for Baseten Inference Stack with Qwen/Qwen3-32B. Baseten Inference Stack is Baseten's solution for production-grade deployments via TensorRT-LLM for Causal Language Models models. (e.g. LLama, Qwen, Mistral) + +With Baseten Inference Stack you get the following benefits by default: +- *Lowest-latency* latency, beating frameworks such as vllm +- *Highest-throughput* inference, automatically using XQA kernels, paged kv caching and inflight batching. +- *distributed inference* run large models (such as LLama-405B) tensor-parallel +- *json-schema based structured output for any model* +- *chunked prefilling* for long generation tasks + +Optionally, you can also enable: +- *speculative decoding* using an external draft model or self-speculative decoding +- *fp8 quantization* deployments on H100, H200 and L4 GPUs +- *fp4 quantization* deployments on B200 GPUs to get even more speed + + +# Examples: +This deployment is specifically designed for the Hugging Face model [Qwen/Qwen3-32B](https://huggingface.co/Qwen/Qwen3-32B). +Suitable models can be identified by the `ForCausalLM` suffix in the model name. Currently we support e.g. LLama, Qwen, Mistral models. + +Qwen/Qwen3-32B is a text-generation model, used to generate text given a prompt. \nIt is frequently used in chatbots, text completion, structured output and more. + + +## Deployment with Truss + +Before deployment: + +1. Make sure you have a [Baseten account](https://app.baseten.co/signup) and [API key](https://app.baseten.co/settings/account/api_keys). +2. Install the latest version of Truss: `pip install --upgrade truss` + + +First, clone this repository: +```sh +git clone https://github.com/basetenlabs/truss-examples.git +cd 11-embeddings-reranker-classification-tensorrt/BISV2-qwen-qwen3-32b-fp4-mlp-only +``` + +With `11-embeddings-reranker-classification-tensorrt/BISV2-qwen-qwen3-32b-fp4-mlp-only` as your working directory, you can deploy the model with the following command. Paste your Baseten API key if prompted. + +```sh +truss push --publish +# prints: +# ✨ Model BISV2-qwen-qwen3-32b-fp4-mlp-only-truss-example was successfully pushed ✨ +# 🪵 View logs for your deployment at https://app.baseten.co/models/yyyyyy/logs/xxxxxx +``` + +## Call your model + +### OpenAI compatible inference +This solution is OpenAI compatible, which means you can use the OpenAI client library to interact with the model. + +```python +from openai import OpenAI +import os + +client = OpenAI( + api_key=os.environ['BASETEN_API_KEY'], + base_url="https://model-xxxxxx.api.baseten.co/environments/production/sync/v1" +) + +# Default completion +response_completion = client.completions.create( + model="not_required", + prompt="Q: Tell me everything about Baseten.co! A:", + temperature=0.3, + max_tokens=100, +) + +# Chat completion +response_chat = client.chat.completions.create( + model="", + messages=[ + {"role": "user", "content": "Tell me everything about Baseten.co!"} + ], + temperature=0.3, + max_tokens=100, +) + +# Structured output +from pydantic import BaseModel + +class CalendarEvent(BaseModel): + name: str + date: str + participants: list[str] + +completion = client.beta.chat.completions.parse( + model="not_required", + messages=[ + {"role": "system", "content": "Extract the event information."}, + {"role": "user", "content": "Alice and Bob are going to a science fair on Friday."}, + ], + response_format=CalendarEvent, +) + +event = completion.choices[0].message.parsed + +# If you model supports tool-calling, you can use the following example: +tools = [{ + "type": "function", + "function": { + "name": "get_weather", + "description": "Get current temperature for a given location.", + "parameters": { + "type": "object", + "properties": { + "location": { + "type": "string", + "description": "City and country e.g. Bogotá, Colombia" + } + }, + "required": [ + "location" + ], + "additionalProperties": False + }, + "strict": True + } +}] + +completion = client.chat.completions.create( + model="not_required", + messages=[{"role": "user", "content": "What is the weather like in Paris today?"}], + tools=tools +) + +print(completion.choices[0].message.tool_calls) +``` + + +## Config.yaml +By default, the following configuration is used for this deployment. This config uses `quantization_type=fp4_mlp_only`. This is optional, remove the `quantization_type` field or set it to `no_quant` for float16/bfloat16. + +```yaml +model_metadata: + example_model_input: + max_tokens: 512 + messages: + - content: Tell me everything you know about optimized inference. + role: user + stream: true + temperature: 0.5 + tags: + - openai-compatible +model_name: BISV2-qwen-qwen3-32b-fp4-mlp-only-truss-example +python_version: py39 +resources: + accelerator: B200 + cpu: '1' + memory: 10Gi + use_gpu: true +trt_llm: + build: + checkpoint_repository: + repo: Qwen/Qwen3-32B + revision: main + source: HF + quantization_config: + calib_dataset: baseten/quant_calibration_dataset_v1 + quantization_type: fp4_mlp_only + runtime: + max_batch_size: 32 + max_num_tokens: 32768 + max_seq_len: 32768 + +``` + +## Support +If you have any questions or need assistance, please open an issue in this repository or contact our support team. diff --git a/11-embeddings-reranker-classification-tensorrt/BISV2-qwen-qwen3-32b-fp4-mlp-only/config.yaml b/11-embeddings-reranker-classification-tensorrt/BISV2-qwen-qwen3-32b-fp4-mlp-only/config.yaml new file mode 100644 index 00000000..b10adb67 --- /dev/null +++ b/11-embeddings-reranker-classification-tensorrt/BISV2-qwen-qwen3-32b-fp4-mlp-only/config.yaml @@ -0,0 +1,31 @@ +model_metadata: + example_model_input: + max_tokens: 512 + messages: + - content: Tell me everything you know about optimized inference. + role: user + stream: true + temperature: 0.5 + tags: + - openai-compatible +model_name: BISV2-qwen-qwen3-32b-fp4-mlp-only-truss-example +python_version: py39 +resources: + accelerator: B200 + cpu: '1' + memory: 10Gi + use_gpu: true +trt_llm: + inference_stack: v2 + build: + checkpoint_repository: + repo: Qwen/Qwen3-32B + revision: main + source: HF + quantization_config: + calib_dataset: baseten/quant_calibration_dataset_v1 + quantization_type: fp4_mlp_only + runtime: + max_batch_size: 32 + max_num_tokens: 32768 + max_seq_len: 32768 diff --git a/11-embeddings-reranker-classification-tensorrt/BISV2-qwen-qwen3-4b-calib-dataset-fp8/README.md b/11-embeddings-reranker-classification-tensorrt/BISV2-qwen-qwen3-4b-calib-dataset-fp8/README.md new file mode 100644 index 00000000..41e8cf0a --- /dev/null +++ b/11-embeddings-reranker-classification-tensorrt/BISV2-qwen-qwen3-4b-calib-dataset-fp8/README.md @@ -0,0 +1,171 @@ +# Baseten Inference Stack with Qwen/Qwen3-4B-calib-dataset + +This is a Deployment for Baseten Inference Stack with Qwen/Qwen3-4B-calib-dataset. Baseten Inference Stack is Baseten's solution for production-grade deployments via TensorRT-LLM for Causal Language Models models. (e.g. LLama, Qwen, Mistral) + +With Baseten Inference Stack you get the following benefits by default: +- *Lowest-latency* latency, beating frameworks such as vllm +- *Highest-throughput* inference, automatically using XQA kernels, paged kv caching and inflight batching. +- *distributed inference* run large models (such as LLama-405B) tensor-parallel +- *json-schema based structured output for any model* +- *chunked prefilling* for long generation tasks + +Optionally, you can also enable: +- *speculative decoding* using an external draft model or self-speculative decoding +- *fp8 quantization* deployments on H100, H200 and L4 GPUs +- *fp4 quantization* deployments on B200 GPUs to get even more speed + + +# Examples: +This deployment is specifically designed for the Hugging Face model [Qwen/Qwen3-4B](https://huggingface.co/Qwen/Qwen3-4B). +Suitable models can be identified by the `ForCausalLM` suffix in the model name. Currently we support e.g. LLama, Qwen, Mistral models. + +Qwen/Qwen3-4B is a text-generation model, used to generate text given a prompt. \nIt is frequently used in chatbots, text completion, structured output and more. + +This model is quantized to FP8 for deployment, which is supported by Nvidia's newest GPUs e.g. H100, H100_40GB or L4. Quantization is optional, but leads to higher efficiency. + +## Deployment with Truss + +Before deployment: + +1. Make sure you have a [Baseten account](https://app.baseten.co/signup) and [API key](https://app.baseten.co/settings/account/api_keys). +2. Install the latest version of Truss: `pip install --upgrade truss` + + +First, clone this repository: +```sh +git clone https://github.com/basetenlabs/truss-examples.git +cd 11-embeddings-reranker-classification-tensorrt/BISV2-qwen-qwen3-4b-calib-dataset-fp8 +``` + +With `11-embeddings-reranker-classification-tensorrt/BISV2-qwen-qwen3-4b-calib-dataset-fp8` as your working directory, you can deploy the model with the following command. Paste your Baseten API key if prompted. + +```sh +truss push --publish +# prints: +# ✨ Model BISV2-qwen-qwen3-4b-calib-dataset-fp8-truss-example was successfully pushed ✨ +# 🪵 View logs for your deployment at https://app.baseten.co/models/yyyyyy/logs/xxxxxx +``` + +## Call your model + +### OpenAI compatible inference +This solution is OpenAI compatible, which means you can use the OpenAI client library to interact with the model. + +```python +from openai import OpenAI +import os + +client = OpenAI( + api_key=os.environ['BASETEN_API_KEY'], + base_url="https://model-xxxxxx.api.baseten.co/environments/production/sync/v1" +) + +# Default completion +response_completion = client.completions.create( + model="not_required", + prompt="Q: Tell me everything about Baseten.co! A:", + temperature=0.3, + max_tokens=100, +) + +# Chat completion +response_chat = client.chat.completions.create( + model="", + messages=[ + {"role": "user", "content": "Tell me everything about Baseten.co!"} + ], + temperature=0.3, + max_tokens=100, +) + +# Structured output +from pydantic import BaseModel + +class CalendarEvent(BaseModel): + name: str + date: str + participants: list[str] + +completion = client.beta.chat.completions.parse( + model="not_required", + messages=[ + {"role": "system", "content": "Extract the event information."}, + {"role": "user", "content": "Alice and Bob are going to a science fair on Friday."}, + ], + response_format=CalendarEvent, +) + +event = completion.choices[0].message.parsed + +# If you model supports tool-calling, you can use the following example: +tools = [{ + "type": "function", + "function": { + "name": "get_weather", + "description": "Get current temperature for a given location.", + "parameters": { + "type": "object", + "properties": { + "location": { + "type": "string", + "description": "City and country e.g. Bogotá, Colombia" + } + }, + "required": [ + "location" + ], + "additionalProperties": False + }, + "strict": True + } +}] + +completion = client.chat.completions.create( + model="not_required", + messages=[{"role": "user", "content": "What is the weather like in Paris today?"}], + tools=tools +) + +print(completion.choices[0].message.tool_calls) +``` + + +## Config.yaml +By default, the following configuration is used for this deployment. This config uses `quantization_type=fp8_kv`. This is optional, remove the `quantization_type` field or set it to `no_quant` for float16/bfloat16. + +```yaml +model_metadata: + example_model_input: + max_tokens: 512 + messages: + - content: Tell me everything you know about optimized inference. + role: user + stream: true + temperature: 0.5 + tags: + - openai-compatible +model_name: BISV2-qwen-qwen3-4b-calib-dataset-fp8-truss-example +python_version: py39 +resources: + accelerator: H100 + cpu: '1' + memory: 10Gi + use_gpu: true +trt_llm: + build: + checkpoint_repository: + repo: Qwen/Qwen3-4B + revision: main + source: HF + quantization_config: + calib_dataset: baseten/quant_calibration_dataset_v1 + quantization_type: fp8_kv + runtime: + max_batch_size: 32 + max_num_tokens: 32768 + max_seq_len: 32768 + +``` + +## Support +If you have any questions or need assistance, please open an issue in this repository or contact our support team. diff --git a/11-embeddings-reranker-classification-tensorrt/BISV2-qwen-qwen3-4b-calib-dataset-fp8/config.yaml b/11-embeddings-reranker-classification-tensorrt/BISV2-qwen-qwen3-4b-calib-dataset-fp8/config.yaml new file mode 100644 index 00000000..f01ee910 --- /dev/null +++ b/11-embeddings-reranker-classification-tensorrt/BISV2-qwen-qwen3-4b-calib-dataset-fp8/config.yaml @@ -0,0 +1,31 @@ +model_metadata: + example_model_input: + max_tokens: 512 + messages: + - content: Tell me everything you know about optimized inference. + role: user + stream: true + temperature: 0.5 + tags: + - openai-compatible +model_name: BISV2-qwen-qwen3-4b-calib-dataset-fp8-truss-example +python_version: py39 +resources: + accelerator: H100 + cpu: '1' + memory: 10Gi + use_gpu: true +trt_llm: + inference_stack: v2 + build: + checkpoint_repository: + repo: Qwen/Qwen3-4B + revision: main + source: HF + quantization_config: + calib_dataset: baseten/quant_calibration_dataset_v1 + quantization_type: fp8_kv + runtime: + max_batch_size: 32 + max_num_tokens: 32768 + max_seq_len: 32768 diff --git a/11-embeddings-reranker-classification-tensorrt/README.md b/11-embeddings-reranker-classification-tensorrt/README.md index 28e231cf..b2e9ce43 100644 --- a/11-embeddings-reranker-classification-tensorrt/README.md +++ b/11-embeddings-reranker-classification-tensorrt/README.md @@ -103,7 +103,9 @@ Examples: - [Qwen/Qwen3-32B-Briton](https://github.com/basetenlabs/truss-examples/tree/main/11-embeddings-reranker-classification-tensorrt/Briton-qwen-qwen3-32b-fp4) - [Qwen/Qwen3-32B-Briton](https://github.com/basetenlabs/truss-examples/tree/main/11-embeddings-reranker-classification-tensorrt/Briton-qwen-qwen3-32b-fp4-mlp-only) - [Qwen/Qwen3-32B-BISV2](https://github.com/basetenlabs/truss-examples/tree/main/11-embeddings-reranker-classification-tensorrt/BISV2-qwen-qwen3-32b-fp4) + - [Qwen/Qwen3-32B-BISV2](https://github.com/basetenlabs/truss-examples/tree/main/11-embeddings-reranker-classification-tensorrt/BISV2-qwen-qwen3-32b-fp4-mlp-only) - [Qwen/Qwen3-4B-BISV2](https://github.com/basetenlabs/truss-examples/tree/main/11-embeddings-reranker-classification-tensorrt/BISV2-qwen-qwen3-4b-fp8) + - [Qwen/Qwen3-4B-calib-dataset-BISV2](https://github.com/basetenlabs/truss-examples/tree/main/11-embeddings-reranker-classification-tensorrt/BISV2-qwen-qwen3-4b-calib-dataset-fp8) - [Qwen/Qwen3-8B-min-latency-Briton](https://github.com/basetenlabs/truss-examples/tree/main/11-embeddings-reranker-classification-tensorrt/Briton-qwen-qwen3-8b-min-latency-fp8) - [deepseek-ai/DeepSeek-R1-Distill-Llama-70B-Briton](https://github.com/basetenlabs/truss-examples/tree/main/11-embeddings-reranker-classification-tensorrt/Briton-deepseek-ai-deepseek-r1-distill-llama-70b-fp8) - [deepseek-ai/DeepSeek-R1-Distill-Llama-70B-BISV2](https://github.com/basetenlabs/truss-examples/tree/main/11-embeddings-reranker-classification-tensorrt/BISV2-deepseek-ai-deepseek-r1-distill-llama-70b-fp4) @@ -119,10 +121,12 @@ Examples: - [meta-llama/Llama-3.2-3B-Instruct-Briton](https://github.com/basetenlabs/truss-examples/tree/main/11-embeddings-reranker-classification-tensorrt/Briton-meta-llama-llama-3.2-3b-instruct-fp8) - [meta-llama/Llama-3.2-3B-Instruct-BISV2](https://github.com/basetenlabs/truss-examples/tree/main/11-embeddings-reranker-classification-tensorrt/BISV2-meta-llama-llama-3.2-3b-instruct-fp8) - [meta-llama/Llama-3.2-3B-Instruct-BISV2](https://github.com/basetenlabs/truss-examples/tree/main/11-embeddings-reranker-classification-tensorrt/BISV2-meta-llama-llama-3.2-3b-instruct-fp4-mlp-only) + - [meta-llama/Llama-3.2-3B-Instruct-calib-dataset-BISV2](https://github.com/basetenlabs/truss-examples/tree/main/11-embeddings-reranker-classification-tensorrt/BISV2-meta-llama-llama-3.2-3b-instruct-calib-dataset-fp8) - [meta-llama/Llama-3.2-3B-Instruct-calib-dataset-Briton](https://github.com/basetenlabs/truss-examples/tree/main/11-embeddings-reranker-classification-tensorrt/Briton-meta-llama-llama-3.2-3b-instruct-calib-dataset-fp8) - [meta-llama/Llama-3.3-70B-Instruct-Briton](https://github.com/basetenlabs/truss-examples/tree/main/11-embeddings-reranker-classification-tensorrt/Briton-meta-llama-llama-3.3-70b-instruct-fp8) - [meta-llama/Llama-3.3-70B-Instruct-Briton](https://github.com/basetenlabs/truss-examples/tree/main/11-embeddings-reranker-classification-tensorrt/Briton-meta-llama-llama-3.3-70b-instruct-fp4) - [meta-llama/Llama-3.3-70B-Instruct-BISV2](https://github.com/basetenlabs/truss-examples/tree/main/11-embeddings-reranker-classification-tensorrt/BISV2-meta-llama-llama-3.3-70b-instruct-fp4) + - [meta-llama/Llama-3.3-70B-Instruct-calib-dataset-BISV2](https://github.com/basetenlabs/truss-examples/tree/main/11-embeddings-reranker-classification-tensorrt/BISV2-meta-llama-llama-3.3-70b-instruct-calib-dataset-fp4) - [meta-llama/Llama-3.3-70B-Instruct-tp4-Briton](https://github.com/basetenlabs/truss-examples/tree/main/11-embeddings-reranker-classification-tensorrt/Briton-meta-llama-llama-3.3-70b-instruct-tp4-fp8) - [microsoft/phi-4-Briton](https://github.com/basetenlabs/truss-examples/tree/main/11-embeddings-reranker-classification-tensorrt/Briton-microsoft-phi-4-fp8) - [mistralai/Mistral-7B-Instruct-v0.3-Briton](https://github.com/basetenlabs/truss-examples/tree/main/11-embeddings-reranker-classification-tensorrt/Briton-mistralai-mistral-7b-instruct-v0.3) diff --git a/11-embeddings-reranker-classification-tensorrt/templating/generate_templates.py b/11-embeddings-reranker-classification-tensorrt/templating/generate_templates.py index da131c02..d5c8bd68 100644 --- a/11-embeddings-reranker-classification-tensorrt/templating/generate_templates.py +++ b/11-embeddings-reranker-classification-tensorrt/templating/generate_templates.py @@ -1559,6 +1559,7 @@ def llamalike_config_v2( repoid="meta-llama/Llama-3.3-70B-Instruct", max_batch_size: int = 32, calib_size: Optional[int] = None, + calib_dataset: str = None, ): # config for meta-llama/Llama-3.3-70B-Instruct (FP8) build_kwargs = dict() @@ -1567,6 +1568,11 @@ def llamalike_config_v2( if calib_size is not None: build_kwargs["quantization_config"] = dict(calib_size=calib_size) + if calib_dataset is not None: + if "quantization_config" not in build_kwargs: + build_kwargs["quantization_config"] = dict() + build_kwargs["quantization_config"].update(dict(calib_dataset=calib_dataset)) + config = TRTLLMConfigurationV2( build=TrussTRTLLMBuildConfiguration( checkpoint_repository=CheckpointRepository( @@ -1629,6 +1635,19 @@ def llamalike_config_v2( ) ), ), + Deployment( + "meta-llama/Llama-3.3-70B-Instruct-calib-dataset", + "meta-llama/Llama-3.3-70B-Instruct", + Accelerator.B200, + TextGen(), + solution=BISV2( + trt_config=llamalike_config_v2( + repoid="meta-llama/Llama-3.3-70B-Instruct", + quant=TrussTRTLLMQuantizationType.FP4, + calib_dataset="baseten/quant_calibration_dataset_v1", + ) + ), + ), Deployment( "meta-llama/Llama-3.3-70B-Instruct-tp4", "meta-llama/Llama-3.3-70B-Instruct", @@ -1666,6 +1685,19 @@ def llamalike_config_v2( ) ), ), + Deployment( + "meta-llama/Llama-3.2-3B-Instruct-calib-dataset", + "meta-llama/Llama-3.2-3B-Instruct", + Accelerator.H100_40GB, + TextGen(), + solution=BISV2( + trt_config=llamalike_config_v2( + repoid="meta-llama/Llama-3.2-3B-Instruct", + quant=TrussTRTLLMQuantizationType.FP8_KV, + calib_dataset="baseten/quant_calibration_dataset_v1", + ) + ), + ), Deployment( "meta-llama/Llama-3.2-3B-Instruct-calib-dataset", "meta-llama/Llama-3.2-3B-Instruct", @@ -1868,6 +1900,19 @@ def llamalike_config_v2( ) ), ), + Deployment( + "Qwen/Qwen3-32B", + "Qwen/Qwen3-32B", + Accelerator.B200, + TextGen(), + solution=BISV2( + trt_config=llamalike_config_v2( + repoid="Qwen/Qwen3-32B", + quant=TrussTRTLLMQuantizationType.FP4_MLP_ONLY, + calib_dataset="baseten/quant_calibration_dataset_v1", + ) + ), + ), Deployment( "Qwen/Qwen3-4B", "Qwen/Qwen3-4B", @@ -1880,6 +1925,19 @@ def llamalike_config_v2( ) ), ), + Deployment( + "Qwen/Qwen3-4B-calib-dataset", + "Qwen/Qwen3-4B", + Accelerator.H100, + TextGen(), + solution=BISV2( + trt_config=llamalike_config_v2( + repoid="Qwen/Qwen3-4B", + quant=TrussTRTLLMQuantizationType.FP8_KV, + calib_dataset="baseten/quant_calibration_dataset_v1", + ) + ), + ), Deployment( "nvidia/Qwen3-8B-FP4", "nvidia/Qwen3-8B-FP4",