C++ Inference Code by Using ONNX model

[shubhammore1310]

Hi Team, I am trying to get the timestamps with BFA in C++. I have converted the checkpoint model to ONNX model and taking output with ONNX runtime (ort). I am currently facing issue to convert the ViterbiDecoder class and its method in C++. I am attaching the Python file i am referring for inference, C++ code made so far and the exported ONNX model.

bfaonnx.py

main.cpp

[shubhammore1310]

Following Python code I used to export the checkpoint to ONNX model

`import torch
import torch.nn as nn
from bfaSingle import ContextFreePhonemeRecognizer # Import your classes

class CUPEPredictONNX(nn.Module):
def init(self, cupe_ckpt_path):
super().init()
self.model = ContextFreePhonemeRecognizer()

    # Load config and build model exactly as in CUPEEmbeddingsExtractor
    checkpoint = torch.load(cupe_ckpt_path, map_location='cpu', weights_only=True)
    if 'model_config' not in checkpoint:
        raise ValueError("Model config not found in checkpoint")
    
    self.model.load_config_state_dict(checkpoint['model_config'])
    self.model.make_model()
    
    # Load weights exactly as in CUPEEmbeddingsExtractor
    state_dict = checkpoint['state_dict']
    state_dict = {k.replace('model.', ''): v for k, v in state_dict.items()}
    # Filter valid keys (same as original)
    valid_prefixes = [
        'feature_extractor.', 'freq_attention.', 'temporal_stream.', 
        'spectral_stream.', 'fusion.', 'window_processor.', 
        'classifier.', 'group_classifier.'
    ]
    state_dict = {k: v for k, v in state_dict.items() 
                 if any(prefix in k for prefix in valid_prefixes)
                 and not any(skip in k for skip in ['quantizer.', 'prediction_head.', 'final_proj.'])}
    
    self.model.load_state_dict(state_dict, strict=False)
    self.model.eval()

def forward(self, audio_batch):
    """Exact replica of CUPEEmbeddingsExtractor.predict() without training noise/groups_only"""
    # Input: [batch_size, wav_length]
    x = audio_batch.unsqueeze(1)  # [B, 1, T]
    
    # Feature extraction
    features = self.model.feature_extractor(x)
    att = self.model.freq_attention(features)
    features = features * att
    
    # Dual streams
    temporal = self.model.temporal_stream(features)
    spectral = self.model.spectral_stream(features)
    
    # Fusion
    fused = torch.cat([temporal, spectral], dim=1)
    fused = self.model.fusion(fused)
    fused = fused.transpose(1, 2)  # [B, frames, dim]
    
    # Transformer + classifiers
    embeddings = self.model.window_processor(fused)
    logits_class = self.model.classifier(embeddings)
    logits_group = self.model.group_classifier(embeddings)
    
    return logits_class, logits_group, embeddings

Export

def export_to_onnx(cupe_ckpt_path, output_path="cupe_predict_english.onnx", dummy_wav_len=1891):
model = CUPEPredictONNX(cupe_ckpt_path)
device = next(model.parameters()).device
model.eval()

# Dummy input matching your example
dummy_input = torch.randn(2, dummy_wav_len, dtype=torch.float32).to(device)

torch.onnx.export(
    model,
    dummy_input,
    output_path,
    opset_version=18,
    do_constant_folding=True,
    input_names=['audio'],
    output_names=['logits_class', 'logits_group', 'embeddings'],
    dynamic_axes={
        'audio': {0: 'batch_size', 1: 'wav_length'},  # Variable audio length
        'logits_class': {0: 'batch_size', 1: 'frames'},
        'logits_group': {0: 'batch_size', 1: 'frames'},
        'embeddings': {0: 'batch_size', 1: 'frames'}
    },
    verbose=False
)
print(f"Exported {output_path}")
return output_path

Usage

ckpt_path = "en_libri1000_ua01c_e4_val_GER3D0.2186.ckpt"
onnx_path = export_to_onnx(ckpt_path)
`

[tabahi]

Thanks for the code. I have tried to improve it further.

Try the latest C++ port in
bournemouth_aligner/cpp_onix/main.cpp

There is also a compiled binary for linux with CUDA12 at ./bournemouth_aligner/cpp_onix/compiled_binary_x64_linux

You will need to set paths

int main()
{
    init_reverse_lookups();

    // Configuration — update paths to match your setup
    std::string onnx_path = "../../models/en_libri1000_ua01c_e4_val_GER=0.2186.ckpt.onnx";
    std::string audio_path = "../../examples/samples/audio/109867__timkahn__butterfly.wav";

First export the model with bournemouth_aligner/cpp_onix/export_cupe_to_onnx.py

Currently missing the espeak integration. This example has pre-computed phonemes.

[shubhammore1310]

Thanks for the new code, it is working perfectly with proper phoneme level alignment.

Aligned 17 phonemes in 5083 ms:
1: ð ( ð) 280.9 - 297.5 ms
2: ɪ ( ɪ) 347.0 - 363.6 ms
3: s ( s) 380.1 - 396.6 ms
4: ɪ ( ɪ) 512.3 - 528.8 ms
5: z ( z) 545.4 - 561.9 ms
6: t ( t) 677.6 - 694.1 ms
7: ɛ ( ɛ) 776.7 - 793.3 ms
8: s ( s) 859.4 - 875.9 ms
9: t ( t) 925.5 - 958.5 ms
10: ɪ ( ɪ) 1008.1 - 1024.6 ms
11: ŋ ( ŋ) 1041.1 - 1057.7 ms
12: ʌ ( ʌ) 1107.3 - 1123.8 ms
13: v ( v) 1140.3 - 1156.8 ms
14: f ( f) 1189.9 - 1206.4 ms
15: i: ( i:) 1272.5 - 1289.0 ms
16: tʃ (tʃ) 1421.2 - 1437.8 ms
17: ɚ ( ɚ) 1503.9 - 1520.4 ms

But I am using ONNX model I exported. And it is taking time,
Inference time: 5080 ms
Logits class: [124, 10, 67]
Logits group: [124, 10, 17]
Stitched class: 41540, group: 10540
Spectral length: 110
Aligned 17 phonemes in 5083 ms

Can You share the model you exported, or requirement.txt for the environment.

Because for new exported model I got runtime error as,
Windows flattened: 124 x 1920
2026-02-17 11:55:34.646067038 [E:onnxruntime:, sequential_executor.cc:516 ExecuteKernel] Non-zero status code returned while running Reshape node. Name:'node_view' Status Message: /home/tian/Yxh/onnxruntime/onnxruntime/core/providers/cpu/tensor/reshape_helper.h:45 onnxruntime::ReshapeHelper::ReshapeHelper(const onnxruntime::TensorShape&, onnxruntime::TensorShapeVector&, bool) input_shape_size == size was false. The input tensor cannot be reshaped to the requested shape. Input shape:{10,124,1536}, requested shape:{10,1,3,512}

terminate called after throwing an instance of 'Ort::Exception'
what(): Non-zero status code returned while running Reshape node. Name:'node_view' Status Message: /home/tian/Yxh/onnxruntime/onnxruntime/core/providers/cpu/tensor/reshape_helper.h:45 onnxruntime::ReshapeHelper::ReshapeHelper(const onnxruntime::TensorShape&, onnxruntime::TensorShapeVector&, bool) input_shape_size == size was false. The input tensor cannot be reshaped to the requested shape. Input shape:{10,124,1536}, requested shape:{10,1,3,512}

Aborted (core dumped)

[al-zatv]

Is onnx version faster? On CPU/GPU.

[tabahi]

Yes. It's faster on both. But right now it's a work in progress.