diff --git a/Tetrahedral_remeshing/benchmark/Tetrahedral_remeshing/Benchmark_comparison/README.md b/Tetrahedral_remeshing/benchmark/Tetrahedral_remeshing/Benchmark_comparison/README.md
new file mode 100644
index 000000000000..7002ffb59f58
--- /dev/null
+++ b/Tetrahedral_remeshing/benchmark/Tetrahedral_remeshing/Benchmark_comparison/README.md
@@ -0,0 +1,230 @@
+# Benchmark Comparison Tool
+
+This tool compares benchmark results across multiple benchmark runs, providing statistical analysis and visualization of performance metrics, quality measurements, and resource usage data.
+
+## Features
+
+- **Flexible Comparison Modes**: Compare within same executable or across different implementations
+- **Statistical Analysis**: Automatic outlier detection and variance analysis
+- **Interactive Charts**: Plotly-based charts with hover information and filtering
+- **Comprehensive Reports**: HTML reports with detailed metric comparisons
+- **Resource Usage Tracking**: Memory and CPU usage analysis (when available)
+
+## Configuration Options
+
+### Comparison Modes
+
+The tool supports two distinct comparison scenarios controlled by the `group_by_executable` setting:
+
+#### 1. Cross-Executable Comparison (`group_by_executable: false`)
+**Use Case**: Compare different implementations that perform the same operation
+- Compares `benchmark_tetrahedral_remeshing.exe` vs `benchmark_refactored_tetrahedral_remeshing.exe`
+- Groups results by input arguments only (ignoring executable metadata)
+- Perfect for comparing current vs refactored implementations
+- Shows performance differences between algorithm variants
+
+**Example**: Compare the current CGAL implementation against your refactored elementary operations framework
+
+#### 2. Same-Executable Comparison (`group_by_executable: true`)
+**Use Case**: Fine-tune and analyze variations within a single implementation
+- Compares only runs from the same executable
+- Groups results by input arguments AND executable metadata
+- Perfect for performance tuning and consistency analysis
+- Shows performance variations across different runs of the same code
+
+**Example**: Analyze performance consistency of the refactored implementation across multiple runs
+
+#### 3. Ignoring Specific Arguments (`exclude_from_grouping`)
+**Use Case**: Compare runs that differ only in one argument that should not affect grouping — for example, comparing a sequential run (`threads=1`) against a parallel run (`threads=8`) of the same algorithm variant.
+
+By default, two runs are placed in the same comparison row only when **all** input arguments match. Adding argument keys to `exclude_from_grouping` strips those keys before matching, so runs that differ only on those arguments are treated as the same configuration.
+
+```yaml
+analysis:
+ group_by_executable: false
+ exclude_from_grouping: [threads] # sequential and parallel land in the same row
+```
+
+### Basic Configuration
+
+```yaml
+results_root: /path/to/Benchmark_results # Root directory containing benchmark runs
+include_reports: [] # Specific subdirs to include (empty = all)
+
+analysis:
+ significant_change_threshold: 0.0 # Show metrics with >X% change (0 = all)
+ outlier_threshold: 2.0 # Highlight outliers >X std deviations
+ group_by_executable: false # false = cross-executable, true = same-executable
+ exclude_from_grouping: [] # Input argument keys to ignore when matching runs.
+ # e.g. [threads] to compare sequential (threads=1)
+ # vs parallel (threads=8) in the same table row.
+
+report:
+ chart_grid_cols: 2 # Chart grid layout columns
+
+charts:
+ - kind: line
+ y: metrics.Performance.Total_Time.Value
+ x: run_metadata.input_arguments.target_edge_factor
+ title: "Performance vs Edge Factor"
+ style: "o-"
+```
+
+## Usage Examples
+
+### Comparing Current vs Refactored Implementation
+
+1. **Run both benchmarks with identical parameters**:
+ ```yaml
+ # In Benchmark_creation/config.yaml
+ benchmarks:
+ - name: current
+ exec: benchmark_tetrahedral_remeshing.exe
+ sweep:
+ input_mesh: [elephant.mesh]
+ num_iterations: [50]
+ target_edge_factor: [0.25, 0.5, 1.0, 2.0]
+ threads: [1]
+ - name: refactored
+ exec: benchmark_refactored_tetrahedral_remeshing.exe
+ sweep:
+ input_mesh: [elephant.mesh]
+ num_iterations: [50]
+ target_edge_factor: [0.25, 0.5, 1.0, 2.0]
+ threads: [1]
+ ```
+
+2. **Configure comparison for cross-executable analysis**:
+ ```yaml
+ # In Benchmark_comparison/config.yaml
+ analysis:
+ group_by_executable: false # Enable cross-executable comparison
+
+ charts:
+ - kind: line
+ y: metrics.Performance.Total_Time.Value
+ x: run_metadata.input_arguments.target_edge_factor
+ title: "Current vs Refactored: Performance by Edge Factor"
+ style: "o-"
+ ```
+
+3. **Run comparison**:
+ ```bash
+ python compare_benchmark_results.py --config config.yaml
+ ```
+
+### Fine-Tuning Single Implementation
+
+1. **Configure for same-executable analysis**:
+ ```yaml
+ analysis:
+ group_by_executable: true # Compare only within same executable
+ outlier_threshold: 1.0 # Sensitive outlier detection for consistency analysis
+ ```
+
+2. **Run multiple times with same parameters** to analyze consistency
+3. **Analyze variance and outliers** in the generated report
+
+## Chart Types and Filtering
+
+### Time-Series Charts
+```yaml
+- kind: line
+ y: metrics.Performance.Total_Time.Value
+ title: "Performance Over Time"
+ # No x specified = uses timestamp
+```
+
+### Parameter-Based Charts
+```yaml
+- kind: line
+ y: metrics.Performance.Total_Time.Value
+ x: run_metadata.input_arguments.threads
+ title: "Performance vs Thread Count"
+```
+
+### Filtered Charts
+```yaml
+- kind: line
+ y: metrics.Performance.Total_Time.Value
+ x: run_metadata.input_arguments.target_edge_factor
+ title: "Performance for Large Meshes Only"
+ filter:
+ run_metadata.run_info.Mesh.Name: elephant
+```
+
+## Understanding Results
+
+### Metric Comparison Tables
+- **Green cells**: Low variance (CV < 5%)
+- **Yellow cells**: Medium variance (5% ≤ CV ≤ 20%)
+- **Red cells**: Outliers (beyond threshold standard deviations)
+- **Hover tooltips**: Detailed statistics (mean, std dev, percentiles)
+
+### Interactive Charts
+- **Series**: Automatically grouped by benchmark + input parameters
+- **Hover information**: Shows benchmark name, input arguments, and metric values
+- **Legend**: Distinguishes between different implementations or parameter sets
+- **Responsive**: Adapts to different screen sizes
+
+### Statistical Analysis
+- **Coefficient of Variation (CV%)**: Relative variability measure
+- **Outlier Detection**: Z-score based identification
+- **Percentile Analysis**: Min, Q1, Median, Q3, Max values
+- **Cross-Implementation Comparison**: When enabled, shows both implementations in same tables
+
+## File Structure
+
+```
+Benchmark_results/
+├── 2024-01-15_10-30-00/ # Timestamp-based run directories
+│ └── pipeline_results.json
+├── 2024-01-15_11-45-00/
+│ └── pipeline_results.json
+└── benchmark_comparison_report.html # Generated report
+```
+
+## Troubleshooting
+
+### No Comparison Data Available
+**Problem**: Report shows "No comparison data available"
+**Solutions**:
+- Ensure both executables use identical input parameters
+- Check that `group_by_executable` setting matches your comparison intent
+- Verify that `pipeline_results.json` files contain both implementations
+
+### Charts Not Showing Expected Comparisons
+**Problem**: Charts don't show both implementations
+**Solutions**:
+- Set `group_by_executable: false` for cross-executable comparison
+- Ensure both executables have runs with identical input arguments
+- Check chart filters aren't excluding one implementation
+
+### High Variance Warnings
+**Problem**: Many metrics marked as high variance
+**Solutions**:
+- Run more iterations to reduce natural variance
+- Check for system interference during benchmarking
+- Consider if differences are due to algorithmic improvements vs measurement noise
+
+## Advanced Features
+
+### Custom Grouping
+The tool automatically handles different executable metadata when `group_by_executable: false`, allowing meaningful comparison between:
+- Different compilation flags
+- Different algorithm implementations
+- Different versions of the same code
+
+### Resource Usage Integration
+When resource monitoring is enabled, the tool automatically includes:
+- Peak memory usage
+- Average CPU utilization
+- Memory allocation patterns
+- All integrated into the same comparison framework
+
+### Extensible Chart System
+Charts support various customizations:
+- Multiple metric types (performance, quality, resource usage)
+- Flexible filtering by any result field
+- Multiple chart types (line, scatter, etc.)
+- Custom styling and layout options
\ No newline at end of file
diff --git a/Tetrahedral_remeshing/benchmark/Tetrahedral_remeshing/Benchmark_comparison/benchmark_comparison_config.yaml b/Tetrahedral_remeshing/benchmark/Tetrahedral_remeshing/Benchmark_comparison/benchmark_comparison_config.yaml
new file mode 100644
index 000000000000..e875940cc9db
--- /dev/null
+++ b/Tetrahedral_remeshing/benchmark/Tetrahedral_remeshing/Benchmark_comparison/benchmark_comparison_config.yaml
@@ -0,0 +1,23 @@
+results_root: C:/Users/iason.manolas/Documents/learning/gsoc2025-Tetra_remeshing_parallel-imanolas/Tetrahedral_remeshing/build_debug_tetrahedral_remeshing/BenchmarkResults # Folder containing multiple benchmark run folders
+include_reports: [] # Optional: list of subfolders to include, or leave empty for all
+# metrics:
+ # - metrics.Performance.Total_Time.Value
+ # - metrics.Quality.Edge_Length.Mean
+group_by: [] # (Unused, now auto-detected)
+significant_change_threshold: 0.0 # Only show metrics with >1% change (set to 0 to show all)
+outlier_threshold: 3.0 # Highlight changes >3x stddev as outliers
+
+# Chart grid configuration
+report_chart_grid_cols: 2 # Number of columns in the chart grid (charts will be scaled accordingly)
+
+charts:
+ - kind: line
+ y: Performance.Total_Time.Value
+ title: "Total Time across versions"
+ save_as: "total_time_timeseries.png"
+ style: "o-"
+ - kind: line
+ y: Performance.Memory.Value
+ title: "Memory across versions"
+ save_as: "memory_timeseries.png"
+ style: "o-"
\ No newline at end of file
diff --git a/Tetrahedral_remeshing/benchmark/Tetrahedral_remeshing/Benchmark_comparison/compare_benchmark_results.py b/Tetrahedral_remeshing/benchmark/Tetrahedral_remeshing/Benchmark_comparison/compare_benchmark_results.py
new file mode 100644
index 000000000000..89c3b0bcac2b
--- /dev/null
+++ b/Tetrahedral_remeshing/benchmark/Tetrahedral_remeshing/Benchmark_comparison/compare_benchmark_results.py
@@ -0,0 +1,1168 @@
+import os
+import glob
+import yaml
+import json
+import numpy as np
+from jinja2 import Environment
+import plotly.graph_objects as go
+import plotly.express as px
+from plotly.offline import plot
+import plotly.io as pio
+import hashlib
+import io
+import sys
+import argparse
+from datetime import datetime
+import statistics
+
+def validate_config(config):
+ """Validate the configuration file."""
+ if not isinstance(config, dict):
+ raise ValueError("Config must be a dictionary")
+
+ required_keys = ['results_root']
+ for key in required_keys:
+ if key not in config:
+ raise ValueError(f"Missing required config key: {key}")
+
+ return True
+
+def load_metadata(report_dir):
+ path = os.path.join(report_dir, "environment_metadata.json")
+ if not os.path.isfile(path):
+ return {}
+ with open(path, "r") as f:
+ return json.load(f)
+
+def collect_metric_paths(d, prefix=()):
+ """Recursively collect all metric paths and their values from a nested dict."""
+ paths = {}
+ for k, v in d.items():
+ if isinstance(v, dict):
+ paths.update(collect_metric_paths(v, prefix + (k,)))
+ else:
+ paths[prefix + (k,)] = v
+ return paths
+
+def load_pipeline_results(results_dir, group_by_executable=True, exclude_from_grouping=None):
+ run_map = {}
+ # First check for pipeline_results.json in the top level directory
+ pipeline_results_file = os.path.join(results_dir, "pipeline_results.json")
+ if os.path.isfile(pipeline_results_file):
+ with open(pipeline_results_file, "r") as f:
+ results_list = json.load(f)
+ # Handle list of results
+ for data in results_list:
+ # ==================================================================
+ # CONFIGURABLE: Include or exclude exec_metadata in similarity comparison
+ # - group_by_executable=True: Only compare runs with same exec_metadata
+ # (same executable, compile flags, optimizations, etc.)
+ # - group_by_executable=False: Compare runs across different executables
+ # (useful for comparing different implementations)
+ # ==================================================================
+ input_args = data.get("run_metadata", {}).get("input_arguments", {})
+ exec_metadata = data.get("run_metadata", {}).get("exec_metadata", {})
+ preprocessor_macros = data.get("run_metadata", {}).get("preprocessor_macros", [])
+
+ # Create comparable args from all input arguments
+ excluded = set(exclude_from_grouping) if exclude_from_grouping else set()
+ if input_args:
+ processed_items = []
+ for key, value in input_args.items():
+ if key in excluded:
+ continue
+ # Process values to extract filename stems for file paths
+ if isinstance(value, str) and ('/' in value or '\\' in value or '.' in value):
+ # Likely a file path - extract stem (filename without extension)
+ filename = os.path.basename(value)
+ stem = os.path.splitext(filename)[0]
+ processed_items.append((key, stem))
+ else:
+ processed_items.append((key, value))
+
+ comparable_args = dict(processed_items)
+ else:
+ comparable_args = {}
+
+ # Normalize preprocessor macros for comparison (sort to ensure consistent ordering)
+ comparable_macros = sorted(preprocessor_macros) if preprocessor_macros else []
+
+ # Create run key - conditionally include exec_metadata based on config
+ if group_by_executable:
+ # Include exec_metadata - separate groups for different executables
+ run_key_data = {
+ 'input_args': comparable_args,
+ 'exec_metadata': exec_metadata,
+ 'preprocessor_macros': comparable_macros
+ }
+ else:
+ # Exclude exec_metadata - allows comparison across different executables
+ run_key_data = {
+ 'input_args': comparable_args,
+ 'preprocessor_macros': comparable_macros
+ }
+
+ run_key_str = str(run_key_data)
+
+ if run_key_str not in run_map:
+ run_map[run_key_str] = []
+
+ # Add benchmark source directory information to the result
+ data["_benchmark_source_dir"] = os.path.basename(results_dir)
+
+ run_map[run_key_str].append(data)
+ return run_map
+
+def compare_metrics(run_metrics_list):
+ """Given a list of result objects (one per report), compare all numeric leaves in their metrics and resource usage."""
+ all_paths = set()
+ for result in run_metrics_list:
+ # Extract metrics
+ metrics = result.get("metrics", {})
+ all_paths.update(collect_metric_paths(metrics).keys())
+
+ # Extract resource usage data
+ resource_usage = result.get("resource_usage", {})
+ if resource_usage:
+ resource_paths = collect_metric_paths(resource_usage, prefix=("resource_usage",))
+ all_paths.update(resource_paths.keys())
+
+ all_paths = sorted(all_paths)
+ results = []
+ for path in all_paths:
+ values = []
+ for result in run_metrics_list:
+ # Try metrics first
+ metrics = result.get("metrics", {})
+ val = collect_metric_paths(metrics).get(path, None)
+
+ # If not found in metrics, try resource usage
+ if val is None and path[0] == "resource_usage":
+ resource_usage = result.get("resource_usage", {})
+ if resource_usage:
+ resource_paths = collect_metric_paths(resource_usage, prefix=("resource_usage",))
+ val = resource_paths.get(path, None)
+
+ values.append(val)
+ results.append((path, values))
+ return results
+
+def safe_extract_value_by_path(data, path):
+ """Safely extract a value from nested dictionary structure using dot notation path."""
+ try:
+ current = data
+ for key in path.split('.'):
+ if isinstance(current, dict) and key in current:
+ current = current[key]
+ else:
+ return None
+ return current
+ except:
+ return None
+
+def normalize_path_value(value):
+ """Normalize file paths to handle different path separators consistently."""
+ if isinstance(value, str) and ('/' in value or '\\' in value):
+ # This looks like a file path - normalize it
+ return os.path.normpath(value).replace('\\', '/')
+ return value
+
+def get_benchmark_id_from_result(result):
+ """Extract benchmark identifier from a result using exec_metadata primarily."""
+ benchmark_id = "Unknown_Benchmark"
+
+ # First try to use benchmark_name if available (most reliable)
+ benchmark_name = result.get("benchmark_name")
+ if benchmark_name:
+ benchmark_id = benchmark_name
+ else:
+ # Try to extract from exec_metadata (different executables = different benchmarks)
+ exec_metadata = result.get("run_metadata", {}).get("exec_metadata", {})
+ if exec_metadata:
+ # Look for executable path or name in exec_metadata
+ exec_path = exec_metadata.get("executable_path", exec_metadata.get("exec_path"))
+ if exec_path:
+ # Extract executable name without extension
+ exec_name = os.path.splitext(os.path.basename(exec_path))[0]
+ benchmark_id = exec_name
+ else:
+ # Fallback to first available exec_metadata field
+ for key, value in exec_metadata.items():
+ if isinstance(value, str) and value:
+ benchmark_id = f"{key}_{value}"[:20] # Truncate for readability
+ break
+
+ # Add git commit info for version differentiation
+ env_metadata = result.get("env_metadata", {})
+ git_commit = env_metadata.get("git_commit")
+ git_dirty = env_metadata.get("git_dirty", False)
+ source_dir = result.get("_benchmark_source_dir", "")
+
+ if git_commit:
+ # Use short git hash (first 8 characters)
+ git_hash = git_commit[:8]
+ if git_dirty:
+ git_hash += "*"
+
+ # Combine benchmark name with git hash
+ if source_dir:
+ benchmark_id = f"{benchmark_id}_{git_hash}_{source_dir}"
+ else:
+ benchmark_id = f"{benchmark_id}_{git_hash}"
+ else:
+ # Fallback to source directory name if available
+ if source_dir and benchmark_id == "Unknown_Benchmark":
+ benchmark_id = source_dir
+ elif source_dir:
+ benchmark_id = f"{benchmark_id}_{source_dir}"
+
+ # If still no identifier, use timestamp
+ if benchmark_id == "Unknown_Benchmark":
+ benchmark_id = env_metadata.get("timestamp", "Unknown_Benchmark")
+
+ # If no env_metadata timestamp, try run_metadata timestamp
+ if benchmark_id == "Unknown_Benchmark":
+ run_timestamp = result.get("run_metadata", {}).get("timestamp", "Unknown")
+ if run_timestamp != "Unknown":
+ # Truncate to minute to group runs from same benchmark execution
+ try:
+ benchmark_id = run_timestamp[:16] # YYYY-MM-DDTHH:MM
+ except:
+ benchmark_id = run_timestamp
+
+ return benchmark_id
+
+def apply_chart_filter(result, chart_filter):
+ """Apply filter conditions to a result. Returns True if result passes all filters."""
+ if not chart_filter:
+ return True
+
+ for filter_path, expected_value in chart_filter.items():
+ actual_value = safe_extract_value_by_path(result, filter_path)
+
+ # Handle different comparison types
+ if actual_value is None:
+ return False
+
+ # Convert to string for comparison to handle different types
+ if str(actual_value) != str(expected_value):
+ return False
+
+ return True
+
+
+
+def generate_charts(comparison_results, config):
+ """Generate unified charts that support both time-series and parameter-based analysis."""
+ charts = config.get("charts", [])
+ if not charts:
+ print("No charts defined in config.")
+ return {}
+
+ chart_files = {}
+
+ # Process each chart configuration
+ for chart_idx, chart_config in enumerate(charts):
+ y_metric = chart_config.get("y")
+ x_param = chart_config.get("x") # If not specified, we'll use timestamp
+ chart_filter = chart_config.get("filter", {})
+ title = chart_config.get("title", f"{y_metric} vs {x_param or 'Time'}")
+ style = chart_config.get("style", "o-")
+
+ if not y_metric:
+ print(f"Warning: Chart {chart_idx} missing required 'y' field. Skipping.")
+ continue
+
+ # If no x parameter specified, use timestamp (time-series chart)
+ if not x_param:
+ x_param = "run_metadata.timestamp"
+
+ print(f"\nGenerating chart: {title}")
+ print(f"Y metric: {y_metric}")
+ print(f"X parameter: {x_param}")
+ if chart_filter:
+ print(f"Filters: {chart_filter}")
+
+ # Collect data points for this chart, grouped by benchmark execution
+ benchmark_data = {}
+ total_results = 0
+ filtered_results = 0
+
+ for run_key, results_list in comparison_results.items():
+ for result in results_list:
+ total_results += 1
+
+ # Apply chart filter first
+ if not apply_chart_filter(result, chart_filter):
+ continue
+
+ filtered_results += 1
+
+ # Extract x value using the unified approach
+ x_value = safe_extract_value_by_path(result, x_param)
+ if x_value is None:
+ continue
+
+ # Extract y value from metrics or resource usage using full path
+ y_value = safe_extract_value_by_path(result, y_metric)
+ if y_value is None or not isinstance(y_value, (int, float)):
+ continue
+
+ # Get benchmark identifier
+ benchmark_id = get_benchmark_id_from_result(result)
+
+ # Universal grouping: use run_key as the base grouping mechanism
+ # Results are already grouped by input_args + exec_metadata
+ data_key = (run_key, benchmark_id)
+
+ if data_key not in benchmark_data:
+ benchmark_data[data_key] = {
+ 'points': [],
+ 'run_key': run_key,
+ 'benchmark_id': benchmark_id
+ }
+
+ benchmark_data[data_key]['points'].append({
+ 'x': x_value,
+ 'y': y_value
+ })
+
+ if chart_filter:
+ print(f"Filter applied: {filtered_results}/{total_results} results passed filter")
+
+ if not benchmark_data:
+ print(f"Warning: No data points found for chart '{title}' after filtering")
+ continue
+
+ # Universal grouping rule: group by input_arguments + exec_metadata,
+ # excluding any variables used as x or y in the chart
+ chart_series = {}
+
+ # Extract variable names from x and y paths that should be excluded from grouping
+ excluded_vars = set()
+
+ # Extract variable from x path (e.g., "run_metadata.input_arguments.threads" -> exclude "threads")
+ if x_param and 'input_arguments' in x_param:
+ x_var_name = x_param.split('.')[-1] # Get the last part (e.g., "threads")
+ excluded_vars.add(x_var_name)
+
+ # Y is typically a metric path, not an input variable, so usually no exclusion needed
+ # But if y_metric contains input_arguments, we should exclude that too
+ if 'input_arguments' in y_metric:
+ y_var_name = y_metric.split('.')[-1]
+ excluded_vars.add(y_var_name)
+
+ for data_key, data_info in benchmark_data.items():
+ run_key = data_info['run_key']
+ benchmark_id = data_info['benchmark_id']
+
+ # Create series ID based on input_args + exec_metadata, excluding x/y variables
+ try:
+ run_key_data = eval(run_key)
+ input_args = run_key_data.get('input_args', {})
+ exec_metadata = run_key_data.get('exec_metadata', {})
+
+ # Filter input_args to exclude x/y variables
+ filtered_input_args = {k: v for k, v in input_args.items() if k not in excluded_vars}
+
+ # Create series identifier from benchmark + filtered args + exec_metadata
+ series_parts = [benchmark_id]
+
+ # Add filtered input arguments
+ if filtered_input_args:
+ args_str = "_".join(f"{k}={v}" for k, v in sorted(filtered_input_args.items()))
+ series_parts.append(args_str)
+
+ # Add exec_metadata (usually distinguishes different executables)
+ if exec_metadata:
+ exec_str = "_".join(f"{k}={v}" for k, v in sorted(exec_metadata.items()))
+ series_parts.append(f"exec_{exec_str}")
+
+ series_id = "_".join(series_parts)
+
+ except Exception as e:
+ # Fallback: use benchmark_id + run_key
+ series_id = f"{benchmark_id}_{run_key}"
+
+ if series_id not in chart_series:
+ chart_series[series_id] = {
+ 'points': [],
+ 'run_key': run_key,
+ 'benchmark_id': benchmark_id,
+ 'label': benchmark_id # Will be cleaned up later
+ }
+
+ chart_series[series_id]['points'].extend(data_info['points'])
+
+ print(f"Found {len(chart_series)} series for this chart")
+
+ if not chart_series:
+ print(f"Warning: No series found for chart '{title}'")
+ continue
+
+ # Set chart title with filters
+ chart_title = title
+ if chart_filter:
+ filter_str = ", ".join(f"{k}={v}" for k, v in chart_filter.items())
+ chart_title = f"{title}\nFiltered by: {filter_str}"
+
+ # Store the chart
+ chart_key = f"chart_{chart_idx}"
+ if chart_key not in chart_files:
+ chart_files[chart_key] = []
+
+ # Create interactive Plotly chart
+ print(f"Creating interactive Plotly chart for '{title}'")
+
+ fig = go.Figure()
+
+ # Add traces for each series
+ for series_id, series_data in chart_series.items():
+ points = series_data['points']
+
+ # Sort points by x value for proper line connection
+ sorted_points = sorted(points, key=lambda p: p['x'])
+ x_vals = [p['x'] for p in sorted_points]
+ y_vals = [p['y'] for p in sorted_points]
+
+ # Create more compact hover text using plotly's customdata approach
+ run_key = series_data['run_key']
+ try:
+ run_key_data = eval(run_key)
+ input_args = run_key_data.get('input_args', {})
+ exec_metadata = run_key_data.get('exec_metadata', {})
+
+ customdata = []
+ for i, (x_val, y_val) in enumerate(zip(x_vals, y_vals)):
+ # Get all input arguments, formatted nicely
+ config_parts = []
+ for key, value in input_args.items():
+ # For file paths, show only the filename or last folder
+ if isinstance(value, str) and ('/' in value or '\\' in value):
+ display_value = os.path.basename(value)
+ else:
+ display_value = str(value)
+
+ config_parts.append(f"{key}:{display_value}")
+
+ # Create config text with intelligent wrapping
+ if config_parts:
+ # If too many parameters, use line breaks for wrapping
+ if len(config_parts) > 3:
+ # Split into groups of 2-3 for better wrapping
+ config_text = ", ".join(config_parts[:3])
+ if len(config_parts) > 3:
+ config_text += "
" + ", ".join(config_parts[3:]) # Add spacing for alignment
+ else:
+ config_text = ", ".join(config_parts)
+ else:
+ config_text = "Default"
+
+ # Store data as list for custom hover template (benchmark, config)
+ customdata.append([
+ series_data['benchmark_id'], # Already contains git hash info
+ config_text
+ ])
+
+ except Exception as e:
+ customdata = [[series_data['benchmark_id'], 'N/A'] for _ in x_vals]
+
+ # Determine marker and line style
+ if 'o' in style:
+ mode = 'lines+markers'
+ marker_size = 8
+ elif '-' in style:
+ mode = 'lines'
+ marker_size = 6
+ else:
+ mode = 'markers'
+ marker_size = 8
+
+ # Use different markers for different series
+ marker_symbol = 'circle'
+ if 'd' in style:
+ marker_symbol = 'diamond'
+ elif 's' in style:
+ marker_symbol = 'square'
+
+ # Create a clean series name for the legend (shorter version)
+ legend_name = series_id.replace('_', ' ').title()
+ if len(legend_name) > 50:
+ legend_name = legend_name[:47] + "..."
+
+ fig.add_trace(go.Scatter(
+ x=x_vals,
+ y=y_vals,
+ mode=mode,
+ name=legend_name,
+ customdata=customdata,
+ hovertemplate=
+ f"X: %{{x}}
" +
+ f"Y: %{{y}}
" +
+ "Benchmark: %{customdata[0]}
" +
+ "Input Arguments: %{customdata[1]}" +
+ "",
+ marker=dict(
+ size=marker_size,
+ symbol=marker_symbol
+ ),
+ line=dict(width=2)
+ ))
+
+ # Update layout for better appearance
+ fig.update_layout(
+ title=chart_title,
+ xaxis_title=x_param,
+ yaxis_title=y_metric,
+ hovermode='closest',
+ height=600,
+ legend=dict(
+ orientation="v",
+ yanchor="top",
+ y=1,
+ xanchor="left",
+ x=1.02,
+ font=dict(size=10) # Smaller font for multi-column layout
+ ),
+ margin=dict(r=180, l=60, t=80, b=60), # Adjusted margins for multi-column
+ template="plotly_white", # Clean white background
+ hoverlabel=dict(
+ bgcolor="rgba(255,255,255,0.95)",
+ bordercolor="rgba(0,0,0,0.3)",
+ font_size=12,
+ font_family="Arial",
+ namelength=-1, # Show full hover text
+ align="left"
+ )
+ )
+
+ # Convert to embeddable HTML div (not full HTML document)
+ div_id = f"plotly-div-{hash(chart_title)}"
+
+ # Create embeddable HTML with plotly CDN - use full width
+ plotly_chart_html = f"""
+
+
+
+ """
+
+ # Debug: Show what the plotly HTML looks like
+ print(f"Plotly HTML starts with: {plotly_chart_html[:100]}...")
+
+ # Store Plotly HTML directly
+ chart_files[chart_key].append(plotly_chart_html)
+ print(f"Successfully created interactive Plotly chart for '{title}' - HTML length: {len(plotly_chart_html)}")
+
+ # Handle file saving if requested
+ save_as = chart_config.get("save_as")
+ if save_as:
+ # Save as complete HTML file
+ html_filename = save_as.replace('.png', '.html').replace('.jpg', '.html').replace('.jpeg', '.html')
+ full_html = f"""
+
+
+
+ {chart_title}
+
+
+
+ {chart_title}
+
+
+
+
+"""
+ with open(html_filename, 'w', encoding='utf-8') as f:
+ f.write(full_html)
+ print(f"Saved interactive chart: {html_filename}")
+
+ return chart_files
+
+def calculate_outliers(comparison_data, outlier_threshold):
+ """
+ Pre-calculate statistical information for each metric row including outliers, variance, etc.
+ Returns a dict with statistical analysis for each value.
+ """
+ outlier_info = {}
+
+ for run_key, metrics in comparison_data.items():
+ outlier_info[run_key] = {}
+
+ for path, values in metrics:
+ # Extract numeric values only
+ numeric_values = [v for v in values if isinstance(v, (int, float)) and v is not None]
+
+
+
+ # Calculate statistics if we have enough data points
+ row_outliers = [False] * len(values) # Default: no outliers
+ variance = 0
+ coefficient_of_variation = 0
+ percentiles = {}
+
+ if len(numeric_values) > 0:
+ try:
+ row_mean = statistics.mean(numeric_values)
+
+ if len(numeric_values) > 1:
+ row_std = statistics.stdev(numeric_values)
+ variance = statistics.variance(numeric_values)
+
+ # Coefficient of variation (CV) - relative variability
+ if row_mean != 0:
+ coefficient_of_variation = (row_std / abs(row_mean)) * 100
+
+ # Calculate percentiles if we have enough data
+ if len(numeric_values) >= 3:
+ sorted_values = sorted(numeric_values)
+ percentiles = {
+ 'p25': statistics.quantiles(sorted_values, n=4)[0] if len(sorted_values) >= 4 else sorted_values[0],
+ 'p50': statistics.median(sorted_values),
+ 'p75': statistics.quantiles(sorted_values, n=4)[2] if len(sorted_values) >= 4 else sorted_values[-1],
+ 'min': min(sorted_values),
+ 'max': max(sorted_values)
+ }
+
+ # Detect outliers only when there is a meaningful relative
+ # spread. Runs that are effectively identical still produce a
+ # std of ~1e-16 from floating-point noise (not exactly 0), and
+ # for n=2 the z-score is a fixed ratio independent of how tiny
+ # the actual difference is - so a bare `row_std > 0` guard would
+ # flag spurious outliers on metrics that don't really differ.
+ rel_std = (row_std / abs(row_mean)) if row_mean != 0 else row_std
+ if row_std > 0 and rel_std > 1e-9:
+ for i, value in enumerate(values):
+ if isinstance(value, (int, float)) and value is not None:
+ z_score = abs(value - row_mean) / row_std
+ row_outliers[i] = z_score > outlier_threshold
+ else:
+ row_std = 0
+ row_mean = numeric_values[0]
+
+ except Exception as e:
+ row_mean = statistics.mean(numeric_values) if numeric_values else 0
+ row_std = 0
+ else:
+ row_mean = 0
+ row_std = 0
+
+ # Store comprehensive statistical info for this path
+ path_key = '.'.join(str(p) for p in path)
+ outlier_info[run_key][path_key] = {
+ 'outliers': row_outliers,
+ 'mean': row_mean,
+ 'std': row_std,
+ 'variance': variance,
+ 'coefficient_of_variation': coefficient_of_variation,
+ 'percentiles': percentiles,
+ 'count': len(numeric_values),
+ 'outlier_count': sum(row_outliers)
+ }
+
+ return outlier_info
+
+def main():
+ parser = argparse.ArgumentParser(description="Compare benchmark results as specified in a YAML config.")
+ parser.add_argument('--config', type=str, default='config.yaml', help='Path to the YAML config file')
+ args = parser.parse_args()
+
+ # Load and validate config
+ try:
+ with open(args.config) as f:
+ config = yaml.safe_load(f)
+ validate_config(config)
+ except Exception as e:
+ print(f"Error loading config: {e}")
+ sys.exit(1)
+
+ results_root = config.get("results_root", "")
+ if not results_root:
+ print("Error: 'results_root' not specified in config file")
+ sys.exit(1)
+
+ if not os.path.exists(results_root):
+ print(f"Error: Results root directory not found: {results_root}")
+ print("Please ensure the results_root path in the config is correct.")
+ sys.exit(1)
+
+ include_reports = config.get("include_reports", [])
+
+ # Get analysis configuration (backward compatibility)
+ analysis_config = config.get("analysis", {})
+ significant_change_threshold = analysis_config.get("significant_change_threshold",
+ config.get("significant_change_threshold", 0.0))
+ outlier_threshold = analysis_config.get("outlier_threshold",
+ config.get("outlier_threshold", 3.0))
+ group_by_executable = analysis_config.get("group_by_executable",
+ config.get("group_by_executable", True))
+ exclude_from_grouping = analysis_config.get("exclude_from_grouping", [])
+
+ # Get report configuration (backward compatibility)
+ report_config = config.get("report", {})
+ chart_grid_cols = report_config.get("chart_grid_cols",
+ config.get("report_chart_grid_cols", 2))
+
+ charts = config.get("charts", [])
+
+ # Find all report directories
+ if include_reports:
+ report_dirs = [os.path.join(results_root, d) for d in include_reports if os.path.isdir(os.path.join(results_root, d))]
+ if not report_dirs:
+ print(f"Warning: None of the specified report directories found in {results_root}")
+ print(f"Specified directories: {include_reports}")
+ else:
+ try:
+ all_dirs = [d for d in os.listdir(results_root) if os.path.isdir(os.path.join(results_root, d))]
+ report_dirs = [os.path.join(results_root, d) for d in all_dirs]
+ if not report_dirs:
+ print(f"Warning: No subdirectories found in {results_root}")
+ except OSError as e:
+ print(f"Error accessing results directory: {e}")
+ sys.exit(1)
+
+ print(f"Found {len(report_dirs)} report directories to process:")
+ for report_dir in report_dirs:
+ print(f" - {os.path.basename(report_dir)}")
+
+ print(f"\nComparison mode: {'Same executable only' if group_by_executable else 'Cross-executable comparison'}")
+
+ # Load pipeline results and metadata for each report
+ report_metrics = {}
+ report_timestamps = {}
+ report_input_args = {}
+
+ for report_dir in report_dirs:
+ print(f"\nProcessing: {os.path.basename(report_dir)}")
+ run_key_to_metrics_map = load_pipeline_results(report_dir, group_by_executable, exclude_from_grouping)
+ if run_key_to_metrics_map:
+ report_metrics[report_dir] = run_key_to_metrics_map
+ metadata = load_metadata(report_dir)
+ timestamp = metadata.get("timestamp", "unknown")
+ report_timestamps[report_dir] = timestamp
+ for run_key, data_list in run_key_to_metrics_map.items():
+ if run_key not in report_input_args:
+ # Extract just the input_args part from the run_key for display
+ try:
+ run_key_data = eval(run_key) # Convert string back to dict
+ input_args = run_key_data.get('input_args', {})
+ report_input_args[run_key] = input_args
+ print(f" Debug: Extracted input_args from run_key: {input_args}")
+ except Exception as e:
+ # Fallback: extract from first result's run_metadata
+ fallback_args = data_list[0].get("run_metadata", {}).get("input_arguments", {})
+ report_input_args[run_key] = fallback_args
+ print(f" Debug: Using fallback input_arguments: {fallback_args}")
+ print(f" Debug: run_key parsing failed: {e}")
+ print(f" Debug: run_key content: {run_key[:200]}...")
+ print(f" Loaded {len(run_key_to_metrics_map)} run configurations")
+ else:
+ print(f" No results found in {report_dir}")
+
+ if not report_metrics:
+ print("\nError: No benchmark results found in any of the report directories.")
+ print("Please ensure that:")
+ print("1. The results_root path is correct")
+ print("2. The directories contain pipeline_results.json files")
+ print("3. The results were generated by the benchmark pipeline")
+ sys.exit(1)
+
+ # Compare metrics across reports
+ comparison_results = {}
+ for report_dir, run_key_to_metrics_map in report_metrics.items():
+ for run_key, metrics_list in run_key_to_metrics_map.items():
+ if run_key not in comparison_results:
+ comparison_results[run_key] = []
+ # Collect all results that have the same input arguments AND exec_metadata
+ comparison_results[run_key].extend(metrics_list)
+
+ # Generate comparison data - only for run_keys that have multiple results to compare
+ comparison_data = {}
+ for run_key, results_list in comparison_results.items():
+ if len(results_list) > 1:
+ # Only compare if we have multiple results with the same input arguments and exec_metadata
+ comparison_data[run_key] = compare_metrics(results_list)
+ try:
+ run_key_data = eval(run_key)
+ input_args = run_key_data.get('input_args', {})
+ exec_metadata = run_key_data.get('exec_metadata', {})
+ print(f"Comparing {len(results_list)} runs with input arguments: {input_args}")
+ if exec_metadata:
+ print(f" Exec metadata: {exec_metadata}")
+
+ # Debug outlier detection
+ metrics_comparison = compare_metrics(results_list)
+ if metrics_comparison:
+ print(f" Outlier detection debug (threshold={outlier_threshold}):")
+ for path, values in metrics_comparison[:2]: # Show first 2 metrics
+ numeric_values = [v for v in values if isinstance(v, (int, float))]
+ if len(numeric_values) > 1:
+ mean_val = sum(numeric_values) / len(numeric_values)
+ variance = sum((v - mean_val) ** 2 for v in numeric_values) / (len(numeric_values) - 1)
+ std_dev = variance ** 0.5 if variance > 0 else 0
+ outliers = [v for v in numeric_values if std_dev > 0 and abs(v - mean_val) / std_dev > outlier_threshold]
+ print(f" {'.'.join(path)}: {len(outliers)}/{len(numeric_values)} outliers detected")
+
+ # Debug: Print some metric values for outlier detection testing
+ metrics_comparison = compare_metrics(results_list)
+ if metrics_comparison:
+ print(f" Sample metrics for outlier detection:")
+ for path, values in metrics_comparison[:3]: # Show first 3 metrics
+ numeric_values = [v for v in values if isinstance(v, (int, float))]
+ if len(numeric_values) > 1:
+ mean_val = sum(numeric_values) / len(numeric_values)
+ variance = sum((v - mean_val) ** 2 for v in numeric_values) / (len(numeric_values) - 1)
+ std_dev = variance ** 0.5 if variance > 0 else 0
+ max_z = max(abs(v - mean_val) / std_dev if std_dev > 0 else 0 for v in numeric_values)
+ print(f" {'.'.join(path)}: values={numeric_values}, mean={mean_val:.3f}, std={std_dev:.3f}, max_z={max_z:.2f}")
+ except:
+ print(f"Comparing {len(results_list)} runs with run key: {run_key[:100]}...")
+ else:
+ print(f"Skipping comparison for run key - only {len(results_list)} result(s) found")
+
+ # Generate charts
+ chart_files = generate_charts(comparison_results, config)
+
+ # Print chart generation info
+ print("\nChart generation completed:")
+ print(f"Number of charts generated: {len(chart_files)}")
+ print("Using interactive Plotly charts with hover information and legends")
+ print("Chart file keys:", list(chart_files.keys()) if chart_files else "None")
+
+ # Generate HTML report
+ output_file = os.path.join(results_root, "benchmark_comparison_report.html")
+ try:
+ env = Environment(
+ autoescape=True,
+ enable_async=False,
+ keep_trailing_newline=False,
+ extensions=['jinja2.ext.do', 'jinja2.ext.loopcontrols']
+ )
+ template = env.from_string("""
+
+
+
+ Benchmark Comparison
+
+
+
+ Benchmark Comparison
+
+
+
+
Configuration Summary
+
Results Root: {{ config.get('results_root', 'Not specified') }}
+
Analysis: Significant change threshold: {{ config.get('analysis', {}).get('significant_change_threshold', config.get('significant_change_threshold', 0.0)) }}%,
+ Outlier threshold: {{ config.get('analysis', {}).get('outlier_threshold', config.get('outlier_threshold', 3.0)) }} sigma
+
Comparison Mode: {{ 'Same executable only' if config.get('analysis', {}).get('group_by_executable', config.get('group_by_executable', True)) else 'Cross-executable comparison' }}
+
Charts: {{ config.get('charts', [])|length }} charts configured, {{ chart_grid_cols }} columns
+
+
Charts
+
+ {% for key in chart_files.keys() %}
+
+ {% for chart in chart_files[key] %}
+ {{ chart|safe }}
+ {% endfor %}
+
+ {% endfor %}
+
+
Detailed Comparisons
+ {% for run_key, metrics in comparison_data.items() %}
+ {% set run_results = comparison_results[run_key] %}
+
+
Run Configuration {{ loop.index }}
+
Input Arguments
+
+
+ | Parameter |
+ Value |
+
+ {% if run_key in report_input_args %}
+ {% for key, value in report_input_args[run_key].items() %}
+
+ | {{ key }} |
+ {{ value }} |
+
+ {% endfor %}
+ {% endif %}
+
+
+
+ {% set first_result = run_results[0] if run_results else {} %}
+ {% set preprocessor_macros = first_result.get('run_metadata', {}).get('preprocessor_macros', []) %}
+ {% if preprocessor_macros %}
+
Preprocessor Macros
+
+
+ | Defined Macros |
+
+ {% for macro in preprocessor_macros %}
+
+ | {{ macro }} |
+
+ {% endfor %}
+
+ {% endif %}
+
+
+ {% if not group_by_executable %}
+
+
Executable Metadata (All Implementations)
+ {% set all_exec_metadata = {} %}
+ {% for result in run_results %}
+ {% set exec_metadata = result.get('run_metadata', {}).get('exec_metadata', {}) %}
+ {% set benchmark_id = get_benchmark_id_from_result(result) %}
+ {% if exec_metadata and benchmark_id not in all_exec_metadata %}
+ {% do all_exec_metadata.update({benchmark_id: exec_metadata}) %}
+ {% endif %}
+ {% endfor %}
+
+ {% if all_exec_metadata %}
+ {% for benchmark_id, exec_metadata in all_exec_metadata.items() %}
+
{{ benchmark_id }}
+
+
+ | Property |
+ Value |
+
+ {% for key, value in exec_metadata.items() %}
+
+ | {{ key }} |
+ {{ value }} |
+
+ {% endfor %}
+
+ {% endfor %}
+ {% endif %}
+ {% else %}
+
+ {% set first_result = run_results[0] if run_results else {} %}
+ {% set exec_metadata = first_result.get('run_metadata', {}).get('exec_metadata', {}) %}
+ {% if exec_metadata %}
+
Executable Metadata
+
+
+ | Property |
+ Value |
+
+ {% for key, value in exec_metadata.items() %}
+
+ | {{ key }} |
+ {{ value }} |
+
+ {% endfor %}
+
+ {% endif %}
+ {% endif %}
+
+ {% set categories = {} %}
+ {% for path, values in metrics %}
+ {% set category = path[0] %}
+ {% if category not in categories %}
+ {% do categories.update({category: []}) %}
+ {% endif %}
+ {% do categories[category].append((path, values)) %}
+ {% endfor %}
+ {% for category, category_metrics in categories.items() %}
+
{{ category }} Metrics
+
+
+ | Metric |
+ Variance (CV%) |
+ {% for result in run_results %}
+ {{ get_benchmark_id_from_result(result) }} |
+ {% endfor %}
+ Statistics |
+
+ {% for path, values in category_metrics %}
+ {% set path_key = '.'.join(path) %}
+ {% set outlier_data = outlier_info.get(run_key, {}).get(path_key, {}) %}
+ {% set variance = outlier_data.get('variance', 0) %}
+ {% set cv = outlier_data.get('coefficient_of_variation', 0) %}
+ {% set percentiles = outlier_data.get('percentiles', {}) %}
+ {% set outlier_count = outlier_data.get('outlier_count', 0) %}
+
+
+ | {{ '.'.join(path[1:]) }} |
+
+ {{ "%.2f"|format(cv) }}%
+ |
+ {% for value in values %}
+ {# Use pre-calculated outlier information #}
+ {% set outliers_list = outlier_data.get('outliers', []) %}
+ {% set is_outlier = outliers_list[loop.index0] if loop.index0 < outliers_list|length else False %}
+ {% set row_mean = outlier_data.get('mean', 0) %}
+ {% set row_std = outlier_data.get('std', 0) %}
+ {% set row_count = outlier_data.get('count', 0) %}
+
+ {# Calculate z-score for display #}
+ {% set z_score = 0 %}
+ {% if value is not none and value != "" and not (value is string) and row_std > 0 %}
+ {% set z_score = ((value|float - row_mean)|abs / row_std) %}
+ {% endif %}
+
+
+ {% if value is not none and value != "" %}
+ {% if value is string %}
+ {{ value }}
+ {% else %}
+ {% set float_val = value|float %}
+ {% if float_val != 0 and (float_val|abs < 0.000001 or float_val|abs >= 1000000) %}
+ {{ "%.3e"|format(float_val) }}
+ {% else %}
+ {{ "%.6f"|format(float_val) }}
+ {% endif %}
+ {% endif %}
+ {% else %}
+ N/A
+ {% endif %}
+ |
+ {% endfor %}
+
+ μ={{ "%.3f"|format(outlier_data.get('mean', 0)) }}
+ |
+
+ {% endfor %}
+
+ {% endfor %}
+
No comparison data available. This could mean:
+
+ - No runs with identical input arguments and exec_metadata were found across different benchmark executions
+ - Only single runs were found for each configuration
+ - The results contain different exec_metadata (different executables/compile options)
+
+ {% endif %}
+
+
+ """)
+
+ # Calculate outlier information before rendering
+ outlier_info = calculate_outliers(comparison_data, outlier_threshold)
+
+ html = template.render(
+ comparison_data=comparison_data,
+ comparison_results=comparison_results,
+ chart_files=chart_files,
+ report_input_args=report_input_args,
+ config=config,
+ outlier_threshold=outlier_threshold,
+ outlier_info=outlier_info,
+ chart_grid_cols=chart_grid_cols,
+ group_by_executable=group_by_executable,
+ get_benchmark_id_from_result=get_benchmark_id_from_result
+ )
+
+ with open(output_file, "w", encoding='utf-8') as f:
+ f.write(html)
+
+ print(f"\nHTML report generated successfully: {output_file}")
+
+ except Exception as e:
+ print(f"Error generating HTML report: {e}")
+ sys.exit(1)
+
+if __name__ == "__main__":
+ main()
\ No newline at end of file
diff --git a/Tetrahedral_remeshing/benchmark/Tetrahedral_remeshing/Benchmark_comparison/config.yaml b/Tetrahedral_remeshing/benchmark/Tetrahedral_remeshing/Benchmark_comparison/config.yaml
new file mode 100644
index 000000000000..129af433a1cc
--- /dev/null
+++ b/Tetrahedral_remeshing/benchmark/Tetrahedral_remeshing/Benchmark_comparison/config.yaml
@@ -0,0 +1,48 @@
+results_root: /path/to/Benchmark_results # Folder containing multiple benchmark run folders
+include_reports: [] # Optional: list of subfolders to include, or leave empty for all
+
+# Analysis configuration
+analysis:
+ significant_change_threshold: 0.0 # Only show metrics with >X% change (set to 0 to show all)
+ outlier_threshold: 2.0 # Highlight values >N standard deviations from mean as outliers
+ group_by_executable: false # false: compare across different executables (e.g. original vs refactored)
+ # true: compare only within the same executable (repeatability)
+ exclude_from_grouping: [] # Input argument keys to ignore when matching runs for comparison.
+ # Use e.g. [threads] to compare sequential (threads=1) vs parallel (threads=8)
+ # in the same table row.
+
+# Report configuration
+report:
+ chart_grid_cols: 2 # Number of columns in the chart grid (charts will be scaled accordingly)
+
+# Unified charts configuration - supports both time-series and parameter-based charts
+charts:
+ # Time-series chart (no x specified, defaults to timestamp)
+ - kind: line
+ y: metrics.Performance.Total_Time.Value
+ title: "Total Time across versions"
+ # save_as: "total_time_timeseries.png"
+ # filter:
+ # benchmark_name: uniform
+ style: "o-"
+
+ # Parameter-based charts - compare metrics across different parameter values
+ # Results are automatically grouped by input_arguments + exec_metadata
+ - kind: line
+ y: metrics.Performance.Total_Time.Value
+ x: run_metadata.input_arguments.threads # The parameter to vary on x-axis
+ title: "Total Time vs Number of Threads"
+ style: "o-"
+
+ - kind: line
+ y: metrics.Performance.Memory.Value
+ x: run_metadata.input_arguments.threads
+ title: "Memory Usage vs Number of Threads"
+ style: "o-"
+
+ - kind: line
+ y: resource_usage.memory.peak_rss_mb
+ title: "Peak Memory Usage Across Runs"
+ # save_as: "peak_memory_timeseries.png"
+ style: "d-" # diamonds with lines
+
\ No newline at end of file
diff --git a/Tetrahedral_remeshing/benchmark/Tetrahedral_remeshing/Benchmark_creation/README.md b/Tetrahedral_remeshing/benchmark/Tetrahedral_remeshing/Benchmark_creation/README.md
new file mode 100644
index 000000000000..28e4285687a5
--- /dev/null
+++ b/Tetrahedral_remeshing/benchmark/Tetrahedral_remeshing/Benchmark_creation/README.md
@@ -0,0 +1,217 @@
+# Generic Benchmarking Pipeline
+
+This repository provides a **generic, schema-driven benchmarking pipeline** for scientific and engineering workflows. It is designed to be easily adapted to any domain by simply editing a YAML schema file and configuration.
+
+---
+
+## Features
+
+- **Schema-driven validation**: Define your required results structure in a simple YAML file.
+- **Configurable parameter sweeps**: Run benchmarks over any parameter grid.
+- **Automatic results aggregation**: Collects and flattens results into a CSV and a full-fidelity JSON.
+- **Interactive visualization**: Generate dynamic HTML reports with embedded charts and data tables.
+- **Cross-run comparison**: Compare results across multiple benchmark runs with automatic chart generation.
+- **Extensible and domain-agnostic**: No hardcoded field names or domain logic.
+- **Rich metadata tracking**: Automatically captures environment info, git commit, and run configuration.
+
+---
+
+## Quickstart
+
+1. **Prepare your benchmark executables**
+ Your executables should output a JSON file with a structure matching your schema (see below).
+
+2. **Write your config file**
+ Example: `config.yaml`
+ ```yaml
+ pipeline:
+ results_dir: ./Results
+ run_benchmarks: true
+ export_charts: true
+ report: true
+ report_chart_grid_cols: 2 # Configure chart grid layout
+
+ benchmarks:
+ - name: my_benchmark
+ exec: ./my_benchmark_exec
+ input_dir: ./input_data
+ macros_config_file: ./my_benchmark_macros.h # Optional
+ sweep:
+ param1: [1, 2]
+ param2: [A, B]
+
+ charts:
+ - y: "metrics.Performance.Total_Time.Value"
+ x: "run_metadata.run_info.Mesh.Cells" # Optional, defaults to timestamp
+ title: "Performance vs Mesh Size"
+ style: "o-" # Optional plotting style
+ ```
+
+3. **Write your results schema**
+ Example: `results_schema.yaml`
+ ```yaml
+ - metrics:
+ - Performance
+ - Quality
+ - run_metadata:
+ - status
+ - run_info
+ - input_arguments
+ ```
+
+4. **Run the pipeline**
+ ```sh
+ python run_benchmarks.py --config config.yaml
+ ```
+
+5. **Compare results across runs**
+ ```sh
+ python Benchmark_comparison/compare_benchmark_results.py --config Benchmark_comparison/config.yaml
+ ```
+
+---
+
+## Directory Structure
+
+```
+.
+├── config.yaml
+├── results_schema.yaml
+├── run_benchmarks.py
+├── parse_results.py
+├── generate_charts.py
+├── generate_report.py
+├── generate_run_matrix.py
+├── Benchmark_comparison/
+│ └── compare_benchmark_results.py
+├── Results/
+│ ├── /
+│ │ ├── my_benchmark/
+│ │ │ ├── /
+│ │ │ │ ├── ..._results.json
+│ │ │ ├── benchmark_results.csv
+│ │ ├── pipeline_results.csv
+│ │ ├── pipeline_results.json
+│ │ ├── environment_metadata.json
+│ │ ├── Charts/
+│ │ │ ├── ...
+│ │ ├── report.html
+│ │ └── benchmark_comparison_report.html
+```
+
+---
+
+## Main Scripts
+
+- **run_benchmarks.py**: Orchestrates the pipeline, runs benchmarks, patches results, triggers aggregation, charting, and reporting.
+- **parse_results.py**: Flattens and validates results JSONs into a CSV and a full-fidelity JSON, using your schema.
+- **generate_charts.py**: Creates charts from the aggregated CSV, driven by your config.
+- **generate_report.py**: Produces a self-contained HTML report with embedded charts, parameter sweeps, and raw results.
+- **generate_run_matrix.py**: Generates the parameter sweep grid.
+- **compare_benchmark_results.py**: Compares results across multiple runs, generating interactive HTML reports with embedded charts and data tables.
+
+---
+
+## Results Schema
+
+- The schema is a YAML file describing the required structure of your results JSON.
+- Each dash/indentation level corresponds to a level in the JSON.
+- Only the presence of keys is checked; extra keys are ignored.
+
+Example:
+```yaml
+- metrics:
+ - Performance
+ - Quality
+- run_metadata:
+ - status
+ - run_info
+ - input_arguments
+```
+
+---
+
+## Preprocessor Macros Configuration
+
+The pipeline supports tracking preprocessor macros that are defined when building your benchmarks. This allows you to:
+
+- **Tag results** with the compile-time configuration used
+- **Compare runs** with different macro configurations
+- **Group results** by macro combinations for analysis
+
+### Configuration
+
+Add a `macros_config_file` field to your benchmark configuration:
+
+```yaml
+benchmarks:
+ - name: my_benchmark
+ exec: ./my_benchmark_exec
+ macros_config_file: ./my_benchmark_macros.h # Path to header file with macros
+ sweep:
+ param1: [1, 2]
+```
+
+### Macros Config File Format
+
+The macros config file should be a simple C/C++ header file containing `#define` statements:
+
+```c
+#define FEATURE_A_ENABLED
+#define FEATURE_B_ENABLED
+// #define FEATURE_C_ENABLED // Commented out = not defined
+#define OPTIMIZATION_LEVEL 2
+```
+
+### Important Notes
+
+- **Assumption**: The pipeline assumes that macros defined in the config file are actually used in your executable. **We do not validate that the macros are actually used in the code**.
+- **Purpose**: Macros are parsed and used to **tag results** for comparison and grouping purposes only.
+- **Parsing**: Only active `#define` statements are parsed; commented out defines (e.g., `// #define MACRO`) are ignored.
+- **Comparison**: Runs are only compared if they have the same input arguments, exec metadata, AND the same set of defined macros.
+
+### Results Structure
+
+Parsed macros appear in the results JSON as:
+
+```json
+{
+ "run_metadata": {
+ "preprocessor_macros": [
+ "FEATURE_A_ENABLED",
+ "FEATURE_B_ENABLED",
+ "OPTIMIZATION_LEVEL"
+ ]
+ }
+}
+```
+
+---
+
+## Data Aggregation and Comparison
+
+- After each pipeline run, both a CSV (`pipeline_results.csv`) and a JSON (`pipeline_results.json`) are exported.
+- **pipeline_results.json** is the canonical, full-fidelity data source for all downstream analysis and comparison.
+- The CSV is provided for spreadsheet compatibility and quick tabular inspection.
+- Environment metadata is captured in `environment_metadata.json`.
+- Use `compare_benchmark_results.py` to generate interactive comparison reports across multiple runs.
+
+---
+
+## Extending and Customizing
+
+- To add or remove required fields, just edit `results_schema.yaml`.
+- To change parameter sweeps, edit the `sweep` section in your config.
+- To add new charts or change chart types, edit the `charts` section in your config.
+- To adjust chart grid layout, set `report_chart_grid_cols` in the pipeline config.
+
+---
+
+## Troubleshooting
+
+- If a run fails schema validation, you'll get a clear error message showing the missing key and its path.
+- You can specify a different schema file with `--schema` if needed.
+- The pipeline automatically captures environment metadata to help reproduce and debug issues.
+
+---
+
diff --git a/Tetrahedral_remeshing/benchmark/Tetrahedral_remeshing/Benchmark_creation/config.yaml b/Tetrahedral_remeshing/benchmark/Tetrahedral_remeshing/Benchmark_creation/config.yaml
new file mode 100644
index 000000000000..98e2e73b23c5
--- /dev/null
+++ b/Tetrahedral_remeshing/benchmark/Tetrahedral_remeshing/Benchmark_creation/config.yaml
@@ -0,0 +1,55 @@
+# Tetrahedral Remeshing Benchmark — example configuration.
+#
+# Copy this file, fill in the absolute paths, then run from this directory:
+# python run_benchmarks.py --config config.yaml
+#
+# Executable positional args:
+#
+#
+
+pipeline:
+ results_dir: /path/to/Benchmark_results
+ run_benchmarks: true
+ num_repeats: 3
+ export_csv: true
+ export_charts: false
+ report: true
+ report_chart_grid_cols: 2
+ resource_monitoring: false
+
+profiling:
+ analysis_types: []
+ profile_runs: none
+
+benchmarks:
+ - name: original_sequential
+ exec: /path/to/build/Release/benchmark_original_tetrahedral_remeshing.exe
+ sweep:
+ input_mesh:
+ - /path/to/bear.mesh
+ - /path/to/sphere.mesh
+ num_iterations: [3]
+ target_edge_factor: [0.25]
+ smooth_constrained_edges: [1]
+ threads: [1]
+
+ - name: refactored_sequential
+ exec: /path/to/build/Release/benchmark_refactored_sequential_tetrahedral_remeshing.exe
+ sweep:
+ input_mesh:
+ - /path/to/bear.mesh
+ - /path/to/sphere.mesh
+ num_iterations: [3]
+ target_edge_factor: [0.25]
+ smooth_constrained_edges: [1]
+ threads: [1]
+
+ - name: parallel_8threads
+ exec: /path/to/build/Release/benchmark_refactored_parallel_tetrahedral_remeshing.exe
+ sweep:
+ input_mesh:
+ - /path/to/bear.mesh
+ num_iterations: [3]
+ target_edge_factor: [0.25]
+ smooth_constrained_edges: [1]
+ threads: [8]
diff --git a/Tetrahedral_remeshing/benchmark/Tetrahedral_remeshing/Benchmark_creation/external/nlohmann/json.hpp b/Tetrahedral_remeshing/benchmark/Tetrahedral_remeshing/Benchmark_creation/external/nlohmann/json.hpp
new file mode 100644
index 000000000000..8b72ea6539f4
--- /dev/null
+++ b/Tetrahedral_remeshing/benchmark/Tetrahedral_remeshing/Benchmark_creation/external/nlohmann/json.hpp
@@ -0,0 +1,24765 @@
+// __ _____ _____ _____
+// __| | __| | | | JSON for Modern C++
+// | | |__ | | | | | | version 3.11.3
+// |_____|_____|_____|_|___| https://github.com/nlohmann/json
+//
+// SPDX-FileCopyrightText: 2013-2023 Niels Lohmann
+// SPDX-License-Identifier: MIT
+
+/****************************************************************************\
+ * Note on documentation: The source files contain links to the online *
+ * documentation of the public API at https://json.nlohmann.me. This URL *
+ * contains the most recent documentation and should also be applicable to *
+ * previous versions; documentation for deprecated functions is not *
+ * removed, but marked deprecated. See "Generate documentation" section in *
+ * file docs/README.md. *
+\****************************************************************************/
+
+#ifndef INCLUDE_NLOHMANN_JSON_HPP_
+#define INCLUDE_NLOHMANN_JSON_HPP_
+
+#include // all_of, find, for_each
+#include // nullptr_t, ptrdiff_t, size_t
+#include // hash, less
+#include // initializer_list
+#ifndef JSON_NO_IO
+ #include // istream, ostream
+#endif // JSON_NO_IO
+#include // random_access_iterator_tag
+#include // unique_ptr
+#include // string, stoi, to_string
+#include // declval, forward, move, pair, swap
+#include // vector
+
+// #include
+// __ _____ _____ _____
+// __| | __| | | | JSON for Modern C++
+// | | |__ | | | | | | version 3.11.3
+// |_____|_____|_____|_|___| https://github.com/nlohmann/json
+//
+// SPDX-FileCopyrightText: 2013-2023 Niels Lohmann
+// SPDX-License-Identifier: MIT
+
+
+
+#include
+
+// #include
+// __ _____ _____ _____
+// __| | __| | | | JSON for Modern C++
+// | | |__ | | | | | | version 3.11.3
+// |_____|_____|_____|_|___| https://github.com/nlohmann/json
+//
+// SPDX-FileCopyrightText: 2013-2023 Niels Lohmann
+// SPDX-License-Identifier: MIT
+
+
+
+// This file contains all macro definitions affecting or depending on the ABI
+
+#ifndef JSON_SKIP_LIBRARY_VERSION_CHECK
+ #if defined(NLOHMANN_JSON_VERSION_MAJOR) && defined(NLOHMANN_JSON_VERSION_MINOR) && defined(NLOHMANN_JSON_VERSION_PATCH)
+ #if NLOHMANN_JSON_VERSION_MAJOR != 3 || NLOHMANN_JSON_VERSION_MINOR != 11 || NLOHMANN_JSON_VERSION_PATCH != 3
+ #warning "Already included a different version of the library!"
+ #endif
+ #endif
+#endif
+
+#define NLOHMANN_JSON_VERSION_MAJOR 3 // NOLINT(modernize-macro-to-enum)
+#define NLOHMANN_JSON_VERSION_MINOR 11 // NOLINT(modernize-macro-to-enum)
+#define NLOHMANN_JSON_VERSION_PATCH 3 // NOLINT(modernize-macro-to-enum)
+
+#ifndef JSON_DIAGNOSTICS
+ #define JSON_DIAGNOSTICS 0
+#endif
+
+#ifndef JSON_USE_LEGACY_DISCARDED_VALUE_COMPARISON
+ #define JSON_USE_LEGACY_DISCARDED_VALUE_COMPARISON 0
+#endif
+
+#if JSON_DIAGNOSTICS
+ #define NLOHMANN_JSON_ABI_TAG_DIAGNOSTICS _diag
+#else
+ #define NLOHMANN_JSON_ABI_TAG_DIAGNOSTICS
+#endif
+
+#if JSON_USE_LEGACY_DISCARDED_VALUE_COMPARISON
+ #define NLOHMANN_JSON_ABI_TAG_LEGACY_DISCARDED_VALUE_COMPARISON _ldvcmp
+#else
+ #define NLOHMANN_JSON_ABI_TAG_LEGACY_DISCARDED_VALUE_COMPARISON
+#endif
+
+#ifndef NLOHMANN_JSON_NAMESPACE_NO_VERSION
+ #define NLOHMANN_JSON_NAMESPACE_NO_VERSION 0
+#endif
+
+// Construct the namespace ABI tags component
+#define NLOHMANN_JSON_ABI_TAGS_CONCAT_EX(a, b) json_abi ## a ## b
+#define NLOHMANN_JSON_ABI_TAGS_CONCAT(a, b) \
+ NLOHMANN_JSON_ABI_TAGS_CONCAT_EX(a, b)
+
+#define NLOHMANN_JSON_ABI_TAGS \
+ NLOHMANN_JSON_ABI_TAGS_CONCAT( \
+ NLOHMANN_JSON_ABI_TAG_DIAGNOSTICS, \
+ NLOHMANN_JSON_ABI_TAG_LEGACY_DISCARDED_VALUE_COMPARISON)
+
+// Construct the namespace version component
+#define NLOHMANN_JSON_NAMESPACE_VERSION_CONCAT_EX(major, minor, patch) \
+ _v ## major ## _ ## minor ## _ ## patch
+#define NLOHMANN_JSON_NAMESPACE_VERSION_CONCAT(major, minor, patch) \
+ NLOHMANN_JSON_NAMESPACE_VERSION_CONCAT_EX(major, minor, patch)
+
+#if NLOHMANN_JSON_NAMESPACE_NO_VERSION
+#define NLOHMANN_JSON_NAMESPACE_VERSION
+#else
+#define NLOHMANN_JSON_NAMESPACE_VERSION \
+ NLOHMANN_JSON_NAMESPACE_VERSION_CONCAT(NLOHMANN_JSON_VERSION_MAJOR, \
+ NLOHMANN_JSON_VERSION_MINOR, \
+ NLOHMANN_JSON_VERSION_PATCH)
+#endif
+
+// Combine namespace components
+#define NLOHMANN_JSON_NAMESPACE_CONCAT_EX(a, b) a ## b
+#define NLOHMANN_JSON_NAMESPACE_CONCAT(a, b) \
+ NLOHMANN_JSON_NAMESPACE_CONCAT_EX(a, b)
+
+#ifndef NLOHMANN_JSON_NAMESPACE
+#define NLOHMANN_JSON_NAMESPACE \
+ nlohmann::NLOHMANN_JSON_NAMESPACE_CONCAT( \
+ NLOHMANN_JSON_ABI_TAGS, \
+ NLOHMANN_JSON_NAMESPACE_VERSION)
+#endif
+
+#ifndef NLOHMANN_JSON_NAMESPACE_BEGIN
+#define NLOHMANN_JSON_NAMESPACE_BEGIN \
+ namespace nlohmann \
+ { \
+ inline namespace NLOHMANN_JSON_NAMESPACE_CONCAT( \
+ NLOHMANN_JSON_ABI_TAGS, \
+ NLOHMANN_JSON_NAMESPACE_VERSION) \
+ {
+#endif
+
+#ifndef NLOHMANN_JSON_NAMESPACE_END
+#define NLOHMANN_JSON_NAMESPACE_END \
+ } /* namespace (inline namespace) NOLINT(readability/namespace) */ \
+ } // namespace nlohmann
+#endif
+
+// #include
+// __ _____ _____ _____
+// __| | __| | | | JSON for Modern C++
+// | | |__ | | | | | | version 3.11.3
+// |_____|_____|_____|_|___| https://github.com/nlohmann/json
+//
+// SPDX-FileCopyrightText: 2013-2023 Niels Lohmann
+// SPDX-License-Identifier: MIT
+
+
+
+#include // transform
+#include // array
+#include // forward_list
+#include // inserter, front_inserter, end
+#include