Skip to content
Draft
Show file tree
Hide file tree
Changes from 96 commits
Commits
Show all changes
101 commits
Select commit Hold shift + click to select a range
6ede096
feat: add parent id to existing SimParticle EDM
paulgessinger Apr 28, 2026
eb9835a
feat: Make ScopedTimer threadsafe
paulgessinger Apr 24, 2026
37b3ec6
particle docs fixes
paulgessinger May 12, 2026
06005ef
clang-format
paulgessinger May 12, 2026
59290dd
MERGE
paulgessinger May 12, 2026
32d6228
feat: Initial arrow/parquet support
paulgessinger Apr 17, 2026
970f19d
experiment with arrow object library
paulgessinger Apr 22, 2026
d2080b8
clean up symbol visibility in wrapper target
paulgessinger Apr 22, 2026
f331c3b
make the isolated arrow absorption optional
paulgessinger Apr 24, 2026
3ce451b
add parquet option to full chain odd
paulgessinger Apr 24, 2026
99478a4
updated particle arrow schema based on colliderml
paulgessinger Apr 24, 2026
c28a93c
particle arrow converter writes parent id
paulgessinger Apr 28, 2026
5fbc174
use row indices as particle ids
paulgessinger Apr 29, 2026
5801bf5
add edm4hep to parquet conversion script
paulgessinger Apr 29, 2026
299d69e
update output converters to produce proper nulls
paulgessinger Apr 29, 2026
74cd1ea
add sim hit output converter + connect to track hit_ids
paulgessinger Apr 29, 2026
c6b9587
update detector resolver
paulgessinger Apr 29, 2026
a163863
add jobs arg to full chain odd
paulgessinger May 4, 2026
a824336
drop separate generated particles output
paulgessinger May 4, 2026
bcd8891
add plan for edm4hep input perf opt
paulgessinger May 4, 2026
4bbc029
clang-format
paulgessinger May 5, 2026
2de4baf
initial calo conversion
paulgessinger May 5, 2026
c421bc2
validated calo output
paulgessinger May 5, 2026
8c72b08
optimization for calo hits and averaging timers
paulgessinger May 5, 2026
fddfd47
some timing for edm4hepsiminput
paulgessinger May 5, 2026
fed4480
add proper detector encoding, speedup
paulgessinger May 5, 2026
ffbd6a8
restore pythia script (?)
paulgessinger May 7, 2026
fac62a5
use acts units more
paulgessinger May 7, 2026
b15e10c
dataset system shards files
paulgessinger May 7, 2026
5865ab1
address large number of propagation to perigee failures
paulgessinger May 7, 2026
d240655
test updates, fix init bug
paulgessinger May 7, 2026
33d01b4
lint / ci fixes
paulgessinger May 12, 2026
2240629
arrow schema (evolution) story follow up
paulgessinger May 12, 2026
5c1e993
add C-API / ABI safe interop with python
paulgessinger May 12, 2026
8bafc7c
bridge arrow schema into python cleanly
paulgessinger May 12, 2026
5d16d1a
add schema validation to ParquetWriter
paulgessinger May 12, 2026
c817d51
drop calo hit code
paulgessinger May 12, 2026
e94fd31
make misconfig a hard error
paulgessinger May 12, 2026
5648de8
clang-format
paulgessinger May 12, 2026
d890ddd
reduce diff
paulgessinger May 12, 2026
4720f3d
feat: Initial arrow/parquet support
paulgessinger Apr 17, 2026
3a09ce1
experiment with arrow object library
paulgessinger Apr 22, 2026
ebdde33
clean up symbol visibility in wrapper target
paulgessinger Apr 22, 2026
230727b
make the isolated arrow absorption optional
paulgessinger Apr 24, 2026
080bf22
add parquet option to full chain odd
paulgessinger Apr 24, 2026
2159d6f
updated particle arrow schema based on colliderml
paulgessinger Apr 24, 2026
c444056
particle arrow converter writes parent id
paulgessinger Apr 28, 2026
18fb443
use row indices as particle ids
paulgessinger Apr 29, 2026
e07bac0
add edm4hep to parquet conversion script
paulgessinger Apr 29, 2026
308778b
update output converters to produce proper nulls
paulgessinger Apr 29, 2026
f75fa18
add sim hit output converter + connect to track hit_ids
paulgessinger Apr 29, 2026
e3ee308
update detector resolver
paulgessinger Apr 29, 2026
683da75
add jobs arg to full chain odd
paulgessinger May 4, 2026
4396547
drop separate generated particles output
paulgessinger May 4, 2026
ce2c5c9
add plan for edm4hep input perf opt
paulgessinger May 4, 2026
b4b4971
clang-format
paulgessinger May 5, 2026
527780c
initial calo conversion
paulgessinger May 5, 2026
fc28cba
validated calo output
paulgessinger May 5, 2026
8b7a984
optimization for calo hits and averaging timers
paulgessinger May 5, 2026
a940d6b
some timing for edm4hepsiminput
paulgessinger May 5, 2026
38811b1
add proper detector encoding, speedup
paulgessinger May 5, 2026
4603d0c
restore pythia script (?)
paulgessinger May 7, 2026
8e836f4
use acts units more
paulgessinger May 7, 2026
ed599b5
dataset system shards files
paulgessinger May 7, 2026
0cfde9a
address large number of propagation to perigee failures
paulgessinger May 7, 2026
7786cd1
test updates, fix init bug
paulgessinger May 7, 2026
a94812a
lint / ci fixes
paulgessinger May 12, 2026
a3ccceb
arrow schema (evolution) story follow up
paulgessinger May 12, 2026
1cea9ae
add C-API / ABI safe interop with python
paulgessinger May 12, 2026
94f045a
bridge arrow schema into python cleanly
paulgessinger May 12, 2026
6824502
add schema validation to ParquetWriter
paulgessinger May 12, 2026
0c382f2
drop calo hit code
paulgessinger May 12, 2026
552a7d8
make misconfig a hard error
paulgessinger May 12, 2026
ee33a72
clang-format
paulgessinger May 12, 2026
ce2ab37
reduce diff
paulgessinger May 12, 2026
421fa0d
cmake lint
paulgessinger Jun 2, 2026
5f14a9a
Merge remote-tracking branch 'paulgessinger/feat/arrow-plugin+convers…
benjaminhuth Jun 2, 2026
7811dc8
add ColliderML → ACTS EDM converter and Kalman truth tracking demo
benjaminhuth Jun 2, 2026
1651eb4
fix geo map matching and simhit/measurement correspondence; switch to…
benjaminhuth Jun 3, 2026
cc89ed9
store geo map as Parquet; absorb ColliderMLInputConverter into Arrow …
benjaminhuth Jun 3, 2026
77000b5
refactor: ColliderML scripts to run-function style; separate slides
benjaminhuth Jun 4, 2026
019c026
fix: serialize histograms to numpy arrays before pickling
benjaminhuth Jun 4, 2026
a76eed0
fix: handle all three histogram wrapper types in _serialize_hists
benjaminhuth Jun 4, 2026
33b2fc8
fix: clean up performance plot style
benjaminhuth Jun 4, 2026
0e34a38
feat: timestamp PDF filename in performance plots script
benjaminhuth Jun 4, 2026
1821e68
fix: geo map matching + hard errors for g2l/unknown-geoId
benjaminhuth Jun 4, 2026
9b1230f
fix: reliable geo map generation + hard errors in converter
benjaminhuth Jun 4, 2026
b905684
feat: proto-track efficiency evaluation + seeding vs KF comparison
benjaminhuth Jun 4, 2026
44c837f
fix: evaluate efficiency against truth_seeded_particles denominator
benjaminhuth Jun 4, 2026
355ec56
feat: title slide + step-function plot style
benjaminhuth Jun 4, 2026
e8fc04d
Merge upstream/main into feature/collider-ml-reader-arrow
benjaminhuth Jun 5, 2026
1b7c564
update
benjaminhuth Jun 5, 2026
134fa89
update unused files
benjaminhuth Jun 6, 2026
a9391d2
lint
benjaminhuth Jun 9, 2026
5fce52a
Merge remote-tracking branch 'upstream/main' into feature/collider-ml…
benjaminhuth Jun 9, 2026
494f445
remove unrelated stuff
benjaminhuth Jun 9, 2026
b1cf5ed
update
benjaminhuth Jun 9, 2026
a3e6abd
restore odd.py
benjaminhuth Jun 9, 2026
4e995fc
address PR review comments: schema validation, cleanup
benjaminhuth Jun 9, 2026
2636986
remove redundant schema validation from execute()
benjaminhuth Jun 9, 2026
476a802
update
benjaminhuth Jun 12, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CI/check_unused_files.py
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,7 @@
".yml",
".xml",
".sh",
".parquet",
)


Expand Down
Binary file added Examples/Configs/colliderml_geo_map.parquet

@paulgessinger paulgessinger Jun 9, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this the volume mapping? Should that be parquet? I think there's benefit in having this be ASCII, no? How large is it?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had it in CSV and thought I put it in parquet because its CSV with ten-thousands of lines of samples... but can do CSV as well

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, the CSV is roughly 500KB, vs 131KB, so CSV is acceptable.

Binary file not shown.
55 changes: 55 additions & 0 deletions Examples/Io/Parquet/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -78,3 +78,58 @@ target_link_libraries(
)

acts_compile_headers(ExamplesIoParquet GLOB include/ActsExamples/**/*.hpp)

# ColliderML input converter — absorbed into libActsPluginArrow like the
# other Parquet TUs. It needs ExamplesDigitization (for createMeasurement and
# smearing types), so we add that as a PRIVATE link to ActsPluginArrow and
# include its headers via genex (not a link-level dep on the OBJECT target).
add_library(
Comment thread
benjaminhuth marked this conversation as resolved.
Outdated
ActsExamplesIoColliderML_obj
OBJECT
src/ColliderMLInputConverter.cpp
)
target_include_directories(
ActsExamplesIoColliderML_obj
PRIVATE
$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>
$<TARGET_PROPERTY:Acts::PluginArrow,INTERFACE_INCLUDE_DIRECTORIES>
$<TARGET_PROPERTY:Acts::ExamplesDigitization,INTERFACE_INCLUDE_DIRECTORIES>
)
target_link_libraries(
ActsExamplesIoColliderML_obj
PRIVATE Acts::ExamplesFramework Acts::ArrowLinkage
)
set_target_properties(
ActsExamplesIoColliderML_obj
PROPERTIES POSITION_INDEPENDENT_CODE ON
)
if(ACTS_ARROW_ISOLATED)
set_target_properties(
ActsExamplesIoColliderML_obj
PROPERTIES CXX_VISIBILITY_PRESET hidden VISIBILITY_INLINES_HIDDEN YES
)
endif()

target_sources(
ActsPluginArrow
PRIVATE $<TARGET_OBJECTS:ActsExamplesIoColliderML_obj>
)
target_link_libraries(ActsPluginArrow PRIVATE Acts::ExamplesDigitization)
# ExamplesDigitization is PRIVATE-linked by ActsPluginArrow; add it to the
# export set for the same reason ExamplesFramework is exported above.
install(TARGETS ActsExamplesDigitization EXPORT ActsPluginArrowTargets)

# INTERFACE alias so downstream code using Acts::ExamplesIoColliderML still
# resolves to libActsPluginArrow (no separate .so produced).
acts_add_library(ExamplesIoColliderML INTERFACE)
target_include_directories(
ActsExamplesIoColliderML
INTERFACE $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>
)
target_link_libraries(
ActsExamplesIoColliderML
INTERFACE
Acts::PluginArrow
Acts::ExamplesFramework
Acts::ExamplesDigitization
)
Original file line number Diff line number Diff line change
@@ -0,0 +1,148 @@
// This file is part of the ACTS project.
//
// Copyright (C) 2016 CERN for the benefit of the ACTS project
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this
// file, You can obtain one at https://mozilla.org/MPL/2.0/.

#pragma once

#include "Acts/Geometry/GeometryIdentifier.hpp"
#include "Acts/Utilities/Logger.hpp"
#include "ActsExamples/Digitization/DigitizationConfig.hpp"
#include "ActsExamples/EventData/Measurement.hpp"
#include "ActsExamples/EventData/SimHit.hpp"
#include "ActsExamples/EventData/SimParticle.hpp"
#include "ActsExamples/EventData/TruthMatching.hpp"
#include "ActsExamples/Framework/DataHandle.hpp"
#include "ActsExamples/Framework/IAlgorithm.hpp"
#include "ActsPlugins/Arrow/ArrowUtil.hpp"

#include <cstdint>
#include <filesystem>
#include <memory>
#include <string>
#include <unordered_map>

namespace Acts {
class TrackingGeometry;
}

namespace ActsExamples {

/// Packed ColliderML geometry key: det(8b)|vol(8b)|layer(16b)|surface(32b).
Comment thread
benjaminhuth marked this conversation as resolved.
Outdated
inline std::uint64_t colliderMLGeoKey(std::uint8_t detector,
std::uint8_t volume, std::uint16_t layer,
std::uint32_t surface) {
return (static_cast<std::uint64_t>(detector) << 40) |
(static_cast<std::uint64_t>(volume) << 32) |
(static_cast<std::uint64_t>(layer) << 16) |
static_cast<std::uint64_t>(surface);
}

/// Load a ColliderML geometry ID map from a CSV file.
Comment thread
benjaminhuth marked this conversation as resolved.
Outdated
///
/// Expected columns (with header): detector, volume, layer, surface,
/// acts_geo_id. The @c acts_geo_id value is parsed as a hex or decimal
/// unsigned 64-bit integer (e.g. @c 0x1000000000010001).
///
/// @param path Path to the CSV file.
/// @return Map from packed ColliderML key to @c Acts::GeometryIdentifier.
std::unordered_map<std::uint64_t, Acts::GeometryIdentifier>
loadColliderMLGeoIdMap(const std::filesystem::path& path);

/// Convert ColliderML Arrow tables to ACTS EDM types.
///
/// Reads two Arrow tables placed on the whiteboard by @c ParquetReader —
/// one for particles and one for tracker hits — and emits any combination
/// of @c SimParticleContainer, @c SimHitContainer, and
/// @c MeasurementContainer depending on which output keys are non-empty.
///
/// Modes (controlled by which output keys are set in the config):
/// - Particles only: set @c outputParticles, leave hits keys empty.
/// - + SimHits: also set @c outputSimHits.
/// - + Measurements: also set @c outputMeasurements; requires
/// @c trackingGeometry and @c digiConfig.
///
/// @note SimHit momentum fields are zero-filled; ColliderML does not
/// record per-hit momentum.
class ColliderMLInputConverter : public IAlgorithm {
public:
struct Config {
/// Whiteboard key for the particles Arrow table (from ParquetReader).
std::string inputParticlesTable;
/// Whiteboard key for the tracker-hits Arrow table (from ParquetReader).
std::string inputHitsTable;

/// Output key for @c SimParticleContainer. Empty = skip.
std::string outputParticles;
/// Output key for @c SimHitContainer. Empty = skip.
std::string outputSimHits;
/// Output key for @c MeasurementContainer. Empty = skip.
std::string outputMeasurements;
/// Output key for @c MeasurementSubset covering all measurements (required
/// by CKF / SpacePointMaker). Empty = skip.
std::string outputMeasurementSubset;
/// Output key for @c MeasurementSimHitsMap. Empty = skip.
std::string outputMeasSimHitsMap;
/// Output key for @c MeasurementParticlesMap (measurement→particle).
/// Empty = skip.
std::string outputMeasParticlesMap;
/// Output key for @c ParticleMeasurementsMap (particle→measurements).
/// Required by TruthTrackFinder / TruthSeeding. Empty = skip.
std::string outputParticleMeasurementsMap;

/// Required when @c outputMeasurements is non-empty.
std::shared_ptr<const Acts::TrackingGeometry> trackingGeometry;

/// Digitisation config used to determine subspace and covariance per
/// surface. Required when @c outputMeasurements is non-empty.
/// Load with @c readDigiConfigFromJson (acts.examples.json module).
/// The hierarchy map is queried per hit with full fallback
/// (sensitive → layer → volume).
DigiConfigContainer digiConfig;

/// ColliderML (det, vol, layer, surf) → ACTS GeometryIdentifier.
/// Optional. When empty, geometry IDs are constructed directly from the
/// ColliderML (volume, layer, surface) fields — only correct when the data
/// was produced from a geometry whose ID scheme matches the current build.
/// Load with @c loadColliderMLGeoIdMap().
std::unordered_map<std::uint64_t, Acts::GeometryIdentifier> geoIdMap;
};

ColliderMLInputConverter(const Config& cfg,
std::unique_ptr<const Acts::Logger> logger);

ColliderMLInputConverter(const Config& cfg, Acts::Logging::Level level);

~ColliderMLInputConverter() override;

ProcessCode execute(const AlgorithmContext& ctx) const final;

const Config& config() const { return m_cfg; }

private:
Config m_cfg;

ReadDataHandle<ActsPlugins::ArrowUtil::ArrowTable> m_inputParticles{
this, "InputParticles"};
ReadDataHandle<ActsPlugins::ArrowUtil::ArrowTable> m_inputHits{this,
"InputHits"};

WriteDataHandle<SimParticleContainer> m_outputParticles{this,
"OutputParticles"};
WriteDataHandle<SimHitContainer> m_outputSimHits{this, "OutputSimHits"};
WriteDataHandle<MeasurementContainer> m_outputMeasurements{
this, "OutputMeasurements"};
WriteDataHandle<MeasurementSubset> m_outputMeasurementSubset{
this, "OutputMeasurementSubset"};
WriteDataHandle<MeasurementSimHitsMap> m_outputMeasSimHitsMap{
this, "OutputMeasSimHitsMap"};
WriteDataHandle<MeasurementParticlesMap> m_outputMeasParticlesMap{
this, "OutputMeasParticlesMap"};
WriteDataHandle<ParticleMeasurementsMap> m_outputParticleMeasurementsMap{
this, "OutputParticleMeasurementsMap"};
};

} // namespace ActsExamples
Loading
Loading