Skip to content

Q3 Features and Fixes#26

Merged
Wei (Celia) Xu (celiafish) merged 177 commits into
NSLS2:3.0from
psavery:nsls2-q3
May 28, 2026
Merged

Q3 Features and Fixes#26
Wei (Celia) Xu (celiafish) merged 177 commits into
NSLS2:3.0from
psavery:nsls2-q3

Conversation

@psavery

@psavery Patrick Avery (psavery) commented May 25, 2026

Copy link
Copy Markdown

This contains the Q3 features and fixes. The biggest change is a major pipeline redesign, which provides the foundation for branching pipelines and selectively persistent intermediate outputs (cached in memory or on disk).

Updated documentation and tutorial videos covering the new features in this PR are published at the 3.0 docs site: https://tomviz.readthedocs.io/en/3.0/.

The docs are mostly complete and still being refined.

Brief bullet points of all new features are provided below:

Q3 SOW features

  • Pipeline overhaul: typed node graph (sources / transforms / sinks) with a new vertical pipeline-strip widget.
  • Pipeline templates: save a pipeline's transform graph and reapply it to any source; auto-discovered from a templates directory (override via TOMVIZ_PIPELINE_TEMPLATES_PATH).
  • Branching pipelines: fan a single output port out into multiple downstream branches via drag-to-link, or fan multiple output ports inward for operators that require multiple inputs (e. g., similarity metrics, deconvolution/denoise).
  • Persistent intermediate outputs: per-port modes (transient / in-memory / on-disk) selectable from the pipeline controls toolbar or port context menu.
  • Mid-pipeline execution: breakpoints, Play/Pause/Stop controls, and resume.
  • State files: save/load .tvsm and .tvh5 with the new pipeline (including molecule and table ports); old 2.x state files still load.
  • Per-node external Python environments: each transform can run in a separate conda env (the hook for AI/ML and other heavy deps).
  • Custom Transforms auto-rescan: directory re-scanned on every menu open; no restart needed. Operator source code loaded each time it is added from the menu (so it can be edited externally). Search paths overridable via TOMVIZ_CUSTOM_TRANSFORMS_PATH.
  • Batch processing via tomviz-pipeline CLI / Python package, with glob, file-list, and per-source overrides.
  • Cylindrical crop: interactive 3D widget.
  • Stability: memory-leak fixes, hardened port-data access, Windows crash stack traces

Additional features beyond the SOW

  • LabelMap port type plus segmentation colormap generator; segmentation operators wired to the typed port.
  • Operator search dialog: type-aware; incompatible operators hidden under "Unavailable".
  • Drag-and-drop loading for data and state files.
  • New file formats: DICOM read/write, MRC write, HyperSpy HDF5, .npy, .mat.
  • Reconstruction: SIRT, ART, TV TomoPy methods; GPU SIRT and MLEM.
  • PyXRF / Ptycho: refactored as SourceNodes with embedded widgets

@alesgenova
clean slate new pipeline lib, mostly abstract
8508d88
@alesgenova
Add volume data class and PythonTransform node
e90fc9f
@alesgenova
adde ReaderSource nodes
5d8ad05
@alesgenova
initial pretty good version of a pipeline visualization widget
9f300db
@alesgenova
a few more tweaks to the pipeline widget
5450762
@alesgenova
adde threaded executor for the new pipeline
af2693f
@alesgenova
Initial implementation of the various modules as sink nodes
fefe0cb
@alesgenova
add volume port properties widget, and demo app
a2983fa
@alesgenova
add sink properties widgets
b569d5f
@alesgenova
add custom properties widget for python transform nodes
a158686
@alesgenova
initial very incomplete integration of new pipeline in tomviz
78ce4ad
@alesgenova
fix python transform scripts that inherit from Operator
34d0986
@alesgenova
fix a few reactivity bugs in the sink nodes
c48882e
@alesgenova
initial pass at adding back colormap widget and logic
8f9a2a3
@alesgenova
improve responsiveness when interacting with sinks and opacity editor
43b1c86
@alesgenova
minor appearance bugfixes
7c2526d
@alesgenova
replace DataSource with SourceNode for all load data paths
0f756ab
@alesgenova
improvements to the pipeline widget and sink toggle
094ff2f
@alesgenova
few fixes to LegacyPythonTransform handling of parameters
58da82b
@alesgenova
add link selection and context menus in the pipelineStripWidget
2da9f2b
@alesgenova
add interactive link creation and deletion
c068578
@alesgenova
improve pipeline widget layout and ordering (dfs)
640af05
@alesgenova
integrate some interactive pipeline editing in the main app
d56a97a
@alesgenova
minor improvements and bugfixes to pipelineStripWidget
6676892
@alesgenova
bring back operator edit dialogs
bac47a1
@alesgenova
add custom widgets of a few python transforms
e30f682
@alesgenova
add specialized versions of volume input/output ports
f566e9d
@alesgenova
add very basic indentation to pipelineStripWidget
5a16261
@alesgenova
improve output port typing of python transforms and data reader
71f0b31
@alesgenova
remove a lot of legacy code from ActiveObject and cascade cleanup
369c2ea
@psavery
Ensure we use pip from conda
406572f 
Pip doesn't always come with conda. We have to install it. It
was picking up a system pip before.

Signed-off-by: Patrick Avery <patrick.avery@kitware.com>
@psavery
Re-enable LegacyDataset tests
0f42074 
Signed-off-by: Patrick Avery <patrick.avery@kitware.com>
@psavery
Update old python tests to use LegacyDataset
fade44a 
This has the API they are expecting. Also, we needed to update the
random particles test to use a SourceNode.

Signed-off-by: Patrick Avery <patrick.avery@kitware.com>
@psavery
Try to avoid Windows store stub for Python version
c9d3895 
$(which python) should hopefully return the correct Python for Windows,
which may not ship "python3".

Signed-off-by: Patrick Avery <patrick.avery@kitware.com>
@psavery
Fix "Custom Transform" and "Custom ITK Transform"
38d63fa 
These now work, and no longer need AddExpressionReaction, so those
files were removed.

Signed-off-by: Patrick Avery <patrick.avery@kitware.com>
@psavery
Fix "Color By" other arrays for contour viz
00bda27 
Signed-off-by: Patrick Avery <patrick.avery@kitware.com>
@psavery
Increase default operator dialog size
b679c68 
Signed-off-by: Patrick Avery <patrick.avery@kitware.com>
@psavery
Fix occasional warning from SliceSink deletion
6b1ae9b 
Signed-off-by: Patrick Avery <patrick.avery@kitware.com>
@psavery
Make default B&C editor width larger
c36b2e0 
Signed-off-by: Patrick Avery <patrick.avery@kitware.com>
@psavery
Delete pipeline node on "Delete" key press
480c3d1 
Signed-off-by: Patrick Avery <patrick.avery@kitware.com>
@psavery
Delete contents of sink node group
30c43ba 
If a sink node group is deleted, also delete its contents.

Signed-off-by: Patrick Avery <patrick.avery@kitware.com>
@psavery
Check if port data is valid before accessing
baa75b1 
This prevents segmentation faults like the following:

    terminate called after throwing an instance of 'std::bad_any_cast'
      what():  bad any_cast

    Loguru caught a signal: SIGABRT
    Stack trace:
    39      0x64e7885debb5 _start + 37
    38      0x79f835a2a28b __libc_start_main + 139
    37      0x79f835a2a1ca /lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca) [0x79f835a2a1ca]
    36      0x64e7885da1ac main + 1116
    35      0x79f83d9afd92 QCoreApplication::exec() + 178
    34      0x79f83d9b2b8c QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) + 412
    33      0x79f83dcbc315
    QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) + 117
    32      0x79f831f7157e g_main_context_iteration + 46
    31      0x79f831f70e07
    /home/patrick/virtualenvs/conda/envs/tomviz/lib/libglib-2.0.so.0(+0x5fe07)
    [0x79f831f70e07]
    30      0x79f831f6d77a
    /home/patrick/virtualenvs/conda/envs/tomviz/lib/libglib-2.0.so.0(+0x5c77a)
    [0x79f831f6d77a]
    29      0x79f81e3aa7f0 /home/patrick/virtualenvs/conda/envs/tomviz/lib/qt6/plugins/platf
    orms/../../../libQt6XcbQpa.so.6(+0x5d7f0) [0x79f81e3aa7f0]
    28      0x79f83e24fa24
    QWindowSystemInterface::sendWindowSystemEvents(QFlags<QEventLoop::ProcessEventsFlag>) +
    180
    27      0x79f83e1e730c
    QGuiApplicationPrivate::processMouseEvent(QWindowSystemInterfacePrivate::MouseEvent*) +
    2860
    26      0x79f83d9a550a QCoreApplication::notifyInternal2(QObject*, QEvent*) + 362
    25      0x79f83eb87fcc QApplicationPrivate::notify_helper(QObject*, QEvent*) + 140
    24      0x79f83ec08960
    /home/patrick/virtualenvs/conda/envs/tomviz/lib/libQt6Widgets.so.6(+0x208960)
    [0x79f83ec08960]
    23      0x79f83ec05d9c
    /home/patrick/virtualenvs/conda/envs/tomviz/lib/libQt6Widgets.so.6(+0x205d9c)
    [0x79f83ec05d9c]
    22      0x79f83eb9736c QApplicationPrivate::sendMouseEvent(QWidget*, QMouseEvent*,
    QWidget*, QWidget*, QWidget**, QPointer<QWidget>&, bool, bool) + 1516
    21      0x79f83d9a550a QCoreApplication::notifyInternal2(QObject*, QEvent*) + 362
    20      0x79f83eb988d5 QApplication::notify(QObject*, QEvent*) + 3541
    19      0x79f83eb87fcc QApplicationPrivate::notify_helper(QObject*, QEvent*) + 140
    18      0x79f83ebf65f0 QWidget::event(QEvent*) + 656
    17      0x64e7888acb75
    tomviz::pipeline::PipelineStripWidget::mousePressEvent(QMouseEvent*) + 1765
    16      0x64e78863cc70
    tomviz::pipeline::PipelineStripWidget::nodeSelected(tomviz::pipeline::Node*) + 64
    15      0x79f83da06f61
    /home/patrick/virtualenvs/conda/envs/tomviz/lib/libQt6Core.so.6(+0x206f61)
    [0x79f83da06f61]
    14      0x64e78866ec51 tomviz::ActiveObjects::setActiveNode(tomviz::pipeline::Node*) +
    81
    13      0x64e78863ca33 tomviz::ActiveObjects::activeNodeChanged(tomviz::pipeline::Node*)
     + 67
    12      0x79f83da06f61
    /home/patrick/virtualenvs/conda/envs/tomviz/lib/libQt6Core.so.6(+0x206f61)
    [0x79f83da06f61]
    11      0x64e78861e6d5 tomviz::MainWindow::onActiveNodeChanged(tomviz::pipeline::Node*)
    + 837
    10      0x64e78888fca9
    tomviz::pipeline::NodePropertiesPanel::NodePropertiesPanel(tomviz::pipeline::Node*,
    tomviz::pipeline::Pipeline*, QWidget*) + 233
    9       0x64e78881e874 tomviz::pipeline::LegacyPythonTransform::createPropertiesWidget(t
    omviz::pipeline::Pipeline*, QWidget*) + 628
  8       0x64e78888ec8c
  tomviz::pipeline::PythonNodeEditorWidget::PythonNodeEditorWidget(tomviz::pipeline::Node*,
  tomviz::pipeline::Pipeline*, QString const&, QString const&, QString const&, QMap<QString,
  QVariant> const&, QString const&, QString const&,
  std::function<tomviz::pipeline::CustomPythonNodeWidget* (QWidget*)>, bool, QWidget*) + 4140
  7       0x64e7885bad53
  /home/patrick/virtualenvs/conda/envs/tomviz/src/tomviz-build/bin/tomviz(+0x1a4d53)
  [0x64e7885bad53]
  6       0x79f835ec1297
  /home/patrick/virtualenvs/conda/envs/tomviz/lib/libstdc++.so.6(+0xc1297) [0x79f835ec1297]
  5       0x79f835eae62e std::unexpected() + 0
  4       0x79f835ec0ffa
  /home/patrick/virtualenvs/conda/envs/tomviz/lib/libstdc++.so.6(+0xc0ffa) [0x79f835ec0ffa]
  3       0x79f835eaeaa4
  /home/patrick/virtualenvs/conda/envs/tomviz/lib/libstdc++.so.6(+0xaeaa4) [0x79f835eaeaa4]
  2       0x79f835a288ff abort + 223
  1       0x79f835a4527e gsignal + 30
  0       0x79f835a9eb2c pthread_kill + 284
  2026-05-20 15:31:01.871 ( 207.660s) [main thread     ]                       :0     FATL|
  Signal: SIGABRT

Signed-off-by: Patrick Avery <patrick.avery@kitware.com>
@psavery
Remove "too many unique values" warning
1e90651 
We also now accept any number of values for a segmentation colormap.
We just loop back around to the starting point and the same colors
will be re-used again.

Signed-off-by: Patrick Avery <patrick.avery@kitware.com>
@psavery
Fix warning when Clip or Slice is cleared
112ccac 
This should fix the following warning:

Generic Warning: In vtkNonOrthoImagePlaneWidget.cxx, line 505 set interactor and Enabled before changing interaction...

Signed-off-by: Patrick Avery <patrick.avery@kitware.com>
@psavery
Ensure warning doesn't appear on slice/clip delete
67648cb 
Signed-off-by: Patrick Avery <patrick.avery@kitware.com>
@psavery
Fix "Image Viewer Mode" to use new pipeline sinks
e1362e3 
Signed-off-by: Patrick Avery <patrick.avery@kitware.com>
@psavery
Stage several old features for removal
c0aea88 
This hides and stages several features for removal that we don't
anticipate being used. We can always bring them back if we find
real use cases for them.

Signed-off-by: Patrick Avery <patrick.avery@kitware.com>
@psavery
Simplify error printing in python transform
e53f0a5 
This prints only the line number and the exception from the python
operator that caused the exception.

Signed-off-by: Patrick Avery <patrick.avery@kitware.com>
@psavery
Stage registration operators for removal
c85c870 
This includes manual manipulation and registration. I'm not aware
of them being used. We can add them back in and update them if
users need them.

Signed-off-by: Patrick Avery <patrick.avery@kitware.com>
@psavery
Stronger guards against seg faults
297e857 
Ensure the data is valid before using.

Signed-off-by: Patrick Avery <patrick.avery@kitware.com>
@psavery
Make PortData::value<T>() non-throwing to prevent bad_any_cast crashes
d7d3264 
This makes the application more stable overall

Signed-off-by: Patrick Avery <patrick.avery@kitware.com>
@psavery
Allow "missing" entries to be checked
948a415 
They will be downloaded automatically if they are checked.

The import will fail if the download fails, or if "skip downloads"
is checked and some "missing" ones are checked.

Signed-off-by: Patrick Avery <patrick.avery@kitware.com>
@psavery
Add "executeWhenIdle()" to prevent re-execution
e0ffe5e 
This prevents re-execution of a pipeline that is already running,
when things like a slice module are added.

Example of the previous issue: a long-running reconstruction is
going, and the user adds a slice module. This triggers the whole
pipeline to re-execute immediately.

Now, it waits until the pipeline is finished, and if all the results
are valid, it won't re-execute.

Signed-off-by: Patrick Avery <patrick.avery@kitware.com>
@psavery
Allow missing scans to be checked and re-downloaded in PyXRF UI
8222674 
Signed-off-by: Patrick Avery <patrick.avery@kitware.com>
@psavery
Guard against invalid JSON values
9906d57 
Previously, invalid JSON values that users would input could crash
the application. This guards against them.

Signed-off-by: Patrick Avery <patrick.avery@kitware.com>
@psavery
Ensure combo box is updated in main thread
28c7da7 
We can't update the combo box from another thread. It can cause
a crash. Only update the combo box from the main thread.

Signed-off-by: Patrick Avery <patrick.avery@kitware.com>
@psavery
Trigger CI for PRs against branch 3.0
1ceeecb 
This 3.0 branch is for the new release we have planned. Ensure the
CI runs for it, while we are using it.

Signed-off-by: Patrick Avery <patrick.avery@kitware.com>
Copilot AI reviewed May 28, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

@celiafish Wei (Celia) Xu (celiafish) merged commit cc99279 into NSLS2:3.0 May 28, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@psavery @celiafish @alesgenova