HyperspyUI for 4D STEM

[CSSFrancis]

@ericpre I was wondering what changes might be necessary to properly handle lazy datasets/4D STEM.

I think with the Lazy plotting and annotations this might be a little easier/ smoother than when first tried.

General Idea:

1 - The navigator should be reused for all transformations of some lazy signal using the map function to not repeatedly compute the navigator.
2 - Any map-reduce functions should only compute when asked to (ie. Virtual imaging)
3 - Vectors signals are just displayed over the signal.

This should allow use to port most of the functions in pyxem into HyperspyUI and have pretty good functionality there.

One thing that might be nice is to have the UI running a seperate event loop and then have dask run all of the parallelization etc. I'm not sure that is the case at the moment. But with lazy signals you tend to get a lot of spinning wheels.

[CSSFrancis]

@magnunor Any thoughts as well?

I've increasingly been having problems getting people to use pyxem/ hyperspy and there is a fairly big force that wants to do processing on the computer next to the microscope.

[ericpre]

1 - The navigator should be reused for all transformations of some lazy signal using the map function to not repeatedly compute the navigator.

Would it better to handle this in hyperspy?

2 - Any map-reduce functions should only compute when asked to (ie. Virtual imaging)

Could it be done by adding a docked widget which allow the user to define manually or automatically?

3 - Vectors signals are just displayed over the signal.

Do you mean adding a plugin to support a specify workflow (if I infer correctly: handling several signals, one being a vector signal?), otherwise, I guess this should all be handling in hyperspy/pyxem?

This should allow use to port most of the functions in pyxem into HyperspyUI and have pretty good functionality there.

You should already be able to run everything that you do in pyxem within HyperspyUI by using the embedded qtconsole or a script.
One think that would be good is to integrate with jupyter notebook and they could two ways:

• running notebook within HyperSpyUI, a good example of that is notebook in spyder
• Use HyperSpyUI as "matplotlib backend" to manage the matplotlib figure within a jupyter notebook

In both cases, the dependent process needs to connect the already running jupyter kernel when starting.

One thing that might be nice is to have the UI running a seperate event loop and then have dask run all of the parallelization etc. I'm not sure that is the case at the moment. But with lazy signals you tend to get a lot of spinning wheels.

It should be possible to do that within HyperSpyUI because we should have good control of the event loop. In Ipython qtconsole or jupyter lab/notebook, we may not have that much control.

[ericpre]

• Use HyperSpyUI as "matplotlib backend" to manage the matplotlib figure within a jupyter notebook

xref #129

[CSSFrancis]

Would it better to handle this in hyperspy?

Yes I think I generally the navigation image should be reused especially for lazy images.

2 - Any map-reduce functions should only compute when asked to (ie. Virtual imaging)

Could it be done by adding a docked widget which allow the user to define manually or automatically?

That's a really good idea. Although we have to be better about not freezing the application when someone tries to compute when they shouldn't.

3 - Vectors signals are just displayed over the signal.

Do you mean adding a plugin to support a specify workflow (if I infer correctly: handling several signals, one being a vector signal?), otherwise, I guess this should all be handling in hyperspy/pyxem?

Ultimately I would like to be able to do template matching or strain mapping etc.

The problem with operations that return vectors is how do you display them in order to have a selectable signal object. We can now just plot them over a signal.

This should allow use to port most of the functions in pyxem into HyperspyUI and have pretty good functionality there.

You should already be able to run everything that you do in pyxem within HyperspyUI by using the embedded qtconsole or a script.

Yea but it might be nice to have the ability to right click and apply any function that the signal can apply.

One think that would be good is to integrate with jupyter notebook and they could two ways:

• running notebook within HyperSpyUI, a good example of that is notebook in spyder

• Use HyperSpyUI as "matplotlib backend" to manage the matplotlib figure within a jupyter notebook

In both cases, the dependent process needs to connect the already running jupyter kernel when starting.

Hmm I might look into how to do this!

One thing that might be nice is to have the UI running a seperate event loop and then have dask run all of the parallelization etc. I'm not sure that is the case at the moment. But with lazy signals you tend to get a lot of spinning wheels.

It should be possible to do that within HyperSpyUI because we should have good control of the event loop. In Ipython qtconsole or jupyter lab/notebook, we may not have that much control.

Another thing to look into I guess.

[ericpre]

That's a really good idea. Although we have to be better about not freezing the application when someone tries to compute when they shouldn't.

It should be possible and there is functionality for this, see for example:

hyperspyUI/hyperspyui/uiprogressbar.py

Lines 60 to 70 in fce8b44

	def _wrap(*args, **kwargs):
	"""
	Replacement function for hyperspy.external.progressbar.progressbar().
	Causes a UIProgressBar() to be made, which the MainWindow can connect to
	in order to create a progress indicator. It is important that the
	connection is made with QtCore.Signals, as they are thread aware, and the
	signal is processed on the GUI main event loop, i.e. the main thread. This
	is necessary as all UI operations have to happen on the main thread, and
	the hyperspy processing might be pushed to a worker thread "threaded.py".
	"""
	return UIProgressBar(*args, **kwargs)

It may be rough on the edge quite possible some code needs to be updated here to take into account changes in hyperspy.

Any operation that it long enough should have a progress bar, for example in case of the decomposition:

hyperspyUI/hyperspyui/plugins/mva.py

Line 255 in fce8b44

t = ProgressThreaded(self.ui, do_threaded, lambda: callback(ns), label=label)

[CSSFrancis]

Just some other thoughts related to this:

@ericpre I'm curious about the "tree"/ DataViewWidget. What I would like to do is to "connect" signals together where you might have a workflow like this:

4D STEM Signal --> Centered 4D STEM Signal -->Polar 4D STEM Signal --> Orientation Match

In this case, it would be nice to represent this entire workflow as a single signal, with the ability to switch between viewing each of the different "steps".

[CSSFrancis]

Just curious if this is worth doing? If I was starting new for 4D STEM the way that I would tackle this is:

1. Add typing in hyperspy/pyxem. This would allow us to auto create dialogues for different functions.
2. Run everything through dask. The full lazy workflow including markers now in hyperspy would allow us to interact with data without computing for any function using map.
3. Make every fuction editable as well. Meaning that you could edit some functions parameters and it would update the resultant signals. This would make it easier to build up a full workflow.
4. Try to implement more gpu integrations. That would make the plotting a bit more interactive as you could speed up the calculations on a single chunk.

[ericpre]

4D STEM Signal --> Centered 4D STEM Signal -->Polar 4D STEM Signal --> Orientation Match

In this case, it would be nice to represent this entire workflow as a single signal, with the ability to switch between viewing each of the different "steps".

This could be done with some widgets (traitsui and ipwidgets so that they can be used in hyperspyui but also in a notebook) but these would different signals, maybe sharing the same navigator for convenience?
On changing parameter or navigation position, it could compute the current coordinate for preview purposes.
This should be possible without too much work - there is a bit of learning curve for implementing traitsui/ipywidgets widgets but this is not too much work.

I don't know if typing will be enough to generate widgets automatically. It may be worth checking what/how napari are doing.

For interactive plotting on lazy signal, it may be worth revisiting caching in dask, as you already mentioned in hyperspy/hyperspy#3326. ;)
Matplotlib can be a significant bottleneck (even with the qt backend in hyperspyui) when images are large or "many artists to draw"...