Allow to add files to analysis browser which got returned by a python script

[t-b]

• Test on CI machine
• Chat about UI with Tim:
  Result: Add button to open a generic modal window where users can paste files, folders, cellids
• Can we not hardcode the credentials?
  Result: Yes don't hardcode them. Store in packages settings? Somehow encrypted or XXX?
• Fix the scripts in all necessary regards
• Add uv to tools
• Use uv from tools, add CI test

Close #2591

[Copilot]

Pull request overview

This PR introduces a new public API function AB_AddFolder to enable adding folders to the analysis browser programmatically, particularly for use from Python scripts. The change refactors existing button handler code to use this new function and makes the internal helper function AB_AddElementToSourceList static to clarify the API boundary.

Key Changes:

• Adds new public function AB_AddFolder(string win, WAVE/T folders) for adding folders to the analysis browser
• Refactors AB_ButtonProc_AddFolder to use the new AB_AddFolder function
• Makes AB_AddElementToSourceList static to restrict it to internal use only

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Suppressed comments (10)

tools/nwb-read-tests/nwbv2-read-test.py:52

• Same as above for dandi validation: check=True will raise before the stdout/stderr is printed, so the validation output is lost on failures. Restoring explicit return-code handling keeps the error output visible while still failing the script.

    comp = run(
        ["dandi", "validate", "--ignore", "(NWBI|DANDI)", path],
        stdout=PIPE,
        stderr=STDOUT,
        text=True,
        timeout=120,
        check=True,
    )

    print(f"dandi validation output: {comp.stdout}", file=sys.stdout)

Packages/Python/limspath_from_cellname.py:16

• This script hardcodes LIMS DB connection credentials/host in source. That creates an immediate secret-management risk and also makes it impossible to run against different environments without editing the file.

Please load connection settings from environment variables (or a config file) and fail fast with a clear error when required variables are missing.

import sys

import pg8000
from pg8000.native import literal


def limspath_from_cellname(cellnames: list):

    with (
        pg8000.connect(
            user="limsreader",
            host="limsdb2",
            database="lims2",
            password="limsro",
            port=5432,
        ) as conn,

Packages/Python/limspath_from_cellname.py:32

• The SQL query uses LIKE with a user-provided cell name. Even though literal(...) quotes safely, LIKE still treats %/_ as wildcards, which can unintentionally match multiple records and return an arbitrary row via fetchone().

If the input is meant to be an exact cell ID/name, use equality instead (and prefer is not None for the result check).

        for cell in cellnames:
            cur.execute(
                f"""SELECT err.storage_directory AS path
                FROM specimens cell
                JOIN ephys_roi_results err ON err.id = cell.ephys_roi_result_id
                WHERE cell.name LIKE {literal(cell)}"""
            )

            result = cur.fetchone()

            if result != None:
                paths.append(result[0])

tools/nwb-read-tests/nwbv2-read-test.py:40

• Using subprocess.run(..., check=True) here prevents printing the captured validator output on failure: CalledProcessError is raised before the print(...) line runs. This makes CI/debugging harder compared to the previous behavior where the tool output was emitted on stderr and a non-zero code was returned.

Consider switching back to explicit return-code handling (or catching CalledProcessError and printing e.stdout) so failures still report the validation output and return 1.

This issue also appears on line 43 of the same file.

    comp = run(
        ["pynwb-validate", path],
        stdout=PIPE,
        stderr=STDOUT,
        text=True,
        timeout=120,
        check=True,
    )

    print(f"pynwb validation output: {comp.stdout}", file=sys.stdout)

tools/nwb-read-tests/nwbv2-read-test.py:111

• The top-level exception handler prints the exception and then re-raises it, which will typically result in duplicate output (message + traceback). If the intent is to fail with a helpful traceback, printing the full traceback once and exiting with a non-zero status is clearer.

if __name__ == "__main__":
    try:
        sys.exit(main())
    except Exception as e:
        print(e, file=sys.stderr)
        raise

Packages/MIES/MIES_Python.ipf:19

• PY_CreateVirtEnv is declared as Function/S (string return type) but it does not return a string. This makes the API misleading and can mask errors (callers may assume a meaningful return value).

Function/S PY_CreateVirtEnv()

Packages/MIES/MIES_Python.ipf:40

• PY_ActivateVirtEnv is declared as Function/S (string return type) but it does not return a string. Use a non-string function signature (or return a meaningful value such as the activated venv path).

Function/S PY_ActivateVirtEnv()

Packages/MIES/MIES_Python.ipf:29

• The venv is created with --python 3.14, while the generated requirements.txt in tools/lims-query is compiled for --python-version 3.11 (and the developer docs mention 3.13 for tooling). Pinning a different interpreter version than the one used to compile hashes can lead to resolution/install failures.

Consider aligning the venv python version with the compiled requirements target (3.11), or regenerating requirements.txt for the intended version.

	sprintf cmd, "uv venv --clear --no-project --no-config --relocatable --managed-python --python 3.14 \"%s\"", HFSPathToWindows(venv)

Packages/MIES/MIES_AnalysisBrowser.ipf:3287

• AB_GatherFoldersFromLIMS is currently an empty stub, but the PR title/linked issue indicate LIMS integration to fetch experiment paths given a cell ID. As-is, there is no implementation to gather folders or connect it to the analysis browser.

Either remove this placeholder until it’s implemented, or implement it to call the Python-backed fetch function (Igor 10+) and return a text wave of folders.

Function AB_GatherFoldersFromLIMS(WAVE/T cellIDs)

End

Packages/MIES/MIES_Python.ipf:9

• This new procedure file is missing the standard Doxygen-style file header (/// @file ... / /// @brief ...) that is present in other MIES modules (e.g. MIES_Configuration.ipf, MIES_Replay.ipf). Adding it keeps generated documentation consistent and makes it easier to discover the module’s purpose.

Function/S PY_GetPackageFolder(string packageName)

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

Suppressed comments (9)

Packages/Python/limspath_from_cellname.py:15

• Database connection parameters (including credentials) are hardcoded in the repository. This is a security risk and also makes local/CI usage brittle; prefer reading these values from environment variables (or another external secret/config mechanism) and fail fast if they are not provided.

            user="limsreader",
            host="limsdb2",
            database="lims2",
            password="limsro",
            port=5432,

Packages/MIES/MIES_Python.ipf:21

• folder is declared but never used, which adds noise and can trigger linter warnings. Please remove it from the local variable list.

	string venv, cmd, folder, reqFolder, packageName, pkgFolder

Packages/MIES/MIES_Python.ipf:29

• The venv creation pins Python 3.14, but the compiled requirements file for this tool was generated with --python-version 3.11 (tools/lims-query/requirements.txt). Please align these so dependency resolution is consistent/reproducible.

	sprintf cmd, "uv venv --clear --no-project --no-config --relocatable --managed-python --python 3.14 \"%s\"", HFSPathToWindows(venv)

Packages/MIES/MIES_Python.ipf:59

• The Python script path is hardcoded to a developer-specific absolute path (e:/...). This will fail on other machines and CI. Please build the script path relative to the installed MIES Packages folder (similar to other FunctionPath usages).

	PythonFile/Z file="e:/projekte/mies-igor/Packages/Python/limspath_from_cellname.py", array={"paths", results}, args=list

Packages/Python/limspath_from_cellname.py:31

• Use is not None for None checks. result != None can be fooled by custom equality implementations and is not idiomatic Python.

            if result != None:

Packages/Python/limspath_from_cellname.py:62

• These commented-out exit-code lines are misleading: the script currently exits with 0 by default, so the note about “Not return zero” is inconsistent with the actual behavior. Please either implement the intended non-zero behavior or remove the stale comment.

    # @todo Not return zero here due to WM bug #8570
    # sys.exit(0)

Packages/MIES/MIES_Python.ipf:8

• This new IPF file is missing the standard Doxygen file header (e.g. /// @file / /// @brief) used throughout MIES, which makes generated documentation incomplete.

This issue also appears in the following locations of the same file:

• line 21
• line 29

#ifdef AUTOMATED_TESTING
#pragma ModuleName = MIES_PY
#endif // AUTOMATED_TESTING

Function/S PY_GetPackageFolder(string packageName)

Packages/MIES/MIES_AnalysisBrowser.ipf:3287

• AB_GatherFoldersFromLIMS is currently an empty function with no documentation or behavior. If it is meant as a placeholder, add a clear TODO and mark the parameter unused; otherwise, remove it until implemented to avoid dead code.

Function AB_GatherFoldersFromLIMS(WAVE/T cellIDs)

End

tools/nwb-read-tests/nwbv2-read-test.py:35

• Using check=True here raises CalledProcessError and bypasses the script’s previous error-path that printed validator output. That makes failures harder to diagnose in CI. Consider returning a non-zero code while still emitting the tool output to stderr on failure.

    comp = run(
        ["pynwb-validate", path],
        stdout=PIPE,
        stderr=STDOUT,
        text=True,

[Copilot]

Pull request overview

Copilot reviewed 17 out of 18 changed files in this pull request and generated 6 comments.

Suppressed comments (7)

Packages/Python/limspath_from_cellname.py:15

• Database connection parameters are hardcoded in the repository. This exposes credentials and makes it hard to run in different environments; read connection info from environment variables (or a secure settings mechanism) instead.

            user="limsreader",
            host="limsdb2",
            database="lims2",
            password="limsro",
            port=5432,

Packages/Python/limspath_from_cellname.py:32

• Use is not None for None checks; != None can be incorrect when objects define custom equality.

            result = cur.fetchone()

            if result != None:
                paths.append(result[0])

Packages/MIES/MIES_Python.ipf:49

• The venv is created with Python 3.14, but tools/lims-query/requirements.txt was compiled for Python 3.11 (see header comment) and other tooling in this repo also uses 3.11 for uv-managed environments. Pinning a different interpreter version is likely to cause dependency resolution/install issues.

	pyVersion   = "3.14"

Packages/MIES/MIES_Python.ipf:59

• Package installation happens via uv pip install ... before the created venv is activated/selected. Without an activated venv (or an explicit target interpreter), this can install into an unintended environment and leave the venv empty.

	sprintf cmd, "%s pip install --no-config --require-hashes --exact --directory \"%s\" --requirements \"%srequirements.txt\"", HFSPathToWindows(uv), HFSPathToWindows(pkgFolder), HFSPathToWindows(reqFolder)
	print cmd
	ExecuteScriptText/B/Z cmd
	ASSERT(!V_Flag, "Could not fill the python environment")

Packages/MIES/MIES_Python.ipf:130

• dostuff() looks like a local debug helper and is currently part of the shipped procedure file. Please remove it (or hide it behind a debug/testing define) to avoid leaking ad-hoc commands into production code.

Function dostuff()

	WAVE/T results = PY_FetchFilesFromLims({"fake"})
	print results
End

Packages/MIES/MIES_AnalysisBrowser.ipf:3288

• AB_GatherFoldersFromLIMS is currently an empty public function, which is easy to accidentally call and silently do nothing. Either remove it until implemented or make the unimplemented state explicit (e.g., via FATAL_ERROR).

Function AB_GatherFoldersFromLIMS(WAVE/T cellIDs)

End

tools/nwb-read-tests/nwbv2-read-test.py:119

• Catching Exception and then raise will print a stack trace after printing the error message, which is noisy for CI/test output. If the intent is just a non-zero exit with a concise message, exit with code 1 instead of re-raising.

    except Exception as e:
        print(e, file=sys.stderr)
        raise