Migrate all modules off docval + real signatures for generated classes (Phases 3-4 of #1129)

CHANGELOG.md

@@ -7,6 +7,12 @@
 - `hdmf.utils.get_docval` now also works on functions with type-hinted signatures: docval-compatible argument specs are synthesized from the signature, type hints, and Google-style docstring. This keeps the downstream splice pattern `@docval(*get_docval(Parent.__init__, ...))` working as HDMF (and downstream libraries) migrate off `@docval` module by module. Existing `@docval`-decorated functions are unaffected. @bendichter [#TBD]
 - New required dependencies: `beartype`, `numpydantic`, and `docstring_parser`. @bendichter [#TBD]
 - Migrated `hdmf.data_utils`, `hdmf.query`, `hdmf.validate.validator`, `hdmf.common.sparse`, `hdmf.common.resources`, and `hdmf.common.hierarchicaltable` off `@docval` to type-hinted signatures with `@validated` (includes `GenericDataChunkIterator`, `DataChunkIterator`, `DataChunk`, `DataIO`, `CSRMatrix`, `HERD`, and the validator classes). Downstream code is unaffected: `get_docval` on these functions returns equivalent specs synthesized from the type hints, so patterns like `@docval(*get_docval(DataIO.__init__), ...)` (used by `H5DataIO` and extensions) keep working, positional/keyword calling conventions are unchanged, and error messages keep the docval format. @bendichter [#TBD]
+- Migrated `hdmf.common.table` (`VectorData`, `VectorIndex`, `ElementIdentifiers`, `DynamicTable`, `DynamicTableRegion`, `EnumData`, `MeaningsTable`), `hdmf.common.alignedtable`, and `hdmf.common.multi` off `@docval` to type-hinted signatures. The three `AlignedDynamicTable` methods that splice parent specs via `@docval(*get_docval(...))` intentionally remain on `@docval` and consume specs synthesized from the migrated parents. @bendichter [#TBD]
+- Migrated the `hdmf.spec` subsystem (`spec.py`, `namespace.py`, `catalog.py`, `write.py`) off `@docval` to type-hinted signatures. The shared module-level spec-argument lists (`_attr_args`, `_group_args`, ...) are inlined into the signatures and removed. Note that error messages for *missing or unrecognized arguments* on migrated functions now use Python's native wording (e.g. "missing a required argument: 'doc'" instead of "missing argument 'doc'"); type and value error messages are unchanged. @bendichter [#TBD]
+- Migrated `hdmf.container` (`AbstractContainer`, `Container`, `Data`, `DataRegion`, `Table` base classes) off `@docval` to type-hinted signatures. The dynamically composed decorators in `MultiContainerInterface` (`__make_get`/`__make_add`/`__make_create`/...) and the `Row`/`Table` row-class machinery intentionally remain on `@docval` until it is removed. @bendichter [#TBD]
+- Migrated the remaining runtime modules off `@docval`: `hdmf.build` (`builders`, `manager`, `objectmapper`, `errors`), `hdmf.backends` (`io`, `utils`, `hdf5.h5_utils` including `H5DataIO`, `hdf5.h5tools` including `HDF5IO`), `hdmf.common.__init__`, `hdmf.common.io.table`, `hdmf.validate.errors`, and `hdmf.term_set`. The only remaining `@docval` uses are the spec-splicing composition sites (which consume synthesized specs) and `build/classgenerator.py`, which generates constructors for schema-derived classes and migrates in a later phase. @bendichter [#TBD]
+- Schema-generated classes (from `get_class`/`TypeMap.get_dt_container_cls`, including all classes auto-generated by extensions) and the dynamically composed `MultiContainerInterface` methods (`add_*`, `create_*`, `get_*`, `__init__`) now have **real type-hinted signatures** instead of `(self, **kwargs)`, visible to IDEs, `help()`, and `inspect.signature`. They are built by the new `hdmf.typing.signature_function`, which converts docval-style argument specs into signatures at runtime (falling back to legacy `@docval` for non-identifier argument names). Their docstrings no longer include docval's synthetic signature line. @bendichter [#TBD]
+- Added a `py.typed` marker: hdmf now declares inline type information to static type checkers. Annotation coverage is partial and will grow. @bendichter [#TBD]
 
 ## HDMF 6.1.0 (June 25, 2026)
 

docs/source/conf.py

@@ -101,6 +101,8 @@
     # Is[is_array_data], Is[has_shape_...], etc., which are not documented targets
     ('py:class', r'beartype\..*'),
     ('py:class', r'(is|has_shape)_\w*'),
+    ('py:class', r'h5py\._hl\..*'),
+    ('py:class', r'types\.(Lambda|Function)Type'),
 ]
 
 suppress_warnings = ["config.cache"]

src/hdmf/backends/hdf5/h5_utils.py

@@ -19,7 +19,9 @@
 from ...data_utils import DataIO, AbstractDataChunkIterator, append_data
 from ...query import HDMFDataset, ReferenceResolver, ContainerResolver, BuilderResolver
 from ...spec import SpecWriter, SpecReader
-from ...utils import docval, getargs, popargs, get_docval, get_data_shape
+from ...utils import get_docval, get_data_shape
+from typing import Any
+from ...typing import Bool, Int, TypeName, validated
 
 
 class HDF5IODataChunkIteratorQueue(deque):
@@ -83,12 +85,16 @@ def append(self, dataset, data):
 
 
 class H5Dataset(HDMFDataset):
-    @docval({'name': 'dataset', 'type': Dataset, 'doc': 'the HDF5 file lazily evaluate'},
-            {'name': 'io', 'type': 'hdmf.backends.hdf5.h5tools.HDF5IO',
-             'doc': 'the IO object that was used to read the underlying dataset'})
-    def __init__(self, **kwargs):
-        self.__io = popargs('io', kwargs)
-        super().__init__(**kwargs)
+    @validated
+    def __init__(self, dataset: Dataset, io: TypeName['hdmf.backends.hdf5.h5tools.HDF5IO']):  # noqa: F821
+        """Initialize this object.
+
+        Args:
+            dataset: the HDF5 file lazily evaluate
+            io: the IO object that was used to read the underlying dataset
+        """
+        self.__io = io
+        super().__init__(dataset=dataset)
 
     @property
     def io(self):
@@ -183,14 +189,16 @@ def get_object(self, h5obj):
 
 class AbstractH5TableDataset(DatasetOfReferences):
 
-    @docval({'name': 'dataset', 'type': Dataset, 'doc': 'the HDF5 file lazily evaluate'},
-            {'name': 'io', 'type': 'hdmf.backends.hdf5.h5tools.HDF5IO',
-             'doc': 'the IO object that was used to read the underlying dataset'},
-            {'name': 'types', 'type': (list, tuple),
-             'doc': 'the IO object that was used to read the underlying dataset'})
-    def __init__(self, **kwargs):
-        types = popargs('types', kwargs)
-        super().__init__(**kwargs)
+    @validated
+    def __init__(self, dataset: Dataset, io: TypeName['hdmf.backends.hdf5.h5tools.HDF5IO'], types: list | tuple):  # noqa: F821
+        """Initialize this object.
+
+        Args:
+            dataset: the HDF5 file lazily evaluate
+            io: the IO object that was used to read the underlying dataset
+            types: the types of the elements in the dataset
+        """
+        super().__init__(dataset=dataset, io=io)
         self.__refgetters = dict()
         for i, t in enumerate(types):
             if t is Reference:
@@ -317,9 +325,14 @@ class H5SpecWriter(SpecWriter):
 
     __str_type = special_dtype(vlen=str)
 
-    @docval({'name': 'group', 'type': Group, 'doc': 'the HDF5 file to write specs to'})
-    def __init__(self, **kwargs):
-        self.__group = getargs('group', kwargs)
+    @validated
+    def __init__(self, group: Group):
+        """Initialize this object.
+
+        Args:
+            group: the HDF5 file to write specs to
+        """
+        self.__group = group
 
     @staticmethod
     def stringify(spec):
@@ -344,9 +357,14 @@ def write_namespace(self, namespace, path):
 class H5SpecReader(SpecReader):
     """Class that reads cached JSON-formatted namespace and spec data from an HDF5 group."""
 
-    @docval({'name': 'group', 'type': Group, 'doc': 'the HDF5 group to read specs from'})
-    def __init__(self, **kwargs):
-        self.__group = popargs('group', kwargs)
+    @validated
+    def __init__(self, group: Group):
+        """Initialize this object.
+
+        Args:
+            group: the HDF5 group to read specs from
+        """
+        self.__group = group
         source = "%s:%s" % (os.path.abspath(self.__group.file.name), self.__group.name)
         super().__init__(source=source)
         self.__cache = None
@@ -378,78 +396,56 @@ class H5DataIO(DataIO):
     for data arrays.
     """
 
-    @docval({'name': 'data',
-             'type': (np.ndarray, list, tuple, Dataset, Iterable),
-             'doc': 'the data to be written. NOTE: If an h5py.Dataset is used, all other settings but link_data' +
-                    ' will be ignored as the dataset will either be linked to or copied as is in H5DataIO.',
-             'default': None},
-            {'name': 'maxshape',
-             'type': tuple,
-             'doc': 'Dataset will be resizable up to this shape (Tuple). Automatically enables chunking.' +
-                    'Use None for the axes you want to be unlimited.',
-             'default': None},
-            {'name': 'chunks',
-             'type': (bool, tuple),
-             'doc': 'Chunk shape or True to enable auto-chunking',
-             'default': None},
-            {'name': 'compression',
-             'type': (str, bool, int),
-             'doc': 'Compression strategy. If a bool is given, then gzip compression will be used by default.' +
-                    'http://docs.h5py.org/en/latest/high/dataset.html#dataset-compression',
-             'default': None},
-            {'name': 'compression_opts',
-             'type': (int, tuple),
-             'doc': 'Parameter for compression filter',
-             'default': None},
-            {'name': 'fillvalue',
-             'type': None,
-             'doc': 'Value to be returned when reading uninitialized parts of the dataset',
-             'default': None},
-            {'name': 'shuffle',
-             'type': bool,
-             'doc': 'Enable shuffle I/O filter. http://docs.h5py.org/en/latest/high/dataset.html#dataset-shuffle',
-             'default': None},
-            {'name': 'fletcher32',
-             'type': bool,
-             'doc': 'Enable fletcher32 checksum. http://docs.h5py.org/en/latest/high/dataset.html#dataset-fletcher32',
-             'default': None},
-            {'name': 'link_data',
-             'type': bool,
-             'doc': 'If data is an h5py.Dataset should it be linked to or copied. NOTE: This parameter is only ' +
-                    'allowed if data is an h5py.Dataset',
-             'default': False},
-            {'name': 'allow_plugin_filters',
-             'type': bool,
-             'doc': 'Enable passing dynamically loaded filters as compression parameter',
-             'default': False},
-            {'name': 'shape',
-             'type': tuple,
-             'doc': 'the shape of the new dataset, used only if data is None',
-             'default': None},
-            {'name': 'dtype',
-             'type': (str, type, np.dtype),
-             'doc': 'the data type of the new dataset, used only if data is None',
-             'default': None}
-            )
-    def __init__(self, **kwargs):
-        # Get the list of I/O options that user has passed in
-        ioarg_names = [name for name in kwargs.keys() if name not in ['data', 'link_data', 'allow_plugin_filters',
-                                                                      'dtype', 'shape']]
-
-        # Remove the ioargs from kwargs
-        ioarg_values = [popargs(argname, kwargs) for argname in ioarg_names]
+    @validated
+    def __init__(self,
+                 data: np.ndarray | list | tuple | Dataset | Iterable | None = None,
+                 maxshape: tuple | None = None,
+                 chunks: Bool | tuple | None = None,
+                 compression: str | Bool | Int | None = None,
+                 compression_opts: Int | tuple | None = None,
+                 fillvalue: Any = None,
+                 shuffle: Bool | None = None,
+                 fletcher32: Bool | None = None,
+                 link_data: Bool = False,
+                 allow_plugin_filters: Bool = False,
+                 shape: tuple | None = None,
+                 dtype: str | type | np.dtype | None = None):
+        """Initialize this object.
+
+        Args:
+            data: the data to be written. NOTE: If an h5py.Dataset is used, all other settings but link_data
+                will be ignored as the dataset will either be linked to or copied as is in H5DataIO.
+            maxshape: Dataset will be resizable up to this shape (Tuple). Automatically enables chunking.
+                Use None for the axes you want to be unlimited.
+            chunks: Chunk shape or True to enable auto-chunking
+            compression: Compression strategy. If a bool is given, then gzip compression will be used by
+                default. http://docs.h5py.org/en/latest/high/dataset.html#dataset-compression
+            compression_opts: Parameter for compression filter
+            fillvalue: Value to be returned when reading uninitialized parts of the dataset
+            shuffle: Enable shuffle I/O filter. http://docs.h5py.org/en/latest/high/dataset.html#dataset-shuffle
+            fletcher32: Enable fletcher32 checksum. http://docs.h5py.org/en/latest/high/dataset.html#dataset-fletcher32
+            link_data: If data is an h5py.Dataset should it be linked to or copied. NOTE: This parameter is
+                only allowed if data is an h5py.Dataset
+            allow_plugin_filters: Enable passing dynamically loaded filters as compression parameter
+            shape: the shape of the new dataset, used only if data is None
+            dtype: the data type of the new dataset, used only if data is None
+        """
+        # the h5py io settings (everything except the DataIO/link args)
+        ioargs = dict(maxshape=maxshape, chunks=chunks, compression=compression,
+                      compression_opts=compression_opts, fillvalue=fillvalue, shuffle=shuffle,
+                      fletcher32=fletcher32)
         # Consume link_data parameter
-        self.__link_data = popargs('link_data', kwargs)
+        self.__link_data = link_data
         # Consume allow_plugin_filters parameter
-        self.__allow_plugin_filters = popargs('allow_plugin_filters', kwargs)
+        self.__allow_plugin_filters = allow_plugin_filters
         # Check for possible collision with other parameters
-        if not isinstance(getargs('data', kwargs), Dataset) and self.__link_data:
+        if not isinstance(data, Dataset) and self.__link_data:
             self.__link_data = False
             warnings.warn('link_data parameter in H5DataIO will be ignored', stacklevel=3)
         # Call the super constructor and consume the data parameter
-        super().__init__(**kwargs)
+        super().__init__(data=data, shape=shape, dtype=dtype)
         # Construct the dict with the io args, ignoring all options that were set to None
-        self.__iosettings = {k: v for k, v in zip(ioarg_names, ioarg_values) if v is not None}
+        self.__iosettings = {k: v for k, v in ioargs.items() if v is not None}
         if self.data is None:
             self.__iosettings['dtype'] = self.dtype
             self.__iosettings['shape'] = self.shape