Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 9 additions & 7 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,17 +6,19 @@ All notable changes to this project will be documented in this file.

### Added

- Introduce `MetricReader` for iterating metrics from any text-IO source (#41)
- Add an `encoding` kwarg on `MetricReader.open` and `MetricWriter.open` (#43)
- Add `RecordModel`, a mixin-free base class providing the default tabular read/write surface without the `fgmetric.converters` mixins; `Metric` now subclasses it (#10)
- Introduce `ModelReader` for iterating records from any text-IO source (#41)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion
Consider adding **Breaking:** prefixes where relevant.

- Add an `encoding` kwarg on `ModelReader.open` and `ModelWriter.open` (#43)
- Transparent gzip/bz2/xz support via `xopen` (#44)
- Infer the delimiter from the file extension in `MetricReader.open`/`MetricWriter.open`; pass `delimiter=` to override (#61)
- Infer the delimiter from the file extension in `ModelReader.open`/`ModelWriter.open`; pass `delimiter=` to override (#61)

### Changed

- `MetricWriter` is IO-first; open a path with `MetricWriter.open()` rather than passing it to the constructor (#42)
- `Metric.read` is now a thin wrapper over `MetricReader.open` and reads eagerly, returning a `list` instead of a lazy generator; it accepts `str` paths in addition to `Path` and gains transparent decompression and the `encoding` kwarg. Use `MetricReader.open` to stream metrics without holding them all in memory (#62)
- `MetricReader.open` and `MetricWriter.open` eagerly validate that the path is readable/writable on context entry (#66)
- `MetricWriter.open` now refuses to overwrite an existing file by default, raising `FileExistsError`; pass `overwrite=True` to truncate and overwrite it (#69)
- **Breaking:** Rename `MetricReader`/`MetricWriter` to `ModelReader`/`ModelWriter`; the IO classes now read and write any `RecordModel` subclass, not just `Metric` (#10)
- **Breaking:** `ModelWriter` is IO-first; open a path with `ModelWriter.open()` rather than passing it to the constructor (#42)
- **Breaking:** `Metric.read` is now a thin wrapper over `ModelReader.open` and reads eagerly, returning a `list` instead of a lazy generator; it accepts `str` paths in addition to `Path` and gains transparent decompression and the `encoding` kwarg. Use `ModelReader.open` to stream metrics without holding them all in memory (#62)
- `ModelReader.open` and `ModelWriter.open` eagerly validate that the path is readable/writable on context entry (#66)
- **Breaking:** `ModelWriter.open` now refuses to overwrite an existing file by default, raising `FileExistsError`; pass `overwrite=True` to truncate and overwrite it (#69)

## [0.3.0] - 2026-05-12

Expand Down
29 changes: 21 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ for metric in AlignmentMetric.read(path):
Define a class to represent each row:

```python
from fgmetric import Metric, MetricReader, MetricWriter
from fgmetric import Metric, ModelReader, ModelWriter


class AlignmentMetric(Metric):
Expand All @@ -100,25 +100,25 @@ metrics = [
AlignmentMetric(read_name="read1", mapping_quality=60),
AlignmentMetric(read_name="read2", mapping_quality=30, is_duplicate=True),
]
with MetricWriter.open(AlignmentMetric, "output.tsv") as writer:
with ModelWriter.open(AlignmentMetric, "output.tsv") as writer:
writer.writeall(metrics)
```

`Metric.read()` reads the whole file into a list. To stream metrics one at a time without holding them all in memory, use `MetricReader.open()`:
`Metric.read()` reads the whole file into a list. To stream metrics one at a time without holding them all in memory, use `ModelReader.open()`:

```python
# Streaming from a path
with MetricReader.open(AlignmentMetric, "alignments.tsv") as reader:
with ModelReader.open(AlignmentMetric, "alignments.tsv") as reader:
for metric in reader:
print(f"{metric.read_name}: MQ={metric.mapping_quality}")
```

To read from an open file handle or any other text IO source (e.g. `StringIO`), use `MetricReader` directly:
To read from an open file handle or any other text IO source (e.g. `StringIO`), use `ModelReader` directly:

```python
# Reading from an IO source
with open("alignments.tsv") as handle:
reader = MetricReader(AlignmentMetric, handle)
reader = ModelReader(AlignmentMetric, handle)
for metric in reader:
print(f"{metric.read_name}: MQ={metric.mapping_quality}")
```
Expand Down Expand Up @@ -149,7 +149,7 @@ Pass `delimiter=` to override inference, or to read or write a file whose extens

```python
# Pipe-delimited file with an unrecognized extension
with MetricWriter.open(MyMetric, "output.dat", delimiter="|") as writer:
with ModelWriter.open(MyMetric, "output.dat", delimiter="|") as writer:
...
```

Expand All @@ -161,7 +161,7 @@ Reading and writing transparently handle gzip, bzip2, and xz files based on the
for metric in AlignmentMetric.read("alignments.tsv.gz"):
...

with MetricWriter.open(AlignmentMetric, "output.tsv.bz2") as writer:
with ModelWriter.open(AlignmentMetric, "output.tsv.bz2") as writer:
writer.writeall(metrics)
```

Expand Down Expand Up @@ -229,6 +229,19 @@ class BaseCountMetric(Metric):
# BaseCountMetric(position=1, counts=Counter({Base.A: 10, Base.C: 5, ...}))
```

### Plain records (`RecordModel`)

`Metric` is a batteries-included subclass of `RecordModel`, the mixin-free base. `RecordModel` does the default tabular parsing only — one model field per column — without the converter behaviors above (empty-string-to-`None`, delimited lists, counter pivots). Subclass it directly when you want plain field-per-column parsing and nothing else; `ModelReader`, `ModelWriter`, and the inherited `read()` work on any `RecordModel`.

```python
from fgmetric import RecordModel


class SampleRecord(RecordModel):
name: str
note: str | None # an empty cell stays "" here; on a Metric it would become None
```

## Contributing

See the [contributing guide](./CONTRIBUTING.md) for development setup and testing instructions.
10 changes: 6 additions & 4 deletions fgmetric/__init__.py
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
from fgmetric.metric import Metric
from fgmetric.metric_reader import MetricReader
from fgmetric.metric_writer import MetricWriter
from fgmetric.model_reader import ModelReader
from fgmetric.model_writer import ModelWriter
from fgmetric.record_model import RecordModel

__all__ = [
"Metric",
"MetricReader",
"MetricWriter",
"ModelReader",
"ModelWriter",
"RecordModel",
]
45 changes: 43 additions & 2 deletions fgmetric/converters/_counter_pivot_table.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@
from typing import final
from typing import get_args

from pydantic import BaseModel
from pydantic import SerializationInfo
from pydantic import SerializerFunctionWrapHandler
from pydantic import model_serializer
Expand All @@ -13,9 +12,15 @@

from fgmetric._typing_extensions import is_counter
from fgmetric._typing_extensions import is_optional
from fgmetric.record_model import RecordModel


class CounterPivotTable(BaseModel):
# NB: This mixin inherits `RecordModel` rather than `BaseModel` (unlike the other converters) so
# that `_default_header_fieldnames` is in scope for its `_header_fieldnames` override. It is the
# only converter that rewrites the on-disk column set. Any future converter that also overrides
# `_header_fieldnames` should build on `_default_header_fieldnames` (not `super()`) so the result
# does not depend on MRO ordering.
class CounterPivotTable(RecordModel):
"""
A mixin to support pivot table representations of Counters.

Expand Down Expand Up @@ -201,6 +206,42 @@ def _get_counter_enum(cls) -> type[StrEnum] | None: # noqa: C901

return enum_cls

@final
@classmethod
def _header_fieldnames(cls) -> list[str]:
"""
Return the header fieldnames with the Counter field pivoted into per-member columns.

Overrides `RecordModel._header_fieldnames` to match this mixin's wide serialization: the
single `Counter[T]` field has no on-disk column of its own, so it is dropped from the
default header and replaced by one column per enum member (in enum declaration order),
matching the output of `_pivot_counter_values`.

Returns:
The list of fieldnames to use as the header row.

Example:
Given a model with ``name: str`` and ``counts: Counter[Color]`` where
``Color`` has members ``RED``, ``GREEN``, ``BLUE``:

```python
cls._header_fieldnames()
# -> ["name", "red", "green", "blue"]
```
"""
default_header = cls._default_header_fieldnames()

if cls._counter_fieldname is None:
# Short circuit if we don't have a Counter field
return default_header

# The Counter field has no alias (aliases are rejected for Counter fields), so its
# serialized name equals its field name; drop it from the default header and replace it
# with one column per enum member.
assert cls._counter_enum is not None # not None iff _counter_fieldname is not None
non_counter = [name for name in default_header if name != cls._counter_fieldname]
return non_counter + [member.value for member in cls._counter_enum]

@final
@model_validator(mode="before")
@classmethod
Expand Down
137 changes: 13 additions & 124 deletions fgmetric/metric.py
Original file line number Diff line number Diff line change
@@ -1,43 +1,41 @@
from abc import ABC
from collections.abc import Sequence
from pathlib import Path
from typing import ClassVar
from typing import Self

from pydantic import BaseModel

from fgmetric.converters import CounterPivotTable
from fgmetric.converters import DelimitedList
from fgmetric.converters import NullSentinels
from fgmetric.metric_reader import MetricReader
from fgmetric.record_model import RecordModel


class Metric(
DelimitedList,
CounterPivotTable,
NullSentinels,
BaseModel,
ABC,
RecordModel,
):
"""
Abstract base class for defining structured "metric" data models.

This class combines Pydantic's `BaseModel` with `ABC` to provide a foundation for creating
type-safe metric classes that can be easily read from and written to delimited text files (e.g.,
TSV, CSV). It leverages Pydantic's automatic validation and type conversion while providing
convenient class methods for parsing metrics from files.
`Metric` is a `RecordModel` with the default `fgmetric.converters` mixins layered on. It
provides the same file-reading and -writing surface as `RecordModel` (see `RecordModel.read`,
`ModelReader`, and `ModelWriter`) plus the conversion behaviors below. Subclasses should define
their fields using Pydantic field annotations.

Metrics are delimited files containing a header and zero or more rows for metric values. When
using a `Metric` to read a delimited file, the `Metric`'s fields correspond to the columns and
header of the file. Subclasses should define their fields using Pydantic field annotations.
header of the file.

Relative to a plain `RecordModel`, `Metric` adds the following custom
serialization/deserialization behaviors:

`Metric` includes the following custom serialization/deserialization behaviors:
1. **Null sentinels.** Empty strings in Optional fields are converted to `None` before field
validation. The sentinel values converted to `None` can be overridden by the
`null_sentinels` class variable. See `NullSentinels`.
2. **Delimited lists.** Any field typed as `list[T]` will be parsed from and serialized to a
delimited string. The list delimiter may be controlled by the `collection_delimiter` class
variable.
3. **Counter pivot tables.** A single field typed as `Counter[T]` (where `T` is a `StrEnum`)
is pivoted into one column per enum member during serialization and folded back during
validation.

Class Variables:
collection_delimiter: A single-character delimiter used to split and join `list` fields
Expand All @@ -59,112 +57,3 @@ class AlignmentMetric(Metric):
"""

null_sentinels: ClassVar[frozenset[str]] = frozenset({""})

@classmethod
def read(
cls,
path: Path | str,
# NB: these defaults mirror `MetricReader.open()`; keep them in sync.
delimiter: str = "\t",
fieldnames: Sequence[str] | None = None,
encoding: str = "utf-8-sig",
) -> list[Self]:
"""
Read all Metric instances from a file path.

Eager wrapper around `MetricReader.open()`: the file is opened, parsed, and closed
before this method returns, collecting every row into a list. Because reading happens
up front, IO and validation errors surface here at the call site rather than partway
through iteration.

See `MetricReader.open()` for the file-handling behavior shared by both APIs - encoding,
automatic decompression of `.gz`/`.bz2`/`.xz` files, and how `fieldnames` selects
header vs. headerless parsing. To stream a large file without holding every row in
memory, or to read from an already-open handle or other text IO source, use
`MetricReader` directly.

Args:
path: Filesystem path to the input file.
delimiter: The input file delimiter.
fieldnames: Optional sequence of field names. If provided, the input is treated as
headerless and these names are used as the column headers.
encoding: The text encoding used to decode the file.

Returns:
A list of instances of the calling Metric subclass, one per data row.

Raises:
FileNotFoundError: If `path` does not exist.
LookupError: If `encoding` is not a known codec.
UnicodeDecodeError: If the file's bytes cannot be decoded using `encoding`.
ValueError: If `fieldnames` is supplied and the first row matches it (a likely
forgotten header).
ValidationError: If a row fails `Metric` validation, e.g. a missing required field
or a value of the wrong type.

Example:
`read` is eager and returns a list, so the whole file is available at once:

```python
metrics = AlignmentMetric.read("metrics.txt")
print(f"read {len(metrics)} rows")
for m in metrics:
print(m.read_name, m.mapping_quality)
```
"""
with MetricReader.open(
cls,
path,
delimiter=delimiter,
fieldnames=fieldnames,
encoding=encoding,
) as reader:
return list(reader)

@classmethod
def _header_fieldnames(cls) -> list[str]:
"""
Return the fieldnames to use as a header row when writing metrics to a file.

This method is used by `MetricWriter` to construct the underlying `csv.DictWriter`.
It returns the fieldnames that will appear in serialized output, which may differ from
the model's field names when aliases are used.

Note:
This method is deliberately not used during reading/validation; see
`MetricReader` for the headerless-input path.

Returns:
The list of fieldnames to use as the header row.

Example:
Given a model with ``name: str`` and ``counts: Counter[Color]`` where
``Color`` has members ``RED``, ``GREEN``, ``BLUE``:

```python
cls._header_fieldnames()
# -> ["name", "red", "green", "blue"]
```
"""
# TODO: support returning the set of fields that would be constructed if the class has a
# custom model serializer

# Resolve each field to the key it serializes under (its alias, when one is set), so the
# header matches the keys produced by `model_dump(by_alias=True)`. The Counter field, if
# present, is dropped here and replaced by its pivoted enum-member columns below.
fieldnames: list[str]
fieldnames = [
info.serialization_alias or info.alias or name
for name, info in cls.model_fields.items()
if name != cls._counter_fieldname
]

if cls._counter_fieldname is None:
# Short circuit if we don't have a Counter field
return fieldnames

# Add the enum members
assert cls._counter_enum is not None # type narrowing
fieldnames += [member.value for member in cls._counter_enum]

return fieldnames
Loading