test: add pytest suite + CI; fix empty-shard crash in JsonLineDataset

[ajinkyajawale14499]

What

Adds the repository's first automated test suite and CI, plus a small fix that the tests surfaced.

Tests (CPU-only, no GPU/model required)

• tests/test_config.py — ConfigNode attribute access, the recursive to_config_node / config_to_plain_dict converters, jsonable, CustomJSONEncoder (function/type/torch.dtype/Path/Namespace/ConfigNode), the --opts override parser (including the KeyError/TypeError paths), and load_config.
• tests/test_jsonl_dataset.py — JsonLineDataset mmap line indexing: length with and without a trailing newline, per-record decoding, multi-file global indexing, out-of-range bounds, and the on-disk line-start cache (write + reuse). CACHE_DIR is redirected to a temp dir so tests are hermetic.
• tests/test_sampling.py — deterministic surfaces of the sampling helpers: greedy/temperature-0 and softmax logits_to_probs, sample_tokens, gather_token_probs, sample_from_probs, and sample_residual (including the matched-distribution fallback).

CI

• .github/workflows/ci.yml runs pytest on Python 3.10 and 3.12 with CPU-only torch, so it stays fast and needs no CUDA/triton. A root conftest.py keeps the project importable without an editable install.

Fix

• While adding the dataset tests, JsonLineDataset was found to crash when a shard is 0 bytes: ValueError: cannot mmap an empty file. Empty shards are now treated as 0 records (with a matching regression test, including an empty shard alongside populated files).

Verification

All 42 tests pass locally (python -m pytest tests). Happy to adjust scope, naming, or the Python matrix to match your preferences.

[rajpratham1]

Great addition! The new pytest suite significantly improves confidence in the core utilities, and the regression test for the empty JSONL shard fixes a real edge case that could otherwise crash dataset initialization. The CI workflow is lightweight and appropriate for CPU-only validation. Nice work improving both reliability and maintainability.