fix(synthetic): align _SyntheticTextExamplesIterable with HuggingFace…

[arturofredes]

… datasets API

• Add n_shards property (alias of num_shards). HuggingFace IterableDataset expects n_shards; without it, loading synthetic data raises NotImplementedError.
• Change shard_data_sources(worker_id, num_workers) to match the base _BaseExamplesIterable API; the previous signature (num_shards, index, contiguous) caused 'unexpected keyword argument worker_id' when using multiple DataLoader workers. Fixes synthetic data loader when used with datasets.IterableDataset.

Summary

Details

• [ ]

Test Plan

Related Issues

• Resolves #

---

• "I certify that all code in this PR is my own, except as noted below."

Use of AI

• Includes AI-assisted code completion
• Includes code generated by an AI application
• Includes AI-generated tests (NOTE: AI written tests should have a docstring that includes ## WRITTEN BY AI ##)

[dbutenhof]

Can you tell us what version of the datasets package you're using?

The latest, 4.8.2, aligns with the code in the current GuideLLM. The change you're proposing seems to revert a datasets change made in October 2024, which renamed n_shards to num_shards (although it retained n_shards as an alias for backwards compatibility) and changed the signature of shard_data_sources from what you're proposing to what GuideLLM now uses.

GuideLLM currently depends on the datasets package without any minimum version, but apparently actually depends on a version somewhat later than you're using. (It appears that PR first appeared in datasets 3.1.0.)

Is there another constraint requiring you to use an old version of datasets? And, if not, can you try upgrading?

[sjmonson]

See above

[dbutenhof]

We've chosen to resolve this with #650, which explicitly requires a minimum datasets version of 4.1.0. This will avoid the possibility of accidentally running GuideLLM with the extremely old datasets version that caused your problem.