[BUG] load_datasets() cannot use the generated Arrow cache correctly

[Nexround]

Describe the bug

The datasets library cannot use the generated Arrow cache correctly, seemingly due to a flaw in the internal hash symbol calculation logic.
The following code provides verification.
I am trying to locate the specific code position, and if there are further developments, I will update here.

Steps to reproduce the bug

python -c "
from datasets.load import LocalDatasetModuleFactory
mod = LocalDatasetModuleFactory('/cache/datasets/imagenet-1k')
dm = mod.get_module()
print('Hash:', dm.hash)
"
Resolving data files: 100%|█████████| 294/294 [00:00<00:00, 23895.46it/s]
Resolving data files: 100%|██████████| 28/28 [00:00<00:00, 221168.57it/s]
Hash: 9e9925e0f7d48775
python -c "
from datasets.load import LocalDatasetModuleFactory
mod = LocalDatasetModuleFactory('/cache/datasets/imagenet-1k')
dm = mod.get_module()
print('Hash:', dm.hash)
"
Resolving data files: 100%|█████████| 294/294 [00:00<00:00, 26252.91it/s]
Resolving data files: 100%|██████████| 28/28 [00:00<00:00, 188205.95it/s]
Hash: 9af23f3d5d488660

Expected behavior

The expected behavior is that the above code should give the same hash value.

Environment info

• datasets version: 4.6.1
• Platform: Linux-6.8.0-100-generic-x86_64-with-glibc2.39
• Python version: 3.12.12
• huggingface_hub version: 1.5.0
• PyArrow version: 23.0.1
• Pandas version: 3.0.1
• fsspec version: 2026.2.0

[Nexround]

PYTHONHASHSEED=1 uv run python -c
"
from datasets.load import LocalDatasetModuleFactory
mod = LocalDatasetModuleFactory('/cache/datasets/imagenet-1k')
dm = mod.get_module()
print('Hash:', dm.hash)
"
Resolving data files: 100%|█████████| 294/294 [00:00<00:00, 20584.68it/s]
Resolving data files: 100%|██████████| 28/28 [00:00<00:00, 118149.41it/s]
Hash: 65e572373d037b39
PYTHONHASHSEED=1 uv run python -c "
from datasets.load import LocalDatasetModuleFactory
mod = LocalDatasetModuleFactory('/cache/datasets/imagenet-1k')
dm = mod.get_module()
print('Hash:', dm.hash)
"
Resolving data files: 100%|█████████| 294/294 [00:00<00:00, 24316.24it/s]
Resolving data files: 100%|██████████| 28/28 [00:00<00:00, 171948.04it/s]
Hash: 65e572373d037b39

When PYTHONHASHSEED is set to a fixed value, the issue is resolved. So this should be related to internal calls concerning Python hash.

[lhoestq]

The hash comes from a pickle dump of dataset_info and builder config:

datasets/src/datasets/load.py

Line 486 in 81027be

hash = Hasher.hash({"dataset_infos": dataset_infos, "builder_configs": builder_configs})

The dataset_info is empty in your case, and the builder config doesn't change across session except maybe its data_files.

The data_files are made of DataFilesList objects which contain

• the data file paths as strings
• some origin metadata to uniquely identify the files (e.g. with the latest modified time for local files)

So in practice the hash should only change if the data files changes (either the list of file paths or if a file was modified)

Not sure why the hash changes in your case or why the PYTHONHASHSEED would impact this tbh :/

[Nexround]

#8034 (comment)
Update my progress, hoping to help solve this issue:

datasets/src/datasets/packaged_modules/__init__.py

Line 129 in 81027be

_ALL_METADATA_EXTENSIONS = list({_ext for _exts in _MODULE_TO_METADATA_EXTENSIONS.values() for _ext in _exts})

This line uses a set comprehension to deduplicate extensions before converting to a list. In Python 3.6+, the iteration order of a set depends on PYTHONHASHSEED (which defaults to a random value per process). Therefore, the order of _ALL_METADATA_EXTENSIONS is non-deterministic across different process invocations.

1. _ALL_METADATA_EXTENSIONS has non-deterministic order
2. → _ALL_ALLOWED_EXTENSIONS = _ALL_EXTENSIONS + _ALL_METADATA_EXTENSIONS has non-deterministic trailing extensions
3. → DataFilesDict.from_patterns() passes allowed_extensions to DataFilesPatternsList
4. → DataFilesPatternsList.filter_extensions() appends these extensions to each pattern's allowed_extensions attribute
5. → BuilderConfig.data_files contains allowed_extensions with non-deterministic order
6. → In LocalDatasetModuleFactory.get_module(), Hasher.hash({"dataset_infos": ..., "builder_configs": ...}) performs dill.dumps() + xxhash on builder_configs (which includes data_files)
7. → The resulting hash differs on every run
8. → Cache directory path .../0.0.0/{hash}/ changes every time
9. → Cache is never reused; Parquet → Arrow conversion is re-executed on every run

Run the code below to observe the impact:

for i in 1 2 3 4 5; do
  python -c "from datasets.packaged_modules import _ALL_METADATA_EXTENSIONS; print(_ALL_METADATA_EXTENSIONS)"
done

Expected the same order every time, but actual orders differ (e.g., ['.manifest', '.idx', '.txn'] vs ['.manifest', '.txn', '.idx'])

To fix this, we can replace the set comprehension with sorted():

_ALL_METADATA_EXTENSIONS = sorted({_ext for _exts in _MODULE_TO_METADATA_EXTENSIONS.values() for _ext in _exts})

In my case, this fix works. If you agree with this fix, I'd be happy to create a pull request.

[lhoestq]

That makes sense ! yes feel free to open a PR

[lhoestq]

Moreover it seems we can set allowed_extensions=None when config_data_files is not None here

datasets/src/datasets/load.py

Lines 326 to 329 in 81027be

	config_data_files_dict = DataFilesPatternsDict.from_patterns(
	config_patterns,
	allowed_extensions=_ALL_ALLOWED_EXTENSIONS,
	)

i.e. when the data_files are specified in the README.md we don't need to apply an additional filter.

This should also make the hash less likely to change for such cases across datasets versions