LiTEx-NLI-extension

Dataset and Implementation of the ACL 2026 Findings paper "Agree, Disagree, Explain: Decomposing Human Label Variation in NLI through the Lens of Explanations"

Overview

Natural language inference annotations often vary across humans. This project studies that variation by looking not only at NLI labels, but also at the explanations annotators provide for their decisions.

The released data includes explanation-level annotations for two NLI sources:

• annotations_varierr.jsonl: VariErr examples annotated with NLI labels, free-text explanations, explanation categories, and annotator_ids.
• annotations_livenli.jsonl: LiveNLI examples annotated with labels, explanations, explanation categories, and worker_ids.

The analysis notebook summarizes label variation, explanation-category distributions, category-conditioned label distributions, pair-level variation, and annotator-level patterns.

Repository Contents

• annotator_tracking.ipynb: Reproducible analysis notebook using repository-relative paths.
• annotations_varierr.jsonl: Public VariErr annotation file.
• annotations_livenli.jsonl: Public LiveNLI annotation file.
• Agree_Disagree_Explain.pdf: Paper PDF.

Generated tables and figures are written to outputs/ when the notebook is run.

Reproducing the Analysis

Install the Python dependencies:

pip install -r requirements.txt

Then open and run:

jupyter notebook annotator_tracking.ipynb

The notebook assumes it is run from the repository root. It writes CSV summaries and figures such as label distributions, explanation-category distributions, category-conditioned label distributions, pair-level variation summaries, and annotator-level distributions to outputs/.

Citation

If you use this code&data, please cite the papers below:

Agree, Disagree, Explain: Decomposing Human Label Variation in NLI through the Lens of Explanations

@article{DBLP:journals/corr/abs-2510-16458,
  author       = {Pingjun Hong and
                  Beiduo Chen and
                  Siyao Peng and
                  Marie{-}Catherine de Marneffe and
                  Benjamin Roth and
                  Barbara Plank},
  title        = {Agree, Disagree, Explain: Decomposing Human Label Variation in {NLI}
                  through the Lens of Explanations},
  journal      = {CoRR},
  volume       = {abs/2510.16458},
  year         = {2025},
  url          = {https://doi.org/10.48550/arXiv.2510.16458},
  doi          = {10.48550/ARXIV.2510.16458},
  eprinttype   = {arXiv},
  eprint       = {2510.16458},
  timestamp    = {Sat, 15 Nov 2025 15:31:37 +0100},
  biburl       = {https://dblp.org/rec/journals/corr/abs-2510-16458.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}