Dog-Whistle Detection and Disambiguation

Open-weight models for detecting, ingroup-categorizing, and explaining coded dog-whistle language on Reddit, built on the silent_signals corpus (Kruk et al., ACL 2024).

A dog-whistle is a phrase whose surface meaning is innocuous but signals an ingroup ideological position to those who recognise it. Examples range from the surface-ambiguous (absent fathers, globalist, urban) to the conventionally pejorative (SJW, libtard). This deniability defeats standard toxicity classifiers — and the gap is the motivation for this project.

Research questions

• RQ-A — Binary disambiguation: given a Reddit comment containing a dog-whistle phrase, is the phrase used in its coded or literal sense?
• RQ-B — Multiclass ingroup attribution: which of 17 ingroups does the dog-whistle target?
• RQ-C — Structured generation: produce parseable JSON with four fields — dog_whistle_root, ingroup, definition, explanation — given a comment.

Headline results

Task	Model	Metric	Score	Baseline
RQ-A	RoBERTa-base	F1 (held-out human-gold, n=124)	0.707 ± 0.015	0.701 (majority-class)
RQ-A	RoBERTa-base	Accuracy (held-out human-gold)	0.696 ± 0.010	0.540 (+15.6 pp)
RQ-A	RoBERTa-base	PR-AUC (held-out human-gold)	0.802 ± 0.010	0.50 (random)
RQ-B	RoBERTa-base	macro-F1 (root-stratified test)	0.353 ± 0.007	0.058 (majority-class)
RQ-B	RoBERTa-base	macro-F1 (leaky alt-split, comparison)	0.996	64 pp gap is the central RQ-B finding
RQ-C	Flan-T5-XL + LoRA	ingroup macro-F1 (held-out roots)	0.379 ± 0.014	0.143 (random-7-class)
RQ-C	Flan-T5-XL + LoRA	JSON parse rate	0.97	—

All headline numbers are mean ± std over seeds {42, 123, 7}. The 64 pp drop between RQ-B's grouped and leaky splits is documented as the central methodological finding: the underlying glossary maps each dog-whistle term to exactly one ingroup, so splits that don't hold out roots are measuring glossary memorisation rather than generalisation. We recommend root-stratified evaluation as the default protocol on this corpus.

Models on Hugging Face

• calerio/silent-signals-rqa — RQ-A binary disambiguator (6 variant branches across input arm × seed)
• calerio/silent-signals-rqb — RQ-B 17-class ingroup classifier (6 variant branches)
• calerio/silent-signals-rqc — RQ-C LoRA adapters for Flan-T5-XL (6 variant branches: 3 seeds × {Path A unbalanced, Path B class-balanced})
• calerio/silent-signals-demo — interactive Gradio demo

Repository structure

notebooks/        EDA, negatives mining, training pipeline (Jupyter)
scripts/          SLURM submit scripts + training entry points
data/             train/val/test splits, manifests, processed intermediates
resources/        dog-whistle glossaries and canonical definitions
results/          per-seed model outputs and aggregated metrics
docs/             methodology reports (RQ-A/B/C) and design rationale

The trained model artefacts live on Hugging Face (above); only metric JSONs and predictions are committed.

Reproducing the results

The models are already trained and live on HF Hub, so cloning lets you re-run inference and error analysis without retraining. For full retraining you need a GPU with ≥40 GB VRAM.

git clone https://github.com/calerio/dog-whistle-detection.git
cd dog-whistle-detection
pip install -r requirements.txt          # off-cluster (bracket-pinned, see file header)
jupyter lab notebooks/pipeline.ipynb     # canonical training-pipeline entrypoint

For training on a SLURM cluster, the submit scripts under scripts/hpc/ are templates: replace <YOUR_MATRICOLA> and <YOUR_EMAIL> with your cluster account and email, and adjust --partition / --account for your environment. Original development was on the Bocconi student HPC; see docs/references/bocconi_hpc.md for that specific cluster's conventions and docs/HPC_INVENTORY.md for the exact env package list.

Documentation

• docs/DESIGN_DEFENSE.md — every methodological decision and its defence (D1–D7)
• docs/IMPLEMENTATION_PLAN.md — section-by-section pipeline blueprint
• docs/DATASETS.md — dataset profiles and processing decisions
• docs/negative_report.md — three-stage RQ-A negatives mining (heuristic → LLM adjudication → balancing)
• docs/rq_a_report.md, rq_b_report.md, rq_c_report.md — per-task methodology + results writeups
• docs/hf_space_report.md — design log for the HF demo Space
• docs/reproduction_report.md — what it takes to re-run the pipeline end-to-end on a fresh box
• docs/HPC_INVENTORY.md — snapshot of the cluster footprint at the time of release

Citation

This was a group project for Università Bocconi course 20597 (Natural Language Processing), Spring 2026.

Authors: Facundo Lucero, Alissa Sharuda, Jan Szkulepa, Valerio Costa, Cecilia Lo Cicero.

The underlying dataset is published by Kruk et al. (2024):

@inproceedings{kruk2024silent,
  title     = {Silent Signals, Loud Impact: LLMs for Word Sense Disambiguation of Coded Dog Whistles},
  author    = {Kruk, Julia and Marchini, Michela and Magu, Rijul and Ziems, Caleb and Muchlinski, David and Yang, Diyi},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year      = {2024},
  publisher = {Association for Computational Linguistics}
}

License

MIT — see LICENSE.