MBIB Quality Observatory

This repository audits the Media Bias Identification Benchmark (MBIB). It keeps the summary tables, figures, task cards, and LLM review exports under results/ and reports/. Large dataset caches and downloaded model weights are not tracked in git.

The pipeline includes:

• dataset loading and normalization for all 8 MBIB tasks
• label balance, sentence length, and cross-task overlap checks
• near-duplicate detection
• three embedding-based cleanlab probes for label-noise estimation
• zero-shot transfer with a BABE-fine-tuned RoBERTa baseline
• LLM review of flagged rows

Quick start

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

If you want to run notebook 06, add your API key to .env:

OPENAI_API_KEY=...

Reproducing the full pipeline

Run the notebooks in order:

To regenerate the markdown task cards from the committed summaries:

python -m scripts.regenerate_reports

To review the saved results, open notebooks/07_results_walkthrough.ipynb.

External downloads

These files are required to rerun the pipeline, but they are not tracked here.

Notes:

• 01_load_mbib.ipynb downloads the dataset and writes local parquet caches to data/processed/.
• sentence-transformers and transformers download model weights to your local cache on first use.
• The repo keeps the analysis outputs, not the raw cached inputs.

Results Summary

The tracked summaries cover 810,953 rows across 8 MBIB tasks.

• The largest overlap pairs are linguistic_bias ↔ racial_bias (8,940 shared sentences), gender_bias ↔ linguistic_bias (8,574), gender_bias ↔ racial_bias (7,056), gender_bias ↔ hate_speech (6,816), and cognitive_bias ↔ fake_news (5,230).
• The highest mean noise rates are in linguistic_bias, cognitive_bias, and fake_news. The lowest is gender_bias.
• Near-duplicate rates stay low across the benchmark. The highest sampled near-duplicate rate is gender_bias at 1.16%.
• The highest macro-F1 is political_bias (0.6856). The lowest is racial_bias (0.3492).
• LLM disagreement is highest for political_bias (71%) and lowest for racial_bias (23%).

Key findings

1. Cross-task leakage is substantial

The largest quality problem is not class imbalance or duplication. It is task leakage across the benchmark.

• linguistic_bias ↔ racial_bias: 8,940 shared sentences
• gender_bias ↔ linguistic_bias: 8,574
• gender_bias ↔ racial_bias: 7,056
• gender_bias ↔ hate_speech: 6,816
• hate_speech ↔ linguistic_bias: 6,414
• cognitive_bias ↔ fake_news: 5,230

This can inflate transfer results when the same sentence appears across tasks.

2. The three cleanlab probes agree on which tasks are noisiest

Mean issue rates from the three embedding probes:

The ordering is stable across probes. linguistic_bias, cognitive_bias, and fake_news consistently look the least reliable. gender_bias is consistently the cleanest.

3. Duplicates are not the main problem

Exact duplicate rates are all below 0.24%. Sampled near-duplicate rates remain low as well, between 0.00% and 1.16%.

That means the benchmark issues are mostly about annotation quality and task boundaries, not bulk repetition.

4. Zero-shot transfer is highly uneven

Macro-F1 of the BABE-fine-tuned RoBERTa baseline on the 5k evaluation sample:

political_bias has the highest macro-F1 and racial_bias the lowest. That gap matches the broader variation across tasks.

5. LLM review shows high disagreement on many flagged rows

Disagreement between the committed labels and the LLM review on 100 flagged rows per task:

Disagreement is highest in political_bias, linguistic_bias, and cognitive_bias. racial_bias is lower, but still not negligible.

6. Combined View

Across the pipeline, the same pattern keeps showing up:

• the largest tasks are also where cross-task reuse is most visible
• label-noise estimates and LLM disagreement both point to unstable annotations in a small subset of tasks
• duplicate rates stay low, so the benchmark’s weak points are not simple deduplication issues
• the benchmark should be read as a mixed-quality collection of tasks rather than a uniformly clean shared standard

Representative examples

These examples come from the saved flagged pool and LLM review outputs. They show how the embedding filter and the LLM review differ.

A. High-consensus flags that look like clear annotation mistakes

B. "Word trap" false alarms dismissed by the LLM

The smaller models often react to surface words without understanding the actual claim.

C. Political criticism is not the same as hate speech

The LLM layer is useful when the sentence is aggressive but not identity-targeted.

D. Formally Worded Racist Claims That Small Models Missed

Some hard cases use formal or technical wording rather than explicit slurs.

These rows are a good example of where embeddings alone are not enough.

E. Some rows appear to be assigned to the wrong task

F. Rows Where the LLM Changed or Confirmed the Label

Why These Examples Matter

• the filter stage surfaces a broad candidate pool
• the review stage removes many keyword-based false alarms
• some benchmark problems are annotation errors, while others are task-assignment errors
• the hardest cases combine label quality, task boundaries, and wording

Repository layout

.
├── data/
│   └── processed/
│       └── .gitkeep
├── notebooks/
├── reports/
│   ├── cognitive_bias.md
│   ├── fake_news.md
│   ├── gender_bias.md
│   ├── hate_speech.md
│   ├── linguistic_bias.md
│   ├── political_bias.md
│   ├── racial_bias.md
│   └── text_level_bias.md
├── results/
├── scripts/
│   ├── export_top_flagged_to_csv.py
│   ├── regenerate_reports.py
│   └── run_extra_validation.py
├── src/
├── tests/
├── .gitignore
├── README.md
└── requirements.txt

Notes for this repo

• results/ is versioned so the analysis can be reviewed without rerunning the notebooks.
• data/processed/ is ignored because the local parquet cache is large and can be rebuilt.
• The active pipeline uses the three-model cleanlab workflow.