Cross-Constituency Aggregation for Community Notes

changes continuously — new notes, ratings, and status updates are added over time — so re-downloading it today and re-running ingest will not reproduce the exact same master_full.parquet, and downstream clustering/scoring numbers may shift as a result.

The snapshot behind this repository's master_full.parquet (and therefore behind every committed data/interim/, data/processed/, and data/llm_validation/ artifact) was retrieved on 2026-05-08. Anyone who needs this exact historical result set should use the committed data/interim/, data/processed/, and data/llm_validation/ outputs (or fetch master_full.parquet from Hugging Face above) rather than re-downloading raw inputs from X.

How To Run

Local Setup

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

Place the Community Notes snapshot and FastText files under raw/; see raw/PUT_SOURCE_FILES_HERE.md.

Running From Scratch

How much you need to fetch or run depends on what you are trying to do:

1. Just read the results / build the paper. No download needed. Cloning

this repository already gives you data/processed/, data/interim/ (except the two oversized ratings files), and data/llm_validation/ — everything the paper and its figures are built from. Skip straight to Build The Paper.

1. Re-run clustering/scoring/topics to check the pipeline. You do not need

the raw X snapshot. Run scripts/fetch_interim_from_hf.sh to pull master_full.parquet and the two interim ratings files (see Large Local Data), then start from notebooks/01_clustering.ipynb onward — data/master_full.parquet is 01_clustering.ipynb's input.

1. Reproduce ingestion itself from raw X data. Download a fresh snapshot

from the official links in raw/PUT_SOURCE_FILES_HERE.md and the FastText language model, place them under raw/, then run 00_ingest.ipynb onward. Read Raw Data Provenance first: X's snapshot changes over time, so a newly downloaded snapshot will not reproduce this repository's exact master_full.parquet or its downstream numbers.

Smoke Test

Set TEST_MODE=1 in config.txt. Smoke runs use a small input slice and write only under .artifacts/smoke/.

for stage in ingest clustering scoring topics plots paper_visuals; do
  jobs/submit_stage.sh --dry-run "$stage"
done

Submit the stages in order:

jobs/submit_stage.sh ingest
jobs/submit_stage.sh clustering
jobs/submit_stage.sh scoring
jobs/submit_stage.sh topics
jobs/submit_stage.sh plots
jobs/submit_stage.sh paper_visuals

Production Run

1. Put the raw snapshot and FastText files in the root raw/ directory.
2. Install any missing packages in the active Python environment, including pyamg.
3. Set TEST_MODE=0 in config.txt.
4. Submit the six stages in the order shown above.

The production clustering configuration is fixed at 100,000 notes and 200,000 users. Downstream stages always consume Method-B reassigned clusters from data/interim/user_clusters_method_b_voteprofile.parquet.

jobs/submit_stage.sh and jobs/submit_llm_validation.sh target an SGE-style scheduler (qsub/qstat/qacct); adapt the queue names and resource flags to whatever job scheduler is available, or run the underlying Python entry points directly for a local run. Logs and status files are written under .artifacts/logs/.

Resume From Interim Data

scripts/fetch_interim_from_hf.sh

This populates data/interim/ratings_filtered.parquet and data/interim/ratings_clustered.parquet. The remaining clustering artifacts, especially the Method-B user assignment, must also exist before starting from scoring.

Canonical Gemma Validation

The canonical validation is implemented under notebooks/llm_validation/ and evaluates the frozen 13,655-note universe with Gemma 4 31B IT in BF16 on two L40S-class GPUs. Prompts contain only note text: no tweet retrieval, URL fetching, Gabriel scaffold, or upstream selection score is supplied to the model.

Install the pinned runtime and model, then submit the 128-note Stage 1 smoke:

jobs/submit_llm_validation.sh setup
jobs/submit_llm_validation.sh smoke
jobs/submit_llm_validation.sh stage1-array 2-7 3 64

The three frozen prompts are reproduced verbatim, with their hashes, in data/llm_validation/PROMPTS.md. See notebooks/llm_validation/README.md for adaptive batch sizing, parallel Stage 1 shards, checkpoint behavior, and output schemas.

The separately reported Stage 1.5 sensitivity analysis rechecks only the 1,703 resolved Gemma opinion notes for a substantial sourced factual core. It keeps canonical Stage 1 immutable and writes to a separate run directory:

jobs/submit_llm_validation.sh --dry-run stage1-5 1703 64
jobs/submit_llm_validation.sh stage1-5 1703 64

Its 280 additional admissions join the 10,096 strict sourced notes in a 10,376-note expanded Stage 2 run. The completed and merged run retains 8,527 strict-route notes and 31 Stage 1.5 recall-route notes at the frozen threshold of 50. The canonical final rescue set therefore contains 8,558 notes. The admission route is audited but never shown to the model. Operational and audit details are in notebooks/llm_validation/README.md.

Build The Paper

cd paper
latexmk -pdf main.tex

The five figures and the three frozen prompt files the appendix quotes are pulled in from figures/ and data/, so a fresh clone builds without any extra downloads. See paper/README.md.

Methodology

1. Ingestion

notebooks/00_ingest.ipynb reads the official Community Notes notes, ratings, and status-history files.

• FastText retains English notes.
• Tweets must have at least three English notes.
• Ratings are joined to note metadata and status history.
• Output: data/master_full.parquet (large, ignored by Git).

2. Clustering

notebooks/01_clustering.ipynb selects a dense slice with 100,000 notes and 200,000 users. Only ratings cast within 48 hours of note creation are eligible, and notes require at least three eligible ratings.

The clustering implementation in src/spectral_fast.py:

• builds a sparse mean-centered user-note matrix;
• constructs a symmetric 15-nearest-neighbor affinity graph;
• computes spectral embeddings with AMG for production;
• evaluates k=2..7 using silhouette and repeated stability diagnostics.

Clusters smaller than max(1% of users, 500 users) are treated as outliers. Method A assigns those users by embedding-centroid distance. Method B assigns them by correlation with the main camps' note-level vote profiles. Method B is the canonical downstream assignment; the initial spectral label remains available as initial_cluster.

3. Scoring And Representative Selection

notebooks/02_scoring.ipynb computes global and camp-conditional approval. Notes need at least three total ratings. A bridge score is computed only when every camp has at least three ratings for the note:

bridge score = geometric mean of camp-conditional approval rates

For each tweet, the Representative strategy selects the note with the highest bridge score. A non-helpful note counts as rescued when its bridge score is strictly above 0.5.

Topic input is restricted to notes whose maximum-minus-minimum camp approval is at least 0.90, again requiring at least three ratings from every camp.

4. Representative Topics

notebooks/03_topics.ipynb runs BERTopic on the diagnostic note subset using all-MiniLM-L6-v2, cosine UMAP, and the repository's fixed text-cleaning rules.

Three auditable layers:

• fine-grained topics;
• a 30-topic parent layer;
• a paper-facing super-parent layer selected from candidate counts 6, 8, 10, 12.

HDBSCAN outliers remain visible in the strict parent output. A separate reassigned table maps them to the nearest parent centroid and records similarity diagnostics. Process/meta vocabulary is removed from labels without removing notes from the corpus.

5. Gemma Validation And Figures

The canonical Gemma pipeline first classifies all 13,655 Representative rescue candidates, applies a targeted Stage 1.5 recall check to resolved opinion notes, and scores the resulting 10,376 sourced candidates for rescue worthiness. The final rescue definition is passes_rescue_threshold == True in the merged expanded Stage 2 result, which yields 8,558 notes.

notebooks/06_plots.ipynb produces exploratory analysis plots, and notebooks/07_paper_visuals.ipynb is the historical manuscript-figure notebook.

Every figure in the current paper is produced by a standalone script in figures/script_figures/, where the script and its .pdf and .png outputs share one basename. Regenerate any of them in place:

python figures/script_figures/fig1-rater-dominance.py        # Figure 1
python figures/script_figures/fig2-dataset-construction.py   # Figure 2
python figures/script_figures/fig3-rescue-panels.py          # Figure 3
python figures/script_figures/cn-gemma-validation-funnel.py  # Figure 4
python figures/script_figures/cn-topic-signatures.py         # Figure 5

All but Figure 2 run on a fresh clone; that one also needs data/interim/ratings_clustered.parquet from Hugging Face. See figures/README.md.

Smoke mode changes scale and output location only. It does not redefine the production methodology.

Tests

python -m pip install -r requirements.txt
python run_tests.py          # all 68 tests
python run_tests.py -v       # per-test names

Three suites run: the aggregation rule on synthetic data, the paper's headline numbers re-derived from the committed parquet files, and the Gemma validation package with every model call mocked. The tests themselves need only pandas, numpy, and pyarrow from that requirements file; pytest is not a dependency, and no data has to be downloaded. tests/RESULTS.md records the last full run and lists what each suite asserts.

Licensing

The underlying Community Notes data remains subject to X's own terms; no raw snapshot files are redistributed here. To cite this work, see CITATION.cff.

Acknowledgments

The authors acknowledge support by the local computing resources through the core facility SCCKN.

References

• Paper: paper/README.md
• Reproduction map: REPRODUCING.md
• Data catalogue: data/README.md
• Figures: figures/README.md
• Presentations: docs/README.md