Methodology
This page discloses every methodological decision. Each layer is anchored in research, every finding traceable through verbatim evidence — especially for readers who are initially skeptical of our results.
What does Klyptra measure?
Language and framing in the news — across 6 dimensions with 27 sub-patterns.
↓ Read section
Who decides the values?
Three language models in parallel, with verbatim evidence for every finding.
↓ Read section
How reliable are the values?
Three models judge independently; spread and confidence are openly reported per analysis.
↓ Read section
The scale
The six dimensions and the 0–10 scale are Klyptra's operationalization of the Media Bias Taxonomy (Spinde et al. 2023) — not a direct part of BABE, which annotates binarily (biased/neutral). The scale is calibrated against BABE-style expert benchmarks. The five bands are not equally wide: news-agency reporting (dpa, AFP, Reuters) typically hits 8–9 — a value of 10 would be pure fact-listing with no narrative selection at all, practically unreachable. Downward, by contrast, there is more room: the propaganda band (0–2.9), three points wide, is the largest.
The labels in this table are exactly the strings the analyzer outputs in the JSON and the permalink UI (in German) — no UI mapping in between.
The six dimensions — in depth
Which perspective is declared the narrative norm? Who is subject, who is object?
Operationalization
Which words carry judgments without marking them? Loaded language in the narrow sense.
Operationalization
How many voices are quoted directly, how politically broad is the spectrum?
Operationalization
In the book (Spinde 2025, Ch. 2), source/selection bias is a reporting-level construct that strictly measures across articles. Klyptra approximates it on the single text — the full cross-outlet analysis sits at Tier 2 (not in the score).
Is evaluation linguistically separated from observation — or sold as fact?
Operationalization
Its own dimension, because German news language interweaves evaluation and observation especially tightly at the syntactic level (nominalization, modal verbs, subjunctive I/II) — the English BABE annotation only covers this indirectly.
What is left out? Which relevant background or counter-positions are missing?
Operationalization
Omission/spin bias is in part reporting-level in the book (what is missing across several articles). Klyptra assesses the gaps recognizable in the single text; the cross-article level is Tier 2.
How strongly is it emotionally charged? Sensational or outrage language?
Operationalization
27 sub-categories
The six dimensions are the top-level axes. Beneath them, Klyptra maintains 27 specific bias patterns following the BiasScanner taxonomy (Menzner & Leidner 2024). Per article, 0–N patterns are identified — each with verbatim evidence, position in the text and a strength assessment.
Sub-categories are qualitative markers, not numeric sub-scores. If a top dimension such as “Completeness” is rated low, the sub-layer shows which concrete pattern carries the finding — e.g. Cherry-Picking or Whataboutism.
Lexical level — words that evaluate without marking the evaluation as such.
ExampleA “migrant” is consistently called an “intruder”.
Example“Nightmare scenario”, “shock diagnosis”, “mood of doom” as routine vocabulary.
ExampleGeneralization about groups (“typical of …-migrants”), mentioning origin without relevance.
Example“scandalous attempt” for one party vs. “bold initiative” for the other — for the same kind of action.
Narrative constructs — how a matter is framed in storytelling, independent of individual words.
Example“The left wants every migrant to get a house immediately.” A caricature of the opposing position is attacked.
Example“Either we cut taxes — or the country collapses.” Two options suggested where many exist.
Example“Just like back in 1933 …” for a current political debate with a loose connection.
Example“Why is the chancellor silent on the accusations?” — without the accusations themselves being substantiated.
Example“5% growth was expected, now it's 8% — so a failure.” The yardstick adjusted after the result.
ExampleConsistent “we Germans” vs. “them” — collective assignment of guilt or virtue.
Source quality — who is heard, how they are quoted, whether the voices are classifiable.
ExampleOnly one party's press office is quoted; the other side not at all or only paraphrased.
ExampleA lobbyist is introduced as an “independent expert” without naming their interests.
Example“Circles report …”, “according to insiders …” carry the central argument of the piece.
Linguistic discipline — whether evaluation is marked as evaluation or sold as fact.
Example“The government's catastrophic policy …” — an evaluative adjective in news mode.
Example“This will undoubtedly end in disaster.” Forecast without subjunctive, without source.
Example“Millions are affected” — a number without evidence, source or method.
Example“They only care about power.” Attribution of motive as a statement of fact.
Example“It is illegal because it breaks the law.” The justification repeats the claim.
What is missing — relevant context, counter-arguments, evidence that goes unmentioned.
ExampleOne study is cited; three methodologically comparable studies with the opposite result are not.
Example“Ms. M. from Hamburg says …” carries a trend finding; statistical data are missing.
ExampleConsistent deflection from the main accusation onto the behavior of other actors.
ExampleClimate scientists and climate deniers are presented as equivalent voices — although the evidence base is asymmetric.
Affective charge — how strongly and in which direction the text colors emotionally.
Example“The incompetent minister …” — the person attacked instead of the argument refuted.
Example“Since X has governed, Y has fallen — so X is to blame.” Correlation as causation, without a mechanism.
Example“All politicians lie”, “the media” as a monolithic actor.
ExampleA product report without distance; editorial content not separated from advertising.
ExampleA consistent camp tendency across fact selection, word choice and sources.
Every sub-finding passes the same verbatim gate as the top-level analysis: findings without a quote that can be verified in the original text are discarded. In the JSON export of an analysis, the layer appears as sub_categories[] with parent_dimension, verbatim_quote, char_offset and bias_strength.
Actor analysis — PFA-light
Person-Oriented Framing Analysis (Felix Hamborg 2023) extracts the named actors per article and describes how they are talked about. That is more concrete than any holistic score — and makes systematic asymmetries between actors visible.
Each identified person (politician, scientist, citizen, …) gets four fields:
mentions_count
How often the actor appears — across all designations (see Coreference).
sentiment_score
Aggregated tone toward the actor on a scale from −1 (negative) to +1 (positive).
framing_devices
Up to five recurring stylistic devices per actor — e.g. “attribution of blame”, “hero narrative”, “victim staging”.
representative_quotes
Three to five verbatim quotes that carry the framing — the verbatim gate ensures each quote appears 1:1 in the text.
From the individual actors, a distribution observation is computed — whether the article treats the people comparably in language or not.
sentiment_disparity
Difference between the most positive and most negative actor sentiment in the article — computed over all actors with mentions_count ≥ 2.
disparity = max(sentiments) − min(sentiments)
balance_assessment
A qualitative classification of the disparity: balanced / slightly_asymmetric / strongly_asymmetric. With fewer than 2 actors with sufficient mentions, the field returns not_applicable — instead of inventing a value.
Aggregation from models
cross_person_analysis is not taken from the language model but recomputed deterministically from the filtered actor data. This keeps the disparity metric always consistent with the reported sentiment values — even if the model would summarize differently internally.
Coreference documentation
Political texts reference the same entity in several ways — by name, by role, by pronoun. Klyptra documents these cross-references explicitly so that mention counts and actor sentiment are not distorted by mere synonymy.
Per article, a list coreference_documentation.entities[] is reported. Each entity has a canonical_name and a list of all all_mentions[] found in the text.
mention_count is then recomputed deterministically as the sum of non-overlapping substring matches of all mentions in the text — not taken from the language model.
In a report on Ukraine policy:
canonical_name
Volodymyr Zelensky
all_mentions
mention_count
7
Without this resolution, the actor would land in three different buckets — and the sentiment aggregation in the PFA layer would be distorted.
Pipeline
Every step is independently testable and logged. Anyone who questions a result can trace the chain back to the individual piece of evidence in the text.
Submitted text
The article text to be checked is submitted directly (paste or file) — optionally with a title and source label. Klyptra analyzes exactly this text, not the outlet behind it.
3 models in parallel
Three language models independently rate the same text on all six dimensions and extract the detail layers in parallel: sub-category findings, actor mentions with sentiment, and coreference clusters. A chain-of-verification (5 control questions) reduces hallucinations.
Median + agreement
Numeric scores: median across the three models. Labels: majority vote. The spread of the individual judgments is reported per dimension as model agreement — it stays visible instead of vanishing into the average.
Evidence checked literally
Every finding must carry a quote that appears exactly in the original text — otherwise the evidence is discarded (the assessment remains marked as model-based). Verified evidence is highlighted in the text with a character offset. The result is a permalink with a 30-day TTL.
One analyzer, two uses
The same analyzer code runs in two contexts:
/analyse): the actual analysis of a submitted text — with a permalink, 30-day TTL.Ensemble
Language models have their own, model-specific biases. Klyptra picks three models from three different pre-training pipelines so that the blind spots of a single model become visible through the others. Aggregation is by median (numeric) and majority vote (labels) — all models carry equal weight, none is preferred.
OpenAI (USA)
A lean, low-latency variant of the GPT-5.4 line — it carries the on-demand analysis without giving up the score consistency of the larger models.
Mistral AI (France / EU)
A European pre-training pipeline from an independent provider — it brings a different training basis than the US models.
DeepSeek (China)
A third, independent pre-training corpus — if this model deviates from the consensus, the spread becomes visible as model agreement.
Numeric scores
Median across all 3 models. Robust against outliers of a single model.
Categorical labels
Majority vote. On a 1:1:1 split, the label falls back to the score nearest the median.
Model agreement
The spread of the three judgments is reported per dimension as high, medium or low agreement — separate from the model's confidence.
Scientific foundations
Klyptra is not a self-construction. Every methodological layer references a peer-reviewed source with a DOI. The differentiation from BiasScanner (our direct predecessor) lies in the three-model ensemble, the German-language specialization, the verbatim gate and the PFA layer.
Concept & 6 dimensions
The six dimensions operationalize the bias types of the Media Bias Taxonomy. BABE (Spinde et al. 2021, EMNLP-Findings) serves as an expert-annotated validation benchmark (binary biased/neutral) and IRR yardstick — the 0–10 scale itself is Klyptra's operationalization, not part of BABE.
Spinde et al. (2023) Media Bias Taxonomy, ACM Comput. Surv., arXiv:2312.16148 · BABE: EMNLP-Findings 2021, DOI: 10.18653/v1/2021.findings-emnlp.101 · consolidated in Spinde (2025), Springer, Open Access, DOI: 10.1007/978-3-658-47798-1
→ 6 dimensions27 sub-patterns
The 27 specific bias categories are placed as a sub-layer beneath the six Klyptra dimensions. BiasScanner is Klyptra's direct scientific predecessor. Not part of Spinde's work — an independent foundation.
Menzner & Leidner (2024) “Improved Models for Media Bias Detection and Subcategorization”, NLDB 2024, pp. 181–196. DOI: 10.1007/978-3-031-70239-6_13
→ 27 sub-patternsActor layer
PFA-light extracts named actors with mention counts, sentiment and framing devices. The cross-person disparity makes systematic asymmetries visible.
Hamborg (2023) “Revealing Media Bias in News Articles”, Springer. DOI: 10.1007/978-3-031-17693-7
→ PFA sectionLimits
Methodological honesty requires naming the weak points. This list is not exhaustive — contributions are welcome.
Language models have documented biases (Horych et al. 2025). Ensemble + verbatim requirement reduce this but do not eliminate it. Klyptra is not an arbiter — it is a systematic, verifiable indicator.
Klyptra rates exactly the submitted text — not the outlet, newsroom or author behind it. A single result is not a verdict on an outlet; no general “tendency” of a source can be derived from one analysis.
Klyptra measures language and framing — not whether claimed facts are correct. Fact-checking is a separate task (see Correctiv, dpa fact-check).
The verbatim gate keeps only quotes that appear exactly in the text. Paraphrases are not output as evidence — some dimensions therefore deliberately appear without an evidence quote and are marked as a model-based assessment. Honesty over forced evidence.
The six top-level scores are aggregated across all three models (median) and reported with their spread. The detail layers (sub-categories, actor analysis, coreference) currently come from one of the three models — a true union/dedupe aggregation across all models is planned as the next stage. A deliberate MVP decision, not a bug.
The three models are primarily pre-trained in English. Idiom, subjunctive discipline and irony detection can be thinner in German than in English. Few-shot examples compensate in part; a systematic German ground-truth evaluation is on the research roadmap.
The six dimensions are calibrated for news reporting. If a commentary is submitted, word choice and emotional balance swing strongly, as expected — Klyptra cannot yet reliably tell whether a text should be classified as a report or a commentary. A genre detection as a pre-stage is documented in the methodology roadmap.
Even trained experts reach only an agreement of Krippendorff α ≈ 0.40 on bias labels (Spinde 2025, Ch. 4) — that is the documented maximum of the domain, not a weakness signal. There is no strong “ground truth”; Klyptra's consistency aim targets this expert level, not objective truth.
Automated methods can report patterns that are statistically tangible but substantively irrelevant (Spinde 2025, Ch. 8). A score is a systematic indicator, not a final verdict on the significance of a text.
Readers perceive texts that contradict their position as more biased than comparable texts on their own side (hostile-media effect). Studies show that even bias visualizations do not resolve this effect (Spinde 2025, Ch. 7) — a result is read filtered through one's own stance.
Spinde (2025, Ch. 7) shows that a political classification does not increase bias perception — it communicates stance, not distortion. Klyptra's political, economic and social descriptors are a tendency indication and do not feed into the objectivity score.
Version history
Every single analysis carries a methodology_version tag. Existing analyses keep their version — methodology updates create no score drift in historical data.
Reproducibility
Klyptra discloses its methodology: model versions, bias dimensions and the underlying research are documented on this page. Every analysis also carries a signature (system-prompt hash) that records its exact methodological state.
Methodology
Bias dimensions, scale and aggregation logic are described in full on this page.
Data
Every analysis is exportable as JSON and PDF for its creator.
Prompts
Versioned prompt templates incl. few-shot examples. Diff log for every change.