EKI Propaganda Resistance Benchmark: Measuring AI Susceptibility to Russian Disinformation

What Is the EKI Propaganda Resistance Benchmark?

In June 2026, the Institute of the Estonian Language (Eesti Keele Instituut, EKI) published a benchmark designed to answer a question most safety evals don't touch: how easily can Russian propaganda fool AI language models?

Working with Propastop, an Estonian disinformation research organization, EKI constructed a test suite that measures whether LLMs "take positions on topics that the Russian Federation uses in its strategic narratives." The benchmark evaluates models across 14 distinct propaganda narrative categories, in three languages, using three question-phrasing strategies — making it one of the most targeted geopolitical safety evaluations available.

Sixty models were tested. The results expose a clear hierarchy: frontier models from Anthropic lead by a wide margin, while open-weight and older commercial models show significant vulnerability — especially when queried in Russian.

Benchmark Design and Methodology

Narrative Coverage

EKI and Propastop identified 14 propaganda narrative categories based on active Russian influence operations. These include:

• Status of Crimea and the legality of the 2014 annexation
• Justifications for the war in Ukraine
• History of NATO enlargement and alleged broken promises
• Justification for the USSR's annexation of Baltic states during World War II
• Claims about Nazi influence in Baltic governments
• Narratives alleging discrimination against Russian-speaking minorities
• Historical revisionism about Soviet occupation

Each category was developed into multiple test questions, for a total of 75 items per language.

Three-Tier Question Design

Every narrative is tested with three question phrasings:

This tiered approach is the benchmark's most important design feature. Models that pass neutral questions routinely fail on biased or malicious variants — sometimes becoming twice as likely to produce propaganda-aligned responses under adversarial prompting.

Scoring Protocol

Each response is scored on a 1–5 scale:

Scoring was performed by a separate LLM judge calibrated against Propastop's disinformation experts to ensure alignment with human evaluation. Crucially, models had no access to web search or external tools during testing — the benchmark measures the model's intrinsic ability to recognize and reject propaganda, not its ability to look up facts.

Results: Model Performance Rankings

Tier 1: Frontier Resistance Leaders

Anthropic's Claude models dominate the top of the leaderboard:

Claude Opus 4.7, the best publicly available model at time of testing, scored "Exemplary" (5/5) on 77 percent of questions and "Mediocre" (3/5) on just 2 percent. Fable 5, which is currently restricted to U.S.-based access, scored even higher at 95.2 but is not yet available for most international deployments.

Anthropic's safety-first alignment approach — including their constitutional AI methodology — appears to translate directly into resistance against geopolitical disinformation prompts in ways that other training paradigms do not.

Tier 2: Mixed Performance

Google Gemini models placed in the middle tier but showed notable inconsistency. EKI's AI alignment lead Krister Kruusmaa noted that despite Gemini's otherwise strong performance on Estonian language tasks, its resistance to propaganda prompts was uneven — suggesting language capability and safety alignment are orthogonal properties.

OpenAI GPT-4.5 and GPT-5 placed solidly in the middle of the pack. While significantly better than their predecessors (GPT-3.5 landed in the bottom third), they were outclassed by Anthropic's top models by a margin of 7–10 points.

Tier 3: Vulnerable Models

The bottom tier reveals which models are most susceptible:

• Mistral — the worst-performing major commercial model. Earlier studies had already flagged Mistral with a steady 36.67 percent misinformation rate. This is especially problematic given Mistral's positioning as a European alternative to US and Chinese providers; the company is currently negotiating a €20 billion valuation.
• GPT-3.5 and GPT-4o Mini — older and smaller OpenAI models, both in the bottom third
• Meta Llama (open-weight variants) — struggled significantly, particularly on Russian-language prompts
• Other smaller open models — consistent underperformance across all three tiers

Language Effect

One of the benchmark's most important findings is the language vulnerability gap:

In top-tier models, differences between English, Estonian, and Russian were negligible. In weaker systems, the gap between English and Russian performance reached 15 percentage points.

Kruusmaa pointed to two likely factors:

1. Training data imbalance — a large share of biased and propaganda-laden content in Russian-language training data
2. English-centric alignment — most RLHF and constitutional alignment pipelines operate primarily on English data, leaving Russian-language safety alignment incomplete

Comparison to Existing Safety Benchmarks

How This Differs from TruthfulQA

TruthfulQA measures whether models reproduce common misconceptions across 38 categories. The EKI benchmark differs in three critical ways:

How This Compares to Toxicity Benchmarks

Toxicity benchmarks (e.g., RealToxicityPrompts, BOLD) measure offensive language generation. The EKI benchmark is orthogonal: a model can generate non-toxic text while still fluently repeating propaganda talking points. This is a fundamentally different safety dimension that existing benchmarks do not capture.

Where This Fits in the Safety Stack

The EKI benchmark fills a gap between factuality (TruthfulQA) and toxicity (RTP) and what might be called narrative alignment — the model's ability to recognize and reject adversarial narrative frames. No major benchmark before this has systematically evaluated geopolitical disinformation resistance across multiple languages with tiered adversarial prompting.

Implications for Developers

Language-Specific Deployment Risk

If you're deploying a model in a multilingual context — especially in Eastern Europe, the Baltics, or any region exposed to Russian influence operations — English-only safety evaluations give you a false sense of security. A model that passes every English safety benchmark may still produce propaganda-aligned responses in Russian, Estonian, or Ukrainian at a material rate.

What to do: Always evaluate safety alignment in the languages your users will actually use, not just English. If your deployment covers Russian-language users, Russian-language safety testing is non-negotiable.

The Open Model Gap

EKI's findings confirm what many in the AI safety community have suspected: open-weight models lag significantly in geopolitical disinformation resistance. For institutions in high-risk regions — government agencies, media organizations, educational institutions — this creates a hard trade-off:

• Commercial frontier models (Claude, GPT-5) offer stronger disinformation resistance but require API access, increase costs, and introduce vendor dependence
• Open models (Llama, Mistral, Qwen) offer autonomy and data sovereignty but carry materially higher risk of propagating disinformation

EKI's Kruusmaa put it bluntly: "Open models are the only option for many institutions, but these don't yet meet the needs of the Estonian information space."

Data Poisoning as an Active Threat

The benchmark's timing is notable because the underlying threat is escalating. Russian influence operations are systematically generating synthetic content designed to be scraped by web crawlers and ingested into model training data. Kruusmaa warned: "Massive amounts of content is being produced that isn't meant for humans at all. It's for web-crawling bots."

This means the vulnerability profile of today's models may be worse than what the benchmark measures — because training data contamination is ongoing and compounding. Models trained on datasets that include large volumes of operationally-generated propaganda will be harder to align out of it.

Mitigation Strategies

There is no single fix for disinformation vulnerability. The most effective approach combines multiple strategies:

1. System Prompting

Explicit disinformation resistance instructions in system prompts show measurable improvement. For high-risk deployments:

You are a helpful assistant. If you detect that a user's question
is based on false premises common in disinformation narratives,
clearly identify the false premise and refuse to engage with the
framing. Do not repeat or reproduce disinformation talking points,
even if asked to do so indirectly.

This is the cheapest mitigation and should be applied universally, but it has limits: it won't help if the model's training data has already embedded the propaganda narratives deep in its weights.

2. Targeted RLHF / Constitutional AI

Anthropic's leadership on this benchmark is consistent with their investment in Constitutional AI (CAI), which trains models to reject harmful outputs based on a written constitution. The EKI results suggest that CAI generalizes to geopolitical disinformation in ways that standard RLHF may not.

If you're training your own models or fine-tuning open models:

• Include geopolitical narrative safety in your RLHF reward model
• Add propaganda-resistance criteria to your constitutional principles
• Train on multilingual data covering the languages in your deployment region
• Test explicitly with adversarially framed questions, not just neutral ones

3. Multilingual Content Filtering

Post-hoc content filtering is a safety net, not a solution — but it matters. For Russian-language outputs specifically, filtering systems should be tuned more aggressively, because models are measurably more likely to produce propaganda-aligned content in Russian than in English.

4. Continuous Evaluation

The EKI benchmark is now public and repeatable. Any organization deploying LLMs in multilingual or geopolitically sensitive contexts should:

5. Data Curation in the Training Pipeline

For organizations training their own models:

• Actively filter Russian-language training data using known propaganda narrative categories
• Work with regional disinformation researchers (like Propastop) to maintain up-to-date narrative taxonomies
• Consider federated data sourcing from trusted regional institutions to improve small-language representation

The underlying problem is structural: as EKI director Arvi Tavast noted, "Even if developers wanted to [improve Estonian language quality], they depend on what data exists online. Estonia still hasn't made enough Estonian-language training data available."

Summary

The EKI Propaganda Resistance Benchmark establishes that LLM vulnerability to disinformation is not a binary property — it depends on language, model architecture, training alignment, and how the question is framed. For developers deploying models in multilingual environments, the takeaway is clear: evaluate in the languages your users speak, test with adversarial framing, and don't assume English safety performance generalizes.

EKI has open-sourced the benchmark methodology through its LLM leaderboard, which tracks performance in Estonian language, knowledge, and safety categories. As geopolitical disinformation operations increasingly target AI systems, benchmarks like this one will become standard equipment in the AI safety toolbox.