Under the hood

Same story, twenty sources

This section is currently available in English only.

When a species comes back from the brink, every nature outlet covers it. Twenty versions of the same event land in our pipeline within hours. Which one do you keep? And what do the other nineteen tell you?

These are two different questions. The first is about removing noise. The second is about detecting a signal. We handle them with two separate mechanisms: deduplication and corroboration.

The dedup problem

Our pipeline collects from over a thousand sources across 50 countries. Every four hours, new articles arrive. When something genuinely newsworthy happens, a dozen outlets publish within the same day. Showing all of them wastes your attention.

The naive approach is keyword matching: if two titles share enough words, they're the same story. We tried this. Title Jaccard similarity (the overlap between the sets of words in two titles, divided by their union) seemed promising. In practice, it barely works.

Why naive approaches fail

We analyzed duplicate pairs across several weeks of data. The Jaccard scores between titles that humans would call "the same story" ranged from 0.00 to 0.33. That's essentially random.

Why so low? Because different outlets frame the same event with entirely different vocabulary. One writes "Wolf population rebounds in Germany." Another writes "Predator numbers surge in Central Europe." Same story. Zero word overlap after removing stop words.

Multilingual sources make this worse. The same event appears in English, Dutch, German, and Spanish with no lexical overlap at all. Semantic similarity (comparing meaning, not words) would help, but requires embedding every title on every run. At our scale, that's a real cost.

What we actually do

Deduplication happens in two places, at two different scales:

Upstream (NexusMind): The scoring service detects duplicates across runs using content hashing and URL normalization. If the same article appears in multiple collection cycles, only the first instance is scored. This catches exact and near-exact duplicates before they reach us.

At build time (Chief Editor): A second, title-comparison pass (Jaccard similarity) exists in the editorial layer but is currently switched off — in practice, paraphrased titles defeated it. Cross-source duplicates are instead folded upstream by cluster detection: when several outlets cover the same story, one representative survives and the others become corroborating sources on its page.

This approach catches most duplicates. It misses some. A story about forest recovery in the Amazon might appear from both a Brazilian and a European source with completely different framing. We accept this trade-off: a duplicate is annoying, but a missed story is invisible.

Corroboration: the flip side

Here's the insight that changed our approach: duplicate coverage is not just noise. When multiple independent sources report the same story, that's evidence of reliability. A reef recovery story covered by one blog could be aspirational. The same story covered by Reuters, a marine research institute, and a local newspaper is corroborated.

When several outlets report the same story (grouped by NexusMind's cross-source analysis), we show one version. It is the version NexusMind picked to represent the story, if we have a summary for it; otherwise the highest-ranked version we can show. Being covered by several outlets does not move a story up the feed.

The result: you see one version of each story. The article page shows how many sources were grouped into the story, counting the one you are reading. It does not name them: when we checked whether a named outlet really covered the same story, it was right somewhere between about half and six times in ten, whether we picked the most credible outlet or one at random. That is not reliable enough to name an organisation, so we stopped showing names.

Noise vs. signal

Dedup and corroboration solve opposite problems using the same raw material:

What we haven't solved

Cross-lingual dedup remains an open problem. A story about the same forest recovery project might appear in the Recovery lens from a Brazilian source in Portuguese and from a German source in German. Our title comparison works on English translations, which helps, but the translations themselves can diverge enough to slip past the Jaccard threshold.

Semantic dedup (comparing article meaning rather than title words) is the likely next step. We're watching embedding costs come down. Until then, the current system catches most duplicates while staying fast enough to run twelve times a day.

Last updated: July 2026