39,000 Torrents, a Green Benchmark, and the Bug the Data Never Caught

[Torrent911] Inception.2010.TRUEFRENCH.1080p.BluRay.x264-YIFY

That's a real filename. The goal: extract "Inception" and "2010" from it, query the TMDB API (The Movie Database, the open reference for film and TV metadata), and display the right card. Poster, rating, synopsis. Automatically, for a library of a few hundred files.

Context: a self-hosted PHP media server. Think Plex, but lighter, no cloud, no account. The interface looks vaguely like Netflix. The hard part is automatic recognition. Torrent release names follow no formal standard: [Torrent911] can come before the title, TRUEFRENCH can interrupt it, a release group trails at the end. Films have a year. TV shows almost never do. Popular anime have episode numbers with four digits: One.Piece.S01E1164.VOSTFR.

I had accumulated 39,188 real torrent names over the years: 22,058 films, 17,130 series. Enough to build an honest test bench. I used an AI assistant to replay variants quickly and analyze where the metrics leaked.

Two things happened. First, a clean data-driven optimization: measure, iterate, stop at the right time. Then, a classic trap: the benchmark was green while a silent bug turned every single TV show into a film match.

Two different lessons about what data measures — and what it doesn't.

A minefield inside a string

The pipeline has two distinct stages.

Stage one: extract a clean title and a year from the filename. This is the foundation. If it returns "Inception 1080p BluRay x264" instead of "Inception", TMDB finds nothing useful. Release naming puts the title first and technical metadata after, but the variations are endless: site prefixes ([Torrent911]), language tags (VOSTFR, TRUEFRENCH), codec (x264, H.265), resolution (1080p, 4K), release groups.

Stage two: query TMDB with the extracted title and pick the right result. A search for "Breaking Bad" returns dozens of results, including films and documentaries with the same name. You have to score and decide.

Three metrics to measure extraction quality: cleanliness (% of titles with no residual technical tag — a stray 1080p, WEB-DL, or S01E02 that should have been stripped), coverage (% of non-empty titles), and year recall (how often the year is correctly extracted). Cleanliness is the main metric. A "dirty" title produces false positives or zero results on TMDB. Coverage was already near 100% from V0.

V0 → V4: measure before you optimize

The test bench is entirely offline. It replays the extraction algorithm on all 39,188 corpus filenames, computes the metrics, and returns a report in a few seconds. Deterministic, 100% replayable, no network calls. This is where the AI assistant genuinely helps: building the bench, writing the variants, analyzing the 65 remaining leaks to find patterns. Not "the AI optimized the algorithm" — more like "the AI let me iterate five times in one hour".

V0 applies basic separator normalization. 10.84% cleanliness, 34,939 leaks. Catastrophic baseline, but that's the honest starting point.

V1 strips 8 known technical tags. 50%. The corpus has far more than eight conventions. This approach doesn't scale: every site and every release group has its own patterns.

V2 introduces the central idea: don't try to enumerate all known tags, just truncate at the first technical marker found. Release naming places the title at the start and technical metadata after. Cutting at the first tag brings cleanliness from 10.84% to 99.83%. 65 leaks remain across 39,000 names. This single shift in approach delivers 99% of the total gain.

// Step 1 — bracket prefixes are stripped first, no enumeration needed:
// [Torrent911], [HorribleSubs], [FW]... one pattern covers them all.
$clean = preg_replace('/\[.*?\]/', '', $clean);
$clean = preg_replace('/[._()[\]{}:]+/', ' ', $clean);

// Step 2 — everything after (and including) the first technical marker is discarded.
// This is what takes cleanliness from 10.8% to 99.8%.
$title = preg_replace(
    '/\b(multi|vff|truefrench|french|vostfr|bluray|web-?dl|hdtv|x264|x265|hevc'
  . '|10bit|remux|2160p|1080p|720p|480p|uhd|hdr|dts|aac|ac3|proper|repack'
  . '|extended|directors?-?cut|complete|s\d{1,2}e?\d{0,4}|e\d{2,4})\b.*/i',
    '', $clean
);
// s\d{1,2}e?\d{0,4}: the {0,4} (instead of {0,2}) catches 3-4 digit episodes
// (One Piece S01E1164, Plus Belle La Vie S12E248) — found by analyzing 39,000 torrents.

V3 fixes a case revealed by analyzing the 65 remaining leaks. Three and four digit episode numbers weren't captured: S01E1164 for One Piece, S12E248 for Plus Belle La Vie. One change in the regex (\d{0,2} → \d{0,4}) cuts the leak count in half, from 65 to 28. You can't guess this by looking at ten examples manually. The corpus forces it to appear in the statistics.

V4 tests more aggressive normalization. Result: 29 leaks — one more than V3. The data says stop. Adding complexity to regress is a no.

Three lessons from this part.

The big lever was binary. Going from "keep everything" to "cut at the first technical tag": 10.8% → 99.8% cleanliness in one idea. Refinement (V3, V4) comes after, not before.

Large-scale data finds what the eye can't see. Without 39,000 examples, four-digit episode numbers stay invisible. The corpus forces them into the statistics.

Know when to stop. V4 is more complex and slightly worse than V3. YAGNI proven by data, not assumed.

TMDB matching and the premature optimization

Extracting a clean title is half the work. The other half: query TMDB and pick the right result from the candidates.

The scoring combines several signals. Title similarity counts for 65% (normalized comparison across all title variants). Year gap: 15%. Type coherence (film or TV): 10 to 18% depending on confidence. TMDB popularity (vote count): 12%, so obscure namesakes lose to well-known titles. Above 55: strong match. Between 35 and 55: quarantine for manual review. Below: rejected.

When the exact query doesn't produce a good result, the algorithm shortens it word by word until the confidence threshold is reached. A title with unusual punctuation may not match directly: we drop words from the end one at a time and keep the longest query that clears the threshold.

// Releases often append a subtitle or language tag after the actual title.
// Drop words FROM THE END, one at a time; keep the LONGEST query that clears
// the confidence threshold (55). The title is always at the start.
$words    = explode(' ', trim($title));
$minWords = max(1, count($words) - 4);   // at most 5 tries
$best = null; $bestScore = 0;
for ($n = count($words); $n >= $minWords; $n--) {
    $query = implode(' ', array_slice($words, 0, $n));
    foreach (tmdb_search($query, $year, $preferTv) as $cand) {
        $score = score_candidate($query, $year, $cand, $preferTv);
        if ($score > $bestScore) { $bestScore = $score; $best = $cand; }
    }
    if ($bestScore >= 55) break; // reliable match → stop truncating
}

This handles cases like "Avengers: Age of Ultron" (the colon disappears during cleanup) or foreign-language titles that don't match the English filename. Each iteration costs an API call, but avoids zero-match on titles with punctuation or special characters.

This is where an "optimization" had quietly slipped into the code. The variable $preferTv was supposed to signal to TMDB to prioritize TV results. To save API calls, I'd set the algorithm to query the "probable" type first (movie by default, unless the filename explicitly looked like a series), and only query tv if there were no candidates at all. The network test bench showed flattering numbers: 90% match rate for films, 96.7% for series.

I navigated it in real conditions. Every single TV show matched a film.

• Breaking Bad → Breaking Bad Wolf (an indie film with the same name)
• Game of Thrones → Game of Thrones: The Last Watch (an HBO documentary)
• The Mandalorian → The Mandalorian and Grogu (an upcoming film)
• One Piece → One Piece: Gold (a feature film)

0 correct out of 11 series.

Why the benchmark was showing green

The offline bench only measured extraction. Fair enough, that's its scope. It knew nothing about matching.

The network bench, though, provided the content type (movie or tv) directly from the known category of each corpus entry. It passed $preferTv = true for series directly. It never exercised the automatic type detection, because it didn't need to: the right answer was already baked into the request parameters. The auto-detection that ran in production had never been tested. The aggregate metric was green on a path that real navigation never took.

A benchmark tests what you tell it to test. Its blind spots are silent.

The fix: four levers

1. Always query both types.

Movie and tv are now queried every time. Scoring decides. Breaking Bad (TV series on TMDB, high popularity) beats any obscure namesake film on popularity score alone. This was the central regression: saving one API call had broken the selection logic. The fix is literally removing the condition that blocked the second type.

2. Folder structure as a signal.

Parsing the filename to guess film or series is fragile. A show organized in seasons — "Show/Season 1/", "Show/01/", "Show/02/" — may have no S01E01 in its folder name at all. The folder structure doesn't lie.

// A folder is a SERIES if it has multiple episodes (≥ 2 videos at top level)
// OR numbered season subdirectories (Season 1, Saison 3, 01, 02…).
// A film has a single video file. Folder SHAPE beats parsing the filename.
$videoCount = 0; $seasonish = 0;
foreach (scandir($folder) as $sub) {
    if ($sub[0] === '.') continue;
    if (is_dir("$folder/$sub")) {
        if (preg_match('/^(s\d{1,2}\b|saison|season|saga|arc|part|\d{1,3}$)/i', $sub))
            $seasonish++;
    } elseif (in_array(strtolower(pathinfo($sub, PATHINFO_EXTENSION)), $VIDEO_EXTS, true)) {
        $videoCount++;
    }
}
$isSeries = ($videoCount >= 2 || $seasonish >= 1);

This signal is used to weight the type coherence bonus in the score. It doesn't replace querying both types; it refines the decision.

3. Multilingual title merging.

TMDB returns multiple title variants per result: original, local, alternative. When you search for "Attack on Titan", TMDB may return "L'Attaque des Titans" (French) and "Shingeki no Kyojin" (Japanese). Deduplication by ID was keeping only the first title encountered: low similarity with the English query, potentially missed match.

Fix: aggregate all title variants by ID and score against the best one. Unexpected bonus: Spirited Away went from a weak match to a strong match, because "Sen to Chihiro no Kamikakushi" (the Japanese original title) was already in TMDB data and contributed to the similarity score.

4. Reinforced type coherence bonus.

When the folder structure clearly confirms a series (numbered subdirectories, dozens of video files), the tv type bonus in the score goes from 10% to 18%. The structural signal amplifies the scoring decision without overriding it.

Result: 11 series and anime matched correctly at high confidence, 18 films stay films. 0 → 11 on series.

Full source code
ShareBox is open source. The algorithm described in this article lives in functions.php: extract_title_year() (extraction), tmdb_match() (word-removal loop), tmdb_score_candidate() (scoring).

Conclusion

The large-scale data brilliantly optimized extraction. From 10.8% to 99.9% cleanliness in three measured iterations, across 39,000 names. The corpus surfaced a blind spot (S01E1164) no human would test manually, and it said stop when a more complex variant regressed by one unit.

The matching failed differently. An API cost optimization caused a regression invisible to the aggregate metrics, because the benchmark was feeding itself the right answer. Thirty seconds of real navigation found what a 96.7% success rate had hidden.

The real lesson: data optimizes brilliantly what it measures. A benchmark measures one path. It doesn't know others exist. The aggregate metric was an honest indicator; it wasn't a complete one. Testing in real conditions stays irreplaceable, even when everything is green.