Research · v2
Which Recommender Architecture Wins, at Which Stage, Under Which Data Regime
Seventeen complete recommendation pipelines measured stage by stage across nine datasets, including feeds whose logs record impressions rather than only engagements.
Recommender benchmarks usually compare models. Production recommenders are not models; they are pipelines — an eligible catalogue narrowed by retrieval, ordered by a ranker, adjusted by policy, served as a slate. A benchmark that scores one model end to end cannot say which stage the quality came from, and cannot evaluate the hybrid configurations teams actually deploy.
This benchmark evaluates each stage against its own objective and then evaluates complete pipelines. Where a dataset logs impressions we separate not shown from shown and ignored; where a dataset randomises exposure we use it to check off-policy estimators against ground truth rather than assuming they are unbiased.
What the report measures
12 of 12
Top systems that are hybrids
every system in the top twelve pairs a retriever from one family with a ranker from another; no single-family system ranks above 13th.
−0.30
Offline vs. true ranking
rank correlation between naive offline evaluation and randomised-exposure ground truth. Offline evaluation does not merely inflate scores — it picks a different winner.
−11.7%
Cost of diversity reranking
mean relevance cost of MMR across 36 runs, against −1.7% for a per-supplier cap. Controls are not uniformly cheap.
Inside
- Seventeen complete pipelines ranked by a composite score whose weights were fixed and hashed before any results existed.
- Each stage judged against its own objective: retrieval on candidate recall, ranking on identical candidate sets, policies on what they cost.
- Scenario leaderboards for cold users, sparse behaviour, large catalogues, latency budgets and multi-signal feeds.
- Impression-set ranking with AUC over exposed-but-ignored negatives, where the logs support it.
- Off-policy estimators checked against randomised-exposure ground truth rather than assumed unbiased.
- Two compute tracks — fixed budget, and best achievable with GPU training — reported separately.
- Every prose claim re-derived from the stored result files before publication.
Datasets
All datasets are public and used under their respective research licences. None is redistributed, and no customer data was used in any experiment.