NSL Research · Benchmark v3.2 · 19 September 2026
Benchmark methodology
Download the results
Free, ungated, CC BY 4.0. The /v3.2/ path is stable and never changes once published; /latest/ always points at the current release. Cite the versioned one.
One row per system: rank, index, the six dimension scores, win rate, cross-dataset spread, cost, latency, model size and provenance.
/research/benchmark/v3.2/leaderboard.csv
The board, the scenario boards, the secondary boards, the dimension weights and metric definitions, the datasets, the robustness check and every caveat, with the licence and citation embedded.
/research/benchmark/v3.2/leaderboard.json
The unnormalised measurements: one row per system, dataset and metric. This is what the index is computed from, so it is what to use if you disagree with the index.
/research/benchmark/v3.2/per-dataset-metrics.csv
Cold users, cold items, sparse and dense histories, long tail, coverage, novelty, diversity and multi-objective value, with the whole field ranked on each.
/research/benchmark/v3.2/scenarios.csv
The ten public datasets with user, item and interaction counts, Gini coefficient, median history length, whether impressions are logged, and the licence each is used under.
/research/benchmark/v3.2/datasets.csv
Everything the published page is rendered from, unmodified. Nothing on the page is hand-written, so this file and the page cannot disagree.
/research/benchmark/v3.2/benchmark-full.json
How to cite this benchmark
Licensed CC BY 4.0: reuse it anywhere, including commercially, with attribution. Cite a version rather than the page, because the page shows the current release and the numbers move between releases.
Plain citation
NeuronSearchLab. "NSL Recommender Leaderboard v3.2: which complete recommender system produces the best recommendations?" 19 September 2026. https://www.neuronsearchlab.com/research/recommender-leaderboard
BibTeX
@misc{nsl_recommender_leaderboard_v32,
title = {NSL Recommender Leaderboard v3.2: Which Complete Recommender System Produces the Best Recommendations?},
author = {{NeuronSearchLab}},
year = {2026},
howpublished = {\url{https://www.neuronsearchlab.com/research/recommender-leaderboard}},
note = {Benchmark v3.2, generated 19 September 2026; index weights fingerprint 52706396658de50c; calibration fingerprint 05eb9f3f645e581b. Accessed: YYYY-MM-DD.}
}If you are quoting a single number, quote the version and the metric with it. Index v3.2 and index v3.0 are not the same scale, and the changelog below says so explicitly.
The datasets
10 public datasets, chosen to span the regimes recommenders behave differently in: sparse long-tailed commerce, dense short-video feeds with impression logs, extremely long music sequences, news with permanent cold start. All used under their stated licences; none redistributed.
| Dataset | Regime it covers | Users | Items | Interactions | Median history | Impressions | Licence |
|---|---|---|---|---|---|---|---|
| amazon-beauty-2023 | e-commerce reviews; very sparse, very long-tailed, rich item text | 24,979 | 15,588 | 204,659 | 6 | No | Amazon Reviews 2023 (McAuley Lab, UCSD) — research use; see dataset page. |
| goodreads-poetry | books; medium catalogue, slow-moving, strong content signal | 5,791 | 2,903 | 67,954 | 7 | No | Goodreads datasets (UCSD McAuley Lab) — academic use only; redistribution not permitted. |
| hm-fashion | fashion e-commerce; seasonal catalogue turnover and heavy supplier concentration | 32,343 | 39,979 | 1,053,294 | 16 | No | H&M Personalized Fashion Recommendations (Kaggle competition data) — competition rules apply; research use. |
| kuairand-pure | short-video feed; impression logs; eight engagement signals; a randomised-exposure slice | 25,103 | 7,539 | 863,799 | 23 | Yes | KuaiRand (Gao et al., CIKM 2022) — CC BY 4.0. |
| kuairec | short-video feed; near-fully observed user-item matrix, so exposure bias can be measured directly | 1,341 | 1,821 | 84,911 | 33 | Yes | KuaiRec (Gao et al., CIKM 2022) — CC BY 4.0. |
| lastfm-1k | music; extremely long, repetitive sequences — the regime sequential models were designed for | 761 | 42,638 | 13,310,148 | 9,016 | No | Last.fm-1K (Celma, 2010) — research use. Mirror hosted on Hugging Face. |
| mind-small | news; impressions; a catalogue that turns over in days, so almost every target item is cold | 27,095 | 3,711 | 171,444 | 4 | Yes | Microsoft MIND — Microsoft Research License Terms, research use only. |
| ml-1m | movies; small catalogue, dense histories; the field's most-reported dataset | 5,970 | 3,125 | 574,376 | 54 | No | GroupLens / MovieLens Terms of Use — research use permitted, redistribution of the dataset not permitted without permission. |
| ml-32m | movies; large catalogue, long time span, sparse per-user coverage | 35,073 | 10,476 | 3,093,573 | 43 | No | GroupLens / MovieLens Terms of Use — research use permitted, redistribution of the dataset not permitted without permission. |
| retailrocket | e-commerce; sparse sessions, large catalogue, strong recency | 1,499 | 2,156 | 19,062 | 7 | No | RetailRocket e-commerce dataset — CC BY-NC-SA 4.0. |
Metric definitions
What each measurement is, stated so a number quoted from this benchmark can be checked against the definition it was computed under. All ranking metrics are computed over the full catalogue with no sampled negatives.
NDCG@10
Recall@20
MRR@10
Hit rate@10
Catalogue coverage@20
Novelty@20
Diversity (ILD@20)
Tail share@20
Popularity percentile@20
Provider Gini
Multi-objective weighted utility
Multi-objective negative signal rate
How the index is weighted
The Recommender Performance Index is quality only. Weights were fixed and hashed on 2026-08-22, fingerprint 52706396658de50c, before any result existed.
Cost, latency, model size and retraining burden are deliberately not in it. They are reported beside the index and combined only in the separate Recommender Value Index, because the trade-off between quality and cost is a business decision rather than a measurement. If you weight the dimensions differently - most teams do - recompute the index from per-dataset-metrics.csv with your own weights. The exports exist so you can disagree with ours.
Cohort and cold-start definitions
“Cold start” means several different things and benchmarks rarely say which. These are the exact cohorts behind every cold-start and robustness number on the board.
Cold users (1-4 interactions)
Cold items
New items since training
Sparse histories
Dense histories
Warm users (5-19) and heavy users (20+)
Tail target items
Reproducing a score
Everything needed to recompute any number on the board by hand, or to score your own system against this benchmark without rerunning it.
The data
The measurements
The scoring
The weights
The robustness check
The implementations
Model versions and original sources
Each entrant, the paper or system it comes from, and the confidence level of our implementation. A result from an architecture proxy is evidence about the architecture, never about a named commercial product - and each proxy states what it is not.
ItemKNN + Multi-task ranker
Reproductionmulti-stageMa et al., 'Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts', KDD 2018; Zhao et al., 'Recommending What Video to Watch Next', RecSys 2019.
ItemKNN + MLP ranker
Referencemulti-stageSarwar et al., WWW 2001 (retrieval); Covington, Adams & Sargin, RecSys 2016 (the deep-ranking stage this follows).
VS-KNN
ReferenceneighbourhoodLudewig & Jannach, 'Evaluation of Session-based Recommendation Algorithms', UMUAI 2018.
EASEᴿ
Referencelinear-autoencoderSteck, 'Embarrassingly Shallow Autoencoders for Sparse Data', WWW 2019.
SASRec
ReproductionsequentialKang & McAuley, 'Self-Attentive Sequential Recommendation', ICDM 2018; trained with the full softmax of Klenitskiy & Vasilev, RecSys 2023.
VS-KNN + GBDT
Referencemulti-stageLudewig & Jannach, UMUAI 2018; Ke et al., NeurIPS 2017 (LightGBM).
Recency-Weighted Popularity
Referencenon-personalisedJi et al., 'A Re-visit of the Popularity Baseline', SIGIR 2020.
LightGCN
ReproductiongraphHe et al., 'LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation', SIGIR 2020.
ItemKNN
ReferenceneighbourhoodSarwar et al., WWW 2001; shrinkage per Bell & Koren, ICDM 2007.
BERT4Rec
ReproductionsequentialSun et al., 'BERT4Rec', CIKM 2019; training regime per Petrov & Macdonald, RecSys 2022.
PLUM-style SID generative retrieval
Proxygenerative-semantic-idSohn et al., 'PLUM: Adapting Pre-trained Language Models for Industrial-scale Generative Recommendations', arXiv:2510.07784 / WWW 2026.
Not the production system. PLUM adapts a large pre-trained language model and a multimodal item encoder on YouTube-scale logs. This entrant trains a small decoder-only transformer from scratch on public data with a TF-IDF/SVD content encoder. It shares the mechanism, not the scale, the data, or the pre-training.
Two-Tower + DCN-v2
Referencemulti-stage-neuralYi et al., 'Sampling-Bias-Corrected Neural Modeling for Large Corpus Item Recommendations', RecSys 2019 (retrieval); Wang et al., 'DCN V2', WWW 2021 (ranking).
GRU4Rec
ReproductionsequentialHidasi et al., 'Session-based Recommendations with Recurrent Neural Networks', ICLR 2016.
Cheap full pass + Option-Attention reranker
Reproductionmulti-stage-neuralZhou et al., KDD 2018 (target attention); Yi et al., RecSys 2019 (the pooled two-tower stage this degenerates to).
TIGER
Reproductiongenerative-semantic-idRajput et al., 'Recommender Systems with Generative Retrieval', NeurIPS 2023 (arXiv:2305.05065).
Two-Tower + RankMixer-style ranker
Proxymulti-stage-neuralZhang et al., 'RankMixer: Scaling Up Ranking Models in Industrial Recommenders', CIKM 2025 (arXiv:2507.15551).
Not the production system. RankMixer is a billion-parameter ranker over ByteDance's own sparse feature space with a sparse mixture-of-experts variant, trained on trillion-scale logs. This entrant implements the per-token feed-forward and token-mixing design over this benchmark's shared feature space, at a size where its scaling argument cannot be exercised.
iALS
Referencematrix-factorisationHu, Koren & Volinsky, ICDM 2008; tuning per Rendle et al., 'Revisiting the Performance of iALS', RecSys 2022.
EASEᴿ + LambdaMART
Referencemulti-stageSteck, WWW 2019 (retrieval); Burges, 'From RankNet to LambdaRank to LambdaMART', MSR-TR-2010-82 (ranking).
Option-Attention Scorer (full catalogue)
Reproductionoption-scoringZhou et al., 'Deep Interest Network for Click-Through Rate Prediction', KDD 2018 (target-conditioned attention); Zaratiana et al., 'GLiNER2', arXiv:2507.18546 (schema-conditioned one-pass scoring).
HSTU (Generative Recommenders)
Reproductiongenerative-sequentialZhai et al., 'Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations', ICML 2024 (arXiv:2402.17152); reference code at github.com/meta-recsys/generative-recommenders.
Calibrated Multi-Outcome Option Scorer
Reproductionoption-scoringKumar, Sarawagi & Jain, 'Trainable Calibration Measures from Kernel Mean Embeddings', ICML 2018; Platt, 'Probabilistic Outputs for Support Vector Machines', 1999; Guo et al., 'On Calibration of Modern Neural Networks', ICML 2017; Ma et al., KDD 2018 (multi-outcome heads).
Netflix-style foundation model
Proxyfoundation-modelNetflix Technology Blog, 'Foundation Model for Personalized Recommendation', March 2025; Netflix Personalization, Search and Recommendation workshop, 2025.
Not the production system. Netflix's model is trained on their own interaction logs at a scale and context length not reproducible here, and its published description omits most implementation detail. This entrant implements the four design choices Netflix stated publicly — unified autoregressive user model, multi-token prediction, multi-layer readout, content-aware item embeddings — at benchmark scale.
OneRec-style session generator
Proxygenerative-sequentialDeng et al., 'OneRec Technical Report', arXiv:2506.13695; 'OneRec-V2', arXiv:2508.20900.
Not the production system. OneRec is a production system trained on Kuaishou's logs with a multimodal tokenizer, a mixture-of-experts decoder at a scale far beyond anything here, and a reward model built from proprietary engagement data. This entrant implements session-wise generation and preference alignment at benchmark scale on public data.
ItemKNN + Option-Attention reranker
Reproductionmulti-stage-neuralZhou et al., KDD 2018 (target attention); Covington, Adams & Sargin, RecSys 2016 (the stage this replaces).
Popularity
Referencenon-personalisedCremonesi, Koren & Turrin, RecSys 2010 (TopPop).
Content TF-IDF
ReferenceneighbourhoodStandard content-based baseline; Lops et al., Recommender Systems Handbook 2011.
Phoenix-style unified transformer
Proxyunified-transformerxAI, 'x-algorithm' (github.com/xai-org/x-algorithm), phoenix/README.md, January 2026.
Not the production system. Phoenix in production is a Grok-derived transformer with a 2560-dimensional embedding, eight layers, 1022 history positions and a 64-action taxonomy, trained on X's own logs with a multimodal post encoder. This entrant implements the published architecture at benchmark scale, with the harness's own semantic-ID tokenizer and whatever action taxonomy the dataset actually logs — often a single one. It is not X's model.
Changelog
Every release, what changed, and - the part most benchmarks leave out - whether scores are comparable with the release before it.
v3.2
19 September 2026Scores comparable with previousCurrent release. New entrants added; every previously published score unchanged, because the scoring field was frozen in v3.1 and adding a system does not move a percentile in a frozen field.
- · 27 complete recommender systems across 10 public datasets.
- · Scores comparable with v3.1: same frozen field, same weights, same fingerprint.
- · Machine-readable CSV and JSON exports published at versioned URLs for the first time.
v3.1
2026-08-22Scores not comparable with previousScoring rebuilt on a frozen-field percentile. Not comparable with v3.0, which was withdrawn before publication.
- · Normalisation changed to a frozen-field percentile: a score is where a measurement falls in the pooled distribution of every entrant's measurement on that metric and dataset, published as a fingerprinted constant set.
- · Because the field is frozen, adding an entrant no longer moves any published score, and no system is pinned to either end of the scale.
- · Weights frozen 2026-08-22, fingerprint 52706396658de50c.
- · Calibration constants fingerprinted 05eb9f3f645e581b (167 constants).
v3.0
2026-08-01Scores not comparable with previousWithdrawn before publication. Recorded here because a benchmark that hides its own defects is not a source.
- · Used a two-point anchored scale dividing by (ItemKNN − Popularity).
- · On two of ten datasets that difference was negative, which inverted the scale and rewarded systems for being worse.
- · Where the difference was small, a worse result could reach −381 before being floored.
- · The reference system could never show the fault, because it sat on 100 by arithmetic. Scores are not comparable across the change.
What this benchmark cannot tell you
The boundary of what the measurements support. A result quoted outside it is a wrong result, and these are reproduced here rather than only on the board so a reader arriving at the data files still meets them.
- 01
On the one dataset that inserts randomly-chosen items into real feeds — the only place an unbiased estimate is available — the offline ordering and the unbiased ordering correlate at 0.17 and name different winners. That is a limit on what any offline recommender benchmark, including this one, can tell you.
- 02
Public offline datasets are not production environments. Offline relevance is a proxy for user value, and the mapping between them is platform-specific.
- 03
Several entrants are architecture proxies for proprietary systems. They implement a published mechanism at benchmark scale and are not those companies' production systems.
- 04
No single recommender wins every scenario. The winner changes 5 ways across the scenario boards.
- 05
Entrants added after the calibrating release are measured in their own run rather than beside the systems they are ranked against. Every board is unaffected — a score is a percentile in a frozen distribution, not a rank against whoever ran that day — but a head-to-head significance cell exists only where both systems' result files carry per-user vectors. Which cells those are is published in `pairwise_provenance`. Cost and latency use the same instance types across runs, not the same job.
- 06
Serving latency is offline batch throughput on the benchmark host, not a production p99. Multi-stage systems additionally build their ranker's features online here, which a deployed pipeline precomputes — so their measured serving cost is an upper bound and the cost gap to single-stage systems is overstated.
- 07
Scale is the untested variable: the central claim of the industrial generative systems is a scaling claim this benchmark is too small to test.
- 08
The index scores each measurement by its percentile in a frozen, published distribution. That records where a system placed, not by how much: doubling the best cold-start result and beating it by a hair score the same. Raw metric values are published beside every score, and the scenario boards are stated in raw metrics.
- 09
This is index version v3.1 and its scores are not comparable to v3.0's. The v3.0 scale rescaled every metric so Popularity read 0 and ItemKNN 100; it was withdrawn before publication because on two of ten datasets Popularity beat ItemKNN, which inverted the scale, and where the two were close a worse result could reach -381. The effect was to flatter ItemKNN by about six places. Under v3.1 nothing is pinned and ItemKNN places 9th.
Interpreting the results
Does the top of this leaderboard tell me what to deploy?
Why is a well-known commercial system not ranked here?
Are scores comparable between releases?
Does a higher index mean better recommendations for my users?
Why are cost and latency not in the index?
Can I cite a specific version?
Can I add my own system?
Related reading
Add a system, or correct one
A new architecture is a class in the harness and a rerun. Because every score is a percentile in a frozen field, adding one leaves every published score exactly where it was.