Which Complete Recommender System Produces the Best Recommendations?

Recommender comparisons are usually comparisons of models. A model is not a recommender: a deployed system is an architecture plus a serving path plus whatever it does about cold users, long-tail items and competing objectives, and two systems built on the same model can differ more than two different models do.

So V3 asks one question — given the same available data and catalogue, which complete recommender system produces the best recommendations? — gives every entrant an identical data contract, and requires an identical output: a ranked Top-K slate. Nothing is required to use retrieval-then-ranking. A closed-form linear autoencoder, a graph propagation, an autoregressive transformer and a semantic-ID decoder all satisfy the contract in their own way, and are compared on what they produce rather than on how they are organised.

Quality is scored on its own. The Recommender Performance Index covers relevance, cold start, long tail, novelty and diversity, multi-objective value and robustness across data regimes. A system's score on a metric is its percentile in a frozen field — the pooled distribution of what every entrant measured on that metric and dataset, published as a fingerprinted constant set and never recomputed. Because the field is fixed, adding a system never moves an existing score; because nothing is scaled against a chosen system, no entrant sits at a fixed position, the two reference systems included. Training cost, inference cost, latency, model size and retraining burden are reported beside it and combined only in a separate Recommender Value Index.

The paper documents a scale that was withdrawn on the way here. Version 3.0 rescaled every metric so a popularity baseline read 0 and ItemKNN 100 — and on two of ten datasets the popularity baseline beat ItemKNN, which inverted the scale and rewarded systems for being worse. The reference itself could never show the fault, because it sat on 100 by arithmetic. Section 6.3 sets out what it cost, how it was caught, and why the results are not comparable across the change.

What the report measures

Inside

• Twenty-three complete recommender systems ranked by an index whose weights were fixed and hashed before any result existed.
• Every entrant labelled official, reference, reproduction or architecture proxy — with each proxy stating exactly what it is not.
• Scenario leaderboards for cold users, cold items, sparse and dense histories, long tail, coverage, novelty, diversity and multi-objective value.
• Stage-level diagnostics explaining why each system wins, including a batch-invariance test for ranking architectures that claim it.
• A full cost study: training cost, inference cost per thousand slates, latency, model size and retraining requirements against a stated reference deployment.
• Exposure-bias analysis on the datasets whose logs record impressions rather than only engagements, including a randomised-exposure check of whether offline evaluation picks the same winner as an unbiased one.
• Two compute tracks reported separately, so a weak architecture can be told apart from an under-trained one.
• The whole ranking re-scored under a second normalisation as a robustness check, plus the one defect we found and fixed before publishing.

Datasets

All datasets are public and used under their respective research licences. None is redistributed, and no customer data was used in any experiment.