Rethinking Ranking in the LLM Era [Testμ 2026]

Three Travellers, One Query

Three people type the same query into a hotel search engine, and their priorities have almost nothing in common.

• A conference traveller - proximity to the venue, reliable Wi-Fi, breakfast and flexible check-in.
• A family with a toddler and elderly parents - family rooms and a quiet neighbourhood.
• A backpacker - price, nightlife and proximity to public transport.

Asked which of the three rankings is correct, her answer is all three.

That is why production ranking balances relevance, business goals, diversity, price and ratings at the same time. No single model carries all of it.

Today’s architecture runs in stages. Retrieval narrows millions of items to a manageable candidate set, a first-stage ranker predicts signals such as relevance or engagement, and a re-ranker applies additional business objectives and constraints before the final ordering.

She is complimentary about it. The design scales extremely well and has served the industry for years.

The Objective Function Problem

The strain shows when product requirements get nuanced. Her example is a product manager asking to prioritise hotels that are great for families unless quality drops significantly, while still encouraging discovery without hurting long-term engagement.

Every clause in that sentence is reasonable. Someone then has to turn it into a mathematical objective function with thresholds and optimisation weights.

Those weights come from offline experiments, parameter tuning and countless A/B tests. As the objectives get more nuanced, the translation gets harder.

Which raises the question the rest of the talk answers. What if the system optimised for language directly, with no weights, no equations and no handcrafted optimisation function, just intent expressed in natural language.

Expressive, and immediately loaded with engineering questions. Can language be optimised reliably, how precise is unless quality drops significantly, can models reason consistently, can those decisions be trusted, and how should they be evaluated.

She frames the rest of the session as an exercise, replacing a traditional re-ranker with an LLM, which decomposes into five questions.

• Is prompting alone enough - given a general-purpose LLM, does a prompt get you there?
• How do you teach it your domain - if prompting falls short, what supplies the missing knowledge?
• How do you refine its behaviour - once it understands the domain, how do you capture more nuanced preferences?
• How do you make it fast enough - what gets it inside a production latency budget?
• How do you know it is better - what evidence separates a better ranker from a different one?

Prompting a Base Model

The simplest approach is a general-purpose LLM and a plain prompt: you are a hotel ranking assistant, rank these hotels for an anniversary trip to Napa, and explain your reasoning.

The response is already good. Without any additional training the model understands romance, vineyards, luxury and memorable experiences, and infers that a boutique vineyard resort fits better than a business hotel.

Traditional ranking models struggle with exactly that, because they have no access to broad world knowledge.

Her own test is whether you would deploy it tomorrow, and her answer is probably not. The behavioural data tells a different story.

Users planning anniversary trips still book the Marriott. They may be familiar with the brand, they may have loyalty points, they may prefer a lower price point, they may want flexible cancellation.

None of those preferences appeared in the prompt, and most cannot be inferred from world knowledge alone. A base LLM understands language well; it does not yet understand your platform or your users.

Behaviour Versus Intent

Behavioural models learn from millions of historic interactions, and they are strong at predicting what users are likely to do. They can learn that families consistently choose the hotel with the free shuttle to Disneyland even when the query never mentions a shuttle.

LLMs excel differently. Asked for a romantic anniversary getaway, they infer that a boutique vineyard hotel with a spa is the better experience without any historical interaction data behind that judgment.

Her point is that the two are not competing on the same problem. One predicts behaviour, the other reasons about intent, and modern ranking systems increasingly combine them.

Behavioural models capture implicit preferences revealed through user behaviour. LLMs capture explicit intent expressed through language.

Teaching the Model Your Platform

Supervised fine-tuning is how the missing knowledge gets in. The prompt states the task, rank hotels by booking likelihood, and the labels teach what booking likelihood actually means on your platform.

Across millions of examples the model picks up patterns nobody wrote down: brand familiarity matters, loyalty programmes influence booking decisions, users often trade luxury for affordability, free cancellation increases conversion.

She flags that this should feel familiar, because it is conceptually what traditional learn-to-rank models already do. The difference is that the supervision arrives as complete ranking examples rather than as engineered features.

From Imitation to Preference

Optimising for historical behaviour is not always what you want. If a user says outright that they do not mind spending more for a memorable experience, the fine-tuned model still leans towards the Marriott, because that is what historical booking data suggests.

Ask a human the same query and they pick the boutique vineyard resort, because they have taken in context that goes beyond average historical behaviour. This is where supervised learning reaches its limit.

Preference data is how that gets collected. Instead of asking annotators to produce a perfectly ranked list, which is surprisingly difficult, you show them two rankings and ask which is better.

People are far more consistent comparing two alternatives than ordering a full list, and those comparisons capture subtle judgments that historical logs miss.

Reinforcement learning with human feedback, or more recently AI feedback, splits the training into two stages. Human preferences train a reward model that predicts which rankings people prefer, then reinforcement learning optimises the LLM to produce rankings with a higher predicted reward.

The reward model becomes the learned objective function, and once you have it you can improve the ranker against it repeatedly. It also brings a great deal of complexity.

Direct preference optimisation is the simpler path. DPO skips the explicit reward model and the reinforcement learning loop, optimising the language model straight from the preference pairs by raising the probability of preferred rankings and lowering the probability of rejected ones.

On many tasks it reaches quality comparable to RLHF while dramatically simplifying the training pipeline.

Hard and Soft Constraints

Everything up to this point optimises a single objective, identify the best hotel for this user and query. Production does not work that way.

Her list of what arrives next will be familiar to anyone who has worked with a product manager: also optimise for affordability, promote new inventory, promote sustainable hotels, and do not hurt revenue.

Read closely, each of those is underspecified. Prioritise family-friendly hotels unless quality drops significantly leaves open how much quality you are willing to trade, and whether a 4.5-rated family-friendly hotel beats a 4.8-rated one that is not family friendly.

Prefer walkable hotels means something different to each person reading it. Stay within my budget of 250 if possible does not say whether 260 is reasonable or 290 is acceptable.

Her answer splits them in two rather than trying to resolve the ambiguity everywhere.

• Hard constraints - enforced deterministically before or after LLM ranking, with no reliance on the model to satisfy them.
• Soft constraints - left ambiguous on purpose, because the ambiguity is where the expressive power of an LLM comes from, with the requirement being consistent reasoning across requests rather than a fixed threshold.

Consistency on the soft side is bought with human preference alignment through techniques like DPO, plus continuous evaluation of the model.

Making It Fast Enough

Every component added so far improves ranking quality and adds computation. Larger models need more GPU memory, longer prompts increase context processing, reasoning generates more output tokens, and multi-stage pipelines increase end-to-end latency.

Her first question is whether a 70-billion-parameter model is needed for every ranking request. Many ranking decisions are relatively straightforward, and if a smaller model produces similar rankings most of the time, latency and infrastructure cost drop sharply.

Knowledge distillation is supervised fine-tuning again, with one substitution. A larger and more capable teacher model generates the labels instead of humans, reasoning through the ranking problem and producing both a ranking and an explanation.

Those outputs become training examples for a smaller student model, repeated offline, and the student is what gets deployed. Much of the teacher’s capability is retained at a fraction of the inference cost.

Prefix caching attacks the same bill from the other side. Requests carry different users and different hotels, but the system prompt is identical, and in many production systems that shared prefix runs past a thousand tokens.

The serving engine caches the model’s internal state after processing the prefix once, then restores it whenever a new request starts with the same token sequence. The model does not know it is happening, and most serving engines now ship with it, so it is not something a practitioner has to build.

Two more techniques adapt the compute to the query rather than to the fleet.

• Routing - decide online whether a query is complex enough to need a language model or a reasoning model at all.
• Cascading - let a smaller model handle the query first and escalate to a more capable model when the output does not meet the bar.

Evaluating the Reasoning

Traditional ranking metrics keep their value. NDCG, mean reciprocal rank and precision at K tell you whether the model produces rankings that agree with historical or human relevance judgments.

What they do not tell you is why the model preferred one ranking over another. With an LLM in the loop, the reasoning also has to line up with human preferences and explanations.

Having humans compare thousands of rankings by hand is taxing, so another LLM does the comparing. The judge receives the user query, the candidates, the ranking your model produced and an evaluation prompt, with the criteria themselves written in natural language.

The example she walked through has three parts worth copying.

• An evaluation rubric - the criteria spelled out explicitly, so the judgment stays consistent across examples.
• A request to explain its reasoning - chain of thought, which is known to make an LLM reason more consistently.
• A structured output - so judgments can be pulled and aggregated programmatically over many examples instead of read one at a time.

The judge rates multiple aspects of ranking quality against those criteria, rather than returning a bare score with nothing behind it.

Four Engineering Tensions

She closes by stepping back from the individual techniques to the design principles connecting them, framed as four tensions.

• Prediction versus reasoning - traditional models predict user behaviour from historical interactions, while LLMs add semantic reasoning that captures intent and context beyond those behavioural signals.
• Precision versus expressiveness - natural language makes ranking objectives far easier to express, and production systems still require precise guarantees through constraints and structured outputs.
• Generality versus controllability - foundation models bring general world knowledge, which then has to be aligned with the specific objectives, policies and requirements of your platform.
• Quality versus cost - larger models and deeper reasoning improve ranking quality while increasing latency and infrastructure cost, which is what distillation and adaptive inference are balancing.

Her final thought rejects the question she gets asked most, which is whether LLMs will replace traditional ranking models. After working through the whole life cycle she does not think that is the interesting one.

One side supplies efficient statistical prediction, robust infrastructure and decades of production experience. The other supplies semantic reasoning, natural language objectives and richer ways of aligning models with human preferences.

The design work is combining the two, and she left the audience with a reading list rather than a verdict.

Q & A Session

The Q&A box filled up during the talk, and the answers cover ground the slides did not.

• Could an agentic AI model work as a production re-ranker, or is that more capability than the job needs?

  Rhea Goel: More agentic styles are genuinely needed in production, because ranking has several parts. Her example is an e-commerce shopper looking for dresses to wear to a wedding in Hawaii, where the model needs skills to look up what the weather is like there, which dress styles a guest wears, and which colours to avoid. One component talks to the database, another does the reasoning, and that is how she thinks about production ranking too.

• Could an agent that makes humans more effective count as successful even if it completes fewer tasks autonomously?

  Rhea Goel: She noted the question was not about ranking, then answered it anyway. An agent’s success criteria can be that it automates some human task end to end, and she would not call it unsuccessful for failing to do everything. Any efficiency gain counts.

• How do you decide what makes an LLM answer better when two answers can both be correct?

  Rhea Goel: Direct preference optimisation is how the industry has been solving it. Collect pairwise preferences from humans, train a model to mimic the human annotator and act as the judge, expand the preference data set with it, then run DPO so the ranker mimics the human preferences. When you can articulate a rubric you put it in the prompt, and when you cannot articulate what makes one answer better, you ask humans which they prefer and let the model learn it.

• Do you anticipate new tools that act as ranking agents more efficiently and accurately than ChatGPT?

  Rhea Goel: Yes, and several already exist. The reading list she shared includes tools, models and techniques built specifically for ranking on top of foundation models, and many of those papers lean on distillation and smaller language models precisely because of the productionisation constraint. Every ranking problem she can think of carries subsecond latency requirements.

• Traditional rankers get scheduled retraining on fresh engagement data, so what does continuous improvement look like for an LLM ranking loop?

  Rhea Goel: All three of prompt refinement, fine-tuning and updating a distilled model. Most systems she has seen or read about are still hybrid, a traditional ranker plus an LLM re-ranker, so improvement continues on both sides: scheduled retraining on new engagement data for the traditional ranker, and a similar cadence for the fine-tuning component of the re-ranker. Prompt refinement can be automated too, and she pointed to an emerging area where agents evaluate the customer experience continuously and feed their reasoning back into the LLM ranker.

This session was part of Testμ Conf 2026, which ran across three days of sessions on agentic engineering and quality. Registrations for the next edition are already open on the Testμ Conference 2027 page.