G-Eval: How It Scores LLM Output, and What Its Data Shows

What Is G-Eval?

G-Eval is an LLM evaluation metric in which a judge model scores generated text against plain-language criteria, guided by evaluation steps the model writes itself, and the final score is the average of the possible ratings weighted by their token probabilities. It needs no reference answer: in the SummEval setup, the judge reads the source article and the summary, or the summary alone for fluency.

Yang Liu and five co-authors at Microsoft Azure AI introduced the method in G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment, published in the EMNLP 2023 proceedings. They tested it on news summaries (SummEval), dialogue responses (Topical-Chat) and the QAGS factual-consistency benchmark.

Compared with a judge prompt that asks for a bare number, G-Eval adds:

• Criteria in words - each dimension, such as coherence or fluency, gets a short written definition and the scale it uses.
• Model-written steps - the judge first turns the criteria into numbered evaluation steps, which the paper calls auto chain-of-thoughts, and the same steps are reused for every item.
• Weighted score - the result is a probability-weighted sum over the allowed ratings, which yields decimals such as 3.4 instead of a stack of 3s.

G-Eval is one way to structure an LLM-as-a-judge setup: one dimension per prompt, on a fixed scale. It grades text, such as the wording of a summary, a reply or a dialogue turn.

How Does G-Eval Score an Output?

A G-Eval call is one prompt per dimension. The authors' released coherence prompt shows its parts in order:

• Task introduction and criteria - the prompt opens with "You will be given one summary written for a news article" and defines the dimension as "Coherence (1-5) - the collective quality of all sentences."
• Evaluation steps - generated once by the model from the criteria, then kept fixed. The coherence prompt's last step reads "Assign a score for coherence on a scale of 1 to 5, where 1 is the lowest and 5 is the highest based on the Evaluation Criteria."
• Input and form - the source article and the summary are inserted, and the prompt ends with "Evaluation Form (scores ONLY):" followed by "- Coherence:", so the reply starts with the score.
• Scoring function - the score is the sum, over every allowed rating, of the probability the model assigns to that rating multiplied by the rating, which is Equation 1 in the paper.

In a made-up example, the judge puts probability 0.6 on 3, 0.3 on 4, 0.05 on 2 and 0.05 on a token that is not a score. Renormalized over the three valid ratings, the weighted score is 3.26; without renormalizing, the stray token would pull every score down.

The function below computes that from the top log-probabilities an API returns for the score token. Renormalizing is an implementation choice, since the paper's formula does not say what to do with probability left on other tokens:

// G-Eval score from the top log-probabilities returned
// for the score token of the judge's reply
function gevalScore(top, scale = [1, 2, 3, 4, 5]) {
  let mass = 0;
  let sum = 0;
  for (const { token, logprob } of top) {
    const s = Number(token.trim());
    if (!scale.includes(s)) continue;
    const p = Math.exp(logprob);
    mass += p;
    sum += p * s;
  }
  // renormalize over the valid score tokens only
  return mass > 0 ? sum / mass : null;
}

Why Does G-Eval Weight Scores by Probability?

In the arXiv version of the G-Eval paper, the authors explain why a bare score falls short: they write that "one digit usually dominates the distribution of the scores", and that LLMs "usually only output integer scores, even when the prompt explicitly requests decimal values."

Weighting by probability answers both: the score becomes continuous, and summaries that would all receive a 3 separate by how much probability the judge put on 2 or 4. The paper says the method obtains "more fine-grained, continuous scores that better reflect the quality and diversity of the generated texts."

The same paper reports a side effect. In its results table, G-Eval-4 scored without probabilities has the higher Kendall correlation on every SummEval dimension, which the authors put down to ties: direct scoring "can lead to many ties, which are not counted as either concordant or discordant." They treat the higher Spearman correlation with probabilities as the fairer reading.

• Report Spearman next to Kendall - a judge that ties many items can post a higher Kendall's tau than a judge that ranks them better, so a Kendall gain alone does not show a better judge.
• Log the tie rate - record the share of tied item pairs in every run; the recount below shows how far averaging moves it.

Recomputing G-Eval's Published SummEval Correlations

The authors published their GPT-4 replies for all four SummEval dimensions in the nlpyang/geval repository, under the MIT licence, with its last commit on 4 February 2024. A script run on 2026-10-19 recomputed the paper's correlations from those files and tested what changes when fewer replies are used.

G-Eval-4 on SummEval: released replies vs paper
dim          rho    paper  tau    paper  ties
coherence    0.571  0.582  0.463  0.457  4.0%
consistency  0.501  0.507  0.420  0.425  2.3%
fluency      0.506  0.506  0.455  0.455  33.9%
relevance    0.564  0.547  0.453  0.433  2.5%
mean of 4    0.535  0.535  0.448  0.443
paper AVG column    0.514         0.418

# gpt4_eval.py, nlpyang/geval (excerpt)
temperature=2,
max_tokens=5,
top_p=1,
# ... (penalty and stop arguments cut)
# logprobs=40,
n=20

One reply vs the paper's - Probs row (rho / tau)
coherence    first 0.491/0.425  mode 0.544/0.475
             paper 0.560/0.472
consistency  first 0.469/0.423  mode 0.510/0.468
             paper 0.501/0.459
fluency      first 0.506/0.483  mode 0.512/0.490
             paper 0.505/0.473
relevance    first 0.491/0.425  mode 0.523/0.456
             paper 0.511/0.444

coherence    unparsed 0    replies differ 82.4%
consistency  unparsed 159  replies differ 95.1%
fluency      unparsed 7    replies differ 38.0%
relevance    unparsed 0    replies differ 90.6%

How Do You Run G-Eval When the API Returns No Token Probabilities?

Check the API first, since some return what G-Eval needs. The Chat Completions parameters in the openai-python type definitions include logprobs and top_logprobs, "An integer between 0 and 20 specifying the maximum number of most likely tokens to return at each token position", though not every model accepts them.

Google's Gemini API reference lists responseLogprobs and a logprobs setting whose number "must be in the range of [0, 20]". Where an API offers neither, sample: request n replies at a non-zero temperature and average the ones that parse.

The SummEval replies that the repository README points to show how many sampled replies are worth paying for. Spearman correlation by the number of replies averaged per summary, from the same run of the recount script:

Spearman rho by replies averaged per summary
n    coh    con    flu    rel    ties, coh
1    0.492  0.467  0.498  0.485  38.3%
2    0.528  0.482  0.510  0.522  24.5%
4    0.550  0.493  0.515  0.543  14.4%
5    0.556  0.495  0.513  0.550  12.0%
10   0.566  0.499  0.513  0.559  6.7%
20   0.571  0.501  0.506  0.564  4.0%

• Five replies - on coherence, consistency and relevance, five replies recover more than 80% of the Spearman gain that 20 replies give over one.
• Fluency - on the 1 to 3 fluency scale the gain stopped early: Spearman rose from 0.498 with one reply to its best, 0.515, at four replies, and 20 replies gave 0.506.
• Cost - extra replies multiply output tokens, and the openai-python definition of n warns that "you will be charged based on the number of generated tokens across all of the choices." The released script capped each reply at five tokens, which keeps the extra replies short.

A fallback that averages only the replies that parse, so a stray reply is skipped instead of counted as 0:

// Fallback for an API with no log-probabilities:
// sample n replies, average the ones that parse
function sampledScore(replies, min = 1, max = 5) {
  const vals = replies
    .map((r) => /^\s*(\d+(?:\.\d+)?)/.exec(r))
    .filter(Boolean)
    .map((m) => Number(m[1]))
    .filter((v) => v >= min && v <= max);
  if (vals.length === 0) return null;
  return vals.reduce((a, b) => a + b, 0) / vals.length;
}

Treat the sample count and the temperature as part of the metric: a run averaged over five replies and one averaged over 20 are not comparable item by item.

How Do You Write G-Eval Criteria and Evaluation Steps?

Three of the four released prompts share one template: one dimension per prompt, a definition, numbered evaluation steps and a form that ends on the dimension name. The fluency prompt has no steps and instead defines each point of its 1 to 3 scale, starting with "1: Poor. The summary has many errors that make it hard to understand or sound unnatural."

• One dimension per prompt - the authors ran a separate prompt and results file for each dimension, which keeps every score tied to one definition.
• Anchor every scale point - write what a 1, a 3 and a 5 look like; the released coherence prompt says only that 1 is the lowest and 5 the highest.
• Freeze the steps - generate the evaluation steps once, review them and store them with the prompt. DeepEval's documentation says that relying on criteria alone "makes the metric regenerate fresh CoT steps every run", so two runs can grade against different steps.
• Give the judge what the criterion needs - the released fluency prompt shows only the summary, while the consistency prompt and the relevance prompt include the source article, since those criteria compare the two.
• Constrain the reply - end the prompt on the field name, cap the output length and parse strictly, then count unparsed replies per run.
• Calibrate against people - have people score a sample of the same items and measure agreement before trusting a new prompt; the judge-against-human check with Cohen's kappa is laid out in the guide to chatbot evaluation.
• Hold the items fixed - score every prompt version on the same reviewed, versioned golden dataset, so a score change can come only from the judge or the system.

When Is G-Eval the Wrong Choice?

G-Eval rates the wording of an output against a written definition, so whether a fact, a format or an action is right needs a different check.

• Factual claims against a source - a 1 to 5 consistency score hides which sentence was wrong. Split the output into claims and check each against the source, as the guide to groundedness shows.
• Format, length and exact values - valid JSON, required fields or a date pulled from a receipt need code, which returns the same answer on every run at no model cost.
• Outputs from the judge's own model family - the paper's own analysis found that G-Eval-4 "always gives higher scores to GPT-3.5 summaries than human-written summaries, even when human judges prefer human-written summaries."
• Scores compared across judge versions - hosted judge models change: Chen, Zaharia and Zou found GPT-4's accuracy at telling prime from composite numbers fell from 84% in its March 2023 version to 51% in its June 2023 version, so compare G-Eval scores only across runs with a pinned, dated model.
• Agents that act - an agent's reply that it changed an order's shipping address can be coherent, fluent and false. Grade it with a judge that can read the changed records, the design described in agent-as-a-judge.

G-Eval vs QAG vs DAG

QAG, short for question-answer generation, names metrics built on the idea behind the QAGS metric of Wang, Cho and Lewis (ACL 2020), which asks questions about a summary and its source and expects "similar answers if the summary is factually consistent with the source." DAG is the name DeepEval gives its decision-graph metric, a "deep acyclic graph" whose task and judgement nodes "still use an LLM" and which can run a G-Eval metric as one of its nodes.

For retrieval pipelines, the faithfulness and context scores built on the claim-checking idea are compared in RAG evaluation metrics. The open-source Ragas library ships such scores as classes, and the guide to Ragas metrics lists each class and which ones need an LLM.

Checking Agent Actions That G-Eval Scores Cannot Confirm

A G-Eval coherence prompt would rate the reply "I've moved the order to the new shipping address and emailed the customer" highly whether or not either happened. When an agent's tools write to real systems, the check has to read those systems.

One npm command installs Rook CLI, the command line for TestMu AI's Agent Assurance, which runs record-level checks on agents you build before they ship. The current release, still pre-alpha, is publicly installable:

npm install -g @testmuai/rook

Releases target macOS and Linux, and 64-bit Windows through npm or WSL. Installing with npm needs a Node.js 20 or newer runtime already present; Homebrew and a checksum-verifying shell installer, which cover macOS and Linux only, ship with a Node runtime of their own.

• Criteria from the agent itself - Rook CLI derives scenarios, each carrying acceptance criteria, from the agent's code or, when the source is unavailable, from a PRD, a docs folder or an API spec; the default mix covers the functional and adversarial classes.
• Model, probe or rule - each criterion is scored by a model judge (check: llm_judge), by a read-only probe of the system the agent changed (check: mcp_probe), or by a rule such as a regex.
• Evidence from the run - every observed tool call is matched to the tools the agent declares, its must-not-call entries included, and files are read under the paths its profile declares.
• A claim is not evidence - a reply announcing the new shipping address never passes a criterion by itself; the order record has to show it.
• Pass, Fail or Unable to Verify - a criterion the run left no evidence for comes back as Unable to Verify, and the pass rate counts only scenarios with a decided verdict.

Most eval and observability tools score the words an agent produced and the trace it logged, the layer a G-Eval rubric works on. Agent Assurance also reads the files and records a run touched, and reports any criterion it could not decide as a separate verdict. Point runs at a staging environment, since whatever the agent writes there is a genuine write that no run undoes.

Runs happen before release; monitoring production traffic, blocking actions at runtime and benchmarking the underlying model are jobs for other tools. The Agent Assurance overview ranks observed effects above the agent's account of its work, calling an agent's own reply "the weakest available signal."

How to Validate a First G-Eval Prompt Before Comparing Versions

Pick the one quality your users complain about most, write its definition and scale anchors, generate the evaluation steps once and freeze them. Score a set of items your team has already rated, read the agreement and the tie rate, and only then use the prompt to compare versions.

If your judge API returns log-probabilities, weight by them; if not, start at five sampled replies per item and keep that number fixed, since in the coherence recount above, averaging five replies cut tied summary pairs from 38.3% to 12.0%. When the output being judged is an action, follow the Rook CLI installation guide and run the CLI against the agent's staging deployment, so Agent Assurance checks the records the run changed.