← 返回首页

AI Contract Review Accuracy Measured on Real Clauses: How Legal AI Bench Scores Precision, Recall and F1

Count the unit before reading the score

A bare accuracy claim is not enough for a clause-level review. The buyer must know what the system was asked to identify and what the reference calls positive. A label should name a clause type, not stand for a general judgment that a contract is accurate. A positive prediction on a positive example is a true positive (tp); a positive prediction on a negative example is a false positive (fp); and a positive example that the system misses is a false negative (fn). These are the counts behind the reference metrics.

A contract can be evaluated against several clause types, so an agreement may contribute to more than one label decision. A true positive for cuad_governing_law is not a true positive for cuad_assignment. The agreement, a page or a highlighted passage is therefore not the automatic unit. The protocol must state whether it is evaluating a clause-type decision, a span or another object.

LegalBench’s task names show the possible granularity. Examples include contract_nli_confidentiality_of_agreement, contract_nli_limited_use, contract_nli_no_licensing, contract_nli_notice_on_compelled_disclosure, cuad_cap_on_liability, cuad_expiration_date, cuad_termination_for_convenience and cuad_governing_law. They identify categories to spot, not interchangeable names for a whole agreement.

Precision, recall and F1 in clause review

For a specified clause type, precision is tp / (tp + fp): of the items flagged as positive, the proportion that are true positives. In the scikit-learn reference, precision measures the classifier’s ability not to label a negative sample as positive. Recall is tp / (tp + fn): of the positive items in the reference, the proportion the system finds. It measures the ability to find all positive samples.

The same reference describes F-beta as a weighted harmonic mean of precision and recall. When beta = 1.0, F1 gives the two measures equal importance. In standard notation, it is the equal-weight harmonic mean, 2PR / (P + R), where P is precision and R is recall. The reference states that the best F-beta value is 1 and the worst is 0. F1 is a compact summary, not a third set of observations and not a replacement for the underlying counts.

The denominators need to be disclosed. Precision is undefined when tp + fp = 0; recall is undefined when tp + fn = 0; and the F-score is undefined when tp + fn + fp = 0. A score shown without those conditions can conceal an empty class or an undefined result.

Precision and recall must be read together because they answer different questions. High precision is compatible with low recall, while high recall is compatible with false positives. Precision describes the reliability of the positive output. Recall describes coverage of the positive reference. F1 compresses that tradeoff, but it cannot show whether the underlying errors were extra flags or missed clauses.

What counts as one hit?

The answer depends on the labeling convention. The original CUAD corpus contains annotations on long documents and is intended for evaluating extraction with span-prediction models. The CUAD paper describes over 13,000 annotations created with dozens of legal experts from The Atticus Project. Its task is to highlight salient portions of a contract that are important for a human to review.

LegalBench’s full text says it restructures that corpus into a binary classification task for each type of contractual clause. The stated purpose is to test whether a model can identify the distinguishing features of different clause types. Under that convention, one hit is one true-positive decision for one specified clause-type task. It is an identification count; it does not automatically prove that the exact original span was located.

If an evaluation retains span extraction, a predicted span must be compared with an annotated span under a declared matching rule. The cited material does not specify a universal exact-match, partial-match or overlap threshold. A vendor saying that it found a clause must therefore state whether it is reporting a binary clause decision, a span match or another unit. Without that disclosure, the hit count is not reproducible.

Finding a clause and proving a legal outcome are different measurements. LegalBench’s rule-application evaluation separately assesses whether an explanation is correct and whether it contains analysis. A clause-level identification score should not be presented as a score for legal reasoning, drafting quality or an explanation merely because both outputs came from the same tool.

Why one number cannot describe a legal-AI result

Even valid precision, recall and F1 values depend on how clause labels are combined. The scikit-learn reference defines support as the number of occurrences of each class in the true labels. Its macro calculation takes an unweighted mean across labels and does not account for label imbalance. Its weighted calculation averages by support; the documentation warns that the resulting F-score can be outside the interval between precision and recall. Its micro calculation counts true positives, false negatives and false positives globally.

Those choices expose different weaknesses. A support-weighted headline can be dominated by common clause types while performance on rare types remains weak. An unweighted macro result can reveal uneven performance but gives every label equal weight. A global micro result can hide which clause types generated the errors. The practical implication is that a single F1 can conceal strong performance on common labels and weak performance on rare ones. An overall score is not evidence that every clause category works equally well.

The scope of the benchmark matters as well. LegalBench contains 162 tasks gathered from 40 contributors and covers six types of legal reasoning. Its paper reports an empirical evaluation of 20 open-source and commercial LLMs. The project says each task has a dataset of input-output pairs and that task datasets can be used to evaluate how frequently a model generates the desired output. That is a broad evaluation structure, not one contract-review score. The paper’s evaluation of those models is historical benchmark evidence, not a current ranking of commercial products.

CUAD’s authors describe Transformer performance as nascent and say there is substantial room for improvement. That is the authors’ assessment of the research described in the paper, not a current measurement of a vendor. The LegalBench full text lists further limits: the initial tasks did not involve longer documents because context windows were constrained to a few pages; the benchmark considers English tasks, is skewed toward American law and contracts, focuses on questions with objectively correct answers, and evaluates IRAC abilities independently rather than as a multi-hop matter. A result from that setting cannot be generalized to every contract, jurisdiction or legal task.

What to ask a vendor before trusting a score

Ask for the following in enough detail to reproduce the result:

A defensible reading

A useful score tells the buyer what was counted, how labels were assigned, how a hit was matched, what the per-label tp, fp and fn were, how support was handled, and whether the output was identification, extraction, outcome or explanation. Read precision and recall together; treat F1 as a summary rather than the conclusion.

No precision, recall or F1 score for a named commercial product is reported here because the available benchmark evidence contains no per-tool measurement. Without the counting convention and those underlying counts, a single headline number cannot establish how a legal-AI tool performs on the clauses that matter to a particular review.