Claude vs ChatGPT for Legal Work: Contract Drafting, Clause Spotting and Where Each Model Breaks

The comparison must be dated before it can be interpreted. The reproducible, openly documented head-to-head legal-reasoning evidence behind this article is LegalBench’s evaluation of GPT-4 and Claude-1 (v1.3), not tests of today’s Claude and ChatGPT products. Results for OpenAI’s GPT-4 and GPT-3.5 (text-davinci-003) were retrieved between May and August 2023; Claude-1 results were retrieved in July 2023. This is historical evidence from an older model generation, not a current product ranking. It supports conclusions about workflow patterns and failure modes, but it cannot identify a winner among current offerings.

Why a general benchmark score cannot choose a drafting model

A general benchmark compresses unlike abilities into one result. That makes it useful for broad comparison, but poorly suited to deciding which model to use for contract drafting.

Anthropic’s announcement of Claude 3.5 Sonnet, for example, said the model outperformed competitor models and Claude 3 Opus across broad evaluations. The named benchmarks were GPQA, MMLU and HumanEval—not legal benchmarks. That is a vendor claim about general capabilities, not evidence of superiority in contract review or drafting.

OpenAI’s GPT-4 technical report described a simulated bar-examination result around the top 10% of test takers. That likewise does not measure whether a model can identify a contract issue, recall the governing rule, apply it correctly, interpret a long clause or generate usable contract language. The report itself also stated that GPT-4 remained less capable than humans in many real-world scenarios.

LegalBench’s own conclusion is more useful for tool selection: assessments must be made task by task and informed by the nuances of the task. A single average cannot supply that evidence.

Six legal workflows, not one leaderboard

LegalBench reported separate category averages rather than one overall legal score. Table 2 covered issue spotting, rule recall, rule conclusion, interpretation and rhetorical understanding. A separate table assessed rule-application correctness and analysis. The rule-application responses were evaluated manually by a law-trained individual.

• Issue spotting: GPT-4 recorded 82.9, compared with 60.9 for GPT-3.5 and 58.1 for Claude-1. The paper reported GPT-4’s lead at p < 0.001 and described an absolute margin of more than 20 points. But its results varied sharply by task: balanced accuracy was 70.6% on learned_hands_torts, while three other learned-hands tasks each exceeded 95%. The relevant failure mode is therefore task-dependent consistency, not uniformly weak issue detection.

• Rule recall: GPT-4 scored 59.2, GPT-3.5 46.3 and Claude-1 57.7. Claude-1 outperformed GPT-3.5 at p < 0.05 and appeared almost level with GPT-4. It also scored higher than GPT-4 on rule_qa, international_citizenship_questions and nys_judicial_ethics. This was the only category in which the paper found Claude-1 comparable to GPT-4. At the same time, the authors said the overall results supported reports that LLMs often misstate law or cases. The failure mode is legal rule or authority misstatement, not a vendor-specific hallucination rate.

• Rule application: GPT-4 had 82.2 for correctness and 79.7 for analysis. The corresponding pairs were 58.5 and 44.2 for GPT-3.5, and 61.4 and 59.0 for Claude-1. GPT-4 outperformed both other models on correctness at p < 0.01. The documented errors were calculation failure—incorrect arithmetic or comparisons, including treating $50,000 as greater than $75,000—and rule-location failure, such as reasoning from a mandatory disclosure in 16 CFR § 310.3(a)(1) while citing the misrepresentation provision in 16 CFR § 310.3(a)(2).

• Rule conclusion: The category averages were 89.9 for GPT-4, 78.0 for GPT-3.5 and 79.5 for Claude-1. GPT-4 had the highest reported average, but the paper did not establish that it would remain superior on an unseen contract-drafting task. It separately cautioned that legal terms occur less frequently in general training corpora and that legal prompting may require additional background. It also framed rule-conclusion testing around whether a model can rely on latent knowledge of a rule. These are prompt-and-context cautions, not evidence of a universal rule-conclusion error.

• Clause interpretation: GPT-4 scored 75.2, GPT-3.5 72.6 and Claude-1 67.4. GPT-4 outperformed GPT-3.5 at p < 0.01, while GPT-3.5 outperformed Claude-1 at p < 0.01. The paper found the larger API models particularly effective on binary classification involving short clauses. Its named weaknesses were long-text degradation and multi-class classification degradation.

• Rhetorical understanding: The averages were 79.4 for GPT-4, 66.7 for GPT-3.5 and 68.9 for Claude-1. Nearly all evaluated models struggled with function_of_decision_section and oral_argument_question_purpose, which required assigning legal roles to questions or opinion excerpts across six or more categories. The failure mode is fine-grained legal-role classification.

A model-specific failure map, without invented attribution

GPT-4’s clearest measured weakness was inconsistency across issue-spotting tasks, while rule recall was its lowest category average. Claude-1’s issue-spotting and rule-recall averages were 58.1 and 57.7, and its interpretation result of 67.4 trailed both GPT-4 and GPT-3.5. Its rule-recall strength, however, prevents any claim that Claude-1 was uniformly inferior.

The arithmetic, rule-citation and legal-misstatement findings concern LLM outputs generally. The paper did not provide a Claude-specific or GPT-4-specific hallucination rate, so those errors should not be converted into vendor-wide percentages.

Why clause spotting is not contract drafting

The closest measured analogue to clause spotting is LegalBench’s interpretation category. Across the 38 CUAD contract-clause tasks, GPT-4, GPT-3.5 and Claude-1 all achieved balanced accuracy of at least 88%. On proa statutory-clause tasks, GPT-4 and GPT-3.5 both exceeded 90%.

Those results support short-clause classification: the model is being asked to identify or label language. Drafting requires generated contract text, and no cited study measures the quality of contracts produced by either GPT-4 or Claude-1. Rule-application analysis is also not a drafting test; producing an explanation after applying a rule does not establish that a model can draft a complete clause.

Length and label complexity further separate the tasks. On the Supply Chain Disclosure tasks, involving one- to two-page disclosures, the large commercial models averaged 74–75% balanced accuracy. On the multi-class MAUD merger-deal questions, GPT-4 fell to 47.8% accuracy. A strong short-clause spotting result therefore cannot be carried over to long, multi-class interpretation or drafting.

What this evidence cannot tell you

This evidence does not establish:

• which current Claude or ChatGPT model is better for legal work;
• which current model writes better contracts;
• a current vendor-specific hallucination rate;
• a current cross-vendor legal benchmark winner; or
• a procurement ranking for any present product.

The LegalBench paper also cautioned that some tasks might have entered model pretraining data, and its rule-application assessment depended on a law-trained human evaluator. Its results therefore require replication with representative present-day products and an organization’s own clause types, instruction patterns and review standards.

The defensible lesson is methodological: test issue spotting, rule recall, rule application, clause interpretation and rhetorical work separately; score correctness and analysis independently; include long and multi-class examples; and inspect calculations and rule citations. The historical results show why those distinctions matter, but they do not justify declaring a current drafting-model winner.