Technology news and discussion on TalkFuse.
“Can AI Ace Your Exam? A Penn Law Experiment,” by Prof. R. Polk Wagner
A controlled experiment tested a frontier AI model on 11 real University of Pennsylvania Carey Law School final exams from spring 2026, with its answers graded blind alongside students’ work. Across the 10 exams for which grades were returned, the model passed every exam, matched or exceeded the class median on each, and placed in the top tenth on seven. The results raise questions about whether timed law exams measure course mastery or skill at producing the kind of answers the exams reward.
The model took each exam with and without the course syllabus and outline; the materials helped most on multiple-choice questions, while essay results varied. On two exams scored on an open scale, it outscored every enrolled student. Graders often spotted AI answers through formatting clues that could be corrected, though some also noted a consistent writing style unlike students’ timed work; one grader declined to mark them. The experiment’s author argues that faculty should be clearer about what each assessment is designed to measure and how that relates to legal practice.
In short
- The AI passed all 10 graded exams and reached the top decile on seven.
- Course materials helped on multiple-choice questions, with mixed effects on essays.
- The findings prompt questions about what law-school exams actually measure.
AI Politics Research Science Technology