The University of Minnesota law school tested ChatGPT on exams — it was a C+ student

• The University of Minnesota law school gave ChatGPT its law school exams alongside real students.
• The chatbot was a C student, with a better aptitude for essays than multiple choice.

A group of University of Minnesota Law School instructors gave the AI bot ChatGPT four exams alongside real students and found the bot — considered the world's most advanced example of generative artificial intelligence — to be a C student on paper.

Curious about how ChatGPT might be used to both help students cheat and aid teachers in teaching, professors decided to test its capabilities.

Associate law school professor Jon Choi ran exam questions through ChatGPT, reformatted them to fit the exam, and mixed the resultant exams in with students' tests.

ChatGPT earned a C+ average and "a low but passing grade" in Constitutional Law, Employee Benefits, Taxation, and Torts, according to Choi. (It came in 36th out of 40 students in Con Law, 18th out of 19 in Employee Benefits, 66th out of 67 in Tax, and dead-last at 75 out of 75 students in Torts, according to the paper).

It performed better on essays than on multiple choice questions. The bot performed particularly poorly at multiple choice questions involving math, "which ChatGPT bombed."

In essays, it could recite legal rules and correctly describe cases (sans citations), but failed at "spotting issues" and providing deep reasoning or analysis, Choi said.

Of the experiment's three human graders, all of whom graded the tests blindly, at least two told Choi they had a hunch about which exams were from AI — and they were right. ChatGPT responses, for instance, used perfect grammar and were more repetitive. (Choi's co-authors — Kristin E. Hickman, Amy Monahan, and Daniel Schwarcz — acted as the three graders).

For now, spotting the difference in exams is relatively easy. "I think professors would get it right almost all the time," he told Insider. But, a ChatGPT-generated rough draft that a student revises would be, he said, "a lot more difficult to spot."

While exams are graded on a curve, and ChatGPT ranked as one of the worst students in class, Choi pointed out that he believed University of Minnesota Law School students to be relatively fierce competition. A considerable 97% of first-time test takers pass the Minnesota Bar exam, according to the university, and, in the fall of 2020, the pass rate for Minnesota Law grads taking the bar for the first time was the third best in the nation, behind Harvard and Yale.

"I would say just on knowledge of the law and the ability to synthesize and make legal arguments, Minnesota students are going to be some of the best in the country," Choi told Insider. "ChatGPT is performing near the bottom of a class in which 99% of the students will pass the bar exam. Almost all of the students are going to be successful in their careers. So, it's a tough comparison." While the results would have landed ChatGPT on academic probation, it would still likely be good enough to earn a law degree.

"Overall, ChatGPT wasn't a great law student acting alone," Choi tweeted on January 23, "but we expect that collaborating with humans, language models like ChatGPT would be very useful to law students taking exams and to practicing lawyers."

Choi believes ChatGPT could be a boon to his students and peers if used intentionally. Choi used it to write code for the paper's statistics and graphs, and said it worked "VERY well," noting it "produced simple code that would be tedious to write (e.g. bootstrapping p-values, creating graph grids)."

While some school districts have banned ChatGPT, some teachers have embraced it as an educational tool.

Rather than pretending ChatGPT doesn't exist, Choi focused on how law schools can adapt to a changed environment. To do that, professors may need to ditch exams that mainly test simple knowledge of the law.

"If lawyers start to use these tools to augment their abilities to be lawyers, then we need to appropriately change our teaching and our tests," Choi told Insider.

"When they invented calculators, they didn't force students to continue doing math by hand," Choi said. "They allowed them to use calculators and just changed the nature of the test. So I think that's the task for legal education."

OpenAI, the company responsible for ChatGPT, recently received an eye-watering "multi-billion dollar investment" from Microsoft. In a post, the company said the investment "will allow us to continue our independent research and develop AI that is increasingly safe, useful, and powerful."

Email tips on all things internet to mleighton@insider.com.