“Can Generative AI Pass the Legal Ethics Exam? Earlier this year, research found that GPT-4 could surpass law students in passing the Uniform Bar Exam. Our study builds on this discovery, testing whether generative AI models can also navigate the rules and fact patterns around legal ethics. The Research:

• We challenged OpenAI’s GPT-4 and GPT-3.5, Anthropic’s Claude 2, and Google’s PaLM 2 Bison to 100 simulated exams, composed of questions crafted by Professor Stevenson to model the Multistate Professional Responsibility Exam (MPRE).
• GPT-4 performed best, answering 74% of questions correctly, an estimated 6% better than the average human test-taker.
• GPT-4 and Claude 2 both scored above the approximate passing threshold for the MPRE, estimated to range between 56-64% depending on the jurisdiction.
• We note that model performance varied by subject area, and there are opportunities for improvement through domain-specific knowledge and lawyer-led validation.”