Accurate, Focused Research on Law, Technology and Knowledge Discovery Since 2002

Category Archives: Legal Research

Re-Evaluating GPT-4’s Bar Exam Performance

Martínez, Eric [Massachusetts Institute of Technology (MIT), Re-Evaluating GPT-4’s Bar Exam Performance (May 8, 2023). Available at SSRN: https://ssrn.com/abstract=4441311 or http://dx.doi.org/10.2139/ssrn.4441311

“Perhaps the most widely touted of GPT-4’s at-launch, zero-shot capabilities has been its reported 90th-percentile performance on the Uniform Bar Exam, with its reported 80-percentile-points boost over its predecessor, GPT-3.5, far exceeding that for any other exam. This paper investigates the methodological challenges in documenting and verifying the 90th-percentile claim, presenting four sets of findings that suggest that OpenAI’s estimates of GPT-4’s UBE percentile, though clearly an impressive leap over those of GPT-3.5, appear to be overinflated, particularly if taken as a “conservative” estimate representing “the lower range of percentiles,” and more so if meant to reflect the actual capabilities of a practicing lawyer. First, although GPT-4’s UBE score nears the 90th percentile when examining approximate conversions from February administrations of the Illinois Bar Exam, these estimates are heavily skewed towards repeat test-takers who failed the July administration and score significantly lower than the general test-taking population. Second, data from a recent July administration of the same exam suggests GPT-4’s overall UBE percentile was ~68th percentile, and ~48th percentile on essays. Third, examining official NCBE data and using several conservative statistical assumptions, GPT-4’s performance against first-time test takers is estimated to be ~63rd percentile, including ~41st percentile on essays. Fourth, when examining only those who passed the exam (i.e. licensed or license-pending attorneys), GPT-4’s performance is estimated to drop to ~48th percentile overall, and ~15th percentile on essays. Taken together, these findings carry timely insights for the desirability and feasibility of outsourcing legally relevant tasks to AI models, as well as for the importance for AI developers to implement rigorous and transparent capabilities evaluations to help secure safe and trustworthy AI.”

When do your employees need to disclose their use of ChatGPT?

HR Brew: “…As ChatGPT and other generative AI technologies provide a helping hand to employees, HR teams are grappling with policies regarding its use, including disclosure. Some companies have banned or restricted employees from the tech. Others are embracing the possibilities the tech can offer to employee productivity and see it as a tool to… Continue Reading

Why won’t Google give a straight answer on whether Bard was trained on Gmail data?

Skiff Blog: “… Google’s Smart Compose feature was trained on Gmail users’ private emails.Bard is not Google’s only language-focused machine learning model. Anyone who’s used Gmail in the past few years knows about the Smart Compose and Smart Reply features, which auto-complete sentences for you as you go.According to Google’s 2019 paper introducing Smart Compose, the feature was trained… Continue Reading

FTC Finds Amazon Ring Cameras Responsible for “Egregious Violations of Users’ Privacy,” Requires Data Deletion

EPIC: “In a proposed consent order released today, the Federal Trade Commission will require Amazon to “delete data products such as data, models, and algorithms derived from videos it unlawfully reviewed,” implement new privacy and security measures, and pay a fine of $5.8 million. The proposed order was published alongside a complaint finding that Amazon… Continue Reading

Congressional Research Service Syndication Feed

Disruptive Library Technology Jester; “One of the hidden gems of the Library of Congress is the Congressional Research Service (CRS). With a staff of about 600 researchers, analysts, and writers, the CRS provides “policy and legal analysis to committees and Members of both the House and Senate, regardless of party affiliation.” It is kind of… Continue Reading

Who Is Liable if AI Violates Your Human Rights?

Impakter: “Artificial intelligence (AI) has become part of our daily life and will become more important over the years. It can solve complex problems, reduce human errors and increase productivity. However, if legal frameworks and paths of accountability do not adapt and evolve alongside swift technological advancements, AI may encounter human rights issues. This is… Continue Reading

Poor writing, not specialized concepts, drives processing difficulty in legal language

Eric Martínez, Francis Mollica, Edward Gibson, Poor writing, not specialized concepts, drives processing difficulty in legal language, Cognition, Volume 224, 2022, 105070 [h/t Pete Weiss]. “Despite their ever-increasing presence in everyday life, contracts remain notoriously inaccessible to laypeople. Why? Here, a corpus analysis (n ≈10 million words) revealed that contracts contain startlingly high proportions of… Continue Reading

Sound Recordings of Supreme Court of the United States Now Fully Digitized

NARA: “The Moving Image and Sound Branch is pleased to announce that the sound recordings of RG 267: Records of the Supreme Court of the United States have been fully digitized and are available for listening and download through the National Archives Catalog. The audio recordings in Record Group 267 are organized into three series, … Continue Reading

Top Cyberattacks Revealed in New Threat Intelligence Report

Dark Reading: “New report provides actionable intelligence about attacks, threat actors, and campaigns. Ismael Valenzuela, Vice President of Threat Research & Intelligence, BlackBerry. We recognize that in today’s world, security leaders must expand their focus beyond technologies and their vulnerabilities. To effectively manage risk, security leaders must continually analyze the global threat landscape and understand… Continue Reading

Best Practices for Disclosure and Citation When Using Artificial Intelligence Tools

Shope, Mark, Best Practices for Disclosure and Citation When Using Artificial Intelligence Tools (January 26, 2023). Available at SSRN: https://ssrn.com/abstract=4338115 “This article is intended to be a best practices guide for disclosing the use of artificial intelligence tools in legal writing. The article focuses on using artificial intelligence tools that aid in drafting textual material,… Continue Reading