Jason Reinecke, Co-Authors Ask 'Can AI Hold Office Hours?'

Jason Reinecke, with co-authors Lisa Larrimore Ouellette, Amy R. Motomura and Jonathan S. Masur, recently published "Can AI Hold Office Hours?" Rapid improvements in AI tools offer transformative opportunities in legal education, including the possibility of students using AI tools to answer questions they might otherwise ask during office hours. But a critical challenge is the accuracy of these tools' responses, write the authors. "Both general-purpose and law-specific AI models have been shown to 'hallucinate' incorrect responses to a range of legal questions," they continue. In this paper, the authors evaluate current capabilities of AI models when given the more constrained task of answering questions about a specific legal text. "We found that a substantial number of responses were unacceptable in the sense of being harmful for learning, and many more responses failed to fully answer the question or had minor errors," they wrote. The paper is forthcoming in the Journal of Legal Education.