A Law School Course in Applied Legal Analytics and AI

Jaromir Savelka  
Carnegie Mellon University
Matthias Grabmair
Technical University of Munich


Technological advances in artificial intelligence (AI) are affecting the legal profession. Machine learning (ML) and natural language processing (NLP) enable new legal apps that, to some extent, can analyze contracts, answer legal questions, or predict the outcome of a case or issue. While it is hard to predict the extent to which these techniques will change law practice, two things are certain: legal professionals will need to understand the new text analysis techniques and how to use and evaluate them, and law faculties face the question of how to teach law students the required skills and knowledge to do so. At the University of Pittsburgh School of Law, the authors have co-designed a semester-long course entitled, Applied Legal Data Analytics and AI, and twice taught it to combined groups of law students and students from technical departments. The course provides a hands-on practical introduction to applying ML and NLP to extract information from legal text data, the ways text analytics have been applied to support the work of legal professionals, researchers, and administrators, and the techniques for evaluating how well they work.

The article introduces the new text analytic techniques and briefly surveys law schools’ current efforts to incorporate instruction on computer programming and machine learning in legal education. Then it describes the 2020 version of the course, including the students, instructors, and course sessions in overview. We explain how we taught law students skills of programming and experimental design and engaged them in assignments that involve using Python programming environments to analyze legal data.

The course culminated in joint projects engaging small teams of law and technical students in applying machine learning and data analytics to legal problems. The article explains how the instructors prepare the students for the final course projects, beginning early in the term with project ideas and databases of text, forming teams, working on the projects as a team and obtaining interim feedback, and finally completing the projects and reporting results. We draw some salient comparisons between the 2019 and 2020 versions of the course and report what worked well and what did not, the students’ reactions, and lessons learned for future offerings of the course.


  1. Aletras, N., Tsarapatsanis, D., Preoţiuc-Pietro, D. and Lampos, V. 2016. “Predicting judicial decisions of the European Court of Human Rights: A natural language processing perspective.” PEERJ Computer Science 2: e93. https://peerj.com/articles/cs-93/?utm_source=mandiner&utm_medium=link&utm_campaign=mandiner_201912 Accessed 27/11/2020
  2. Angwin, J., Larson, J., Mattu, S., and Kirchner, L. 2016. “Machine Bias,” ProPublica, 23 May. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing Accessed 7/8/2019
  3. Ashley, K. 2017. Artificial Intelligence and Legal Analytics. New Tools for Law Practice in the Digital Age. Cambridge, UK: Cambridge University Press.
  4. Ashley, K. 2019. “Automatically Extracting Meaning from Legal Texts: Opportunities and Challenges.” Ga. St. U. L. Rev. 35: 1117-1151.
  5. Ashley, K. and Walker, V. 2013. “From Information Retrieval (IR) to Argument Retrieval (AR) for Legal Cases: Report on a Baseline Study.” In K. Ashley (ed.), 26th Int’l Conf. on Legal Knowledge and Information Systems. Jurix-2013. Amsterdam: IOS Press pp. 29-38.
  6. Bennett, Z., Russell-Rose, T., and Farmer, K. 2017. “A scalable approach to legal question answering.”, In Proceedings ICAIL-17. New York: ACM, pp. 269-270.
  7. Berman, D. and Hafner, C. 1986. “Obstacles to the Development of Logic-Based Models of Legal Reasoning.” In C. Walter (ed.) Computer Power and Legal Language. Santa Barbara: Praeger. pp. 183-214.
  8. Bhattacharya, P., Paul, S., Ghosh, K., Ghosh, S., and Wyner, A. 2019. “Identification of Rhetorical Roles of Sentences in Indian Legal Judgments.” In M. Araszkiewicz and V. Rodríguez-Doncel (ed.), 32d Int’l Conf. on Legal Knowledge and Information Systems, Jurix-19. Amsterdam: IOS Press. pp. 3-12.
  9. Bishop, C. 2006. Pattern Recognition and Machine Learning. New York: Springer.
  10. Branting, K. 2017. “Data-centric and logic-based models for automated legal problem solving.” Artificial Intelligence and Law, 25 (1): 5-27.
  11. Brostoff, T. and Sinsheimer, A. 2013. United States Legal Language and Culture: An Introduction to the US Common Law System. Oxford, UK: Oxford University Press.
  12. Cardellino, C., Teruel, M., Alemany, L., and Villata, S. 2017. “A low-cost, high-coverage legal named entity recognizer, classifier and linker.” In Proceedings ICAIL-17. New York: ACM, pp. 9–18.
  13. Chalkidis, I., Androutsopoulos, I., and Aletras, N. 2019. Neural Legal Judgement Prediction in English, Athens University of Economics and Business, https://arxiv.org/pdf/1906.02059 Accessed 26/11/2020
  14. Conrad, J., and Al-Kofahi, K. Scenario analytics: Analyzing jury verdicts to evaluate legal case outcomes. 2017 In Proceedings ICAIL-17. New York: ACM, pp. 29-37.
  15. Contreras, A. and McGrath, J. 2020. “Law, Technology, and Pedagogy: Teaching Coding to Build a “Future-Proof” Lawyer.” Minn. J.L. Sci. & Tech. 21: 2 297-332.
  16. Corbett-Davies, S., Pierson, E., Feller, A., and Goel, S. 2016. “A computer program used for bail and sentencing decisions was labeled biased against blacks. It’s actually not that clear.” The Washington Post, Monkey Cage. Oct. 17. https://www.washingtonpost.com/news/monkey-cage/wp/2016/10/17/can-an-algorithm-be-racist-our-analysis-is-more-cautious-than-propublicas/?noredirect=on Accessed 7/7/2020.
  17. Council, J. 2019. “Top Law Schools Add AI Courses.” WSJ PRO Artificial Intelligence. https://www.wsj.com/articles/top-law-schools-add-ai-courses-11555925401 Accessed 3/10/2019
  18. Crichton, D. 2015. “With Judge Analytics, Ravel Law Starts to Judge the Judges.” TechCrunch. April 16. https://techcrunch.com/2015/04/16/who-judges-the-judges/ Accessed 26/11/2020
  19. Dalton, B. “Cognifying Legal Education.” 2019. Above the Law: Law 2020. https://abovethelaw.com/law2020/cognifying-legal-education/ Accessed 2/10/2019
  20. Devlin, J., Chang, M., Lee, K., and Toutanova, K. 2018. “Bert: Pre-training of deep bidirectional transformers for language understanding.” ARXIV Preprint arXiv:1810.04805.
  21. Domingos, P. 2012. “A few useful things to know about machine learning.” Communications of the ACM 55: (10) 78-87.
  22. Eicks, J. 2012. “Educating Superior Legal Professionals: Successful Modern Curricula Join Law and Technology.” In O. Goodenough and M. Lauritsen (eds.), Educating the Digital Lawyer 12-1: 5-1 – 5-14, https://www.academia.edu/9202158/Educating_the_Digital_Lawyer Accessed 8/1/2020
  23. Federal Automated Vehicles Policy. 2016. Accelerating the Next Revolution in Roadway Safety, NHTSA, US Dept. Transportation. (https://www.transportation.gov/AV/federal-automated-vehicles-policy-september-2016) Accessed 26/11/2020, pp. 5-14, 17-19.
  24. Fenwick, M., Kaal, W., and Vermeulen, E. 2018 “Legal Education in a Digital Age: Why 'Coding for Lawyers' Matters.” Lex Research Topics in Corporate Law & Economics Working Paper No, 2018-4, U of St. Thomas (Minnesota) Legal Studies Research Paper No. 18-21. Pp. 0-31 SSRN: https://ssrn.com/abstract=3227967 or http://dx.doi.org/10.2139/ssrn.3227967 Accessed 26/11/2020
  25. Ferrucci, D., Brown, E., Chu-Carroll, J., Fan, J., Gondek, D., Kalyanpur, A., Lally, A., Murdock, J., Nyberg, E., Prager, J., Schlaefer, N. and Welty, C. 2010. “Building Watson: An Overview of the DeepQA Project.” AI Magazine, Fall, 31: 59-79.
  26. Grabmair, M., Ashley, K., Chen, R., Sureshkumar, P., Wang, C., Nyberg, E., and Walker, V. 2015. “Introducing LUIMA: an experiment in legal conceptual retrieval of vaccine injury decisions using a UIMA type system and tools.” In Proceedings ICAIL-15. New York: ACM. pp. 69-78.
  27. Gretok, E., Langerman, D. and Oliver, W. 2020. “Transformers for Classifying Fourth Amendment Elements and Factors Tests.” 33d Int’l Conf. on Legal Knowledge and Information Systems, Jurix-2020. Amsterdam: IOS Press pp. 63-72.
  28. Halevy, A., Norvig, P., and Pereira, F. 2009. “The unreasonable effectiveness of data.” IEEE Intelligent Systems, 24 (2): 8-12.
  29. Halterman. R. 2018 Fundamentals of Python Programming. Southern Adventist University (2018). https://archive.org/details/2018Fundamentals.ofPython. Accessed 27/11/2020.
  30. Haselager, P. 2019. “Mediated action and the risk of entrapment”, invited speech at ICAIL-2019, Montreal, June 18.
  31. Hudgins, V. 2020. “Casetext Launches New Brief-Writing Automation Platform Compose.” LegalTech News. Feb. 25. https://www.law.com/legaltechnews/2020/02/25/casetext-launches-new-brief-writing-automation-platform-compose/ Accessed 26/11/2020
  32. Katz, D., Bommarito, I., and Blackman, J. 2017. “A general approach predicting the behavior of the Supreme Court of the United States.” PLOS One. April 12. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0174698 Accessed 27/11/2020
  33. Kohavi, R., and Provost, F. 1998. “Glossary of Terms.” Machine Learning, 30 (2-3): 271–274.
  34. Larson, J., Mattu, S., Kirchner, L., and Angwin, J. 2016. “How We Analyzed the COMPAS Recidivism Algorithm.” ProPublica May 23. https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm Accessed 7/7/2020.
  35. Lauderdale, B., and Clark, T. 2012. “The Supreme Court's many median justices.” American Political Science Review, 106 (4): 847-866.
  36. Linna, Jr., D. 2018. “Training Lawyers to Assess Artificial Intelligence and Computational Technologies.” LegalTech Lever 1 https://www.legaltechlever.com/2018/09/training-lawyers-assess-artificial-intelligence-computational-technologies/ Accessed 3/10/2019.
  37. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. 2019. “Roberta: A robustly optimized BERT pretraining approach.” ARXIV Preprint arXiv:1907.11692.
  38. Medvedeva, M., Vols, M., and Wieling, M. 2020. “Using machine learning to predict decisions of the European Court of Human Rights.” Artificial Intelligence and Law, 28(2): 237-266.
  39. Mikolov, T., Sutskever, I., Chen, K., Corrado, G., and Dean, J. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, 3111-3119.
  40. Miller, S. 2019. “Artificial Intelligence and Law.” Colorado Law, https://www.colorado.edu/law/2019/05/03/artificial-intelligence-and-law Accessed 3/10/2019
  41. Mochales, R. and Moens, M. 2011. “Argumentation mining.” Artificial Intelligence and Law, 19(1), 1–22.
  42. Murphy, M. and Pearce, R. “Algorithms, Ethics, and Legal Services: How Artificial Intelligence Will Disrupt Legal Ethics and Professional Responsibility.” Unpublished manuscript on file with author.
  43. Nooteboom, L. 2017. “Child-Friendly Autonomous Vehicles; Designing Autonomy with all road users in mind.” In Developing Human Interactions with Autonomous Systems. Nov. 13. https://medium.com/@HumanisingAutonomy/child-friendly-autonomous-vehicles-2880ca74165f Accessed 29/3/2020.
  44. Norvig, P. 2007. “How to Write a Spelling Corrector.” https://norvig.com/spell-correct.html Accessed 7/7/2020.
  45. O’Connor, N. 2018. “Reforming the U.S. Approach to Data Protection and Privacy.” Council of Foreign Relations, Digital and Cyberspace Policy Program. https://www.cfr.org/report/reforming-us-approach-data-protection Accessed 26/11/2020
  46. O’Grady, J. 2018. “Suffolk Law School: Leading Transformation of Legal Education.” Practice Innovations 14. http://static.legalsolutions.thomsonreuters.com/static/images/newsletters/pracinno/Mar18_PracticeInnovations.pdf Accessed 3/10/ 2019
  47. Pavlus, J. 2019. “Machines Beat Humans on a Reading Test. But Do They Understand?” Quanta Magazine. Oct. 17. https://www.quantamagazine.org/machines-beat-humans-on-a-reading-test-but-do-they-understand-20191017/ Accessed 7/7/2020.
  48. Pennington, J., Socher, R., and Manning, C. 2014. “Glove: Global vectors for word representation.” In Proceedings of the 2014 Conf. on Empirical Methods in Natural Language Processing. EMNLP. pp. 1532-1543.
  49. Perlman, A. 2017. “Reflections on the Future of Legal Services.” Suffolk University Law School Research Paper No. 17-10. https://ssrn.com/abstract=2965592 Accessed 3/10/2019. pp. 1-11.
  50. Pivovarov, V. 2019. “Future Law School. What Does It Look Like?” Forbes 5 https://www.forbes.com/sites/valentinpivovarov/2019/02/12/futurelawschool/#67a0dc2f6a84 Accessed 3/10/2019
  51. Reed, C., Kennedy, E., and Silva, S. 2016. “Responsibility, Autonomy and Accountability: Legal Liability for Machine Learning.” Queen Mary School of Law Legal Studies Research Paper No. 243/2016: 1-17, 26-31. October 17. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2853462 Accessed 26/11/2020
  52. Reid, M. 2018. “A Call to Arms: Why and How Lawyers and Law Schools Should Embrace Artificial Intelligence.” U. Tol. L. Rev. 50: 477-489.
  53. Saravanan, M. and Ravindran, B. 2010. “Identification of rhetorical roles for segmentation and summarization of a legal judgment.” Artificial Intelligence and Law 18, 1: 45–76.
  54. Savelka, J. 2019. Statutory_Interpretation, https://github.com/jsavelka/statutory_interpretation Accessed 7/7/2020
  55. Savelka, J. and Ashley, K. 2018. “Segmenting US Court Decisions into Functional and Issue Specific Parts.” In Proceedings of the 31st Int’l Conf. on Legal Knowledge and Information Systems. Jurix-2018. Amsterdam: IOS Press pp. 111-120.
  56. Savelka, J., Walker, V., Grabmair, M., and Ashley, K. 2017. “Sentence Boundary Detection in Adjudicatory Decisions in the United States.” Traitement Automatique des Langues. 58: 21-45.
  57. Savkar, V. 2019. “How Will Artificial Intelligence Change Law Schools? How law schools can evolve using artificial intelligence and machine learning.” Above the Law. https://abovethelaw.com/legal-innovation-center/2019/06/20/how-will-artificial-intelligence-change-law-schools/ Accessed 3/10/2019
  58. Sergot, M., Sadri, F., Kowalski, R., Kriwaczek, F., Hammond, P. and Cory, H., 1986. “The British Nationality Act as a logic program.” Communications of the ACM, 29(5), pp. 370-386.
  59. Shulayeva, O., Siddharthan, A., and Wyner, A. 2017. “Recognizing cited facts and principles in legal judgements.” Artificial Intelligence and Law 25 1: 107–126.
  60. Simon, M. Lindsay, A., Sosa, L., and Comparato, P. 2018. “Lola v. Skadden and the Automation of the Legal Profession.” Yale J.L. & Tech. 20: 234-310.
  61. Surdeanu, M., Nallapati, R., Gregory, G. Walker, J. and Manning, C. 2011. “Risk Analysis for Intellectual Property Litigation.” In Proceedings ICAIL-11. p. 116-120.
  62. Surden, H. 2014. “Machine Learning and Law”, Wash. L. Rev. 89: 87-116.
  63. Walker, V. 2007. “A default-logic paradigm for legal fact-finding.” Jurimetrics 47: 193-244.
  64. Zentgraf, D. 2015. “What Every Programmer Absolutely, Positively Needs to Know About Encodings and Character Sets to Work with Text.” Kunstube, https://kunststube.net/encoding/ Accessed 7/7/2020
  65. Zhang, P. and Koppaka, L. 2007. “Semantics-based legal citation network.” In Proceedings ICAIL-07). New York: ACM. pp. 123–130.
How to Cite
Savelka J, Grabmair M, Ashley K. A Law School Course in Applied Legal Analytics and AI. LiC [Internet]. 2021Jan.14 [cited 2021Mar.1];37(1):134-7. Available from: https://journals.latrobe.edu.au/index.php/law-in-context/article/view/125

Send mail to Author

Send Cancel