Biostatistics Dept Seminar: Does AI help humans make better decisions? A methodological framework for experimental evaluation
Biostatistics Department Seminar
Title: Does AI help humans make better decisions? A methodological framework for experimental evaluation
Abstract: The use of Artificial Intelligence (AI), or more generally data-driven algorithms, has become ubiquitous in today’s society. Yet, in many cases and especially when stakes are high, humans still make final decisions. The critical question, therefore, is whether AI helps humans make better decisions compared to a human-alone or AI-alone system. We introduce a new methodological framework to experimentally answer this question without additional assumptions. We measure a decision maker’s ability to make correct decisions using standard classification metrics based on the baseline potential outcome. We consider a single-blinded experimental design, in which the provision of AI-generated recommendations is randomized across cases with humans making final decisions. Under this experimental design, we show how to compare the performance of three alternative decision-making systems — human-alone, human-with-AI, and AI-alone. We also show when to provide a human-decision maker with AI recommendations and when they should follow such recommendations. We apply this methodology to the data from our own randomized controlled trial of a pretrial risk assessment instrument. We find that the risk assessment recommendations do not improve the classification accuracy of a judge’s decision to impose cash bail. Our analysis also shows that the risk assessment-alone decisions generally perform worse than human decisions with or without algorithmic assistance.
Preprint of the corresponding paper on arXiv.
Speakers
Eli Ben-Michael is an assistant professor in the Department of Statistics & Data Science and the Heinz College of Information Systems & Policy at Carnegie Mellon University.
Zoom Registration
If you would like to join via Zoom, please register here.
2024-2025 Monday Seminar Series
All seminars are held at 12:05 PM via Zoom and onsite. View all seminar information here.