Leveraging Artificial Intelligence for Objective and Individualized Skills Assessment

Schmieder, Jacob; Alvarado, Cristina; Green, Dawn; Pappada, Scott; Vargas, Karen M

This project aims to develop an asynchronous assessment platform to support healthcare training, addressing the growing need for consistent evaluation of nursing and physician assistant (PA) students. Using natural language processing (NLP ) and later, large language models (LLMs), the study focuses on assessing IV insertion skills using learners’ spoken words, with plans to expand to other procedures and computer vision to assess procedural accuracy. The platform provides both objective scores and individualized feedback, enabling students to improve without requiring faculty presence. This increases experiential learning opportunities and reduces faculty workload. It is hypothesized that an combined NLP and logic-based algorithm will detect 85% or more of IV insertion procedure events verbalized by learners. We further hypothesize that an under development LLM-based detection will improve upon detection accuracy.

Overview