Digital Assistance in Angiography: Assessing the Speed and Usability in Accessing Information of Standard Operating Procedures with a Large Language Model Enhanced Chatbot Versus Conventional Paper Printouts – a Randomized Pilot Study

Aim and Research Question(s)

The aim of this master thesis is to evaluate the efficiency and usability of a Large Language Model (LLM)-enhanced Chatbot compared to traditional paper printouts for accessing Standard Operating Procedures (SOPs) in angiography. Research Questions: How does the use of an LLM-enhanced Chatbot affect the speed of medical professionals in answering questions about SOPs compared to conventional paper printouts? How do users assess the usability of an LLM-enhanced Chatbot compared to traditional paper printouts?

Background

Healthcare professionals often face challenges in quickly accessing SOPs, which are crucial for guiding medical procedures and ensuring the availability of necessary materials. Traditionally, these SOPs are available as paper printouts, which can be time-consuming to navigate [1].

Methods

Study Design: Randomized pilot trial. Participants: Recruited from the Departement of angiography in Klinik Ottakring, randomly assigned to use either the digital Chatbot or paper printouts. Procedure: Participants answered 10 questions related to angiography SOPs using either the Chatbot or paper printouts. Metrics: Usability assessed through the System Usability Scale (SUS) and response times measured in seconds.

Results and Discussion

Response Time: The Chatbot achieved a marginally faster response rate with a mean of 57.6 ± 23.9 seconds compared to 59.2 ± 23.2 seconds for paper printouts. Usability Scores: The SUS scores averaged 87 for the chatbot, indicating excellent usability, compared to 67.5 for paper printouts. Feedback: Participants noted issues with language accuracy in the chatbot's responses, highlighting areas for improvement Discussion: The Chatbot demonstrated a slight improvement in response time and higher usability scores, suggesting a preference for digital information retrieval methods among users. However, accuracy issues in the Chatbots responses indicate the need for further refinement. By using newer LLMs and better infrastructure the response time could be further improved.

Conclusion

This study highlights the potential of LLM-enhanced Chatbots to improve the efficiency and usability of accessing SOPs in clinical settings. Future research should focus on enhancing the accuracy and reliability of Chatbot responses.

References

[1] C. Chen, T. Kan, S. Li, C. Qiu, and L. Gui, ‘Use and implementation of standard operating procedures and checklists in prehospital emergency medicine: a literature review’, Am J Emerg Med, vol. 34, no. 12, pp. 2432–2439, Dec. 2016, doi: 10.1016/j.ajem.2016.09.057.