Honors Thesis 2024 - Ellie Paek

Emora Assistant Bot: Revolutionizing Task Automation and Chatbot Evaluation

Ellie Paek

Highest Honor in Computer Science

Abstract

The Emora Assistant Bot, part of the Emora Chat: College Companion project, was developed by members of the Emory NLP Research Lab. Motivated by the call for efficiency in executing tasks, Emora excels in managing both general administrative tasks and those specific to classroom settings, all while facilitating seamless communication among multiple users. Leveraging the Emora State Transition Dialogue Manager framework and OpenAI's GPT-3.5 Turbo API, the Assistant Bot executes seventeen different tasks through natural language interaction, offering users a conversational and efficient experience. An innovative automated evaluation approach utilizing the GPT-3.5 language model is used to evaluate this task-oriented chatbot, providing valuable insights into Emora's performance and highlights areas for improvement.

Conducting automatic evaluations revealed limitations with the STDM framework, yet Emora demonstrated successes in information extraction and task categorization, underscoring her capability for seamless task execution. Moreover, despite occasional inconsistencies, the GPT simulation emerged as a promising method for evaluating task-oriented chatbots. Between two professor and 20 student profiles, Emora had an average success rate of 94.3% for task execution, and 94% for natural language understanding. The GPT simulation displayed an average success rate of about 81%.

Through this research, the Emora Assistant Bot project emerges as a pioneering solution for automating administrative tasks, showcasing the potential of large language models in both task execution and evaluation within the realm of chatbots.

Department / School

Computer Science / Emory University

Degree / Year

BS / Spring 2024

Committee

Jinho D. Choi, Computer Science and QTM, Emory University (Chair)
Bethany Mamola, Music, Emory University
Talea Mayo, Mathematics, Emory University

Links

Anthology | Paper | Presentation

Jinho Choi, Ellie Paek, Bethany Mamola (screen), <span style="caret-color: rgb(40, 50, 78);">Tales</span>&nbsp;Mayo