Available at: https://digitalcommons.calpoly.edu/theses/3245
Date of Award
3-2026
Degree Name
MS in Computer Science
Department/Program
Computer Science
College
College of Engineering
Advisor
Ayaan Kazerouni
Advisor Department
Computer Science
Advisor College
College of Engineering
Abstract
Community chat platforms such as Discord and Slack support spontaneous, collaborative communication but make it difficult to retrieve previously discussed information. As conversations accumulate, valuable exchanges become buried, leading to repeated questions and sustained burden on experienced community members.
This work contributes a set of design requirements for question-answering systems operating over unstructured chat data, a Discord bot prototype implementing those requirements named Echo, and an empirical evaluation of how such a system affects user trust. Rather than encoding discrete question-answer pairs or generating synthetic responses with a language model, Echo indexes conversation topics for semantic retrieval and presents results as constructed exchanges of original messages attributed to their authors. This design preserves conversational context and grounds responses in source material.
A user study evaluated how interaction with Echo affected trust-related perceptions of automated question-answering systems. Ten participants performed information-seeking tasks within a synthetic Discord server and completed trust surveys before and after interaction. All participants exhibited increased composite trust scores following use of the system, with a mean increase of 8.4 points. Findings suggest high perceived value in semantic retrieval over keyword search and in exchange-based responses, as well as the importance of source attribution in building user trust.