Available at: https://digitalcommons.calpoly.edu/theses/3338
Date of Award
6-2026
Degree Name
MS in Computer Science
Department/Program
Computer Science
College
College of Engineering
Advisor
Franz Kurfess
Advisor Department
Computer Science
Advisor College
College of Engineering
Abstract
With the recent popularization of large language models (LLMs), natural language has become one of the most accessible and powerful ways for people to interact with creative tools. Although they have become common in mainstream domains like image and audio editing, there is currently no robust AI-based system that can reliably turn free-form language into edits for symbolic musical scores. This gap represents a missed opportunity to improve human workflows for creating and editing sheet music, but it is also a fundamental limitation for other agentic music systems; without a robust mechanism for translating free-form language into structured scores, AI agents developed for high-level composition and arrangement work are unable to reliably translate their ideas into output.
To address this gap, this thesis proposes ScoreSpeak, a comprehensive LLM-based agentic system that enables full natural language control of musical scores as MusicXML files. With over 80 score editing tools available to the agent, ScoreSpeak supports both the creation and precise editing of virtually every aspect of a score, including notation, instrumentation, articulations, dynamics, expressions, and advanced structural layouts. In order to evaluate our system, we synthetically generate and manually validate a benchmark of score-editing tasks across multiple well-known scores of varying complexity, pairing natural language prompts with executable reference edits to measure instruction following accuracy, latency, and cost. On our 752-case precise-edit benchmark, the best ScoreSpeak agent configuration achieved a 99.60% pass rate, compared with 75.40% for a Cursor CLI coding agent editing baseline. To support reproducibility and future research, we open-source the benchmark dataset, agent code, and our agent-based sheet music web editor. Together, these contributions establish natural language as a viable interface for editing symbolic musical scores—both for humans and for future agentic composition systems.