Date of Award

12-2025

Degree Name

MS in Computer Science

Department/Program

Computer Science

College

College of Engineering

Advisor

Ayaan Kazerouni

Advisor Department

Computer Science

Advisor College

College of Engineering

Abstract

Background and Context

Software testing is a fundamental component of computer science education, forming the basis for students’ ability to ensure program correctness and reliability. Despite its importance, many students struggle to design test cases that effectively expose faults and achieve meaningful test coverage. Traditional instructional approaches often emphasize code coverage metrics such as line or branch coverage, but these metrics may not adequately capture the quality of student tests. Mutation analysis, which measures how well tests detect small, artificial faults (mutants) introduced into the program, offers a potentially richer measure of test effectiveness. However, little is known about how feedback based on mutation coverage affects student learning compared to more conventional coverage-based feedback.

Objectives

This thesis investigates how different forms of automated feedback impact students’ underlying ability to construct effective test suites. Specifically, it examines whether exposure to mutation-based feedback or branch coverage feedback better improves students’ conceptual understanding and independent application of software testing principles. In other words, rather than asking which feedback produces stronger test suites during training, this study asks which feedback better helps students learn to become stronger testers.

Method

To investigate these questions, a controlled experiment was conducted with students in undergraduate software engineering courses. Participants were randomly assigned to one of two treatment groups: one receiving automated feedback based on branch coverage, and the other receiving feedback based on mutation coverage. The study employed a pre/post-test design to measure learning gains independent of feedback exposure. In the beginning, all students completed a set of baseline (pre-test) problems in which they wrote test suites without receiving any feedback. During the intervention phase, each group practiced test writing while receiving their respective feedback type. Afterward, students completed a new set of problems (post-test) again without feedback, allowing direct measurement of how much their testing ability improved as a result of the practice sessions. Data from all three phases (pre-test, post-test, practice) was analyzed to assess changes in the correctness and completeness of the test suites, as well as additional statistical characteristics such as the quantity and redundancy of the test cases.

Findings

The results showed that students who received mutation coverage feedback demonstrated a greater improvement in test-writing ability between the pre- and post-tests compared to those who received branch coverage feedback. Specifically, the mutation feedback group achieved a higher increase in fault detection and completeness, as well as a larger gain in mutation coverage from pre- to post-tests. During the practice phase, students receiving mutation feedback also wrote more tests on average than those receiving branch feedback, reflecting a more exploratory and fault-oriented testing approach. These outcomes suggest that exposure to mutation-based feedback encourages deeper reasoning about program behavior and more persistent refinement of test suites, leading to stronger transfer of testing skills beyond the feedback environment.

Implications

Although results remain preliminary, this work contributes to the growing body of research on assessment and feedback in computing education by examining how different adequacy metrics can shape students’ testing practices. The findings are expected to inform instructional design decisions regarding the integration of advanced testing metrics, such as mutation analysis, into computer science curricula. Ultimately, this research aims to provide actionable insights for educators seeking to improve how software testing concepts are taught and assessed in undergraduate computer science programs.

Download

Share

COinS