Predicting Location and Training Effectiveness (PLATE)

Author

Erik Rolf BruennerFollow

Available at: https://digitalcommons.calpoly.edu/theses/2746

Date of Award

6-2023

Degree Name

MS in Computer Science

Department/Program

Computer Science

College

College of Engineering

Advisor

Alex Dekhtyar

Advisor Department

Computer Science

Advisor College

College of Engineering

Abstract

Abstract Predicting Location and Training Effectiveness (PLATE)
Erik Bruenner

Physical activity and exercise have been shown to have an enormous impact on many areas of human health and can reduce the risk of many chronic diseases. In order to better understand how exercise may affect the body, current kinesiology studies are designed to track human movements over large intervals of time. Procedures used in these studies provide a way for researchers to quantify an individual’s activity level over time, along with tracking various types of activities that individuals may engage in. Movement data of research subjects is often collected through various sensors, such as accelerometers. Data from these specialized sensors may be fed into a deep learning model which can accurately predict what movements a person is making based on aggregated sensor data. However, in order for prediction models to produce accurate classifications of activities, they must be ‘trained’. Training occurs through the process of supervised learning on large amounts of data where movements are already known. These training data sets are also known as ‘validation’ data or ‘ground truth’.

Currently, generation of these ground truth sets is very labor-intensive. To generate these labeled data sets, research assistants must analyze many hours of video footage with research subjects. These research assistants painstakingly categorize each video, second by second, with a description of the activity the subject was engaging in. Using only labeled video, the PLATE project facilitates the generation of ground truth data by developing an artificial intelligence (AI) that predicts video quality labels, along with labels that denote the physical location that these activities occurred in.

The PLATE project builds on previous work by a former graduate student, Roxanne Miller. Miller developed a classification system to categorize subject activities into groups such as ‘Stand’, ‘Sit’, ‘Walk’, ‘Run’, etc. The PLATE project focuses instead on development of AI to generate ground truth training in order to accurately detect and identify the quality of video data, and the location of the video data. In the context of the PLATE project, video quality refers to whether or not a test subject is visible in the frame. Location classifications include categorizing ‘indoors’, ‘outdoors’, and ‘traveling’. More specifically, indoor categories are further identified as ‘house’, ‘office’, ‘school’, ‘store’ or ‘commercial’ space. Outdoor locations are further classified as ‘commercial space’, ‘park/greenspace’, ‘residential’ or ‘neighborhood’.

The nature of our location classification problem lends itself particularly well to a hierarchical classification approach, where general indoor, outdoor, or travel categories are predicted, then separate models predict the subclassifications of these categories. The PLATE project uses three convolutional neural networks in its hierarchical location prediction pipeline, and one convolutional neural network to predict if video frames are high or low quality.

Results from the PLATE project demonstrate that quality can be predicted with an accuracy of 96%, general location with an accuracy of 75%, and specific locations with an accuracy of 31%. The findings and model produced by the PLATE project are utilized in the PathML project as part of a ground truth prediction software for activity monitoring studies.

PathML is a project funded by the NIH as part of a Small Business Research Initiative. Cal Poly partnered with Sentimetrix Inc, a data analytics/machine learning company, to build a methodology for automated labeling of human physical activity. The partnership aims to utilize this methodology to develop a software tool that performs automatic labeling and facilitates the subsequent human inspection. Phase I (proof of concept) of the project took place from September 2021 to August 2022, Phase II (final software production) is pending. This thesis is part of the research that took place during Phase I lifetime, and continues to support Phase II development.