Electrical and Computer Engineering ETDs

Publication Date

Fall 9-9-2021

Abstract

Human activity recognition in videos is a challenging problem that has drawn a lot of interest, particularly when the goal requires the analysis of a large video database. The Advancing Out-of-School Learning in Mathematics and Engineering (AOLME) project provides a collaborative learning environment for middle school students to explore mathematics, computer science, and engineering by processing digital images and videos. As part of this project, around 2200 hours of video data were collected for analysis. This data was collected to understand how children learn in situations involving mathematical and programming challenges so as to recognize best teaching practices that support broadening the participation of underrepresented students in STEM fields. Because of the size of the dataset, it is hard to analyze all the videos of the dataset manually. Thus, there is a huge need for reliable computer-based methods that can detect activities of interest.

My thesis is focused on the development of accurate methods for detecting and tracking objects in collaborative learning environments in long videos (>1 hour). Long-term object detection and tracking face fundamental challenges due to occlusion, illumination variations, and pose variations.

For collaborative learning groups, the thesis contributes robust methods for computer keyboard detection, tracking, and student hand detection. For hand detection, the thesis integrates object detection with clustering and time-projections for accurate, long-term assessment of student participation. The hand detection method was integrated into a writing detection system and can also be used for later research on recognizing student gestures.

All the models are validated on videos from 7 different sessions, ranging from 45 minutes to 90 minutes. The keyboard detector achieved a very high average precision (AP) of 92% at 0.5 intersection over union (IoU). Furthermore, a combined system of the detector with a fast tracker KCF (159fps) was developed so that the algorithm runs significantly faster without sacrificing accuracy. For a video of 23 minutes having resolution 858 x 480 @ 30 fps, the detection alone runs at 4.7x the real-time, and the combined algorithm runs at 21x the real-time for an average IoU of 0.84 and 0.82, respectively. The hand detector achieved average precision (AP) of 72% at 0.5 intersection over union (IoU). The detection results were improved to 81% using optimal data augmentation parameters. The hand detector runs at 4.7x the real-time with AP of 81% at 0.5 intersection over union. The hand detection method was integrated with projections and clustering for accurate proposal generation. This approach reduced the number of false-positive hand detections by 80% by improving IoU ratios from 0.2 to 0.5. The overall hand detection system runs at 4x the real-time, capturing all the activity regions of the current collaborative group.

Keywords

Object Detection, Data Augmentation, Video Analysis, Projections, ivpcl

Sponsors

This material is based upon work supported by the National Science Foundation under Grant No. 1613637 and No. 1842220.

Document Type

Thesis

Language

English

Degree Name

Computer Engineering

Level of Degree

Masters

Department Name

Electrical and Computer Engineering

First Committee Member (Chair)

Marios S. Pattichis

Second Committee Member

Balasubramaniam Santhanam

Third Committee Member

Ramiro Jordan

Fourth Committee Member

Sylvia Celedon-Pattichis

Download

Share

COinS