Columbia DirecT Lab

GCN Analysis

The first task that I was assigned when starting at the lab was to compare three newly published Graph Convolution Networks (GCNs) on a collected dataset of stock videos showing people using mobility aids. This problem had two main challenges, first I needed to find a way to process the raw videos I was given into a format that can be processed by a GCN and second I needed to adopt each GCN to work on our dataset.

To turn raw videos into a format suitable for GCN processing, I developed a preprocessing pipeline that included frame extraction, normalization, and data augmentation techniques. We first break each video into 10 second chunks and vectorize each frame into a feature vector using mediapipe which can extract all needed keypoints per frame. We then ensure that none of these vectors our empty (meaning no one is in frame) and assign a ground truth label to each.

After getting the preprocessed data, I trained and evaluated the GCNs I had to adopt each open source GCN to work with our data, this involved adding a custom graph definition for our datatype and adding in a 5-class data feeder. After adopting each GCN, I was able to train and evaluate each one on our dataset effectively and show which one works best for our domain.

Gait Analysis

After training the GCNs, I was then tasked with seeing if we can use gait parameters to predict mobility aid usage instead of the GCNs. The first step was to decide what gait features would be most relevant for this task. For this task I did a literature review, along with some experimental work to find the best features. With this I was able to identify 16 features that gave me the best accuracy.

After identifying the best features, I then trained a number of different machine learning models to predict mobility aid usage from these features. I found that a random forest classifier gave me the best accuracy and was able to achieve an accuracy of 92% on our validation set.

Aidsight

Finally, with these two projects finished I combined them to developed AidSight, a privacy-aware system for automatically labeling pedestrian mobility-aid use from naturalistic roadside video. This system leverages the best GCN I identified along with the lightweight gait analysis model that are wrapped in with a device triage and reinforcement learning policy that achieves 82% accuracy on roadside videos.

This system was formalized in a research paper submitted to the 106th Annual Meeting of the Transportation Research Board (TRB) and is currently under review; we are also planning on submitting it for publication to TRB Part C journal.