David Kaber, Ph.D., the Dean’s Leadership Professor and chair of the UF Department of Industrial & Systems Engineering (ISE), has received funding from the Defense Advanced Research Projects Agency (DARPA) in support of his research to use augmented reality (AR) to aid in operational tasks conducted in extreme environments.
The project will utilize multiple lightweight wearable devices, including a smartwatch and eye-tracking glasses, to measure operators’ heart rate variability, galvanic skin response, eye blink rate, and pupil size. This data will then be used to develop an algorithm that will determine cognitive arousal and workload during a target task performance. Based on the assessment of the operator’s physical and physiological state and stress levels, AR devices (headsets) will be employed to provide knowledge-task guidance in real-time to aid the operator with the task at hand.
“The primary challenge for UF ISE in this project is creating a data-to-knowledge ‘pipeline’ from wearable devices to an intelligent agency that determines what information or guidance to provide to a human operator, all in real-time,” Dr. Kaber said. “Getting raw data streams from the devices to Python is done but sanitizing and smoothing the data streams for feature extraction, which drives an operator state classifier, at the speed of milliseconds is difficult to do. Fortunately, we have some outstanding graduate students at UF, who are working on the project.”
Dr. Kaber, who is a Co-PI on the project, is working with a collaborative team that includes Jaime Ruiz, Ph.D., and Lisa Anthony, Ph.D., both associate professors in the UF Department of Computer & Information Science & Engineering (CISE), James Fairbanks, Ph.D., a CISE assistant professor, Maryam Zahabi, Ph.D., an assistant professor in the Department of Industrial and Systems Engineering at Texas A&M, and Evan Patterson, Ph.D., a research scientist at the Topos Institute. Yunmei Liu, a UF ISE Ph.D. candidate, will be assisting Dr. Kaber with ISE’s portion of the project.
“After we collect the subjects’ physiological signals through wearable devices, I will need to find features that can accurately reflect any changes in the operator’s cognitive workload. Then, I can determine an algorithm that is suitable for us to complete the work of predicting cognitive workload through physiological signals,” Liu said.
This new technology will have a broad impact in that it can be used by civilian operators as well as the armed forces as they engage in high-complexity tasks in adverse environments, such as military helicopter pilot checklist completion and Army field-medic care triage for soldier amputee cases.
“This type of technology has the potential to prevent faults and failures in high-consequence systems and to save lives, under certain circumstances,” Dr. Kaber said.