Funding: NASA Human Research Program (HRP), Grant #80NSSC19K0656
Period of Performance: 2019 – 2025
PI: Daniel Selva, Texas A&M University
Overview
Future space exploration missions beyond low Earth orbit will require crews to execute complex operations and respond to off-nominal events without real-time support from Mission Control. The Human Factors and Behavioral Performance Element (HFBP) of NASA’s Human Research Program has emphasized the need to mitigate risks of adverse outcomes due to inadequate human systems integration architecture (HSIA). This project, part of NASA’s Human Capabilities Assessment for Autonomous Missions (HCAAM) research project, addressed these challenges by developing and evaluating Daphne-AT, a virtual assistant (VA) tailored to spacecraft anomaly treatment. The project measured the impact of AI virtual assistants on astronaut performance, cognitive workload (CW), situational awareness (SA), and trust, with the goal of informing standards and guidelines for similar agents on future missions.
The Daphne-AT Virtual Assistant
Daphne-AT follows a micro-services architecture consisting of a web-based front-end interface, a front-end server (the “Daphne brain”) that accepts user requests in natural language and routes them to the appropriate skills, a set of skills (anomaly Detection, Diagnosis, and Resolution), and a set of back-end services performing statistical and logical reasoning. Data sources include a real-time spacecraft telemetry feed from NASA JSC’s Habitat System Simulator (HSS), an expert knowledge graph covering habitat subsystems, anomalies, symptoms, and resolution procedures, and a historical database.
Two versions of the VA were developed and evaluated:
- Baseline VA: Included question answering, anomaly detection, diagnosis, and procedure-guided resolution. Used a template-based restricted-domain NLP system with a small neural network for question classification—fast and reliable for known question types, but limited in flexibility.
- Enhanced VA: Added confidence scores and natural language explanations to the diagnosis output. The chat agent was substantially upgraded to leverage GPT-4 and Retrieval Augmented Generation (RAG) for more flexible, scalable question answering.
Research Aims and Experiments
The project pursued three specific aims through four experiments—two laboratory studies and two analog deployments in NASA’s Human Exploration Research Analog (HERA) facility.
Lab Experiment 1 — Baseline VA
A counterbalanced within-subjects design where participants resolved representative anomaly scenarios with and without the baseline VA. Results showed significant positive effects of the VA on human performance, workload, and situational awareness. Published in the Journal of Aerospace Information Systems.
Lab Experiment 2 — Enhanced VA with Explanations
Participants were divided into three groups by VA accuracy level (high, medium, low) and resolved anomaly sets with and without explanations, under both high- and low-uncertainty conditions. Explanations had a significant positive effect on performance, situational awareness, and trust without increasing workload. Results published in two papers in the Journal of Cognitive Engineering and Decision Making—one on the overall effect of explanations and one focused on uncertainty.
HERA Campaign 6 — Baseline VA in Analog
The baseline VA was deployed in HERA Campaign 6 under a similar experimental design to Lab Experiment 1. No significant effects of the VA were found on any metrics; crewmembers found the anomaly scenarios easy to solve with the standard emergency chart. Overall performance was higher and workload lower than in the lab, suggesting the analog context reduced the difficulty of the task.
HERA Campaign 7 — Enhanced VA in Analog
The enhanced VA was deployed in HERA Campaign 7. Again, no statistically significant effects of the VA were found. Compared to C6, higher situational understanding was observed in C7 (consistent with Lab Experiment 2), but also higher attentional and mental demand—a cost not seen in the lab environment.
Key Findings
Across all four studies, results demonstrate potential for VA-based support in anomaly resolution for long-duration exploration missions, but also highlight that gains are strongly context-dependent. For straightforward, known anomalies, a simple emergency chart is a sufficient aid. VAs are most valuable for diagnosing more complex, novel anomalies where the knowledge graph and reasoning capabilities provide a real advantage over static reference materials. The project also produced recommendations for standards and guidelines for the development of virtual assistants for human spaceflight.
Team
PI: Daniel Selva (Texas A&M University Aerospace Engineering)
Co-Investigators: Ana Diaz-Artiles (AERO), Bonnie J. Dunbar (AERO), Raymond K. Wong (STAT)
Graduate Students: Prachi Dutta, Poonam Josan, Antoni Viros-i-Martin, Renee Abbott, Kazuki Toma, Joshua Elston, Logan Kluis (all AERO), Mahima Ganni (CSEN)
Publications
- Dutta P, Josan PK, Wong RKW, Dunbar BJ, Diaz-Artiles A, Selva D. Effects of Explanations and Accuracy on Human Performance and Trust in AI-Assisted Anomaly Diagnosis Tasks. Journal of Cognitive Engineering and Decision Making, 19(4), 453–473, 2025. https://doi.org/10.1177/15553434251338433
- Dutta P, Josan PK, Wong RKW, Dunbar BJ, Diaz-Artiles A, Selva D. Are Explanations Helpful Under Uncertainty? Effects of Uncertainty in AI-Assisted Spacecraft Anomaly Diagnosis. Journal of Cognitive Engineering and Decision Making, 2025.
- Josan PK, Dutta P, Abbott R, Viros-i-Martin A, Dunbar BJ, Wong RKW, Selva D, Diaz-Artiles A. Virtual Assistant for Spacecraft Anomaly Resolution: Effects on Human Performance Metrics. Journal of Aerospace Information Systems, 22(4), 264–274, 2025. https://doi.org/10.2514/1.I011449
- Dutta P, Josan PK, Wong RKW, Dunbar BJ, Diaz-Artiles A, Selva D. Effect of Explanations in AI-Assisted Anomaly Treatment for Human Spaceflight Mission. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 66, No. 1, pp. 697–701, 2022.
- Woodruff R, Beebe N, Josan PK, Wong RKW, Dunbar BJ, Selva D, Diaz-Artiles A. 3D Interactive Model of HERA to Support ECLSS Anomaly Resolution Using a Virtual Assistant. 2021 IEEE Aerospace Conference. https://doi.org/10.1109/AERO50100.2021.9438341
- Josan PK, Beebe N, Kluis L, York K, Viros A, Woodruff R, Dunbar BJ, Wong RKW, Selva D, Diaz-Artiles A. Experimental Design & Pilot Testing for ECLSS Anomaly Resolution Using Daphne-AT Virtual Assistant. 2021 IEEE Aerospace Conference. https://doi.org/10.1109/AERO50100.2021.9438497