Paul Brenner and Charles Vardeman
University of Notre Dame
3:30 PM
Click for Zoom Link
Hybrid Multimodal Agentic Teams Bridging Simulations and the Real World
We will detail research on Multimodal Agentic Teaming for Mission Planning and Digital Twin applications stemming from foundational drivers in DoD strategy and the emergence of Large Language Models (LLMs). Objectives include building prototype agentic systems and developing specialized cognitive agents with multimodal LLMs, reasoning, and temporal/spatial awareness. Key milestones included initial AI agent scenario planning, evaluation of open-weight LLMs (Mixtral, LLAMA, Phi), and code base enhancements for specific use cases. The work explores integrating simulation platforms with customizable, trusted LLMs and showcases practical applications in real- world prototype robotics for Search and Rescue and IED response scenarios. The ultimate goal is enhancing autonomous mission completion through integrated digital twins and embodied LLMs.
Click to view Poster