Methods of AI for Multimodal Sensing and Action for Complex Situations

Artificial intelligence (AI) seeks to emulate human reasoning, but is still far from achieving such results for actionable sensing in complex situations. Instead of emulating human situation understanding, machines can amplify intelligence by accessing large amounts of data, filtering unimportant information, computing relevant context, and prioritizing results (for example, answers to human queries) to provide human‐machine shared context. Intelligence support can come from many contextual sources that augment data reasoning through physical, environmental, and social knowledge. We propose a decisions‐to‐data multimodal sensor and action through contextual agents (human or machine) that seek, combine, and make sense of relevant data. Decisions‐to‐data combines AI computational capabilities with human reasoning to manage data collections, perform data fusion, and assess complex situations (that is, context reasoning). Five areas of AI developments for context‐based AI that cover decisions‐to‐data include: (1) situation modeling (data at rest), (2) measurement control (data in motion), (3) statistical algorithms (data in collect), (4) software computing (data in transit), and (5) human‐machine AI (data in use). A decisions‐to‐data example is presented of a command‐guided swarm requiring contextual data analysis, systems‐level design, and user interaction for effective and efficient multimodal sensing and action.