TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs

Abstract

In recent years, artificial intelligence (AI) has made incredible progress. Advanced foundation models such as ChatGPT can offer powerful conversation, in-context learning, and code generation abilities for a broad range of open-domain tasks. They can also generate high-level solution outlines for domain-specific tasks based on their acquired common-sense knowledge. Nonetheless, they still face difficulties in specialized tasks because they lack sufficient domain-specific data during pretraining and can make errors in neural network computations requiring accurate execution. However, many existing models and systems can perform domain-specific tasks very well, although they are not easily accessible or compatible with foundation models because of the different implementations or working mechanisms. Therefore, there is a clear and pressing need for a mechanism that can leverage foundation models to propose task solution outlines and then automatically match the subtasks in the outlines to off-the-shelf models and systems with special functionalities to complete these subtasks. Inspired by this, we introduce TaskMatrix.AI as a new AI ecosystem that connects foundation models to millions of application programming interfaces (APIs) for task completion. Unlike most previous studies, which aimed to improve a single AI model, TaskMatrix.AI focuses on using an existing foundation model (as a brain-like central system) and APIs of other AI models and systems (as subtask solvers) to realize diversified tasks in both the digital and physical domains.