DARPA Mind’s Eye Program

: Image via Wikipedia

The Mind’s Eye program seeks to develop in machines a capability that currently exists only in animals: visual intelligence. Humans in particular perform a wide range of visual tasks with ease, which no current artificial intelligence can do in a robust way. Humans have inherently strong spatial judgment and are able to learn new spatiotemporal concepts directly from the visual experience. Humans can visualize scenes and objects, as well as the actions involving those objects. Humans possess a powerful ability to manipulate those imagined scenes mentally to solve problems. A machine‐based implementation of such abilities would be broadly applicable to a wide range of applications.

This program pursues the capability to learn generally applicable and generative representations of action between objects in a scene directly from visual inputs, and then reason over those learned representations. A key distinction between this research and the state of the art in machine vision is that the latter has made continual progress in recognizing a wide range of objects and their properties—what might be thought of as the nouns in the description of a scene. The focus of Mind’s Eye is to add the perceptual and cognitive underpinnings for recognizing and reasoning about the verbs in those scenes, enabling a more complete narrative of action in the visual experience.

One of the desired military capabilities resulting from this new form of visual intelligence is a smart camera, with sufficient visual intelligence that it can report on activity in an area of observation. A camera with this kind of visual intelligence could be employed as a payload on a broad range of persistent stare surveillance platforms, from fixed surveillance systems, which would conceivably benefit from abundant computing power, to camera‐equipped perch‐and‐stare micro air vehicles, which would impose extreme limitations on payload size and available computing power. For the purpose of this research, employment of this capability on man‐portable unmanned ground vehicles (UGVs) is assumed. This provides a reasonable yet challenging set of development constraints, along with the potential to transition the technology to an objective ground force capability.

Mind’s Eye strongly emphasizes fundamental research. It is expected that technology development teams will draw equally from the state of the art in cognitive systems, machine vision, and related fields to develop this new visual intelligence. To guide this transformative research toward operational benefits, the program will also feature flexible and opportunistic systems integration. This integration will leverage proven visual intelligence software to develop prototype smart cameras. Integrators will contribute an economical level of effort during the technology development phase, supporting participation in phase I program events (PI meetings, demonstrations, and evaluations) as well as development of detailed systems integration concepts that will be considered by DARPA at appropriate times for increased effort in phase II systems integration.