Perception describes how robots sense and process data to obtain an understanding of their environment. For robots to operate safely and effectively in unstructured environments, where they currently are not applied, the ability to perceive is a fundamental requirement. Perception represents a complex interaction between sensors and software, algorithms and data representations. Robots also need to have human-like scene understanding and situational awareness. True scene understanding can require an element of imagination, for example, being able to imagine consequences, and this represents a completely new frontier challenge for robots. The slow pace of progress in perception limits the usefulness and uptake of robotics in most sectors of the Australian economy, making it an obvious area to target [AAS18].

A robot’s perception must process and fuse sensor data at a sufficient rate for the task. For example, a robot moving around people must be able to detect and avoid collisions in a fraction of a second. Perception systems must also parse useful and relevant sensor data about a robot’s environment from irrelevant data. For a self-driving car to navigate it must perceive where it is in relation to the road, surroundings and other actors such as cars, pedestrians and cyclists. It may need to ignore or subtract distracting elements like snow or rain to enhance its understanding of its immediate environment. Creating a representation of a robot’s environment is a key part of perception. It is required so that the robot can reason about which action to take, and must interface with the robot’s action systems [AAS18].

Today, most large-scale applications of robots rely on very limited perception capabilities or no perception capability at all. Industrial robots are most effective when the operating environment is carefully designed. Tasks must be carefully limited so that the robot only needs to capture information directly from specific simple sensors and uncertainty and surprise is engineered out. The challenge is to go beyond these limitations and for the robot to operate in unstructured environments, fusing data from many sensors and adapting dynamically to changing circumstances. Key to a robot’s success is whether it can perceive its own place in the environment (where am I?), the likely effects on the environment from any actions the robot plans to take (what is here and how can I interact?) and the ability to measure the effect of its own actions (what have I done?) [AAS18].

Future research should tackle how sensor data is processed, how a robot’s environment is represented, deeper integration of perception with action and cognition including machine learning, especially deep learning (section 11.4), and biological mimicking (section 11.7). Over the next twenty years, it is reasonable to expect to see many work-arounds applied to address some of the limitations currently imposed by robot perception. For example, controlling the robot’s environment or making it well-understood through widespread continuous mapping, deployment of multiple sensors and improved robot-robot communications [AAS18].