What do we mean by machine perception & interaction?

Machine perception and interaction refers to the communication between the environment and a machine, through which the machine takes in data, and presents information to users. A communication channel to take in the data, and present the data to the outer world. This communication channel can be broken down into two parts: sensing and interaction. Sensing refers to the machine's ability to collect information about users, and its environment, while interaction refers to the machine's ability to present information to users.

The computing industry has been working to take both domains forward for the past 70 years. The technologies we have today have the potential to enable ambient systems and HMI, but not in the current form. We need to rethink the way these technologies are used and embedded in our systems. We need to look at these technologies with fresh eyes and explore new ways of using them.

We will go deep into the perception part in this note. For interaction, please read this note.

Sensing → Perception

In the theory of situational awareness, sensing, is seen as the first step of a wider perception process. It is aimed at perceiving the status, attributes, and dynamics of relevant elements in the environment. It involves the processes of monitoring, cue detection, and simple recognition, which lead to an awareness of multiple situational elements (objects, events, people, systems, environmental factors) and their current states (locations, conditions, modes, actions).

Further, we can define perception as the process of deriving meaning and significance from raw sensory input. This process relies on the initial step of sensing, or collecting data from the environment. But perception goes beyond mere data collection, as it involves analyzing and interpreting that data to create a meaningful representation of the world. In order for this process to take place, a system must have advanced cognitive capabilities.

💡 It's an open question for all of us whether we'll ever be able to create machines that perceive the environment in the same way humans do.

I believe ambient systems give us all the new ways to collect the data from environment, whereas the HMI paradigm challenges us to think about innovative ways to utilize the information in enabling (add something from quote).

One way to study sensing in-depth is to divide it into two segments: 1) user input, and 2) environment input. I’m totally aware that the segmentation can also be done in a ample of different ways. Let’s deep dive into them -

1. User inputs

User input is the process by which a user provides data or instructions to a machine. The history of user input goes back to the earliest days of technical systems, when our mechanical systems used to have levers, and gears. In digital systems, the segment started with keyboard focussed command lines. Over time, user input segments have become increasingly sophisticated and diverse, making them more fluid.

As we move forward, it's important to focus on reducing bottlenecks and enabling more natural and fluid methods of interaction. Ideally, we want machines to understand and interpret the user's input, rather than forcing the user to conform to the machine's requirements. We're already seeing this with the rise of gesture-based and voice-based input methods, which are much more intuitive and efficient than older methods like keyboards and mice. These new technologies are helping us move toward a future where machines can truly understand and respond to human input in a natural way.

With the advancement of AR and VR technologies, we're seeing a more fluid and immersive form of human-machine interaction. Our current state of the art methods are advanced enough to enable an immersive and fluid experience. The next step is to make it as natural and intuitive as possible, so that humans feel as if they're interacting with another person rather than a machine. In doing so, it's important to prioritize privacy and security,

The release of Apple's "Vision Pro" technology signified a shift in how the tech industry viewed human-computer interaction. It sparked a focus on more natural and intuitive forms of interaction, such as using eye movements and hand gestures. This shift marked a significant departure from traditional input methods like touchscreens and keyboards.

2. Environment inputs

While user inputs are used to directly control a system, environmental inputs are used to provide context for the system's actions. This context allows the device to perform tasks more effectively and serve the user more intelligently.

For true perception, it's crucial to consider not only user input, but also environmental input. This is because future technology will not only act on user input, but also respond to environmental cues. These inputs assist us primarily in two ways:

1. Ability to take actions based on environmental data cues
2. Generate enough context of the environment so that the system can work better for the user.

Technically speaking, we (humans) derive the vast majority of our understanding of the environment from two types of data: visual and audio. The remaining perception is informed by touch, taste, and smell. With this statistic in mind, it seems we have the tools to create systems that can understand their environment and act accordingly.

💡 Imagine giving a machine the capability to perceive a picture of a subway just as you would do it.

3. How ambient computing enable much easier sensing capabilities?

An ambient system is one that operates autonomously in the background, unobtrusively collecting required data from its environment to fulfill a purpose. Though the biggest notable difference is in their ability to perceive a situation, from data collection to data processing, these systems doesn’t require central human attention. Due to this capability, these systems can play a huge role in wider machine perception.

Where is the work needed?

As also mentioned above we believe we have advanced and varied type of sensors to get in data required for machine perception. I believe the major work is required in the below-mentioned fields:

1. The placement of sensors and tools is crucial to enable a multimodal data stream in a privacy-conscious manner.
2. Fundamentally rethinking perception models to enable a much more diverse and immersive understanding.