Gesture Recognition in a Natural User Interface

Samuel Wilson

What is a Natural User Interface?

“The most profound technologies are those that disappear. They weave themselves into the fabric of the everyday until they are indistinguishable from it.”

A Natural User Interface is the name given to a method of interacting with a computer in a way that is natural to the user. In short, the user interacts with the computer using traditional methods of communication, such as speech or gestures. These are non-trivial ways of entering data into a computer, as they provide many forms of analogue input, but it does have a number of benefits.

A good demonstration of the benefits of an NUI was demonstrated using a glove which records finger movement. This glove was able to provide simple digital commands, such as pressing the enter key on the computer, but was also able to provide more advanced analogue commands, such as those needed to successfully fly a quadcopter.

Semi-autonomous and remote controlled robots require a human to control them, and even autonomous robots often require a person to teach them how to behave. This is almost always done using a computer, and the Natural User Interface allows this to be done in a way that is far more simple for the person doing it.

Of course, the fundamental problems haven’t changed. It still takes a lot of processing power to achieve the tasks at hand. The only difference is that due to the rate of increase in processing power of computers, they are able to do more of the hard work, making it easier for the user.

Another example is 3D modelling. There are a number of programs that you can download for free that will allow you to build models in 3D, but it takes a lot of practice. One of the main constraints is that you are using a mouse, which only provides a two dimensional input. The human hand has 27 degrees of freedom, which would allow a far greater rate of data transfer.

Applications in Prosthetics

The other major application of this research is in the area of prosthetics. By listening to the sounds that the muscles in the arm make when they are tensed, we can predict some of the movements of the hand. This is possible whether or not the hand is there by analysing these sound waves, known as Mechanomyography. We hope that by increasing the number of signals that we can determine which muscle the signal originated in and therefore what hand gesture the user is attempting to make.