Operating machines or playing
videogames just with our thoughts is not science fiction anymore. And Anton Nijholt (see photo on the left), full professor of computer science at the University of Twente, is one of the
researchers working at building so called brain-computer interfaces for these
At CHI 2008, he is one of the organizers of the workshop on Brain-Computer Interfaces for HCI and Games that takes place today. I’ve discuss with him about these topics and a lot of interesting things came out as you can read below.
Could you explain what a
"Brain-Computer Interface" (BCI) is and why it is an important field
“In Brain-Computer Interfacing (BCI), so-called EEG caps are used to measure brain activity. An EEG cap is a cap which has electrodes attached to it and these electrodes measure activity from different regions of the brain. Unfortunately, these electrodes can’t measure activity other than activity close to the skull. Moreover, due to user’s movement (e.g., something as simple as winking of the eye), brain signals can get disturbed. Nevertheless, it is possible to distinguish brain activity coming from different regions of the brain.
Different regions of the brain have different functions. There are parts of the brain that become more active when an anticipated event is perceived. There are also parts of the brain, in the so-called motor cortex, that are concerned with movements your body and limbs can make. In fact, moving the tip of your tongue or imagining to move the tip of your tongue gives already rise to brain activity that can be distinguished from other activity. There is a mapping from body part movements (hands, feet, legs, arms, tongue, whatever) to electrical activity in regions of the brain. Making these movements, observing these movements and, interestingly, imagining these movements all lead to activity in a particular region of the brain. When we can measure and distinguish this activity, then we can also translate measurements to different commands for the computer or to adaptations of the interface.
BCI is an important field of research for different reasons. First of all, there are medical applications. Disabled persons who are not able to move arms or legs can activate particular brain activity, e.g., imagine a particular movement, to control a prosthetic device. But there are also professions, e.g., a pilot of a fighter plane, where the professional can add something to his capabilities by not only using his hands, feet, and voice to issue commands, but also by activating particular regions of his brain. This is very useful, but another application area where we think that BCI can play an interesting role is in games and entertainment.”
And games are indeed one of the
applications your CHI 2008 workshop is investigating. How will gaming be made
more fun by BCIs?
“There are several reasons why the combination of games and BCI is interesting. First of all, we can think of entirely new games that are completely controlled by brain signals. For example, the game Pong, which is a classical virtual tennis game, can be controlled by brain commands only. There are similar games, e.g. moving an avatar in a game environment or in Second Life, commanding an avatar to perform certain actions in the environment, etc. Secondly, probably more importantly, we can look at commands issued from particular brain activity as an extra to the already existing ways of controlling game environments. Consider a situation where we use already our fingers, hands, and maybe voice to control a game, and this added way of controlling an aspect of the game allows us to win the game! Clearly, when gamers have access to such an extra facility, they can improve their scores and their status in the game community. Thirdly, from the measured brain activity in different regions of the brain we can get information about the ‘affective state’ of the user. Is the user bored by the game, is he irritated or maybe angry, is he excited, etc.? Hence, this is about the experience of the gamer, and knowledge about this affective state allows us to adapt the game, the interface, and the ways of interaction to what we know or able to calculate about the users’ affective state. Knowledge about this affective state allows us to make real-time changes in the game in order to keep the gamer ‘in the flow’ of the game.
Unlike medical application where we need to solve a problem, in games we have much more freedom to exploit the possibilities of brain-computer interfacing. We need to build challenging environments that put demands on the user’s cognitive capabilities rather than making it as simple as possible for the user as we want in the case of a disabled person that uses BCI. In a game, all shortcomings from BCI can be translated into challenging features.
Some people consider games as the killer application for BCI. When gamers adopt BCI, then we have a mass market immediately and we have an audience that is ready to explore new ideas and insights immediately. While training time generally is considered to be one problem that often prevents disabled users to make good use of BCI, for gamers this training time can become part of the game, going from one level to a next level in the game or beating an opponent.”
Could you give me a few examples of particularly interesting work that will be presented at your workshop?
“Among others, there will be new results on how to evoke brain activity in a controlled way by letting things happen on a display. This allows the user to choose among alternatives. Others report about experiments where measurement of brain activity is done without needing a tedious preparation time, using so-called 'dry electrodes' technology. Researchers from Microsoft report about their experiments where they measure engagement and stress of a user in order to adapt the interface and the interaction to the mental state of the user. Further experiments with controlling a wheelchair with BCI are presented by researchers from the University of Graz and their colleagues.”
What sorts of new applications do you
imagine for the future (besides gaming), thanks to the research that is being
discussed at your workshop?
“Medical applications of BCI will of course remain extremely important. Applications in gaming and entertainment are very interesting because the application domain is rather about experience, fun, challenges, excitement, etc., than about efficiency. This allows us to design game environments where the use of BCI can be exploited in all kinds of creative ways rather than that having to solve a problem. It is not necessarily the efficiency, but it is the experience to which BCI should contribute. However, many other applications are waiting and in a research context many other applications have already been looked at. As an example, look at the control of your home environment. There are several types of BCI commands that can be used in your smart home environment. By letting different things happen on a screen, leading to different brain activity, the user can choose among alternatives. In that way, commands can be issued to your home environment (open the door, turn on the lights, increase temperature, etc.). One may also look at situations where a user (e.g. a pilot in a fighter plane, a surgeon that controls his devices) is already using hands and voice to perform his tasks and using BCI allows him to get more control on the devices and the environment.”
far away are the applications of BCI?
“Medical applications are already there. There are patients that use BCI to type e-mail, to control a disease or to control a prosthetic device. But then we speak of almost laboratory situations. Applications on a larger scale will only become possible when the measurement of brain signals can be done wireless, when no special preparation for measurements have to be done and when we can reduce the training time that is needed for controlling some kinds of brain activity. It depends on the brain signals that are needed for the application, but generally there are many problems in distinguishing the brain signals that we are after from other signals that are related to other activities of the user. For example, sometimes brain activity related to the blinking of the eyes may disrupt certain patterns that we are trying to measure.”
© 2008, Il Sole 24 Ore. Web report from CHI 2008.