Surfing the web with electrical brain signals: The Brain Web Surfer (BWS) for the completely paralysed
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
PREPRINT VERSION OF CONFERENCE PAPER TO APPEAR IN: PROCEEDINGS OF THE 2ND WORLD CONGRESS OF THE
INTERNATIONAL SOCIETY OF PHYSICAL AND REHABILITATION MEDICINE – ISPRM, 2003
Surfing the web with electrical brain signals: The Brain
Web Surfer (BWS) for the completely paralysed
J. Mellinger1, T. Hinterberger1, M. Bensch2, M. Schröder2,
N. Birbaumer1,3
1
Institute for Medical Psychology and Behavioral Neurobiology
University of Tübingen, Germany
2
Department of Computer Engineering
University of Tübingen, Germany
3
Center of Cognitive Neuroscience, Trento, Italy
Abstract ries will severely impair motor function,
and lead to a state of complete paralysis,
A state of complete paralysis, with higher
with higher cortical functions remaining
cortical functions remaining relatively in-
relatively intact. Examples of respective
tact, is associated with the final stages of a
diseases include cerebral palsy, brain stem
number of neurological disorders, e.g.,
strokes, certain spinal-cord injuries, and
amyotrophic lateral sclerosis (ALS). For
amyotrophic lateral sclerosis (ALS). Pa-
patients suffering from such a disease,
tients suffering from such a disease cannot
means of communication independent of
rely on functional motor channels for com-
functional motor channels are of utter im-
munication; direct brain-computer commu-
portance. With the advent of direct brain-
nication utilizing Brain-Computer Inter-
computer communication (Brain-Computer
faces (BCIs) provide means that, to some
Interfaces, BCIs), such means have become
extent, compensate for the absence of motor
available. While letter spelling is a state-of-
function in communicating with their envi-
the-art application for BCI systems today,
ronment.
interactive access to the World Wide Web
(WWW) is, amongst further applications, In the late 1990s, Birbaumer et al. [1, 2]
one of the most promising, as it enables a were the first to provide ALS patients with
completely paralysed patient to participate a BCI, the so-called Thought Translation
in the broad portion of life reflected by the Device (TTD). The TTD is a non-invasive,
WWW. Since 1999, BCI mediated access EEG-based BCI driven by Slow Cortical
to web browsing and electronic mail has Potentials (SCP) which humans can learn to
been implemented and used with patients in control in an operant conditioning proce-
Tübingen. We discuss the technical and dure [3].
user interface difficulties one encounters The history of providing WWW access to
when connecting a BCI to a web browser, ALS patients dates back to 1999 when the
modelling the BCI as a slow, noisy, low TTD was used to operate a standard web
capacity communication channel; and we browser window [4]. The browser window
present an implementation of a novel web would be shown for a certain amount of
browser user interface that makes better use time (about 120s), then a navigation screen
of the information available via the BCI would present the links from the current
whilst overcoming many of the difficulties web page as leaves in a tree navigated by
present in previous approaches. the binary brain responses from the BCI
Keywords: BCI, WWW, low bandwidth system, side by side with a feedback area
input visualizing the SCP response. Until re-
cently, this system, referred to as 'Des-
Introduction
cartes', has been in use with patients, and
A number of neurological disorders, degen- has proven a valuable evaluation tool for
erative disease and other conditions or inju-
1
PREPRINT VERSION OF CONFERENCE PAPER TO APPEAR IN: PROCEEDINGS OF THE 2ND WORLD CONGRESS OF THE
INTERNATIONAL SOCIETY OF PHYSICAL AND REHABILITATION MEDICINE – ISPRM, 2003
practical problems associated with thought- situation which cannot be handled by the
controlled web browsing. Descartes or by Mankoff et al.'s approach:
The user views a web page that contains a
Recently, Mankoff et al. [5] proposed a web
list of items from a web shop. For each
browser modification requiring a five-class
item, the user may, by clicking a symbol
BCI that separates the screen into three ar-
associated with the item, execute some ac-
eas: One containing the main browser win-
tion, e.g. buy it. If the links do have a tex-
dow, one presenting browser functions and
tual description, it will say 'Buy' for all
links as an array of buttons, and one show-
items, so the user will not know which item
ing a preview of the target page associated
she buys when choosing a link. With
with the currently highlighted link.
Mankoff et al.'s approach, the user will in
More general accessibility approaches for principle be able to choose the correct link
low-bandwidth input are available, though but will have to navigate through a number
not as prominent as solutions for visually of descriptions which are all the same;
impaired web users (cf. [5] for an over- moreover, for building the thumbnail pre-
view); the 'conventional' input method that, view, all temporarily encountered 'Buy'
by its characteristics, most closely matches links will be followed, triggering actions
a BCI system is a 'switch interface' consist- not intended by the user.
ing of one or more physical switches corre-
The scanning approach, operating directly
sponding to different answers a person can
on an interaction surface identical for both
give. With such a system, links and com-
unimpaired and impaired users, circum-
mand options are automatically scanned in
vents the need to maintain a separate list of
a serial fashion, allowing the switch user to
labels. In our own approach, we will avoid
select the element that currently has the
textual labels in a similar way.
focus; or, if two switches are available, one
can be used to move the focus, the other to Explicit low-level navigation
select. Typically, a BCI system provides a number
Shortcomings of current approaches of distinguishable classes (brain responses)
which is less than the number of simultane-
Obtaining labels for selectable options
ous options (e.g. links on a web page, or
In the Descartes as well as in Mankoff et letters from an alphabet) a user can choose
al.'s approach, links and other options are on the application level. Put in the termi-
presented to the user separate from their nology of information theory [6], this re-
location in the browser window. This im- quires some sort of 'encoding' of the high-
plies the need of obtaining a label to serve level 'alphabet symbols' (choices) into se-
as an identifier to the link showing up in the quences of low-level 'channel symbols'
separate list of options; moreover, in the (brain responses). The BCI's user interface
Descartes case, the identifier must be tex- acts as a 'transducer'; in accordance with the
tual because in navigating the tree of links concept of a transducer, it performs state
these must be grouped together according transitions depending on input symbols
to their labels' position in an alphabetical (brain responses) and, less frequently, pro-
sorting. Due to modern web coding prac- duces output symbols (e.g. following links)
tices, it is quite often impossible to obtain a on certain state transitions. From this point
textual label associated with a link on a web of view, a BCI's user interface implements
page. Very often, links are associated with an abstract 'encoding', i.e. a mapping of
graphics instead of a descriptive text, but simultaneous choices onto different se-
even for a textual link its semantic content quences of brain responses, cf. figure 1.
may depend on its position on the web
Because classification errors are rather fre-
page.
quent in a BCI system, there must be a cer-
As an illustration, consider a quite common tain amount of redundancy involved in this
2PREPRINT VERSION OF CONFERENCE PAPER TO APPEAR IN: PROCEEDINGS OF THE 2ND WORLD CONGRESS OF THE
INTERNATIONAL SOCIETY OF PHYSICAL AND REHABILITATION MEDICINE – ISPRM, 2003
encoding to keep confusion between high- its mean selection time grows only loga-
level symbols below a reliability limit. rithmically with the number of options.
Both the Descartes and Mankoff et al.'s The present Brain Web Surfer (BWS)
approach make low-level encoding explicit approach
to the user in form of navigating a tree re- In the work presented here, our goal has
spectively an array of choices, requiring the been to overcome the difficulties discussed
user to keep track of and explicitly alter the in the previous section. Within our ap-
system's current transducer state (figure 1). proach, we use graphical markers 'in-place',
From the user's side, this requires a consid- i.e. on the browser's web page display, in
erable amount of concentration competing the form of coloured frames placed around
with the already demanding task of navigat- user selectable items, circumventing any
ing the WWW itself. This is very much an need to maintain a separate presentation of
issue in the Descartes case where the user choices (cf. figure 2). The frame colours are
needs to keep track of her position in a bi- firmly assigned to the possible brain re-
nary tree but also in the Mankoff et al. case sponses; the current colour of an item's
the reduplication of selectable items, and frame indicates the low-level symbol (brain
the competition of high- and low-level response) currently associated with choos-
navigation structures may be considered a ing the item.
drawback. The user, instead of focusing the
link she intends to follow, additionally This presentation scheme allows the user to
needs to process information about links focus on the link she wants to follow, giv-
she does not intend to follow, and handle ing a series of brain responses as indicated
low-level system states she is not interested by the frame around that link, neglecting
in. any knowledge about the system's low-level
('transducer') state. The user will experience
Inefficient encoding the system asking her a sequence of ques-
The sequential scanning approach does not tions – What colour does 'your' link have
suffer from concurrent navigation structures right now? – which the user answers by
but is still difficult to manage for a low- giving the brain response associated with
bandwidth user with error-prone communi- the colour in question. This way, the system
cation facilities. While it is indeed possible 'gathers information' and follows a link
for the user to focus on the option she wants once it established the user's choice with
to choose and fully ignore all irrelevant some certainty.
options, choosing an option requires an Keeping the system's transducer state out of
amount of time growing linearly with the the user's mental focus has an important
number of simultaneous options. This is consequence: The system may encode high-
equivalent to a rather inefficient low-level level (link) choices into low-level symbols
symbol encoding with a still quite high (brain responses) without constraints be-
probability of transmission errors. cause the user need not understand anything
Mankoff et al.'s suggestion to arrange items about the encoding if she only tries to re-
in a rectangular array results in a mean se- spond as suggested for the link she wants to
lection time linear in the square root of the follow. This implies a number of possible
number of options, and, making better use optimisations:
of the redundancy involved in the proposed
- By reducing the occurrence of a certain
selection method, a smaller error probabil-
brain response in favour of another one,
ity.
an encoding may implement the opti-
From the encoding perspective, the Des- mum low-level symbol frequencies to
cartes approach of arranging high-level achieve maximum channel capacity [6];
options into a binary tree is optimal because
- Asking more than the minimally re-
3PREPRINT VERSION OF CONFERENCE PAPER TO APPEAR IN: PROCEEDINGS OF THE 2ND WORLD CONGRESS OF THE
INTERNATIONAL SOCIETY OF PHYSICAL AND REHABILITATION MEDICINE – ISPRM, 2003
quired number of 'questions', the system step, some redundancy is added to this en-
may employ a redundant encoding to al- coding, allowing for tacit error correction;
low for automatic (i.e. tacit) error cor- this is done by inserting a parity bit after
rection [7]; each n-th binary digit of the Huffman code,
forcing a retransmission of the previous n
- The encoding may dynamically adapt to
bits upon a parity error. For good BCI per-
the user's current performance in terms
formance (small error probability) n is ad-
of BCI classification errors (cf. below
justed towards greater values. Estimates of
for an example);
the current BCI performance could, in prin-
- A-priori knowledge about high-level ciple, be derived from parity error statistics,
choice probabilities (e.g. users more of- and used for adjusting the value of n dy-
ten choose the 'back' button than they namically. However, this straight handling
choose a link) may be used for optimisa- of redundancy is still suboptimal; further
tion (Huffman encoding [8]). work will be performed, following [13], to
Implementation investigate optimal encoding strategies in
depth on the ground of information theo-
For a model implementation of the ap- retic modelling.
proach discussed in the previous section,
we defined an application protocol for data Conclusions
exchange between a BCI and an external The Brain Web Surfer (BWS) system is
application designed to work within the currently being evaluated with healthy per-
TTD [9] and the BCI 2000 framework [10], sons. With the user interface presented here,
and we adapted the open source 'Mozilla' a number of problematic issues from previ-
web browser [11] to implement this proto- ous approaches to low-bandwidth web
col on the application side. The browser browsing have been resolved. By its sim-
will display selectable items with colour plicity for the user, this interface results in
frames indicating which brain response a more freedom in the encoding aspect of a
user should give if she intends to select a BCI application. General information theo-
respective link. Upon receiving classifica- retical modelling of BCI systems allows
tion results (low-level symbols) from the evaluation and optimisation of BCI applica-
BCI, the browser will either turn to high- tions, thereby helping motor impaired and
light a different selection of items or, if the even completely paralysed patients to ac-
selection process is complete – i.e. the tively stay in touch with the world.
browser has obtained enough information
from the user –, it will follow the selected Acknowledgements
link. This work was supported by the German
Bundesministerium für Bildung und For-
For a communication interface such as the
schung (BMBF) and the United States Na-
SCP driven TTD used to operate the modi-
tional Institutes of Health (NIH).
fied web browser, there are two possible
brain responses – cortical positivity vs cor- References
tical negativity, corresponding to red and 1 KÜBLER A, KOTCHOUBEY B,
green item frames – with symmetric confu- HINTERBERGER T et al. The Thought
sion probabilities, resulting in equal symbol Translation Device: A neurophysiological
probabilities for any optimal encoding. For approach to communication in total motor
some browser interface elements, such as paralysis. Experimental Brain Research
the 'back' button, we know that they are 124: 223-232; 1999.
more likely to be selected than others; from
probability estimates based on actual user 2 BIRBAUMER N, KÜBLER A,
behaviour [12] we derive a Huffman encod- GHANAYIM N et al. The Thought Trans-
ing of the high level choices. In a further lation Device (TTD) for Completely Para-
4PREPRINT VERSION OF CONFERENCE PAPER TO APPEAR IN: PROCEEDINGS OF THE 2ND WORLD CONGRESS OF THE
INTERNATIONAL SOCIETY OF PHYSICAL AND REHABILITATION MEDICINE – ISPRM, 2003
lyzed Patients. IEEE Transactions on Re- 8 HUFFMAN D A. A method for the con-
habilitation Engineering, 8: 190-193, 2000. struction of minimum redundancy codes.
Proc IRE 40: 1098-1101, 1952.
3 KÜBLER A, KOTCHOUBEY B,
SALZMANN H P et al. Self-regulation of 9 HINTERBERGER T, MELLINGER J,
slow cortical potentials in completely para- BIRBAUMER N. The Thought Translation
lyzed human patients. Neuroscience Let- Device: Structure of a multimodal brain-
ters, 252: 171-174, 1998. computer communication system. IEEE
Proceedings of the EMBS conference 2003.
4 HINTERBERGER T, KAISER J,
KÜBLER A, NEUMANN N, 10 SCHALK G et al, The BCI 2000 Home-
BIRBAUMER N. The Thought Translation page. http://www.bciresearch.org
Device and its Applications to the Com- 11 THE MOZILLA ORGANIZATION.
pletely Paralyzed. In: DIEBNER H, The Mozilla Open Source Project.
DRUCKREY T, WEIBEL P (eds) Sciences http://www.mozilla.org
of the Interface. Tübingen: Genista, 2001;
232-240. 12 CATLEDGE L D, PITKOW J E. Char-
acterizing Browsing Strategies in the
5 MANKOFF J, DEY A, BATRA U, World-Wide Web. Proceedings of the 3rd
MOORE M. Web Accessibility for Low International World Wide Web Conference,
Bandwidth Input. Proceedings of ASSETS Darmstadt, Germany, 1995.
2002.
13 PERELMOUTER J, BIRBAUMER N.
6 SHANNON C. A Mathematical Theory A Binary Spelling Interface with Random
of Communication. The Bell System Tech- Errors. IEEE Transactions on Rehabilita-
nical Journal 27: 379-423, 623-656, 1948. tion Engineering, 8: 227-232, 2000.
7 HAMMING R W. Error detecting and
error correcting codes. The Bell System
Technical Journal 29: 147-160, 1950.
5PREPRINT VERSION OF CONFERENCE PAPER TO APPEAR IN: PROCEEDINGS OF THE 2ND WORLD CONGRESS OF THE
INTERNATIONAL SOCIETY OF PHYSICAL AND REHABILITATION MEDICINE – ISPRM, 2003
Figure 1: Information channels in a general BCI system
A BCI system contains a presentation channel of capacity CP and a selection channel of capacity CSPREPRINT VERSION OF CONFERENCE PAPER TO APPEAR IN: PROCEEDINGS OF THE 2ND WORLD CONGRESS OF THE
INTERNATIONAL SOCIETY OF PHYSICAL AND REHABILITATION MEDICINE – ISPRM, 2003
Figure 2: A modified web browser displaying a sample web page with colour markers
Each link is provided with a colour frame; to the user, the colour indicates which brain response she should
give next if she intends to follow this link. When used with the Thought Translation Device [9], an SCP
feedback area is displayed to the left of the browser window.
7You can also read
