Hacking the nintendo Wii remote
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
The Hacking Tradition
Hacking the
Nintendo Wii Remote
The global hacking community has collectively reverse-engineered a
significant portion of one of the world’s most sophisticated and common
input devices. And they’re putting it to uses its designers never intended.
I
n November 2006, Nintendo released its torials to demonstrate interaction techniques
fifth home videogame console, the Nin- supported or enabled by the Wii remote. In just
tendo Wii. The company’s previous game a few weeks, these tutorials have received over
console, the Gamecube, hadn’t fared six million unique views and generated over
well in terms of market share against 700,000 software downloads. In this article,
the much higher-powered alternatives released I will talk about the Wii remote’s technology,
by its competitors, Microsoft and Sony. At first cover what’s involved in developing custom ap-
the Wii also seemed significantly underpowered plications, describe intended and unintended
relative to its competitors. However, one year interaction techniques, and outline additional
later it became the market leader of its console uses of the device.
generation, selling over 20 million units world-
wide.1 This success is largely attributable to the Inside the Wii remote
innovative interactive technology and game-play Although the Wii remote’s official specifications
capabilities introduced by the console’s game are unpublished, the global hacking community
controller, the Wii remote, shown in Figure 1. has collectively reverse-engineered a significant
The Nintendo Wii remote, or Wiimote, is a portion of the technical information regarding
handheld device resembling a television remote, its internal workings. Much of this work has
but in addition to buttons, it been collected in online wikis at http://wiili.org
Johnny Chung Lee contains a 3-axis accelerom- and http://wiibrew.org. The body of knowledge
Carnegie Mellon University eter, a high-resolution high- at these sites represents contributions from nu-
speed IR camera, a speaker, a merous individuals and constitutes the source
vibration motor, and wireless for most of the information presented in this
Bluetooth connectivity. This section.
technology makes the Wii remote one of the Because many low-level details are available
most sophisticated PC-compatible input de- online and, furthermore, are likely to be refined
vices available today; together with the game and updated as more information is uncovered,
console’s market success, it’s also one of most the following descriptions of each major Wii re-
common. At a suggested retail price of US$40, mote component represent only higher-level de-
the Wii remote is an impressively cost-effective tails relevant to building custom applications.
and capable platform for exploring interaction
research. Software applications developed for Infrared camera tracker
it have the additional advantage of being read- In the tip of each Wii remote is an IR camera sen-
ily usable by millions of individuals around the sor manufactured by PixArt Imaging, shown in
world who already own the hardware. Figure 2. The camera chip features an integrated
I’ve recently begun using Internet video tu- multiobject tracking (MOT) engine, which
Published by the IEEE CS n 1536-1268/08/$25.00 © 2008 IEEE P ER VA SI V E computing 39
The Hacking Tradition
Broadcom designed for devices that
conform to the Bluetooth Human In-
terface Device standard, such as key-
boards and mice.
The remote isn’t 100 percent compli-
ant with the HID standard, but it can
connect to many Bluetooth-capable
computers.
Internal flash memory
Figure 1. The Nintendo Wii remote game Figure 2. The PixArt IR camera chip. The onboard memory is approximately
controller. (Copyright for all photos, Integrated multiobject tracking 5.5 Kbytes. It’s used for adjusting the
Figures 1–10, Johnny Chung Lee.) minimizes wireless data transmission. device settings, maintaining output
state, and storing data. Nintendo de-
signed it to let users transport and
provides high-resolution, high-speed providing a trigger-like affordance for store a personal profile, called a Mii.
tracking of up to four simultaneous IR the index finger. The remaining seven This memory allows data and identity
light sources. The camera sensor’s ex- buttons are intended to be used by the to be physically associated to a given
act specifications are unpublished, but thumb. The remote design is symmet- remote.
it appears to provide location data with ric, allowing use in either the left or
a resolution of 1,024 × 768 pixels, more right hand. Expansion port
than 4 bits of dot size or light intensity, At the base of the remote is a proprie-
a 100 Hz refresh rate, and a 45 degree Vibration motor (tactile feedback) tary six-pin connector used to commu-
horizontal field of view. The integrated A small vibration motor provides tactile nicate with and power extension con-
hardware object tracking minimizes feedback. The motor is similar to those trollers such as the Nintendo Nunchuk,
the data transmitted over the wireless used in cell phones. The motor state Classic Controller, or a guitar control-
connection and greatly simplifies the has only binary control (on and off), ler. These extensions provide alterna-
implementation of camera-based track- but you can vary the feedback intensity tive form factors and additional input
ing applications. by pulsing the motor activation—that capabilities.
These specifications outperform com- is, by rapidly turning the motor on and The port provides 3.3 V of power
parably priced webcams, which typi- off at different duty cycles. and 400 KHz of I2C serial communi-
cally provide 640 × 480 tracking at 30 cation, to which a microcontroller can
Hz. Webcams also require significant Light-emitting diodes easily interface and effectively provide
CPU power to perform real-time com- (visual feedback) a Bluetooth-to-I2C bridge.
puter-vision tracking. Specialized IR Four blue LEDs at the bottom of the
camera trackers, such as those from Nat- remote are typically used to indicate Batteries
ural Point Systems (www.naturalpoint. player IDs (1 to 4). Each LED’s state The Wii remote uses two AA batteries
com), can provide 710 × 288 tracking is individually addressable. Similarly to and has an operating time between 20
at 120 Hz, but at a significantly higher the vibration motor, pulsing the state and 40 hours, depending on the num-
cost of $180. creates varying levels of brightness. ber of active components. Approxi-
mately 8 bits of battery-level resolution
Accelerometer Speaker (auditory feedback) are available.
Analog Devices manufactures the A small speaker in the remote’s cen-
ADXL330, a 3-axis linear accelerom- ter supports in-game sound effects Developing
eter that provides the Wii remote’s mo- and user feedback. The audio data custom applications
tion-sensing capability. It has a +/−3 g streams directly from the host with Although Nintendo offers a relatively
sensitivity range, 8 bits per axis, and a 4-bit, 4 KHz sound similar in quality inexpensive development kit for the Wii
100 Hz update rate. to a telephone. console, its legal agreement severely
limits the types of applications you’re
Buttons Bluetooth connectivity permitted to develop using its tools. Al-
The Wii remote has 12 buttons. Four Communication runs over a wireless ternatively, you can quite easily connect
are arranged in a standard directional- Bluetooth connection. The connection the Wii remote to a personal computer
pad layout. One button is on the bottom uses a Broadcom 2042 chip, which via Bluetooth and immediately begin
40 P ER VA SI V E computing www.computer.org/pervasive
developing custom applications. The bar” is a misnomer because the device text of bowling, boxing, or playing ten-
remote’s compatibility with the Blue- doesn’t contain sensors; rather, it con- nis, baseball, or golf. The game appears
tooth HID specification manifests on tains two groups of infrared LEDs. The to register subtle variations in swing dy-
the host computer as a joystick. Wii remote’s IR camera sees the two namics and thus affect the simulation.
Software libraries for connecting to groups and provides a method of laser- Though the motion recognition might
a Wii remote, parsing the input report pointer-style input. The software can not necessarily be accurate, the expe-
data, and configuring the controller are transform the x, y coordinate dot pairs rience is quite compelling. However,
available for nearly every major devel- to provide an averaged x, y coordinate the majority of existing games simply
opment platform on Windows, Mac pair, a rotation, and a distance. The x, employ shake recognition to trigger an
OS, and Linux. The open development y, and rotation values correspond to the event similar to a button press.
community has created these libraries, controller’s yaw, pitch, and roll, respec- As in other game consoles, the but-
and you can download them for free. tively, and the distance is estimated us- tons of the Wii remote are heavily
Because these software APIs are in ac- ing the known physical separation of employed for triggering input events.
tive development and might change rap- the two IR groups and the camera’s Frequently, games use the Nunchuk at-
idly, I won’t discuss them in detail. Visit fixed field of view. tachment, which is designed to be held
http://wiili.org and http://wiibrew.org Common Wii game interactions us- in the nondominant hand and adds
for more information. ing the controller as a pointer include more buttons, an analog joystick, and
Accessing the data is usually as sim- selection, navigation, aiming a weapon another accelerometer for independent
ple as reading values from an array or or tool, drawing, rotating objects, and motion sensing in each hand. In total,
an appropriately named member vari- push-pull interactions. Although the the Wii remote with Nunchuck attach-
able of a Wii remote class object, such remote is frequently used as a pointer, ment provides 13 digital inputs, 12 ana-
as accelX = remote.accelerometer.x. The com- no game currently makes a significant log controls, and auditory, visual, and
puter receives input reports 100 times attempt to ensure that the cursor’s on- tactile feedback.
per second, providing low-latency data. screen position accurately matches the
As the software libraries evolve, they screen plane’s intersection with the ray Remote interaction
might support event queues, derivative defined by the axis of the Wii remote. techniques without
values, and utilities that compute useful Assumptions are made regarding the the Wii console
transformation matrices or recognize screen’s visual angle and the scale of The Wii remote’s rich level of input
gestures, thereby simplifying applica- movement. However, this doesn’t ap- and output combined with the ease of
tion development. pear to have a significant impact on us- PC connectivity have made it a popu-
In many cases, the most difficult part ers’ pointing ability in most contexts. lar platform for exploring alternative
of this process is getting the Bluetooth This might be because the game pro- control schemes for existing applica-
pairing to occur successfully. Because vides constant visual feedback of the tions. Many initial projects in the de-
the Wii remote isn’t 100 percent HID cursor position, which lets users rely veloper community involved using the
compliant, it might work only with cer- on relative movements rather than ab- motion- and tilt-sensing capabilities for
tain Bluetooth chipsets and driver soft- solute aiming. robotic control and synthesized musical
ware. However, once a pairing is suc- Use of the accelerometer data within performance. For example, see the Wii
cessful, the configuration is typically
quite reliable. After you’ve connected
the Wii remote and installed the soft- The open development community has created
ware library, developing custom appli-
cations is straightforward. software libraries for connecting to a Wii remote
The projects I describe in this article
are C# Windows software applications for nearly every major development platform.
using Brian Peek’s Managed Library
for the Wiimote. 2 Wii games varies from basic shake trig- remote’s use in making the iSobot per-
gering, to tilt-and-balance control, to form combat motions (www.robodance.
Wii console simple gesture recognition. WiiSports, com/Nintendo-wii-i-sobot.php) and in
interaction techniques the mini-game that comes with the Wii composing music in the Kyma X devel-
Wii users hold the remote controller console, might currently involve the opment environment (www.youtube.
in one hand and point it at a television most intricate use of the accelerometer com/watch?v=ESDzYYl0__s).
that has a Wii sensor bar either above data. WiiSports encourages players to Software libraries to replicate the
or below the screen. The term “sensor swing the remote in the imaginary con- remote’s cursor-pointing capabilities
July–September 2008 P ER VA SI V E computing 41
The Hacking Tradition
Figure 3. Finger tracking. (a) The Wii
remote’s IR LED array illuminator.
(b) A reflector tag increases visibility for
tracking.
pairs to provide an on-screen cursor
(a) (b) and then trigger events by pinching
them together.
Attaching the retroreflective markers
for controlling mouse input were also nificant amount of near-IR light to be to gloves or other wearable accessories
among the early projects. As a result, detected—for example, an IR LED. can help create removable, highly reus-
some people began using the device However, instrumenting surfaces or able markers. The onset of fatigue is
for media navigation or to play mouse- objects with active LED emitters can very rapid in mid-air hand manipula-
based PC games. However, these uses be mechanically prohibitive or unde- tion, so this approach might be practi-
have been somewhat limited because sirable due to battery weight and size cal for only some application types or
the Wii remote requires an IR sensor constraints. better used on more horizontal surfaces
bar to enable pointer tracking. Hands and fingers are good examples for productivity applications.
My work has so far focused on how of surfaces that benefit from minimiz- The technique can also be used to
custom IR emitters can extend the use- ing tracking instrumentation. Camera- track arbitrary objects such as sporting
fulness of the controller’s IR camera be- based motion-capture systems often equipment, physical input devices, or
yond merely distant pointer tracking. employ a technique that uses special- even animals. However, unintentional
When you hold the Wii remote in your ized markers to increase the visibility IR illuminator reflections can generate
hand, the camera sees the IR dot move- of tracked points. The systems can fur- spurious tracking data, complicating
ments primarily in correspondence to ther increase visibility by using retrore- retroreflecting marker tracking. Thus, if
the controller’s yaw, pitch, and roll. The flective tags and colocating specialized instrumentation of the object or surface
tracking data is relatively insensitive to light sources with the tracking camera is acceptable, then active LED markers
translational movement. rather than the tracked point. Vicon will provide less tracking interference
However, when the remote is sta- motion-capture systems use this ap- and tracking at longer distances.
tionary and the IR emitters move, this proach (www.vicon.com).
property is reversed. Dot movement Figure 3 shows the Wii remote LED Interactive whiteboards
corresponds primarily to translation, array and the use of reflective tags to and tablet displays
and the tracking data is relatively in- track fingers. This approach provides By constraining the movement of IR
sensitive to orientation. This is the ar- simple, reliable tracking of multiple emitters to a planar display surface,
rangement that motion-capture systems objects. It could work without the re- you can map the Wii remote camera’s
typically use. Thus, using the remote in flective tags, but the tracking data can coordinate system to the display’s co-
this manner transforms it into a rela- be noisy and the working volume is ordinates. For example, if you point
tively high-performing, commodity small and adjacent to the front of the the camera at a projected image on a
motion-tracking system. Wii remote. wall and then place an IR-emitting light
The rest of this section explores proj- You can trigger events by curl- pen on the surface, you can use the IR
ect applications of this configuration. ing and extending the finger to make camera data to compute which display
Video demonstrations of all these ap- points appear and disappear. The diffi- pixels correspond to the pen’s location.
plications are available on my projects culty of hitting screen targets without This lets you interact with the projected
Web site at http://johnnylee.net. (For a persistent on-screen cursor poses a image as if it were an interactive white-
a useful overview of general tracking usability issue. One approach to re- board system, as shown in Figure 4.
technologies, techniques, and issues, solving this is to have the software To discover the correspondence be-
see B. Danette Allen and her colleagues’ respond to a point’s disappearance tween the camera and projector coordi-
Siggraph course notes.)3 rather than its appearance. This would nates, you use a four-point calibration
be similar to making graphical buttons process typical for any touch-screen sys-
Finger and object tracking respond when you release the mouse tem. First, you display four crosshairs at
Because the Wii remote camera is button events instead of when you known locations in each corner of the
sensitive only to bright sources of IR press the mouse button. Alternatively, projected display, then you activate the
light, tracked objects must emit a sig- you could track thumb and forefinger pen at each of these crosshair locations
42 P ER VA SI V E computing www.computer.org/pervasive
Figure 4. Interactive display. (a) Infrared
LED pens used as a stylus for (b) an
interactive whiteboard.
to register the corresponding camera
coordinates. From these four registered
points, you can compute a homogra-
phy, a warping matrix for mapping any
new point visible to the camera to the (a) (b)
correct pixel location in the projected
image.4 This approach also works with
any flat display surface, such as an LCD
or plasma television. However, displays
that have a thick glass surface can cause
unwanted reflections that result in er-
ratic tracking behavior.
The homography calculation is robust
against display orientation and mirror-
ing, so it supports a variety of camera- (a) (b)
projector geometric relationships. Ad-
ditionally, because the Wii remote can Figure 5. Desktop VR. (a) Rigid IR emitters on glasses together with (b) the Wii
track up to four points, you can track remote can render view-angle-dependent displays that simulate motion parallax
multiple pens simultaneously, creating and a changing field of view.
multitouch interactive surfaces.
The software that performs the four-
point touch calibration and mouse em- the sensor bar: x, y coordinates, rota- Because the software renders a cus-
ulation is available at my projects site, tion, and estimated distance. If you tom viewpoint for the person wear-
along with the video demonstration place the Wii remote adjacent to the ing the IR glasses, the perspective
of this work. The software has been display in a known location and a set will be incorrect for other observers.
downloaded more than 500,000 times of wearable IR emitters on a user’s Some method of using a split screen
as of 1 March 2008. Several educators head, you can track the head’s location or shutter-glass technology could sup-
are already using it in their classrooms relative to the display and render view- port multiple users simultaneously, but
as a low-cost interactive whiteboard angle-dependent views of a virtual en- implementing such an approach would
alternative for certain applications. vironment. Figure 5 shows a system depend on the display technology. The
The approach’s primary limitations implementation that uses glasses with Wii remote’s horizontal field of view
are a maximum tracking resolution of IR emitters. By responding to head might limit the range of movement to a
1,024 × 768 and the high sensitivity movement, the display can simulate smaller usable volume than desired for
of tracking quality to camera position the behavior of a window providing certain applications. However, multi-
and occlusions. Thus, the Wii remote’s motion parallax and a changing field ple remotes could increase the field of
placement is key to obtaining good per- of view, thus increasing the illusion of view. Additionally, conflicting stereo
formance. Overhead or off-to-the-side depth and realism. depth cues from each eye can weaken
placement will reduce the likelihood of Using the known physical separation the illusion. Combining head-tracking
obstructions but also reduce tracking of the IR emitters, you can estimate the with polarized or shutter stereo-vision
uniformity. If a rear-projected arrange- head’s distance from the screen. Simi- goggles could enhance the 3D experi-
ment is possible, it provides ideal track- larly, using the display’s known physi- ence. However, implementing stereo-
ing performance. Multiple Wii remotes cal dimensions, you can calculate the vision techniques can be difficult, de-
could also increase performance. remaining values of vertical and hori- pending on the display technology.
zontal head displacement at the appro-
Head tracking priate scale. Several game and data- Spatial augmented reality
for desktop VR displays visualization companies are already You can augment the appearance of
Two rigidly connected IR points pro- exploring the use of this technique in physical objects by using projected light
vide the same tracking capabilities as future products. to present colocated information on
July–September 2008 P ER VA SI V E computing 43
The Hacking Tradition
Figure 6. Foldable augmented reality
displays projected onto (a) a foldable
newspaper and (b) a foldable fan.
controls, to support an even larger
numbers of objects. The IR transmis-
sion would be visible to the camera that
provides location data, and we could
(a) (b) use the high-speed receiver to demodu-
late the data.
nearby surfaces. This field of research the camera pose would enable registra- IR glyphs
is called spatial augmented reality. tion onto surfaces in 3D space. Because the Wii remote can track up
For fixed objects, you can manu- to four points simultaneously, we could
ally align projected imagery onto a Other projects use spatial and temporal multiplexing
surface’s physical features. 5 However, I hope these few projects I’ve described of IR emitters to create unique identi-
projecting imagery onto moving ob- so far have demonstrated the immense fiers. This would let the remote discover
jects requires very-low-latency, high- utility the Wii remote provides. The fol- the identity of an object it’s pointing at,
resolution tracking to ensure sufficient lowing are additional project concepts which means it could control arbitrary
registration quality to make the illusion that haven’t yet been implemented, but instrumented objects in the environ-
compelling. The Wii remote’s low-cost, could further increase the remote’s pos- ment simply by pointing. The remote
high-performing camera provides an sible applications. could manipulate lights, electronic
attractive option for this application. doors, vehicles, appliances, or other
Unfortunately, the remote can track 3D motion tracking objects in the environment.
only up to four points simultaneously, By using two Wii remotes, we can ap- If the identifiers are associated with
which limits the number of objects you ply stereo-vision techniques to acquire computer displays, individuals could
can track and the geometric complex- 3D tracking data from individual IR use their personal Wii remotes to inter-
ity of surfaces before you must make emitters. Multiple remotes could cover a act with any participating display in an
significant assumptions. larger tracking volume and a wider range intuitive and immediate manner. Using
For example, four points are sufficient of occlusion conditions. IR illumination each controller’s unique identity, seam-
to track a quadrilateral surface’s general and reflective tags could support the as- less file manipulation and management
orientation. However, if the surface is sembly of a low-end motion-capture sys- across displays and computers would
known to be square or is constrained tem for a couple hundred dollars. be possible through a centralized in-
to a table surface, you can use fewer formation server.
points to track the surface orientation Tracking objects with ID
and use the remaining points for input. One limitation of camera-based track- Laser tag
Figure 6 shows examples of a display ing is the inability to easily detect emit- If IR emitters are attached to each re-
projected to a foldable newspaper and ter identity. Researchers have explored mote, each remote can see the others’
a folding fan. A video demonstration of temporal variations in emitter behavior locations. This would support a laser-
this work is available at www.cs.cmu. to communicate identity. However, the tag-style interaction in which individu-
edu/~johnny/academic. data rate of ID transmission is directly als hold their own remotes. Your in-
You can use a four-point homogra- related to a camera’s frame rate, which tended target could be discovered by
phy calibration, similar to the interac- has typically been in the range of 30 blinking each player’s IR emitter in
tive whiteboard application, to regis- Hz. The Wii remote’s 100 Hz refresh some identification or hit-validation
ter surfaces that are constrained to a and several bits of IR dot size or inten- pattern.
plane. However, if you know the pro- sity provide an opportunity for higher
jector parameters, you could leverage data rates resulting in faster recognition Gesture recognition
the epipolar geometry of the projector- of a larger set of trackable and identifi- The gesture recognition in Nintendo Wii
camera pair and compute the funda- able objects. games, using either the accelerometer
mental matrix.6 Given four points of Alternatively, we could couple the data or camera-tracking data, has been
known geometric relationship, com- Wii remote with high-speed IR receiv- rather limited relative to what’s possible
bining the matrix with a solution for ers, similar to those used for remote in contemporary research systems. We
44 P ER VA SI V E computing www.computer.org/pervasive
might explore how to adapt gesture- the AUTHORS
recognition algorithms to the particular Johnny Chung Lee is a researcher in the Applied Sciences group at microsoft-
characteristics of the accelerometer and Hardware, although the work reported in this article was done while he was a
orientation data that the Wii remote’s phD student at Carnegie mellon university. His research focuses on exploring
novel techniques that enhance the practicality and accessibility of interactive
camera provides. The data from these technology. lee received his phD in human-computer interaction from Carn-
inputs presents unique challenges for egie mellon university. Contact him at johnny@cs.cmu.edu.
recognition systems to properly param-
eterize variations in speed, size, and ori-
entation for a given gesture. A number
of developers are currently exploring
this issue, but it still remains an open
research problem. However, a robust community’s energy and imagination www.cs.unc.edu/~tracker/media/pdf/
method for performing accelerometer- will lead to countless more uses than I SIGGRAPH2001_CoursePack_11.pdf.
based gesture recognition would be a could possibly list here. 4. E.W. Weisstein, “Homography,” Math-
significant contribution to a wide vari- World—A Wolfram Web Resource, http://
ety motion-sensing applications in both mathworld.wolfram.com/Homography.
html.
the industrial and research domains.
REFERENCES
5. R. Raskar, G. Welch, and K.-L. Low,
T
1. Nintendo, Consolidated Financial High- “Shader Lamps: Animating Real Objects
lights, 24 Jan. 2008, www.nintendo. with Image-Based Illumination,” Proc.
he Wii remote’s rich I/O ca- Eurographics Workshop on Rendering,
co.jp/ir/pdf/2008/080124e.pdf.
pabilities clearly support a Springer, 2001, pp. 89–102.
wide range of potential ap- 2. Managed Library for Nintendo’s Wiimote,
blog, http://blogs.msdn.com/coding4fun/ 6. R. Hartley and A. Zisserman, Multiple
plications beyond its origi- View Geometry in Computer Vision,
archive/2007/03/14/1879033.aspx.
nal intended use. Its low cost and easy Cambridge Univ. Press, 2003.
Bluetooth connectivity have made it an 3. B.D. Allen, G. Bishop, and G. Welch,
ideal platform for the developer com- “Tracking: Beyond 15 Minutes of
Thought,” Proc. 28th Ann. Conf. Com- For more information on this or any other com-
munity to create custom control and puter Graphics and Interactive Tech- puting topic, please visit our Digital library at
tracking applications. I’m certain the niques (Siggraph 01), ACM Press, 2001, www.computer.org/csdl.
Call for Articles Be on the Cutting Edge of Artificial Intelligence!
IEEE Intelligent Systems
seeks papers on all aspects of
artificial intelligence, focusing on the
development of the latest research
into practical, fielded applications.
For guidelines, see www.computer.
org/mc/intelligent/author.htm.
The #1 AI Magazine
www.computer.org/intelligent
IEEE
July–SEptEmbEr 2008 P ER VA SI V E computing 45
You can also read
