A Survey of Mobile Phone Sensing - AD HOC AND SENSOR NETWORKS
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
LANE LAYOUT 8/24/10 10:43 AM Page 140
AD HOC AND SENSOR NETWORKS
A Survey of Mobile Phone Sensing
Nicholas D. Lane, Emiliano Miluzzo, Hong Lu, Daniel Peebles, Tanzeem Choudhury,
and Andrew T. Campbell, Dartmouth College
ABSTRACT as a platform for sensing research has been dis-
cussed for a number of years now, in both indus-
Mobile phones or smartphones are rapidly trial [8] and research communities [9, 10], there
becoming the central computer and communica- has been little or no advancement in the field
tion device in people’s lives. Application delivery until recently.
channels such as the Apple AppStore are trans- All that is changing because of a number of
forming mobile phones into App Phones, capa- important technological advances. First, the
ble of downloading a myriad of applications in availability of cheap embedded sensors initially
an instant. Importantly, today’s smartphones are included in phones to drive the user experience
programmable and come with a growing set of (e.g., the accelerometer used to change the dis-
cheap powerful embedded sensors, such as an play orientation) is changing the landscape of
accelerometer, digital compass, gyroscope, GPS, possible applications. Now phones can be pro-
microphone, and camera, which are enabling the grammed to support new disruptive sensing
emergence of personal, group, and community- applications such as sharing the user’s real-time
scale sensing applications. We believe that sen- activity with friends on social networks such as
sor-equipped mobile phones will revolutionize Facebook, keeping track of a person’s carbon
many sectors of our economy, including busi- footprint, or monitoring a user’s well being. Sec-
ness, healthcare, social networks, environmental ond, smartphones are open and programmable.
monitoring, and transportation. In this article we In addition to sensing, phones come with com-
survey existing mobile phone sensing algorithms, puting and communication resources that offer a
applications, and systems. We discuss the emerg- low barrier of entry for third-party programmers
ing sensing paradigms, and formulate an archi- (e.g., undergraduates with little phone program-
tectural framework for discussing a number of ming experience are developing and shipping
the open issues and challenges emerging in the applications). Third, importantly, each phone
new area of mobile phone sensing research. vendor now offers an app store allowing develop-
ers to deliver new applications to large popula-
INTRODUCTION tions of users across the globe, which is
transforming the deployment of new applications,
Today’s smartphone not only serves as the key and allowing the collection and analysis of data
computing and communication mobile device of far beyond the scale of what was previously possi-
choice, but it also comes with a rich set of ble. Fourth, the mobile computing cloud enables
embedded sensors, such as an accelerometer, developers to offload mobile services to back-end
digital compass, gyroscope, GPS, microphone, servers, providing unprecedented scale and addi-
and camera. Collectively, these sensors are tional resources for computing on collections of
enabling new applications across a wide variety large-scale sensor data and supporting advanced
of domains, such as healthcare [1], social net- features such as persuasive user feedback based
works [2], safety, environmental monitoring [3], on the analysis of big sensor data.
and transportation [4, 5], and give rise to a new The combination of these advances opens the
area of research called mobile phone sensing. door for new innovative research and will lead to
Until recently mobile sensing research such the development of sensing applications that are
as activity recognition, where people’s activity likely to revolutionize a large number of existing
(e.g., walking, driving, sitting, talking) is classi- business sectors and ultimately significantly
fied and monitored, required specialized mobile impact our everyday lives. Many questions
devices (e.g., the Mobile Sensing Platform remain to make this vision a reality. For exam-
[MSP]) [6] to be fabricated [7]. Mobile sensing ple, how much intelligence can we push to the
applications had to be manually downloaded, phone without jeopardizing the phone experi-
installed, and hand tuned for each device. User ence? What breakthroughs are needed in order
studies conducted to evaluate new mobile sens- to perform robust and accurate classification of
ing applications and algorithms were small-scale activities and context out in the wild? How do we
because of the expense and complexity of doing scale a sensing application from an individual to
experiments at scale. As a result the research, a target community or even the general popula-
which was innovative, gained little momentum tion? How do we use these new forms of large-
outside a small group of dedicated researchers. scale application delivery systems (e.g., Apple
Although the potential of using mobile phones AppStore, Google Market) to best drive data
140 0163-6804/10/$25.00 © 2010 IEEE IEEE Communications Magazine • September 2010
LANE LAYOUT 8/24/10 10:43 AM Page 141
collection, analysis and validation? How can we
exploit the availability of big data shared by
applications but build watertight systems that Ambient light
protect personal privacy? While this new
research field can leverage results and insights
Proximity
from wireless sensor networks, pervasive com-
puting, machine learning, and data mining, it
presents new challenges not addressed by these
communities.
Dual cameras
In this article we give an overview of the sen-
sors on the phone and their potential uses. We
discuss a number of leading application areas and
sensing paradigms that have emerged in the liter- GPS
ature recently. We propose a simple architectural
framework in order to facilitate the discussion of
the important open challenges on the phone and Accelerometer
in the cloud. The goal of this article is to bring
the novice or practitioner not working in this field
quickly up to date with where things stand. Dual microphones
SENSORS Compass
As mobile phones have matured as a computing
platform and acquired richer functionality, these Gyroscope
advancements often have been paired with the
introduction of new sensors. For example,
accelerometers have become common after being Figure 1. An off-the-self iPhone 4, representative of the growing class of sensor-
initially introduced to enhance the user interface enabled phones. This phone includes eight different sensors: accelerometer,
and use of the camera. They are used to automat- GPS, ambient light, dual microphones, proximity sensor, dual cameras, com-
ically determine the orientation in which the user pass, and gyroscope.
is holding the phone and use that information to
automatically re-orient the display between a
landscape and portrait view or correctly orient tinct patterns within the accelerometer data can
captured photos during viewing on the phone. be exploited to automatically recognize different
Figure 1 shows the suite of sensors found in activities (e.g., running, walking, standing). The
the Apple iPhone 4. The phone’s sensors include camera and microphone are powerful sensors.
a gyroscope, compass, accelerometer, proximity These are probably the most ubiquitous sensors
sensor, and ambient light sensor, as well as other on the planet. By continuously collecting audio
more conventional devices that can be used to from the phone’s microphone, for example, it is
sense such as front and back facing cameras, a possible to classify a diverse set of distinctive
microphone, GPS and WiFi, and Bluetooth sounds associated with a particular context or
radios. Many of the newer sensors are added to activity in a person’s life, such as using an auto-
support the user interface (e.g., the accelerome- matic teller machine (ATM), being in a particu-
ter) or augment location-based services (e.g., the lar coffee shop, having a conversation, listening
digital compass). to music, making coffee, and driving [11]. The
The proximity and light sensors allow the camera on the phone can be used for many
phone to perform simple forms of context recog- things including traditional tasks such as photo
nition associated with the user interface. The blogging to more specialized sensing activities
proximity sensor detects, for example, when the such as tracking the user’s eye movement across
user holds the phone to her face to speak. In the phone’s display as a means to activate appli-
this case the touchscreen and keys are disabled, cations using the camera mounted on the front
preventing them from accidentally being pressed of the phone [12]. The combination of
as well as saving power because the screen is accelerometer data and a stream of location esti-
turned off. Light sensors are used to adjust the mates from the GPS can recognize the mode of
brightness of the screen. The GPS, which allows transportation of a user, such as using a bike or
the phone to localize itself, enables new loca- car or taking a bus or the subway [3].
tion-based applications such as local search, More and more sensors are being incorporat-
mobile social networks, and navigation. The ed into phones. An interesting question is what
compass and gyroscope represent an extension new sensors are we likely to see over the next
of location, providing the phone with increased few years? Non-phone-based mobile sensing
awareness of its position in relation to the physi- devices such as the Intel/University of Washing-
cal world (e.g., its direction and orientation) ton Mobile Sensing Platform (MSP) [6] have
enhancing location-based applications. shown value from using other sensors not found
Not only are these sensors useful in driving in phones today (e.g., barometer, temperature,
the user interface and providing location-based humidity sensors) for activity recognition; for
services; they also represent a significant oppor- example, the accelerometer and barometer make
tunity to gather data about people and their it easy to identify not only when someone is
environments. For example, accelerometer data walking, but when they are climbing stairs and in
is capable of characterizing the physical move- which direction. Other researchers have studied
ments of the user carrying the phone [2]. Dis- air quality and pollution [13] using specialized
IEEE Communications Magazine • September 2010 141
LANE LAYOUT 8/24/10 10:43 AM Page 142
project [4] or the Mobile Millennium project [5]
UbitFit Garden Garbage Watch Participatory Urbanism (a joint initiative between Nokia, NAVTEQ, and
the University of California at Berkeley) are
being used to provide fine-grained traffic infor-
mation on a large scale using mobile phones that
facilitate services such as accurate travel time
estimation for improving commute planning.
SOCIAL NETWORKING
Millions of people participate regularly within
online social networks. The Dartmouth
CenceMe project [2] is investigating the use of
sensors in the phone to automatically classify
events in people’s lives, called sensing presence,
and selectively share this presence using online
social networks such as Twitter, Facebook, and
MySpace, replacing manual actions people now
Individual Group Community perform daily.
ENVIRONMENTAL MONITORING
Figure 2. Mobile phone sensing is effective across multiple scales, including: a Conventional ways of measuring and reporting
single individual (e.g., UbitFit Garden [1]), groups such as social networks or environmental pollution rely on aggregate statis-
special interest groups (e.g., Garbage Watch [23]), and entire communities/ tics that apply to a community or an entire city.
population of a city (e.g., Participatory Urbanism [20]). The University of California at Los Angeles
(UCLA) PEIR project [3] uses sensors in phones
to build a system that enables personalized envi-
sensors embedded in prototype mobile phones. ronmental impact reports, which track how the
Still others have embedded sensors in standard actions of individuals affect both their exposure
mobile phone earphones to read a person’s and their contribution to problems such as car-
blood pressure [14] or used neural signals from bon emissions.
cheap off-the-shelf wireless electroencephalogra-
phy (EEG) headsets to control mobile phones HEALTH AND WELL BEING
for hands-free human-mobile phone interaction The information used for personal health care
[36]. At this stage it is too early to say what new today largely comes from self-report surveys and
sensors will be added to the next generation of infrequent doctor consultations. Sensor-enabled
smartphones, but as the cost and form factor mobile phones have the potential to collect in
come down and leading applications emerge, we situ continuous sensor data that can dramatically
are likely to see more sensors added. change the way health and wellness are assessed
as well as how care and treatment are delivered.
The UbiFit Garden [1], a joint project between
APPLICATIONS AND APP STORES Intel and the University of Washington, captures
New classes of applications, which can take levels of physical activity and relates this infor-
advantage of both the low-level sensor data and mation to personal health goals when presenting
high-level events, context, and activities inferred feedback to the user. These types of systems
from mobile phone sensor data, are being have proven to be effective in empowering peo-
explored not only in academic and industrial ple to curb poor behavior patterns and improve
research laboratories [11, 15–22] but also within health, such as encouraging more exercise.
startup companies and large corporations. One
such example is SenseNetworks, a recent U.S.- APP STORES
based startup company, which uses millions of Getting a critical mass of users is a common
GPS estimates sourced from mobile phones problem faced by people who build systems,
within a city to predict, for instance, which sub- developers and researchers alike. Fortunately,
population or tribe might be interested in a spe- modern phones have an effective application dis-
cific type of nightclub or bar (e.g., a jazz club). tribution channel, first made available by Apple’s
Remarkably, it has only taken a few years for App Store for the iPhone, that is revolutionizing
this type of analysis of large-scale location infor- this new field. Each major smartphone vendor
mation and mobility patterns to migrate from has an app store (e.g., Apple AppStore, Android
the research laboratory into commercial usage. Market, Microsoft Mobile Marketplace, Nokia
In what follows we discuss a number of the Ovi). The success of the app stores with the pub-
emerging leading application domains and argue lic has made it possible for not only startups but
that the new application delivery channels (i.e., small research laboratories and even individual
app stores) offered by all the major vendors are developers to quickly attract a very large number
critical for the success of these applications. of users. For example, an early use of app store
distribution by researchers in academia is the
TRANSPORTATION CenceMe application for iPhone [2], which was
Traffic remains a serious global problem; for made available on the App Store when it opened
example, congestion alone can severely impact in 2008. It is now feasible to distribute and run
both the environment and human productivity experiments with a large number of participants
(e.g., wasted hours due to congestion). Mobile from all around the world rather than in labora-
phone sensing systems such as the MIT VTrack tory controlled conditions using a small user
142 IEEE Communications Magazine • September 2010
LANE LAYOUT 8/24/10 10:43 AM Page 143
study. For example, researchers interested in sta-
tistical models that interpret human behavior Mobile computing cloud
from sensor data have long dreamed of ways to
collect such large-scale real-world data. These
app stores represent a game changer for these Big sensor data
types of research. However, many challenges
remain with this new approach to experimenta-
tion via app stores. For example, what is the best
way to collect ground-truth data to assess the
accuracy of algorithms that interpret sensor
data? How do we validate experiments? How do
we select a good study group? How do we deal
with the potentially massive amount of data
made available? How do we protect the privacy
of users? What is the impact on getting approval
for human subject studies from university institu- Inform, share and
tional review boards (IRBs)? How do persuasion
researchers scale to run such large-scale studies?
For example, researchers used to supporting
small numbers of users (e.g., 50 users with b
Application
mobile phones) now have to construct cloud ser- Fi1j1
Rij
Fi2j2
Rij
Finjn
Rij
distribution
vices to potentially deal with 10,000 needy users. Learn Y{l} Y{l} Y{l}
This is fine if you are a startup, but are academic
{l} {l} {l}
ij Mij ij Mij ij Mij
{l} {l} {l}
research laboratories geared to deal with this?
SENSING SCALE AND PARADIGMS
Sense
Future mobile phone sensing systems will oper-
ate at multiple scales, enabling everything from
personal sensing to global sensing as illustrated
in Fig. 2 where we see personal, group, and com-
munity sensing — three distinct scales at which
mobile phone sensing is currently being studied
by the research community. At the same time
researchers are discussing how much the user
(i.e., the person carrying the phone) should be
actively involved during the sensing activity (e.g.,
taking the phone out of the pocket to collect a
sound sample or take a picture); that is, should
the user actively participate, known as participa-
tory sensing [15], or, alternatively, passively par-
ticipate, known as opportunistic sensing [17]?
Each of these sensing paradigms presents impor-
tant trade-offs. In what follows we discuss differ- Figure 3. Mobile phone sensing architecture.
ent sensing scales and paradigms.
SENSING SCALE sensing information freely or with privacy pro-
Personal sensing applications are designed for a tection. There is an element of trust in group
single individual, and are often focused on data sensing applications that simplify otherwise diffi-
collection and analysis. Typical scenarios include cult problems, such as attesting that the collect-
tracking the user’s exercise routines or automating ed sensor data is correct or reducing the degree
diary collection. Typically, personal sensing appli- to which aggregated data must protect the indi-
cations generate data for the sole consumption of vidual. Common use cases include assessing
the user and are not shared with others. An excep- neighborhood safety, sensor-driven mobile social
tion is healthcare applications where limited shar- networks, and forms of citizen science. Figure 2
ing with medical professionals is common (e.g., shows GarbageWatch [23] as an example of a
primary care giver or specialist). Figure 2 shows group sensing application where people partici-
the UbitFit Garden [1] as an example of a person- pate in a collective effort to improve recycling by
al wellness application. This personal sensing capturing relevant information needed to
application adopts persuasive technology ideas to improve the recycling program. For example,
encourage the user to reach her personal fitness students use the phone’s camera to log the con-
goals using the metaphor of a garden blooming as tent of recycling bins used across a campus.
the user progresses toward their goals. Most examples of community sensing only
Individuals who participate in sensing appli- become useful once they have a large number of
cations that share a common goal, concern, or people participating; for example, tracking the
interest collectively represent a group. These spread of disease across a city, the migration
group sensing applications are likely to be popu- patterns of birds, congestion patterns across city
lar and reflect the growing interest in social net- roads [5], or a noise map of a city [24]. These
works or connected groups (e.g., at work, in the applications represent large-scale data collection,
neighborhood, friends) who may want to share analysis, and sharing for the good of the commu-
IEEE Communications Magazine • September 2010 143LANE LAYOUT 8/24/10 10:43 AM Page 144
taking a picture). An advantage is that complex
operations can be supported by leveraging the
intelligence of the person in the loop who can
solve the context problem in an efficient man-
ner; that is, a person who wants to participate in
collecting a noise or air quality map of their
Raw data Extracted features Classification inferences neighborhood simply takes the phone out of
their bag to solve the context problem. One
Figure 4. Raw audio data captured from mobile phones is transformed into drawback of participatory sensing is that the
features allowing learning algorithms to identify classes of behavior (e.g., driv- quality of data is dependent on participant
ing, in conservation, making coffee) occurring in a stream of sensor data, for enthusiasm to reliably collect sensing data and
example, by SoundSense [11]. the compatibility of a person’s mobility patterns
to the intended goals of the application (e.g.,
collect pollution samples around schools). Many
nity. To achieve scale implicitly requires the of these challenges are actively being studied.
cooperation of strangers who will not trust each For example, the PICK project [23] is studying
other. This increases the need for community models for systematically recruiting participants.
sensing systems with strong privacy protection Clearly, opportunistic and participatory rep-
and low commitment levels from users. Figure 2 resent extreme points in the design space. Each
shows carbon monoxide readings captured in approach has pros and cons. To date there is lit-
Ghana using mobile sensors attached to taxicabs tle experience in building large-scale participato-
as part of the Participatory Urbanism project ry or opportunistic sensing applications to fully
[20] as an example of a community sensing appli- understand the trade-offs. There is a need to
cation. This project, in conjunction with the N- develop models to best understand the usability
SMARTs project [13] at the University of and performance issues of these schemes. In
California at Berkeley, is developing prototypes addition, it is likely that many applications will
that allow similar sensor data to be collected emerge that represent a hybrid of both these
with phone embedded sensors. sensing paradigms.
The impact of scaling sensing applications
from personal to population scale is unknown.
Many issues related to information sharing, pri- MOBILE PHONE SENSING
vacy, data mining, and closing the loop by pro-
viding useful feedback to an individual, group,
ARCHITECTURE
community, and population remain open. Today, Mobile phone sensing is still in its infancy. There
we only have limited experience in building scal- is little or no consensus on the sensing architec-
able sensing systems. ture for the phone and the cloud. For example,
new tools and phone software will be needed to
SENSING PARADIGMS facilitate quick development and deployment of
One issue common to the different types of sens- robust context classifiers for the leading phones
ing scale is to what extent the user is actively on the market. Common methods for collecting
involved in the sensing system [12]. We discuss and sharing data need to be developed. Mobile
two points in the design space: participatory sens- phones cannot be overloaded with continuous
ing, where the user actively engages in the data sensing commitments that undermine the perfor-
collection activity (i.e., the user manually deter- mance of the phone (e.g., by depleting battery
mines how, when, what, and where to sample) and power). It is not clear what architectural compo-
opportunistic sensing, where the data collection nents should run on the phone and what should
stage is fully automated with no user involvement. run in the cloud. For example, some researchers
The benefit of opportunistic sensing is that it propose that raw sensor data should not be
lowers the burden placed on the user, allowing pushed to the cloud because of privacy issues. In
overall participation by a population of users to the following sections we propose a simple archi-
remain high even if the application is not that tectural viewpoint for the mobile phone and the
personally appealing. This is particularly useful computing cloud as a means to discuss the major
for community sensing, where per user benefit architectural issues that need to be addressed.
may be hard to quantify and only accrue over a We do not argue that this is the best system
long time. However, often these systems are architecture. Rather, it presents a starting point
technically difficult to build [25], and a major for discussions we hope will eventually lead to a
resource, people, are underutilized. One of the converging view and move the field forward.
main challenges of using opportunistic sensing is Figure 3 shows a mobile phone sensing archi-
the phone context problem; for example, the tecture that comprises the following building
application wants to only take a sound sample blocks.
for a city-wide noise map when the phone is out
of the pocket or bag. These types of context SENSE
issues can be solved by using the phone sensors; Individual mobile phones collect raw sensor data
for example, the accelerometer or light sensors from sensors embedded in the phone.
can determine if the phone is out of the pocket.
Participatory sensing, which is gaining inter- LEARN
est in the mobile phone sensing community, Information is extracted from the sensor data by
places a higher burden or cost on the user; for applying machine learning and data mining tech-
example, manually selecting data to collect (e.g., niques. These operations occur either directly on
lowest petrol prices) and then sampling it (e.g., the phone, in the mobile cloud, or with some
144 IEEE Communications Magazine • September 2010LANE LAYOUT 8/24/10 10:43 AM Page 145
partitioning between the phone and cloud. writing common data processing components,
Where these components run could be governed such as signal processing routines, or performing Most of the
by various architectural considerations, such as computationally costly inference due to the
smartphones on the
privacy, providing user real-time feedback, resource constraints of the phone. Early sensor-
reducing communication cost between the phone enabled phones (i.e., prior to the iPhone in market are open and
and cloud, available computing resources, and 2007) such as the Symbian-based Nokia N80 programmable by
sensor fusion requirements. We therefore con- included an accelerometer, but there were no
sider where these components run to be an open open application programming interfaces (APIs) third party
issue that requires research. to access the sensor signals. This has changed developers and offer
significantly over the last few years. Note that
INFORM, SHARE, AND PERSUASION SDKs, APIs, and
phone vendors initially included accelerometers
We bundle a number of important architectural to help improve the user interface experience. software tools. It is
components together because of commonality or Most of the smartphones on the market are easy to cross-compile
coupling of the components. For example, a per- open and programmable by third-party develop-
sonal sensing application will only inform the user, ers, and offer software development kits (SDKs), code and leverage
whereas a group or community sensing application APIs, and software tools. It is easy to cross-com- existing software
may share an aggregate version of information pile code and leverage existing software such as
such as established
with the broader population and obfuscate the established machine learning libraries (e.g.,
identity of the users. Other considerations are how Weka). machine learning
to best visualize sensor data for consumption of However, a number of challenges remain in libraries.
individuals, groups, and communities. Privacy is a the development of sensor-based applications.
very important consideration as well. Most vendors did not anticipate that third par-
While phones will naturally leverage the dis- ties would use continuous sensing to develop
tributed resources of the mobile cloud (e.g., new applications. As a result, there is mixed API
computation and services offered in the cloud), and operating system (OS) support to access the
the computing, communications, and sensing low-level sensors, fine-grained sensor control,
resources on the phones are ever increasing. We and watchdog timers that are required to devel-
believe that as resources of the phone rapidly op real-time applications. For example, on Nokia
expand, one of the main benefits of using the Symbian and Maemo phones the accelerometer
mobile computing cloud will be the ability to returns samples to an application unpredictably
compute and mine big data from very large num- between 25–38 Hz, depending on the CPU load.
bers of users. The availability of large-scale data While this might not be an issue when using the
benefits mobile phone sensing in a variety of accelerometer to drive the display, using statisti-
ways; for example, more accurate interpretation cal models to interpret activity or context typi-
algorithms that are updated based on sensor cally requires high and at least consistent
data sourced from an entire user community. sampling rates.
This data enables personalizing of sensing sys- Lack of sensor control limits the management
tems based on the behavior of both the individu- of energy consumption on the phone. For
al user and cliques of people with similar instance, the GPS uses a varying amount of
behavior. power depending on factors such as the number
In the remainder of the article we present a of satellites available and atmospheric condi-
detailed discussion of the three main architec- tions. Currently, phones only offer a black box
tural components introduced in this section: interface to the GPS to request location esti-
• Sense mates. Finer-grained control is likely to help in
• Learn preserving battery power and maintaining accu-
• Inform, share, and persuasion racy; for example, location estimation could be
aborted when accuracy is likely to be low, or if
the estimate takes too long and is no longer use-
SENSE: THE MOBILE PHONE AS A ful.
As third parties demand better support for
SENSOR sensing applications, the API and OS support
As we discussed, the integration of an ever will improve. However, programmability of the
expanding suite of embedded sensors is one of phone remains a challenge moving forward. As
the key drivers of mobile phone applications. more individual, group, and community-scale
However, the programmability of the phones applications are developed there will be an
and the limitation of the operating systems that increasing demand placed on phones, both indi-
run on them, the dynamic environment present- vidually and collectively. It is likely that abstrac-
ed by user mobility, and the need to support tions that can cope with persistent spatial queries
continuous sensing on mobile phones present a and secure the use of resources from neighbor-
diverse set of challenges the research community ing phones will be needed. Phones may want to
needs to address. interact with other collocated phones to build
new sensing paradigms based on collaborative
PROGRAMMABILITY sensing [12].
Until very recently only a handful of mobile Different vendors offer different APIs, mak-
phones could be programmed. Popular plat- ing porting the same sensing application to mul-
forms such as Symbian-based phones presented tivendor platforms challenging. It is useful for
researchers with sizable obstacles to building the research community to think about and pro-
mobile sensing applications [2]. These platforms pose sensing abstractions and APIs that could be
lacked well defined reliable interfaces to access standardized and adopted by different mobile
low-level sensors and were not well suited to phone vendors.
IEEE Communications Magazine • September 2010 145LANE LAYOUT 8/24/10 10:43 AM Page 146
CONTINUOUS SENSING Continuous sensing raises considerable chal-
Different vendors lenges in comparison to sensing applications that
Continuous sensing will enable new applications require a short time window of data or a single
offer different APIs, across a number of sectors but particularly in snapshot (e.g., a single image or short sound clip).
making porting the personal healthcare. One important OS require- There is an energy tax associated with continuous-
same sensing appli- ment for continuous sensing is that the phone ly sensing and potentially uploading in real time to
supports multitasking and background process- the cloud for further processing. Solutions that
cation to multi-ven- ing. Today, only Android and Nokia Maemo limit the cost of continuous sensing and reduce
dor platforms phones support this capability. The iPhone 4 OS, the communication overheard are necessary. If the
while supporting the notion of multitasking, is interpretation of the data can withstand delays of
challenging. It is use- inadequate for continuous sensing. Applications an entire day, it might be acceptable if the phone
ful for the research must conform to predefined profiles with strict can collect and store the sensor data until the end
community to think constraints on access to resources. None of these of the day and upload when the phone is being
profiles provide the ability to have continuous charged. However, this delay-tolerant model of
about and propose access to all the sensors (e.g., continuous sensor sampling and processing severely limits the
sensing abstractions accelerometer sampling is not possible). ability of the phone to react and be aware of its
While smartphones continue to provide more context. Sensing applications that will be success-
and APIs that could computation, memory, storage, sensing, and com- ful in the real world will have to be smart enough
be standardized and munication bandwidth, the phone is still a to adapt to situations. There is a need to study the
adopted by different resource-limited device if complex signal process- trade-off of continuous sensing with the goal of
ing and inference are required. Signal processing minimizing the energy cost while offering suffi-
mobile phone and machine learning algorithms can stress the cient accuracy and real-time responsiveness to
vendors. resources of the phones in different ways: some make the application useful.
require the CPU to process large volumes of sen- As continuous sensing becomes more com-
sor data (e.g., interpreting audio data [12]), some mon, it is likely that additional processing sup-
need frequent sampling of energy expensive sen- port will emerge. For example, the Little Rock
sors (e.g., GPS [3]), while others require real-time project [28] underway at Microsoft Research is
inference (e.g., Darwin [12]). Different applica- developing hardware support for continuous
tions place different requirements on the execu- sensing where the primary CPU frequently
tion of these algorithms. For example, for sleeps, and digital signal processors (DSPs) sup-
applications that are user initiated the latency of port the duty cycle management, sensor sam-
the operation is important. Applications (e.g., pling, and signal processing.
healthcare) that require continuous sensing will
often require real-time processing and classifica- PHONE CONTEXT
tion of the incoming stream of sensor data. We Mobile phones are often used on the go and in
believe continuous sensing can enable a new class ways that are difficult to anticipate in advance.
of real-time applications in the future, but these This complicates the use of statistical models
applications may be more resource demanding. that may fail to generalize under unexpected
Phones in the future should offer support for con- environments. The background environment or
tinuous sensing without jeopardizing the phone actions of the user (e.g., the phone could be in
experience; that is, not disrupt existing applica- the pocket) will also affect the quality of the sen-
tions (e.g., to make calls, text, and surf the web) or sor data that is captured. Phones may be exposed
drain batteries. Experiences from actual deploy- to events for too short a period of time, if the
ments of mobile phone sensing systems show that user is traveling quickly (e.g., in a car), if the
phones which run these applications can have event is localized (e.g., a sound) or the sensor
standby times reduced from 20 hours or more to requires more time than is possible to gather a
just six hours [2]. For continuous sensing to be sample (e.g., air quality sensor). Other forms of
viable there need to be breakthroughs in low-ener- interfering context include a person using their
gy algorithms that duty cycle the device while phone for a call, which interferes with the ability
maintaining the necessary application fidelity. of the accelerometer to infer the physical actions
Early deployments of phone sensing systems of the person. We collectively describe these
tended to trade off accuracy for lower resource issues as the context problem. Many issues remain
usage by implementing algorithms that require open in this area.
less computation or a reduced amount of sensor Some researchers propose to leverage co-
data. Another strategy to reduce resource usage located mobile phones to deal with some of
is to leverage cloud infrastructure where differ- these issues; for example, sharing sensors tem-
ent sensor data processing stages are offloaded porarily if they are better able to capture the
to back-end servers [12, 26] when possible. Typi- data [12]. To counter context challenges
cally, raw data produced by the phone is not sent researchers proposed super-sampling [13] where
over the air due to the energy cost of transmis- data from nearby phones are collectively used to
sion, but rather compressed summaries (i.e., lower the aggregate noise in the reading. Alter-
extracted features from the raw sensor data) are natively, an effective approach for some systems
sent. The drawback to these approaches is that have been sensor sampling routines with admis-
they are seldom sufficiently energy-efficient to sion control stages that do not process data that
be applied to continuous sensing scenarios. is low-quality, saving resources, and reducing
Other techniques rely on adopting a variety of errors (e.g., SoundSense [11]).
duty cycling techniques that manage the sleep While machine learning techniques are being
cycle of sensing components on the phone in used to interpret mobile phone data, the reliabil-
order to trade off the amount of battery con- ity of these algorithms suffer under the dynamic
sumed against sensing fidelity and latency [27]. and unexpected conditions presented by every-
146 IEEE Communications Magazine • September 2010LANE LAYOUT 8/24/10 10:43 AM Page 147
day phone use. For example, a speaker identifi- dow could be useful for separating standing and
cation algorithm maybe effective in a quiet office walking classes). Supervised learning is feasible A natural question is
environment but not a noisy cafe. Such problems for small-scale sensing applications, but unlikely
can be overcome by collecting sufficient exam- to scale to handle the wide range of behaviors how well can mobile
ples of the different usage scenarios (i.e., train- and contexts exhibited by a large community of phones interpret
ing data). However, acquiring examples is costly users. Other forms of learning algorithms, such human behavior
and anticipating the different scenarios the as semi-supervised (where only some of the data
phone might encounter is almost impossible. is labeled) and unsupervised (where no labels (e.g., sitting in con-
Some solutions to this problem straddle the are provided by the user) ones, reduce the need servation) from low-
boundary of mobile systems and machine learn- for labeled examples, but can lead to classes that
ing and include borrowing model inputs (i.e., do not correspond to the activities that are use- level multimodal
features) from nearby phones, performing col- ful to the application or require that the unla- sensor data? Or, sim-
laborative multi-phone inference with models beled data only come from the already labeled ilarly, how accurately
that evolve based on different scenarios encoun- class categories (e.g., an activity that was never
tered, or discovering new events that are not encountered before can throw off a semi-super- can they infer the
encountered during application design [12]. vised learning algorithm). surrounding context
Researchers show that a variety of everyday
human activities can be inferred most successful- (e.g., pollution,
LEARN: INTERPRETING SENSOR DATA ly from multimodal sensor streams For example, weather, noise
The raw sensor data able to be acquired by [29] describes a system which is capable of recog- environment)?
phones, irrespective of the scale or modality (e.g., nizing eight different everyday activities (e.g.,
accelerometer, camera) are worthless without brushing teeth, riding in an elevator) using the
interpretation (e.g., human behavior recogni- Mobile Sensing Platform (MSP) [6] — an impor-
tion). A variety of data mining and statistical tant mobile sensing device that is a predecessor
tools can be used to distill information from the of sensing on the mobile phone. Similar results
data collected by mobile phones and calculate are demonstrated using mobile phones that infer
summary statistics to present to the users, such everyday activities [2, 3, 30], albeit less accurately
as, the average emissions level of different loca- and with a smaller set of activities than the MSP.
tions or the total distance run by a user and their The microphone, accelerometer, and GPS
ranking within a group of friends (e.g., Nike+). found on many smartphones on the market have
Recently, crowd-sourcing techniques have proven to be effective at inferring more complex
been applied to the analysis of sensor data which human behavior. Early work on mobility pattern
is typically problematic; for example, image pro- modeling succeeds with surprisingly simple
cessing when used in-the-wild is notoriously dif- approaches to identify significant places in peo-
ficult to maintain high accuracy. In the ple’s lives (e.g., work, home, coffee shop). More
CrowdSearch [21] project, crowd sourcing and recently researchers [31] have used statistical
micro-payments are adopted to incentivize peo- techniques to not only infer significant places but
ple to improve automated image search. In [21] also connect these to activities (e.g., gym, waiting
human-in-the-loop stages are added to the pro- for the bus) using just GPS traces. The micro-
cess of image search with tasks distributed to the phone is one of the most ubiquitous sensors and
user population. is capable of inferring what a person is doing
We discuss the key challenges in interpreting (e.g., in conversation), where they are (e.g., audio
sensor data, focusing on a primary area of inter- signature of a particular coffee shop) — in
est: human behavior and context modeling. essence, it can capture a great deal both about a
person and their surrounding ambient environ-
HUMAN BEHAVIOR AND CONTEXT MODELING ment. In SoundSense [11] a general-purpose
Many emerging applications are people-centric, sound classification system for mobile phones is
and modeling the behavior and surrounding con- developed using a combination of supervised and
text of the people carrying the phones is of par- unsupervised learning. The recognition of a static
ticular interest. A natural question is how well set of common sounds (e.g., music) uses super-
can mobile phones interpret human behavior vised learning but augmented with an unsuper-
(e.g., sitting in conversation) from low-level mul- vised approach that learns the novel frequently
timodal sensor data? Or, similarly, how accurate- recurring classes of sound encountered by differ-
ly can they infer the surrounding context (e.g., ent users. Finally, the user is brought into the
pollution, weather, noise environment)? loop to confirm and provide a textual description
Currently, supervised learning techniques are (i.e., label) of the discovered sounds. As a result,
the algorithms of choice in building mobile SoundSense extends the ability of the phone to
inference systems. In supervised-learning, as recognize new activities.
illustrated in Fig. 4, examples of high-level
behavioral classes (e.g., cooking, driving) are SCALING MODELS
hand annotated (i.e., labeled). These examples, Existing statistical models are unable to cope
referred to as training data, are then provided to with everyday occurrences such as a person using
a learning algorithm, which fits a model to the a new type of exercise machine, and struggle
classes (i.e., behaviors) based on the sensor data. when two activities overlap each other or differ-
Sensor data is usually presented to the learning ent individuals carry out the same activity differ-
algorithm in the form of extracted features, ently (e.g., the sensor data for walking will look
which are calculations on the raw data that very different for a 10-year-old vs. a 90-year-old
emphasize characteristics that more clearly dif- person). A key to scalability is to design tech-
ferentiate classes (e.g., the variance of the niques for generalization that will be effective for
accelerometer magnitude over a small time win- entire communities containing millions of people.
IEEE Communications Magazine • September 2010 147LANE LAYOUT 8/24/10 10:43 AM Page 148
To address these concerns current research using a web portal where sensor data and infer-
Existing statistical directions point toward models that are adaptive ences are easily displayed. This offers a familiar
and incorporate people in the process. Automati- and intuitive interface. For the same reasons, a
models are unable to cally increasing the classes recognized by a model number of phone sensing systems connect with
cope with everyday using active learning (where the learning algo- existing web applications to either enrich existing
occurrences such as rithm selectively queries the user for labels) is applications or make the data more widely acces-
investigated in the context of heath care [23]. sible [12, 23]. Researchers recognize the strength
a person using a Approaches have been developed in which train- of leveraging social media outlets such as Face-
new type of exercise ing data sourced directly from users is grouped book, Twitter, and Flickr as ways to not only dis-
based on their social network [12]. This work seminate information but build community
machine, and demonstrates that exploiting the social network of awareness (e.g., citizen science [20]). A popular
struggle when two users improves the classification of locations such application domain is fitness, such as Nike+.
activities overlap as significant places. Community-guided learning Such systems combine individual statistics and
[30] combines data similarity and crowd-sourced visualizations of sensed data and promote com-
each other or when labels to improve the classification accuracy of the petition between users. The result is the forma-
different individuals learning system. In [30] hand annotated labels are tion of communities around a sensing
no longer treated as absolute ground truth during application. Even though, as in the case of
carry out the same the training process but are treated as soft hints Nike+, the sensor information is rather simple
activity differently. as to class boundaries in combination with the (i.e., just the time and distance of a run), people
observed data similarity. This approach learns still become very engaged. Other applications
classes (i.e., activities) based on the actual behav- have emerged that are considerably more sophis-
ior of the community and adjusts transparently to ticated in the type of inference made, but have
the changes in how the community performs had limited up take. It is still too early to predict
these activities — making it more suitable for which sensing applications will become the most
large-scale sensing applications. However, if the compelling for user communities. But social net-
models need to be adapted on the fly, this may working provides many attractive ways to share
force the learning of models to happen on the information.
phone, potentially causing a significant increase in
computational needs [12]. PERSONALIZED SENSING
Many questions remain regarding how learn- Mobile phones are not limited to simply collect-
ing will progress as the field grows. There is a ing sensor data. For example, both the Google
lack of shared technology that could help accel- and Microsoft search clients that run on the
erate the work. For example, each research iPhone allow users to search using voice recogni-
group develops their own classifiers that are tion. Eye tracking and gesture recognition are
hand coded and tuned. This is time consuming also emerging as natural interfaces to the phone.
and mostly based on small-scale experimentation Sensors are used to monitor the daily activi-
and studies. There is a need for a common ties of a person and profile their preferences and
machine learning toolkit for mobile phone sens- behavior, making personalized recommendations
ing that allows researchers to build and share for services, products, or points of interest possi-
models. Similarly, there is a need for large-scale ble [32]. The behavior of an individual along
public data sets to study more advanced learning with an understanding of how behavior and pref-
techniques and rigorously evaluate the perfor- erences relate to other segments of the popula-
mance of different algorithms. Finally, there is tion with similar behavioral profiles can radically
also a need for a repository for sharing datasets, change not only online experiences but real
code, and tools to support the researchers. world ones too. Imagine walking into a pharma-
cy and your phone suggesting vitamins and sup-
plements with the effectiveness of a doctor. At a
INFORM, SHARE, AND PERSUASION: clothing store your phone could identify which
items are manufactured without sweatshop labor.
CLOSING THE SENSING LOOP The behavior of the person, as captured by sen-
How you use inferred sensor data to inform the sors embedded in their phone, become an inter-
user is application-specific. But a natural question face that can be fed to many services (e.g.,
is, once you infer a class or collect together a set targeted advertising). Sensor technology person-
of large-scale inferences, how do you close the alized to a user’s profile empowers her to make
loop with people and provide useful information more informed decisions across a spectrum of
back to users? Clearly, personal sensing applica- services.
tions would just inform the individual, while social
networking sensing applications may share activi- PERSUASION
ties or inferences with friends. We discuss these Sensor data gathered from communities (e.g.,
forms of interaction with users as well as the fitness, healthcare) can be used not only to
important area of privacy. Another topic we inform users but to persuade them to make posi-
touch on is using large-scale sensor data as a per- tive behavioral changes (e.g., nudge users to
suasive technology — in essence using big data to exercise more or smoke less). Systems that pro-
help users attain goals using targeted feedback. vide tailored feedback with the goal of changing
users’ behavior are referred to as persuasive
SHARING technology [33]. Mobile sensing applications
To harness the potential of mobile phone sens- open the door to building novel persuasive sys-
ing requires effective methods of allowing peo- tems that are still largely unexplored.
ple to connect with and benefit from the data. For many application domains, such as
The standard approach to sharing is visualization healthcare or environmental awareness, users
148 IEEE Communications Magazine • September 2010LANE LAYOUT 8/24/10 10:43 AM Page 149
commonly have desired objectives (e.g., to lose Privacy for group sensing applications is based
weight or lower carbon emissions). Simply pro- on user group membership. For instance, The risks from
viding a user with her own information is often although social networking applications like
not enough to motivate a change of behavior or Loopt and CenceMe [2] share sensitive informa- location-based
habit. Mobile phones are an ideal platform capa- tion (e.g., location and activity), they do so within attacks are fairly well
ble of using low-level individual-scale sensor groups in which users have an existing trust rela- understood given
data and aggregated community-scale informa- tionship based on friendship or a shared common
tion to drive long-term change (e.g., contrasting interest such as reducing their carbon footprint. years of previous
the carbon footprint of a user with her friends Community sensing applications that can col- research. However,
can persuade the user to reduce her own foot- lect and combine data from millions of people
print). The UbiFit Garden [1] project is an early run the risk of unintended leakage of personal our understanding of
example of integrating persuasion and sensing information. The risks from location-based the dangers of other
on the phone. UbiFit uses an ambient back- attacks are fairly well understood given years of modalities (e.g.,
ground display on the phone to offer the user previous research. However, our understanding
continuous updates on her behavior in response of the dangers of other modalities (e.g., activity activity inferences,
to desired goals. The display uses the metaphor inferences, social network data) are less devel- social network data)
of a garden with different flowers blooming in oped. There are growing examples of reconstruc-
response to physical exercise of the user during tion type attacks where data that may look safe are less developed.
the day. It does not use comparison data but and innocuous to an individual user may allow
simply targets the individual user. A natural invasive information to be reverse-engineered.
extension of UbiFit is to present community For example, the UIUC Poolview project shows
data. Ongoing research is exploring methods of that even careful sharing of personal weight data
identifying and using people in a community of within a community can expose information on
users as influencers for different individuals in whether a user’s weight is trending upward or
the user population. A variety of techniques are downward [35]. The PEIR project evaluates dif-
used in existing persuasive system research, such ferent countermeasures to this type of scenario,
as the use of games, competitions among groups such as adding noise to the data or replacing
of people, sharing information within a social chunks of the data with synthetic but realistic
network, or goal setting accompanied by feed- samples that have limited impact on the quality
back. Understanding which types of metaphors of the aggregate analysis [3].
and feedback are most effective for various per- Privacy and anonymity will remain a signifi-
suasion goals is still an open research problem. cant problem in mobile-phone-based sensing for
Building mobile phone sensing systems that inte- the foreseeable future. In particular, the second-
grate persuasion requires interdisciplinary hand smoke problem of mobile sensing creates
research that combines behavioral and social new privacy challenges, such as:
psychology theories with computer science. • How can the privacy of third parties be
The use of large volumes of sensor data pro- effectively protected when other people
vided by mobile phones presents an exciting wearing sensors are nearby?
opportunity and is likely to enable new applica- • How can mismatched privacy policies be
tions that have promise in enacting positive managed when two different people are
social changes in health and the environment close enough to each other for their sensors
over the next several years. The combination of to collect information from the other party?
large-scale sensor data combined with accurate Furthermore, this type of sensing presents even
models of persuasion could revolutionize how larger societal questions, such as who is respon-
we deal with persistent problems in our lives sible when collected sensor data from these
such as chronic disease management, depression, mobile devices cause financial harm? Stronger
obesity, or even voter participation. techniques for protecting the rights of people as
sensing becomes more commonplace will be nec-
PRIVACY essary.
Respecting the privacy of the user is perhaps the
most fundamental responsibly of a phone sens-
ing system. People are understandably sensitive
CONCLUSION
about how sensor data is captured and used, This article discusses the current state of the art
especially if the data reveals a user’s location, and open challenges in the emerging field of
speech, or potentially sensitive images. Although mobile phone sensing. The primary obstacle to
there are existing approaches that can help with this new field is not a lack of infrastructure; mil-
these problems (e.g., cryptography, privacy-pre- lions of people already carry phones with rich
serving data mining), they are often insufficient sensing capabilities. Rather, the technical barri-
[34]. For instance, how can the user temporarily ers are related to performing privacy-sensitive
pause the collection of sensor data without caus- and resource-sensitive reasoning with noisy data
ing a suspicious gap in the data stream that and noisy labels, and providing useful and effec-
would be noticeable to anyone (e.g., family or tive feedback to users. Once these technical bar-
friends) with whom they regularly share data? riers are overcome, this nascent field will
In personal sensing applications processing advance quickly, acting as a disruptive technolo-
data locally may provide privacy advantages com- gy across many domains including social net-
pared to using remote more powerful servers. working, health, and energy. Mobile phone
SoundSense [11] adopts this strategy: all the audio sensing systems will ultimately provide both
data is processed on the phone, and raw audio is micro- and macroscopic views of cities, commu-
never stored. Similarly, the UbiFit Garden [1] nities, and individuals, and help improve how
application processes all data locally on the device. society functions as a whole.
IEEE Communications Magazine • September 2010 149You can also read
