ARE WE THERE YET? THOUGHTS ON IN-CAR HMI - Version 1.0 July 2014 Produced by ustwo @ustwo @autoustwo
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Version 1.0
ARE WE THERE YET? July 2014
Produced by ustwo
THOUGHTS ON IN-CAR HMI @ustwo
@autoustwo
FIVE PART BLOG
With the ongoing integration of advanced computer systems into We touched upon interaction patterns (covering issues with muscle
vehicles, the interfaces within are becoming ever-more complex. memory and regulations), took a qualitative look at current on-screen
interaction and visual design, briefly gazed into what the near future
As the prime points of interaction between driver and car, these might look like (we see haptic and aural feedback), and finally said a
‘human–machine interfaces’ (HMI) demand great care and attention few words about our own approach to in-car HMI design here at ustwo
from manufacturers, designers and engineers alike. studio.
ustwo wrote a blog series focussing on this subject, posting one part This eBook consolidates the series into once designed, considered
each day for five days. In this five part thought series we looked into this document which is optimised for both desktop and iPad viewing.
phenomenon of growing in-car HMI complexity. We explored our belief
that an interface should match the evocative nature and elegance of From all of us at ustwo, we hope you enjoy it.
an automobile’s exterior and interior design. We also outlined our key
thoughts on how, in partnership with manufacturers, we can bring an
extra layer of care into the design of compelling experiences, in order
to tame the beast that is in-car HMI.
Are we there yet? Thoughts on in-car HMI 02
THE AUTHORS
Tim Smith Tim is a design specialist at ustwo
with ten years of visual design
His client experience includes
Sony PlayStation, Amex and Evian.
experience and 14 awards and Tim’s current area channelling his
Visual Design honours to his name (although OCD is fitness and diet, where he
& UI perspective 13 of those awards were made is undergoing a regime so strict,
up by himself). that he forbids himself to be within
@mypoorbrain 50 metres of any unhealthy food.
Tim is known for his illustration
work and ability to expertly
translate brand into interactive
digital experience designs.
Are we there yet? Thoughts on in-car HMI 03
Harsha Vardhan
Interaction &
UX perspective
@wabisabifiction
Harsha is an interaction and Between 2012-13, Harsha
service designer with a background worked at Service Innovation
in industrial design and engineering. Labs in Berlin, solving urban
He is a graduate from the mobility issues (which included
Copenhagen Institute of Interaction parking and car-sharing) for
design (CIID) and the Indian two of Germany’s largest car
Institute of Technology (IIT), Delhi. manufacturers. Since joining
ustwo in late 2013, he has been
During his time at the CIID saving up for a BMW F800 on
consultancy he worked with ustwo’s save and match scheme.
Toyota’s Kansei division.
Are we there yet? Thoughts on in-car HMI 04
Barnaby Malet
Developer
perspective
@barnabymalet
Barnaby started building for After 12 months he realised that
the web in 2006 and hasn’t it wasn’t for him and headed to
stopped since. He graduated from startup land. He now spends his
Imperial College with a masters in time writing JavaScript, Ruby and
computer science before pursuing Objective C. He has a passion
a PhD in distributed systems. for cutting edge technology and
product development driven by
solid user feedback and analytics.
At social events he avoids bottled
lager and instead prefers to go
home to get an early night.
Are we there yet? Thoughts on in-car HMI 05
David Mingay David has over 15 years
experience in digital design,
Before long, the interweb
appeared and David became
spanning a wealth of industry involved in all things web
Process & petrol head verticals, engagement types related, including branding and
perspective and user touch-points. After user experience design, Flash
graduating from Northumbria programming and animation. Now
@dmingay University with a design for at ustwo, he prides himself in
industry degree, he immediately being the 3rd oldest person in the
jumped into the digital world of company, with the greyest beard.
set-top boxes, CD-ROMS and
kiosk design.
Are we there yet? Thoughts on in-car HMI 06
CONTENTS
08 39
INTRODUCTION: LOOKING AHEAD:
THE NEAR FUTURE DESIGNING FOR THE
OF IN-CAR HMI IN-CAR HMI
12 64
LOOKING BACK: OUR EXPERIENCE:
WHERE WE ARE NOW A NEW APPROACH
& WHERE WE WERE
25 86
LOOKING AT THE NOW: CONCLUSION:
CHANGING PATTERNS WHAT LIES AHEAD?
IN HMI
Are we there yet? Thoughts on in-car HMI 07
INTRODUCTION THE NEAR FUTURE OF IN-CAR HMI Are we there yet? Thoughts on in-car HMI 08
INTRODUCTION
Which consumer technology market has changed and advanced
most in the past decade? It has to be the mobile phone sector. “ Cars are the biggest
and oldest mobile
Thanks to game-changing user experiences delivered through devices... the face of
touchscreens the market has rapidly evolved, giving rise to tablets, mobility. We’ve [Ford]
phablets and the like. Along with advances in chip, display and web- been around for over a
based technologies, mobile devices have become more and more century. But we welcome
affordable, achieving unprecedented mass adoption. It seems only
natural that they have found their way into the cars we drive today,
the competition from
replacing physical buttons and knobs with pixels. newcomers like Apple
and Samsung.
There is much potential for in-car HMI, but we have yet to see a similar
revolution in the UX and UI of the automotive industry. Paraphrasing John Ellis (Head of Ford’s
developer program) at CES 2013
A recent survey by software supplier Cisco Systems suggests that the
automotive industry is not meeting consumer demand in respect of
technology. In order to bring in-car HMI up to speed, the industry needs
to gear up.
Are we there yet? Thoughts on in-car HMI 09
HTC car stereo clip
Pervasive and distributed computing, affordable and accurate sensors
for measuring physiological and mechanical systems, and the notion
of networked objects (the Internet of Things) all play a part — and all
are available through smartphones.
And therein lies the challenge: as people become ever more dependent
on their smart devices, they begin to expect more of embedded technology
elsewhere in their lives. Sadly, these expectations are not being met by
the current generation of in-car HMI.
Furthermore, patterns of ownership have changed. The young urban
populace is moving away from car ownership and towards ‘pay as you go’
or car-sharing schemes, throwing up all kinds of new design considerations.
However, in the haste to get on-trend, car manufacturers have simply
used screens to replicate what has been before, rather than taking
an empathetic, intelligent approach. Skeuomorphism abounds,
where physical buttons are replaced with lookalikes on a screen —
Are we there yet? Thoughts on in-car HMI 10familiarity is retained, but at the expense of tactile feedback. Current Tesla vs the Boeing 757-300:
touchscreen HMIs are often simply ill-considered re-appropriated solutions a comparison where screens abound,
developed for completely different contexts (which we will discuss later). but in the aircraft cockpit, physicality
remains. This says a lot about the
‘learnability of purely screen-based
We believe that there is a need to approach this topic anew, in light controls such as those seen in the
of these challenges and others we will highlight later on in this series. Tesla’s centre console.
ustwo is a design studio specialising in the creation of digital products
and services across multiple technological platforms. When it comes to
in-car HMI, we’ve been learning and adapting as we go — finding our feet
and getting stuck in. Our experience in meaningfully connecting people
and technology, through the rapid prototyping and testing of conceptual
ideas gives us a solid base from which to challenge the status quo.
In this series we focus on recognising the issues, exploring the opportunities
and offering solutions for in-car HMI. We also describe the approach
we took in developing our own concepts for the in-car experience when
working with one of the world’s leading automotive manufacturers.
Are we there yet? Thoughts on in-car HMI 11LOOKING BACK: WHERE WE ARE NOW & WHERE WE WERE Are we there yet? Thoughts on in-car HMI 12
ORGANISATIONAL
& LEGACY ISSUES
Photo credit :
thecarscoop.blogspot.com
The velocity of growth in digital large scale manufacturing and
technology and its adoption has materials sciences).
not been matched by automotive
manufacturers. Another roadblock is the siloed
team structure within manufacturing
Although expert in the design industries, where engineering is
and build of highly functional and often isolated from interior UI and
emotional hardware, automotive interaction design teams.
manufacturers often ignore the
critical interactions and opportunities When working with organisations
between car hardware and HMI. that have this structure, we’ve
experienced a narrowing approach
This is generally due to home- to problem solving and opportunities.
grown complexities — long cycles As a result, cutting-edge technology
of engineering and manufacture is not used to its best advantage
continually put them behind the and the basic theories underlying
curve of computational evolution cognition and human–machine
(Moore’s Law does not apply to interaction are not fully applied.
Are we there yet? Thoughts on in-car HMI 13Samsung “smart fridge”
with embedded touchscreen
“ And today we are
adding the digital user
interface. The two large
monitors in the new
S-Class, each measuring
12.3 inches, currently
represent a benchmark.
Designer interview on the Mercedes
Benz website
Even when teams do work together, This approach, although effective,
it is often too late to share disruptive does not suit concepts that
product and experience ideas. require a high degree of user
This is evidenced by the ‘stick a interaction with new technology.
touchscreen on it’ phenomenon, This is because in order to get
seen on many appliances and these experiences right, you need
in-car interfaces. to develop the user experience
with real users, in near-real
The motor industry also tends to situations. We believe that a
rely on a ‘couture’ methodology methodology is needed that
for car design, whereby very high allows concept, user experience
level concepts are brought to life, and user interface ideas to be
with the aim of wowing motor developed in a collaborative and
show attendees. In order to get effective manner. We will cover
some semblance of the concept this in detail later in this series.
into an actual production vehicle,
designers have to ‘over-egg’ the
idea in anticipation of its dilution.
Are we there yet? Thoughts on in-car HMI 14COGNITION & MUSCLE
MEMORY ISSUES
1922 Austin Seven
The nature of driving a car has What has been changing significantly
not changed since the Austin 7 is the integration of electronics
of the early 1920’s. and, more recently computers,
into the HMI (e.g. GPS, telematics,
The archetypical automobile, ADAS, and infotainment systems).
with its specific driving position, These elements have introduced
steering and control scheme, new layers of complexity to
gauges and even its reliance on interactivity, completely changing
the internal combustion engine cognitive models and expectations.
has not really taken any major
evolutionary steps in the last
80 years.
This dogged consistency has
helped the automotive industry
over the years — a user needs only
to learn to drive one car in order to
drive any car.
Are we there yet? Thoughts on in-car HMI 15Consider the interior of a Mercedes SL Controls: tools or devices which Mapping: developing a ‘feel’ Left: Mercedes SL 1970s
from the 1970s compared with a offer control of in-car functions for controls — the ability to Right: Mercedes SL 2013
present day Mercedes SL. (e.g. a knob on a music player). understand what a control does
Affordance: the nature of and where it is located
In a typical interaction design manipulation a control offers, (an accomplished pianist can play
fashion, we can break down the while performing an action the instrument blind, by ‘feel’ or
analysis of the past and present (e.g. a knob can be turned about muscle memory).
HMI by looking at: on an axis).
Learnability: the ability to
Feedback: the change or reaction understand the way a control
brought about by the controller behaves over time (e.g. turn the
(e.g. the change in volume when knob clockwise to increase volume).
the knob is turned).
Modes: the number of ways a
tool or device can be used and
repurposed by switching to a new
function (e.g. the same knob can
also be used to control brightness).
Are we there yet? Thoughts on in-car HMI 161970 2013
Knobs, switches, sliders offer Multi-modal screen with GUI
direct control / manipulation Indirect control via knobs & switches
In the car interiors of yesteryear, Learnability was always a factor,
the spatial arrangement of the dials but a sparse set of controls meant
and knobs allowed for a mental an inherent simplicity.
map of the HMI to be developed
over time and in turn built into With modern HMI it’s a completely
muscle memory. different story. There are many more
controllable elements within cars,
Since there were fewer electrical numbering in the hundreds —
complexities, mechanical elements navigation systems, telematics,
such as knobs, switches and ride control and infotainment
sliders provided both control systems to name but a few.
and direct mechanical feedback
— a tangible user interface. There
were far fewer modes, because
there was direct communication The spatial arrangement
between the elements on the car provides for a mental map
and the controls. of the controls over time Knob Button Slider
Are we there yet? Thoughts on in-car HMI 17Single function controls Control knob turns through, selects Touchscreen
Calls, seat / drive position modes and operates the GUI Direct manipulation of GUI
Nowadays, there is a mix of both This adds a fivefold complexity in 3. Affordances: how does moving 5. Consistency and muscle
tangible and graphical user interfaces cognition: a circular knob relate to movement memory: if a person changes car
(GUI), with the GUI behaving as the between modes on the screen? models or even manufacturers,
primary source of feedback with 1. Shifting between modes: Can we bring back the direct control they have to relearn some of the
multiple modes for different in-car (and learning where they are) does seen in the cars of yesteryear? basic controls from scratch.
functions and systems. Elements not allow for a single mental map
within the GUI can be controlled to be built up over time. By way of 4. Feedback: at present there is a With the advent of these multi-
either indirectly via mouse-like an example, see the video for how reliance on visual feedback in the modal HMIs, people are now faced
devices or more recently, by to move between sat-nav and seat GUI which can be very distracting. with an unprecedented level of
manipulating elements via touch. position control. Complimenting visual feedback complexity, as well as the added
with aural and/or haptic feedback pressures of congested driving.
This short clip shows how the 2. Ease of Mapping or Learnability might be the answer. (Haptic This plays a huge role in decision
control knob indirectly controls of the GUI: Where are the controls technologies are tactile feedback making due to the limited nature
modes and elements within the and what are their functions? systems that take advantage of of data storage and access in the
GUI of the Mercedes SL 2013. Can they be learnt easily? the sensation of touch). human mind.
Are we there yet? Thoughts on in-car HMI 18LIMITATIONS OF COGNITION &
‘THE WORKING MEMORY MODEL’
The ‘Working Memory Model’ put forward
by Baddeley & Hitch in 2000 (first in 1974)
To understand the limitations of and forms, and muscle memory,
cognition, we first need to look whereas fluid memory tends more
into how people consciously to the visual and auditory. CENTRAL
engage with the world through EXECUTIVE
the construction of long-term For further information, see these
and short-term memories and related papers and talks.
their retrieval over time. Visuospacial EPSIODIC Phonological
Sketchpad BUFFER Loop
The ‘Working Memory Model’ put
forward by Baddeley & Hitch in
2000 (first in 1974) describes the
interplay between a ‘crystallized’ Visual Semantics Language Episodic LTM
(shapes - forms) (Muscle Memory)
long-term memory system, a
‘fluid’ short-term memory system
and a limited capacity ‘episodic
buffer’. The crystallised long- Crystallized systems Fluid systems for short
for long term memory term & conscious tasks
term memory system comprises
language, knowledge of shapes
Are we there yet? Thoughts on in-car HMI 19frustration on the part of the driver The Gestalt principle of ‘Proximity’-
due to cognitive overload. notice the grouping of elements in a,
a.
b & c, based on how close they are in
relation to each other
Over the years human–computer
interaction studies have postulated
several key methods for tackling
b.
the problem of cognitive overload.
One such method is the concept of
‘Chunking’ as proposed by George
Miller in 1956, whereby an individual
c. can remember or process only seven
chunks of information, in their correct
serial-order, in his working memory.
This essentially means that grouped
“ The Episodic buffer is
assumed to be a limited
One can consider the act of driving
to be crystallised, since it uses long-
term muscle memory, whereas the
items are easier to recall, because
grouping assists phonological and
visuospatial memory.
capacity temporary information consumed around that
storage system capable task of driving — Who’s walking
of integrating information in front of the car? Where am I +445232367456
driving to? What am I listening to? Linear
from a variety of sources.
— can be considered to be part of
Allen Baddely (2000) from a conscious experience and in the +44 (523) 2367 456
‘The Episodic buffer: A new episodic buffer at any particular time. Chunked
component of working memory’
The episodic buffer has a limited The concept of chunking also seems
capacity for taking in visual, auditory to apply to visual forms , similarly
and motor cues, so a HMI which does grouped shapes or patterns are also
not allow for the easy construction easier to remember. This is known as
of skill or muscle memory can the Gestalt principle of proximity.
easily overtax it. This in turn leads Further evidence for this concept
to bad decision-making and is suggested in studies here.
Are we there yet? Thoughts on in-car HMI 20FRUSTRATION, CONFUSION
& SAFETY ISSUES
The complexities of in-car HMI,
combined with the limitations
of human memory may result in
“ The system should
be designed to support “ The system should be
designed in such a way “ The system should
be designed so as not
cognitive overload for drivers. the driver and should not that the allocation of driver to distract or visually
give rise to potentially attention to the system entertain the driver.
As a result, minimising distraction hazardous behaviour displays or controls
European statement of Principles on
and reducing driver error has been by the driver or other remain compatible with in-vehicle HMI, Commission of the
the focus of law and policy makers
worldwide. Efforts are being made
road users. the attentional demand European Communities
to curb the use of distracting of the driving situation.
devices (eg mobile interfaces)
which in turn also influences the
design of HMI.
For instance, the European statement
of principles on in-vehicle HMI
issued by the Commission of the
European Communities states:
Are we there yet? Thoughts on in-car HMI 21“ The driver should be
“ The distraction
“ Any task performed
Or the NHSTA guidelines (USA),
a summary of which is present
able to keep at least one induced by any secondary by a driver should be
here. The guidelines are based
upon a number of fundamental hand on the steering task performed while interruptible at any time.
principles: wheel while performing driving should not exceed
a secondary task that associated with a
(both driving-related baseline reference task
and non-driving related). (manual radio tuning).
Are we there yet? Thoughts on in-car HMI 22“ The driver, not the
system / device, should “ Displays should be
easy for the driver to see
So, it is fair to say that many of the
issues we currently face in HMI
design are borne of the legacy
This leads on to learnability issues,
not to mention frustration and the
obvious safety concerns as put
control the pace of task and content presented of the space and the automotive forward by regulatory agencies.
interactions. should be easily industry infrastructure. The siloed Though manufacturers are definitely
discernible. nature of design teams means aware of these issues, it is our
that while new functions may be view that many of them could
NHTSA Guidelines introduced, organisational structure be mitigated early and rapidly,
might not allow for the sensitive if a “one-team” collaborative
incorporation of these features. approach was adopted, where
they can design, build and test
ideas with real users to learn and
iterate accordingly.
Are we there yet? Thoughts on in-car HMI 231969 Bertone BMW Spicup Concept
LOOKING AT THE NOW: CHANGING PATTERNS IN HMI Are we there yet? Thoughts on in-car HMI 25
CONNECTED CARS:
A LONG TERM SOLUTION?
Tim Smith Harsha Vardhan
Design Specialist Interaction Designer
Smartcar provides climate control
both within the Tesla S and the home
In the past year we’ve seen Apple, As we see it, there are three
Microsoft and Google enter the models at present:
in-car space with their own
custom HMI concepts. Model 1 : The radically
connected car
The media has focused on the Such as the Tesla S, where the
potential for new user experiences hardware, software and HMI is
within the connected car and the all custom-built; integrated from
likely emergence of a strong in-car the ground up. This model allows
app market. the deep integration of not only
the multimedia systems, but also
Take a look at Apple’s ‘Carplay‘ diagnostics and control systems
and ‘Android Auto’. inside the car with the driver.
Connectivity (see these tie-ups
The notion of a connected car with network service providers)
has a significant impact on and an API also allow interesting
the way HMI is conceived and custom apps and services to be
implemented. built by independent developers.
Are we there yet? Thoughts on in-car HMI 26Model 3: Integration of same-
Top: Apple CarPlay platform devices to onboard HMI
Middle: Google Open Automotive Alliance
as done by Apple and Microsoft.
Bottom: Microsoft Window in-car
While this might mean less
fragmentation of operating
systems, the user is wedded to
one platform both in-car and
in everyday life if they want
everything to integrate.
So which model of connected car
is best?
Model 3, while exciting, is borne
Model 2: Custom HMI modules
that use software platforms like
‘Microsoft Embedded Automotive 7‘
“ Apple, Ford, Android
and others — are targeting
of different motives to Models 1
and 2. Apple, Google, Microsoft et
al are all looking for a gap in the
or Blackberry’s QNX to manage and the car as the next frontier market or, more precisely, an extra
integrate different devices and for mobile development. touch point in which to sell more
sensors in a car into one platform. devices. The design community
For example, Ford used Microsoft’s Adario Strange, Mashable may well be all fired up designing
platform for their relatively long apps for cars, but what are the
running ‘Ford Sync’ HMI systems. potential pitfalls of an app-based
In this model custom interfaces approach?
for drivers are created, which means
that the status quo, fragmentation First and foremost, drivers are
of interfaces and experiences, still beholden to a hardware/software
remains. However, the inherent manufacturer. To make Apple CarPlay
independence allows for the work on the Mercedes Benz C-class,
creation of custom experiences for the driver has to plug in their iPhone.
users while being agnostic to the Similarly, to use ‘Auto’ you’ll need
driver’s mobile device. an Android phone.
Are we there yet? Thoughts on in-car HMI 27Will people start to choose cars platforms into existing cars, than
“ If you have (HMI) ideas Being beholden to a hardware/soft-
based on their handset of choice? to create tailored solutions in a ware manufacturer – Plugging in the
now and you want to bring iPhone to make Apple ‘CarPlay’ work
fragmented market.
Compounding this phenomenon,
it to the vehicle soon, on the Mercedes Benz C-class as
seen in the demo video previously or
car manufacturers seem all too Car manufacturers benefit too — we try to accomplish it
your Android phone with ‘Auto’.
keen to jump into bed with these not only do they save on HMI costs, with this team by taking
initiatives — Honda, Audi, General they also benefit from the fast pace advantage of mobile
Motors and Hyundai, to name but a of technological advancement in
devices and connecting
few, are already members of Google’s the mobile and software space.
Open Automotive Alliance (OAA). Lengthy automotive design process them to the car to bridge
cycles would otherwise hold back that gap.
There’s a “me too” sense of the integration of a contemporary
urgency too; it’s quicker and more HMI, as pointed out by Vera Schmidt, Vera Schmidt, Senior Manager of
Advanced UX Design at Mercedes
scalable for Apple, Google and Senior Manager of Advanced UX
[source]
Microsoft to get their existing Design at Mercedes.
Are we there yet? Thoughts on in-car HMI 28And speed is of the essence — A 2 dollar smartphone mount
these companies, software giants via niftycurly @ Intructables.com
and automotive stalwarts alike,
need to establish their presence
in this field as quickly as possible
if they’re to get ahead of the trend.
In the short term at least, Model 3
may be the quickest and most viable
way of getting a contemporary
UI into a car by providing better
usability and consistency across
car types. It also plays into people’s
behaviour and the ways smartphones
are already being used — for 1. Given the lengthy production
instance using Google Maps cycles and that purchase cycles
instead of traditional Sat-Nav are less than one vehicle per two
via a cunning dash-mount hack years, it will take many years for the
model to become truly pervasive.
But there are obvious drawbacks According to estimates from GSMA
to Model 3, which might hold and others, by 2018 there will
it back: be over 60 million connected cars
on the road globally, driving some
£30 billion ($51 billion) in annual
revenue. That’s just 6% of the
cars on the road, which surpassed
1 billion in 2010 (according to a
study by Ward’s Auto).
Are we there yet? Thoughts on in-car HMI 291 BILLION CARS ON THE ROAD
6% (60 MILLION) CONNECTED
£30 BILLION ANNUAL REVENUE
2. Being beholden to a certain The best ‘iPhone’ experience and mobile devices/software to be truly viable and attain
hardware/software provider, might not translate well for reach obsolescence far sooner. any level of meaningfulness.
might be detrimental to user everyone who does not use it. Would this mean a new paradigm
experience. For instance, demos in replaceable and modular HMI? This leads us on to the radically
of Apple’s CarPlay system speak 3. Updates to mobile devices connected car and custom module
about the use of the iPhone should happen at an equal pace 4. Access to deep telemetric data HMI car, outlined in Model 1 and 2
in a car. It predetermines the to that of the in-car HMI (hardware and control systems of the vehicle respectively, which use a bespoke,
use of a certain kind of device and software). Update patterns in (eg seat positions, remote ignition integrated HMI approach.
and an understanding of the consumer mobile devices cannot control) in an app-based model This could be a better more
idiosyncrasies of that operating carry over into cars, since that could prove difficult with regards meaningful long-term option,
system, which would make sense could present safety issues. Rules to driver and passenger safety and but like Model 3, also has some
only for Apple users. What we need to be communicated to security. Regulations will need to existing issues.
actually need is synchronisation independent app makers who wish change quickly to suit this reality.
across multiple platforms to to develop software for cars. (See Hacking into the Tesla S).
be more inclusive across world Similarly, we need to consider the
markets. (Car makers are looking rapid change in mobile hardware We feel that these are just some
into this issue — Press release in comparison to the car in itself. of the challenges that need to be
from Audi). Cars are meant to last for years overcome in order for this model
Are we there yet? Thoughts on in-car HMI 30BLACK & BLUE
Tim Smith
Design Specialist
In the first and second model,
car manufacturers’ internal
design teams create their own
“ For those (automotive)
manufacturers looking
some potential (discussed later),
but currently is poorly executed.
and unintuitive—the exact opposite
feelings that they should evoke.
And then there’s the fact that
“tailored” HMI (though often the to go it alone, I don’t The problem with current internally they all end up looking the same.
GUI and even physical buttons expect much. designed in-car HMIs is that they
are repurposed from vehicle to either look hard to use or they are HMI UIs have their fair share
Geoff Teehan
vehicle). hard to use (probably the same of practical, legal and safety
thing) — and let’s not even get constraints, of course, which can
The screen and, more recently, the started on Pixel Perfect Precision. make a designer’s job very difficult.
multi-touch screen seem to be the However, it is fair to say that current
default thought on HMI problem They are often very moody and HMIs have no aesthetic beauty,
solving today. dark, with neon blue highlights no identity of their own, no character.
and metallic textures. This creates They are an emotionless,
In his article The State on In-Car UX, another problem: they have an ugly, utilitarian, afterthought;
Geoff Teehan does not exude overtly masculine or sci-fi aesthetic a seemingly out of character
confidence in this approach. which can alienate both women approach for automotive companies
Contrary to Geoff Teehan’s opinion, and men. The UI’s approachability that expend so much effort on
we believe the model itself does have is thus often daunting, intimidating exteriors and interiors.
Are we there yet? Thoughts on in-car HMI 31The current in-car UI
landscape
Are we there yet? Thoughts on in-car HMI 32BRANDING:
A MISSED OPPORTUNITY
Tim Smith
Design Specialist
Ferrari California 2008 HMI
This “black and blue” aesthetic HMI from the Ferrari California
seems to be born out of a separate 2008 feature no red, but instead
design stream to that of the lots of blue (admittedly, the
rest of the car and indeed the selected hardware is restrictive)?
larger brand.
We are not suggesting that HMI
As with almost all companies in should necessarily follow brand
the world, automotive companies books religiously, but we do feel
have strict brand guidelines that there is a wasted opportunity
on which they base all of their here. HMI should take inspiration
communications, from logo from the entirety of the car, not
treatment and typography to just in the choice of colour, but
specific brand colour values. in other aspects of both form
You can imagine that a Ferrari and function – this is the point
brochure would feature lots in which we interact with their
of red, as would a Ferrari website, product after all.
a Ferrari app and so on. Why then,
does the centre console of the
Are we there yet? Thoughts on in-car HMI 33Photo credit:
Otis Blank for Petrolicious
There’s real potential in blurring
the lines between the exterior,
interior and HMI design of a
The use of skeuomorphism in
many car HMIs is primarily based
on a need to solve the lack of
“ They (automotive
manufacturers) should
car, creating a coherent user affordance and feedback offered see it (in-car UI) as a
and brand experience, from the by the screens that have replaced major opportunity to
colours to the look and feel, to buttons (as discussed earlier). bring moments of joy
interactions. With that in mind, However, this skeuomorphism and delight to customers.
neither the blue of the HMI in the seems to bleed into the entirety of
example above, nor the square the interface, which again pushes
These are systems that
shape nor the cold, dry look and any branding into the back seat. allow us to physically
feel, fit in with the warm, sporty interact with their brand.
and exciting interior and exterior
of the car. Geoff Teehan
Are we there yet? Thoughts on in-car HMI 34SKEUOMORPHISM:
IT’S NOT ALL BAD
Tim Smith
Design Specialist
Skeuomorphism
applied to a UI
An aesthetic common in current To explain, let’s take a random
automotive HMI UI’s, and indeed button from a UI. Using stylistic
with many UI’s of the past skeuomorphism you might give
decade, is skeuomorphism. it a glass texture, perhaps some
bevels around the edges, maybe
However, unlike contemporary UIs, even some light reflections and
the in-car HMI has failed to move glare. While this may help it look
with the times and shed the more like a physical object, it’s not
skeuomorphism — perhaps an necessary for the understanding
artefact of the lengthy car production of the function.
cycles we discussed earlier.
However by applying semantic
We’ve always thought skeuomorphism to that same
skeuomorphism was too generic button, you can imply that the
a term though, preferring instead button has depth, which helps
to split this term into two types; communicate that it is pressable.
stylistic skeuomorphism and
semantic skeuomorphism.
Are we there yet? Thoughts on in-car HMI 35“ Affordance is an object’s ability to convey
its function through sensory means, for example
a button suggests that you press it by being
slightly raised; This technique can also be used
in digital design to lead users into interacting
with objects.
Pixel Perfect Precision, ustwo
Attachment ‘paperclip’ icon
from email clients
This simple shadow or edge adds Semantic skeuomorphism Within today’s car dashboard,
an ‘affordance’ to the graphical works with visual metaphors skeuomorphism also manifests
user interface, which is pivotal to communicate a meaning, itself in these two distinct ways.
in helping users find operational much like many icon designs. Just as skeuomorphic UI buttons
cues. (We cover this in more detail For example, the attachment look like analogue buttons of
in our ustwo PPP document for ‘paperclip’ icon in email clients: old, car HMIs are derived from a
best practices in GUI.) it’s not an aesthetic style, nor legacy of analogue or mechanical
is it literally demonstrating the interpretations of pre-digital car
function. It is a metaphor that dashboards.
effectively communicates the
function — which is now so well- The first, stylistic skeuomorphism,
known that is has been adopted presents itself in the form of
as a standard. metallic sheens and bezelled
graphics. This only acts to
compound the masculine or
sci-fi look we discussed earlier.
Are we there yet? Thoughts on in-car HMI 36Toyota GT 86 speedometer
So, it is, in a way, a legacy; the So, the current state of in-car
familiar representation carried HMIs suffers largely from the
over from a time when that emergence of the screen and the
method was the best approach multi-touch screen in the in-car
within technological and environment, or more precisely,
mechanical constraints. the poor appropriation of the
technology. Car manufacturers’
The familiar is a powerful eagerness to use the screen
signifier of meaning. Just as a solution to mapping and
look at the paperclip example learnability problems has meant
discussed earlier. However, that new problems have emerged
learned behaviours and new in the infancy of its adoption.
signifiers are being adopted
all the time, especially in this As this technology continues
You could argue that this aesthetic pin left or right depending on era of the rapidly evolving to advance and the experiences
is informed by the materials used the exertion put upon the engine. smart device, and especially if mature, we may start seeing some
within the car’s interior, or that in Though these mechanics are they’re efficient, a pleasure to changing patterns in in-car HMI
fact the faux leather and wood no longer used, the same method use and easily understandable. that finally make sense.
effects are a form of skeuomorphism to communicate velocity is The transformation from the
themselves. Regardless, the styling, still adopted in nearly all cars. skeuomorphism of iOS6 to the flat,
the blurring of the line between Is a dial pointing at a number motion oriented design of iOS7 is
the design of the car itself and really the most effective way of an obvious example.
that of the UI is something we communicating a driving speed?
find interesting.
There is an argument that it
The second, semantic signifies a relative position — akin
skeuomorphism, is apparent to ‘How many minutes to 2 o’clock?’
in the familiar speedometer. as seen with traditional watch-faces.
But there are new and more relevant
The early speedometers worked measures for speedometers now,
directly from the engine: a cable such as ‘How close you are to the
connecting the two pulled the speed limit?’
Are we there yet? Thoughts on in-car HMI 37Photo credit: Otis Blank for Petrolicious
LOOKING AHEAD: DESIGNING FOR THE IN-CAR HMI Are we there yet? Thoughts on in-car HMI 39
HARD & SOFT
INTERACTIONS
Harsha Vardhan
Interaction Designer
Before we discuss our thoughts
on the best approach for in-car
interactions, we will touch upon
Soft interactions can be defined
as the actions performed by the
machine as non-deliberate inputs
“ A machine is beautiful
when it’s legible, when
the types of interactions that provided by the user. its form describes how
exist in the in-car environment. it works. It isn’t simply
Self-cancelling turn signals a matter of covering the
In-car interactions can be split into are an example of a soft technical components
two distinct types: hard and soft. interaction — where the machine
autocompletes a sequence of
with an outer skin.
Hard interactions can be defined actions without any user input. Konstantin Grcic (2007) via
as deliberate manipulative elasticspace.com
actions performed by the driver. The latter type of interaction
Examples are: changing the drive especially is coming to
position using a button, using an prominence with the advent of
infotainment system via a GUI or embedded interior sensors and
inputting location data into the the notion of the connected car.
sat-nav.
Are we there yet? Thoughts on in-car HMI 40Hard and soft
interactions
A. Haptic controls with B. Touchscreens with
embedded touch controls possible Haptic feedback
Some of the possibilities have We believe that combination
been exploited with contextual of meaningful hard and soft
information displayed in HUDs interactions is the key to getting
(Heads-Up Displays), auto the best out of HMI in a car.
C. Gesture control with D. Voice control
dimming of interior lighting, visual & aural feedback loops & feedback
and even experimental tracking In the sketch (right) we have
of closed eyelids. As an aside, laid out what could be the set
we feel soft interactions require of possible interaction paradigms.
the greatest amount of care and This outlines some of our
appropriateness in execution own research into near-future
since there is a thin line between interaction possibilities using Curved road
being assistive and in being a current technology. ahead
distraction.
100m
We will now delve into each of
the above interaction paradigms.
E. Soft interactions aided F. Contextual information on
by computer vision secondary displays (e.g. HUD’s)
Are we there yet? Thoughts on in-car HMI 41The Hype Cycle for emerging Less than 2 years
technologies Source: Gartner
(July 2013) 2 to 5 years
5 to 10 years
10+ years
Gamification
Consumer 3D Printing
Wearable User Interfaces
Big Data Complex-Event Processing
Natural Language Question Answering
Content Analytics
Internet of Things
Speech-to-Speech Translation In-Memory Database
Management Systems
Mobile Robots
Virtual Assistants
3D Scanners
Neurobusiness
Biochips
Autonomous Vehicles
EXPECTATIONS
Speech
Recognition
Prescriptive Analytics Augmented Reality
Machine-to-Machine Communication Services Consumer Predictive
Affective Computing Mobile Health Monitoring Telematics Location Analytics
Electrovibration NFC Intelligence
Volumetric & Holographic Displays Biometric Authentication
Human Augmentation Mesh Network Methods
Sensor
Brain-Computing Interface
3D Bioprinting
Cloud Computing Enterprise 3D Printing
Quantified Self Activity Streams
Gesture Control
Quantum
Computing In-Memory
Virtual Analytics
Reality
Smart Dust
Bioacoustic
Sensing
INNOVATIONS PEAK OF TROUGH OF SLOPE OF PLATEAU OF
TRIGGER INFLATED DISILLUSIONMENTS ENLIGHTENMENTS PRODUCTIVITY
EXPCT’NS
TIME
Are we there yet? Thoughts on in-car HMI 42A. Haptic controllers with Early BMW i-Drive Touch, late 2000’s. BMW went on to introduce the Improved BMW i-Drive Touch in 2013
embedded touch surfaces: Control knob next to the shifter improved i-Drive Touch in 2013 to
hybrid interfaces and screen alleviate some of these issues by
introducing a touch interface on
Let’s take a closer look at the the control knob itself (as shown
i-Drive controller that BMW in the video).
employed in the late 2000’s.
This form of hybrid interaction
In this video, note the issues with presents a significant
modes — affordance and mapping improvement because it allows
a circular motion into a linear more active and tangible control
output on the screen. of on-screen GUI.
Are we there yet? Thoughts on in-car HMI 43B. Touch Screens with An interesting set of experiments Left: Porsche 918 HMI
Haptic feedback being carried out at Disney Research
points to the way forward, where Middle: Tactile Rendering of 3D Features
on Touch Surfaces by Disney Reserach
Touch screens are being put forward tactile rendering algorithms are
as the sole modes of control in used to simulate rich 3D geometric Right: An array of 1D Gaussian bumps
automotive HMI, as demonstrated features (such as bumps, ridges, are rendered to create DVDs, books and
in the large-screen iterations in the edges, protrusions and texture) other stack of things. Disney Research
Porsche 918 and the Tesla Model S. on touchscreen surfaces.
Although they appear to offer a If applied meaningfully, this could
simple alternative, they are in allow a muscle memory or ‘feel’
fact problematic with respect to for controls to develop over time.
learnability, as discussed earlier.
They can also be very distracting, This technology can be seen
because the driver has to rely on on the Disney Research website.
visual feedback all the time and
cannot form a muscle memory or
map of the controls over time.
Are we there yet? Thoughts on in-car HMI 44C. Gesture control with visual 3D gesture control as a concept is There are literally hundreds of 3D A notion taking root in people’s minds,
and aural feedback loops also taking root in people’s minds, gestures possible and it takes thanks to these guys
because of gadgets like the Leap time to learn and understand a
Using gestures to control certain Motion and Kinect controllers. set pattern and thus in its present
aspects of HMIs is an exciting state cannot be relied on as a pure
concept. This is primarily because We can detect not just macro interaction — especially with
it presents an opportunity to changes in physical characteristics, regards to safety.
bring back the direct control and like nodding, facial position and
feedback which existed in early hand gestures, but also micro This was indeed a key issue in our
cars, although it is not without changes like eye movements. initial experiments using both the
problems. However, the new interaction Leap Motion controller and the
patterns emerging from low cost Kinect as primary modes of in-car
The sensing of 3D gestural data is computer vision have not been control. We found, as with any new
getting progressively easier, not fully cataloged and understood, control, gestural interaction is not
only because of low cost sensors which poses a challenge when necessarily intuitive.
and processors, but also as better mapping and learning a gestural
algorithms become available. interface.
Are we there yet? Thoughts on in-car HMI 45The rich feedback of physical A concept using the Leap motion con- This issue with fine grained control The AIREAL device by Disney Reserach
interactions — clicking buttons, troller as shown below from Denso and has been the focus of research emits a ring of air called a vortex to-
the movement of levers, gears Chaoticmoon institutions over the last few years wards a user’s hand. The vortex can im-
part a force on the user’s hand, enabling
falling into place — has not and we find this exploration by
a range of dynamic free air sensations.
translated well into the fuzzy Disney research to be amongst the
digital space. ‘Minority Report’ interesting ones—‘Aireal: Interactive
style interfaces remain a fallacy tactile experiences in free air’.
(The fallacy of a ‘Minority Report’
style interface). Here they prototype a new low cost,
highly scalable haptic technology
Also watch a concept using the that delivers expressive tactile
Leap motion controller from Denso sensations in mid air as part of
and Chaoticmoon. There are no their long term vision for creating
buttons — only visual feedback is large-scale augmented environments
employed — which introduces the which can deliver compelling
problem of fine grained control interactive experiences
and learnability. everywhere and at anytime.
Are we there yet? Thoughts on in-car HMI 46D. Voice control and feedback Siri and Google Now in mobile OS’s HAL 9000 from 2001: A Space Odyssey
have also been playing a strong
Voice based interfaces have occupied role with in-car interactivity with
imagination ever since the pop companies such as Nuance
culture exposure to the eponymous supplying their software expertise
HAL 9000 and more recently in the to manufacturers such as Ford —
movie ‘Her’. Though we are far from seen in their Sync range of HMI.
achieving human-like conversations
with machines, due to continuous The promise of voice control lies
advances in natural language with two factors, one in replacing
processing and recognition, the last physical and digital controls moving
few years have seen a number of into the land of no UI, where one
high-fidelity consumer applications can freely converse with HMI and
seeing the light of day (in essence secondly minimising the distractions
this is a form of AI though some which come from the manual
people might argue that it is not — operation of HMI, targeting
a strong case of the ‘AI effect’). increased safety.
Are we there yet? Thoughts on in-car HMI 47Vocal interaction design is a new control while changing volume, Voice input ‘Siri’ available in
challenge. It is easy to say that voice which operates over a range e.g. Apple CarPlay
could be a no-brainer in terms of ‘Increase volume… make it higher…
next generation user interfaces, higher…’, as it operates as an
but we need to critically understand abstract, analogue, inexact notion.
implications before designing for
the same. In our research we find We then get into the fuzzy area of
the following factors (among many) allowing a user to set presets so
to be quite important to consider; that a computer understands what
he/she is trying to achieve or time
1. Discrete and continuous control: based learning where the computer
This is the difference between the on understands intention by gauging
and off states of a button and the past interactions e.g. ‘higher’ can
continuous rotation of a knob. Voice mean increase by 20%. This fuzziness
can play a large role in functioning could lead to increased confusion
as an effective discrete control e.g. and frustration if not dealt with
‘turn on radio’ or ‘radio’, but may not carefully. A study into voice
be as effective as a continuous interaction and distraction.
Are we there yet? Thoughts on in-car HMI 482. The problem with ‘Strings’ operations and the subjects of the
“ The destination entry
Left: Driver frustration in
and ‘lists’: study rated these parts of voice research carried out at the
MIT AGELAB and the New task was the most time
There is a challenge in dealing with interfaces to be as demanding as
the input of strings of sentences using knobs and buttons.
England Transportation Center consuming, requiring an
(alphanumeric data entry e.g Sat- average of 111 seconds
Right: Voice input available
Nav) and cognitive load it poses The complexities in the in Android Auto to complete in the first two
on a driver. interface design arise from many studies. Task completion
multimodal demands posed by
Though one would think this is the technology. Among them
time was not a matter
where voice input could be ideal, are having to remember lists of problematic speech
by eliminating the need to enter of information as spelt out by recognition... it was a
text via a keypad, studies point the interface. One of them is a matter of interface design.
to the contrary. Research carried behavior called the ‘orienting
out at the MIT AGELAB and the New response’ — which often took Seeing Voices, MIT
England Transportation Center, the form of subtle, seemingly
point out that the distraction and unconscious shifts in posture as
engagement levels of voice are the driver spoke to the HMI. A case
comparable to that of manual of personification of technology.
Are we there yet? Thoughts on in-car HMI 49A way to effectively deal with 3. Recognising emotion in voice:
“ Designers must remain
Voice input available in Ford Sync
the above issues as found by the This seems to be the next step 2015 Ford Mustang
aware that, although they
study is by offering appropriate in Natural Language processing
confirmations — both visual and — where mood and emotion are may be targetting the ear
aural. Treating a person as a whole triggers for in-car reactions. and the voice, what they
rather than just focussing on Approaches being taken by are really dealing with is
targeting the ear and voice. Google and Nuance. the entire person.
The orienting response often relies Seeing Voices, MIT
on visual feedback to the verbal
input on the driver’s part. For
instance, Apple with it’s CarPlay
has tried to address the issue
by deactivating the UI whenever
possible. But the implications of
these modes of automatic behavior
on part of the interface have not
been studied in detail as yet.
Are we there yet? Thoughts on in-car HMI 50E. Soft interactions aided Infrequent movement can signify The Kinect One and ‘Eulerian video Left: Capabilities in the
by Computer Vision a tired driver and thus a car might magnification’ have both been new Kinect to detect micro-
prompt the driver to take a break used to non-invasively measure fluctuations in physiological data
The ability of cameras to track or offer the driver directions to the heart rate (BPM) in a research
Right: Enabling cameras to
micro-movements in pixel data nearest motorway services. setting. Happily there is already measure BPM - Eulerian Video
allows sensors in the car’s interior an element of consumer trust magnification - MIT , Quanta
to detect a driver’s physiological Asynchronous reactions are time- when it comes to such personal research center Cambridge
data. This can produce both based. For example, tracking a metrics — a survey carried out
synchronous (real-time) and driver’s heart rate over a journey and by Cisco Systems revealed that
asynchronous reactions presenting them with hot spots 60% of car owners would be
(with a deliberate time delay). where there are data spikes. Much willing to share biometrics such
like how a car’s fuel consumption as fingerprints and even DNA
By synchronous reactions over a journey could be mapped samples if it would improve car
we mean immediate and real- and studied, we can study and security.
time reactions to changes in learn from physiological data.
physiology, like the movement
of a driver’s eyelids or reactions
to gaze detection.
Are we there yet? Thoughts on in-car HMI 51Curved road
ahead
100m
Eye tracker
1. 2.
3.
F. Contextual information on Temporal reactions. This information can also be Spatial reactions.
secondary displays (e.g. HUD’s) Turn-by-turn navigation displayed where the driver is An embedded eye tracker
on a HUD only when GPS looking, via gaze detection on the dashboard activates
is activated screens the driver is looking
We could break the visual emphasis techniques.
at only, minimising distraction
towards a central console and
provide displays for the driver What technological implementations
based on when data is required have we seen so far?
(temporal, left) and where it is
required (spatial, right). The first main implementation
is the use of secondary displays
For instance, information can in cars, like HUDs, for providing
be broken down into a number of information on or near the driver’s
displays, to provide turn-by-turn line of sight have been in use
navigation data when a driver since the late 1980s.
requires it, perhaps using HUDs.
Are we there yet? Thoughts on in-car HMI 52The Land Rover Discovery invisible Land Rover Discover: Invisible In a similar vein, the BMW ‘Vision BMW Vision Future Luxury
bonnet concept is a more recent bonnet concept Future Luxury’ speaks about
idea, where a combination of the ‘contact-analogue’ HUD for
contextual on-road information the driver which augments the
and actual off-road imagery from real-world view by projecting
grille cameras is viewed through information directly within the line
a HUD. Digital immersion through of sight. Buildings, traffic signs
the use of cameras is something and hazards can be highlighted
we expect to see more and more directly in the real-world
in in-car HMI, used mainly by environment, selectively directing
augmented reality. the driver’s attention to specific
information.
Are we there yet? Thoughts on in-car HMI 53Then we have gaze detection or Eye tracking technology It’s early days yet, but quite a few Caterpillar and Seeing
eye-tracking displays, where specific software companies are working to integrate machines collaboration
portions of the GUI become active trackers into driver assistance
depending on where the driver systems, tackling issues like
or passenger is looking (spatial driver fatigue, especially for
reactions). This has the potential large commercial vehicles.
for minimising distractions and,
coupled with the temporal reactions, For instance a Caterpillar and
can be quite powerful. Seeing Machines collaboration.
These systems are built using a
combination of software (face and
gaze tracking algorithms) and
hardware (cameras and processing
units to integrate with on-board
assistive systems).
Are we there yet? Thoughts on in-car HMI 54In a similar vein, Tobii has eye
“ The integration of eye Tobii eye tracking
tracking systems which they enabled game
tracking into the game
have experimented with in cars
and games. experience is literally a
game changer – not only
for the gamers themselves
but developers.
Bruce Hawver, CEO of SteelSeries
Are we there yet? Thoughts on in-car HMI 55You can also read
