ARTICLE Beyond Happy-or-Not: Using Emoji to Capture Visitors' Emotional Experience
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
ARTICLE
Beyond Happy-or-Not: Using Emoji to Capture Visitors’ Emotional
Experience
DANIELA DE ANGELI , RYAN M. KELLY, AND EAMONN O’NEILL
Abstract Museums are emotionally driven sites. People visit museums to feel and their emotions
influence how the museum and its artefacts are perceived. Thus, evaluating emotional states are
increasingly important for museums. However, evaluating visitors’ experiences is increasingly challenging,
especially with the introduction of new and emerging technology. Moreover, people’s behaviour is not
strictly objective and rational. While emotional states are subjective and hard to verbalize or observe, emoji
are often used to express emotions on mobile and smartphone messaging applications. In this paper we
investigate whether emoji can capture emotional states elicited by museum experiences, supporting
traditional methods such as interviews. While other non-verbal self-report methods have been used to
evaluate emotions, this is the first tool of this kind designed specifically to measure emotions elicited by
museum experiences. We designed a set of 9 emoji illustrating a variety of emotional states beyond happy-
or-not. Then, we confirmed that participants understood our emoji’s intended concept using a word
association task. Finally, we used our 9 emoji to evaluate an interactive museum experience. We also run
interviews and we investigated the correspondence between participants’ comments and the emoji they
chose. Through this study we gained a better understanding of how the emoji can be deployed to capture a
range of visitors’ emotional experiences. Our findings suggest that emoji can capture which emotional
states participants felt beyond the happy-or-not dichotomy, but that they should be complemented with
traditional methods such as interviews to understand why specific emotions were felt.
INTRODUCTION objects, emoji illustrating facial expressions are
particularly popular in everyday life to express
In the past couple of decades, emoji have emotions (Barbieri, Ronzano, & Saggion, 2016;
been growing in types and number (G€ ulsßen, G€ulsßen, 2016).
2016), becoming part of everyday language and Face emoji have been increasingly deployed
ulsßen, 2016; Oxford Dictionar-
everyday life (G€ to capture users’ satisfaction, both online and
ies, 2015). Emoji are text-based pictographic offline. For example, emoji have been used to
characters illustrating facial expressions and evaluate consumers’ emotions regarding the
abstract concepts such as emotions (Rodrigues design of new products or brands such as food
et al., 2017). According to Rodrigues et al. and drinks (Desmet, Overbeeke, & Tax, 2001;
(2017), emoji are used to help us to visually Jaeger et al., 2017). An increasingly common
express our emotions, especially in social net- example of this practice is the ‘smiley terminal’
works (Kelly & Watts, 2015; Vidal, Ares, & or ‘satisfaction kiosk’ often seen in airports.
Jaeger, 2016). While there are multiple types of Emoji have recently been introduced in muse-
emoji, ranging for example from animals to ums too to capture visitor satisfaction, but they
Daniela De Angeli (dada21@bath.ac.uk) is a postdoctoral researcher at the Centre for the Analysis of Motion,
Entertainment Research and Applications (CAMERA) at the University of Bath. Ryan M. Kelly (ryan.kelly@unimelb.
edu.au) is a Research Fellow in the School of Computing and Information Systems at the University of Melbourne.
Eamonn O’Neill (maseon@bath.ac.uk) is Professor of Human-Computer Interaction at the University of Bath.
© 2020 The Authors. Curator: The Museum Journal published by Wiley Periodicals, Inc. 1
This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution
and reproduction in any medium, provided the original work is properly cited.
Figure 1. Visitor satisfaction kiosks from a museum in Spain (left and center) and Germany (right). Photographs
taken by Daniela De Angeli in 2018.
are far less common (Figure 1). These satisfac- have limitations when it comes to capturing
tion kiosks usually display 3 to 5 emoji ranging emotions (Foster, 2008; “Visitor Evaluation
from happy to unhappy, so are limited to Guidelines” 2015) (see A Visitor-Centred
expressing the dichotomy of happy-or-not. Museum). Emotions are not easy to verbalize
While this may be enough to judge whether a or to observe (Desmet, Overbeeke, & Tax,
user is satisfied with the service in an airport 2001; Hein, 1998; Mehrabian, 1995). More-
bathroom or in a shop, museum visitor experi- over, the aforementioned methods tend to be
ences are more complex as they are very personal time consuming both for the museum and the
and influenced by a number of factors, including visitors. Image-based methods illustrating, for
emotions (Brent Ritchie et al., 2011; Norris & example, facial expressions are often more reli-
Tisdale, 2013; de Rojas & Camarero, 2008). able in capturing emotional states. However,
As visitors are more and more interested in these methods also have limitations (see Evalu-
‘feeling’ rather than ‘learning’ (Munro, 2014), it ation of Visitors’ Experiences). For example,
is becoming critical for museums to understand they may present too many emotions and
which emotions their public feel, where, and require visitors too much time and effort to
why. Once museums have a better understand- use. Emoji may offer an approach that is
ing of visitors’ emotional experiences, they can quicker but still effective but there is very little
plan for narratives that are more engaging, formal research to support whether or not
meaningful, and ultimately satisfying (Galani emoji can effectively capture visitors’ emotions.
et al., 2011; Hansen, Kortbek, & Grønbæk, This paper explores the potential of a range
2012; Simon, 2016). For example, memorials of emoji specifically designed for evaluating visi-
and history museums may wish to elicit empa- tors’ emotional experience beyond simply
thy with historical events and with victims of happy-or-not. We present an initial investiga-
atrocities (Savenije & de Bruijn, 2017). Local tion of an emoji-based method tailored specifi-
museums may want to connect with their com- cally to museums, to evaluate whether museum
munities at a deeper emotional level (Munro, visitors are satisfied (i.e. happy) or not with their
2014). experience. The method must be rapid and intu-
While understanding visitors’ emotional itive, so it should include a limited number of
experience is such a key factor in improving visi- emoji. Thus, we designed a set of 9 emoji icons
tors’ satisfaction, traditional methods such as illustrating key emotions elicited during a
questionnaires, interviews and observations can museum visit (see Design Process for Emoji).
2 Article: Beyond Happy-or-Not: Using Emoji to Capture Visitors’ Emotional Experience
Here, we use the term emoji to identify icons specific item. Here, when we talk about emo-
that are pictorial (i.e. not textual) and are not tions, we refer to a range of emotional experi-
necessarily digital. Since emoji may be misinter- ences including boredom, anger and inspired.
preted (Caicedo & Van Beuzekom, 2006), we Researchers also agree that emotions are
set out to validate both the method and the multidimensional. While these dimensions can
emoji themselves. First, we evaluated whether sometimes vary across fields and researchers,
the emoji we designed presented the intended there seems to be agreement on two main dimen-
meaning (see Emoji Validation: Word Associa- sions: arousal and valence (Baas, De Dreu, &
tion Task). Then, we further validated emoji as Nijstad, 2008; Russell, 1980). Arousal happens
a method to evaluate visitors’ emotional when a person ‘feels’ something, meaning their
responses through an emoji-based evaluation of senses are stimulated or a physiological/psycho-
an augmented reality sandbox developed for the logical state is awoken. In other words, it identi-
UK’s National Trust (see Validation of an fies the level of reactivity to an event or stimuli
Emoji-based Tool to Capture Emotional Reac- (Russell, 1980). Then, we interpret this arousal
tions). Based on findings from this study, we to determine why it occurs and what it is, often in
discuss how emoji can capture visitors’ emo- relation to context. For example, if we are having
tional experiences beyond happy-or-not (see a negative experience, we may be sad or angry.
Discussion). The differentiation between positive and nega-
tive experiences or events is known as valence
BACKGROUND (Barrett, Lewis, & Haviland-Jones, 2008).
According to Ortony et al. (1988), every emotion
Defining Emotions has a valence, a polarity. Thus, there is no neutral
emotion. However, an emotion can still be asso-
The question ‘what is an emotion?’ is still ciated with different valences (e.g. surprise).
very complex to answer (Solomon, 1993). Emo- Moreover, emotions can present with different
tions are often confused with other terms such as levels of intensity, which is the degree to which
feelings or other affective states such as attitudes emotions are experienced regardless of valence
and moods that impact human behaviour less but (van Goozen et al., 1994). Indeed, survey tools to
last longer (Scherer, 2005). Nevertheless, evaluate emotions often use different levels of
researchers usually agree that emotions are intensity to gain a deeper understanding of per-
prompt responses to events or stimuli. Indeed, sonal emotions. For example, Scherer (2005)
while feelings and moods have long term effects, arranges emotions with five degrees of intensity,
emotions are more immediate and can change Bradley and Lang (1994) propose a nine-point
rapidly (Borod, 2000). For example, Borod scale, and Desmet (2005) uses a three-point scale
(2000) defines emotions as changes in conscious (not-felt, light, intense).
subjective feelings in response to an evaluation of
external or internal events. Scherer (2000) high- A Visitor-Centred Museum
lights how emotions are short affective episodes
that change over time in response to internal or Traditionally, museums considered their
external events. More recently, Del Chiappa key mission to preserve human knowledge and
et al. (2014) defined emotions as affective vari- educate (Murphy, 2007). However, recent
ables elicited by an experience or by the use of a advances in digital technologies are pressuring
Daniela De Angeli, Ryan M. Kelly, Eamonn O’Neill 3
CURATOR: THE MUSEUM JOURNAL
museums into redesigning their exhibits to the museum is telling, more enjoyment, and
share knowledge through more enjoyable, ultimately a deeper and more frequent engage-
entertaining experiences in order to drive visits ment with the institution. (Dindar, 2015)
and support traditional goals such as education
(Tallon & Walker, 2008). This is happening for In particular, Filippini-Fantoni explains
a variety of reasons. First, digital technologies how a visitor-centred approach to exhibition
have driven changes in society influencing how development can increase visitors’ satisfaction
we live, communicate, learn and how we per- (Dindar, 2015). Understanding visitors is key
ceive the world (Bryce, 2001; Greenfield, 2014; when we consider that their museum experience
Siemens, 2005). Museums need to maintain is often based on personal experience rather
their relevance in a changing society as their than objective values (Gallarza & Saura, 2006).
mission is not only to preserve knowledge but In particular, research has highlighted how
also to share it (Tallon & Walker, 2008). Sec- emotions strongly influence visitors’ satisfaction
ond, digital technology offers us so many (de Rojas & Camarero, 2008). For example, Del
options for our entertainment, some passive Chiappa et al. (2014) interviewed 410 visitors at
(e.g. movies) other more interactive (e.g. games) the National Museum of Archaeology “G.A.
(Greenfield, 2014). While traditional object-fo- Sanna” in Sardinia (Italy) and discovered that
cused exhibitions tend to limit visitors’ interac- higher positive emotions reported corresponded
tion to merely passive observation, this is not to a higher level of satisfaction.
how most people choose to use their free time While the evolution of digital technology
(Simon, 2010). Hence, the simple display of has strongly influenced museums and their
facts and objects is no longer enough to attract approach to exhibition design, the evaluation of
and engage visitors. As a consequence, muse- visitors’ experience and satisfaction is crucial
ums are embracing a more visitor-centred regardless of the nature of the exhibition.
approach where it is increasingly important to Whether an exhibition is digital or not, interac-
understand visitors and how they interact with tive or passive, the evaluation of visitors’ emo-
the artefacts in order to design narratives that tional response to it is key to understanding how
are more engaging and meaningful (Galani they perceive the museum and ultimately to
et al., 2011; Hansen, Kortbek, & Grønbæk, increase engagement. However, evaluating the
2012; Samis, Michaelson, & Baird, 2017; impact of emerging technologies can be particu-
Simon, 2016). larly challenging as both their implementation
According to Silvia Filippini-Fantoni, spe- and impact are less known than long established
cialist in museum audience analysis and former tools (Damala et al., 2013). Although museums
director of interpretation, media and evaluation usually have well-established methods to evalu-
at the Indianapolis Museum of Art: ate traditional passive visits, they may not yet
have a standard protocol to evaluate new and
Museums are institutions that serve the emerging technologies:
public and therefore knowing what people enjoy
and don’t is fundamental to helping these insti- They have yet to cultivate standard protocol
tutions create improved experiences for their for measuring the success of the technologies
visitors. It can result in a better appreciation of they deploy. Exacerbating this challenge is the
the art, better understanding of the stories that notion that evaluation should occur both before
4 Article: Beyond Happy-or-Not: Using Emoji to Capture Visitors’ Emotional Experience
and after technologies are implemented; staff included four A4 pages of questions (Damala
must have thorough understanding of how the et al., 2013).
tools correspond with the museum’s mission and Observational methods such as direct obser-
goals prior to being embraced at scale. Unfortu- vation have also been used in museums, but it is
nately, there are not always concrete precedents difficult to acquire a deep understanding of visi-
for the use of new technologies in the cultural tors’ feelings just by observing them. Indeed, “re-
heritage sector, and museums that are early searchers . . . are limited by what they can
adopters often gamble when trying them (John- actually see” (Hein, 1998, 47:101). Often only
son et al., 2015, 28) strong emotional reactions are clearly visible
(e.g. very angry), and people express and experi-
In the next section we explore which meth- ence emotions differently accordingly to their
ods and strategies museums are deploying to cultural or personal background (Scherer, 1988).
understand their visitors’ experience, whether Emotions can also be measured through
with traditional or emerging tools and tech- interaction with gesture, gaze and auditory
nologies, and whether these methods are effec- stimuli. For instance, Ramanarayanan et al.
tive in providing relevant and useful data. (2015) used a variety of equipment and software
tools including Microsoft Kinect to evaluate the
Evaluation of Visitors’ Experiences quality of public presentations in relation to
speech, face, emotion and body movement. Lu
Recent evaluation guidelines from cultural and Petiot (2016) used a set of auditory stimuli
institutions such as the Smithsonian (“Visitor to convey and assess a set of emotions such as
Evaluation Guidelines” 2015) and the East of funny, serious, relaxed, and depressed. One
England Museum Hub (Foster, 2008) indicate advantage of these techniques is that they are
that the most used methods to evaluate visitors’ unobtrusive as they do not require users to ver-
experiences were and still are observation, ques- balize their feelings. However, the technology
tionnaires and interviews. For example, visitors’ used to sense non-verbal behaviour often has
experience with the exhibition The Hague and limitations. For example, Mueller and Bianchi-
the Atlantic Wall: War in the City of Peace at Berthouze (2015) noted that gesture recogni-
Museon, in the Netherlands, was evaluated tion technology is unpredictable and the set of
using a mixed methodology of observations, movements and gestures cannot be predefined.
questionnaires, and interviews (Damala et al., Moreover, such methods cannot usually mea-
2013). sure mixed emotions and their range is limited
However, subjective and subtle emotions to few basic emotions (Desmet, 2005). Finally,
are difficult to articulate and measure with ver- it is typically not feasible for museums to buy,
bal questionnaires (Desmet, Overbeeke, & Tax, install or persuade visitors to wear devices such
2001; Mehrabian, 1995). Moreover, standard as body trackers, brain or temperature sensing
scales such as Likert tend to gain positive, non- devices.
candid responses (Benedek & Miner, 2002). Visitors’ behaviour and interactions may
While longer questionnaire may provide richer also be video recorded. This method can be
amount of data, they would also be time con- unobtrusive and can provide rich data about vis-
suming for visitors; for example, a questionnaire itors’ behaviour and interactions with the exhi-
used to evaluate visitors’ experience at Museon bition. A video may subsequently be analysed in
Daniela De Angeli, Ryan M. Kelly, Eamonn O’Neill 5
Figure 2. Range of Emotions Wheels. (a) Geneva Emotion Wheel. Source: (Tran, 2004). (b) Plutchik’s Wheel. Source: (Garcia & Hammond, 2016). (c) Plutchik’s Wheel with cartoons. Photo credit: Copy Press. (d) Plutchik’s Wheel with emoji. Source: https://usabilla.com/blog/best-ux-articlesaugust/.
CURATOR: THE MUSEUM JOURNAL
detail in order to collect and interpret informa- power and valence. Each emotion is repre-
tion about visitors and their experience. Never- sented by a different colour and five circles of
theless, this method remains little used, often different size indicating five degrees of inten-
due to ethical concerns or the difficulties in sity, so participants can also indicate the inten-
installing appropriate recording equipment sity of their emotions. Plutchik’s Wheel of
within the museum (Damala et al., 2013). Emotions is also well known and has been used
All of these methods have their merits and for example to detect emotions on social media
limitations, meaning that combinations of differ- (Tromp & Pechenizkiy, 2015). It includes 8
ent techniques (i.e. mixed methods) are often basic emotions that are considered key to our
preferred to evaluate visitor experiences. By survival. These emotions come in pairs and are
using combinations of methods for data collec- located opposite to each other: Joy and Sad-
tion, museums can cover a wider range of people ness; Acceptance and Disgust; Fear and Anger;
and data. For example, interviews are often used Surprise and Anticipation. This arrangement
to support both direct observations and ques- is due to Plutchik’s belief that opposite emo-
tionnaires, providing more in-depth informa- tions cannot be felt at the same time (Plutchik,
tion. For example, the Smithsonian uses a mix 1980).
of qualitative and quantitative methods includ- However, the time taken to apply these
ing interviews, surveys, and direct observation tools including multiple emotions and intensity
to support their visitors’ evaluation (“Visitor levels can be challenging (van Goozen et al.,
Evaluation Guidelines” 2015). Still, combina- 1994) and too time consuming in some contexts
tions of traditional methods provide limited such as museums where evaluation methods
data about emotions and, as discussed above, should not disrupt the visit. Moreover, the out-
visitors’ experience is strongly influenced by comes of verbal methods rely on people’s ability
their emotions, which are not easy to verbalize to express their emotions (Fox, 2008). A way to
(Reijneveld et al., 2003) or to observe (Hein, overcome this limitation of language-based
1998). methods is to visualize emotions as images
rather than words (Foglia, Prete, & Zanda,
Methods to Evaluate Emotions 2008). Indeed, there have been adaptations of
emotion wheels using images such as cartoons
As noted in the previous section, tradi- or emoji rather than text (Figure 2). In general,
tional methods such as interviews and direct image-based methods tend to be more fruitful
observation may struggle to capture visitors’ to capture emotions than verbal ones. For exam-
emotional experiences. However, there are ple, a widely accepted image-based tool is Brad-
other fields such as consumer evaluation where ley and Lang’s (1994) graphical version of
emotions have been successfully evaluated. For Mehrabian’s PAD which used Manikins
example, the Geneva Emotion Wheel is a (graphical characters) combined with a nine-
well-known method to evaluate emotions point scale to make the tool more intuitive to
(Scherer, 2005). Participants indicate which participants. However, it uses graphic charac-
emotion(s) they are experiencing from a ters to represent emotions that are not familiar
wheel-shaped emotion scale (Scherer, 2005). to the general public, and they may require too
Emotions are arranged in this circular pattern much time and effort from visitors to under-
according to two major dimensions: control/ stand. Moreover, this method cannot measure
8 Article: Beyond Happy-or-Not: Using Emoji to Capture Visitors’ Emotional Experiencedifferentiated emotions (Caicedo & Van Beu- cartoon face that best identifies their experience
zekom, 2006). (Desmet, 2005).
Emoji have become increasingly familiar One of the most successful non-verbal
to the general public, notably through mobile self-report methods is PrEmo (Product Emo-
messaging systems. Consequently, they are tion Measuring Instrument), which uses a
increasingly used to express and capture emo- three-point scale (not-felt, light, intense) with
tions in a variety of fields including for 12 cartoon animations to represent emotions
example food consumption (Jaeger et al., (Desmet, 2005) (Figure 2). Each of these
2017; Vidal, Ares, & Jaeger, 2016) and well- emotions was selected to represent emotions
being (Fane et al., 2018). The ‘Emoji-o- elicited by consumer products: desire, satisfac-
meter’ uses emoji to evaluate children’s experi- tion, pride, hope, joy, fascination, disgust, dis-
ences with technology (Read & MacFarlane, satisfaction, shame, fear, sadness and boredom.
2006). ‘Emoji-face’ assessment scales have also Each cartoon was animated and includes sound
been deployed in museums to augment tradi- effects designed to facilitate their interpreta-
tional questionnaire scales (Loizeau, K€ undig, tion. However, this could potentially disrupt
and Oppikofer, 2015; Mittelman and Epstein, visitors’ experience within the museum. It also
2009). Similar emoji systems are often seen in means that PrEmo requires the use of a com-
kiosks at airports as a method to empower puter or mobile device, with corresponding
customers and measure their level of satisfac- oversight and maintenance requirements if
tion (Dickinson, 2018). On one hand, these deployed in museums. Even if the cartoon
tools are a very rapid way to capture users’ were instead printed on paper, PrEmo was
basic experience. On the other hand, they designed exclusively to measure emotions eli-
usually evaluate users’ satisfaction simply in cited by consumer products. While it can be
term of happiness and unhappiness. As a used in other fields by selecting only relevant
result, they capture a very limited range of emotions, emotions relevant to museums (e.g.
emotions, while people’s emotional experiences achievement and social engagement) are miss-
are far more complex and go beyond happy or ing from this set. The purpose of our emoji
unhappy. collection is not to evaluate the product satis-
Pictorial representations of facial expres- faction of a generic consumer but to under-
sion, such as cartoons and photos, have been stand museum visitors’ satisfaction beyond
used effectively to communicate a wider variety simply happy-or-not. In the next section we
of emotions and have been researched as evalua- describe the reasoning behind the design of
tion tools in a variety of fields (Bradley and each emoji in our set and the emotional states
Lang, 1994; Desmet, 2005). For example, they represent.
Microsoft tested a questionnaire using pictures
of six faces as stimuli to get user input on intan- DESIGN PROCESS FOR EMOJI
gible properties such as “desire” and “fun” (Ben-
edek & Miner, 2002). Emofaces uses a series of The overall quality of the museum experi-
female and male faces to represent emotions ence is often based on personal experience rather
ranging from pleasant to unpleasant, and than objective values (Gallarza & Saura,
intense to calm (Posner, Russell, & Peterson, 2006). Personal experience includes social con-
2005); Emocards ask people to choose the text, personal motivation, education andCURATOR: THE MUSEUM JOURNAL
Figure 3. The emoji icons used to capture visitors’ emotional experience.
expectations (D. Bryce et al., 2014; de Rojas & & Chen, 2010), where people feel part of the
Camarero, 2008). Visitors’ perceptions of museum, and the museum part of the commu-
the museum experience are affected by nity. To close the circle, if visitors feel involved
whether their expectations are met, they feel and a sense of achievement, then they will prob-
engaged and involved and they perceive the ably enjoy their visit. Thus, in this paper we
museum service as adequate (Bride, Disegna, & argue that visitors feel satisfied not only if they
Scuderi, 2014; Bryce et al., 2014; Lu, Chi, & have received an excellent service but also if
Liu, 2015). they:
We draw on Thurley’s (2005) account of
the heritage experience as a cycle of understand- • Enjoy their visit;
ing, valuing, caring and enjoying. If the • Achieve and/or acquire new knowledge;
museum narrative is clearly communicated, • Feel inspired;
then visitors are more likely to enjoy it and find • Feel socially involved;
it relevant. If visitors find the narrative is valu- • Feel entertained/engaged;
able to them, they may also learn and feel a • Find the narrative clear, communicated
sense of achievement. If visitors feel a sense of clearly
achievement and that the content they are
interacting with is relevant, then they feel more Accordingly, we designed a set of 9 emoji to
involved and they are more inclined to care for mirror emotional states directly related to the
what the museum has to offer. Involvement above principles of visitors’ satisfaction or
implies some kind of social participation (Chen indicative of dissatisfaction (Figure 3):
10 Article: Beyond Happy-or-Not: Using Emoji to Capture Visitors’ Emotional Experience• Basic enjoyment is illustrated by the emoji and WhatsApp render these icons differently
Happy, Sad, and Angry. (Miller et al., 2016). In order to address this
• Learning outcome is represented by the issue, we ran a validation study to investigate
emoji Achieved; whether participants recognised our emoji’s
• Feeling overwhelmed by information is intended concept or if instead they misinter-
illustrated by Tired; preted the emotion we intended to depict
• Inspiration is illustrated by the emoji with a given emoji. We developed an online
Inspired; word association test using Google Forms. The
• Lack of engagement is illustrated by the survey included standard demographic
emoji Bored; questions such as gender and age. Then, the
• Feeling involved is illustrated by the emoji emoji were displayed one after another. Each
Socially Engaged; emoji was followed by a text box where par-
• Lack of clarity is represented by the emoji ticipants were invited to type the first word
Confused; that occurred to them in response to the
stimulus.
The emoji illustrating happiness, anger and Our method was inspired by Prada et al.
sadness were the easiest to design, possibly (2016) and Rodrigues et al. (2017) who asked
because they were the most familiar ones. For participants to state the first meaning or emo-
example, they are often seen in smiley terminals tion that came to mind related to an emoji.
at airports. The design of other emoji, such as Moreover, as in Pejtersen (1991), we used word
the ones representing social experience or association to identify the meaning of images.
achievement, was more challenging. We Word association is a well known method
researched existing emoji including those used (Jung, 1910; Nielsen & Ingwersen, 1999) which
by Apple iOS, Facebook Messenger and Skype consists of presenting a stimulus and the partici-
in order to facilitate our design but also to make pant answering as quickly as possible with the
them more recognizable. While these icons first word that occurs to her. The responses cre-
were already familiar to many people as they are ate a cluster of associative representations of the
proliferating on mobile messaging apps, they stimulus (Nielsen & Ingwersen, 1999). Word
were also protected by copyright, encouraging association is used to collect information on
us to design our own emoji for use by researchers people’s perceptions, emotional states, mental
and practitioners. The designs were produced models and vocabulary (Nielsen & Ingwersen,
using a Wacom Bamboo graphics tablet and 1999; Roininen, Arvola, & L€ahteenm€aki,
Adobe Illustrator. 2006), and has been used to capture the mean-
ing of icons for graphical user interfaces (Pejter-
EMOJI VALIDATION: WORD sen 1991) and food (Roininen, Arvola, &
ASSOCIATION TASK L€ahteenm€aki, 2006).
Participants were recruited through Ama-
Pictograms such as emoji do not always zon MTurk. The survey took about 5 minutes
clearly depict a specific emotion (Caicedo & to complete, for which each participant received
Van Beuzekom, 2006). Moreover, they can be $1.30. We had 121 participants, 77 males, 43
open to interpretation because social networks females, and 1 who preferred not to say. Partici-
and mobile messaging apps such as Facebook pants were mostly 26-35 years old (63
Daniela De Angeli, Ryan M. Kelly, Eamonn O’Neill 11Table 1.
List of emoji (ID) with their intended meaning. The table also includes the words most frequently associated with each emoji (i.e. Associated meaning) and world clouds displaying all the words
associated with each emoji
ID: Intended Associated ID: Intended Associated
meaning meaning Word clouds meaning meaning Word clouds
A: Anger Angry F: Inspired Idea, inspired,
enlightened
CURATOR: THE MUSEUM JOURNAL
B: Happiness Happy G: Educated,
Achievement proud,
graduated
C: Sadness Sad, crying H: Tiredness Tired, sick
(continued)Table 1 . Continued
ID: Intended Associated ID: Intended Associated
meaning meaning Word clouds meaning meaning Word clouds
D: Boredom Bored, tired, I: Social Friends, love
sleepy experience
Daniela De Angeli, Ryan M. Kelly, Eamonn O’Neill
E: Confusion Confusion
13CURATOR: THE MUSEUM JOURNAL
participants), followed by 18-25 (23), 36-45 capture emotional aspects of a visitor experi-
(22), 46-55 (7), and the remaining 6 partici- ence: an augmented reality (AR) sandbox
pants were over 56 years old. developed for the National Trust in the UK.
The responses were recorded in a spread- We ran this study to investigate whether the
sheet which was then imported into NVivo. emoji can indeed be used in practice to evalu-
We used NVivo to calculate word frequency ate visitors’ emotions. We also wanted to con-
and to group responses into categories. Syno- firm which emotional states each emoji can
nyms (e.g. angry, anger, and annoyed were capture to further validate how the meanings
included in the same category) and singular/ of the emoji are perceived. Can complicated
plural forms (e.g. idea, ideas) were included in concepts such as personal achievement and
the same category. Some responses could not social engagement be represented through
be related to others, so they were not included emoji?; and are they selected by participants
in any category (e.g. yellow, eye, and carrot). when there has been some educational out-
The main results illustrating the most frequent come or positive social experience?
words associated with each emoji are summa-
rized in Table 1. VALIDATION OF AN EMOJI-BASED TOOL
The icons illustrating anger, happiness, TO CAPTURE EMOTIONAL REACTIONS
sadness, and confusion were clearly associated
with one specific category. For example, emoji We further validated the perceived mean-
A was associated with the category including ing of these emoji by designing an emoji-based
words such as anger and angry (#anger) 96 out survey. The survey was used to evaluate how
of 121 times. Emoji B (#happiness) was asso- people perceived an interactive sandbox that the
ciated with happy 116/121 times. Emoji C authors developed for the UK’s National Trust
(#sadness) with sad 114/121 times. Emoji E to commemorate the tercentenary of the land-
(#confusion) was most frequently associated scaper Capability Brown in 2017 (Figure 4).
with confusion, 90/121 times. Other emoji The system was devised to illustrates how
were associated with more than one category. Brown implemented his landscapes and is based
This was true in particular for the emoji illus- on the AR sandbox developed by Reed et al.
trating the concept of social experience, which (2014), which allows users to create topographic
was associated with words such as friend (33/ models by shaping sand. The system augments
121) and love (46/121). Emoji D (#boredom) the sandbox by projecting a topographical map
was frequently associated with both bored (66/ onto the sand using a projector connected to a
212) but also with tired (31/121), which may Microsoft Kinect 3D camera. When the sand is
link to mental tiredness. While emoji I moved around, the Kinect senses the changes in
(#tiredness) was mostly associated with tired the sand’s elevation and changes its projection
(34/121) and sick (23/121), which may relate accordingly. For example, if someone digs a hole
to a physical tiredness. Emoji G (#inspired) in the sand, the system projects a blue surface
was described using words such as idea (70/ representing water in that location. In our study
121) while emoji H (#achievement) with edu- we used two versions of the sandbox: (1) Reed
cated (83/121). et al.’s (2014) digital version augmented by
Having validated the emoji as conveying Microsoft Kinect and a projector; (2) a more
their intended meanings, we used them to traditional ‘analogue’ version without depth
14 Article: Beyond Happy-or-Not: Using Emoji to Capture Visitors’ Emotional ExperienceFigure 4. Photo (left) and schematic (right) of the augmented reality sandbox.
sensing and projection, where participants could identified this as a gap in their audience. The
create a landscape with props such as little introduction of new technologies is seen as a
houses and signs illustrating trees and water. By way to attract a younger audience that is cur-
using two versions of the same sandbox, we rently not visiting their properties. As we were
could evaluate our emoji-based tool with both looking specifically for young adult participants
digital and analogue experiences. who were not necessarily visitors to the National
Trust, we recruited participants on campus,
Participants through university mailing lists and word of
mouth. All participants were students and
We had 24 participants in total: 12 partici- members of the staff at the University of Bath.
pants interacted with the digital version (4 male
and 8 female) and 12 with the analogue one (6 Method
females and 6 males). Participants were aged 22
to 34 years. We were particularly interested in An interactive sandbox was installed in a
this age range because the National Trust room at the University of Bath so that we could
Daniela De Angeli, Ryan M. Kelly, Eamonn O’Neill 15CURATOR: THE MUSEUM JOURNAL
collect data in a controlled environment. Each their experience. The emoji were presented in a
participant was given a printed copy of a topo- paper survey. The survey was initially designed
graphical map. First, they were asked to use the with the emoji in a circular patter opposite each
sandbox to replicate the landscape in the map. If other similar to the Geneva Emotion Wheel (K.
they were interacting with the digital version, R. Scherer, 2005). However, we did not want to
they simply moved the sand around and the pro- imply an opposite valence. Rather, we acknowl-
jection on to the sand automatically changed to edged that different emotions can happen at the
match the landscape they created. If they were same time and do not necessarily exclude each
using the analogue version, they moved the sand other. Thus, we displayed the 9 emoji (Figures 1
and placed the props to add landscape elements and 5) with 3 levels of intensity in order to keep
such as trees and houses. They were then asked the survey short and rapid. PrEmo (Desmet,
to make changes of their choosing to the land- 2005) successfully used a 3 point scale where
scape they had created. For example, they could participants could pick low, medium or high
move the sand around to create a hill or dig a intensity for a specific emotional state. Hence,
lake, or change the position of the props. Each we decided to use a simple three intensity levels
participant interacted with the sandbox for scale for our survey. Similarly to other image-
about 15 minutes. based tools such as the Geneva Emotion Wheel,
At the end of the session, a researcher asked intensity levels were displayed as circles of dif-
participants to complete an emoji-based survey ferent sizes (Figure 5).
and carried out a semi-structured interview. After completing the survey, participants
Each participant was assigned an ID that was were interviewed for about 15 minutes. Each
associated with their survey. The interviews interview was guided by a list of questions to
were audio recorded and saved with the ID of evaluate visitors’ experience and satisfaction
the participant. with the sandbox. Questions ranged from
First, participants completed a emoji-based learning outcomes to usability of the system,
survey by selecting the emoji that best illustrated including: Did you enjoy the experience? What
Figure 5. The survey with nine emoji and three levels of intensity each.
16 Article: Beyond Happy-or-Not: Using Emoji to Capture Visitors’ Emotional Experiencedid you like/not like? Was the system easy to between what participants said during the inter-
use? Do you have any questions? Did you need view and the emoji they chose.
any help? Would you like to play it with your
friends? Results
Data Analysis The most frequently selected emoji were
those representing Happiness, Achievement and
The interviews were transcribed into a Inspired, chosen respectively 21, 19 and 14 times
Word document by the researcher who ran the out of 24 (Figure 6). This general positive atti-
interviews. The participant’s ID was also tude was confirmed during the interviews where
included in the document. Meanwhile, a differ- all participants declared that they had a pleasant
ent researcher (i.e. an analyst) organised the experience. Pt 14 selected happiness and during
emoji selected into a spreadsheet: the document the interview described the sandbox as a magical
included participant ID, emoji selected and at experience: “so funny . . . easy to use. I have
which level. The analyst then carried out a qual- enjoyed it”. Pt 16 selected both Achievement and
itative content analysis of the interviews to gain Inspired, feelings that were confirmed by the
an understanding of the participants’ experience interview, in which the participant explained
with the sandbox. Comments related to emo- how a textbook can give you more knowledge
tional states and subjective experiences were “but you cannot really know what a topographi-
identified. Using participants’ ID as a reference, cal map is [from a book]”.
these comments were then compared with the While traditional surveys tend to attract
selected emoji to investigate the correspondence only positive responses, our emoji were also able
Figure 6. Emoji selected by each participant (1 to 24).
Daniela De Angeli, Ryan M. Kelly, Eamonn O’Neill 17CURATOR: THE MUSEUM JOURNAL
to capture negative emotions. 3 participants had is confusing” and could be improved to better
very negative feelings. Pt 11 selected both Con- indicate “heights”. However, the emoji repre-
fused and Angry, stating: “It is not clear, I am senting boredom was selected instead of the
really confused”, adding later: “It is boring . . . it confusion emoji. This could be because the
is just sand . . . it is like homework”. The fact interaction itself was clear and easy to under-
that this participant “felt tired because it is bor- stand but that the colour mapping was slightly
ing” was also confirmed by the emoji survey annoying and made the experience less engag-
where maximum level of Boredom was selected ing. Pt 2 liked the experience and selected hap-
but not Tiredness. Pt 20 was also not happy with piness, probably because it “helped me
the overall experience, selecting Boredom and understanding different altitudes”, however, “I
Sadness rather than Happiness and Achievement. did not know the exact altitude of each colour”
During the interview, this participant stated and could not “understand what the different
that the sandbox could be “slightly more inter- colours mean”. This was reflected in the emoji
esting” and a textbook would be better, more survey as the participant selected Confusion. Pt 9
educational. Pt 20 also complained that the argued that the sandbox was “quite interesting”
sandbox was too small and about the consis- but that a textbook would be more useful to gain
tency of the sand. Pt 6 and 4 also selected Bore- a deeper understanding of the subject. At the
dom. Pt 6 was “not very excited . . . little bit end, pt 9 was “neither tired nor excited”. These
tired” and selected both Tiredness and Boredom. mixed feelings were also reflected on the selec-
Pt 4 said that the sandbox was “just sand” and tion of maximum intensity of Happiness
could be improved. While participants felt con- together with average intensity of Confusion. Pt
fused almost equally with both the digital and 22 also selected Confusion and during the inter-
analogue versions of the sandbox, the second view confirmed the participant found the tasks
version attracted the majority of negative emo- not so clear. Pt 21 considered the sandbox a
tions. Indeed, the emoji Boredom, Angry, Tired “more direct way to understand (topographical
and Sadness were selected only by participants maps)” but expressed a wish for more specific
who interacted with the analogue sandbox. guidelines. Again, the emoji representing Con-
Participants also selected different levels of fusion was also selected. Pt 19 found the experi-
intensity. In some cases, positive emotions were ence more interesting than a textbook but
selected together with other positive ones. For initially found the colour mapping a “little bit
example, usually people who selected the maxi- confusing”. By the end, that was “not a prob-
mum level of Happiness choose exclusively lem” and did not affect the overall experience.
other positive emoji such as Educated and Indeed, the emoji indicating Confusion was not
Inspired. However, participants also selected selected.
positive and negative emotional states at the
same time, a mixed experience that was con- DISCUSSION
firmed during the interview. For example, par-
ticipants 2, 4 and 22 selected maximum Our study suggests that emoji can be used
Happiness together with a minimum level of effectively to capture visitors’ subjective experi-
Confusion or Boredom. During the interview, Pt ences beyond a simple dichotomy of happiness
4 stated that the sandbox “is fine and a good and unhappiness. It does so by providing an
learning tool, but the colour coding (of the map) understanding of why emoji were selected, how
18 Article: Beyond Happy-or-Not: Using Emoji to Capture Visitors’ Emotional Experiencethey were perceived and how they may be whether the visitor had a playful and happy
deployed as a survey tool. Moreover, the digital experience together with others.
and analogue sandboxes were clearly experi- Only 3 participants interacting with the
enced differently, a point that was successfully analogue sandbox selected Anger, at levels 1, 2
recorded by both interviews and emoji. In par- and 3 respectively. However, participants did
ticular, emoji clearly displayed how one version not describe such a negative experience during
was more enjoyable that the other. This sug- the interview. Participants found the analogue
gests that emoji can capture emotions elicited by sandbox particularly boring and they did not
a range of different museum experiences, and like the fact they could not wash the sand from
also that they can capture both positive and neg- their hands at the end. The results from the vali-
ative emotions, giving candid results. Further- dation study indicate a very clear association
more, our findings demonstrate how emoji can between this emoji and the concept of anger.
capture mixed emotions as in some cases partici- One possibility is that interviewees were trying
pants selected negative emotions (e.g. Bored and to be polite during the interviews but more fully
Confusion) at the same time and at different expressed their emotions through the survey. It
levels. is also important to remember that the emoji
The experiences described during the inter- survey was completed immediately after partici-
views further validated how the emoji are per- pants interacted with the sandbox. By the time
ceived. For example, the emoji illustrating they did the interview, their memory of the
Confusion was usually selected by participants experience and their related emotions may have
who were frustrated with an interface or who already changed somewhat.
did not understand some content, e.g. users who The study raised questions around the opti-
expressed usability or clarity issues with the mal number and individual distinctiveness of
sandbox during the interview. Participants typi- emoji. The set of 9 emoji enabled greater
cally selected Boredom if they found the sandbox expressive ‘bandwidth’ than the simple happy-
not engaging or interesting. Tiredness was or-not dichotomy, while retaining sufficient
selected when a participant felt physical, rather distinctiveness between similar or related emo-
than mental, fatigue. During the validation pro- tions to facilitate participants’ selection of the
cess, this emoji was also associated with sick- emoji that most closely matched a given emo-
ness, so it could be interesting to test whether it tion. For example, the study confirmed the rela-
might be also used as part of evaluating whether tionship between Achievement and Inspired.
an experience is likely to make visitors physically Participants who completed the tasks with the
sick, such as motion sickness in virtual reality sandbox felt a strong sense of achievement and
experiences (LaViola, 2000). Our results also selected Achievement, while Inspired was
clarify the perception of the emoji illustrating selected when participants also felt they had
Social experience. During the validation study learned something new. In principle, creating
(section 4), this emoji was associated with words an even larger set of emoji is appealing since the
such as friends, friendship, love and happiness. further increased bandwidth could represent an
The participants in our case study selected this even wider range of emotions and allow for rep-
emoji when they thought the sandbox was resenting more fine-grained distinctions
potentially fun to play with friends. This sug- between similar or related emotions. However,
gests that this emoji could be used to investigate there are at least two potential problems with
Daniela De Angeli, Ryan M. Kelly, Eamonn O’Neill 19CURATOR: THE MUSEUM JOURNAL
adding more emoji. The first is simply that a key Through the case study we gained a better
requirement for our development of the emoji- understanding of how the emoji can be
based approach was speed and ease of use in the deployed to capture visitors’ emotional experi-
museum visitor setting. This requirement ences. In particular, our results provided
would become harder to meet as the set of emoji insights on the use of polarity (i.e. opposite
grew, almost inevitably leading to less participa- valence of emotions) and intensity levels. Emo-
tion and therefore less visitor experience data tions are dynamic and can present at different
collection in practice. intensities (van Goozen, van de Poll, and Ser-
The second challenge exists with any set of geant 1994), which is why emoji surveys should
emoji and becomes even more challenging if we allow visitors to select multiple emotions and
attempt to use additional emoji to represent different intensity levels. While established
more fine-grained distinctions between similar smiley terminals usually allow users to select
or related emotions. Any two emoji must be suf- only one icon (e.g. happy, neutral or angry), our
ficiently distinct that they are reliably recog- participants could select multiple emoji. Indeed,
nised and distinguished from each other. The our survey recorded a variety of emotional
work reported here validated that this require- responses, positive and negative, as well as their
ment was met for the set of 9 emoji. intensity level. Not only did participants often
However, as with any representational sys- choose more than one emoji, they also selected
tem, there is a trade off between the set of sym- combinations of emotions of different valence,
bols, their semantics, and their expressive such as happiness and confusion. Apparently, par-
power. Having only 9 emoji facilitated the rapid ticipants enjoyed the sandbox despite not always
use of the survey but came at the expense of regarding the system as very usable. This cor-
some expressiveness. For example, participants roborates the claim that usability cannot fully
often selected Sadness to express dissatisfaction explain users’ experience without taking emo-
rather than sadness or grief per se. This makes tions into account (Agarwal and Meyer, 2009).
sense since the sandbox was not designed to Furthermore, participants selected different
provoke strong emotional states or to upset visi- intensity levels, ranging from low (1) to high
tors. This finding suggests that we may need (3). Thus, the tool was able to record emotional
more emoji than this basic set of 9 as users could layers where different emotions happened at the
not distinguish between sadness and dissatisfac- same time, at different levels of intensity, for
tion. However, such an extension of the set of example, a high level of Happiness together with
emoji would require further careful design and low levels of Boredom, Confusion or Sadness.
validation, and is likely to necessitate creating Lastly, our study confirmed that while ver-
visually very distinct emoji even for similar emo- bal methods can be used to describe an experi-
tions. For example, we might propose emoji ence, they are not optimal to capture emotions
depicting Thumbs up and Thumbs Down to illus- (Desmet, Overbeeke, & Tax, 2001; Mehrabian,
trate satisfaction/dissatisfaction in order to 1995; Reijneveld et al., 2003). Indeed, during
maintain a reliable distinction from sadness. the interview participants often talked about
Further work is needed to determine the extent their general experience with the sandbox rather
to which using a wider range of emoji would than their emotional state during the interac-
adversely affect the method’s reliability as well tion. The emoji-based survey can provide a clear
as one of its main strengths, its speed. indication of which emotions were felt, which is
20 Article: Beyond Happy-or-Not: Using Emoji to Capture Visitors’ Emotional Experiencesomething interviews can struggle to do. How- expressions differently according to their cul-
ever, the interviews helped to understand why tural background (Jack, Caldara, & Schyns,
emotions were felt. Thus, we suggest than emoji 2012; Park et al., 2013). Thus, further studies
and other traditional methods such as inter- are needed to investigate emoji validity across
views are used together so that they can comple- visitors with different cultural and social back-
ment each other. Data from the interviews can grounds. END
be mapped to the emoji selected. These com-
bined methods can provide a rich understanding ACKNOWLEDGMENTS
of which emotional states were felt and why.
This work is part of EPSRC Centre for Digital
CONCLUSION AND FUTURE WORK Entertainment (grant EP/G037736/1) and the
National Trust funded research project to investigate
In this paper we tested the ability of emoji next-generation cultural heritage user experiences. A
special thanks go to Xindan Wang who helped col-
as a tool to evaluate visitors’ emotional experi-
lecting data. Daniel J Finnegan and Malcolm Holley
ence. While other non-verbal self-report meth- who helped to develop and installing the sandbox.
ods have been used to evaluate emotions, this is Eamonn O’Neill’s research is partly funded by CAM-
the first tool of this kind designed specifically to ERA, the RCUK Centre for the Analysis of Motion,
measure emotions elicited by museum experi- Entertainment Research and Applications (EP/
ences. Before this study, there was little formal M023281/1).
research to support whether emoji can effec-
REFERENCES
tively capture visitors’ emotions. Hence, we
designed a set of 9 emoji: Happy, Sad, Angry,
Agarwal, A., & Meyer, A. (2009). Beyond usability.
Confused, Achieved, Inspired, Bored, Tired, and In Proceedings of the 27th International Conference
Socially Engaged. We validated their meaning Extended Abstracts on Human Factors in
and confirmed that these emoji were indeed able Computing Systems – CHI EA ’09, 2919. New
to capture the relevant emotions in a controlled York, NY: ACM Press. https://doi.org/10.
environment. This helped us to test the method, 1145/1520340.1520420
ensuring it was rapid, intuitive, and effective Baas, M., De Dreu, C. K. W., & Nijstad, B. A.
(2008). A meta-analysis of 25 years of mood-
before deploying it in the field. Our emoji are
creativity research: Hedonic tone, activation, or
freely available under Creative Commons
regulatory focus? Psychological Bulletin, 134(6),
license from: https://drive.google.com/open? 779–806.
xml:id=1onitMwbFF9echTCBb0QHBBcqil Barbieri, F., Ronzano, F., & Saggion, H. (2016).
vzwp1w What does this emoji mean? A vector space skip-
We are currently developing an online sur- gram model for twitter emojis. In Proceedings of
vey editor that will allow museums to create the Tenth International Conference on Language
their own emoji survey. We intend this tool to Resources and Evaluation (LREC 2016). ELRA
(European Language Resources Association).
be generalizable and applicable to a wide range
Accessed January 08, 2019. Retrieved from
of museums and visitors. While emoji are a glo-
https://repositori.upf.edu/handle/10230/33776?
bal phenomenon and are often interpreted in locale-attribute=en
the same way across different cultures and lan- Barrett, L. F., Lewis, M., & Haviland-Jones, J. M.
guages (Barbieri, Ronzano, & Saggion, 2016), (2008). Handbook of emotions. New York, NY:
people may still understand images and facial Guilford Publications.
Daniela De Angeli, Ryan M. Kelly, Eamonn O’Neill 21CURATOR: THE MUSEUM JOURNAL
Benedek, J., & Miner, T. (2002). Measuring Damala, A., Schuchert, T., Rodriguez, I., Moragues,
desirability: New methods for evaluating J., Gilleade, K., & Stojanovic, N. (2013).
desirability in a usability lab setting. In Exploring the affective museum visiting
Proceedings of Usability Professionals’ Association. experience: Adaptive augmented reality (A 2 R)
Accessed March 01, 2015. Retrieved from and cultural heritage. International Journal of
http://sites.google.com/site/danzinde/Desirab Heritage in the Digital Era, 2(1), 117–42.
ilityToolkit.pdf Desmet, P. (2005). Measuring emotion:
Borod, J. C. (2000). The neuropsychology of emotion. Development and application of an instrument
Oxford, UK: Oxford University Press. to measure emotional responses to products,
Bradley, M. M., & Lang, P. J. (1994). Measuring January, 111–23. Accessed October 24, 2015.
emotion: The self-assessment Manikin and the Retrieved from http://dl.acm.org/citation.cfm?
semantic differential. Journal of Behavior Therapy xml:id=1139008.1139023
and Experimental Psychiatry, 25(1), 49–59. Desmet, P., Overbeeke, K., & Tax, S. (2001).
Brent Ritchie, J. R., Wing Sun Tung, V., & Ritchie, Designing products with added emotional value:
R. J. B. (2011). Tourism experience management Development and appllcation of an approach for
research. International Journal of Contemporary research through design. The Design Journal, 4
Hospitality Management, 23(4), 419–38. (1), 32–47.
Brida, J. G., Disegna, M., & Scuderi, R. (2014). The Dickinson, G. (2018). Those smiley feedback
visitors’ perception of authenticity at the buttons do actually work – and they are changing
museums: Archaeology versus modern art. the way we travel. The Telegraph, 2018.
Current Issues in Tourism, 17(6), 518–38. Accessed May 11, 2018. Retrieved from https://
https://doi.org/10.1080/13683500.2012. www.telegraph.co.uk/travel/news/happyornot-
742042 smiley-buttons-at-airports/
Bryce, D., Curran, R., O’Gorman, K., & Taheri, B. Dindar, S. (2015). Silvia Filippini-Fantoni Q&A:
(2014). Visitors’ engagement and authenticity: How the indianapolis museum of art uses visitor
Japanese heritage consumption. Tourism feedback. QuickTapSurvey. 2015. Accessed
Management, 46, 571–81. https://doi.org/10. November 19, 2019. Retrieved from https://
1016/j.tourman.2014.08.012 www.quicktapsurvey.com/blog/silvia-filippini-
Bryce, J. (2001). The technological transformation of fantoni-qa-how-the-indianapolis-museum-of-
leisure. Social Science Computer Review, 19(1), 7– art-uses-visitor-feedback/
16. Fane, J., MacDougall, C., Jovanovic, J., Redmond,
Caicedo, D. G., & Van Beuzekom, M. (2006). ‘How G., & Gibbs, L. (2018). Exploring the use of
do you feel?’ An assessment of existing tools for emoji as a visual research method for eliciting
the measurement of emotions and their young children’s voices in childhood research.
application in consumer products. Delft Early Child Development and Care, 188(3), 359–
University of Technology. Accessed November 74.
10, 2015. Retrieved from http://citeseerx.ist. Foster, H. (2008). Evaluation toolkit for museum
psu.edu/viewdoc/summary?doi=10.1.1.506. practitioners. Accessed November 12, 2015.
8025 Retrieved from http://visitors.org.uk/wp-conte
Chen, C.-F., & Chen, F.-S. (2010). Experience nt/uploads/2014/08/ShareSE_Evaltoolkit.pdf
quality, perceived value, satisfaction and Fox, E. (2008). Emotion science : Cognitive and
behavioral intentions for heritage tourists. neuroscientific approaches to understanding human
Tourism Management, 31(1), 29–35. emotions. London, UK: Palgrave Macmillan.
Del Chiappa, G., Andreu, L., & Gallarza, M. G. Galani, A., Maxwell, D., Mazel, A., & Sharpe, K.
(2014). Emotions and visitors’ satisfaction at a (2011). Situating cultural technologies outdoors:
museum. International Journal of Culture, Design methods for mobile interpretation of
Tourism and Hospitality Research, 8(4), 420–31. rock art in rural Britain. In J. Trant &
22 Article: Beyond Happy-or-Not: Using Emoji to Capture Visitors’ Emotional ExperienceYou can also read
