Conversational Agent Voting Advice Applications - University ...
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Running head: CONVERSATIONAL AGENT VOTING ADVICE APPLICATIONS
Conversational Agent Voting Advice Applications
The effect of tone of voice in (CA)VAAs and political sophistication on political knowledge,
voting intention, and (CA)VAA evaluation.
Simone van Limpt
Snr 2013738
Master’s Thesis
Communication and Information Sciences (Business Communication and Digital Media)
School of Humanities and Digital Sciences
Tilburg University, Tilburg
Supervisors: Christine Liebrecht and Naomi Kamoen
Second Reader: Rianne Conijn
July 2020
CONVERSATIONAL AGENT VOTING ADVICE APPLICATIONS
Abstract
During election times, more and more citizens consult Voting Advice Applications (VAAs) to
inform themselves about political party standpoints towards relevant political issues. VAAs have
been shown to increase political knowledge and interest in politics and enable people to make a
well-informed voting choice. However, research shows at the same time that VAA users
experience comprehension difficulties when filling out a VAA and make little effort to solve
these problems by searching for additional information. The current study developed and tested a
new type of VAA: a Conversational Agent Voting Advice Application (CAVAA). By using
technology from the field of conversational agents, this study investigated whether the additional
political information provided by CAVAAs enhances voters’ political knowledge, voting
intention, and tool evaluation. Besides, the study examined the role of tone of voice in CAVAAs
and the moderating role of political sophistication. An online experiment (N = 229) was
conducted that contained a 3 (VAA type: traditional VAA, CAVAA with formal tone of voice,
or CAVAA with conversational human voice) x 2 (political sophistication: high versus low)
between-subjects design. Results showed that CAVAAs (regardless of their tone of voice) lead to
more political knowledge and a better evaluation than traditional VAAs. However, no effect was
observed for voting intention. In addition, the study found no interaction effect for political
sophistication. Theoretical implications and practical implications for (CA)VAA developers are
discussed.
Keywords: Voting Advice Applications (VAAs), conversational agents, comprehension
process, political sophistication, political knowledge, voting intention
2
CONVERSATIONAL AGENT VOTING ADVICE APPLICATIONS
Table of contents
Abstract 2
1. Introduction 5
2. Theoretical framework 8
2.1 Voting Advice Applications 8
2.2 The cognitive process of answering VAA statements 10
2.3 Semantic and pragmatic comprehension problems 11
2.4 Comprehension problems and response behavior 12
2.5 Conversational Agent Voting Advice Applications 13
2.6 Tone of voice 15
2.7 Political sophistication 17
2.8 Conceptual model 20
3. Method 20
3.1 Design 21
3.2 Material 21
3.2.1 Political statements 21
3.2.2 Traditional VAA 22
3.2.3 CAVAAs 22
3.2.4 Tone of voice in CAVAAs 24
3.3 Pre-test 26
3.3.1 Pre-test procedure 26
3.4 Participants 27
3.5 Measurements 28
3.5.1 Political sophistication 28
3.5.2 Political knowledge 29
3.5.3 (CA)VAA evaluation 30
3.5.4 Voting intention 30
3.5.5 Factor analysis 31
3.6 Procedure 32
3
CONVERSATIONAL AGENT VOTING ADVICE APPLICATIONS
4. Results 34
4.1 Political knowledge 34
4.2 Voting intention 37
4.3 (CA)VAA evaluation 38
4.4 Additional analyses political sophistication 39
5. Discussion 40
5.1 Conclusion 40
5.2 Theoretical implications 40
5.3 Limitations and suggestions for future research 43
5.4 Practical implications 44
References 46
Appendices
Appendix A: CAVAA statements with explanations
Appendix B: Informed consent and material pretest
Appendix C: Results pretest
Appendix D: Online experiment main study
Appendix E: Chatbot conditions main study
Appendix F: Coding scheme factual political knowledge
Appendix G: Assumptions main study
4
CONVERSATIONAL AGENT VOTING ADVICE APPLICATIONS
1. Introduction
Nowadays, citizens experience difficulties in deciding which political party to vote for.
The increasing number of (new) political parties and candidates in multiparty systems has
resulted in electoral instability (Dalton, 2002). As a consequence, it has become difficult for
voters to stay informed about political issues and to make a well-informed voting decision
(Cedroni & Garzia, 2010). Moreover, there are indications that the political interest among Dutch
citizens is declining. This can be seen from, among other things, the trend of decreasing trend in
turnout rates of the Dutch Parliamentary Elections. Where in 1967, 94.4% of the Dutch citizens
turned out to vote, this percentage had decreased to 81.9% by 2017 (CBS, 2012; Kiesraad, 2017).
Voting Advice Applications (VAAs) might provide a solution for citizens with less
political interest who want to inform themselves about party positions towards relevant political
issues without reading and comparing the election programs of every party. VAAs are
accessible, user-friendly, and time-efficient online survey tools that fulfill two main functions
(Garzia, 2010; Kamoen, Holleman, Krouwel, Van de Pol, & De Vreese, 2015). First, VAAs aim
to inform voters about political parties and their viewpoints (Garzia, 2010). Secondly, VAAs
provide voters personalized voting advice based on a comparison between the political parties’
and user’s opinions on a set of political statements. Indeed, research shows that VAAs increase
political knowledge and interest and enable citizens to make a more conscious and well-informed
political-party choice (e.g., Kamoen et al., 2015).
However, the process of making a well-informed voting decision has some drawbacks.
Kamoen and Holleman (2017) found that VAA users encounter comprehension problems in circa
one in five political statements. For example, the statement: To maintain social services, the OZB
may be increased raises semantic (What does OZB mean?), as well as pragmatic (What are the
5CONVERSATIONAL AGENT VOTING ADVICE APPLICATIONS
current OZB rates?) questions. In general, traditional VAAs do not offer users additional
information and instead of consulting external sources to solve these problems, VAA users often
make assumptions about the meaning of the term, which could affect the voting advice (Kamoen
& Holleman, 2017). However, little attention has been paid to exploring solutions that make
complex political information more understandable in VAAs, and as to whether these solutions
can increase political knowledge.
Therefore, the current study develops and tests a completely new type of VAA that
provides users with additional political information: a Conversational Agent Voting Advice
Application (CAVAA). A CAVAA is a combination of a conversational agent (i.e., chatbot) and
a VAA. Just like traditional VAAs, CAVAA users can give their opinion on a set of political
statements to receive voting advice. However, to solve comprehension problems before
answering a statement, voters can request CAVAAs for additional information. For example, by
asking for the current OZB rate. Therefore, CAVAAs have the potential to help VAA users find
the right information to enlarge their political knowledge compared to traditional VAAs. The
current study is, to the best of our knowledge, the first that applies chatbot technology in a
political context in order to investigate the effect of (CA)VAAs on political knowledge and
voting intention.
Receiving additional information via CAVAAs could be particularly beneficial for people
who are already less informed about political issues. According to the National Voter Survey of
2017, political information should be made more accessible, especially for voters with low levels
of political sophistication. Political sophistication is a measure that combines people’s political
interest, political knowledge, and educational level (Lachat, 2008). Citizens with low levels of
political sophistication are overall less informed about politics, less interested in politics, and
6CONVERSATIONAL AGENT VOTING ADVICE APPLICATIONS
vote less often than citizens with high levels of political sophistication (Lachat, 2007). Since
chatbot research shows that the threshold for asking questions to a chatbot is lower than asking
questions to people (Følstad, Nordheim, & Bjørkli, 2018), interaction with a CAVAA could
make it easier for voters to receive political information and consequently making well-informed
voting decisions.
Since CAVAAs are new VAA tools, it is still unknown how they should respond towards
users. Previous chatbot studies in the customer service context have shown that the use of a more
engaging and personal communication style (i.e., Conversational Human Voice, CHV; Kelleher,
2009) can make a difference in how people experience and evaluate chatbots compared to a more
formal tone of voice (e.g., Araujo, 2018; Liebrecht & van der Weegen, 2019). Therefore, the
current study investigates how users evaluate a CAVAA containing a conversational human
voice compared to a formal tone of voice.
Overall, this study will contribute to a wider understanding of the influence of political
sophistication and tone of voice in (CA)VAAs on political knowledge, voting intention, and
evaluation. The results of the study will provide more insight into the effect of chatbots in the
political domain, but could also help VAA developers to supply and present the political
information to citizens in the most optimal way. In order to investigate this, the following
research question will be examined:
RQ: What is the effect of a traditional VAA, a formal CAVAA, and a CAVAA with
CHV on political knowledge, voting intention, and (CA)VAA evaluation, and is this relationship
moderated by users’ level of political sophistication?
7CONVERSATIONAL AGENT VOTING ADVICE APPLICATIONS
2. Theoretical framework
2.1 Voting Advice Applications
The Dutch political landscape is becoming increasingly complex due to an expansion of
political parties and blurred boundaries between the different ideologies of these parties (Dalton
& Wattenberg, 2002). Due to this complex political landscape, it has become difficult for voters
to make a well-reasoned voting choice (Cedroni & Garzia, 2010). In order to help people to
assess which political parties are most in line with their preferences, VAAs have been developed
(Holleman, Kamoen, Krouwel, Van de Pol, & De Vreese, 2016). Based on people’s answers
about a range of political statements, the VAA gives an overview of political parties that best
match the political preferences of the voter. VAAs have become increasingly popular over the
past decade. For example, in the week before the 2017 Dutch National Elections, Kieskompas
and Stemwijzer together gave voting advice to 6.85 million citizens (De Telegraaf, 2017).
However, VAAs do not only aim to provide clear voting advice. In general, VAAs have
the intention to increase citizens’ political knowledge and political interest to make a well-
informed voting choice. Political knowledge is an important predictor for participation in
politics. According to Westle (2006), only citizens who have at least a basic knowledge of
politics can participate in democratic processes in a meaningful way. The effects of voting aids
on political knowledge have been investigated in several previous studies. For example, Kamoen
et al. (2015), and Ladner, Fivaz, and Pianzola (2012) point out that VAA usage improves
political knowledge. Better political knowledge, in turn, affects voting choice and will lead to a
higher voter turnout during election times (Kamoen et al., 2015; Ladner & Pianzola, 2010).
Lassen (2005) also argues that a lack of political knowledge is the main reason for not voting.
8CONVERSATIONAL AGENT VOTING ADVICE APPLICATIONS
Political knowledge can be divided into people’s perceived (i.e., the feeling of knowing)
and factual (i.e., actually knowing) political knowledge. In the studies on the effects of VAAs
(e.g., Kamoen et al., 2015; Ladner et al., 2012; Ladner & Pianzola, 2010), respondents are
generally asked to point out to what extent they feel that the VAA has enhanced their political
knowledge in a post-VAA survey. These studies then show that VAA users experience a
knowledge increase because of VAA usage (i.e., perceived political knowledge). However, the
question arises whether this perceived political knowledge corresponds to their factual political
knowledge (i.e., having more concrete political knowledge then before). Schultze (2014)
therefore states that systematic consideration of the factual level of voters’ political knowledge is
needed to acquire a more detailed explanation of their voting behavior. The findings of his study
indicate a positive effect of the German VAA (the “Wahl-O-Mat”) on factual political
knowledge about party positions (Schultze, 2014). The current study focuses on perceived, as
well as factual political knowledge because previous studies have shown positive correlations
between perceived political knowledge and voting intention (Ladner & Pianzola, 2010).
Moreover, insight into people’s actual knowledge is needed to determine the exact contribution
of VAAs.
Next to informing citizens about political issues and increasing their knowledge and
interest in politics (Cedroni & Garzia, 2010), a number of studies have examined the effects of
VAAs on voting intention and voting advice (Krouwel, Viteillo, & Wall, 2012). An important
condition for receiving voting advice is that VAA users understand the political statements from
the VAA (Kamoen & Holleman, 2017). Still, people seem to experience a sense of
incomprehension when consulting a VAA (Kamoen & Holleman, 2017). Comprehension
problems can hinder the learning process about political issues. To understand how these
9CONVERSATIONAL AGENT VOTING ADVICE APPLICATIONS
comprehension problems appear, theories about cognitive processes in the field of survey
response psychology can be used.
2.2 The cognitive process of answering VAA statements
The “Tourangeau model” (Tourangeau, Rips, & Rasinski, 2000) is used as a starting
point to theorize about comprehension problems in VAA. The model has its origins in the
psychology of survey response literature and describes the cognitive processes that underlie the
process of answering questions in four different steps. To apply the Tourangeau model to VAAs,
the following example statement is used in this section: The Netherlands must allow the
cultivation of genetically modified crops (GMOs). In the first step, the respondent interprets the
information retrieved from the question by making a logical representation of the question (e.g.,
So apparently, GMOs are still forbidden in the Netherlands). In the second stage, respondents
have to retrieve the relevant information from their long-term memory. This stage can be
described as a sampling process that activates the most accessible beliefs. Sometimes,
respondents are able to directly activate a summary evaluation of their beliefs (e.g., A GMO is an
organism that has had its DNA modified in some way through genetic engineering.). Other
respondents, however, need an extra step (i.e., stage 3) of weighting and scaling individual
beliefs in order to form a judgment (e.g., GMOs can offer a solution for the exponentially
growing world population, so they must be allowed versus in my eyes, GMOs are not safe for
consumption). In the final stage, the respondents have to translate their judgment into an
answering option in the questionnaire (e.g., agree, neutral, disagree, or no opinion.)
To make a connection between the characteristics of a VAA statement and possible
comprehension problems, the understanding process (i.e., the first step of the Tourangeau Model)
10CONVERSATIONAL AGENT VOTING ADVICE APPLICATIONS
is an important starting point. Researchers in the field of discourse studies suggest that
respondents first construct a semantic representation of the literal meaning in their heads. Then,
this semantic representation will be enriched with world knowledge, which provides a pragmatic
representation of the question (Graesser, Singer, & Trabasso, 1994; Kintsch, 1994; Krosnick,
2018). According to Tourangeau et al. (2000), comprehension problems can arise both during the
creation of the semantic representation, as well as during the construction of the pragmatic
representation.
2.3 Semantic and pragmatic comprehension problems
Kamoen and Holleman (2017) found in their study that two-thirds of the comprehension
problems in answering political statements were semantic, meaning that they are related to the
literal meaning of words in VAA statements. For example, making use of political jargon (e.g.,
GMOs) or tax names (e.g., OZB). The other problems were related to the pragmatic
representation of the statements. Here, respondents did understand the literal meaning of all the
terms in the VAA statements, but had too little background knowledge on the issue to answer the
question (e.g., the exact tax height). Hence, when VAA users are facing issues regarding the
construction of a semantic representation, or when they make an interpretation that does not
correspond to the interpretation of political parties, the answers to these questions are an invalid
base for the voting advice (Kamoen & Holleman, 2017).
Since understanding the question is the first step to come to an optimal answer
(Tourangeau et al., 2000), it is likely that problems that appear at this stage will continue in the
following phases (Lenzner, Kaczmirek, & Galesic, 2011). For example, if a respondent does not
understand the term GMO, it is also difficult for this respondent to retrieve the relevant
11CONVERSATIONAL AGENT VOTING ADVICE APPLICATIONS
knowledge and subsequently come to an informed assessment of the statement (Jabine, Straf,
Tanur, & Tourangeau, 1984). In this situation, offering information in a semantic and/or
pragmatic way that meets the demands of the users can improve the retrieval process, and as a
result, make the answers of respondents more accurate (Schober & Conrad, 1997; Schober,
Conrad, & Fricker, 2004).
2.4 Comprehension problems and response behavior
To improve the comprehension and retrieval process (i.e., step 2 of the Tourangeau
model), VAA users may benefit by reading additional information about political statements.
However, VAAs differ in the extent to which they provide information to users. For example,
Stemwijzer gives the opportunity to view how the various political parties think about the issue
with a brief explanation of their positions. However, this type of information is not semantic, nor
pragmatic. Kieskompas, in contrast, does not provide additional political information because
they want voters to form their own opinions without being influenced by information from the
political parties (Krouwel et al., 2012). Hence, this implies that the specific need for semantic
and pragmatic information has not been fulfilled in VAAs. Therefore, respondents have to
consult external information sources of information, such as Google, to receive additional
information when they do not understand a term, which costs more effort (Kamoen & Holleman,
2017).
Furthermore, despite the possibilities that VAAs offer to obtain additional information,
users make little use of this extra information (Lenzner, 2012; Kamoen & Holleman, 2017). This
matches with what is known from traditional survey research. For example, Galesic, Tourangeau,
Couper, and Conrad (2008) showed with an eye-tracking study on survey responding that two-
12CONVERSATIONAL AGENT VOTING ADVICE APPLICATIONS
thirds of the respondents did not look at the definitions, which became visible when they moved
over them with their computer mouse. Additionally, the study showed that the more effort is
needed to access the definition, the less likely participants were to look into the available
information.
The fact that respondents spend only a minimal effort in answering the statements
matches the idea of Krosnick (1991) who stated that survey participants often demonstrate
satisficing behavior. This means that people just spend enough effort to deliver an acceptable
answer that satisfies the survey investigator. Thus, rather than seeking for more information
about unclear terms, survey respondents use sub-optimal response strategies which take little
effort. For example, they (1) make assumptions about the meaning of the term, and (2) opt for
the neutral or non-response answering option (Baka, Figgou & Triga, 2012; Van Outersterp et
al., 2016; Kamoen & Holleman, 2017). This behavior is not desirable and could affect the voting
advice (Van Outersterp et al., 2016). Therefore, it is a challenge to develop VAAs that provide
political information in a least effort way.
2.5 Conversational Agent Voting Advice Applications
Technologies that can offer additional information in a low-threshold way are chatbots,
also known as conversational agents, chatterbots, digital assistants, or virtual agents. Chatbots
are conversational agents that use natural language to interact with humans (Dale, 2016). Based
on input from users, chatbots provide pre-programmed answers. Chatbots have become
increasingly popular over the past decades. Nowadays, they are present in different contexts,
such as business, e-commerce, education, healthcare, and customer services, and offer help for
different purposes (Kerlyl, Hall, & Bull, 2007; Shawar & Atwell, 2007). For example, in the
13CONVERSATIONAL AGENT VOTING ADVICE APPLICATIONS
field of e-commerce and customer service, chatbots can operate as online shopping assistants to
provide customers with extra information about products or services in order to find the products
or services that fits best with their needs (Bogdanovych, Simoff, Sierra, & Berger, 2005). In
addition, chatbots can serve to help customers navigate through websites, decrease the number of
clicks, and reduce the time to find the desired information (Lasek & Jessa, 2013). Due to these
dynamic characteristics of chatbots, users have to make less effort to obtain information
(Brandtzaeg & Følstad, 2017). And this easy accessibility seems to correspond to the mindset of
VAA users when completing a VAA: they want to become informed about the political issues at
stake in an efficient and easy way (Kamoen & Holleman, 2017). Therefore, the current study will
explore the feasibility of chatbots for VAAs in the field of politics. We will name them
Conversational Agent Voting Advice Applications (CAVAAs).
CAVAAs are traditional VAAs implemented in a chatbot. The results of the study
conducted by Galesic et al. (2008) indicate that users are less likely to look at additional
information when the more effort is required to access the definition. Compared to traditional
VAAs, chatbots have the ability to transport data in a simple way from computer to human
without searching several web pages to collect information (Dahiya, 2017). Therefore, users can
easily ask their questions about the content of political statements and receive the required
information in return. It can be assumed that CAVAAs have the potential to provide complex
political information in a cost-efficient and understandable way in order to increase political
knowledge among users and to provide more personalized voting advice, compared to traditional
VAAs. Therefore, in line with Galesic et al. (2008) the following hypothesis is formulated:
14CONVERSATIONAL AGENT VOTING ADVICE APPLICATIONS
H1: After using a CAVAA, users will report more (a) perceived and (b) factual political
knowledge compared to a traditional VAA.
Since CAVAAs are a new phenomenon, it is important to investigate how people evaluate
CAVAAs and their intention to use CAVAAs in the future. Based on the studies of Dale (2016)
and Bogdanovych et al. (2005), CAVAAs can provide a more natural way of interacting
compared to traditional VAAs and can act as a personal assistant to help users find the right
information and enhance their political knowledge. Hence, we propose the following hypothesis:
H2: Users will evaluate CAVAAs more positively than a traditional VAA.
2.6 Tone of voice
Since CAVAAs are completely new tools, it is still unknown how the voting application
should respond to users in order to keep them motivated when filling out CAVAAs. It is
important that people are motivated when conducting CAVAAs, so they can process the political
statements in a careful way. This could be achieved by approaching CAVAA users with a
personal and engaging communication style, also known as tone of voice. Tone of voice is
widely investigated in the domain of organizational communication and it has been shown that
using a Conversational Human Voice (CHV) can make a difference in how people experience
chatbots (e.g., Araujo, 2018; Liebrecht & van der Weegen, 2019). Kelleher (2009) defines CHV
as “an engaging and natural style of organizational communication as perceived by an
organization’s publics based on interactions between individuals in the organization and
individuals in publics” (p. 177). CHV can be operationalized by several linguistic elements,
15CONVERSATIONAL AGENT VOTING ADVICE APPLICATIONS
which can be subdivided into three categories: personalization (e.g., personally addressing the
user; Hi Simone), informal language (e.g., the use of emoticons and interjections; haha :)), and
invitational rhetoric (e.g., stimulating dialogue by using humor or asking questions; Can I help
you?) (Van Noort, Willemsen, Kerkhof, & Verhoeven, 2014; Van Hooijdonk & Liebrecht,
2018).
Empirical research in other domains than politics (e.g., public relations, computer
science, and communication studies) shows that CHV is important in creating positive
evaluations. For example, Kelleher (2009) found a positive effect of CHV in blogs on customer
satisfaction and brand attitude. Here, satisfaction is defined as the extent to which the reader of
the blog is positive about the brand and has positive expectations about the brand. Furthermore,
in the study of Schneider (2015), CHV appears to be a key factor in enhancing positive attitudes
towards the reputation of an organization. Others (Liebrecht & Van der Weegen, 2019) have
highlighted the relevance of CHV usage in human-computer interaction. They found that
chatbots significantly enhanced brand attitude in the domain of customer service. Based on these
results from other domains, it can be expected that the use of CHV in CAVAAs will also have
positive effects on people’s attitudes towards the new tools in the political context. Therefore, the
following hypotheses will be investigated:
H3: Users of CAVAAs will evaluate the tool more positively when CHV is used compared to a
formal tone of voice.
Furthermore, it is known from lexical decision-making literature that personally
addressing users is an important driver for information processing and, in turn, makes
16CONVERSATIONAL AGENT VOTING ADVICE APPLICATIONS
information more understandable (Andrews, 1988). When people are able to process information
in an accessible way, this will improve the understanding of that information (Schwanenflugel et
al., 1988). Additionally, research in the field of surveys shows that people will put more effort
into understanding and answering statements when a personal communication style (e.g., using
personal pronouns) is used (Krosnick, 2000). Therefore, it could be expected that CAVAA users
will put more effort into the understanding and processing of complex political statements when
CAVAAs apply a conversational human voice.
In short, because personal communication is an important factor for information
processing, it is expected that using CHV in CAVAAs will make political information more
understandable and will, in turn, lead to more political knowledge. This reasoning leads to the
following hypothesis:
H4: After using a CAVAA, users will report more (a) perceived and (b) factual political
knowledge when CHV is used compared to a formal tone of voice.
2.7 Political sophistication
The process of translating interests and preferences into a vote involves a certain effort
from citizens: they have to collect information about political party positions, qualities of the
individual candidates, and information about the political topics themselves. Citizens vary in
their amount of cognitive resources to collect and process this multitude of complex information
to cast a meaningful vote (Luskin, 1987). Especially those with limited individual dispositions to
collect relevant information could profit the most from CAVAAs. In this study, the term
17CONVERSATIONAL AGENT VOTING ADVICE APPLICATIONS
“political sophistication” is used to make a classification between different types of voters with
divergent cognitive capacities.
However, there is an ongoing debate about the best operationalization of the concept of
political sophistication. Some measure political knowledge by counting right answers to a series
of knowledge questions (McGraw, Lodge, & Stroh, 1990); others pose a number of questions
about political interest with a measure of educational level (Holleman et al., 2016; Lachat, 2008;
MacDonald, Rabinowitz, & Listhaug, 1995). Luskin (1990), however, suggests three aspects to
measure the individual development of political information, namely: the cognitive ability to
understand information (usually measured with level of education), an informative aspect
(usually measured as political knowledge), and the motivation to put effort into the collection of
this information (usually measured as political interest). This study will take all three indicators
into account to determine voters’ degree of political sophistication because previous studies have
already found positive effects of political interest, political knowledge, and level of education on
electoral participation (Söderlund, Wass, & Blais, 2011). Thus, someone with a high level of
education, a high degree of interest in politics, and much political knowledge is characterized by
a high level of political sophistication. Someone with a low level of political sophistication is
low educated, has little interest in politics, and does not have sufficient political knowledge.
In the current study, we expect a positive contribution of CAVAAs compared to VAAs
on political knowledge and voting intention. Overall, political sophistication lowers the cognitive
cost of voting (Denny & Doyle, 2008). To illustrate, sophisticated voters are more interested in
politics and collect information about the political system themselves, even outside the election
times (Luskin, 1990; Söderlund et al., 2011). Therefore, it can be assumed that this group (higher
sophistication) benefits less from the additional information that is given by CAVAAs.
18CONVERSATIONAL AGENT VOTING ADVICE APPLICATIONS
Furthermore, sophisticated voters generally have more political knowledge than those who stay
at home during elections. When homestayers (i.e., low sophisticated citizens) do consult
CAVAAs to receive more information in order to enhance their political knowledge and
understand the political statements, it could make them more confident and motivated to go vote.
Therefore, it is expected that CAVAAs are especially relevant for people with low levels of
political sophistication. Lastly, educational level seems to have a positive influence on political
knowledge and electoral participation (Blais, 2000; Galego, 2010). A higher level of education
ensures a better development of voters’ cognitive capacities which increases the ability to deal
with complex political information (Armingeon & Schädel, 2015). In sum, it can be stated that
previous research clearly indicates that each of the three indicators of political sophistication by
Luskin (1990) are positively associated with political knowledge and voting intention.
So, compared to highly sophisticated people, people with low levels of political
sophistication have less cognitive capacity which decreases the ability to deal with political
information (Armingeon & Schädel, 2015). In addition, they do not seek political information
themselves (Söderlund et al., 2011). Therefore, it can be assumed that these so-called low
information voters will feel more inclined to use a CAVAA, as they will experience a greater
need for information to simplify their voting decision than voters with high levels of political
sophistication. Therefore, the following hypothesis is formulated:
H5: The effect of CAVAAs versus VAAs on political knowledge and voting intention is
moderated by political sophistication, such that CAVAAs lead to more political knowledge and a
higher voting intention among people with low levels of political sophistication than among
people with high levels of political sophistication.
19CONVERSATIONAL AGENT VOTING ADVICE APPLICATIONS
2.8 Conceptual model
The conceptual model of the current thesis project is shown in Figure 2. The aim of the study is
to examine the effect of CAVAAs and tone of voice on political knowledge, voting intention and
evaluation compared to traditional VAAs. In addition, the moderating role of political
sophistication will be explored.
Voting Advice Application Outcome variables
H4
Formal CAVAA Political knowledge
CAVAA with CHV H3
(CA)VAA evaluation
H1 + H2
Traditional VAA
Voting intention
H5
High
Political
sophistication
Low
Figure 1. Conceptual model
3. Method
20CONVERSATIONAL AGENT VOTING ADVICE APPLICATIONS
3.1 Design
To test the hypotheses, an online experiment was conducted with a 3 (type of VAA: traditional
VAA, CAVAA with formal tone of voice, or CAVAA with conversational human voice) x 2
(political sophistication: high versus low) between-subjects design. Each participant was
randomly assigned to one of the three VAA conditions. The independent variable political
sophistication is a quasi-experimental variable, which was measured afterwards. Every
participant filled out an online questionnaire with items regarding political knowledge, voting
intention and (CA)VAA evaluation, after interacting with the CAVAA or VAA.
3.2 Material
3.2.1 Political statements. Twenty different statements were derived and adapted from
the statements that were used in Kieskompas and StemWijzer during the 2017 Dutch National
Elections. Statements were only included if the topic was still relevant today. For example, the
statement “Stricter climate legislation is needed, even though it is at the expense of economic
growth” was not included, since the Dutch cabinet has already accepted the climate plan
proposal for the period 2021-2030. Besides, additional statements about the corona crisis were
added to the voting aid because this topic is currently dominating the news.
Moreover, only statements that were expected to cause comprehension problems were
selected for this study. Previous studies have shown that difficult concepts (semantic) and
missing information (pragmatic) can lead to problems in completing VAAs (Kamoen &
Holleman, 2017). Therefore, the statements had to include political jargon or tax names
(semantic problems). Besides, words like “increase”, “lower”, or “abolish” were added in the
statements to make an explicit reference to the status quo (e.g., The corporate income tax should
21CONVERSATIONAL AGENT VOTING ADVICE APPLICATIONS
decrease further after 2021). The additional semantic and pragmatic information provided was
based on reliable sources, such as the Dutch Ministry of Defense and Statistics Netherlands
(CBS) (see Appendix A). Furthermore, the explanations were formulated in a neutral way which
means they did not favor one side of the political coin. An overview of example statements
including semantic and pragmatic explanations can be found in Appendix A (in Dutch). The
length of the explanations varied between 20 and 60 words (M = 38.95, SD = 15.06).
3.2.2 Traditional VAA. One of the experimental conditions was the traditional VAA
with 20 political statements. Participants were informed that the VAA was especially designed
for research purposes and that political parties were not involved in the development of the
VAA. The VAA was created and distributed with Qualtrics software. We assured that the layout
was comparable to that of a Dutch VAA (e.g., Stemwijzer or Kieskompas). For example, the
response options were derived and adapted from Stemwijzer (i.e., “agree”, “neutral”, “disagree”,
“no opinion”), and the VAA provided a voting advice based on the answers given by the
participants. However, to underline that the VAA was built for research purposes, the logo of
Tilburg University was visible in the voting aid. To realize the voting advice, the websites of
eight political parties (4 left-wing parties and 4 right-wing parties) were visited to check the
attitudes of the different parties towards every political statement.
3.2.3 CAVAAs. Based on the traditional VAA, two CAVAA versions were developed
with the software of Flow.ai. Flow.ai is a visual bot platform for creating AI chatbots for
Facebook, WhatsApp and internet. The main difference between the traditional VAA condition
and the two CAVAA conditions is the chatbot function. In a CAVAA, users are able to chat with
a chatbot in order to receive more information about the political statements. By using this
software, chatbots can be developed and trained to recognize different formulations of their
22CONVERSATIONAL AGENT VOTING ADVICE APPLICATIONS
conversational partners. Messages from the users activate a certain conversational flow within
the chatbot, which in return sends pre-programmed messages.
The CAVAA interface was developed based on previous chatbot research. For example,
it is important that a chatbot starts with a proactive greeting to get the user’s attention (Thies,
Menon, Magapu, Subramony, & O’neill, 2017). Therefore, the CAVAAs started the
conversation with: "Hello, nice that you are going to fill in this voting aid!” Followed by: "Are
you ready to start?". Users were offered two buttons that said, "Yes, I'm ready!", and "No, I'll
come back another time. In this manner, users were forced to actively participate in the
conversation with the CAVAA. An example of a CAVAA conversation is shown in Figure 2.
In addition, it is important that a chatbot responds invitational when a question is not
understood (Cahn, 2017). When a chatbot does not respond in an invitational way, users could
get stuck in the conversation. For this reason, the following chatbot answer was included: "Sorry,
I don’t know what you want to ask. At least, you can ask me a question about: the meaning of the
term, the current state of affairs, advantages, and disadvantages." In this way, the CAVAA was
given the opportunity to properly interpret new input, allowing the conversation to continue.
Research by Følstad et al. (2018) has shown that it is important to be transparent about the
functions and limitations of a chatbot. That is why the CAVAA indicated where users could ask
questions about.
Lastly, before the CAVAA started with the political statements, an explanation was given
about the response options of the statements. Research about VAAs shows that VAA users most
of the time do not know the difference between the “neutral” and “no opinion” answer options
(Kamoen & Holleman, 2017). For this reason, both answer options were explained before users
started completing the statements, like: “Did you know that the answer option 'no opinion' in
23CONVERSATIONAL AGENT VOTING ADVICE APPLICATIONS
VAAs ensures that the statement is not included in the calculation of the voting advice?”. After
this explanation, the first political statement was offered to the user.
3.2.4 Tone of voice in CAVAAs. For this study, two CAVAA versions have been
developed. One CAVAA used a formal tone of voice, and the other CAVAA used a
conversational human voice. Table 1 describes the characteristics of the two different tone of
voices used by the two CAVAA-versions. They are divided into three categories defined by Van
Noort et al. (2014), namely: personalization, informal language, and inviting rhetoric. The
conversations with the CHV-CAVAA were more personal, informal, and engaging than the
CAVAA that uses a formal tone of voice. To illustrate, some CHV elements were absent in the
formal CAVAA (e.g., smileys), and some words were replaced by more formal words in the
formal version (e.g., word choice). The CAVAA versions only differed in tone of voice only in
the parentheses of the conversation, not in the statements or in the additional information.
An example of the two different conversations can be found in Figure 2 and Appendix E.
As you can see in the figure, the left CHV-CAVAA has an avatar and a name (i.e., Voty).
Compared to the formal-CAVAA (right), the CHV-CAVAA addresses users with their first
name (i.e., Hallo Simone), and makes use of emoticons (i.e., :)), personal pronouns (i.e., je
versus u), and interjections (i.e., top!).
Table 1
Tone of voice characteristics (CHV and formal) translated in Dutch into CAVAAs
Conversational Human Voice- Formal-CAVAA Source
CAVAA
Personalization
• Name (Voty) • Name (VoteBot) • Araujo (2018)
24CONVERSATIONAL AGENT VOTING ADVICE APPLICATIONS
• Avatar • No avatar • Araujo (2018)
• Start the conversation • Start the conversation • Van Hooijdonk &
with a personal greeting with a greeting Liebrecht (2018);
(Hi, name) Liebrecht & Van der
Weegen (2019)
• Using personal • Using personal • Van Hooijdonk &
pronouns (You/Je) pronouns (You/U) Liebrecht (2018)
Informal language
• Emoticons (:-)) • No emoticons • Van Noort et al.
(2014)
• Informal interpunction • No informal • Van Hooijdonk &
(!) interpunction Liebrecht (2018)
• Sound mimicking • No sound mimicking • Van Noort et al.
(Wow) (2014)
• Capitals (YES) • No capitals • Van Hooijdonk &
Liebrecht (2018)
Invitational rhetoric
• Stimulating dialogues • No stimulating • Van Hooijdonk &
(I am happy to help dialogues Liebrecht (2018)
you)
• Humor (Lol!) • No humor • Kelleher & Miller
(2006)
• Respond to thank you • Thank you messages • Van Hooijdonk &
messages (You are will not be recognized Liebrecht (2018)
welcome)
• Well-wishes (Have a • No well-wishes • Van Hooijdonk &
nice day!) • No sympathy Liebrecht (2018)
• Sympathy (Enjoy!)
25CONVERSATIONAL AGENT VOTING ADVICE APPLICATIONS
• Kelleher & Miller
(2006)
Figure 2. Sharing conversations with CHV-CAVAA (left) and formal-CAVAA (right).
3.3 Pre-test
3.3.1. Pre-test procedure. Before the main experiment was carried out, a qualitative pre-
test of the CAVAA was conducted in the form of an interview (Appendix B). The pre-test had
three goals: to verify whether the self-developed chatbots were well programmed, to check if the
additional background information provided with each statement met the objective of making the
26CONVERSATIONAL AGENT VOTING ADVICE APPLICATIONS
political statements more understandable, and to check if information was missing in the
explanations.
Ten participants took part in the pretest. The sample consisted of 4 males and 6 females
with an average age of 34.8 years (SD = 5.36). During the pre-test, participants had a
conversation with a chatbot and answered twenty political statements. After reading each
individual statement, participants were asked if they understood the political statement and if
they had a need for additional information (Appendix B). Thereafter, participants could click on
an information button (if desired) and had to read the explanation. After reading the additional
information, several questions were asked about the clarity and understandability of the
explanations (Appendix B). This procedure was repeated for each political statement. Lastly,
participants were asked to evaluate the new voting tool. The pretest ended with a short thank you
note by the researcher. Based on the results of the pretest, several design changes were made for
the main study, see Appendix C.
3.4 Participants
The online questionnaire was distributed via e-mail and social media by the researcher
(convenience sampling). The data were collected online between May 14th 2020 and June 1
2020. The only requirement to participate in this study was a minimum age of 18 years old (i.e.,
participants should be entitled to vote). A total of 325 participants took part in the main
experiment but only 231 of them completed the full survey. These remaining participants were
checked on straight-lining response behavior (i.e., giving the same answer to all attitude
questions). Based on this criterium, two more participants were removed from the dataset.
27CONVERSATIONAL AGENT VOTING ADVICE APPLICATIONS
Of the remaining 229 participants, 78 were male (34.1%), and 150 female (65.5%). One
person answered that he/she would rather not say (0.4%). The average age was 30.43 years old
(SD = 14.51), ranging from 18 to 75. Most of the participants received an education at University
level, as they finished an undergraduate program (52.9%) or a master’s program (26.2%). The
remaining participants finished intermediate vocational education (12.7%) or high school (8.4%).
Also, the majority of participants was familiar with VAAs (89.5%) and had previous experience
with chatbots (61.6%).
Analyses showed that participants in the three VAA-conditions were comparable for
participants’ gender (χ2 (4) = 3.51, p = .48), level of education (χ2 (12) = 5.99, p = .92), age (F(2,
226) = 3.67, p = .03), familiarity with chatbots (χ2 (2) = .88, p = .64), and previous experience
with voting advice applications (χ2 (2) = 3.03, p = .22). This implies that there are no a priori
differences between participants in the three conditions.
3.5 Measures
3.5.1 Political sophistication. Political sophistication is a theoretical construct that
consists of three aspects: a cognitive aspect, usually measured as educational level, an
informative aspect, usually measured as political knowledge, and a motivational aspect, usually
measured as political interest (Stiers, 2016; Rapeli, 2013; Luskin, 1990). Although previous
studies investigating political sophistication did not always focus on all three aspects, this study
does take all three aspects into account. Before the start of the experiment, VAA users indicated
to what extent they are interested in politics. Political interest was explicitly measured with 3
items (e.g., I am interested in politics) on a 7-point Likert scale (1 = “completely disagree”, 7 =
“completely agree”) based on Lachat (2008) and Shani (2012). This scale showed good
28CONVERSATIONAL AGENT VOTING ADVICE APPLICATIONS
reliability (α = .89, M = 4.78, SD = 1.53). Furthermore, political knowledge was measured with
seven political knowledge questions (e.g., There are 225 members in the House of
Representatives) where people could indicate if they thought the statement was “true” or “false”.
The answers on the seven factual political knowledge questions were recoded. So, people could
receive 1 point if they gave the right answer to the question (highest score = 7). Afterwards, the
question was asked about their highest finished degree of education. Then, all three variables
were combined into a new additive sophistication measure with a seven-point scale (see:
Holleman et al., 2016). The new variable political sophistication was split into a new binary
variable (1= “high level of political sophistication”, 2= “low level of political sophistication”) by
means of a median split (µ = 16.00). Based on the median score, the participant group was
divided into two new subgroups: one with high levels of political sophistication (N = 122) and
one with low levels of political sophistication (N = 107).
Additional analyses show that the participant groups with low versus high levels of
political sophistication were comparable for gender (χ2 (2) = 4.22, p = .12), age (F(1, 227)CONVERSATIONAL AGENT VOTING ADVICE APPLICATIONS
perceived and factual knowledge (e.g., Park, 2001). So, when people feel that they have more
knowledge, it does not necessarily mean that users actually know more than they did before. In
order to examine the exact contribution of CAVAAs to political knowledge, the current study
focuses on both types of knowledge.
Factual political knowledge was measured using six open knowledge questions based on
the political statements presented in the (CA)VAA (i.e., What is a binding referendum? and
What is the current state of affairs regarding the retirement age?). The answers given by the
participants were checked and coded by two people based on a coding scheme (Appendix F).
Every right answer was coded as “1” (e.g., 66.4 or 66 years) and wrong answers were coded as
“0” (e.g., 67 years). To determine consistency among the raters, an interrater reliability using
was calculated. In total, 120 answers (8.7%) were compared, see Table 3 in Appendix F. The
interrater reliability of the coding was found to be κ =.85 (pCONVERSATIONAL AGENT VOTING ADVICE APPLICATIONS
3.5.4 Voting intention. The next dependent variable in the current study was “intention
to vote”. In order to assess this intention, two items from Glynn, Huge, and Lunney (2009) (i.e.,
If there were elections now, I would vote and After consulting the (CA)VAA, I feel sufficiently
informed to vote) and one additional item (i.e., I plan to vote in the upcoming elections on March
17, 2021) were used and adapted in the current thesis. The items were measured on a 7-point
Likert scale ranging from 1 (“strongly disagree”) to 7 (“strongly agree”).
3.5.5 Factor analysis. The factor structure of political interest, perceived political
knowledge, voting intention, and (CA)VAA evaluation was assessed by performing a principal
component analysis with Varimax rotation. The results of this analysis are specified in Table 2.
The analysis revealed four factors that together explained 74% of the variance. The four factors
partially matched the predetermined factor structure.
The three items that were supposed to measure “political interest” indeed clustered well
together, and the scale showed good reliability (α = .89, M = 4.78, SD = 1.53). Regarding
perceived political knowledge, the factor analysis revealed that the three items also clustered
well together. Overall, the scale of perceived political knowledge showed good reliability (α =
.79, M = 4.28, SD = 1.35). Moreover, the five items that were supposed to measure “(CA)VAA
evaluation” revealed a good coherence. The scale had a good reliability, Cronbach’s α = .87 (M =
5.14, SD = 1.14).
The items that were expected to be related to “voting intention”, however, fell apart into
two dimensions: items measuring “voting intention” (VI1, VI2) and items measuring “political
interest” (VI3), see the boldfaced part in Table 2. In addition, reliability analysis using
Cronbach’s alpha of voting intention showed an acceptable reliability (α = .66). However, one
item (i.e., I feel sufficiently informed to go vote) remarkably decreased this reliability. Therefore,
31CONVERSATIONAL AGENT VOTING ADVICE APPLICATIONS
we decided to remove this item. The new voting intention scale consisted of two items (i.e., If
there were elections now, I would vote and I intend to vote in the upcoming elections on March
17, 2021) and showed a good reliability (α = .93, M = 6.35, SD = 0.99).
Table 2
Results principal component analysis with Varimax rotation
Factor 1: Factor 2: Factor 3: Factor 4:
(CA)VAA Political Interest Perceived Voting
evaluation knowledge intention
PI1_interesse .89
PI2_aandacht .81 .22
PI3_volgen .91
PK1_begrijpen .82
PK2_kennis .27 .87
PK3_motiveren .71
VI1_stemmen1 .31 .90
VI2_stemmen2 .26 .90
VI3_geïnformeerd .62
EV1_betrokken .69 .26
EV2_nuttig .83
EV3_leuk .76 .37
EV4_makkelijk .80
EV5_aanbevelen .84 .31
Note. Only factor loadings >.25 are included in the table; the interpretation has been boldfaced.
PI = political interest; PK = perceived political knowledge; VI = voting intention; EV =
evaluation.
3.6 Procedure
32CONVERSATIONAL AGENT VOTING ADVICE APPLICATIONS
At the start of the experiment in Qualtrics (see Appendix D), participants first read an
introductory text stating the aim of the study and procedure of the study. Next, an informed
consent stated that that participants would remain anonymous, that participation was on
voluntary basis, that they had the right to drop out during the experiment at any moment, and that
the study was approved by the Research Ethics and Data Management Committee of Tilburg
University (identification code: REDC # 2020/060). In addition, participants had to declare to be
18 years or older and give permission for data processing and storage.
Then, participants were randomly assigned to one of the three (CA)VAA conditions in
Qualtrics and were asked to provide some demographic information about their age, gender,
level of education, political interest, and familiarity with VAAs and chatbots. Thereafter,
participants were linked to the chatbot conversation or traditional VAA and had to give their
opinion towards 20 political statements. A disclaimer told participants in the CAVAA conditions
that they would temporarily leave the questionnaire and a new screen would open.
When participants clicked on the link, a new window popped-up and the chatbot started
the conversation with a greeting (see Appendix E). After the chatbot gave a description about
how to answer the political statements, a set of twenty political statements followed. At every
statement, participants could ask the chatbot for extra information when they experienced
difficulties regarding the comprehension of the statement. Another opportunity to obtain
additional information was by clicking on the semantic or pragmatic button. After completing the
political statements, participants received a personal voting advice. Subsequently, participants
were informed that they could return to Qualtrics for some additional questions about their
experiences with the (CA)VAA.
33CONVERSATIONAL AGENT VOTING ADVICE APPLICATIONS
Back in Qualtrics, participants evaluated the (CA)VAA. Participants were asked how
they experienced the interaction with the (CA)VAA and whether they would recommend the
(CA)VAA to others. Also, questions were asked to indicate their intention to vote during the
upcoming elections. Next, participants’ perceived and factual political knowledge were measured
with several knowledge questions. Lastly, participants could write some final remarks on the
study. They were thanked for their participation and debriefed. The debriefing consisted of a
short text that explained the research goals and manipulations of the study. Besides, it was
highlighted in the debriefing that created (CA)VAA was developed for research purposes and did
possibly did not provide a valid voting advice. In total, the study in the VAA condition took on
average 10.49 minutes (SD = 10.65), and endured on average 15.12 minutes (SD = 11.98) in the
CAVAA conditions.
4. Results
4.1 Political knowledge
First, the perceived political knowledge was investigated. To test the hypotheses, a two-
way ANOVA was performed with VAA type and political sophistication as independent
variables and perceived political knowledge as dependent variable. An overview of the mean
score and standard deviations of the variables can be found in Table 5. The assumptions were
checked (see Appendix G) and showed some skewness and kurtosis. However, given that our
sample size was reasonable, the ANOVA would be fairly robust against this violation.
The ANOVA showed a significant main effect of VAA type, F(2, 223) = 7.44, p = .001,
ηpartial2 = .06. Pairwise comparisons revealed that the traditional VAA differed significantly from
the formal CAVAA (Mdif = -.58, 95% CI [-.99, -.17], p = .01) and CHV CAVAA (Mdif = -.78,
34You can also read
