An Evaluation of Social Return - using Willingness to Pay December 2017 - Provident Financial
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
An Evaluation
of Social Return
using Willingness to Pay
December 2017
Acknowledgments
This analysis was carried out by
morethanoutputs, led by Tim Goodspeed,
with support from the London School of
Economics and Scotinform.
The data collection surveys were
administered and hosted online by
ScotInform. Data collection was carried out
by ResearchNow and Teamsearch Fieldwork.
2
Contents
1 Summary .................................................................................................................................. 4
2 Introduction ............................................................................................................................. 8
3 Scope ..................................................................................................................................... 10
4 Method .................................................................................................................................. 12
5 Results ................................................................................................................................... 20
6 Social Return .......................................................................................................................... 28
7 Benchmarking ........................................................................................................................ 29
8 Application and Recommendations ...................................................................................... 32
9 Bibliography ........................................................................................................................... 34
3
This evaluation followed the Social Return on
1 Summary Investment (SROI) process, where applicable
to the scope.
When we talk about the value of arts and
In the same way that a financial account is
culture to society, we always start with its
required to inform decisions to grow financial
intrinsic value . . . However, we also
performance; the social return of an activity is
understand that arts and culture has a wider,
much more than just a number – it can
more measurable impact on our economy,
provide a decision-making framework to
health and wellbeing, society and education.
increase the social return.
(Arts Council England, 2014).
1.1 Results
This broader definition of value aligns with
Data from 1,113 people (collected with online
the concept of value used in cost-benefit
surveys and interviews) showed strong
analysis (CBA), the standard evaluation
evidence about the value and positive impact
methodology used in UK policymaking, and
of the Festival across the cohort of interest; to
Social Return on Investment (SROI).
include not only Attendees and Schools, but
also Bradford Residents.
This evaluation was commissioned by
Provident Financial (PFG), sponsor of Bradford
1.1.1 Profile
Literature Festival (BLF), to measure the value
An estimated 17,612 people attended the
and social return that the Festival creates in
Festival during 10 days in July 2017 (including
the following areas of interest (outcomes):
5,745 in the Schools Programme).
Inclusion and diversity
Literacy and engagement
Profiles of Attendees (excluding the Schools
Perception of Bradford City
Programme) showed an audience profile that
Contribute to the tourism economy
4
is young, ethnically diverse and from a range The approach is recognised by UK
of income levels. Two thirds were female, Government in the Green Book (HM Treasury,
and two thirds were children. The Festival 2003 (updated 2011)) and recognised as fit-
succeeded in attracting more people in the for-purpose for use by companies in the Social
subgroup Asian Heritage (as an example of Capitol Protocol (WBCSD, 2017).
Black, Asian and Minority Ethnic groups
(BAME) within the Mosaic data) than would Value of the Festival
be expected given the profile of the regional Survey respondents (Attendees, Residents
population. and Teachers) were asked their willingness to
pay for the Festival (for a year).
1.1.2 Outcomes
Outcomes were measured with Likert scales. Attendees and Residents were asked
Combined Likert scores can be compared for comparable questions. Residents were willing
the outcomes tested, as shown in figure 1. (In to pay less (mean WTP £11.24) than
this presentation, a score of 0 would Attendees (mean WTP £19.63). (This result
represent an average response of neither was statistically significant).
agreeing or disagreeing with the statement. A
score of 1 would represent an average Teachers were asked, in their professional
response of agreeing with the statement, and capacity, how much they (their School) would
so on). be willing to pay for the Schools Programme.
Their result (mean WTP £197.30) is not
comparable with the values for individuals
1.1.3 Value
and households (Attendees and Residents).
Contingent Valuation, Willingness to Pay
(WTP) was found to be a viable approach for
Value of Outcomes
measuring economic values in the context of
Likert sums of outcomes were regressed
cultural institutions (Crossick & Kaszynska,
against the Willingness to Pay responses to
2013) (Bakhshi, 2015) and was selected as the test for a relationship between the value
technique to value outcomes for this scope. stakeholders put on the Festival and the
5
outcomes they achieved. Statistically described by the outcomes tested. (This is to
significant results for a relationship between be expected: WTP for the whole is likely to
Willingness to Pay and all the outcomes were encompass a range of other use and non-use
found for Attendees and Bradford residents. values (Sugden, 2005) (Bateman, 2002)).
Statistically significant results were not found
for the outcomes tested for Schools as the Contribution to the Tourism Economy
sample size (30) was too small. Festival Attendees were asked about their
spend in relation to the Festival in different
An indication of the relative value of each
outcome can be presented by the predicted categories. In these limited terms of
value for someone who agrees or strongly economic impact on local businesses, the
agrees with a Likert statement compared with biggest beneficiary businesses in Bradford are
the predicted value of someone who neither those in the food/drink sectors, whereas little
agrees or disagrees. impact is on transport and accommodation.
Likert sums (Agree)
Attendees Inclusion &
diversity
£1.53 1.2 Social Return
Improved The total value of the Festival is described by
literacy & £2.21 combining the quantities of people who
engagement achieved outcomes with the values of each
Improved
perception outcome (including outcomes not defined).
£1.71
of Bradford
City Table 1 summarises the net impact of each
Residents (Bradford
Civic Pride £4.61 outcome for each stakeholder and shows the
Households)
total values. (This summary only captures
More of Attendees Willingness to Pay for the value for Stakeholders included).
Festival was predicted by achieving Literacy
and Engagement outcomes than other Because, in total, there are 199,296
outcomes. In other words, Attendees valued households in Bradford, who on average are
Literacy and Engagement higher than other willing to pay £11.24 each for the Festival,
outcomes tested.
(compared with, for example, 11,867
Comparing these values with the averages for Attendees each Willing to Pay £19.63), the
Willingness to Pay, we can also derive an value of outcomes for households in Bradford
element of Willingness to Pay that is not
Table 1: Net Impact
Stakeholders Outcomes Net Impact Total Value
Attendees Inclusion & Diversity £24,633
Improved literacy & engagement £10,009
Improved perception of Bradford City £26,616
Other outcomes (not defined) £171,742 £233,000
Residents (Bradford Households) Civic Pride £271,341
Other outcomes (not defined) £1,969,308 £2,240,649
Schools All outcomes £8,879 £8,879
Local Economy Spend £189,868 £189,868
TOTAL £2,672,396
6
becomes by far the biggest impact of the To maximise value, the Willingness to Pay or
Festival. outcomes per person would need to increase.
This report does not recommend any action
The social return, in terms of outcomes and to change delivery of the Festival required for
Willingness to Pay described in this report, for this - it is simply an account of the value
the 2017 Festival is £2,672,396. based on the SROI method. Business planning,
strategic and management processes, and
Against an Investment of £900,000, this review of investment objectives should
creates a social return ratio of 2.97. conclude what actions should be taken now
the value of impacts is known and the most
valuable impact revealed.
1.3 Benchmarking
There was no evidence of an SROI or a 1.4.1 Future years
Willingness to Pay study for a similar festival This analysis is designed as a baseline. In
in our desk research. future years, the value should be measured
with comparable survey instruments to
A methodologically comparable, and relevant monitor the achievement of the outcomes in
study was carried out of the Willingness to this scope and the associated creation of
Pay for the Natural History Museum and the value.
Tate Liverpool (Bakhshi, 2015). BLF compares
favourably with the results of this study.
1.4.2 Accounting for Value
These were the only comparable and relevant
Recommendations can be made to improve
WTP studies found for arts and culture
the account.
activities.
Natural Social Mobility
Tate
BLF History The scope included exploring aspirations and
Liverpool
Museum literacy levels. Literacy levels were measured,
Attendees £19.63 £6.65 £10.83
Non-use £11.24 £2.78 £8.00
but aspirations were not explored to the same
depth in the survey questions. More could be
done in the survey to explore aspirations
1.4 Application and/or the scope modified to include social
Community investment can build local mobility, or evidence that the festival
relationships and support local improves prospects for Attendees. This could
development. This analysis defines and be aligned with relevant parts of PFG’s Social
demonstrates the Social Capital impacts of Purpose (PFG, 2017).
the Festival; the first step in managing the
Social Capital (WBCSD, 2017) of this Missing Outcomes
investment. Most of the Willingness to Pay was not
predicted by the outcomes tested. Missing
Among other uses, this analysis can be used outcomes could be explored by: an open
question about change; further analysis of all
to provide decision-makers with the
identified predictors; and/or testing for
information they need to ensure their outcomes pre-defined by other festivals
activities are delivering improved social
capital impacts (WBCSD, 2015).
7
Over 10 days in July, the Festival delivers over
2 Introduction 300 events. It’s not just about the written or
spoken word; it includes theatre, music,
This evaluation was commissioned by
dance and film. World-renowned authors,
Provident Financial (PFG), sponsor of Bradford
poets, musicians and artists are invited to visit
Literature Festival (BLF), to measure the value
Bradford and share their expertise and
and social return that the Festival creates in
passions.
the following areas of impact (outcomes):
Inclusion and diversity Some events were ticketed and held in
Literacy and engagement venues. Some were free events held in public
Perception of Bradford City spaces.
Contribute to the tourism economy
The Festival is funded by a mix of ticket sales
and sponsorship.
2.1 Background
PFG sponsor the Bradford Literature Festival The hub of the Festival can be found in
and wish to put in place an evaluation model Bradford’s City Park, with its Mirror Pool.
for the Festival to measure the impact and Every year City Park is transformed by literary
value resulting from the Festival over 5 years activity, with live performances, film
of sponsorship. screenings and fun, free events for all the
family.
2.2 Provident Financial
Among other things, the Festival aims to:
[Taken from: www.providentfinancial.com ] support and promote reading for pleasure
for all;
Established in 1880, PFG is one of the UK’s create a neutral space for discourse and
leading suppliers of personal credit products dialogue;
to the non-standard lending market. PFG is a promote intercultural fluency and
FTSE 250 company listed on the London Stock stronger communities.
Exchange, with 3,712 employees serving 2.5 showcase Northern ideas, Yorkshire
million customers. Through a network of identity and strengthening Northern
branches, call-centres and websites, PFG cultural development; and
provide a portfolio of products designed to draw attention to Bradford’s distinct
meet the particular needs of those who want historical and cultural offerings reflecting
credit products. The group delivers non- the cultural sensibilities of the district’s
standard lending through four businesses - diverse population in its entirety and
Vanquis Bank, Provident home credit, reflecting the changing face of
Satsuma Loans and Moneybarn. contemporary Britain.
A programme runs in schools during the
2.3 Bradford Literature Festival Festival offering students of all ages, and their
teachers, access to writers working right now.
[Taken from: www.bradfordlitfest.co.uk ]
The programme is free to schools. There were
4 specific events for schools and authors visit
schools.
8
In 2016, the total audience was over 31,000 This broader definition of value aligns with
people. Roughly half described themselves as the concept of value used in cost-benefit
black, Asian or minority ethnic. About 60% analysis (CBA), the standard evaluation
were children. More than half came from methodology used in UK policymaking and
outside of Bradford. Social Return on Investment (SROI).
Audience figures grew to approx. 50,000 in Social return is about value, rather than
2017. money. Money is simply a common unit and
as such is a useful and widely accepted way of
conveying value.
2.4 Value and Social Return
Bakhshi et al. state There are few more deeply This evaluation followed the Social Return on
contested relationships in cultural policy than Investment (SROI) process, where applicable
that between economics and cultural value. to the scope. (Details of the principles and
The way that we measure the value of cultural process of SROI are available in the Cabinet
institutions matters. They are valued in Office sponsored Guide to SROI (Social Value
distinct ways by those who use it and those UK, 2009)).
that do not, as well as by different groups in
society. Our choice of measurement affects In the same way that a financial account is
whose values we capture. For this reason, required to inform decisions to grow financial
arts funders have recently advocated a performance; the social return of an activity is
‘holistic’ assessment of the benefits of culture, much more than just a number – it can
which goes beyond the economic and provide a decision-making framework to
cultural to encompass the wellbeing, societal increase the social return. However, this
and educational value of culture (Arts Council report does not recommend any action to
England, 2014). (Bakhshi, 2015) change delivery of the Festival. It is simply an
account of the value based on the SROI
method.
9
- raising aspirations and literacy levels in
3 Scope the district through inspiring a love of
reading and writing;
3.1 Activity - aiding the economic regeneration of the
Over 10 days, the Festival delivered over 300 city through cultural regeneration;
events. Some events were ticketed and held - enhancing civic pride and dispelling
in venues. Some were free events held in stereotypes by creating positive
public spaces. A programme ran in schools discussion and PR about Bradford - locally,
during the Festival offering students and their regionally, nationally and internationally;
teachers, access to writers. - drawing attention to Bradford’s distinct
historical and cultural offerings;
- reflecting the cultural sensibilities of the
3.2 Activity Duration district’s diverse population in its entirety
In 2017, the Festival ran from the 30th June to and reflecting the changing face of
the 9th of July. The Schools Programme ran contemporary Britain.
during the Festival. Activities over this period
were evaluated. 3.3.3 Ambition
The Festival’s ambition is that by 2020, it will:
- attract 100,000 visitors per year;
3.3 Festival Objectives - be ranked among the top 5 literary
The aims and objectives of the Festival festivals in the UK;
included: - achieve an iconic international status.
3.3.1 National
- celebrating all expressions of the written 3.4 Evaluation Audiences
and spoken word; The main audiences for the evaluation were:
- showcasing the intimate relationship The organisers: Bradford Literature Festival
between words and other art forms such The Sponsors:
as theatre, music, dance and film; - Provident Financial
- championing literature as well as its - Bradford Council
positive impact on culture and society; - Arts Council England
- supporting and promoting reading for
pleasure for all;
- creating a neutral space for discourse and
3.5 Evaluation Objectives
dialogue; The evaluation aimed to provide an in-depth
- promoting intercultural fluency and baseline of value of the following areas of
stronger communities. interest for the 2017 festival; and
propose an ongoing monitoring system to
measure this value in future years.
3.3.2 Regional
- showcasing Northern ideas, Yorkshire
The baseline evaluation aimed to answer the
identity and strengthening Northern
following questions:
cultural development;
103.5.1 Inclusion and Diversity With so many individual events, it was not
How many people came to the Festival possible to derive the value of individual
and what was their profile? events to attendees as the sample size for any
How inclusive was the Festival? event was too small.
How well did the Festival bring together
diverse groups? Data collection surveys were only available in
English.
3.5.2 Literacy and Engagement
This evaluation did not aim to monitor
Did the Festival raise aspirations and
progress towards the 2020 ambitions of the
literacy levels in the Bradford District?
Festival. Credibility, recognition and profile of
How many people did the Festival
the Festival was evaluated by BLF separately.
introduce to literature for the first time?
In order to test relationships between all
3.5.3 Perception of Bradford City responses on outcomes with the valuation
Did the Festival showcase Yorkshire survey instrument, the survey did not include
identity and promote Bradford City? any open questions.
Did the Festival enhance civic pride for
the people of Bradford? The analysis did not follow all the process or
principles of SROI (Social Value UK Assurance
3.5.4 Value Process). The main differences were:
How much did the Festival contribute to - Outcomes and stakeholders to be
the tourism economy of Bradford City? evaluated were pre-defined (above);
What value did attendees, non-attendees - The causal relationship between the
and teachers in the Schools Programme Festival activities and the outcomes
put on the Festival and the impact on evaluated was not measured; and
them?
- There was no measurement of
Where did the Festival have the greatest duration of outcomes.
impact?
3.6 Stakeholders
The stakeholders for the analysis were:
Attendees
Residents (Bradford Households)
Schools
Local Economy
PFG
Other Sponsors
3.7 Limitations
Volunteers, authors and performers were not
included as stakeholders.
114 Method 4.2 Measuring Impact
Each evaluation question that required impact
data was translated in to outcomes (or
This section describes the evaluation method
changes) that survey respondents were asked
designed to meet the scope and objectives.
if they had experienced or not using Likert
scales. For example, an item in the literacy
The evaluation questions required different
scale was: ‘Please tell us whether you agree or
types of data, as shown in table 2.
disagree with the statement: The Festival has
improved my reading ability’:
4.1 Profiling Strongly Disagree
Survey respondents were profiled with Disagree
standard questions and by Mosaic (UK) Neither
Consumer and Demographic Data. Agree
Strongly Agree’
Mosaic UK is built by Experian and provides a A range of items like this were developed to
detailed understanding of the demographics, form a Likert scale for Literacy. Likert scales
lifestyles, purchasing behaviour, technology were developed for each outcome.
adoption, communication channel
preferences and location of all individuals and Each outcome was measured using Likert
households in the UK. sums (or scores) of each scale.
Table 2: Data types
Profiling
Impact
Spend
Inclusion and Diversity Value
How many people came to the Festival and what was their profile? •
How inclusive was the Festival? • •
How well did the Festival bring together diverse groups? • •
Literacy and Engagement
Did the Festival raise aspirations and literacy levels in the Bradford District? •
How many people did the Festival introduce to literature for the first time? •
Perception of Bradford City
Did the Festival showcase Yorkshire identity and promote Bradford City? •
Did the Festival enhance civic pride for the people of Bradford? •
Value
How much did the Festival contribute to the tourism economy of Bradford City? •
What value did attendees, non-attendees and teachers in the Schools Programme put
on the Festival and the impact on them? •
Where did the Festival have the greatest impact? •
124.3 Valuation service. In line with standard economic
Valuation techniques, their limitations, a short theory, WTP and WTA are considered to be
literature review and existing evidence of the appropriate measures of the value which
value of festivals were reviewed to select a a person derives from a particular change,
valuation method for this analysis. (The because it forces people to take into account
review was summarised in a separate scoping alternative outcomes and their household
report). budget constraints.
Contingent Valuation, Willingness to Pay was A specific contingent valuation question was
found to be a viable approach for measuring added to capture how people valued the
economic values in the context of cultural Festival. The approach asked people to
institutions (Bakhshi, 2015) and was selected directly report their Willingness to Pay (WTP)
as the technique to value outcomes for this to estimate the extent to which respondents
scope. The approach is recognised by UK benefited from the Festival, over and above
Government in the Green Book (HM Treasury, the out of pocket money they pay for it (if
2003 (updated 2011)) and recognised as fit- they paid any). The most common method for
for-purpose for use by companies in the Social measuring the WTP for a specific product was
Capitol Protocol (WBCSD, 2017). used - a direct approach using an open-ended
question (and provide a number to the
HM Treasury’s Green Book is founded on the nearest £).
economic concept of cost-benefit analysis
(CBA). What distinguishes economic The survey targeted three separate cohorts
approaches to appraisal and evaluation is that with different experiences of the Festival
the outcomes of an action are what matter, (Attendees, Residents and teachers from
and these outcomes are measured ultimately school participating to the Festival School’s
in terms of their welfare implications. This Programme). To explore whether the type of
allows us to evaluate the impact of non- payment option had an impact on
market goods, on welfare in terms of the respondents’ willingness to pay for the
amount of money that will leave the Festival, two separate versions of the survey
individual equally well off following, or were used to include: either council tax or and
avoiding, a change in the good (known in donation as alternative payment methods (for
economic terms as compensating/equivalent both Attendees and Residents). The survey
surplus). directed to the teachers in the Schools
Programme used the school budget as
Contingent valuation (CV) surveys elicit the preferred payment method.
monetary value of non-market goods and
services by directly asking people what value In every questionnaire a hypothetical scenario
they attach to them (Bateman, 2002). By was laid out where the Festival could not
means of an appropriately designed afford to provide all events and School’s
questionnaire, a hypothetical market is Programme on ticket sales alone. In the
described where the good or service in unlikely event that grant and sponsorship
question can be traded. People are then funding ceased, the Festival would need to
directly asked about their willingness to pay raise enough money in other ways to support
(WTP) or willingness to accept (WTA) for a its activities and secure its long-term future.
change in the level of provision of the good or
13In WTP surveys, in order to elicit consistent A review of the Festival’s own evaluation of
values throughout the samples it is vital that 2016 enabled target sample sizes to be set.
the hypothetical scenario be believable, and
that it justifies introduction of the The venues used for festival events can have
hypothetical payment mechanism. an impact on the effectiveness of audience
surveys (particularly if they are pressured for
In designing scenarios best practice was time) and introduce a bias to responses. We
followed to avoid known biases. These related surveyed all groups online, away from the
to question wording and ordering, payment Festival. Contact was made via email
vehicle, prompts and use of images (Bateman, addresses from ticket sales and email
2002), cheap talk script, realism and addresses collected at free events by field
alternatives (Champ, 2001), including budget researchers. Teachers were contacted
and substitute reminders, and certainty directly and consulted by phone as the sample
questions (Bedate, 2004). size was small and a higher response rate was
required. (The timing was also critical for
Feasibility testing of the survey was collecting data from teachers as the Festival
conducted with 12 individuals, including a Schools Programme was delivered at the end
number of attendees (5), and non-attendees of the school year, a week or two before
(7). schools broke up for the summer holidays).
The contribution to the tourism economy in On collecting the email addresses and
Bradford City was represented simply by completing the questionnaire at the Festival
aggregating spend data from survey all respondents were informed of who is
respondents. It, therefore, only included storing their data, and its use, to adhere to
visitor (festival attendee) spend and excluded data protections law. Compliance with Data
spend by the festival itself. Protection guidelines was ensured by not
asking for personal data beyond an email
(This value to the local economy created by address if they wanted to participate in the
the Festival is likely to have a high amount of prize draw. Responses were reported in
displacement: local festival attendees will aggregate and it was not possible to identify
have a finite amount to spend on leisure participants from the responses that they
activities during a year. An economic gave, or from the way in which they were
equilibrium model would be required to reported.
evaluate this fully. This was not included.)
£800 of prizes were provided to incentivise
The total social value returned, on the completed questionnaires.
investment, was modelled based on estimates
of funding provided by the Festival. Target sample sizes are summarised in table 3
together with the achieved sample sizes. All
4.4 Data collection targets were met, with a total of 1,113
Data was gathered with a set of surveys. The complete responses from people in the
surveys were designed for each stakeholder (screened) target groups.
group that potentially achieved outcomes.
Festival attendees that responded to the
survey represented over 2,000 adults and
14children (average party size just over 3). about outcomes they might be expected to
Teachers that responded represented over achieve (table 4).
3,000 adults and children in their schools
(average party size over 100). A total of 5,829 Table 4: Expected Outcomes
adults and children were represented by
Perception of
Bradford City
Inclusion and
engagement
Literacy and
survey responses.
diversity
4.5 Survey Content
Attendees • • •
Respondents were screened according to Residents (Bradford •
Stakeholder target group: Households)
a) I have been to a Bradford Literature Schools • •
Festival event in 2017 Local Economy
PFG
b) I live in Bradford (but did not attend a Other Sponsors
Bradford Literature Festival event in 2017)
c) The school I teach at participated in the Overall the final version of the survey covered
Bradford Literature Festival Schools the following indicators:
Programme in 2017 The importance of the Festival for the
d) None of the above respondents (in terms of their willingness
to pay to keep Bradford Literature Festival
The surveys had over 100 questions producing running in the same way as presented to
581 separate values (possible answers) of them in the survey);
different attributes explored. However, Respondents’ experience of, and
respondents were routed through the satisfaction/feelings about, the Festival (in
questions with various options and no terms of community and social inclusion
respondent will have seen all the questions. benefits, literature benefits and others);
The experience was optimised to take no The economic impact on local businesses
more than 20 minutes. Respondents took just (travelling and various expenses when
over 10 minutes to complete the survey on attending the Festival);
average. Respondents’ perceptions of Bradford.
Additional information about the
Not all stakeholders were expected to achieve
respondents themselves (e.g. age, gender,
all outcomes. Stakeholders were only asked
income, ethnic background and health
Table 3: Sample size targets and response rate
ID Residents Attendees Teachers Screened none complete
population (emails) 535 5657 45
Target 300 670 30 1000 1000
Survey Test (Day 1) 973 7 7 7
Attendees Survey 634 29 608 26 663 16 679
Residents Panel Survey 089 362 77 4 443 100 543
Total 391 692 30 1113 116 1229
Representation (including children) 391 2189 3249 5829
15related quality of life using a validated set and test for payment vehicle bias). The
of questions from the Euroquol- vehicles were Council Tax and Donation.
5Dimension questionnaire) and their Responses were monitored daily during the
party were collected together with survey period and when a statically significant
specific data on the events attended sample of responses was achieved, the
(type, date, venue, etc); the latter are Willingness to Pay question was changed in
presented elsewhere. each survey.
(The section on Health and Wellbeing, with The payment vehicle for Schools was the
standard questions, was added at the request school budget.
of London School of Economics, as there was
no existing data in the literature review on
health and wellbeing benefits of festivals). 4.7 Analysis
First, the data were checked for content
Question categories and routes through the validity and implausible figures (outliers) were
surveys for different target groups are set to missing values (Bateman, 2002)
summarised in figure 2. (Mitchell, 1989).
Descriptive statistics of the Festival outcomes
4.6 Value Questions and respondent/their party characteristics
Sample sizes for Attendees and Residents were provided to give an overview of the
were big enough to test two different different cohort structures and to provide
Willingness to Pay questions using different data on the impact of the Festival on the array
payment vehicles (to cross check for accuracy of indicators (health-wellbeing, social,
community, literature and economic
Figure 2: Survey question categories and routes for target groups
a) I have been to a Bradford Literature Festival event in 2017
b) I live in Bradford (but did not attend a Bradford Literature Festival event in 2017)
c) The school I teach at participated in the Bradford Literature Festival School’s Programme in 2017
d) None of the above
Party attending festival
Experience of festival
Health and wellbeing
Travel and expenses
Events attended
Exit, thank you
Bradford pride
About festival
About you
Screen
Intro
WTP
a) 3 4 5 7 8 9 13 14 16 17
b) 3 4 5 10 13 16 17
c) 3 4 6 11 12 13 15 16 17
16outcomes). Data were described across the ticket vs. attendees with paid ticket (as sub-
three cohort of interest (attendees, non- groups).
attendees and teachers) and version of
questionnaires provided (council tax, Outcomes data were analysed for quantity
donation and school budget). and value. The quantity of outcomes was pro
rata to the total population of each
WTP responses were tested with a series of stakeholder group from the survey Likert
certainty questions (Bedate, 2004) (Champ, sums.
2001):
I/my school cannot afford to pay
I don’t mind if this is the last Bradford
Literature Festival
4.8 Population model
I think spending should be on all arts and The total population of festival attendees was
culture, not just this Festival not known. Due to the number of, and nature
The government should pay of, events at the festival it is not possible to
I think the Festival is important
accurately calculate a unique attendee figure.
I get satisfaction from giving to a good
cause (‘warm glow’)
I know I will never really have to pay any Audience figures were estimated by the
extra Festival in the standard way – the audience at
Other festivals and events are just as
each event was estimated, and then these
good
figures for each event totalled. This total
No statistically significant relationships were audience of 50,260 will include many people
found between responses to these test who attended more than one event (back-to-
questions and responses to WTP, confirming back free public events, particularly, attracted
that these issues were not affecting results audiences who attended multiple events).
(not determinants of WTP).
Our surveys evaluated attendees experience
In addition, a series of regression analyses of the Festival as a whole, not of each event,
were estimated to identify the key so we needed to establish the total number of
determinants of respondents’ WTP for the unique individuals who attended the Festival
Festival. for any multipliers in our model.
More analyses were also conducted to An estimation of total attendee populations
compare responses from attendees with free was made, informed by: data on people who
bought more than one ticket; observations at
Table 5: Population Model
Audience Unique attendees
Ticketed Events All ticketed events 13,294 6,702 11,867
Non-ticketed events City Park Family Fun Days 18,400 5,166
Drop-in Workshops 1,995
Other Non-Ticketed Events 4,278
Schools Programme In School Events 6,118 5,745 5,745
Schools events at the University of Bradford 6,175
Total 50,260 17,612 17,612
17consecutive public events; and survey 4.12 Deriving Value of Outcomes
responses about party sizes and multiple
events, to produce a population model shown 4.12.1 Regression model
in table 5. There are many variables in this
The Willingness to Pay questions in the
model, for example data was not available to surveys identified the value that different
establish if people buying multiple tickets stakeholders put on the Festival as a whole.
were buying them for multiple people, or
multiple events for one person. It is only an Likert sums of outcomes were regressed
estimation for building a model of the total against the Willingness to Pay responses to
test for a relationship between the value
value.
stakeholders put on the Festival and the
outcomes they achieved.
(The sensitivity analysis shows that the Social
Return result is sensitive to the population Regression types were reviewed (Cam
model. The Willingness to Pay result is not Donaldson, 1998) and an appropriate model
sensitive to it. If it becomes material to developed.
decisions the population model should be
examined further). In this model we have assumed Likert items
can been combined equally in scales where all
items were assumed to be replications of
4.9 Funding and Inputs each other or in other words items were
A detailed breakdown of funding and inputs considered to be parallel instruments (van
for the 2017 Festival was not available. Alphen, Halfens, Hasman, & Imbos, 1994).
Funding was estimated by the Festival at Secondly, Likert items represent linear scales
£900,000. Non-financial inputs were not (as opposed to ordinal data). Therefore,
identified. scaled items can be summed.
Statistically significant results for a
4.10 Profiling relationship between Willingness to Pay and
The Mosaic profile was run for all survey all the outcomes tested were found for
respondents who had attended an event at attendees (692) and Bradford residents (391).
the Festival and who lived in West Yorkshire.
The West Yorkshire sample was chosen in Statistically significant results were not found
for the outcomes tested for Schools as the
order to provide a meaningful area for
sample size (30) was too small. For Schools,
comparison within available Mosaic datasets the total of all outcomes was used, as the
to judge the relative inclusivity of the Festival. value of individual outcomes could not be
derived accurately enough.
4.11 Modelling quantities of In addition to the outcomes, other attributes
in the survey data that may indicate PFG
outcomes customers (or potential customers) were also
Likert scales identified both the occurrence regressed against Willingness to Pay:
and magnitude of change (outcomes). If the event was free
Outcomes were quantified using Likert sums Gender
from survey data. Age
Income
184.12.2 Predictors stakeholder that answered ‘neither’ and
In addition to the outcomes, test questions ‘agree’ to an outcome Likert item statement
and other attributes (above), the relationship was derived. Similarly, the difference in value
between WTP and other data were also between those that answered ‘neither’ and
explored for predictors, including: ‘strongly agree’ was predicted, and so on, to
ethnicity produce values for a unit increase (or
no. of children in party decrease) in all the outcomes.
disability
‘not for me’
4.13 Value Map
‘for me’
A value map was developed to show the
reason: specific author
inputs and outcomes for each stakeholder, to
reason: near me
enable a calculation of the total social return
reason: atmosphere
on the investment(s).
reason: Fun
reason: for the whole family The total value of each outcome was
reason: to shop produced by taking the Likert scores from the
reason: get booked signed surveys pro rata to the total population and
reason: Free multiplying these by the derived values for a
reason: love literature unit of increase or decrease.
reason: space to meet
reason: children want to go The value to the local economy was
reason: someone else bought tickets represented by aggregating spend data from
reason: other reason survey respondents.
overall satisfaction
Overall, 328 separate attributes were
regressed against WTP (41 attributes for 8
groups and sub groups (table 6).
Table 6: Groups and sub-groups
Council tax + Overall sample
Donation
Council tax Overall sample
Residents
Attendees
subgroup of Attendees
with free ticket
subgroup of Attendees
with paid ticket
Donation Overall sample
School budget Teachers
A fitted model was produced from each
outcome regression to predict the Willingness
to Pay for any given outcome Likert score. In
this way the difference in value between a
195 Results (n) 691
Under 16 0%
Each of the evaluation questions are 16-24 6%
considered, and other results are discussed. 25-34 17%
35-44 21%
45-54 20%
5.1 Inclusion and Diversity 55-64 20%
65+ 14%
5.1.1 How many people came to the Prefer not to say 2%
Festival and what was their
profile?
The Festival recorded total audience figures of Table 9: Household income of Respondents
50,260. The population model (table 5) (Attendees)
estimates that this represents 17,612 people.
(n) 691
Attendees were profiled in 2 different ways Less than £10,000 7%
£10,000 - £20,000 19%
(standard questions and a Mosaic profile), in 2
£20,000 - £30,000 22%
different surveys (this evaluation and the
£30,000 - £40,000 13%
Festival’s own evaluation). Profiling by
£40,000 - £50,000 10%
standard questions is shown in tables 7 – 9
More than £50,000 14%
(excluding Schools Programme Attendees).
Prefer not to say 16%
The Mosaic profiling in shown in table 10.
Table 7: Profile of Attendees
The percentage of households falling into
each Mosaic Group and Type were used to
(n) 2,189
Female 69% calculate an Index. An index of 100 means
Children 65% that the percentages are the same, so the
BAME 48% proportion of survey respondents belonging
Disabled 17% to the Group is the same as the proportion of
Bradford 53%
Regional 34% households in West Yorkshire who belong to
National 12% the Group. An Index of more than 100 means
International 1% that the Group is over-represented within the
survey sample, and an Index of less than 100
means that the Group is under-represented
5.1.2 How inclusive was the Festival? within the survey sample.
The Mosaic profile for attendees used 496
responses with eligible postcodes. The profile The Index figure therefore gives an indication
was then compared to the Mosaic UK profile as to how representative Festival attendees
for households in West Yorkshire. were of the general population in West
Yorkshire and, by inference, how inclusive the
Table 8: Age of Respondents (Attendees) Festival was.
Table 10: Mosaic UK Profile of Festival
Attendees from West Yorkshire
20The index for subgroup Asian Heritage (as an
% of survey
Mosaic UK Group Index example of Black, Asian and Minority Ethnic
respondents
groups (BAME) within the Mosaic data) is 153,
(n) 496 suggesting that the Festival has succeeded in
Urban Cohesion 14% 186 attracting more of this Mosaic Type than
Domestic Success 12% 172 would be expected given the profile of the
Rural Reality 3% 158 regional population.
Rental Hubs 10% 150
Prestige Positions 8% 139 5.1.3 How well did the Festival bring
Country Living 2% 138 together diverse groups?
Aspiring Homemakers 13% 114 The overall profile of attendees suggests a
Suburban Stability 7% 96 good mix. However, as the Festival is not one
Senior Security 8% 89 event, there may not be any mixing of groups
Transient Renters 8% 78 at an individual event level. The analysis
Modest Traditions 4% 68 compared profile of ethnic, social and age
Family Basics 5% 50 groups by event. Most events showed an
Vintage Value 3% 28 appropriate bias according to their target
audience (for example, predominately
The largest groups were Urban Cohesion (14% children at children’s events). Insufficient
of survey respondents), Aspiring Homemakers data (small samples) for each event
(13%), Domestic Success (12%) and Rental prohibited detailed analysis of this. Instead,
Hubs (10%). These Groups, which together attendees were asked directly about mixing
account for nearly half of all respondents, all with diverse groups.
have index values of over 100, suggesting that
they were over-represented in comparison to Likert items in the survey for attendees
the market area as a whole. included:
- I felt a sense of inclusion at the Festival
The Festival also appears to be effective at - I mixed with people at the Festival from
penetrating Prestige Positions (Index of 139), different ethnic, social or age groups
Rural Reality (158) and County Living (138)
Groups. However, the number of respondents Combined Likert scores (table 11) for
falling into these groups is lower; together Inclusion and Diversity questions showed the
they represent just 13% of survey majority (85%) of attendees felt included and
respondents. that the Festival brought together mixed
groups.
In broad terms, the profile suggests an
audience profile that is young, ethnically
diverse and from a range of income levels.
This is confirmed when the Profile is analysed
by Type. The dominant five Mosaic UK Types
for the sample are mostly clustered around
the middle of the Limited Resources –
Affluence axis and seem to live in urban or
suburban neighbourhoods.
21Table 11 approximately half (53%) of attendees felt the
Attendees Festival improved their literacy.
Inclusion and Diversity
Likert Sum Freq Table 12
Attendees
S Disagree 3 1% Literacy
Disagree 28 5% Likert Sum Freq
Neither 51 10%
Agree 201 38% S Disagree 23 4%
S Agree 252 47% Disagree 136 25%
(n) 535 Neither 93 17%
Mean Likert Score 1.25 Agree 211 39%
S Agree 72 13%
Attendees of the Schools Programme were (n) 535
profiled, but not asked the Likert questions. Mean Likert Score 0.32
In contrast, teachers gave higher combined
5.2 Literacy and Engagement Likert scores (table 13) for their students as a
result of the School’s programme; (83%) of
5.2.1 Did the Festival raise aspirations teachers felt the Festival improved pupil’s
and literacy levels in the Bradford literacy.
District?
Attendees, including teachers (on behalf of Table 13
Schools
the Schools Programme attendees) were
Literacy
asked a range of Likert questions.
Likert Sum Freq
Likert items in the survey for attendees
S Disagree 0
included:
Disagree 2 7%
- The Festival has made me want to read
Neither 3 10%
more for pleasure
Agree 19 63%
- The Festival has made me want to write
S Agree 6 20%
creatively more (n) 30
- The Festival has improved my reading Mean Likert Score 0.97
ability
5.2.2 How many people did the Festival
Likert items in the survey for teachers also
included:
introduce to literature for the first
- The Festival School’s Programme has time?
contributed to improving national 135 of the survey sample attended the
curriculum literacy competences and Festival for the first time, representing a pro
targets for students at my school rata of approximately 2,300 people.
Combined Likert scores (table 12) for 20% of attendees thought a literature festival
was not for them before attending. Having
attendees’ Literacy questions showed that
22attended the Festival, 67% of these attendees Table 15
no longer thought it was not for them. Schools
Bradford
Likert Sum Freq
5.3 Perception of Bradford City
S Disagree 0 0%
5.3.1 Did the Festival showcase Disagree 0 0%
Yorkshire identity and promote Neither 5 17%
Bradford City? Agree 9 30%
S Agree 16 53%
Attendees, including teachers (on behalf of
(n) 30
the Schools Programme attendees) were
Mean Likert Score 1.37
asked a range of Likert questions. Likert items
in the surveys included:
- The Festival showcases Yorkshire 5.3.2 Did the Festival enhance civic pride
identity for the people of Bradford?
- The Festival has improved my view of Bradford residents (who did not attend the
Bradford Festival) were asked Likert questions about
the Festival. Items included:
Combined Likert scores (table 14) for - The Festival makes me proud to live
attendees’ questions about Bradford and in Bradford
Yorkshire showed that the majority (84%) of
Attendees felt the Festival showcased Table 16
Yorkshire identity and promoted Bradford Residents
City. Bradford (Civic) Pride
Likert Sum Freq
Table 14
Attendees S Disagree 33 9%
Bradford Disagree 29 8%
Likert Sum Freq Neither 168 44%
Agree 98 26%
S Disagree 2 0% S Agree 55 14%
Disagree 26 5% (n) 383
Neither 59 11% Mean Likert Score 0.30
Agree 233 44%
S Agree 215 40% Teachers were also asked questions about this
(n) 535 (table 17), in this case not on behalf of
Mean Likert Score 1.18 School’s programme attendees or their
students, but on their own behalf. Items
Teachers felt the same: 83% agreed that the included:
Festival Schools Programme showcased - The Festival makes me proud to
Yorkshire identity and promoted Bradford teach in Bradford
City. (Table 15)
235.5 Value
Table 17 5.5.1 How much did the Festival
Teachers
contribute to the tourism
Bradford (Civic) Pride
economy of Bradford City?
Likert Sum Freq
Festival attendees were asked about their
S Disagree 0 0%
spend in relation to the Festival in different
Disagree 1 3% categories:
Neither 6 20% - Transport (bus, rail or taxi)
- Car parking
Agree 11 37%
- Accommodation
S Agree 12 40%
- Food and drink/eating out
(n) 30 - Gifts and special (non-routine) shopping
Mean Likert Score 1.13
691 survey respondents told us about £25,886
of spend for their parties (a total of 1,618
5.4 Comparing Data for Outcomes adults and 571 children): An average
Combined Likert Scores can be compared for spend per adult attendee of £16. The total
the 3 outcomes tested, as shown in figure 1 in (estimated) population of attendees therefore
the Summary. (In this presentation, a score of contributed £189,868 to the local economy in
0 would represent an average response of these categories of spend during the 2017
neither agreeing or disagreeing with the Festival.
statement. A score of 1 would represent an
average response of agreeing with the In these limited terms of economic impact on
statement, and so on). local businesses, the biggest beneficiary
businesses in Bradford were those in the
24food/drink sectors, whereas little impact is on 45 Schools were involved in the School’s
transport and accommodation. programme. The sample of teachers that
responded to the survey (30 teachers
The Festival spends a significant amount representing 30 Schools), therefore,
locally, including hotels for authors and represented a high percentage of the School’s
performers, but this was not included in the involved, but was too small a sample to
scope. The contribution to the tourism achieve statistical significance in the results
economy in Bradford in this report only for regressions.
includes visitor (festival attendee) spend.
Table 19
5.5.2 What value did attendees, non- Mean WTP (School Budget)
Schools £197.30
attendees and teachers in the
Schools programme put on the
Festival and the impact on them?
Value of the Outcomes
Likert sums of outcomes were regressed
Value of the Festival against the Willingness to Pay responses to
Survey respondents (Attendees, Residents test for a relationship between the value
and Teachers) were asked their willingness to stakeholders put on the Festival and the
pay for the Festival (for a year). outcomes they achieved. (As described in
4.12.1).
Attendees and Residents were asked
Statistically significant results for a
comparable questions. Residents were willing
relationship between Willingness to Pay and
to pay less than Attendees. This result was all the outcomes were found for Attendees
statistically significant. The average and Bradford residents.
willingness to pay is shown is table 18.
Statistically significant results were not found
Table 18 for the outcomes tested for Schools as the
Mean WTP (Council Tax) sample size (30) was too small. (For Schools,
the total of all outcomes was used, as the
Attendees £19.63
value of individual outcomes could not be
Residents £11.24
derived accurately enough).
Combined £16.13
An indication of the relative value of each
Teachers were asked, in their professional outcome can be presented by the predicted
capacity, how much they (their School) would value for someone who agrees or strongly
be willing to pay for the School’s programme agrees with a Likert statement compared with
the predicted value of someone who neither
(table 19). Their result is, therefore, not
agrees or disagrees.
comparable with the value individuals and
Households (Attendees and Residents) place Comparing these values with the averages for
on the Festival as it is a value to achieve Willingness to Pay, we can also derive that
outcomes for a larger group of students or there is an element of Willingness to Pay that
Schools Programme attendees. It also used a is not described by the outcomes tested.
different payment vehicle (school budget as
The WTP for the Festival without the
opposed to Council tax).
outcomes (someone that answers ‘neither’ to
all Likerts) is £14.47 for Attendees and £9.88
25for Residents. The WTP for an outcome is 5.5.3 Where did the Festival have the
then what someone is willing to pay for greatest impact?
additional inclusion, literacy, civic pride etc
The total value of the Festival is described by
over and above the WTP without the
outcomes. combining the quantities of people who
achieved outcomes with the values of each
Table 20 summaries the relative value of outcome. For example, although Literacy and
these outcomes. Engagement is more important to people than
other outcomes, the quantity who achieved
Table 20: Relative value of Outcomes
these outcomes is lower than other outcomes
Likert sums (Agree) – the combination of these results, therefore,
Attendees Inclusion &
£1.53 provides the overall picture.
diversity
Improved
literacy & £2.21 This summary only captures value for
engagement Stakeholders included. Value for other
Improved
perception stakeholders is not included. There is no
£1.71 value for Teachers included. (Although they
of Bradford
City were asked if the Festival made them proud
Residents (Bradford
Civic Pride £4.61 to teach in Bradford, this was not valued from
Households)
their perspective; the only value described by
teachers was on behalf of the school for
Other outcomes (not defined) net outcomes for students or Schools Programme
Attendees £14.47 attendee).
Residents (Bradford
£9.88
Households) Because, in total, there were approximately
199,296 households in Bradford, who on
We can, therefore, conclude that at a unit average were willing to pay £11.24 each for
level more of the Willingness to Pay for the
the Festival, the value of outcomes for
Festival was predicted by achieving Literacy
and Engagement outcomes than other households in Bradford becomes by far the
outcomes. In other words, Attendees valued biggest impact of the Festival (compared with,
Literacy and Engagement higher than other for example, 11,867 Attendees each Willing to
outcomes tested. Pay £19.63).
However, the majority of the Willingness to
Pay was not predicted by the outcomes
5.6 Other results
tested. In other words, there were missing
outcomes (or attributes) that were valued In addition to the outcomes, other attributes
more than the outcomes tested. This value in the survey data that may indicate PFG
may be described by other predictors in the customers (or potential customers) were also
data or may not be in the data at all. This is to regressed against Willingness to Pay:
be expected: WTP for the whole is likely to If the event was free
encompass a range of other use and non-use Gender
values (Sugden, 2005) (Bateman, 2002). Age
Income
265.6.1 Paying for an Event (rather than their children) valued the
Attendees who did not pay for the event(s) Festival more (than others).
they attended valued the Festival more (mean
WTP £24.33) than Attendees who paid (mean
WTP £17.63). Attendees who did not pay
included people with free tickets and
Attendees at free events.
5.6.2 Gender
Male Attendees valued the Festival more
(mean WTP £28.94) than Female Residents
(mean WTP £15.90).
Conversely, Female Residents valued the
Festival more (mean WTP £15.19) than Male
residents (mean WTP £6.94).
5.6.3 Age
There was no statistically significant result for
the regression of age of Attendees with their
WTP – Age was not a determinant of WTP for
Attendees.
Younger Residents were willing to pay more
than older Residents. Specifically, 16-24 year
olds paying council tax valued the Festival at a
mean WTP of £85.56.
5.6.4 Income
There was no statistically significant result for
the regression of household income of
Attendees or Residents with their WTP for the
Festival– income was not a determinant of
WTP for either Attendees, Residents or
combined groups.
5.6.5 Other Predictors
Other key determinants of respondents’ WTP
for the Festival included:
Respondents who visited the Festival
... for a specific event or author/
because they love literature/ to shop
/ or because they wanted to go
276 Social Return
If the value of all the investment(s) (or inputs)
for the Festival is included, a Social Return on
Investment (SROI) calculation can be made.
The resulting value map summarises inputs,
outcomes and values for stakeholders.
6.1 Funding and Inputs
A detailed breakdown of funding and inputs
for the 2017 Festival was not available.
Funding was estimated by the Festival at
£900,000. Non-financial inputs were not
identified.
Table 22: Inputs
Financial
Inputs
Ticket Sales £40,000
PFG Sponsorship £400,000
Other Sponsorship £460,000
Total £900,000
6.2 SROI Calculation
The social return, in terms of outcomes and
Willingness to Pay described in this report, for
the 2017 Festival is £2,672,396.
Against an Investment of £900,000, this
creates a social return ratio of 2.97.
28You can also read
