Research Setting the context for using complexity theory in evaluation: boundaries, governance and utilisation
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Evidence & Policy • vol 12 • no 1 • 73–89 • © Policy Press 2016 • #EVPOL
Print ISSN 1744 2648 • Online ISSN 1744 2656 • http://dx.doi.org/10.1332/174426415X14298726247211
This article is distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 license
(http://creativecommons.org/licenses/by-nc/4.0/) which permits adaptation, alteration, reproduction
and distribution for non-commercial use, without further permission
provided the original work is attributed. The derivative works do not
need to be licensed on the same terms.
research
Setting the context for using complexity theory in
evaluation: boundaries, governance and utilisation
Mat Walton, m.d.walton@massey.ac.nz
Massey University, Wellington, New Zealand
IP : 192.168.39.211 On: Tue, 30 Nov 2021 08:23:57
Recent literature has usefully explored the application of complexity theory to evaluation. However,
there is little discussion of the contextual conditions in applying complexity theory. Drawing upon a
single complexity-consistent public health programme evaluation and subsequent policy decisions,
Copyright The Policy Press
this paper considers how programme framing and governance can help or hinder application of
Delivered by Ingenta
complexity theory. A wider framing of the programme and cross-system network governance
arrangements are suggested as important components to a complexity theory-informed policy
practice.
key words complexity theory • evaluation utilisation • governance networks
Introduction
There is a growing literature on the potential and application of complexity theory in
a number of areas including: conceptualising and researching public policy (Morçöl,
2012; Buijs et al, 2009; Cairney, 2012; Geyer, 2012; Eppel, 2012; Eppel et al, 2011);
research utilisation (Lemay and Creso, 2012); and evaluation (Vincent, 2012;Westhorp,
2012; Blackman et al, 2013; Patton, 2011;Walton, 2014). Drawing upon a single public
health programme evaluation and subsequent policy decisions, this paper considers
how programme framing and governance can help or hinder application of complexity
theory to public health evaluation and policy.
An outline of complexity theory
Complexity theory is not one coherent theory or field of research (Keshavarz et
al., 2010; Richardson, 2008), although there is a common set of concepts regarding
how complex systems behave and evolve over time. While first established in the
mathematical and physical sciences (Nicolis and Prigogine, 1989; Kauffman, 1995),
over the last 20 years the field has expanded to applied areas including management
(Stacey, 2012), health (Plsek and Greenhalgh, 2001), public policy (Morçöl, 2012)
73Mat Walton
and evaluation (Morell, 2010). Mowles (2014) suggests the most successful translation
from natural to social sciences of complexity requires both explicit identification
of the strand of complexity theory being applied, and a translation through social
theory. Two broad approaches within complexity theory provide either a search for
generalisable simple rules that govern complex systems, or understanding complex
systems within specific contexts (Byrne and Uprichard, 2012).The discussion below
follows the latter approach and is informed by realist complexity thinking and what
Byrne refers to as a complexity ‘frame of reference’ (Byrne and Callaghan, 2014).
Morçöl suggests that ‘complexity theorists describe emergent holistic systems,
whose properties are not reducible to those of their parts.These systems are integrated
into and co-evolve with their environments’ (Morçöl, 2001, 112). The idea that the
interaction of components within a complex system gives rise to ‘emergent’ properties,
which cannot be understood by examining the individual system components, is a
fundamental aspect of complexity theory (Goldstein, 1999; Byrne and Callaghan,
2014). The process of emergent properties is often described as the whole being
greater than the sum of its parts (for example, Cooper and Geyer, 2008). Emergent
IP : 192.168.39.211 On: Tue, 30 Nov 2021 08:23:57
phenomena appear as ‘integrated wholes’ (Goldstein, 1999, 49). The interaction
between components of a complex system is non-linear, and complex systems self-
organise in response to feedback (Morçöl, 2012). Self-organisation and non-linearity
mean that impacts of an intervention on emergent phenomena are not dependent on
Copyright The Policy Press
the size of the intervention in terms of resource or effort, making it difficult to predict
Delivered by Ingenta
the type or scale of system adaptations to new feedback (Medd, 2002; Room, 2011).
A complex system is open to feedback from the wider environment within which
it exists (Gatrell, 2005). Environments may differ between time, social and geographic
contexts. It is possible, therefore, that the configuration of seemingly similar systems
and their emergent phenomena, can be different across environments (Morçöl, 2012).
Conversely, emergent phenomena may be similar across different contexts and system
configurations (Richardson, 2008). Systems may maintain stability at an ‘attractor
state’ (Mackenzie, 2005), a region towards which a system is pulled (Merry, 1995).
A change in nature of emergent phenomena suggests the system has moved to a
new attractor state. At the point of change, the range of possible attractor states the
system can move to is dependent on the ‘phase’ or ‘state’ space of the system (Byrne
and Callaghan, 2014).
The complexity challenge for evidence and policy
There are a number of implications of a complexity frame of reference for thinking
about evidence of and for policy (Sanderson, 2009; Byrne, 2011). One of the
most obvious is that evidence is contextual. Context contains both temporal and
geographical circumstances, with the history, internal organisation and interaction
with other systems all being important. Open to their environments, no intervention
operates in isolation, challenging the notion that one intervention can be understood
as independent of previous and other concurrent interventions. Stern (2011, ix) notes
that ‘[i]n independent spheres it makes little sense that competing administrative
units, ministries and agencies each commission their own evaluations of “parts of
the whole”’.
Drawing upon a realist ontology, consistent with a ‘complex realist’ perspective,
Room (2013) summarises a number of challenges for the role of ‘evidence’ in policy:
74Setting the context for using complexity theory in evaluation
Not only must the evidence for a given policy intervention consider how
it will work in combination with other policies; it must also differentiate
according to the order in which these policies are introduced, in the various
contexts that characterise the policy maker’s domain, and having regard to
the different timescales of their likely effects. (Room, 2013, 231)
Guidance on applying complexity theory to evidence and policy
Several approaches to evaluation drawing upon a complexity frame of reference have
developed over the last decade, in addition to often sympathetic methodologies from
related systems fields (Williams and Hummelbrunner, 2011) and realist evaluation
which also offers a framework for working with complexity (Pawson, 2013). In
reviewing application of complexity theory to evaluation of development programmes,
Vincent (2012) notes two faces of complexity. The first face uses approaches such
as Qualitative Comparative Analysis (QCA) and Realist Evaluation for retrospective
analysis to identify context, intervention mechanism and outcome combinations.
IP : 192.168.39.211 On: Tue, 30 Nov 2021 08:23:57
These approaches highlight both a generative and configuration understanding of
causality (Byrne and Callaghan, 2014; Stern et al, 2012). The second face applies
complexity theory to action research approaches, highlighting the importance of
local adaptations and action, and utilises rapid evaluation-action cycles as a way of
Copyright The Policy Press
managing uncertainty within complex systems.Vincent concludes:
Delivered by Ingenta
Having explored the two faces of complexity we can see that is both
important to attend to local context, diverse perspectives and negotiations
driving emergent social processes, but also to the wider previously emergent
social factors that frame and influence current negotiations. (Vincent, 2012,
26)
Two other recent publications provide useful guidance from a realist complexity
frame. Westhorp (2012) suggests the use of complexity-consistent theory to help
define evaluation questions and appropriate evidence, and provides a list of complexity
theory concepts that may act as a complexity checklist against which the evaluation
should be able to provide information, listed in Table 1. Buijs et al (2009) provide
methodological guidance for policy research from a complexity frame of reference
utilising case-based methodologies.The two frameworks suggest that evaluation from
a complexity frame requires the following: definition of system boundaries and critical
reflection on these; identifying interactions between elements within the system and
between co-evolving systems; identifying processes of feedback and how these impact
on interactions; identifying change across different scales of the systems from macro to
micro; identifying rules and decisions across levels of the system; and understanding
temporal ordering of feedback, changes in interactions and emergent phenomena.
Trenholm and Ferlie (2013) provide an example of the importance of looking at
different scales of a system and their interaction. In studying the TB response system
in London, it was found that local innovation was at times limited and other times
supported by the macro ‘phase space’ dominated by New Public Management models.
A literature considering research utilisation from a complexity lens is also developing.
Viewing the policy-making process itself as a complex system highlights the context
within which research is conducted and utilised, suggesting that standardised research
75Mat Walton
Table 1: Checklist for assessing evaluations as complexity-consistent
Checklist for assessing evaluations as complexity-consistent.
Does the evaluation identify …
Constituent elements of the system and drawing conceptual boundaries around the system
Interactions amongst system elements
Local rules that govern interactions and results across system levels from interactions
Forms of feedback that constrain or support change
Initial conditions that affect interactions within the system
Controlling parameters of the system
Interactions between levels of the system
Adapted from Westhorp (2012)
strategies and tools for translation will have limited effect (Lemay and Creso, 2012).
Within complex policy systems, multiple actors and multiple relationships will be
IP : 192.168.39.211 On: Tue, 30 Nov 2021 08:23:57
involved in transferring research knowledge.There will be a divergence in the values
and assumptions of these actors that will give primacy to different methodologies,
interpretations and outcomes (Smith and Joyce, 2012; Petticrew, 2013). Group
and dialectical methods are seen as important for translating a complexity theory
Copyright The Policy Press
understanding of a system into action to change the system (Byrne and Callaghan,
Delivered by Ingenta
2014). Processes for determining policy action across multiple perspectives are
required, such as sensemaking (Eppel, 2012; Snowden, 2011) and network governance
arrangements (Klijn and Edelenbos, 2013).
Network governance can be defined as ‘public policy making and implementation
through a web of relationships between government, business and civil society’
(Klijn, 2008, 511) for solving complex problems that cannot be solved by a single
actor (Klijn and Edelenbos, 2013). Network governance arrangements have been
identified as consistent with a complexity frame of reference, with an emphasis on
interaction and relationships amongst members of the network leading to ‘emergent’
outcomes (Morçöl, 2012; Morçöl, 2014). Governance networks may develop from
local level collaborations or be mandated from central government (Bogason and
Musso, 2006; Hertting and Vedung, 2012). This paper suggests that governance
networks tasked with implementation and evaluation of a cross-agency programme
may aid in applying complexity theory to evaluation practice, as well as programme
and policy development.
In the next section, the complexity frame of reference outlined above will be applied
to a case study. The analysis aims to explore implications of a complexity frame of
reference for defining programme system boundaries, governance arrangements and
associated implications for evaluation utilisation.
Case study – New Zealand Fruit In Schools programme and
evaluation
Fruit in Schools (FiS) was a health promotion programme introduced into New
Zealand primary schools in 2006. While not explicitly informed by a complexity
frame of reference, the case was chosen because both the programme and evaluation
designs have a number of features that are complexity consistent. Examining processes
76Setting the context for using complexity theory in evaluation
of programme and evaluation governance, as well as evaluation utilisation, facilitates a
thought experiment of how these processes may be managed if an explicit complexity
frame of reference is adopted.
In developing this case study, both document analysis and key informant interviews
were utilised. Documents included: evaluation reports (Boyd et al, 2009; Boyd and
Moss, 2009; Boyd et al, 2007; Dingle et al, 2009); advice to the Minister of Health
obtained under the New Zealand Official Information Act (Ministry of Health,
2009b; Ministry of Health, 2009a); and media reports. Interviews were undertaken
with seven key informants involved in policy making, programme implementation
and evaluation of FiS. Drafts of this paper were shared with four participants who
indicated an interest, to check both accuracy of description and positioning of the
FiS programme and evaluation within contextual background.
The setting
New Zealand has a population of 4.4 million, of which nearly 15% are Maori
IP : 192.168.39.211 On: Tue, 30 Nov 2021 08:23:57
(indigenous) and 77% are of European or ‘other’ descent. Two significant groups in
this ‘other’ category include those who identify as Asian or Pacific Islands ethnicities
(Statistics New Zealand, 2013). A unitary Westminster-style Parliament sees Cabinet
Ministers able to quickly implement policy and programme changes, often following
Copyright The Policy Press
election of new government administrations on a three-year election cycle (Blank
Delivered by Ingenta
and Burau, 2010).
In the early 2000s two high-profile government strategies were developed to address
key issues of health and nutrition. The Healthy Eating – Healthy Action (HEHA)
strategy (Ministry of Health, 2003) linked aims around obesity prevention, nutrition
and physical activity; and the Cancer Control Strategy (Cancer Control Taskforce,
2005) saw adequate nutrition as an important preventative of cancer. These broad
and ambitious strategies provided the framework for many specific programmes and
projects, including a number based within schools and targeting children (Walton et
al, 2010; Cushman and Clelland, 2012).
The policy and the programme
In 2005 a high-profile Fruit in Schools initiative for schools serving low socioeconomic
communities was announced as part of New Zealand’s Cancer Control action plan
(Cancer Control Taskforce, 2005).The FiS programme provided a piece of fruit per day
for each Year 1–8 child in the targeted schools.The programme also aimed to increase
the focus on health promotion in schools so was supported by a Health Promoting
Schools (HPS) framework, coordinators and associated teacher release and training.
The programme is described more fully in Box 1. The free fruit of FiS was seen by
both the government officials and the programme’s evaluators as a “carrot” (IF 3) or
a “foot in the door to a suite of health promotion activities” (IF 23), as schools had
to “sign up to this wider programme” (IF 23) of Health Promoting Schools.
77Mat Walton
Box 1: The Fruit in Schools Programme
Targeted schools: low socioeconomic profile primary, contributing, and intermediate
schools (Years 1–8). FiS was not mandatory – schools were invited to take part.
Intervention:
• Part 1 – one piece of fruit per school day per student
• Part 2 - Health Promoting Schools programme targeting four key cancer-prevention
areas: nutrition, physical activity, SunSmart (melanoma prevention) and smokefree.
• Programme supports: 25 dedicated FiS / HPS advisors; each school lead-teacher could
access funded release time for training and planning
Timeline: FiS expanded quickly through several phases, with each phase extending the
IP : 192.168.39.211 On: Tue, 30 Nov 2021 08:23:57
number of schools eligible for the programme. Phase One began in October 2005; Phase
Three a year later, by which time there about 268 primary schools in the scheme. Later
Phases (Four and Five) saw an expanded budget and the inclusion of slightly higher
socio-economic status (SES) profile schools and intermediate schools (Years 7 and 8) in
Copyright The Policy Press
the FiS programme.
Delivered by Ingenta
While the Ministry of Health was the lead agency and funder of the Fruit in Schools
programme, it worked closely with an External Reference Group that included
policy and practice representatives of the Ministry of Education, SPARC (Sport and
Recreation New Zealand, now Sport New Zealand) and non-government agencies
(NGOs) that provide programmes in schools targeted at the four health areas of
FiS. The group received regular reports from the evaluation and acted as a problem
solving group to support implementation and align initiatives. The FiS evaluation
report states that:
… a restructuring of the group occurred in late 2007 – causing some concern
among national agency representatives who saw it as a forum for shared
decision-making and accountability. (Boyd et al, 2009, 54)
The phased roll-outs of the programme took place against a moving backdrop of
related policies and activities. Several other high-profile nutrition and physical activity
initiatives were implemented within schools or targeted towards children over the
FiS implementation period. Some, but not all, child nutrition-focused interventions
over this period were evaluated through separate processes, and by separate agencies
(Education Review Office, 2009; Pledger et al, 2012; McLean et al, 2009).
The evaluation
An independent team, comprised of education and health researchers, began a mixed
method evaluation in late 2005. In the early stages, 2005–06, the evaluation was
78Setting the context for using complexity theory in evaluation
concerned with ‘process and formative evaluation’ intended to help stakeholders
improve the programme (Boyd et al, 2009, 4). Between 2006 and 2008 the focus
shifted to impact evaluation.
Evaluation methods included: surveys to students and teachers at most of the
schools at the start of 2006 and then at the end of each of the three years, 2006, 2007
and 2008; questionnaires to lead teachers in each school; 12 school case studies, and
interviews and online surveys with agency partners and stakeholders (Boyd et al,
2009, 10). Student questionnaires tracked a cohort of students, from Year 4 in 2006 to
their Year 6 in 2008, to consider changes in attitudes and behaviours. A comparison
group of schools was also selected.
Informed in theory and methodology by HPS, in part the evaluation design was
a pragmatic response to political positioning and tight implementation timelines.
Constant movement in the policy context created unforeseen programme changes.
Notably, most schools initially in the comparison group of schools not receiving fruit
subsequently received fruit in an added Phase Three of the programme.This left only
seven schools from the original comparison group of 34 schools in the evaluation
IP : 192.168.39.211 On: Tue, 30 Nov 2021 08:23:57
study (Boyd et al, 2009).
Final main evaluation findings were that students in FiS schools showed a slowing
in the expected decline in healthy attitudes and behaviours observed as students get
older. FiS students had maintained positive attitudes to, and awareness of, healthy
Copyright The Policy Press
behaviours; increased their intake of healthy foods and increased physical activity and
Delivered by Ingenta
sunsmart practices. They had positive views about school and took an active role in
promoting their school as a ‘healthy school’.The schools gave students health-related
leadership opportunities, increased their involvement with the programme’s agency
partners, and engaged with other health and wellbeing initiatives. This all raised the
value placed on health promotion in the schools and created a ‘protective climate’
for students.
What happened after the evaluation?
The centre-right government elected in November 2008 immediately ended several
child and youth focused nutrition and physical activity programmes. FiS also came
under review (Treasury, 2010). The Minister of Health made an interim decision to
continue the programme until after the evaluation report was completed (Ministry
of Health, 2009a).
In reporting the FiS evaluation findings to the Minister of Health, the Ministry of
Health used almost verbatim the summary provided in the Evaluators’ report (Boyd
et al, 2009).The Ministry advised that free fruit was ‘well-received in these high needs
school communities and was recognised as a key enabler in raising the importance
placed on health and wellbeing’. It went on to say that : ‘… benefits cannot solely be
attributed to the FiS programme’ (Ministry of Health, 2009b). While the evaluators
emphasised that it was the support and activity surrounding FiS, and not the piece
of fruit in itself which was making the difference, this was minimised in reporting
to the Minister. In line with the incoming government’s focus on ‘frontline services
rather than back-room bureaucracies’ (English, 5 December 2008) the Minister of
Health reframed the wider HPS component as ‘administrative’. Later in 2009 it was
announced that the Government would ‘continue the Fruit in Schools programme,
but is cutting the administration component’ (Ryall, 29 October 2009). The cut
79Mat Walton
‘administration component’ included: FiS co-ordinator and staff salaries; liaison
staff from District Health Boards; teacher-release time; meetings and other support
activities of the HPS approach. The FiS programme continues to deliver free fruit
to eligible schools.
Both the timing of reports to the Minister of Health, and comments from some
key informants, suggest that evaluation findings may have had some bearing on the
decision to carry on funding the fruit provision part of the FiS programme. Having
said this, the decision to stop funding for teacher release, professional development
and coordinator support is counter to headline findings of the FiS evaluation. Key
informants suggested that popularity and symbolic significance of the fruit provision
were amongst reasons for keeping this, while the other FiS programme components
did not have such support. The decision regarding the FiS programme was taken by
the Minister of Health alone.
Fruit in Schools through a complexity frame of reference
IP : 192.168.39.211 On: Tue, 30 Nov 2021 08:23:57
The FiS programme and evaluation were not explicitly informed by a complexity
frame of reference. In fact, many of the developments in the application of complexity
theory to policy and evaluation have been published since the programme and
evaluation establishment in 2005. However, using the seven criteria for complexity
Copyright The Policy Press
consistent theory identified by Westhorp (2012), a brief summary of the FiS evaluation
Delivered by Ingenta
against these criteria is provided. The first criterion is that elements of the system
are identified and boundaries drawn, while the second criterion is that interactions
between system elements are identified. The evaluation explicitly drew upon HPS
and related socioecological systems theory to outline a FiS ‘system’. The evaluation
framework identified different levels of the system under study including schools,
regional agencies, national agencies and policies and programmes operating across these
levels of organisation. At the school level the HPS framework focused attention on
interaction between school policies and practices, curriculum, students, staff, parents
and the wider community. Survey, interview and case study data were collected across
the levels of system and from different perspectives within schools.
The third criterion is that local rules are identified that govern interactions. The
impact of rules on school practices were identified in several areas. A particular focus
was the impact of other programmes and administrative regulations that schools
operate under.These could act as both facilitators and barriers to activity in the four
FiS health areas. The fourth criterion is that forms of feedback are identified. The
evaluation design was cognisant of feedback both within and between levels of the FiS
system. The evaluation design looked for interaction between student level changes,
school level changes and inter-agency relationship changes. Data triangulation was
used to consider the interactions between data at these different levels of the system.
The fifth criterion is that initial conditions that affect how the system operates are
identified. Data analysis had a focus on understanding FiS system trajectories, although
this term is not used by the evaluators. Identifying changes from initial conditions
was a focus of analysis, described by the evaluators as:
80Setting the context for using complexity theory in evaluation
• Comparing the patterns over time for FiS and comparison students
• Data triangulation (that is, looking at what the qualitative and quantitative data
from different stakeholders told us about key aspects of student, school and
interagency practice)
• Comparing changes over time to the expected patterns documented in key
literature
• Comparing the patterns of changes over time for different stakeholder groups
(that is, comparing the responses of stakeholders in different phases of FiS). (Boyd
et al, 2009, 11)
Whilst an understanding of trajectories for school and inter-agency practices was
achieved, an explicit complexity frame of reference would likely see these trajectories
fully explored and featured in key findings included in the executive summary.
Identifying interactions between levels of the system and control parameters that
determine whether a system radically changes pattern are the two final criteria. As
described above, data was gathered from school, local partner agency and national
IP : 192.168.39.211 On: Tue, 30 Nov 2021 08:23:57
partner agency perspectives, with interactions between these levels considered. The
concept of controlling parameters may be one area where a socioecological and HPS
view does not explicitly address concerns from a complexity frame of reference.
In summary, the FiS evaluation appears consistent in many ways with a complexity
Copyright The Policy Press
frame of reference.The evaluation has something to say against six of seven complexity
Delivered by Ingenta
theory criteria identified by Westhorp (2012). Multiple methods were used to gain
insight into changes in practices at different levels within the system, from the
perspective of multiple stakeholders. Selecting the FiS case is not to suggest it is
unique in being consistent with complexity theory. In fact, it is likely that many
evaluations drawing upon socioecological theory in public health would likewise be
consistent in several areas.
While the evaluation itself is consistent with the emphasis a complexity theory
frame provides on evidence, the case highlights contextual issues that limit the
use of the evaluation findings. These contextual issues are also considered from a
complexity theory perspective below. The first is the framing of the programme.
The second is the programme governance. It is suggested that an approach informed
by complexity theory and network governance would allow for fuller utilisation of
evaluation findings.
Programme framing and governance
In complex issues, it is likely that action will be required across traditional agency
boundaries, and between government and non-government (NGO) agencies
(Australian Public Service Commission, 2007). The FiS programme was in effect a
collaboration between the Ministry of Health, Ministry of Education, SPARC, and
NGOs at national level, and their delivery agents (district health boards, schools,
regional offices) at local levels. However, the political governance arrangements placed
political decision-making responsibility with the Minister of Health. Drawing upon
ideas from both complexity theory and network governance, two related issues are
identified. First, expanding the boundaries drawn around the FiS programme and
evaluation. Second, the role of a governance network in utilising the evaluation
findings.
81Mat Walton
Programme framing, defining system boundaries
A key step in many systems and complexity methods is actively defining the boundaries
of the system under study (Byrne and Callaghan, 2014; Midgley, 2000; Williams and
Hummelbrunner, 2011), with recognition that boundaries are constructed around
open and interacting systems. The FiS evaluation did consider how FiS was affected
by other nutrition- focused programmes and activities within schools. Even though
the focus was FiS, as one informant stated “… the other initiatives sort of blended
in with Fruit in Schools because in some ways they are aligned, but it’s very hard
to tease out what was what …” (IF 3). One implication of a complexity frame of
reference is that policy and programme designs need to be aware of interactions.
The final evaluation report does suggest that FiS should be better aligned with HPS
and the school curriculum. Also, that partner agency capacity to support schools is
reviewed (Boyd et al, 2009). In making a decision on FiS the Minister of Health
requested a report on existing HPS capacity to support the FiS programme. This
report was light on detail and did not fully explore the implications of removing the
IP : 192.168.39.211 On: Tue, 30 Nov 2021 08:23:57
FiS coordinator workforce (Ministry of Health, 2009b). No mention was made of
partner agency capacity or curriculum alignment, although arguably these are both
outside the Minister of Health’s remit.
A focus on programme interactions suggests a wider system boundary is required.
Copyright The Policy Press
The FiS is clearly both a health promotion and school system intervention. It has
Delivered by Ingenta
many interacting programmes. This was recognised by establishing the inter-agency
groups. Here it is suggested that network governance theory offers a set of ideas
complementary to complexity theory that can act to implement a complexity
theory perspective. A cross-agency network governance group could be delegated
responsibility for looking across the broader area of school health programmes
including FiS. This would legitimise an interaction focus in the evaluation, and
recommendations to Ministers could consider complementary changes across
programmes.
Network governance – a process for managing interaction
A key mechanism for governance networks to produce solutions to complex problems
lies in the ability to bring multiple perspectives and knowledge into a deliberative
decision-making process (Klijn and Edelenbos, 2013; Hertting and Vedung, 2012).
Meadows, from a systems perspective, identified ‘the ability to transcend paradigms’
as the most effective leverage point to transform systems (Meadows, 1999, 19).
Identifying competing paradigms of actors within a system is required to understand
action within that system and the value placed on certain outcomes.
Because complex systems contain vertical as well as horizontal interactions, a
network governance group should consist of cross-agency and cross-system level
(local, regional, national) actors. Within this network process, some complexity-
informed principles should be considered.The focus would not be insular on a single
programme, but consider a wider view of the systems within which the programme
operates and interaction with both system context and other programmes. Complexity
concepts, such as phase shift, attractor states and system trajectories provide a meta-
theory within which more specific discipline and subject matter assumptions can be
considered (Westhorp, 2012).
82Setting the context for using complexity theory in evaluation
Network governance arrangements provide a process to consider evaluation findings
and make decisions across the two faces of complexity identified by Vincent (2012),
iterative programme development and broader policy setting decisions. In the FiS case,
a network governance group would require delegated authority to consider evaluation
findings and make tactical and programme development decisions. Participants
involved with the FiS case note that while in place, the external reference group were
able to make decisions to problem solve during implementation.
A network governance group could also be tasked with making policy setting
recommendations to relevant Ministers, for example programme termination or
expansion. In the FiS case, recommendations came from only the Ministry of Health
to the Minister of Health. Recommendations to government Ministers from a network
governance group would of course represent a variety of perspectives from across the
system. Collective decision making by multiple Ministers with a stake in the ‘system’
under study would be a logical extension of network governance arrangements.
Related to the programme framing and central importance of interactions within
the school-health ‘system’, the remit of a network governance group would need
IP : 192.168.39.211 On: Tue, 30 Nov 2021 08:23:57
to be wider than only the FiS programme. Policy setting recommendations would
be across multiple programmes within health and education ministerial portfolios.
Evaluation utilisation
Copyright The Policy Press
Delivered by Ingenta
The previous discussion has outlined how complexity theory and network governance
frames could inform programme boundary definitions and governance arrangements.
With reference to the FiS case study, the implications of these frames for evaluation
utilisation are considered.
The Minister of Health received a report on the FiS evaluation that largely reflected
the final evaluation report executive summary. Much of the context-sensitive analysis
contained in the evaluation reports was lost within the executive summary. The
importance of initial conditions and divergent trajectories of FiS schools were not
included when describing key outcomes and mechanisms. If explicitly reflecting
a complexity frame, this information would likely drive the findings summary. Its
absence may reflect the challenge of fitting in the complexity of programme and
context interactions within a summary document. Pawson and Manzano-Santaella
(2012) note that realist exploration inside the ‘black box’ of programmes is difficult
within a tight word limit.A review of the degree to which Complex Adaptive Systems
(CAS) elements are incorporated in health promotion evaluations also suggested that
publishing restrictions may limit discussion of CAS perspectives (Kania et al, 2013).
A network governance group could provide a forum where detailed examination
of findings is possible, and implications of findings for practice, rather than findings
themselves, forms the basis of a summary. Byrne and Callaghan (2014) discuss dialogical
processes as needed to unify views of systems under study, including views generated
through research with those of people living and working in the systems under study.
Sanderson (2009), drawing upon complexity theory and Dewian pragmatism also
recognises the need to engage those ‘doing’ within the system in a move towards
‘practical rationality’. The network governance group could be delegated authority
to make programme adaptation and development decisions and policy setting
recommendations (for example, termination or expansion) to Ministers.
83Mat Walton
To be consistent with a complexity frame of reference, when making programme
development decisions or policy-setting recommendations to Ministers, the network
governance group would seek to clarify the types of outcomes emerging from the
system, and whether the system has undergone any phase shift. For FiS, if it seemed
that school systems (or those on certain trajectories at least) had moved into a
situation where health promoting actions were the norm, then it could be reasoned
that some programmes or aspects of them could be terminated while maintaining
the health and education outcomes of importance. Rather than a blanket approach,
recommendations could consider where this may not be the case, the role of initial
conditions, and configurations of system attributes that appear to be driving any
divergence of outcomes between school system cases.
Conclusion
Drawing upon complexity theory-related literature and the FiS case study of
a complexity consistent programme and evaluation, this paper has argued that
IP : 192.168.39.211 On: Tue, 30 Nov 2021 08:23:57
programme framing and governance arrangements can provide the mechanisms for
applying complexity theory within evaluation, programme and policy processes.
The FiS evaluation provided findings of use from a complexity theory perspective.
A nuanced understanding of the role of free fruit and HPS processes within schools
Copyright The Policy Press
was achieved. While an explicit complexity consistent method would likely provide
Delivered by Ingenta
more detailed understanding of programme and context configurations, the FiS
evaluation did identify several interaction effects with other programmes and
‘emergent’ outcomes.The FiS evaluation showed that sympathetic system perspectives,
in this case socioecological theory utilising a mixed-method design, can provide
complexity-consistent results. This was also demonstrated in evaluating the Welsh
healthy school schemes (Rothwell et al, 2010). Less complexity-consistent was the
framing, governance and evaluation utilisation. It is suggested that network governance
arrangements can be a key vehicle for delivering complexity-consistent evaluation
framing and utilisation.
From a complexity frame, defining the boundaries of the system under study
has implications for legitimacy of evaluation questions, what data are collected and
which programme interactions are legitimate areas for evaluation and policy action
(Byrne and Uprichard, 2012; Midgley, 2000). Governance networks bring together
a variety of different problem definitions, preferred solutions and positions from
which to value outcomes (Hertting and Vedung, 2012). By explicitly bringing these
perspectives together both ideas for, and implementation of, solutions for complex
and ‘wicked’ problems may be achieved (Ferlie et al, 2011; Australian Public Service
Commission, 2007).
The case study highlights different processes required to bring together the two faces
of complexity identified by Vincent (2012). First, evaluation activity should inform
ongoing incremental programme developments. Non-linear interactions within
complex systems and ‘emergent’ system properties imply that unintended impacts
will occur (Eppel, 2012). One task of a programme governance network would be
to help steer systems in a desired direction through refinements across programmes
and actors. This requires timely reporting of evaluation findings and flexibility to
focus on emerging outcomes.
84Setting the context for using complexity theory in evaluation
The other use of evaluation is to inform broader policy settings and decisions
regarding overall programme continuation, adaptation or termination. Here again
it is suggested that the governance network is best placed to fully consider a
context-sensitive evaluation, detailing types of system trajectories and interactions.
Recommendations from the governance network would be made to government
ministers across relevant portfolios for collective decision making.
While this paper has focused on application of complexity theory, it is likely that
the analysis and proposed prescriptions would also apply to the practice and influence
of systems and realist approaches more widely.
There are some obvious limitations to the complexity-framed scenario outlined
here. First, devolving decisions to networks, and sharing political responsibility, requires
political will and trust from government ministers. In New Zealand, recent changes
to public service organisation to foster more cross-government working in complex
areas suggests a recognition that more networked approaches are required (Duncan
and Chapman, 2012). It is yet to be seen how consistent with a complexity frame these
efforts will be in practice. The study by Trenholm and Ferlie (2013) is a reminder of
IP : 192.168.39.211 On: Tue, 30 Nov 2021 08:23:57
how the success or failure of localised programmes can be influenced by established
traditions at a more macro level. Even if more complexity-consistent programmes and
evaluations are commissioned, the influence they have on wider political and policy
systems will likely be limited if reductionist and managerial frames are dominant at
Copyright The Policy Press
the macro level. A number of challenges in the actual FiS evaluation resulted from
Delivered by Ingenta
implementation time pressures, as well as changes in programme management and
political priorities. A complexity-framed, network-governance approach requires
sufficient time for genuine and ongoing stakeholder involvement and dialogue. The
second area of limitation is around the established assumptions of evidence and action
held by policy practitioners and within specific disciplines.
Sanderson (2009) notes that the dominant response to complexity is to increase
technical sophistication of methods, rather than adopt new ways of thinking.Within
the case study, informants identified a tension regarding competing assumptions of
‘good’ evidence between health and education disciplines. The complexity frame
suggested here implies new ways of thinking that challenge several assumptions of
evidence based policy making (EBPM) as often practised (Pawson, 2013). To apply
complexity theory, from evaluation design through to governance and policy design,
will require buy-in across the policy system including politicians, evaluation and policy
practitioners, and those implementing programmes across agencies.The ‘phase space’
of the policy system needs to allow for complexity theory informed policy practice.
Acknowledgements
Thank you to participants for generously giving time for this research; to Angelique
Praat, Sally Boyd, Nan Wehipeihana; Jackie Cumming and Jenny Neale for commenting
on earlier drafts; Marie Russell for contributing to participant interviews and document
review. This research was supported by the Marsden Fund Council from Government
funding, administered by the Royal Society of New Zealand.
85Mat Walton
References
Australian Public Service Commission, 2007, Tackling wicked problems: A public policy
perspective, Canberra: Australian Public Service Commission
Blackman, T, Wistow, J, Byrne, D, 2013, Using qualitative comparative analysis to
understand complex policy problems, Evaluation 19, 126–40
Blank, RH, Burau,V, 2010, Comparative health policy, Basingstoke: Palgrave Macmillan
Bogason, P, Musso, JA, 2006, The democratic prospects of network governance,
American Review of Public Administration 36, 3–18
Boyd, S, Dingle, R, Campbell R, King, J, Corter, A, 2007, Taking a bite of the apple:The
implementation of Fruit in Schools,Wellington: New Zealand Council for Educational
Research (NZCER)
Boyd, S, Dingle, R, Hodgen, E, Kind, J, Moss, M, 2009, The changing face of Fruit
in Schools: 2009 overview report, Wellington: New Zealand NZCER and Health
Outcomes International
Boyd, S, Moss, M, 2009, The changing face of Fruit in Schools: The 2008 case studies,
Wellington: New Zealand NZCER and Health Outcomes International, 98
IP : 192.168.39.211 On: Tue, 30 Nov 2021 08:23:57
Buijs, J-M, Eshuis, J, Byrne, D, 2009, Approaches to researching complexity in public
management, in Teisman, G, Buuren,AV, Gerrits L (eds), Managing complex governance
systems, New York, NY: Routledge, 37–55
Byrne, D, 2011, Applying social science: The role of social research in politics, policy and
Copyright The Policy Press
practice, Bristol: Policy Press
Delivered by Ingenta
Byrne, D, Callaghan, G, 2014, Complexity theory and the social sciences: The state of the
art, Oxford: Routledge
Byrne, D, Uprichard, E, 2012, Useful complex causality, in Kincaid, H (ed), The Oxford
Handbook of Philosophy of Social Science, Oxford: Oxford University Press, 109–29
Cairney, P, 2012, Complexity theory in political science and public policy, Political
Studies Review 10, 346–58
Cancer Control Taskforce, 2005, The New Zealand Cancer Control Strategy: Action Plan
2005–10, Wellington: Ministry of Health
Cooper, H, Geyer, R, 2008, Using ‘complexity’ for improving educational research
in health care, Social Science and Medicine 67, 177–82
Cushman, P, Clelland, T, 2012, Addressing health issues in New Zealand schools,
International Journal of Health Promotion and Education 50, 159–68
Dingle, R, Hodgen, E, Boyd, S, Shapleski, J. Kind, J, Moss, M, 2009, The changing
face of Fruit in Schools: Technical report, Wellington: NZCER and Health Outcomes
International
Duncan, G, Chapman, J, 2012, Better public services? Public management and the
New Zealand model, Public Policy 7, 151–66
Education Review Office, 2009, Schools’ progress towards meeting national administration
guideline 5 on food and nutrition: Part 2, Wellington: Education Review Office
English, B, 2008, Crown accounts show need for economic action – Press release 5
December, www.beehive.govt.nz/release/crown–accounts–show–need–economic–
action
Eppel, E, 2012,What does it take to make surprises less surprising? The contribution
of complexity theory to anticipation in public management, Public Management
Review 14, 881–902
Eppel, E, Matheson,A,Walton, M, 2011,Applying complexity theory to New Zealand
public policy: Principles for practice, Policy Quarterly 7, 48–55
86Setting the context for using complexity theory in evaluation
Ferlie, E, Fitzgerald, L, McGivern, G, Dopson, S, Bennett, C, 2011, Public policy
networks and ‘wicked problems’:A nascent solution? Public Administration 89, 307–24
Gatrell,AC, 2005, Complexity theory and geographies of health:A critical assessment,
Social Science and Medicine 60, 2661–71
Geyer, R, 2012, Can complexity move UK policy beyond ‘evidence–based policy
making’ and the ‘audit culture’? Applying a ‘complexity cascade’ to education and
health policy, Political Studies 60, 20–43
Goldstein J, 1999, Emergence as a construct: History and issues, Emergence 1, 49–72
Hertting, N,Vedung, E, 2012, Purposes and criteria in network governance evaluation:
How far does standard evaluation vocabulary take us? Evaluation 18, 27–46
Kania, A, Patel, AB, Roy, A,Yelland, GS, Nguyen, DTK,Verhoef, MJ, 2013, Capturing
the complexity of evaluations of health promotion interventions:A scoping review,
Canadian Journal of Program Evaluation 27, 65–91
Kauffman, S, 1995, At home in the universe:The search for the laws of self-organization and
complexity, New York, NY: Oxford University Press
Keshavarz, N, Nutbeam, D, Rowling, L, Khavarpour, F, 2010, Schools as social complex
IP : 192.168.39.211 On: Tue, 30 Nov 2021 08:23:57
adaptive systems:A new way to understand the challenges of introducing the health
promoting schools concept, Social Science and Medicine 70, 1467–74
Klijn, EH, 2008, Governance and governance networks in Europe: An assessment of
ten years of research on the theme, Public Management Review 10, 505–25
Copyright The Policy Press
Klijn, EH, Edelenbos, J, 2013, The influence of democratic legitimacy on outcomes
Delivered by Ingenta
in governance networks, Administration and Society 45, 627–50
Lemay, MA, Sá, C, 2012, Complexity sciences: Towards an alternative approach to
understanding the use of academic research, Evidence & Policy 8, 473–94
Mackenzie, A, 2005, The problem of the attractor: A singular generality between
sciences and social theory, Theory Culture Society 22, 45–65
McLean, RM, Hoek, JA, Buckley, S, Croxson, B, Cumming, J, Ehau, TH, Tanuvasa,
AF, Johnston, M, Mann, JI, Schofield, G, 2009, Healthy eating – healthy action:
Evaluating New Zealand’s obesity prevention strategy, BMC Public Health 9, 452
Meadows, D, 1999, Leverage points places to intervene in a system, Hartland, VT:
Sustainability Institute
Medd,W, 2002, Complexity and the social world, Social Research Methodology 5, 71–81
Merry, U, 1995, Coping with uncertainty: Insights from the new sciences of chaos, self-
organization, and complexity, Westport, CT: Praeger
Midgley, G, 2000, Systemic intervention, philosophy, methodology, and practice, New York:
Kluwer Academic / Planum Publishers
Ministry of Health, 2003, Healthy eating – healthy action oranga kai – oranga pumau: A
strategic framework, Wellington: Ministry of Health
Ministry of Health, 2009a, Health Report 20090889, 21 May, Wellington: Ministry
of Health
Ministry of Health, 2009b, Health Report 20091752, 9 October,Wellington: Ministry
of Health
Morçöl, G, 2001, What is complexity science? Postmodernist or postpositivist?
Emergence 3, 104–19
Morçöl, G, 2012, A complexity theory for public policy, New York, NY: Routledge
Morçöl, G, 2014, Complex governance networks: An assessment of advances and
prospects Complexity, Governance and Networks 1, 5–16
87Mat Walton
Morell, JA, 2010, Evaluation in the face of uncertainty, anticipating surprise and responding
to the inevitable, New York, NY: Guilford Press
Mowles, C, 2014, Complex, but not quite complex enough: The turn to the
complexity sciences in evaluation scholarship, Evaluation 20, 160–75
Nicolis, G, Prigogine, I, 1989, Exploring complexity: An introduction, New York, NY:
WH Freeman
Patton, MQ, 2011, Developmental evaluation: Applying complexity concepts to enhance
innovation and use, New York, NY: Guilford Press
Pawson, R, 2013, The science of evaluation, London: Sage
Pawson, R, Manzano-Santaella, A, 2012, A realist diagnostic workshop, Evaluation
18, 176–91
Petticrew, M, 2013, Public health evaluation: Epistemological challenges to evidence
production and use, Evidence & Policy 9, 87–95
Pledger, M, McDonald, J, Cumming, J, 2012, Increases in support structures for healthy
eating especially in low decile schools in New Zealand, Australian and New Zealand
Journal of Public Health 33, 543–9
IP : 192.168.39.211 On: Tue, 30 Nov 2021 08:23:57
Plsek, PE, Greenhalgh, T, 2001, Complexity science: The challenge of complexity in
health care, BMJ 323, 625–8
Richardson, KA, 2008, On the limits of bottom-up computer simulation: Towards
a nonlinear modeling culture, in Dennard, L, Richardson, KA, Morçöl, G (eds),
Copyright The Policy Press
Complexity and policy analysis, Goodyear, AZ: ISCE Publishing, 339–54
Delivered by Ingenta
Room, G, 2011, Complexity, institutions and public policy, Cheltenham: Edward Elgar
Publishing
Room, G, 2013, Evidence for agile policy makers: The contribution of transformative
realism, Evidence & Policy 9, 225–44
Rothwell, H, Shepherd, M, Murphy, S, Burgess, S, Townsend, N, Pimm, C, 2010,
Implementing a socialecological model of health in Wales, Health Education 110,
471–89
Ryall, T, 29 October 2009, Fruit in Schools future confirmed, Wellington: Minister
of Health
Sanderson, I, 2009, Intelligent policy making for a complex world: Pragmatism,
evidence and learning, Political Studies 57, 699–719
Smith, KE, Joyce, KE, 2012, Capturing complex realities: Understanding efforts to
achieve evidence-based policy and practice in public health, Evidence & Policy 8,
57–78
Snowden, D, 2011, Naturalizing sensemaking, in Mosier, K, Fischer, M (eds), Informed
by knowledge: Expert performance in complex situations, New York, NY: Psychology
Press, 223–4
Stacey R, 2012, Tools and techniques of leadership and management: Meeting the challenge
of complexity, Oxford: Routledge
Statistics New Zealand, 2013, New Zealand Official Yearbook 2012, www.stats.govt.
nz/yearbook2012
Stern, E, 2011, Foreword, in Forss, K, Marra, M, Schwartz, R (eds), Evaluating the
complex: Attribution, contribution and beyond, New Brunswick, NJ: Transaction
Publishers, vii–xi
Stern, E, Stame, N, Mayne, J, Forss, K, Davies, R, Befani, B, 2012, Broadening the range
of designs and methods for impact evaluations, London: Department for International
Development
88Setting the context for using complexity theory in evaluation
Treasury, 2010, Annex 6, Vote health expenditure and other reviews: Line-by-line
review of Ministry DE (Departmental Expenditure) and NDE (Non Departmental
Expenditure),Wellington: The Treasury New Zealand
Trenholm, S, Ferlie, E, 2013, Using complexity theory to analyse the organisational
response to resurgent tuberculosis across London, Social Science and Medicine 93,
229–37
Vincent, R, 2012, Insights from complexity theory for evaluation of development
action: Recognising the two faces of complexity, London, PANOS / IKM
(Information and Knowledge Management): Emergent Research Programme, 1–47
Walton, M, 2014, Applying complexity theory: A review to inform evaluation design,
Evaluation and Program Planning 45, 119–26
Walton, M, Waiti, J, Signal, L, Thomson, G, 2010, Identifying barriers to promoting
healthy nutrition in New Zealand primary schools, Health Education Journal 69,
84–94
Westhorp, G, 2012, Using complexity-consistent theory for evaluating complex
systems, Evaluation 18, 405–20
IP : 192.168.39.211 On: Tue, 30 Nov 2021 08:23:57
Williams, B. Hummelbrunner, R, 2011, Systems concepts in action: A practitioner’s toolkit,
Stanford, CA: Stanford University Press
Copyright The Policy Press
Delivered by Ingenta
89You can also read
