How Computational Modeling Can Force Theory Building in Psychological Science
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
970585
research-article2021
PPSXXX10.1177/1745691620970585Guest, MartinModeling Forces Theory Building
ASSOCIATION FOR
PSYCHOLOGICAL SCIENCE
Perspectives on Psychological Science
How Computational Modeling Can Force 2021, Vol. 16(4) 789–802
© The Author(s) 2021
Article reuse guidelines:
Theory Building in Psychological Science sagepub.com/journals-permissions
DOI: 10.1177/1745691620970585
https://doi.org/10.1177/1745691620970585
www.psychologicalscience.org/PPS
Olivia Guest1,2,3 and Andrea E. Martin1,4
1
Donders Centre for Cognitive Neuroimaging, Radboud University; 2Research Centre on Interactive Media,
Smart Systems and Emerging Technologies (RISE), Nicosia, Cyprus; 3Department of Experimental Psychology,
University College London; and 4Max Planck Institute for Psycholinguistics, Nijmegen, The Netherlands
Abstract
Psychology endeavors to develop theories of human capacities and behaviors on the basis of a variety of methodologies
and dependent measures. We argue that one of the most divisive factors in psychological science is whether
researchers choose to use computational modeling of theories (over and above data) during the scientific-inference
process. Modeling is undervalued yet holds promise for advancing psychological science. The inherent demands of
computational modeling guide us toward better science by forcing us to conceptually analyze, specify, and formalize
intuitions that otherwise remain unexamined—what we dub open theory. Constraining our inference process through
modeling enables us to build explanatory and predictive theories. Here, we present scientific inference in psychology
as a path function in which each step shapes the next. Computational modeling can constrain these steps, thus
advancing scientific inference over and above the stewardship of experimental practice (e.g., preregistration). If
psychology continues to eschew computational modeling, we predict more replicability crises and persistent failure
at coherent theory building. This is because without formal modeling we lack open and transparent theorizing. We
also explain how to formalize, specify, and implement a computational model, emphasizing that the advantages of
modeling can be achieved by anyone with benefit to all.
Keywords
computational model, theoretical psychology, open science, scientific inference
Psychology is a science that attempts to explain human a specified, formalized, and implemented computational
capacities and behaviors. This results in a wide range model and use it to model an example in which intu-
of research practices, from conducting behavioral and ition is insufficient in determining a quantity. Next, we
neuroscientific experiments to clinical and qualitative present our path model of how psychological science
work. Psychology intersects with many other fields, should be conducted to maximize the relationship
creating interdisciplinary subfields across science, tech- between theory, specification, and data. The scientific-
nology, engineering, mathematics, and the humanities. inference process is a function from theory to data—but
Here we focus on a distinction within psychological this function must be more than a state function to have
science that is underdiscussed: the difference in explan- explanatory force. It is a path function that must step
atory force between research programs that use formal, through theory, specification, and implementation
mathematical, and/or computational modeling and before an interpretation can have explanatory force in
those that do not, or, more specifically, programs that relation to a theory. Our path-function model also
explicitly state and define their models and those that enables us to evaluate claims about the process of
do not. doing psychological and cognitive science itself,
We start by explaining what a computational model
is, how it is built, and how formalization is required at
Corresponding Author:
various steps along the way. We illustrate how specify- Olivia Guest, Donders Centre for Cognitive Neuroimaging, Radboud
ing a model naturally results in better specified theories University
and therefore in better science. We give an example of E-mail: olivia.guest@ru.nl790 Guest, Martin
pinpointing where in the path questionable ways of view, the most crucial part of the process is creating a
conducting research occur, such as p-hacking (biasing specification—but even just creating an implementation
data analysis or collection to force statistical modeling (programming code) leverages more explicitness than
to return significant p values; e.g., Head et al., 2015). going from framework to hypothesis to data collection
Finally, we believe psychological science needs to use directly.
modeling to address the structural problems in theory
building that underlie the so-called replication crisis in,
for example, social psychology (see Flis, 2019). We What Is a Computational Model?
propose a core yet overlooked component of open And Why Build One?
science that computational modeling forces scientists
Let us calculate, without further ado, and see who
to carry out: open theory.
is right.
A Fork in the Path of Psychological —Gottfried Leibniz (Wiener, 1951)
Science Gottfried Leibniz predicted computational modeling
Psychological scientists typically ascribe to a school of when he envisaged a characteristica universalis that
thought that specifies a framework, a theoretical posi- allows scientists to formally express theories and data
tion, or at least some basic hypotheses that they then (e.g., formal languages, logic, programming languages)
set out to test using inferential statistics (Meehl, 1967; and a calculus ratiocinator that computes the logical
Newell, 1973). Almost every article in psychological consequences of theories and data (e.g., digital comput-
science can be boiled down to introduction, methods, ers; Cohen, 1954; Wiener, 1951). Computational model-
analysis, results, and discussion. The way we approach ing is the process by which a verbal description is
science is nearly identical: We ask nature questions by formalized to remove ambiguity, as Leibniz correctly
collecting data and then report p values, more rarely predicted, while also constraining the dimensions a
Bayes factors or Bayesian inference, or some qualitative theory can span. In the best of possible worlds, model-
measure. Computational models do not feature in the ing makes us think deeply about what we are going to
majority of psychology’s scientific endeavors. Most psy- model (e.g., which phenomenon or capacity), in addi-
chological researchers are not trained in modeling tion to any data, both before and during the creation
beyond constructing statistical models of their data, of the model and both before and during data collec-
which are typically applicable off the shelf. tion. It can be as simple as the scientist asking, “How
In contrast, a subset of researchers—formal, math- do we understand brain and behavior in this context,
ematical, or computational modelers—take a different and why?” By thinking through how to represent the
route in the idea-to-publication pipeline. They construct data and model the experiment, scientists gain insight
models of something other than the data directly; they into the computational repercussions of their ideas in
create semiformalized or formalized versions of scien- a much deeper and explicit way than by just collecting
tific theories, often creating (or least amending) their data. By providing a transparent genealogy for where
accounts along the way. Computational modelers are predictions, explanations, and ideas for experiments
researchers who have the tools to be acutely aware of come from, the process of modeling stops us from
the assumptions and implications of the theory they are atheoretically testing hypotheses—a core value of open
using to carry out their science. This awareness comes, science. Open theorizing, in other words explicitly stat-
ideally, from specification and formalization, but mini- ing and formalizing our theoretical commitments, is
mally, it also comes from the necessity of writing code done by default as a function of the process.
during implementation. Through modeling, even in, or especially in, failures
Involving modeling in a research program has the we hone our ideas: Can our theory be formally speci-
effect of necessarily changing the way the research fied, and if not, why not? Thus, we may check whether
process is structured. It changes the focus from testing what we have described formally still makes sense in
hypotheses generated from an opaque idea or intuition light of our theoretical commitments. It aids both us as
(e.g., a theory that has likely never been written down researchers communicating with each other and those
in anything other than natural language, if that) to test- who may wish to apply these ideas to their work out-
ing a formal model of the theory as well as continuing side science (e.g., in industrial or clinical settings).
to be able to generate and test hypotheses using empiri- One of the core properties of models is that they
cal data. Computational modeling does this by forcing allow us to “safely remove a theory from the brain of
scientists to explicitly document an instance of what its author” (A. J. Wills, personal communication, May
their theory assumes, if not what their theory is. In our 19, 2020; see also Wills et al., 2017; Wills & Pothos,Modeling Forces Theory Building 791
2012), thus allowing the ideas in one’s head to run on
other computers. Modeling also allows us to compare a b
models based on one theory with those based on Two 12-in. Pizzas One 18-in. Pizza
another and compare different parameter values’ effects
within a model, including damaging models in ways
that would be unethical in human participants (e.g.,
“lesioning” artificial neural network models; see Guest
et al., 2020). One of the only situations in which mul-
tiple theories can be distinguished in a formal setting
is when they can make sense of the available data (e.g.,
Levering et al., 2019; see also, however, Cox & Shiffrin,
in press; Navarro, 2019; Wills & Pothos, 2012).
We now walk the reader through building a compu-
tational model from scratch to illustrate our argument
and then present a path function of research in psy-
chological science. We emphasize that often merely
building a formal model of a problem is not enough—
actually writing code to implement a computational
model is required to understand the model itself. Area = 2 × π62 = 226 in.2 Area = π92 = 254 in.2
Fig. 1. The pizza problem. Something like comparing the two
options presented here can appear counterintuitive, although we all
The pizza problem learn the formula for the area of a circle in primary school. Compare
All models are wrong but some are more wrong (a) two 12-in. pizzas and (b) one 18-in. pizza (all three pizzas are to
scale). Which order would you prefer?
than others.
—based on Box (1976) and Orwell (1945) Experience is seeing our ideas being executed by a
computer, giving us the chance to debug scientific think-
Imagine it is Friday night, and your favorite pizzeria ing in a very direct way. If we do not make our thinking
has a special: two 12-in. pizzas for the price of one explicit through formal modeling, and if we do not
18-in. pizza. Your definition of a good deal is one in bother to execute (i.e., implement and run our specifica-
which you purchase the most food. Is this a good deal? tion through computational modeling), we can have
A Twitter post (Fermat’s Library, 2019) said “a useful massive inconsistencies in our understanding of our
counterintuitive fact: one 18 inch pizza has more ‘pizza’ own model(s). We call this issue “the pizza problem.”
than two 12 inch pizzas”—along with an image similar Herein we model the most pizza for our buck—over-
to Figure 1. The reaction to this tweet was largely sur- kill for scientific purposes but certainly not for peda-
prise or disbelief. For example, one follower replied, gogical ones. For any formalized specification, including
“But two pizzas are more than one” (Sykes, 2019). Why that for the pizza orders shown in Figure 1, simplifica-
were people taken aback? tions need to be made, so we choose to represent piz-
When it comes to comparing the two options in zas as circles. Therefore, we define the amount of food
Figure 1, although we all agree on how the area of a φ per order option i as
circle is defined, the results of the “true” model, that
one 18-in. pizza has more surface and therefore is more φi = N i πRi2 , (1)
food, are counterintuitive. Computational modeling can
demonstrate how one cannot always trust one’s gut. To where i is the pizza-order option, N is the number of
start, one must create (a) a verbal description, a con- pizzas in the order, and the rest is the area of a circle.
ceptual analysis, and/or a theory; (b) a formal (or for- We also propose a pairwise decision rule:
malizable) description, that is, a specification using
mathematics, pseudocode, flowcharts, and so on; and i, if φi > φj
(c) an executable implementation written in program- ω( φi , φj ) = , (2)
ming code (for an overview of these steps, see Fig. 2, j , otherwise
red area). This process is the cornerstone of computa-
tional modeling and by extension of modern scientific where the output of the ω function is the order with
thought, enabling us to refine our gut instincts through the most food. This is the model that everyone would
experience. have claimed to have running in their heads, but they792 Guest, Martin
still were surprised—an expectation violation occurred—
when faced with the actual results. How do we ensure ACT-R, Connectionism,
Framework
we are all running the same model? We execute it on SOAR, Working Memory
a computer that is not the human mind. To make this
model computational, we move from specification to
implementation (consider where we are in the path
shown in Fig. 2). We notice Equation 1 is not wrong, Prospect Theory,
Theory
Rescorla-Wagner, SUSTAIN
but could be defined more usefully as
N
φi = ∑πR , 2
j (3) Mathematics, Natural
j =1 Specification
Language, TLA+, Z
where j is the current pizza, allowing us to sum over
all pizzas N within food order i.
This change allows generalization of the model (both Models Written in Code
in the specification above and the implementation Implementation
(e.g., C, Python, R)
below) to account for different radii per order (i.e., in
the future we can compare an 11-in. pizza plus a 13-in.
pizza with one 18-in. pizza). One possible implementa-
tion (in Python) of our pizza model looks like this: “Group A Will Be Faster
Hypothesis
Than Group B”
import numpy as np
import math
def food(ds): ANOVA, Linear Regression,
Data MVPA, SEM, t Test
'''
Amount of food in an order as a function Fig. 2. One of many possible paths (in blue) that can be used
of the diameters per pizza (eq. 3). to understand and describe how psychological research is carried
out, with examples of models at each step shown on the left (in
'''
green). Each research output within psychological science can be
described with respect to the levels in this path. The three levels
return (math.pi * (ds/2)**2).sum() superimposed on a red background (theory, specification, implemen-
tation) are those that are most often ignored or left out from research
descriptions. ACT-R = adaptive control of thought–rational; SUSTAIN =
# Order option a in fig. 1, two 12' ' supervised and unsupervised stratified adaptive incremental network;
pizzas: ANOVA = analysis of variance; MVPA = multivariate pattern analysis;
two_pizzas = np.array([12, 12]) SEM = structural equation modeling.
# Option b, one 18' ' pizza:
one_pizza = np.array([18]) Computational modeling, when carried out the way
we describe herein, is quintessentially open science:
# Decision rule (eq. 2): Verbal descriptions of science, specifications, and
print(food(two_pizzas) > implementations of models are transparent and thus
food(one_pizza)) open to replication and modification. If one disagrees
with any of the formalisms, they can plug in another
Note that this implementation change, which we choose decision rule or definition of the amount of food or
to percolate upward, editing our specification, does not even another aspect of the order being evaluated (e.g.,
affect the verbal description of the model. By the same perhaps they prefer more crust than overall surface).
token, a change in the code to use a for-loop in the defini- Computational modeling—when done the way we
tion of the food() function would affect neither the describe, which requires the creation of specifications
specification nor the theory in this case. This is a core and implementations—affords open theorizing to go
concept to grasp: the relationships between theory, speci- along with open data, open-source code, and so on. In
fication, and implementation—consider our movements contrast to merely stating two 12-in. pizzas offer more
up and down the path as depicted in Figure 2. food than one 18-in. pizza, a computational model canModeling Forces Theory Building 793
be generalized and can show our work clearly. Through only if an expectation violation is resolved by first mov-
writing code, we debug our scientific thinking. ing upward. Downward transitions can be thought of
as functions in which the input is the current layer and
the output is the next. Upward transitions are more
Model of Psychological Science complex and involve adjusting (e.g., a theory given
some data) and can involve changes to levels along the
Theory takes us beyond measurement in a way
way to obtain the required theory-level update. With
which cannot be foretold a priori, and it does so
respect to why we might want to move upward out of
by means of the so-called intellectual experiments
choice and recalling the case of the pizza model above,
which render us largely independent of the defects
we updated the specification (changing Equation 1 to
of the actual instruments.
Equation 3) because we thought about the code/
—Planck (1936, p. 27) implementation more deeply and decided it is worth
updating our formal specification (Equation 3). Down-
In this section, we describe an analytical view of psy- ward motion is not allowed if a violation occurs (e.g.,
chological research, shown in Figure 2. Although other our model at the current step is not in line with our
such models exist for capturing some aspect of the expectations). Once this violation is resolved by moving
process of psychological science (e.g., Haig, 2018; to any step above, we may move downward, respecting
Haslbeck et al., 2019; Kellen, 2019; van Rooij & Baggio, the serial ordering of the levels. For example, when the
2021), our model proposes a unified account that dem- data do not confirm the hypothesis, we must move
onstrates how computational modeling can play a radi- upward and understand why and what needs to be
cal and central role in all of psychological research. amended in the levels above the hypothesis. Attempting
We propose that scientific outputs can be analyzed to “fix” things at the hypothesis level is known as
using the levels shown in the left column of Figure 2. hypothesizing after the results are known (or HARKing;
Scientific inquiry can be understood as a function from Kerr, 1998). In the case of the pizza model, an expecta-
theory to data and back again, and this function must tion violation occurs when we realize that the one pizza
pass through several states to gain explanatory force. is more food. At that point, we reevaluate our unspeci-
The function can express a meaningful mapping, trans- fied/implicit model and move back up to the appropri-
formation, or update between a theory at time t and ate level to create a more sensible account.
that theory at time t + 1 as it passes through specifica- At least implicitly, every scientific output is model-
tion and implementation, which enforces a degree of and theory-laden (i.e., contains theoretical and model-
formalization. We note that each level (in blue) can, ing commitments). By making these implicit models
but does not have to, involve the construction of a explicit via computational modeling the quality, useful-
(computational) model for that level, with examples of ness, and verisimilitude of research programs can be
models shown in the left column (in green) connected secured and ascertained. The three levels with a red
by a dotted line to their associated level. If a level is background in Figure 2 (theory, specification, and
not well understood, making a model of that level helps implementation) are those that we believe are left
to elucidate implicit assumptions, addressing “pizza implicit in most of psychological research—this is espe-
problems.” cially so in parts of psychological science that have
A path function is a function in which the output been most seriously affected by the so-called replica-
depends on a path of transformations the input under- tion crisis. This tendency to ignore these levels results
goes. Path functions are used in thermodynamics to from the same process by which theory and hypothesis
describe heat and work transfer; an intuitive example are conflated (Fried, 2020; Meehl, 1967; Morey et al.,
is distance to a destination being dependent on the 2018) and by which models of the data are taken to
route and mode of transport taken. The path function be models of the theory: “theoretical amnesia”
moves from top to bottom in terms of dependencies, (Borsboom, 2013). When models of the data are seen
but the connections between each level and those adja- as models of the theory, potentially bizarre situations
cent are bidirectional (represented by large blue and can arise—eventually forcing dramatic rethinkings of
small black arrows). Connections capture the adding (sub)fields (e.g., Jones & Love, 2011).
or removing, loosening or tightening, of constraints that
one level can impose on those above or below it.
Framework
In our model depicted in Figure 2, the directionality
of transitions is constrained only when moving down- A framework is a conceptual system of building blocks
ward. Thus, at any point transitions moving upward are for creating facsimiles of complex psychological systems
permissible, whereas moving downward is possible (see Fig. 2, topmost level). A framework is typically794 Guest, Martin
described using natural language and figures but can Specification
also be implemented in code such as ACT-R (or adap-
tive control of thought–rational; Anderson & Lebiere, A specification is a formal (or formalizable) description
1998) and SOAR (Newell, 1992). Some frameworks of a system to be implemented on the basis of a theory
appear superficially simple or narrow, such as the con- (Fig. 2, third level). It provides a means of discriminating
cept of working memory (Baddeley, 2010) or dual- between theory-relevant (i.e., closer to the core claims
systems approaches (Dayan & Berridge, 2014; Kahneman, of the theory) and theory-irrelevant auxiliary assump-
2011), whereas others can be all-encompassing such as tions (Cooper & Guest, 2014; Lakatos, 1976). Specifica-
unified theories of cognition (Newell, 1990) or con- tions provide both a way to check whether a computational
nectionism (McClelland et al., 1986). model encapsulates the theory and a way to create a
In the simplest case a framework is the context—the model even if the theory is not clear enough, simply by
interpretation of the terms of a theory (Lakatos, 1976). constraining the space of possible computational mod-
Frameworks usually require descending further down els. Specifications can be expressed in natural-language
the path before they can be computationally modeled sentences, mathematics, logic, diagrams, and formal
(Hunt & Luce, 1992; Vere, 1992). Although it is possible specification languages, such as Z notation (Spivey &
to avoid explicit frameworks, it is “awkward and unduly Abrial, 1992) and TLA+ (Lamport, 2015).
laborious” (Suppes, 1967, p. 58) to work without one The transition to code from specification has been
and thus depends on the next level down in the path automated in some cases in computer science (Monperrus
to do all the heavy lifting. et al., 2008). In psychological science, creating an
It is not the case that all psychological models are implementation typically involves taking the specifica-
or can be evaluated against data directly. For example, tion implicitly embedded in a journal article and writing
ACT-R is certainly not such a model: We have to code that is faithful to it. Without specifications, we can
descend the path first, creating a specific theory, then neither debug our implementations nor properly test
a specification, then an implementation, and then gen- our theories (Cooper et al., 1996; Cooper & Guest, 2014;
erate hypotheses before any data can be collected (see Miłkowski et al., 2018).
Cooper, 2007; Cooper et al., 1996).
Implementation
Theory An implementation is an instantiation of a model cre-
A theory is a scientific proposition—described by a ated using anything from physical materials, (e.g., a
collection of natural-language sentences, mathematics, scale model of an airplane; Morgan & Morrison, 1999),
logic, and figures—that introduces causal relations with to software (e.g., a git repository; Fig. 2, fourth level).
the aim of describing, explaining, and/or predicting a set A computational implementation is a codebase written
of phenomena (Lakatos, 1976; see Fig. 2, second level). in one or more programming languages that constitutes
Examples of psychological theories are prospect theory a software unit and embodies a computational model.
(Kahneman & Tversky, 1979), the Rescorla-Wagner model Although the concept of an implementation is simple
for Pavlovian conditioning (Rescorla & Wagner, 1972), to grasp—perhaps what most psychologists think when
and SUSTAIN (supervised and unsupervised stratified they hear the term model—it might appear to be the
adaptive incremental network), an account of catego- hardest step. This is arguably not the case. Provided
rization (Love et al., 2004). one follows the steps in Figure 2, a large proportion of
To move to the next level and produce a specifica- the heavy lifting is done by all the previous steps.
tion for a psychological theory, we must posit a plau- In some senses, implementations are the most dis-
sible mechanism for the specification model to define. posable and time-dependent parts of the scientific pro-
As can be seen from our path, direct comparisons to cess illustrated in Figure 2. Very few programming
data can happen only once a model is at the correct languages stay in vogue for more than a decade, ren-
level. However, not all psychological models must be dering code older than even a few months in extreme
(or can be) evaluated against data directly. Theoretical cases unrunnable without amendments (Cooper &
computational models allow us to check whether our Guest, 2014; Rougier et al., 2017). This is not entirely
ideas, when taken to their logical conclusions, hold up damaging to our enterprise because the core scientific
(e.g., Guest & Love, 2017; Martin, 2016, 2020; Martin & components we want to evaluate are theory and speci-
Baggio, 2020; van Rooij, 2008). If a theory cannot lead fication. If the computational model is not reimple-
to coherent specifications, it is our responsibility as mentable given the specification, it poses serious
scientists to amend or, more rarely, abandon it in favor questions for the theory (Cooper & Guest, 2014). This
of one that does. constitutes an expectation violation and must beModeling Forces Theory Building 795
addressed by moving upward to whichever previous p value (e.g., because of a failure to replicate) could be
level can amend the issue. However, it is premature to enough to destroy the research program. Any theories
generalize from the success or failure of one implemen- based on hypohacking will crumble if no bidirectional
tation if it cannot be recreated according to the speci- transitions in the path were carried out, especially
fication because we have no reason to assume it is within the steps highlighted in red in Figure 2. Having
embodying the theory. Whether code appropriately built a computational account researchers can avoid
embodies a theory can be answered only by iterating the confirmation bias of hypohacking, which cheats the
through theory, specification, and implementation. path and skips levels.
Running our computational model’s code allows us
to generate hypotheses. For example, if our model
Data
behaves in a certain way in a given task (e.g., has
trouble categorizing some types of visual stimuli more Data are observations collected from the real world or
than others), we can formulate a hypothesis to test this from a computational model (Fig. 2, sixth level). Data
behavior. Alternatively, if we already know this phe- can take on many forms in psychological science, the
nomenon occurs, computational modeling is a useful most common being numerical values that represent
way to check that our high-level understanding does variables as defined by our experimental design (e.g.,
indeed so far match our observations. If our implemen- reaction times, questionnaire responses, neuroimaging).
tation displays behavior outside what is permitted by Most undergraduate psychology students know some
the specification and theory, then we need to adjust basic statistical modeling techniques. Tests such as
something because this constitutes a violation. It might analysis of variance, linear regression, multivariate pat-
be that the theory is underspecified and that this behav- tern analysis, structural equation modeling, the t test,
ior should not be permissible, in which case we might and mixed-effects modeling (e.g., Davidson & Martin,
need to change both the specification and the imple- 2013) are all possible inferential statistical models of
mentation to match the theory (Cooper & Guest, 2014). data sets.
Such a cycle of adjustments until the theory is captured Because data are theory-laden, they can never be
by the code and the code is a strict subset of the theory free from, or understood outside of, the assumptions
are necessary parts of the scientific process. Loosening implicit in their collection (Feyerabend, 1957; Lakatos,
and tightening theory, specification, and implementa- 1976). For example, functional MRI (fMRI) data rest on
tion never ends—it is the essence of theory develop- our understanding of electromagnetism and of the
ment in science. blood-oxygen-level-dependent signal’s association with
neural activation. If any of the scientific theories that
support the current interpretation of fMRI data change
Hypothesis then the properties of the data will also change.
A hypothesis is a narrow testable statement (Fig. 2, fifth If the data model does not support the hypothesis
level). Hypotheses in psychological science focus on (an expectation violation), we can reject the experi-
properties of the world that can be measured and evalu- mental hypothesis with a certain confidence. However,
ated by collecting data and running inferential statistics. we cannot reject a theory with as much confidence.
Any sentence that can be directly tested statistically can The same caution is advised in the inverse situation
be a hypothesis, for example, “the gender similarities (Meehl, 1967). For example, a large number of studies
hypothesis which states that most psychological gender have collected data on cognitive training over the past
differences are in the close to zero (d ≤ 0.10) or small century, and yet consensus on its efficacy is absent
(0.11 < d < 0.35) range” (Hyde, 2005, p. 581). (Katz et al., 2018). To escape these problems and under-
Hypothesis testing is unbounded without iterating stand how data and hypothesis relate to our working
through theory, specification, and implementation and theory we must ascend the path and contextualize our
creating computational models. The supervening levels findings using computational modeling. These viola-
constrain the space of possible hypotheses to be tested. tions cannot be addressed by inventing new hypoth-
Testing hypotheses in an ad hoc way—what we could eses that conveniently fit our data (i.e., HARKing)
dub hypohacking—is to the hypothesis layer what but by asking what needs to change in our theoretical
p-hacking is to the data layer (Head et al., 2015). understanding.
Researchers can concoct any hypothesis, and given suf-
ficient data a significant result is likely to be found
Harking back to pizza
when comparing, for example, two theoretically base-
less groupings. Another way to hypohack is to atheo- The pizza example (purposefully chosen in part because
retically run pilot studies until something works. When it is simple and devoid of psychological constructs,
research is carried out this way, losing the significant which bias reader’s opinions toward one formalism over796 Guest, Martin
another) can be decomposed readily into the six levels Arguably—and this is one of the core points of this
shown in Figure 2. At the framework level we have the article—had we not ignored the steps in red and cre-
concepts of pizza, food, and order because we want to ated a theory, specification, and implementation explic-
compare the total amount of food per order. These are itly, we would have been on better footing from the
the building blocks for any account that involves decid- start. And so it is demonstrated that applying the path
ing between orders of food made up of pizzas, even if model adds information to the scientific-inference pro-
we disagree on which aspects of the order (e.g., money, cess. Still, we managed to document and update our
speed of delivery), food (e.g., calories, ingredients), or less-than-useful assumptions by going back and for-
pizza (e.g., crust, transportability) we will eventually mally and computationally capturing our ideas. We
formally model and empirically test. should all strive not to ignore these vital steps by
Then at the theory level, there are essentially two directly focusing on them, either ourselves or by mak-
theories: the original (implicit) theory T0 that the num- ing sure the literature contains this explicit formal and
ber of pizzas per order corresponds to the amount of computational legwork.
food in that order and the post hoc corrected (explicit)
T0 that the surface areas of the pizzas per order cor-
What our path-function model offers
respond to the amount of food in that order. To get to
T1, we created a specification, created an implementa- We have denoted the boundaries and functions of levels
tion, and refined the specification—we discuss exactly within the scientific-inference process in psychological
how this happened using the path model of Figure 2. research—many should be familiar with similar layers
Before obtaining T1, we descended the path by going of abstraction from computer science and levels of
from basically framework to hypothesis (bypassing the analysis from Marr and Poggio (1976). Simpler, more
red area completely; recall T0 was not explicitly stated abstract descriptions appear higher up the path,
at all, let alone formalized, at the beginning) to gener- whereas more complex descriptions of psychological
ate the very clear prediction (and thus testable hypoth- science are lower down the path—for example, data
esis) that the order with two pizzas is more food than are much less compressed as a description of an experi-
the order with one. Because we skipped the parts of ment than is a hypothesis. Each level is a renormaliza-
the path that required formalizing our ideas (shown in tion, or coarser description, of those below (DeDeo,
red in Fig. 2), we are faced with an expectation viola- 2018; Flack, 2012; Martin, 2020). Higher levels contain
tion. We believed that two 12-in. pizzas are more food fewer exemplars than lower levels. Moving through the
than one 18-in. pizza (recall Fig. 1), and we also path of scientific inference is a form of dimensionality
believed that the food per order is a function of the reduction or coordinate transformation. Not only are
surface area of the pizzas. Therefore, we realize our there often no substantive nor formalized theories for
own ideas about the amount of food per order are some data sets in practice (causing chaos; see Forscher,
incompatible with themselves (what we dub the pizza 1963), but the principle of multiple realizability (Putnam,
problem), as well as what we know about the world 1967) also implies that for every theory there are infi-
from other sources (imagine if we had weighed the nitely many possible implementations consistent with
pizzas per order, for example). Had this been a real it and data sets that can be collected to test it (Blokpoel,
research program (and not a fictional example), we 2018). This helps to contextualize studies that show
would have descended all the way and collected empir- divergence in data-modeling decisions given the same
ical data on the pizzas by, for example, weighing them. hypotheses (e.g., Botvinik-Nezer et al., 2020; Silberzahn
This act of collecting observations would have further et al., 2018).
solidified the existence of an expectation violation Open theories (i.e., those developed explicitly,
because the two pizzas would have been found to, for defined formally, and explored computationally, in line
example, have less mass, thus falsifying both our with Fig. 2) are more robust to failures of replication
hypothesis and indirectly T0. of any single study from which they might derive some
At the point of an expectation violation, we decided support because of the specific way the path has been
to address the steps we skipped in the red area, so we followed to develop and test them. For example, if the
moved upward to create a formal specification S0 impetus or inspiration for theory development is a
embodied by Equations 1 and 2. We then attempted to single study (that is thereafter found not to be repli-
descend from S0 to create an implementation I0, which cable) because we then move to the red area, refine
led to refining our specification, thus creating S1 (Equa- our ideas, and then drop back down to test them again,
tions 2 and 3). We are now fully in the throes of formal we will avoid dependence on a single (potentially prob-
and computational modeling by cycling through the lematic) study. Failures to replicate can not only be
steps shown in Figure 2 in red. detected but also explained and perhaps even driveModeling Forces Theory Building 797 theory creation as opposed to just theory rejection. made it back safely. In many cases theory development Thus, building a theory explicitly as laid out in Figure involves analysis at the data level, as an inspiration or 2, even if based on some hypo- and p-hacking, means impetus, and then a lot of scientific activity within the once a phenomenon is detected we ascend the path levels: theory, specification, and implementation. This and spend time formalizing our account (e.g., Fried, is why we do not impose any constraints on moving 2020). Our path model asks for formalization using upward in Figure 2, only on moving downward. specifications and implementations (or indeed anything On the other hand, our path-function model allows more comprehensive than an individual study; see us to pinpoint on which level QRPs are taking place Head et al., 2015); thus, when our model is used, “sins” and how to avoid them. Different QRPs occur at differ- that are out of individual scientists’ control—such as ent levels, for example, p-hacking at the data level, questionable research practices (QRPs; see John et al., HARKing at the hypothesis level, and so on. HARKing 2012) committed by other labs or publication bias com- does not resolve expectation violations that occur when mitted by the system as a whole—can be both discov- the data meet the hypothesis—it is not, for example, ered and controlled for in many cases. theorizing after the results are known, which is part of Thinking about our science with reference to Figure the scientific practice of creating modeling accounts. 2 allows us to discuss and decide where in the path To retrofit a hypothesis onto a data set does not con- claims about science are being made (i.e., not only stitute resolving a violation because this de novo allowing us to evaluate claims about the phenomena hypothesis is not generated directly or indirectly by a being examined, modeled, and so on, but also to evalu- theory. If we start out with a hypothesis and collect data ate general claims about how we conduct research or that reject our hypothesis, the violation has not occurred about how not to conduct research). For example, the uniquely at the hypothesis level because the hypothesis claim that “science is posthoc, with results, especially has been generated (via the intervening levels) by the unexpected results, driving theory and new applica- theory. This is essentially the opposite to conjuring a tions” (Shiffrin, 2018) is not incompatible with guarding new hypothesis (HARKing) that exists only in the sci- against HARKing because one cannot have an account entific literature because it has been “confirmed” by of a phenomenon without having access to some data— data—data collected to test a different hypothesis. anecdotal, observational, and/or experimental—that It is at the data and hypothesis levels that preregis- guide one to notice the phenomenon in the first case. tration and similar methods attempt to constrain science Theories in psychology result from protracted to avoid QRPs (e.g., Flis, 2019; Szollosi et al., 2019). To thought about and experience with a human cognitive ensure scientific quality, however, we propose that pre- capacity. Scientists immerse themselves in deep thought registration is not enough because it serves only to about why and how their phenomena of study behave. constrain the data and hypothesis spaces. Researchers This basic principle of developing theories is captured who wish to develop their formal account of a capacity in the example of Wald’s investigation into optimally must ascend the path instead of, or in addition to, for (and thus minimally because of weight) armoring air- example, preregistering analysis plans. Preregistration craft to ensure pilots returned safely during World War cannot on its own evaluate theories. We cannot coher- II (Mangel & Samaniego, 1984). Planes returned after ently describe and thus cannot sensibly preregister what engaging with the Nazis with a smattering of bullet we do not yet (formally and computationally) under- holes that were distributed in a specific way: More holes stand. Indeed, theories can and should be computation- were present in the fuselage than in the engines, for ally embodied and pitted against each other without example. Wald explained post hoc why and how the gathering or analyzing any new data. To develop, evalu- holes were correlated to survival. Contrary to what one ate, and stress-test theories, we need theory-level con- might expect, areas with the least holes would benefit straints on and discussions about our science. Figure 2 from armor. Wald theorized that planes in general were can serve as a first step in the right direction toward likely hit by bullets uniformly, unlike the planes in the such an ideal. data set; aircraft hit in the engines did not make it home By the same token, our path model allows us to and so were not present in the data set; and, therefore, delineate and discuss where computational modeling armor should be placed over the engines, the area with itself has been compromised by QRPs occurring at the the fewest bullet holes. This is not HARKing—this is specification and implementation levels. A typical case formal modeling. Wald moved upward from the data of this is when authors report only partial results of (distribution of bullet holes) to a theory (survivor bias) implementing a specification of their theory; for exam- and created an explicit formal model that could explain ple, only some implementations show the required or and predict the patterns of the bullets in planes that predicted patterns of behavior (Guest et al., 2020). As
798 Guest, Martin
mentioned, the solution is to cycle within the red full scale of the path, reaping the benefits of formally
area of Figure 2 to ensure theory-, specification-, and testing and continuously improving their theories, those
implementation-level details are indeed assigned to who eschew modeling miss out on fundamental scien-
and described at the correct level. Failing to do that, tific insights. By formalizing a research program, we
we propose, is a type of QRP. can search and evaluate in a meticulous way the space
Computational modeling can be seen as mediating of the account proposed (i.e., “theory-guided scientific
between theory and data (Morgan & Morrison, 1999; exploration”; Navarro, 2019, p. 31). As shown using the
Oberauer & Lewandowsky, 2019). Asking if we can pizza example, nonmodelers remain unaware of pizza
build a model of our theory allows us to understand problems and may not realize they are implicitly run-
where our theoretical understanding is lacking. Claims ning a different model (in their head) to what they
are typically not falsifiable—not usually directly testable specify.
at the framework or theory level—but become more so
as we move downward. We thus iterate through theory,
Discussion
specification, and implementation as required until we
have achieved a modeling account that satisfies all of We hope to spark a dialogue on the radical role com-
the various constraints imposed by empirical data, as putational modeling can play by forcing open theoriz-
well as collecting empirical data based on hypotheses ing. We also presented a case study in building a basic
generated from the computational model. Is an imple- computational model, providing a useful guide to those
mentation detail pivotal to a model working? Then it who may not have modeled before. Models, especially
must be upgraded to a specification detail (Cooper when formalized and run on a digital computer, can
et al., 1996; Cooper & Guest, 2014). Mutatis mutandis shine a light on when our scientific expectations are
for details at the specification level and so on—meaning violated. To wit, we presented a path-function model
that details at every level can be upgraded (or down- of science, radically centering computational modeling
graded) as required. This process is even useful in the at the core of psychological science. Computational
case of “false” models; that is, computational accounts models cannot replace, for example, data or verbal
that we do not agree with can still improve our under- theories, but the process of creating a computational
standing of phenomena (e.g., Wimsatt, 2002; Winsberg, account is invaluable and informative.
2006). There are three routes that psychology can take,
As mentioned, cycling through the steps in Figure 2 mirroring Newell (1973): The first is bifurcating between
shines a direct light on what our theoretical commit- research programs that use modeling and those that do
ments are in deep ways. Mathematically specifying and/ not; the second is uniting research programs inasmuch
or computationally implementing models, for example, as they contain some modeling to force the creation,
can demonstrate that accounts are identical or overlap refinement, and rejection of theories; and the third is
even when their verbal descriptions (i.e., informal spec- continuing to ask questions that are not secured to a
ifications) are seemingly divergent. This can result from sound theoretical mooring via computational modeling.
(a) multiple theories being indeed more similar in These are not completely mutually exclusive possibilities—
essence than previously thought, paving the way for some components from each can be seen in the
theoretical unification of a seemingly disjointed litera- present.
ture (e.g., Kelly et al., 2017), or (b) theories that are For bifurcation of the field, theoreticians, scientists
indeed different being less computationally explored who mostly inhabit the red area of Figure 2, will be
and thus less constrained in their current state (e.g., free to practice modeling, for example, without having
Olsson et al., 2004). These kinds of discoveries about to run frequentist statistics on their models if inappro-
how we compartmentalize, understand, and predict priate. No constraints will be put on individual scien-
human capacities are why iterating over—and thus tists to pick a side (e.g., Einstein was active in theoretical
refining—theory, specification, and implementation is and experimental physics). Unlike it is now, it will be
vital. easy to publish work containing only modeling at the
Research programs light on modeling do not have a theory level without direct reference to data (something
clear grasp on what is going on in the area highlighted rare currently, although possible; e.g., Guest & Love,
in red in Figure 2. These areas of psychological science 2017; Martin, 2016, 2020).
might have many, often informal, theories, but this is In the case of uniting research programs, mass coop-
not enough (Watts, 2017). Neither is more data—how- eration to work on “larger experimental wholes”
ever open, they will never solve the issue of a lack of (Newell, 1973, p. 24) is perhaps realistic given projects
formal theorizing. Data cannot tell a scientific story; that involve many labs are commonplace (e.g., Botvinik-
that role falls to theory, and theory needs formalization Nezer et al., 2020; Silberzahn et al., 2018). We advise
to be evaluated. Thus, whereas modelers often use the cautious optimism because these collaborations areModeling Forces Theory Building 799
operating only at the data and hypothesis levels, which experiments that cannot be replicated are clearly impor-
are insufficient to force theory building. Nevertheless, tant issues. However, the same is true for theoretical
such efforts might constitute the first step in under- accounts that cannot be instantiated or reinstantiated as
standing the logistics of multilab projects. On the other code. Should results of preregistered studies count as
hand, modelers often already currently work on a series stronger evidence than results of nonpreregistered stud-
of related experiments and publish them as a single ies? Should results of computationally modeled studies
experimental whole (Shiffrin, 2018). count as stronger evidence than those of studies with
The third possibility, more of the same, is the most only a statistical model? Just as the first question here
dire: “Maybe we should all simply continue playing our has been actively discussed (e.g., Szollosi et al., 2019),
collective game of 20 questions. Maybe all is well . . . the second should be as well.
and when we arrive in 1992 . . . we will have homed Thus, although it may superficially appear that we
in to the essential structure of the mind” (Newell, 1973, are at odds with the emphasis on the bottom few steps
p. 24). Thus, the future holds more time wasting and in our path model (hypothesis testing and data analysis;
crises. Some scientists will spend time attempting to recall Fig. 2) by those who are investigating replicabil-
replicate atheoretical hypotheses. However, asking ity, we are comfortable with this emphasis. We believe
nature 20 questions without a computational model the proposals set out by some to automate or streamline
leads to serious theoretical issues, even if the results the last few steps are part of the solution (e.g., Lakens
are superficially deemed replicable (e.g., Devezer et al., & DeBruine, 2021; Poldrack et al., 2019). Such a divi-
2019; Katz et al., 2018). sion of labor might help to maximize the quality of
theories and showcase the contrast—which Meehl
(1967) and others have drawn attention to—between
A Way Forward substantive theories and the hypotheses they generate.
Psychological science can change if we follow Figure We imagine a “best of all possible” massively collabora-
2 and radically update how we view the place of mod- tive future in which scientists allow machines to carry
eling. The first step is introspective: realizing that we out the least creative steps and thus set themselves free
all do some modeling—we subscribe to frameworks to focus exclusively on computational modeling, theory
and theories implicitly. Without formalizing our assump- generation, and explanation.
tions in the same way we explicitly state the variables
in hypothesis testing, we cannot communicate effi-
ciently. Some have even started to demand this shift in Transparency
our thinking (e.g., Morey et al., 2018; Oberauer & Action Editors: Travis Proulx and Richard Morey
Lewandowsky, 2019; Szollosi et al., 2019; Wills et al., Advisory Editor: Richard Lucas
2017). Editor: Laura A. King
The second step is pedagogical: explaining what Declaration of Conflicting Interests
The author(s) declared that there were no conflicts of
modeling is and why it is useful. We must teach men-
interest with respect to the authorship or the publication
tees that modeling is neither extremely complex nor of this article.
requires extra skills beyond those we already expect Funding
that they master, for example, programming, experi- O. Guest was supported by the Research Centre on Inter-
mental design, literature review, and statistical-analysis active Media, Smart Systems and Emerging Technologies
techniques (e.g., Epstein, 2008; Wills et al., 2017; Wilson (RISE) under the European Union’s Horizon 2020 pro-
& Collins, 2019). gramme (Grant 739578) and the Republic of Cyprus through
The third step is cooperative: working together as a the Directorate General for European Programmes, Coor-
field to center modeling in our scientific endeavors. dination and Development. A. E. Martin was supported by
Some believe the replication crisis is a measure of the the Max Planck Research Group “Language and Computa-
scientific quality of a subfield, and given that it has tion in Neural Systems” and by the Netherlands Organiza-
tion for Scientific Research (Grant 016.Vidi.188.029).
affected areas of psychological science with less formal
modeling, one possibility might be to ask these areas
ORCID iD
to model explicitly. By extension, modelers can begin
to publish more in these areas (e.g., in consumer Olivia Guest https://orcid.org/0000-0002-1891-0972
behavior; see Hornsby et al., 2019).
To ensure experimental results can be replicated and Acknowledgments
reobserved, we must force theory building; replicability We thank Abeba Birhane, Sebastian Bobadilla Suarez,
in part depends causally on things higher up the path Christopher D. Chambers, Eiko Fried, Dermot Lynott, Esther
(see also Oberauer & Lewandowsky, 2019). Data and Mondragón, Richard D. Morey, Karim N’Diaye, Nicolas P.800 Guest, Martin
Rougier, Richard M. Shiffrin, Loukia Tzavella, and Andy J. Fermat’s Library [@fermatslibrary]. (2019, January 7). Here’s
Wills for useful discussions and input on previous versions a useful counterintuitive fact: One 18 inch pizza has
of the article. more ‘pizza’ than two 12 inch pizzas [Image attached]
[Tweet]. Twitter. https://twitter.com/fermatslibrary/sta
tus/1082273172114862083
References Feyerabend, P. K. (1957). An attempt at a realistic interpreta-
Anderson, J., & Lebiere, C. (1998). The atomic components tion of experience. Proceedings of the Aristotelian Society,
of thought. Erlbaum. 58, 143–170. https://www.jstor.org/stable/4544593
Baddeley, A. (2010). Working memory. Current Biology, Flack, J. C. (2012). Multiple time-scales and the developmen-
20(4), R136–R140. tal dynamics of social systems. Philosophical Transactions
Blokpoel, M. (2018). Sculpting computational-level models. of the Royal Society B: Biological Sciences, 367(1597),
Topics in Cognitive Science, 10(3), 641–648. 1802–1810.
Borsboom, D. (2013). Theoretical amnesia. Open Science Flis, I. (2019). Psychologists psychologizing scientific psychol-
Collaboration. http://osc.centerforopenscience.org/ ogy: An epistemological reading of the replication crisis.
2013/11/20/theoretical-amnesia Theory & Psychology, 29(2), 158–181.
Botvinik-Nezer, R., Holzmeister, F., Camerer, C. F., Dreber, Forscher, B. K. (1963). Chaos in the brickyard. Science,
A., Huber, J., Johannesson, M., Kirchler, M., Iwanir, R., 142(3590), 339.
Mumford, J. A., Adcock, R. A., Avesani, P., Baczkowski, B. M., Fried, E. I. (2020). Lack of theory building and testing impedes
Bajracharya, A., Bakst, L., Ball, S., Barilari, M., Bault, progress in the factor and network literature. PsyArXiv.
N., Beaton, D., Beitner, J., . . . Schonberg, T. (2020). https://psyarxiv.com/zg84s
Variability in the analysis of a single neuroimaging data Guest, O., Caso, A., & Cooper, R. P. (2020). On simulating
set by many teams. Nature, 582(7810), 84–88. neural damage in connectionist networks. Computational
Box, G. E. (1976). Science and statistics. Journal of the Brain & Behavior, 3, 289–32. https://doi.org/10.1007/
American Statistical Association, 71(356), 791–799. s42113-020-00081-z
Cohen, J. (1954). On the project of a universal character. Guest, O., & Love, B. C. (2017). What the success of brain
Mind, 63(249), 49–63. imaging implies about the neural code. eLife, 6, Article
Cooper, R. P. (2007). The role of falsification in the develop- e21397. https://doi.org/10.7554/eLife.21397
ment of cognitive architectures: Insights from a Lakatosian Haig, B. D. (2018). An abductive theory of scientific method.
analysis. Cognitive Science, 31(3), 509–533. In Method Matters in Psychology. Studies: Essays in Applied
Cooper, R. P., Fox, J., Farringdon, J., & Shallice, T. (1996). A Philosophy of Science (pp. 35–64). Springer. https://doi
systematic methodology for cognitive modelling. Artificial .org/10.1007/978-3-030-01051-5_3
Intelligence, 85(1–2), 3–44. Haslbeck, J., Ryan, O., Robinaugh, D., Waldorp, L., &
Cooper, R. P., & Guest, O. (2014). Implementations are not Borsboom, D. (2019). Modeling psychopathology: From
specifications: Specification, replication and experimen- data models to formal theories. PsyArXiv. https://doi
tation in computational cognitive modeling. Cognitive .org/10.31234/osf.io/jgm7f
Systems Research, 27, 42–49. Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions,
Cox, G. E., & Shiffrin, R. M. (in press). Computational models of M. D. (2015). The extent and consequences of p-hacking
event memory. In M. J. Kahana & A. Wagner (Eds.), Oxford in science. PLOS Biology, 13(3), Article e1002106. https://
handbook of human memory. Oxford University Press. doi.org/10.1371/journal.pbio.1002106
Davidson, D., & Martin, A. E. (2013). Modeling accuracy as Hornsby, A. N., Evans, T., Riefer, P. S., Prior, R., & Love, B. C.
a function of response time with the generalized linear (2019). Conceptual organization is revealed by consumer
mixed effects model. Acta Psychologica, 144(1), 83–96. activity patterns. Computational Brain & Behavior, 3,
Dayan, P., & Berridge, K. C. (2014). Model-based and 162–173. https://doi.org/10.1007/s42113-019-00064-9
model-free pavlovian reward learning: Revaluation, revi- Hunt, E., & Luce, R. D. (1992). Soar as a world view, not
sion, and revelation. Cognitive, Affective, & Behavioral a theory. Behavioral and Brain Sciences, 15(3), 447–
Neuroscience, 14(2), 473–492. 448.
DeDeo, S. (2018). Origin gaps and the eternal sunshine of Hyde, J. S. (2005). The gender similarities hypothesis.
the second-order pendulum. In A. Aguirre, B. Foster, & Z. American Psychologist, 60(6), 581–592.
Merali (Eds.) Wandering towards a goal: How can mind- John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring
less mathematical laws give rise to aims and intention? the prevalence of questionable research practices with
(pp. 41–61). Springer. incentives for truth telling. Psychological Science, 23(5),
Devezer, B., Nardin, L. G., Baumgaertner, B., & Buzbas, E. O. 524–532.
(2019). Scientific discovery in a model-centric framework: Jones, M., & Love, B. C. (2011). Bayesian fundamental-
Reproducibility, innovation, and epistemic diversity. PLOS ism or enlightenment? On the explanatory status and
ONE, 14(5), Article e0216125. https://doi.org/10.1371/ theoretical contributions of bayesian models of cogni-
journal.pone.0216125 tion [Target article and commentaries]. Behavioral and
Epstein, J. M. (2008). Why model? Journal of Artificial Societies Brain Sciences, 34(4), 169–231. https://doi.org/10.1017/
and Social Simulation, 11(4), Article 12. http://jasss.soc S0140525X10003134
.surrey.ac.uk/11/4/12.html Kahneman, D. (2011). Thinking, fast and slow. Macmillan.You can also read
