The neural career of sensory-motor metaphors
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
The neural career of sensory-motor metaphors
Rutvik H. Desai*, Jeffrey R. Binder*, Lisa L. Conant*, Quintino R. Mano†, Mark S.
Seidenberg‡
*Medical College of Wisconsin, †University of Wisconsin-Milwaukee, ‡University of Wisconsin-Madison
Abstract
The role of sensory-motor systems in support the view that the understanding of metaphoric
conceptual understanding has been controversial. It has action retains a link to sensory-motor systems involved
been proposed than many abstract concepts are in action performance. However, the involvement of
understood metaphorically through concrete sensory- sensory-motor systems in metaphor understanding
motor domains such as actions. Using fMRI, we changes through a gradual abstraction process whereby
compared neural responses to literal action (Lit; The relatively detailed simulations are used for understanding
daughter grasped the flowers), metaphoric action (Met; The unfamiliar metaphors, and these simulations become
public grasped the idea), and abstract (Abs; The public less detailed and involve only secondary motor regions
understood the idea) sentences of varying familiarity. Both as familiarity increases. Consistent with these data, we
Lit and Met sentences activated the left anterior inferior propose that aIPL serves as an interface between
partial lobule (aIPL), an area involved in action sensory-motor and conceptual systems and plays an
planning, with Met sentences also activating a important role in both domains. The similarity of
homologous area in the right hemisphere, relative to abstract and metaphoric sentences in the activation of
Abs sentences. Both Met and Abs sentences activated left superior temporal regions suggests that action
left superior temporal regions associated with abstract metaphor understanding is not completely based on
language. Importantly, activation in primary motor and sensory-motor simulations, but relies also on abstract
biological motion perception regions was inversely lexical-semantic codes.
correlated with Lit and Met familiarity. These results
concepts, provide an interesting opportunity to study
INTRODUCTION
the relationship between these two systems.
The relationship between sensory-motor and Engagement of sensory-motor systems even when
conceptual systems of the brain has been the focus of action language is clearly figurative would suggest a
intense debate in recent years (Pulvermuller, 2005; particularly close relationship between these systems,
Barsalou, 2008; Mahon and Caramazza, 2008). consistent with theories that many abstract concepts are
Neuroimaging, behavioral, and patient studies suggest a understood through analogies to sensation and action
closer link between these systems than previously (Lakoff and Johnson, 1980, 1999; Gibbs, 2006; Bergen,
recognized (Aziz-Zadeh and Damasio, 2008; Fischer 2007).
and Zwaan, 2008; Kemmerer, in press). The precise The few imaging studies on figurative action
nature of the relationship between these systems, language have yielded somewhat inconsistent results.
however, remains unclear. Weak embodiment views Aziz-Zadeh et al. (2006) found somatotopic activation
suggest engagement of sensory-motor systems only in the premotor cortex for literal action sentences, but
when concepts are transparently related to physical not for idiomatic phrases (“biting off more than you can
action. In contrast, strong embodiment assigns sensory- chew”). Boulenger et al. (2009) found somatotopic
motor systems a pervasive role in comprehension, activation for figurative and literal action sentences
including more abstract concepts. involving leg and arm verbs. Raposo et al. (2009) found
Action metaphors (e.g., grab the chance or grasp an idea), activation in premotor/motor regions for isolated action
which convey abstract concepts via analogy to concrete verbs, and to a lesser extent for literal action sentences,
Journal of Cognitive Neuroscience In press 1
but not for figurative sentences using action verbs. Institutional Review Board. Participants were paid for
Three studies have also shown activation in or near participation.
motion processing area MT+ for literal as well as
Stimuli
figurative or fictive motion sentences (“The man fell
under her spell”; “The bridge jumped over the brook”) Stimuli were sentences divided into three main
compared to non-motive sentences (Wallentin et al., conditions: Literal action (Lit), Metaphoric action (Met),
2005; Chen et al., 2008; Saygin et al., 2010). and Abstract (Abs). The stimuli were constructed in
To elucidate the relationship between sensory-motor triples consisting of one sentence from each condition,
and conceptual systems, and to adjudicate between weak with the same syntactic form (examples in Table 1;
and strong views of embodiment, we compared BOLD complete listing provided in the Supplemental Material).
responses to metaphoric action sentences with two Table 1. Example stimuli.
types of non-metaphoric sentences – literal action and Literal The daughter grasped the The thief bashed the table
abstract. We varied the familiarity of each sentence type flowers
to investigate the modulation of activity in sensory- Metaphor The jury grasped the concept The council bashed the
motor regions. In many studies of metaphors, proposal
Abstract The jury understood the The council criticized the
processing difficulty has been a common confounding concept proposal
variable, in that metaphoric stimuli tend to be more
difficult to process (as indexed by response latencies) The Lit sentence used a hand/arm action verb to
than literal control stimuli. Additional activation depict a physical action. The corresponding Met
resulting from processing more difficult stimuli can be sentence used the same verb in a figurative manner,
mistaken as activation specific to metaphors (Schmidt such that no physical action was described. The Abs
and Seger, 2009; Yang et al., 2009). We carefully sentence used an abstract verb to convey a meaning
controlled for processing difficulty and other similar to that of the Met sentence. The agent in each
confounding variables such as syntactic structure and sentence was chosen to imply either a literal or
sentence length, which can also lead to activation in abstract/metaphoric interpretation of the verb. Met and
sensory-motor regions. We hypothesized that relatively Abs sentences always used the same agent. This agent
unfamiliar (literal and metaphoric) action language was an entity that makes literal physical actions unlikely
engages sensory-motor systems because comprehension (e.g., the scandal, the crime, the government). The Lit
of such expressions involves relatively detailed sentences, in contrast, always used a person (the teacher,
simulations of literal actions. As the expression becomes the doctor) as an agent. For example, consider the
more familiar and conventionalized, the reliance on fragments “The captain lifted…” and “The government
sensory-motor simulation diminishes. Alternative lifted…”. In the first example, when lifted is
hypotheses are that sensory-motor systems are not encountered, both a physical action as well as figurative
engaged at all for metaphoric expressions, or are lifting (e.g., of a ban) are possible. In the second
engaged regardless of familiarity. example, it is clear the physical action interpretation is
infelicitous.
METHODS Twenty-seven action verbs were used three times
Participants each to create 81 Lit and Met sentences. Additionally,
there were 81 Abs sentences, 81 Nonsense sentences
Participants in the fMRI experiment were 22 healthy
(e.g., The wedding strummed the introduction; created by
adults (11 women; average age 24 years, range 18-33;
combining action and abstract verbs with inappropriate
average years of education 16, range 12-23), with no
nouns), and 81 nonword sentences of the same form.
history of neurological impairment. One additional
Nonword sentences were composed of pronounceable
participant was removed due to activations in the three
nonwords created using the MCWord database
main contrasts that were more than two s.d. away from
(http://www.neuro.mcw.edu/mcword/) and the ARC nonword
the group mean. Participants were native speakers of
database (Rastle et al., 2002). Finally, there were 54 Filler
English, and were right-handed according to the
sentences that used variable syntax (e.g., All lawyers went
Edinburgh Handedness Inventory (Oldfield, 1971).
on strike). These were used to obscure the triplet
Informed consent was obtained from each participant
construction of the stimuli and to provide syntactic
prior to the experiment, in accordance with a protocol
variability in the stimulus set.
sanctioned by the Medical College of Wisconsin
Journal of Cognitive Neuroscience In press 2Stimulus Norming stimuli was then tested in the Meaningfulness Judgment
One of our principal goals was to equate the three experiment below, to verify that the differences between
main conditions with respect to processing difficulty, to conditions in response time (RT) were indeed
remove the possible confound between figurativeness minimized.
and difficulty. Numerous factors can affect the difficulty Meaningfulness Judgment
of processing sentences. In addition to word frequency,
Participants in the meaningfulness judgment
the frequency of the particular verb-noun combination
experiment were 24 adults (20 women; average age 19
(e.g., “grasp the idea” vs. “grasp the procedure”), and
years, range 18-21; average years of education 12, range
the frequency and familiarity of the verb in a metaphoric
12-14). They were native speakers of English and did
vs. literal sense can affect automaticity of
not participate in the fMRI experiment.
comprehension. As described below, stimuli were pre-
Participants made a meaningfulness judgment
tested using a meaningfulness judgment task, which
(meaningful/not meaningful) for each sentence,
reflects the combined effects of such factors.
presented in two parts as in the preliminary experiment
A two-step procedure was used in developing the
above. Participants were asked to judge the
stimuli. First, a large set of sentences was prepared by
meaningfulness of the sentence as a whole by pressing
combining action and abstract verbs with appropriate
one of two buttons.
nouns to create Lit, Met, and Abs sentences, for use in a
As shown in Table 2, there were no significant
preliminary experiment. Six participants made a
differences between Met, Lit, and Abs conditions, while
meaningfulness judgment (“makes sense” or “does not
the Nonsense condition had longer RTs than all of the
make sense”) for each sentence by pressing one of two
other conditions (all p < 0.0001).
buttons on a response pad. The sentences were
presented in two parts, as shown in Fig. 1. Familiarity Rating
The familiarity of sentences can affect speed and
accuracy of processing. Although familiarity differences
would be expected to affect latencies in the
meaningfulness judgment task mentioned above, we
also collected familiarity ratings to enable a more direct
assessment of familiarity effects in the imaging analysis.
Participants in the familiarity rating experiment were
28 adults (16 women; average age 19 years, range 18-21;
average years of education 13, range 12-16). They were
native speakers of English and did not participate in the
Figure 1. The presentation of the stimuli. The first noun phrase of
fMRI experiment.
the sentence was displayed for 500 ms, followed by the remaining
sentence for 1300 ms. The sentences were separated by variable For each sentence, participants rated each sentence
intervals. on a scale of 1 (not at all familiar) to 7 (very familiar).
The results are shown in Table 2. Lit and Abs stimuli
The first screen displayed a noun phrase (e.g., “The differed in familiarity (p < 0.02), with no other reliable
public”) for 500 ms. This was replaced by the verb differences between types.
phrase (e.g., “grasped the idea”) on the second screen, As expected, the Meaningfulness Judgment RTs and
displayed for 1300 ms. This two-part presentation was the familiarity ratings were correlated (Pearson’s r = -
used to ensure that the first noun phrase, which suggests 0.52, p < 0.0001).
the literal or metaphoric/abstract interpretation of the
verb, was read first. Consistent with previous studies, Lit Table 2. The mean (s.d.) RTs in the Meaningfulness Judgement task,
sentences yielded faster responses than Met and Abs and the Familiarity Ratings (on a scale of 1 to 7) for various
sentences. Sentences were then modified to reduce these conditions.
differences. (e.g., by using a more familiar noun-verb Condition n RT Familiarity
combination to reduce the difficulty of a metaphoric Metaphor 81 1277 (165) 5.24 (0.77)
sentence, or by using a less familiar noun-verb Literal 81 1241 (151) 4.97 (1.04)
combination to replace literal sentences that were very Abstract 81 1265 (179) 5.37 (0.87)
easy). In modifying the sentences, we also reduced Nonsense 81 1399 (167) -
differences in word frequencies and in the number of Pseudoword 81 723 (138) -
phonemes, letters, and syllables. The revised set of Filler 54 1234 (232) -
Journal of Cognitive Neuroscience In press 3Action Association Rating and fifth digit – with their left hand or right hand, or
We intentionally selected verbs that were clearly rested. Prior to scanning, these actions were
associated with actions in the Met/Lit conditions, demonstrated by the experimenter outside the scanner,
whereas verbs used in the Abs condition were relatively without using verbal labels, and the subjects were asked
abstract. It is possible, however, that some of these to repeat these actions, in the same sequence, for
“abstract” verbs also have some association with practice and verification of accuracy. In the scanner, the
actions. Action association ratings were collected to instructions “left,” “right,” or “rest” were displayed at
assess whether abstract verbs were in fact less associated the beginning of each block. Each block was 18 seconds
with actions than Met/Lit verbs. Participants in this long, and three blocks of each condition were presented
rating experiment were 14 adults (7 women; average age in a pseudo-randomized order.
20 years, range 18-22; average years of education 13, Image Acquisition and Analysis
range 12-15), native speakers of English who did not
A 3T GE Excite scanner was used to acquire images.
participate in the fMRI experiment. Participants rated
One volume of T2*-weighted, gradient echo, echo-
each verb, preceded by to (e.g., to understand) on a scale of
planar images (TE = 25 ms, flip angle = 77°, NEX = 1)
1 (not associated with action at all) to 7 (very much
was acquired every 1.8 s. Visual sentence presentation
associated with action). The mean (s.d.) for Met/Lit
was time-locked with the beginning of an acquisition.
verbs was 6.17 (0.42) and for Abs verbs was 3.62 (0.70),
Volumes were composed of 30 axially-oriented 3.5 mm
a difference that was highly significant (p < 0.0001).
slices with a 0.5 mm interslice gap, covering the whole
FMRI Tasks brain, with FOV = 240 mm and 64 × 64 matrix,
The sentences in the imaging experiment were resulting in 3.75 × 3.75 × 4 mm voxel dimensions.
presented in two parts, as in the Meaningfulness Anatomical images of the entire brain were obtained
judgment experiment (Fig 1). Participants were using a 3D spoiled gradient echo sequence (SPGR) with
instructed to read each sentence and make a covert 0.94 × 0.94 × 1 mm voxel dimensions.
meaningfulness decision. A covert task was used to The AFNI software package (Cox, 1996) was used
prevent strong activation of the motor cortex by a for image analysis. Within-subject analysis involved
manual or vocal response. The order of sentences was spatial co-registration (Cox and Jesmanowicz, 1999) and
pseudo-randomized, and the interval between the registration of functional images to the anatomy (Saad et
sentences was varied, to allow optimal statistical al., 2009). Runs were removed from the analysis if d’
separation of the hemodynamic response to each performance on the recognition test after a run was less
condition. The sentences were divided into 9 runs then 1, or if the self-reported attentiveness rating was 5
lasting approximately 5 minutes each. To encourage or less; 11 runs (5.6%) were removed in this manner.
attentiveness, participants were also tested on a Voxel-wise multiple linear regression was performed
recognition task after each run. Fourteen sentences were with reference functions representing each condition.
shown, and for each sentence, participants indicated by Additionally, familiarity ratings for each stimulus (see
pressing one of two buttons whether they had seen the Stimulus Norming, above) were used to create within-
sentence in the preceding run. On average, half of the condition familiarity regressors for the Lit, Met, and Abs
14 sentences were taken from the previous run, whereas conditions. Mean-centered regressors for the number of
the others were not in the experiment. At the end of syllables and phonemes in each sentence were used as
each run, participants were also asked to rate their additional item-wise regressors to account for
attentiveness during the task on a scale of 1 (not differences due to these variables. A standard
attentive at all) to 10 (very attentive). Instructions and hemodynamic response function convolved with the
practice with all tasks were provided outside the scanner reference functions, and its temporal derivative, were
before the scan, and the participants were informed that used. A correction for amplitude bias was applied using
the recognition test would be administered after each the method described by Calhoun et al. (2004). Six
run. motion parameters and the signal extracted from the
ventricles, segmented using the FSL fast program
Motor Localizer Task (Zhang et al., 2001), were included as noise covariates of
After the sentence runs, a localizer task was used to no interest. General linear tests were conducted to
locate hand motor regions of the brain, using a block obtain the Lit-Abs, Met-Lit, and Met-Abs contrasts and
design. Participants performed a repeating sequence of Familiarity × Condition interactions.
actions – make a fist, turn the palm up, touch thumb
Journal of Cognitive Neuroscience In press 4The individual statistical maps and the anatomical was restricted to a mask that excluded areas outside the
scans were projected into standard stereotaxic space brain, as well as deep white matter areas and the
(Talairach and Tournoux, 1988) and smoothed with a ventricles. The data from the motor localizer scan were
Gaussian filter of 5 mm FWHM. In a random effects analyzed as a block design in a similar way. To further
analysis, group maps were created by comparing examine motor areas, two regions of interest (ROIs)
activations against a constant value of 0. The group were defined. One used the area activated by the motor
maps were thresholded at voxelwise p < 0.01 and localizer task as an ROI, and the other used primary
corrected for multiple comparisons by removing clusters motor and sensory cortex (M1 and S1) as defined by the
smaller than 1000 µl to achieve a mapwise corrected HMAT atlas (Mayka et al., 2006). Small volume
two-tailed p < 0.05. The cluster threshold was correction was applied in these ROIs to achieve
determined through Monte Carlo simulations that corrected p < 0.05.
estimate the chance probability of spatially contiguous
voxels exceeding the voxelwise p threshold. The analysis
Figure 2. Areas activated by condition contrasts. Yellow-orange scale shows greater activation for the first condition; blue-
cyan scale shows greater activation for the second condition in the contrast. L = left hemisphere, R = right hemisphere.
Journal of Cognitive Neuroscience In press 5RESULTS computed Familiarity × Condition interactions. The
regions responding only to general difficulty and task
The mean (s.d.) d’ in the post-run test was 2.56
load effects should be modulated similarly in all three
(0.74), suggesting that the participants were attentive to
conditions and therefore would not show interactions.
the stimuli during the scans. We first describe the fMRI
Indeed, middle and inferior frontal lobe regions,
results for the contrasts between the three main
commonly associated with task difficulty effects, were
conditions, Lit, Met and Abs. The results are displayed
negatively correlated with familiarity in each condition
on an inflated brain surface using Caret (Van Essen et
(see Supplemental Material, Part II) and were absent
al., 2001). A complete listing of the activated areas with
from the interaction maps. In addition, because our
coordinates is provided in the Supplemental Material
hypotheses concern correlations with Lit or Met
(Part I).
familiarity, we applied a mask to the interaction maps
Literal - Abstract that included only voxels that showed a significant
The areas activated to a greater extent by the Lit correlation with familiarity in Lit or Met conditions.
condition relative to the Abs condition included the left Literal-Abs × Familiarity
anterior inferior parietal lobule (aIPL; including
supramarginal gyrus and postcentral sulcus), left All the regions in this interaction showed negative
parahippocampal and fusiform gyrus, left precuneus, left correlation with familiarity for Lit sentences, and greater
posterior middle and inferior temporal gyrus and lateral negative correlation for Lit than for Abs sentences (cyan
occipital gyrus, left superior frontal gyrus, left and green in Fig 3a). They included the left anterior
orbitofrontal cortex, bilateral cerebellum and thalamus, inferior frontal gyrus, central sulcus, superior parietal
as well as the right hippocampus and fusiform gyrus gyrus, and posterior superior temporal sulcus; the right
(Fig 2a). aIPL; and bilateral posterior middle and inferior
Abs sentences activated the left superior temporal temporal gyrus, parietal operculum, and SMA.
sulcus and the anterior superior temporal gyrus, cuneus, Metaphor-Abstract × Familiarity
as well as the right angular gyrus. Similar to the previous interaction, all areas here
Metaphor - Abstract showed negative correlations for Met sentences, and
The Met condition activated left aIPL, Rolandic greater negative correlations for Met than for Abs
operculum, superior parietal gyrus, and the superior sentences. These areas included the left posterior
frontal gyrus. The cerebellum and thalamus were superior temporal sulcus and bilateral central sulcus,
activated bilaterally. The right aIPL, parietal operculum, SMA, lingual gyrus, and cuneus (cyan and green in Fig
insula, and parahippocampal gyrus were also activated 3b).
(Fig 2b). No areas were activated to a greater extent for Metaphor-Literal × Familiarity
the Abs sentences. The right supramarginal gyrus and the left superior
Metaphor - Literal parietal gyrus showed a significant interaction. The right
Compared to Lit sentences, the Met sentences supramarginal gyrus showed a positive correlation with
activated the left anterior and posterior cingulate gyrus, familiarity for Met, while the left superior parietal gyrus
cuneus, superior temporal sulcus, and the temporal pole. was negatively correlated in the Lit condition (cyan and
In the right hemisphere, the aIPL, parietal operculum green in Fig 3c).
and the superior parietal gyrus were activated (Fig 2c). Identical analyses were also carried out using RTs,
The Lit condition activated the left parahippocampal rather than familiarity ratings, as regressors. No
and fusiform gyrus relative to the Met sentences. Condition × RT interactions in sensory-motor regions
were found after identical application of corrections for
Correlations with Familiarity multiple comparisons. This suggests that these
To assess the effects of sentence familiarity on the interactions are unlikely to be due to differences only in
activation, we used the Familiarity Rating as a condition- the length of action simulations or some form of pre-
wise regressor in the analysis. Correlations with response motor readiness (although the latter is
familiarity can pinpoint areas involved in the semantic unexpected in any case because no responses were made
processing, but also areas modulated due to general during scanning).
processing difficulty. To isolate, to the extent possible,
activation modulation due to semantic factors, we
Journal of Cognitive Neuroscience In press 6Figure 3. The overlap of condition contrasts, condition x familiarity interactions, and the motor localizer. Interactions in (a) and (b) show
greater negative correlation to familiarity for Lit and Met conditions respectively. Talairach y coordinates are indicated in the upper left corner of
each slice. Stereotaxic x and z axis are shown in white.
Journal of Cognitive Neuroscience In press 7sentences, and whether this engagement changes with
Overlap with Localizer
sentence familiarity.
Figure 3 also shows the overlap of the areas
activated by the hand localizer task, the condition Literal Sentences
contrasts, and the Condition × Familiarity interactions. The Lit > Abs contrast produced results similar to a
Activations in the central sulcus, aIPL, posterior previous study (Desai et al., 2009). The aIPL region,
superior temporal sulcus, posterior middle and inferior overlapping the localizer activation, is a secondary
temporal gyrus, opercular regions, SMA, thalamus, and sensory-motor area associated with action planning and
cerebellum overlapped the localizer activation (magenta, complex hand-object interaction, as suggested by both
cyan and white colors). In contrast, activations in the imaging and lesion studies. It is structurally connected to
dorsomedial prefrontal region, parahippocampal and posterior middle temporal and inferior frontal gyri,
fusiform gyrus, middle superior temporal sulcus, and forming a tool-use network (Ramayya et al., 2009), and
posterior cingulate did not overlap the localizer (red and is strongly linked to action performance, imitation, and
green colors), and occipital regions partly overlapped. semantics (Haaland et al., 2000; Tranel et al., 2003;
The left posterior middle/inferior temporal region was Glover, 2004; Buxbaum et al., 2005; Binder et al., 2009;
the only area to show an overlap between the condition for further discussion, see Desai et al., 2009). Peeters et
contrasts and the familiarity interactions (white color). al. (2009) compared human and trained as well as
A summary of the main results is presented in Table untrained monkeys as they observed actions performed
3, where areas commonly and differentially activated using simple tools, and found that aIPL was uniquely
between the contrasts and familiarity interactions can be activated in humans. They proposed that this region has
seen. evolved only in humans to subserve complex actions.
The present results suggest that this region may be
Table 3. A summary of the main results. ‘×’ indicates regions that unique to humans partly because it serves as an interface
are activated or correlated with familiarity, or overlap with the between language and action, playing a role in both
localizer activation.
domains.
Contrast
Parahippocampal gyrus and surrounding cortex is
Lit > Met > Lit-Abs Met-Abs Loc. most closely associated with episodic and spatial
Area
Abs Abs x Fam x Fam Overlap memory (Squire et al., 2004). Several studies report
activation in this region for processing actions, tools, or
L dorsomedial prefrontal x concrete objects, e.g., for simulating rehearsed actions
L parahippocampal/ (Cross et al., 2006), recognizing and naming actions
x
fusiform g
(Decety et al., 1997; Tranel et al., 2005), for artifacts
L aIPL x x x
compared to living things (Martin, 2007), and for
Bilateral thalamus x x x
concrete compared to abstract words (Binder, 2007).
Bilateral cerebellum x x x
These findings suggest that activation of this region for
L p mid temporal g x x x Lit sentences reflects the retrieval of contextual and
Bilateral parietal operc. x x x spatial information related to actions and the concrete
R aIPL x x x nouns in these sentences.
L sup parietal lobule x x x The posterior middle and inferior temporal gyri are
R p mid temporal g x x associated with linguistic knowledge about tools and
L central s x x x actions (Martin, 2007; Binder et al., 2009). This
L p sup temporal s x x x activation, overlapping with the localizer, was
Bilateral SMA x x x immediately anterior to the visual motion processing
R central s x x area MT/MST, suggesting a role in more abstract
Bilateral Lingual g x x motion processing (Kable et al., 2005; Chen et al., 2008).
The cerebellar and precuneus activation overlapped
DISCUSSION the localizer, whereas the dorsomedial prefrontal
We presented participants with literal action, activation did not. This latter region is frequently
metaphoric action, and abstract sentences to examine activated during processing of concrete semantic
the engagement of sensory-motor areas during their concepts, and is thought to play a role in their retrieval
comprehension. We asked whether sensory-motor areas (Binder et al., 2009).
are engaged even when processing metaphoric action
Journal of Cognitive Neuroscience In press 8Thus, the pattern of activation for Lit sentences meaning should result in a signal that is weaker than the
suggests a role for sensory-motor systems in their signal from sustained activation of the literal meaning
comprehension. This activation is consistent with the for Lit sentence, integrated over the sentence, whereas
view that understanding such sentences involves action here the two signals were of similar magnitude.
simulation, but this simulation is at a relatively abstract Futhermore, the initial noun phrases of the sentences
level, engaging higher level action planning and motion were selected to prime an abstract or concrete meaning
perception areas. of the verb, and this property promotes rapid
suppression of incongruous meanings.
Metaphoric Sentences
Compared to both Lit and Abs conditions, the right
Although the meaning conveyed by metaphoric aIPL was also activated for metaphors. The RH
sentences is abstract, an analogy with a concrete domain activation can be interpreted in terms of Beeman’s
is used to convey the meaning. Some theorists have (1994) fine-coarse coding theory. It suggests that the
suggested that mental simulation is used to understand RH maintains a wider “semantic field,” containing
such action metaphors (Gibbs, 2006; Bergen, 2007), and alternative meanings or distantly related features, while
thus the metaphoric meaning is “grounded” in the literal the LH processes the dominant meanings or features.
meaning. The RH activation may therefore represent access to a
The Met > Abs contrast activated regions associated wider variety of meanings to subserve metaphor
with sensory-motor processing (the aIPL and bilateral interpretation.
cerebellum), which were also activated by the Lit The posterior cingulate activation in the Met > Lit
sentences and localizer task. Notably, Met and Lit contrast is found in many semantic studies (Binder et al.,
sentences activated these regions to a similar extent and 2009). This region has been identified as a connectivity
hence were absent from the Met-Lit comparison. The hub (Sporns et al., 2007; Buckner et al., 2009). Hubs
superior parietal lobule was activated additionally for contain disproportionally numerous connections and are
metaphors, which is associated with control of action hypothesized to integrate diverse informational sources.
and computation of dynamic spatial information The involvement of posterior cingulate in metaphor
(Glover, 2004). If the activation of the left aIPL and comprehension may be related to integrating
bilateral cerebellum is taken as an index of sensory- information from target and base domains.
motor processing during sentence comprehension, this Met sentences also activated the left middle superior
suggests that the understanding of sensory-motor temporal sulcus, similar to the Abs sentences. This
metaphors is not abstracted away from their sensory- activation, which did not overlap with the localizer,
motor origins. As with literal action sentences, a could reflect the computation of abstract meaning
(relatively abstract) motoric simulation is used in conveyed by the Met sentences. This similarity between
comprehension of action metaphors. Grasping an idea is Abs and Met conditions suggests that sensory-motor
understood much like grasping a handle is, using meanings metaphors are not represented entirely in a sensory-
that are based on sensory-motor representations. motor format. Although motoric simulations may be
An alternative interpretation of these activations is used to understand such metaphors, an abstract
that verbs such as grasp are homonyms with two component is also present.
independent meanings, one related to the physical
action and one meaning “to understand.” Both Abstract Sentences
meanings are initially activated during sentence The activation of the left middle and anterior
processing, and the incongruent meaning is later superior temporal sulcus for Abs sentences is consistent
suppressed. In this view, the activation of sensory- with a number of studies comparing abstract to concrete
motor areas during Met simply represents the activation stimuli (for a review, see Binder, 2007). According to
of the incongruous literal meaning, and the metaphoric Pavio’s (1986) dual coding theory, abstract information
meaning is abstract and unrelated to sensory-motor is represented mainly through verbal associations with
systems. However, a number of behavioral studies other words. Andrews et al. (2009) distinguish between
suggest that when processing homonymous or distributional and experiential statistics upon which
polysemous words in sentences, incongruous meanings representations are built. With limited experiential
are either not activated at all, or are suppressed within (sensory-motor) features, abstract concepts may rely
approximately 250-300 ms (Onifer and Swinney, 1981; heavily on distributional information (i.e., statistical
Seidenberg et al., 1982; Pynte et al., 1996; Glucksberg, information about word co-occurrences, also captured
2001). The initial brief activation of an unrelated by computational models such as Latent Semantic
Journal of Cognitive Neuroscience In press 9Analysis; Landauer and Dumais, 1997). For example, a meanings of the base concept are activated, and the
representation of justice may be built gradually through appropriate meaning is assigned to the target. Our
associations with concepts such as fair, law, good, court, results suggest a different picture, at least for sensory-
right, truth, etc. The left-dominant temporal activation is motor metaphors. The target is understood in terms of
also consistent with this view, given the role of this area the base domain through motoric simulations, which
in lexical phonology. Further support for this view gradually become less detailed while still maintaining
comes from aphasic patients with stroke in the left their roots in the base domain. The negative correlation
perisylvian region, who generally show greater of primary motor areas with metaphor familiarity, and
processing deficits for abstract words (Goodglass et al., the activation of secondary motor regions for
1969; Katz and Goodglass, 1990; Franklin et al., 1995). metaphors regardless of familiarity, suggest a gradual
abstraction rather than a switch in the processing mode.
Correlations with Familiarity
The right supramarginal gyrus was correlated
Activation in a number of sensory-motor regions, positively with Met familiarity. For more familiar
overlapping with the localizer activation, were negatively metaphors, this area may play a role in using the wider
correlated with familiarity for both Lit and Met RH semantic field to efficiently combine words in
sentences, even after accounting for the increased phrases such as grasp an idea. For less familiar metaphors,
general difficulty of processing less familiar sentences. such automatic combination is not possible, and more
Notably, SMA and primary motor areas in the central on-line simulation is necessary. This view is supported
sulcus – in the LH for Lit and bilaterally for Met – were by a study in our lab in which this area was activated for
correlated with familiarity. The left posterior superior meaningful two-word phrases such as flower girl relative
temporal sulcus was also correlated with both Lit and to difficult-to-interpret combinations such as girl flower
Met familiarity, and is implicated in biological motion (Graves et al., in press).
(Grossman and Blake, 2002; Saygin, 2007). This While our aim was to investigate the effects of
suggests that to understand less familiar action-related variations in familiarity, other factors, such as amount of
language, a relatively detailed simulation is used that personal experience with specific actions (Lyons et al.,
relies on primary motor areas, and this is true even for 2010), amount of physical effort (Moody and Gennari,
metaphoric language in which no literal action is 2010) and force (Frak et al., 2010) required for an
implied. As familiarity increases, the abstractness of action, and semantic context (van Dam et al., in press)
simulation also increases, involving only the secondary may also modulate sensory-motor areas. The effects of
action-planning region (aIPL) that was activated these variables, and their interactions with familiarity,
regardless of familiarity. Moreover, it is possible that at await further research.
the highest end on the familiarity scale that was not
examined here, as in the case of idioms or the pervasive CONCLUSIONS
“time is space” metaphors, sensory-motor systems are
engaged to a lesser extent or not at all (Kemmerer, A comparison of literal action, metaphoric action,
2005). and abstract sentences revealed activation of secondary
Notably, several areas that were activated consistently sensory-motor areas including the left aIPL, involved in
for Met – bilateral parietal operculum, left superior action planning, for literal and metaphoric action
parietal gyrus, and the right aIPL – were correlated with sentences. The right aIPL was additionally involved for
familiarity for Lit sentences. This suggests that metaphors. This supports the view that the
metaphor processing is not fundamentally different understanding of metaphoric action retains a link to
from literal sentence processing, but is similar to sensory-motor systems involved in action performance.
processing relatively unfamiliar sentences whose The aIPL may be an interface area that serves an
comprehension is more effortful. important role in both conceptual and action domains.
Bowdle and Gentner (2005) proposed the “career of Activation of primary motor and biological motion
metaphor” hypothesis to explain the trajectory of perception areas was inversely correlated with metaphor
metaphor processing as metaphors are familiarity, which is consistent with the view that a
conventionalized. According to this hypothesis, gradual abstraction process, whereby relatively detailed
metaphors are initially understood by comparison or simulations are used for understanding unfamiliar
similarity matching between base and target domains. As metaphors, and that these simulations become less
the metaphor becomes more familiar, there is a switch detailed and involve only secondary regions as the
to categorization mode in which abstract metaphoric familiarity increases. The similarity of abstract and
Journal of Cognitive Neuroscience In press 10metaphoric sentences in the activation of the left Chen E, Widick P, Chatterjee A (2008) Functional-anatomical
temporal regions suggests that action metaphor organization of predicate metaphor processing. Brain Lang 107:194-
202.
understanding is not completely based on sensory- Cox RW (1996) AFNI: Software for analysis and visualization of
motor systems, but contains an abstract element. functional magnetic resonance neuroimages. Computers and
Biomedical Research 29:162-173.
Acknowledgements Cox RW, Jesmanowicz A (1999) Real-time 3D image registration
We thank Dana Krauss for help with ratings collection, and of functional MRI. Magnetic Resonance in Medicine 42:1014-1018.
Edward Possing for help with fMRI scanning. We also thank David Cross ES, Hamilton AF, Grafton ST (2006) Building a motor
Kemmerer, Richard Ivry, and an anonymous reviewer for their simulation de novo: observation of dance by dancers. Neuroimage
helpful comments. Supported by grants R03 DC008416 (RHD), R01 31:1257-1267.
NS033576 (JRB), and R01 DC010783 (RHD) from the NIH. Decety J, Grezes J, Costes N, Perani D, Jeannerod M, Procyk E,
Grassi F, Fazio F (1997) Brain activity during observation of actions:
Supplemental Material Influences of action content and subject's strategy. Brain 120:1763-
www.neuro.mcw.edu/~rhdesai/papers/jocn11_desai_senmet_suppl. 1777.
pdf Desai RH, Binder JR, Conant LL, Seidenberg MS (2009)
Activation of Sensory-Motor Areas in Sentence Comprehension.
Cerebral Cortex.
REFERENCES Fischer MH, Zwaan RA (2008) Embodied language: A review of
Andrews M, Vigliocco G, Vinson D (2009) Integrating the role of the motor system in language comprehension. The
experiential and distributional data to learn semantic representations. Quarterly Journal of Experimental Psychology 61:825 - 850.
Psychol Rev 116:463-498. Frak V, Nazir T, Goyette M, Cohen H, Jeannerod M (2010)
Aziz-Zadeh L, Damasio A (2008) Embodied semantics for Grip force is part of the semantic representation of manual action
actions: Findings from functional brain imaging. Journal of verbs. PLoS ONE 5:e9728.
Physiology-Paris 102:35-39. Franklin S, Howard D, Patterson K (1995) Abstract word
Aziz-Zadeh L, Wilson SM, Rizzolatti G, Iacoboni M (2006) anomia. Cognitive Neuropsychology 12:549-566.
Congruent embodied representations for visually presented actions Gibbs R (2006) Metaphor Interpretation as Embodied
and linguistic phrases describing actions. Curr Biol 16:1818-1823. Simulation. Mind & Language 21:434-458.
Barsalou LW (2008) Grounded cognition. Annu Rev Psychol Glover S (2004) Separate visual representations in the planning
59:617-645. and control of action. Behav Brain Sci 27:3-24; discussion 24-78.
Beeman M, Friedman R, Grafman J, Perez E, Diamond S, Glucksberg S (2001) Understanding Figurative Language. New
Lindsay M (1994) Summation priming and coarse semantic coding in York: Oxford University Press.
the right hemisphere. Journal of Cognitive Neuroscience 6:26-45. Goodglass H, Hyde MR, Blumstein S (1969) Frequency,
Bergen BK (2007) Mental simulation in literal and figurative picturability and availability of nouns in aphasia. Cortex 5:104-119.
language understanding. In: The Literal/Non-Literal Distinction Graves WW, Binder JR, Desai RH, Conant LL, Seidenberg MS
(Coulson S, Lewandowska-Tomaszczyk B, eds). Berlin: Peter Lang. (in press) Neural correlates of implicit and explicit conceptual
Binder JR (2007) Effects of word imageability on semantic combination. Neuroimage.
access: Neuroimaging studies. In: Neural basis of semantic memory Grossman ED, Blake R (2002) Brain Areas Active during Visual
(Hart J, Kraut MA, eds), pp 149-181. Cambridge, UK: Cambridge Perception of Biological Motion. Neuron 35:1167-1175.
University Press. Haaland KY, Harrington DL, Knight RT (2000) Neural
Binder JR, Desai RH, Graves WW, Conant LL (2009) Where Is representations of skilled movement. Brain 123 ( Pt 11):2306-2313.
the Semantic System? A Critical Review and Meta-Analysis of 120 Kable JW, Kan IP, Wilson A, Thompson-Schill SL, Chatterjee A
Functional Neuroimaging Studies. Cereb Cortex. (2005) Conceptual representations of action in the lateral temporal
Boulenger V, Hauk O, Pulvermuller F (2009) Grasping Ideas cortex. Journal of Cognitive Neuroscience 17:1855-1870.
with the Motor System: Semantic Somatotopy in Idiom Katz RB, Goodglass H (1990) Deep dysphasia: Analysis of a rare
Comprehension. Cereb Cortex 19:1905-1914. form of repetition disorder. Brain and Language 39:153-185.
Bowdle B, Gentner D (2005) The career of metaphor. Kemmerer D (2005) The spatial and temporal meanings of
Psychological Review 112:193-216. English prepositions can be independently impaired.
Buckner RL, Sepulcre J, Talukdar T, Krienen FM, Liu H, Neuropsychologia 43:797-806.
Hedden T, Andrews-Hanna JR, Sperling RA, Johnson KA (2009) Kemmerer D (in press) Visual and motor features of the
Cortical hubs revealed by intrinsic functional connectivity: mapping, meanings of action verbs: A cognitive neuroscience perspective. In:
assessment of stability, and relation to Alzheimer's disease. J Verb concepts: Cognitive science perspectives on verb
Neurosci 29:1860-1873. representation and processing (de Almeida RG, Manouilidou C, eds).
Buxbaum LJ, Kyle KM, Menon R (2005) On beyond mirror Oxford, UK: Oxford University Press.
neurons: internal representations subserving imitation and Lakoff G, Johnson M (1980) Metaphors We Live By. Chicago:
recognition of skilled object-related actions in humans. Brain Res University of Chicago Press.
Cogn Brain Res 25:226-239. Lakoff G, Johnson M (1999) Philosophy in the Flesh: The
Calhoun VD, Stevens MC, Pearlson GD, Kiehl KA (2004) fMRI Embodied Mind and its Challenge to Western Thought. New York:
analysis with the general linear model: removal of latency-induced Basic Books.
amplitude bias by incorporation of hemodynamic derivative terms. Landauer TK, Dumais ST (1997) A solution to Plato's problem:
Neuroimage 22:252-257. The Latent Semantic Analysis theory of the acquisition, induction,
and representation of knowledge. Psychological Review 104:211-240.
Lyons IM, Mattarella-Micke A, Cieslak M, Nusbaum HC, Small
SL, Beilock SL (2010) The role of personal experience in the neural
Journal of Cognitive Neuroscience In press 11processing of action-related language. Brain and language 112:214- Tranel D, Kemmerer D, Adolphs R, Damasio H, Damasio A
222. (2003) Neural correlates of conceptual knowledge for actions.
Mahon BZ, Caramazza A (2008) A critical look at the embodied Cognitive Neuropsychology 20:409-432.
cognition hypothesis and a new proposal for grounding conceptual Tranel D, Martin C, Damasio H, Grabowski TJ, Hichwa R
content. Journal of Physiology-Paris 102:59-70. (2005) Effects of noun-verb homonymy on the neural correlates of
Martin A (2007) The representation of object concepts in the naming concrete entities and actions. Brain Lang 92:288-299.
brain. Annu Rev Psychol 58:25-45. van Dam WO, Rueschemeyer SA, Lindermann O, Bekkering H
Mayka MA, Corcos DM, Leurgans SE, Vaillancourt DE (2006) (in press) Context effects in embodied lexical-semantic processing.
Three-dimensional locations and boundaries of motor and premotor Front Cognition.
cortices as defined by functional brain imaging: a meta-analysis. Van Essen DC, Drury HA, Dickson J, Harwell J, Hanlon D,
Neuroimage 31:1453-1474. Anderson CH (2001) An integrated software suite for surface-based
Moody CL, Gennari SP (2010) Effects of implied physical effort analyses of cerebral cortex. Journal of American Medical Informatics
in sensory-motor and pre-frontal cortex during language Association 8:443-459.
comprehension. Neuroimage 49:782-793. Wallentin M, Lund TE, Ostergaard S, Ostergaard L, Roepstorff
Oldfield RC (1971) The assessment and analysis of handedness: A (2005) Motion verb sentences activate left posterior middle
the Edinburgh inventory. Neuropsychologia 9:97-113. temporal cortex despite static context. Neuroreport 16:649-652.
Onifer W, Swinney D (1981) Accessing lexical ambiguity during Yang FG, Edens J, Simpson C, Krawczyk DC (2009)
sentence comprehension: Effects of frequency of meaning and Differences in task demands influence the hemispheric lateralization
contextual bias. Memory and Cognition 9:225-236. and neural correlates of metaphor. Brain Lang 111:114-124.
Paivio A (1986) Mental representations: A dual-coding approach. Zhang Y, Brady M, Smith S (2001) Segmentation of brain MR
New York: Oxford University Press. images through a hidden Markov random field model and the
Peeters R, Simone L, Nelissen K, Fabbri-Destro M, Vanduffel expectation-maximization algorithm. IEEE Trans Med Imaging
W, Rizzolatti G, Orban GA (2009) The representation of tool use in 20:45-57.
humans and monkeys: common and uniquely human features. J
Neurosci 29:11523-11539.
Pulvermuller F (2005) Brain mechanisms linking language and
action. Nat Rev Neurosci 6:576-582.
Pynte J, Besson M, Robichon F-H, Poli J (1996) The time-course
of metaphor comprehension: An event-related potential study. Brain
and Language 55:293-316.
Ramayya AG, Glasser MF, Rilling JK (2009) A DTI
Investigation of Neural Substrates Supporting Tool Use. Cerebral
cortex (New York, NY : 1991):507-516.
Raposo A, Moss HE, Stamatakis EA, Tyler LK (2009)
Modulation of motor and premotor cortices by actions, action words
and action sentences. Neuropsychologia 47:388-396.
Rastle K, Harrington J, Coltheart M (2002) 358,534 nonwords:
The ARC Nonword Database. Quarterly Journal of Experimental
Psychology Section a-Human Experimental Psychology 55:1339-
1362.
Saad ZS, Glen DR, Chen G, Beauchamp MS, Desai R, Cox RW
(2009) A new method for improving functional-to-structural MRI
alignment using local Pearson correlation. Neuroimage 44:839-848.
Saygin AP (2007) Superior temporal and premotor brain areas
necessary for biological motion perception. Brain 130:2452-2461.
Saygin AP, McCullough S, Alac M, Emmorey K (2010)
Modulation of BOLD Response in Motion-sensitive Lateral
Temporal Cortex by Real and Fictive Motion Sentences. Journal of
cognitive neuroscience 22:2480-2490.
Schmidt GL, Seger CA (2009) Neural correlates of metaphor
processing: the roles of figurativeness, familiarity and difficulty. Brain
Cogn 71:375-386.
Seidenberg M, Tanenhaus MK, Leiman JL, Bienkowski M (1982)
Automatic access of the meanings of ambiguous words in contex:
Some limiations of knowledge-based processing. Cognitive
Psychology 14:470-519.
Sporns O, Honey CJ, Kotter R (2007) Identification and
classification of hubs in brain networks. PLoS ONE 2:e1049.
Squire LR, Stark CE, Clark RE (2004) The medial temporal lobe.
Annu Rev Neurosci 27:279-306.
Talairach J, Tournoux P (1988) Co-planar Stereotaxic Atlas of
the Human Brain. New York: Thieme Medical.
Journal of Cognitive Neuroscience In press 12You can also read
