The neural career of sensory-motor metaphors

Page created by Jeremy Edwards
 
CONTINUE READING
The neural career of sensory-motor metaphors
The neural career of sensory-motor metaphors
                   Rutvik H. Desai*, Jeffrey R. Binder*, Lisa L. Conant*, Quintino R. Mano†, Mark S.
                                                    Seidenberg‡
                *Medical   College of Wisconsin, †University of Wisconsin-Milwaukee, ‡University of Wisconsin-Madison

Abstract
        The role of sensory-motor systems in                       support the view that the understanding of metaphoric
conceptual understanding has been controversial. It has             action retains a link to sensory-motor systems involved
been proposed than many abstract concepts are                       in action performance. However, the involvement of
understood metaphorically through concrete sensory-                 sensory-motor systems in metaphor understanding
motor domains such as actions. Using fMRI, we                       changes through a gradual abstraction process whereby
compared neural responses to literal action (Lit; The               relatively detailed simulations are used for understanding
daughter grasped the flowers), metaphoric action (Met; The          unfamiliar metaphors, and these simulations become
public grasped the idea), and abstract (Abs; The public             less detailed and involve only secondary motor regions
understood the idea) sentences of varying familiarity. Both         as familiarity increases. Consistent with these data, we
Lit and Met sentences activated the left anterior inferior          propose that aIPL serves as an interface between
partial lobule (aIPL), an area involved in action                   sensory-motor and conceptual systems and plays an
planning, with Met sentences also activating a                      important role in both domains. The similarity of
homologous area in the right hemisphere, relative to                abstract and metaphoric sentences in the activation of
Abs sentences. Both Met and Abs sentences activated                 left superior temporal regions suggests that action
left superior temporal regions associated with abstract             metaphor understanding is not completely based on
language. Importantly, activation in primary motor and              sensory-motor simulations, but relies also on abstract
biological motion perception regions was inversely                  lexical-semantic codes.
correlated with Lit and Met familiarity. These results

                                                                    concepts, provide an interesting opportunity to study
INTRODUCTION
                                                                    the relationship between these two systems.
    The relationship between sensory-motor and                      Engagement of sensory-motor systems even when
conceptual systems of the brain has been the focus of               action language is clearly figurative would suggest a
intense debate in recent years (Pulvermuller, 2005;                 particularly close relationship between these systems,
Barsalou, 2008; Mahon and Caramazza, 2008).                         consistent with theories that many abstract concepts are
Neuroimaging, behavioral, and patient studies suggest a             understood through analogies to sensation and action
closer link between these systems than previously                   (Lakoff and Johnson, 1980, 1999; Gibbs, 2006; Bergen,
recognized (Aziz-Zadeh and Damasio, 2008; Fischer                   2007).
and Zwaan, 2008; Kemmerer, in press). The precise                       The few imaging studies on figurative action
nature of the relationship between these systems,                   language have yielded somewhat inconsistent results.
however, remains unclear. Weak embodiment views                     Aziz-Zadeh et al. (2006) found somatotopic activation
suggest engagement of sensory-motor systems only                    in the premotor cortex for literal action sentences, but
when concepts are transparently related to physical                 not for idiomatic phrases (“biting off more than you can
action. In contrast, strong embodiment assigns sensory-             chew”). Boulenger et al. (2009) found somatotopic
motor systems a pervasive role in comprehension,                    activation for figurative and literal action sentences
including more abstract concepts.                                   involving leg and arm verbs. Raposo et al. (2009) found
    Action metaphors (e.g., grab the chance or grasp an idea),      activation in premotor/motor regions for isolated action
which convey abstract concepts via analogy to concrete              verbs, and to a lesser extent for literal action sentences,

   Journal of Cognitive Neuroscience                                                                   In press    1
The neural career of sensory-motor metaphors
but not for figurative sentences using action verbs.        Institutional Review Board. Participants were paid for
      Three studies have also shown activation in or near         participation.
      motion processing area MT+ for literal as well as
                                                                  Stimuli
      figurative or fictive motion sentences (“The man fell
      under her spell”; “The bridge jumped over the brook”)           Stimuli were sentences divided into three main
      compared to non-motive sentences (Wallentin et al.,         conditions: Literal action (Lit), Metaphoric action (Met),
      2005; Chen et al., 2008; Saygin et al., 2010).              and Abstract (Abs). The stimuli were constructed in
          To elucidate the relationship between sensory-motor     triples consisting of one sentence from each condition,
      and conceptual systems, and to adjudicate between weak      with the same syntactic form (examples in Table 1;
      and strong views of embodiment, we compared BOLD            complete listing provided in the Supplemental Material).
      responses to metaphoric action sentences with two           Table 1. Example stimuli.
      types of non-metaphoric sentences – literal action and        Literal     The daughter grasped the    The thief bashed the table
      abstract. We varied the familiarity of each sentence type                 flowers
      to investigate the modulation of activity in sensory-         Metaphor The jury grasped the concept   The council bashed the
      motor regions. In many studies of metaphors,                                                          proposal
                                                                    Abstract      The jury understood the   The council criticized the
      processing difficulty has been a common confounding                         concept                   proposal
      variable, in that metaphoric stimuli tend to be more
      difficult to process (as indexed by response latencies)          The Lit sentence used a hand/arm action verb to
      than literal control stimuli. Additional activation         depict a physical action. The corresponding Met
      resulting from processing more difficult stimuli can be     sentence used the same verb in a figurative manner,
      mistaken as activation specific to metaphors (Schmidt       such that no physical action was described. The Abs
      and Seger, 2009; Yang et al., 2009). We carefully           sentence used an abstract verb to convey a meaning
      controlled for processing difficulty and other              similar to that of the Met sentence. The agent in each
      confounding variables such as syntactic structure and       sentence was chosen to imply either a literal or
      sentence length, which can also lead to activation in       abstract/metaphoric interpretation of the verb. Met and
      sensory-motor regions. We hypothesized that relatively      Abs sentences always used the same agent. This agent
      unfamiliar (literal and metaphoric) action language         was an entity that makes literal physical actions unlikely
      engages sensory-motor systems because comprehension         (e.g., the scandal, the crime, the government). The Lit
      of such expressions involves relatively detailed            sentences, in contrast, always used a person (the teacher,
      simulations of literal actions. As the expression becomes   the doctor) as an agent. For example, consider the
      more familiar and conventionalized, the reliance on         fragments “The captain lifted…” and “The government
      sensory-motor simulation diminishes. Alternative            lifted…”.       In the first example, when lifted is
      hypotheses are that sensory-motor systems are not           encountered, both a physical action as well as figurative
      engaged at all for metaphoric expressions, or are           lifting (e.g., of a ban) are possible. In the second
      engaged regardless of familiarity.                          example, it is clear the physical action interpretation is
                                                                  infelicitous.
      METHODS                                                          Twenty-seven action verbs were used three times
      Participants                                                each to create 81 Lit and Met sentences. Additionally,
                                                                  there were 81 Abs sentences, 81 Nonsense sentences
          Participants in the fMRI experiment were 22 healthy
                                                                  (e.g., The wedding strummed the introduction; created by
      adults (11 women; average age 24 years, range 18-33;
                                                                  combining action and abstract verbs with inappropriate
      average years of education 16, range 12-23), with no
                                                                  nouns), and 81 nonword sentences of the same form.
      history of neurological impairment. One additional
                                                                  Nonword sentences were composed of pronounceable
      participant was removed due to activations in the three
                                                                  nonwords created using the MCWord database
      main contrasts that were more than two s.d. away from
                                                                  (http://www.neuro.mcw.edu/mcword/) and the ARC nonword
      the group mean. Participants were native speakers of
                                                                  database (Rastle et al., 2002). Finally, there were 54 Filler
      English, and were right-handed according to the
                                                                  sentences that used variable syntax (e.g., All lawyers went
      Edinburgh Handedness Inventory (Oldfield, 1971).
                                                                  on strike). These were used to obscure the triplet
      Informed consent was obtained from each participant
                                                                  construction of the stimuli and to provide syntactic
      prior to the experiment, in accordance with a protocol
                                                                  variability in the stimulus set.
      sanctioned by the Medical College of Wisconsin

Journal of Cognitive Neuroscience                                                              In press                              2
Stimulus Norming                                                      stimuli was then tested in the Meaningfulness Judgment
          One of our principal goals was to equate the three                experiment below, to verify that the differences between
      main conditions with respect to processing difficulty, to             conditions in response time (RT) were indeed
      remove the possible confound between figurativeness                   minimized.
      and difficulty. Numerous factors can affect the difficulty            Meaningfulness Judgment
      of processing sentences. In addition to word frequency,
                                                                                Participants in the meaningfulness judgment
      the frequency of the particular verb-noun combination
                                                                            experiment were 24 adults (20 women; average age 19
      (e.g., “grasp the idea” vs. “grasp the procedure”), and
                                                                            years, range 18-21; average years of education 12, range
      the frequency and familiarity of the verb in a metaphoric
                                                                            12-14). They were native speakers of English and did
      vs. literal sense can affect automaticity of
                                                                            not participate in the fMRI experiment.
      comprehension. As described below, stimuli were pre-
                                                                                Participants made a meaningfulness judgment
      tested using a meaningfulness judgment task, which
                                                                            (meaningful/not meaningful) for each sentence,
      reflects the combined effects of such factors.
                                                                            presented in two parts as in the preliminary experiment
          A two-step procedure was used in developing the
                                                                            above. Participants were asked to judge the
      stimuli. First, a large set of sentences was prepared by
                                                                            meaningfulness of the sentence as a whole by pressing
      combining action and abstract verbs with appropriate
                                                                            one of two buttons.
      nouns to create Lit, Met, and Abs sentences, for use in a
                                                                                As shown in Table 2, there were no significant
      preliminary experiment. Six participants made a
                                                                            differences between Met, Lit, and Abs conditions, while
      meaningfulness judgment (“makes sense” or “does not
                                                                            the Nonsense condition had longer RTs than all of the
      make sense”) for each sentence by pressing one of two
                                                                            other conditions (all p < 0.0001).
      buttons on a response pad. The sentences were
      presented in two parts, as shown in Fig. 1.                           Familiarity Rating
                                                                                The familiarity of sentences can affect speed and
                                                                            accuracy of processing. Although familiarity differences
                                                                            would be expected to affect latencies in the
                                                                            meaningfulness judgment task mentioned above, we
                                                                            also collected familiarity ratings to enable a more direct
                                                                            assessment of familiarity effects in the imaging analysis.
                                                                                Participants in the familiarity rating experiment were
                                                                            28 adults (16 women; average age 19 years, range 18-21;
                                                                            average years of education 13, range 12-16). They were
                                                                            native speakers of English and did not participate in the
      Figure 1. The presentation of the stimuli. The first noun phrase of
                                                                            fMRI experiment.
      the sentence was displayed for 500 ms, followed by the remaining
      sentence for 1300 ms. The sentences were separated by variable            For each sentence, participants rated each sentence
      intervals.                                                            on a scale of 1 (not at all familiar) to 7 (very familiar).
                                                                            The results are shown in Table 2. Lit and Abs stimuli
          The first screen displayed a noun phrase (e.g., “The              differed in familiarity (p < 0.02), with no other reliable
      public”) for 500 ms. This was replaced by the verb                    differences between types.
      phrase (e.g., “grasped the idea”) on the second screen,                   As expected, the Meaningfulness Judgment RTs and
      displayed for 1300 ms. This two-part presentation was                 the familiarity ratings were correlated (Pearson’s r = -
      used to ensure that the first noun phrase, which suggests             0.52, p < 0.0001).
      the literal or metaphoric/abstract interpretation of the
      verb, was read first. Consistent with previous studies, Lit           Table 2. The mean (s.d.) RTs in the Meaningfulness Judgement task,
      sentences yielded faster responses than Met and Abs                   and the Familiarity Ratings (on a scale of 1 to 7) for various
      sentences. Sentences were then modified to reduce these               conditions.
      differences. (e.g., by using a more familiar noun-verb                       Condition      n         RT            Familiarity
      combination to reduce the difficulty of a metaphoric                          Metaphor     81     1277 (165)         5.24 (0.77)
      sentence, or by using a less familiar noun-verb                                Literal     81     1241 (151)         4.97 (1.04)
      combination to replace literal sentences that were very                       Abstract     81     1265 (179)         5.37 (0.87)
      easy). In modifying the sentences, we also reduced                           Nonsense      81     1399 (167)              -
      differences in word frequencies and in the number of                        Pseudoword 81          723 (138)              -
      phonemes, letters, and syllables. The revised set of                           Filler      54     1234 (232)              -

Journal of Cognitive Neuroscience                                                                        In press                         3
Action Association Rating                                       and fifth digit – with their left hand or right hand, or
          We intentionally selected verbs that were clearly           rested. Prior to scanning, these actions were
      associated with actions in the Met/Lit conditions,              demonstrated by the experimenter outside the scanner,
      whereas verbs used in the Abs condition were relatively         without using verbal labels, and the subjects were asked
      abstract. It is possible, however, that some of these           to repeat these actions, in the same sequence, for
      “abstract” verbs also have some association with                practice and verification of accuracy. In the scanner, the
      actions. Action association ratings were collected to           instructions “left,” “right,” or “rest” were displayed at
      assess whether abstract verbs were in fact less associated      the beginning of each block. Each block was 18 seconds
      with actions than Met/Lit verbs. Participants in this           long, and three blocks of each condition were presented
      rating experiment were 14 adults (7 women; average age          in a pseudo-randomized order.
      20 years, range 18-22; average years of education 13,           Image Acquisition and Analysis
      range 12-15), native speakers of English who did not
                                                                           A 3T GE Excite scanner was used to acquire images.
      participate in the fMRI experiment. Participants rated
                                                                      One volume of T2*-weighted, gradient echo, echo-
      each verb, preceded by to (e.g., to understand) on a scale of
                                                                      planar images (TE = 25 ms, flip angle = 77°, NEX = 1)
      1 (not associated with action at all) to 7 (very much
                                                                      was acquired every 1.8 s. Visual sentence presentation
      associated with action). The mean (s.d.) for Met/Lit
                                                                      was time-locked with the beginning of an acquisition.
      verbs was 6.17 (0.42) and for Abs verbs was 3.62 (0.70),
                                                                      Volumes were composed of 30 axially-oriented 3.5 mm
      a difference that was highly significant (p < 0.0001).
                                                                      slices with a 0.5 mm interslice gap, covering the whole
      FMRI Tasks                                                      brain, with FOV = 240 mm and 64 × 64 matrix,
          The sentences in the imaging experiment were                resulting in 3.75 × 3.75 × 4 mm voxel dimensions.
      presented in two parts, as in the Meaningfulness                Anatomical images of the entire brain were obtained
      judgment experiment (Fig 1). Participants were                  using a 3D spoiled gradient echo sequence (SPGR) with
      instructed to read each sentence and make a covert              0.94 × 0.94 × 1 mm voxel dimensions.
      meaningfulness decision. A covert task was used to                   The AFNI software package (Cox, 1996) was used
      prevent strong activation of the motor cortex by a              for image analysis. Within-subject analysis involved
      manual or vocal response. The order of sentences was            spatial co-registration (Cox and Jesmanowicz, 1999) and
      pseudo-randomized, and the interval between the                 registration of functional images to the anatomy (Saad et
      sentences was varied, to allow optimal statistical              al., 2009). Runs were removed from the analysis if d’
      separation of the hemodynamic response to each                  performance on the recognition test after a run was less
      condition. The sentences were divided into 9 runs               then 1, or if the self-reported attentiveness rating was 5
      lasting approximately 5 minutes each. To encourage              or less; 11 runs (5.6%) were removed in this manner.
      attentiveness, participants were also tested on a               Voxel-wise multiple linear regression was performed
      recognition task after each run. Fourteen sentences were        with reference functions representing each condition.
      shown, and for each sentence, participants indicated by         Additionally, familiarity ratings for each stimulus (see
      pressing one of two buttons whether they had seen the           Stimulus Norming, above) were used to create within-
      sentence in the preceding run. On average, half of the          condition familiarity regressors for the Lit, Met, and Abs
      14 sentences were taken from the previous run, whereas          conditions. Mean-centered regressors for the number of
      the others were not in the experiment. At the end of            syllables and phonemes in each sentence were used as
      each run, participants were also asked to rate their            additional item-wise regressors to account for
      attentiveness during the task on a scale of 1 (not              differences due to these variables. A standard
      attentive at all) to 10 (very attentive). Instructions and      hemodynamic response function convolved with the
      practice with all tasks were provided outside the scanner       reference functions, and its temporal derivative, were
      before the scan, and the participants were informed that        used. A correction for amplitude bias was applied using
      the recognition test would be administered after each           the method described by Calhoun et al. (2004). Six
      run.                                                            motion parameters and the signal extracted from the
                                                                      ventricles, segmented using the FSL fast program
      Motor Localizer Task                                            (Zhang et al., 2001), were included as noise covariates of
          After the sentence runs, a localizer task was used to       no interest. General linear tests were conducted to
      locate hand motor regions of the brain, using a block           obtain the Lit-Abs, Met-Lit, and Met-Abs contrasts and
      design. Participants performed a repeating sequence of          Familiarity × Condition interactions.
      actions – make a fist, turn the palm up, touch thumb

Journal of Cognitive Neuroscience                                                              In press                     4
The individual statistical maps and the anatomical                      was restricted to a mask that excluded areas outside the
      scans were projected into standard stereotaxic space                        brain, as well as deep white matter areas and the
      (Talairach and Tournoux, 1988) and smoothed with a                          ventricles. The data from the motor localizer scan were
      Gaussian filter of 5 mm FWHM. In a random effects                           analyzed as a block design in a similar way. To further
      analysis, group maps were created by comparing                              examine motor areas, two regions of interest (ROIs)
      activations against a constant value of 0. The group                        were defined. One used the area activated by the motor
      maps were thresholded at voxelwise p < 0.01 and                             localizer task as an ROI, and the other used primary
      corrected for multiple comparisons by removing clusters                     motor and sensory cortex (M1 and S1) as defined by the
      smaller than 1000 µl to achieve a mapwise corrected                         HMAT atlas (Mayka et al., 2006). Small volume
      two-tailed p < 0.05. The cluster threshold was                              correction was applied in these ROIs to achieve
      determined through Monte Carlo simulations that                             corrected p < 0.05.
      estimate the chance probability of spatially contiguous
      voxels exceeding the voxelwise p threshold. The analysis

                    Figure 2. Areas activated by condition contrasts. Yellow-orange scale shows greater activation for the first condition; blue-
                    cyan scale shows greater activation for the second condition in the contrast. L = left hemisphere, R = right hemisphere.

Journal of Cognitive Neuroscience                                                                                In press                           5
RESULTS                                                      computed Familiarity × Condition interactions. The
                                                                   regions responding only to general difficulty and task
           The mean (s.d.) d’ in the post-run test was 2.56
                                                                   load effects should be modulated similarly in all three
      (0.74), suggesting that the participants were attentive to
                                                                   conditions and therefore would not show interactions.
      the stimuli during the scans. We first describe the fMRI
                                                                   Indeed, middle and inferior frontal lobe regions,
      results for the contrasts between the three main
                                                                   commonly associated with task difficulty effects, were
      conditions, Lit, Met and Abs. The results are displayed
                                                                   negatively correlated with familiarity in each condition
      on an inflated brain surface using Caret (Van Essen et
                                                                   (see Supplemental Material, Part II) and were absent
      al., 2001). A complete listing of the activated areas with
                                                                   from the interaction maps. In addition, because our
      coordinates is provided in the Supplemental Material
                                                                   hypotheses concern correlations with Lit or Met
      (Part I).
                                                                   familiarity, we applied a mask to the interaction maps
      Literal - Abstract                                           that included only voxels that showed a significant
          The areas activated to a greater extent by the Lit       correlation with familiarity in Lit or Met conditions.
      condition relative to the Abs condition included the left    Literal-Abs × Familiarity
      anterior inferior parietal lobule (aIPL; including
      supramarginal gyrus and postcentral sulcus), left                All the regions in this interaction showed negative
      parahippocampal and fusiform gyrus, left precuneus, left     correlation with familiarity for Lit sentences, and greater
      posterior middle and inferior temporal gyrus and lateral     negative correlation for Lit than for Abs sentences (cyan
      occipital gyrus, left superior frontal gyrus, left           and green in Fig 3a). They included the left anterior
      orbitofrontal cortex, bilateral cerebellum and thalamus,     inferior frontal gyrus, central sulcus, superior parietal
      as well as the right hippocampus and fusiform gyrus          gyrus, and posterior superior temporal sulcus; the right
      (Fig 2a).                                                    aIPL; and bilateral posterior middle and inferior
          Abs sentences activated the left superior temporal       temporal gyrus, parietal operculum, and SMA.
      sulcus and the anterior superior temporal gyrus, cuneus,     Metaphor-Abstract × Familiarity
      as well as the right angular gyrus.                             Similar to the previous interaction, all areas here
      Metaphor - Abstract                                          showed negative correlations for Met sentences, and
          The Met condition activated left aIPL, Rolandic          greater negative correlations for Met than for Abs
      operculum, superior parietal gyrus, and the superior         sentences. These areas included the left posterior
      frontal gyrus. The cerebellum and thalamus were              superior temporal sulcus and bilateral central sulcus,
      activated bilaterally. The right aIPL, parietal operculum,   SMA, lingual gyrus, and cuneus (cyan and green in Fig
      insula, and parahippocampal gyrus were also activated        3b).
      (Fig 2b). No areas were activated to a greater extent for    Metaphor-Literal × Familiarity
      the Abs sentences.                                               The right supramarginal gyrus and the left superior
      Metaphor - Literal                                           parietal gyrus showed a significant interaction. The right
          Compared to Lit sentences, the Met sentences             supramarginal gyrus showed a positive correlation with
      activated the left anterior and posterior cingulate gyrus,   familiarity for Met, while the left superior parietal gyrus
      cuneus, superior temporal sulcus, and the temporal pole.     was negatively correlated in the Lit condition (cyan and
      In the right hemisphere, the aIPL, parietal operculum        green in Fig 3c).
      and the superior parietal gyrus were activated (Fig 2c).         Identical analyses were also carried out using RTs,
          The Lit condition activated the left parahippocampal     rather than familiarity ratings, as regressors. No
      and fusiform gyrus relative to the Met sentences.            Condition × RT interactions in sensory-motor regions
                                                                   were found after identical application of corrections for
      Correlations with Familiarity                                multiple comparisons. This suggests that these
          To assess the effects of sentence familiarity on the     interactions are unlikely to be due to differences only in
      activation, we used the Familiarity Rating as a condition-   the length of action simulations or some form of pre-
      wise regressor in the analysis. Correlations with            response motor readiness (although the latter is
      familiarity can pinpoint areas involved in the semantic      unexpected in any case because no responses were made
      processing, but also areas modulated due to general          during scanning).
      processing difficulty. To isolate, to the extent possible,
      activation modulation due to semantic factors, we

Journal of Cognitive Neuroscience                                                            In press                     6
Figure 3. The overlap of condition contrasts, condition x familiarity interactions, and the motor localizer. Interactions in (a) and (b) show
      greater negative correlation to familiarity for Lit and Met conditions respectively. Talairach y coordinates are indicated in the upper left corner of
      each slice. Stereotaxic x and z axis are shown in white.

Journal of Cognitive Neuroscience                                                                                   In press                           7
sentences, and whether this engagement changes with
      Overlap with Localizer
                                                                           sentence familiarity.
          Figure 3 also shows the overlap of the areas
      activated by the hand localizer task, the condition                  Literal Sentences
      contrasts, and the Condition × Familiarity interactions.                 The Lit > Abs contrast produced results similar to a
      Activations in the central sulcus, aIPL, posterior                   previous study (Desai et al., 2009). The aIPL region,
      superior temporal sulcus, posterior middle and inferior              overlapping the localizer activation, is a secondary
      temporal gyrus, opercular regions, SMA, thalamus, and                sensory-motor area associated with action planning and
      cerebellum overlapped the localizer activation (magenta,             complex hand-object interaction, as suggested by both
      cyan and white colors). In contrast, activations in the              imaging and lesion studies. It is structurally connected to
      dorsomedial prefrontal region, parahippocampal and                   posterior middle temporal and inferior frontal gyri,
      fusiform gyrus, middle superior temporal sulcus, and                 forming a tool-use network (Ramayya et al., 2009), and
      posterior cingulate did not overlap the localizer (red and           is strongly linked to action performance, imitation, and
      green colors), and occipital regions partly overlapped.              semantics (Haaland et al., 2000; Tranel et al., 2003;
      The left posterior middle/inferior temporal region was               Glover, 2004; Buxbaum et al., 2005; Binder et al., 2009;
      the only area to show an overlap between the condition               for further discussion, see Desai et al., 2009). Peeters et
      contrasts and the familiarity interactions (white color).            al. (2009) compared human and trained as well as
          A summary of the main results is presented in Table              untrained monkeys as they observed actions performed
      3, where areas commonly and differentially activated                 using simple tools, and found that aIPL was uniquely
      between the contrasts and familiarity interactions can be            activated in humans. They proposed that this region has
      seen.                                                                evolved only in humans to subserve complex actions.
                                                                           The present results suggest that this region may be
      Table 3. A summary of the main results. ‘×’ indicates regions that   unique to humans partly because it serves as an interface
      are activated or correlated with familiarity, or overlap with the    between language and action, playing a role in both
      localizer activation.
                                                                           domains.
                                          Contrast
                                                                               Parahippocampal gyrus and surrounding cortex is
                                    Lit > Met > Lit-Abs Met-Abs Loc.       most closely associated with episodic and spatial
                Area
                                    Abs Abs x Fam x Fam         Overlap    memory (Squire et al., 2004). Several studies report
                                                                           activation in this region for processing actions, tools, or
     L dorsomedial prefrontal         x                                    concrete objects, e.g., for simulating rehearsed actions
     L parahippocampal/                                                    (Cross et al., 2006), recognizing and naming actions
                                      x
     fusiform g
                                                                           (Decety et al., 1997; Tranel et al., 2005), for artifacts
     L aIPL                           x     x                      x
                                                                           compared to living things (Martin, 2007), and for
     Bilateral thalamus               x     x                      x
                                                                           concrete compared to abstract words (Binder, 2007).
     Bilateral cerebellum             x     x                      x
                                                                           These findings suggest that activation of this region for
     L p mid temporal g               x            x               x       Lit sentences reflects the retrieval of contextual and
     Bilateral parietal operc.              x      x               x       spatial information related to actions and the concrete
     R aIPL                                 x      x               x       nouns in these sentences.
     L sup parietal lobule                  x      x               x           The posterior middle and inferior temporal gyri are
     R p mid temporal g                            x               x       associated with linguistic knowledge about tools and
     L central s                                   x       x       x       actions (Martin, 2007; Binder et al., 2009). This
     L p sup temporal s                            x       x       x       activation, overlapping with the localizer, was
     Bilateral SMA                                 x       x       x       immediately anterior to the visual motion processing
     R central s                                           x       x       area MT/MST, suggesting a role in more abstract
     Bilateral Lingual g                                   x       x       motion processing (Kable et al., 2005; Chen et al., 2008).
                                                                               The cerebellar and precuneus activation overlapped
      DISCUSSION                                                           the localizer, whereas the dorsomedial prefrontal
         We presented participants with literal action,                    activation did not. This latter region is frequently
      metaphoric action, and abstract sentences to examine                 activated during processing of concrete semantic
      the engagement of sensory-motor areas during their                   concepts, and is thought to play a role in their retrieval
      comprehension. We asked whether sensory-motor areas                  (Binder et al., 2009).
      are engaged even when processing metaphoric action

Journal of Cognitive Neuroscience                                                                    In press                     8
Thus, the pattern of activation for Lit sentences          meaning should result in a signal that is weaker than the
      suggests a role for sensory-motor systems in their            signal from sustained activation of the literal meaning
      comprehension. This activation is consistent with the         for Lit sentence, integrated over the sentence, whereas
      view that understanding such sentences involves action        here the two signals were of similar magnitude.
      simulation, but this simulation is at a relatively abstract   Futhermore, the initial noun phrases of the sentences
      level, engaging higher level action planning and motion       were selected to prime an abstract or concrete meaning
      perception areas.                                             of the verb, and this property promotes rapid
                                                                    suppression of incongruous meanings.
      Metaphoric Sentences
                                                                        Compared to both Lit and Abs conditions, the right
          Although the meaning conveyed by metaphoric               aIPL was also activated for metaphors. The RH
      sentences is abstract, an analogy with a concrete domain      activation can be interpreted in terms of Beeman’s
      is used to convey the meaning. Some theorists have            (1994) fine-coarse coding theory. It suggests that the
      suggested that mental simulation is used to understand        RH maintains a wider “semantic field,” containing
      such action metaphors (Gibbs, 2006; Bergen, 2007), and        alternative meanings or distantly related features, while
      thus the metaphoric meaning is “grounded” in the literal      the LH processes the dominant meanings or features.
      meaning.                                                      The RH activation may therefore represent access to a
          The Met > Abs contrast activated regions associated       wider variety of meanings to subserve metaphor
      with sensory-motor processing (the aIPL and bilateral         interpretation.
      cerebellum), which were also activated by the Lit                 The posterior cingulate activation in the Met > Lit
      sentences and localizer task. Notably, Met and Lit            contrast is found in many semantic studies (Binder et al.,
      sentences activated these regions to a similar extent and     2009). This region has been identified as a connectivity
      hence were absent from the Met-Lit comparison. The            hub (Sporns et al., 2007; Buckner et al., 2009). Hubs
      superior parietal lobule was activated additionally for       contain disproportionally numerous connections and are
      metaphors, which is associated with control of action         hypothesized to integrate diverse informational sources.
      and computation of dynamic spatial information                The involvement of posterior cingulate in metaphor
      (Glover, 2004). If the activation of the left aIPL and        comprehension may be related to integrating
      bilateral cerebellum is taken as an index of sensory-         information from target and base domains.
      motor processing during sentence comprehension, this              Met sentences also activated the left middle superior
      suggests that the understanding of sensory-motor              temporal sulcus, similar to the Abs sentences. This
      metaphors is not abstracted away from their sensory-          activation, which did not overlap with the localizer,
      motor origins. As with literal action sentences, a            could reflect the computation of abstract meaning
      (relatively abstract) motoric simulation is used in           conveyed by the Met sentences. This similarity between
      comprehension of action metaphors. Grasping an idea is        Abs and Met conditions suggests that sensory-motor
      understood much like grasping a handle is, using meanings     metaphors are not represented entirely in a sensory-
      that are based on sensory-motor representations.              motor format. Although motoric simulations may be
          An alternative interpretation of these activations is     used to understand such metaphors, an abstract
      that verbs such as grasp are homonyms with two                component is also present.
      independent meanings, one related to the physical
      action and one meaning “to understand.” Both                  Abstract Sentences
      meanings are initially activated during sentence                  The activation of the left middle and anterior
      processing, and the incongruent meaning is later              superior temporal sulcus for Abs sentences is consistent
      suppressed. In this view, the activation of sensory-          with a number of studies comparing abstract to concrete
      motor areas during Met simply represents the activation       stimuli (for a review, see Binder, 2007). According to
      of the incongruous literal meaning, and the metaphoric        Pavio’s (1986) dual coding theory, abstract information
      meaning is abstract and unrelated to sensory-motor            is represented mainly through verbal associations with
      systems. However, a number of behavioral studies              other words. Andrews et al. (2009) distinguish between
      suggest that when processing homonymous or                    distributional and experiential statistics upon which
      polysemous words in sentences, incongruous meanings           representations are built. With limited experiential
      are either not activated at all, or are suppressed within     (sensory-motor) features, abstract concepts may rely
      approximately 250-300 ms (Onifer and Swinney, 1981;           heavily on distributional information (i.e., statistical
      Seidenberg et al., 1982; Pynte et al., 1996; Glucksberg,      information about word co-occurrences, also captured
      2001). The initial brief activation of an unrelated           by computational models such as Latent Semantic

Journal of Cognitive Neuroscience                                                            In press                     9
Analysis; Landauer and Dumais, 1997). For example, a          meanings of the base concept are activated, and the
      representation of justice may be built gradually through      appropriate meaning is assigned to the target. Our
      associations with concepts such as fair, law, good, court,    results suggest a different picture, at least for sensory-
      right, truth, etc. The left-dominant temporal activation is   motor metaphors. The target is understood in terms of
      also consistent with this view, given the role of this area   the base domain through motoric simulations, which
      in lexical phonology. Further support for this view           gradually become less detailed while still maintaining
      comes from aphasic patients with stroke in the left           their roots in the base domain. The negative correlation
      perisylvian region, who generally show greater                of primary motor areas with metaphor familiarity, and
      processing deficits for abstract words (Goodglass et al.,     the activation of secondary motor regions for
      1969; Katz and Goodglass, 1990; Franklin et al., 1995).       metaphors regardless of familiarity, suggest a gradual
                                                                    abstraction rather than a switch in the processing mode.
      Correlations with Familiarity
                                                                        The right supramarginal gyrus was correlated
          Activation in a number of sensory-motor regions,          positively with Met familiarity. For more familiar
      overlapping with the localizer activation, were negatively    metaphors, this area may play a role in using the wider
      correlated with familiarity for both Lit and Met              RH semantic field to efficiently combine words in
      sentences, even after accounting for the increased            phrases such as grasp an idea. For less familiar metaphors,
      general difficulty of processing less familiar sentences.     such automatic combination is not possible, and more
      Notably, SMA and primary motor areas in the central           on-line simulation is necessary. This view is supported
      sulcus – in the LH for Lit and bilaterally for Met – were     by a study in our lab in which this area was activated for
      correlated with familiarity. The left posterior superior      meaningful two-word phrases such as flower girl relative
      temporal sulcus was also correlated with both Lit and         to difficult-to-interpret combinations such as girl flower
      Met familiarity, and is implicated in biological motion       (Graves et al., in press).
      (Grossman and Blake, 2002; Saygin, 2007). This                    While our aim was to investigate the effects of
      suggests that to understand less familiar action-related      variations in familiarity, other factors, such as amount of
      language, a relatively detailed simulation is used that       personal experience with specific actions (Lyons et al.,
      relies on primary motor areas, and this is true even for      2010), amount of physical effort (Moody and Gennari,
      metaphoric language in which no literal action is             2010) and force (Frak et al., 2010) required for an
      implied. As familiarity increases, the abstractness of        action, and semantic context (van Dam et al., in press)
      simulation also increases, involving only the secondary       may also modulate sensory-motor areas. The effects of
      action-planning region (aIPL) that was activated              these variables, and their interactions with familiarity,
      regardless of familiarity. Moreover, it is possible that at   await further research.
      the highest end on the familiarity scale that was not
      examined here, as in the case of idioms or the pervasive      CONCLUSIONS
      “time is space” metaphors, sensory-motor systems are
      engaged to a lesser extent or not at all (Kemmerer,               A comparison of literal action, metaphoric action,
      2005).                                                        and abstract sentences revealed activation of secondary
          Notably, several areas that were activated consistently   sensory-motor areas including the left aIPL, involved in
      for Met – bilateral parietal operculum, left superior         action planning, for literal and metaphoric action
      parietal gyrus, and the right aIPL – were correlated with     sentences. The right aIPL was additionally involved for
      familiarity for Lit sentences. This suggests that             metaphors. This supports the view that the
      metaphor processing is not fundamentally different            understanding of metaphoric action retains a link to
      from literal sentence processing, but is similar to           sensory-motor systems involved in action performance.
      processing relatively unfamiliar sentences whose              The aIPL may be an interface area that serves an
      comprehension is more effortful.                              important role in both conceptual and action domains.
          Bowdle and Gentner (2005) proposed the “career of         Activation of primary motor and biological motion
      metaphor” hypothesis to explain the trajectory of             perception areas was inversely correlated with metaphor
      metaphor        processing     as      metaphors        are   familiarity, which is consistent with the view that a
      conventionalized. According to this hypothesis,               gradual abstraction process, whereby relatively detailed
      metaphors are initially understood by comparison or           simulations are used for understanding unfamiliar
      similarity matching between base and target domains. As       metaphors, and that these simulations become less
      the metaphor becomes more familiar, there is a switch         detailed and involve only secondary regions as the
      to categorization mode in which abstract metaphoric           familiarity increases. The similarity of abstract and

Journal of Cognitive Neuroscience                                                             In press                     10
metaphoric sentences in the activation of the left                            Chen E, Widick P, Chatterjee A (2008) Functional-anatomical
      temporal regions suggests that action metaphor                            organization of predicate metaphor processing. Brain Lang 107:194-
                                                                                202.
      understanding is not completely based on sensory-                             Cox RW (1996) AFNI: Software for analysis and visualization of
      motor systems, but contains an abstract element.                          functional magnetic resonance neuroimages. Computers and
                                                                                Biomedical Research 29:162-173.
      Acknowledgements                                                              Cox RW, Jesmanowicz A (1999) Real-time 3D image registration
           We thank Dana Krauss for help with ratings collection, and           of functional MRI. Magnetic Resonance in Medicine 42:1014-1018.
      Edward Possing for help with fMRI scanning. We also thank David               Cross ES, Hamilton AF, Grafton ST (2006) Building a motor
      Kemmerer, Richard Ivry, and an anonymous reviewer for their               simulation de novo: observation of dance by dancers. Neuroimage
      helpful comments. Supported by grants R03 DC008416 (RHD), R01             31:1257-1267.
      NS033576 (JRB), and R01 DC010783 (RHD) from the NIH.                          Decety J, Grezes J, Costes N, Perani D, Jeannerod M, Procyk E,
                                                                                Grassi F, Fazio F (1997) Brain activity during observation of actions:
      Supplemental Material                                                     Influences of action content and subject's strategy. Brain 120:1763-
      www.neuro.mcw.edu/~rhdesai/papers/jocn11_desai_senmet_suppl.              1777.
      pdf                                                                           Desai RH, Binder JR, Conant LL, Seidenberg MS (2009)
                                                                                Activation of Sensory-Motor Areas in Sentence Comprehension.
                                                                                Cerebral Cortex.
      REFERENCES                                                                    Fischer MH, Zwaan RA (2008) Embodied language: A review of
           Andrews M, Vigliocco G, Vinson D (2009) Integrating                  the role of the motor system in language comprehension. The
      experiential and distributional data to learn semantic representations.   Quarterly Journal of Experimental Psychology 61:825 - 850.
      Psychol Rev 116:463-498.                                                      Frak V, Nazir T, Goyette M, Cohen H, Jeannerod M (2010)
           Aziz-Zadeh L, Damasio A (2008) Embodied semantics for                Grip force is part of the semantic representation of manual action
      actions: Findings from functional brain imaging. Journal of               verbs. PLoS ONE 5:e9728.
      Physiology-Paris 102:35-39.                                                   Franklin S, Howard D, Patterson K (1995) Abstract word
           Aziz-Zadeh L, Wilson SM, Rizzolatti G, Iacoboni M (2006)             anomia. Cognitive Neuropsychology 12:549-566.
      Congruent embodied representations for visually presented actions             Gibbs R (2006) Metaphor Interpretation as Embodied
      and linguistic phrases describing actions. Curr Biol 16:1818-1823.        Simulation. Mind & Language 21:434-458.
           Barsalou LW (2008) Grounded cognition. Annu Rev Psychol                  Glover S (2004) Separate visual representations in the planning
      59:617-645.                                                               and control of action. Behav Brain Sci 27:3-24; discussion 24-78.
           Beeman M, Friedman R, Grafman J, Perez E, Diamond S,                     Glucksberg S (2001) Understanding Figurative Language. New
      Lindsay M (1994) Summation priming and coarse semantic coding in          York: Oxford University Press.
      the right hemisphere. Journal of Cognitive Neuroscience 6:26-45.              Goodglass H, Hyde MR, Blumstein S (1969) Frequency,
           Bergen BK (2007) Mental simulation in literal and figurative         picturability and availability of nouns in aphasia. Cortex 5:104-119.
      language understanding. In: The Literal/Non-Literal Distinction               Graves WW, Binder JR, Desai RH, Conant LL, Seidenberg MS
      (Coulson S, Lewandowska-Tomaszczyk B, eds). Berlin: Peter Lang.           (in press) Neural correlates of implicit and explicit conceptual
           Binder JR (2007) Effects of word imageability on semantic            combination. Neuroimage.
      access: Neuroimaging studies. In: Neural basis of semantic memory             Grossman ED, Blake R (2002) Brain Areas Active during Visual
      (Hart J, Kraut MA, eds), pp 149-181. Cambridge, UK: Cambridge             Perception of Biological Motion. Neuron 35:1167-1175.
      University Press.                                                             Haaland KY, Harrington DL, Knight RT (2000) Neural
           Binder JR, Desai RH, Graves WW, Conant LL (2009) Where Is            representations of skilled movement. Brain 123 ( Pt 11):2306-2313.
      the Semantic System? A Critical Review and Meta-Analysis of 120               Kable JW, Kan IP, Wilson A, Thompson-Schill SL, Chatterjee A
      Functional Neuroimaging Studies. Cereb Cortex.                            (2005) Conceptual representations of action in the lateral temporal
           Boulenger V, Hauk O, Pulvermuller F (2009) Grasping Ideas            cortex. Journal of Cognitive Neuroscience 17:1855-1870.
      with the Motor System: Semantic Somatotopy in Idiom                           Katz RB, Goodglass H (1990) Deep dysphasia: Analysis of a rare
      Comprehension. Cereb Cortex 19:1905-1914.                                 form of repetition disorder. Brain and Language 39:153-185.
           Bowdle B, Gentner D (2005) The career of metaphor.                       Kemmerer D (2005) The spatial and temporal meanings of
      Psychological Review 112:193-216.                                         English      prepositions      can    be     independently     impaired.
           Buckner RL, Sepulcre J, Talukdar T, Krienen FM, Liu H,               Neuropsychologia 43:797-806.
      Hedden T, Andrews-Hanna JR, Sperling RA, Johnson KA (2009)                    Kemmerer D (in press) Visual and motor features of the
      Cortical hubs revealed by intrinsic functional connectivity: mapping,     meanings of action verbs: A cognitive neuroscience perspective. In:
      assessment of stability, and relation to Alzheimer's disease. J           Verb concepts: Cognitive science perspectives on verb
      Neurosci 29:1860-1873.                                                    representation and processing (de Almeida RG, Manouilidou C, eds).
           Buxbaum LJ, Kyle KM, Menon R (2005) On beyond mirror                 Oxford, UK: Oxford University Press.
      neurons: internal representations subserving imitation and                    Lakoff G, Johnson M (1980) Metaphors We Live By. Chicago:
      recognition of skilled object-related actions in humans. Brain Res        University of Chicago Press.
      Cogn Brain Res 25:226-239.                                                    Lakoff G, Johnson M (1999) Philosophy in the Flesh: The
           Calhoun VD, Stevens MC, Pearlson GD, Kiehl KA (2004) fMRI            Embodied Mind and its Challenge to Western Thought. New York:
      analysis with the general linear model: removal of latency-induced        Basic Books.
      amplitude bias by incorporation of hemodynamic derivative terms.              Landauer TK, Dumais ST (1997) A solution to Plato's problem:
      Neuroimage 22:252-257.                                                    The Latent Semantic Analysis theory of the acquisition, induction,
                                                                                and representation of knowledge. Psychological Review 104:211-240.
                                                                                    Lyons IM, Mattarella-Micke A, Cieslak M, Nusbaum HC, Small
                                                                                SL, Beilock SL (2010) The role of personal experience in the neural

Journal of Cognitive Neuroscience                                                                               In press                           11
processing of action-related language. Brain and language 112:214-               Tranel D, Kemmerer D, Adolphs R, Damasio H, Damasio A
      222.                                                                         (2003) Neural correlates of conceptual knowledge for actions.
           Mahon BZ, Caramazza A (2008) A critical look at the embodied            Cognitive Neuropsychology 20:409-432.
      cognition hypothesis and a new proposal for grounding conceptual                 Tranel D, Martin C, Damasio H, Grabowski TJ, Hichwa R
      content. Journal of Physiology-Paris 102:59-70.                              (2005) Effects of noun-verb homonymy on the neural correlates of
           Martin A (2007) The representation of object concepts in the            naming concrete entities and actions. Brain Lang 92:288-299.
      brain. Annu Rev Psychol 58:25-45.                                                van Dam WO, Rueschemeyer SA, Lindermann O, Bekkering H
           Mayka MA, Corcos DM, Leurgans SE, Vaillancourt DE (2006)                (in press) Context effects in embodied lexical-semantic processing.
      Three-dimensional locations and boundaries of motor and premotor             Front Cognition.
      cortices as defined by functional brain imaging: a meta-analysis.                Van Essen DC, Drury HA, Dickson J, Harwell J, Hanlon D,
      Neuroimage 31:1453-1474.                                                     Anderson CH (2001) An integrated software suite for surface-based
           Moody CL, Gennari SP (2010) Effects of implied physical effort          analyses of cerebral cortex. Journal of American Medical Informatics
      in sensory-motor and pre-frontal cortex during language                      Association 8:443-459.
      comprehension. Neuroimage 49:782-793.                                            Wallentin M, Lund TE, Ostergaard S, Ostergaard L, Roepstorff
           Oldfield RC (1971) The assessment and analysis of handedness:           A (2005) Motion verb sentences activate left posterior middle
      the Edinburgh inventory. Neuropsychologia 9:97-113.                          temporal cortex despite static context. Neuroreport 16:649-652.
           Onifer W, Swinney D (1981) Accessing lexical ambiguity during               Yang FG, Edens J, Simpson C, Krawczyk DC (2009)
      sentence comprehension: Effects of frequency of meaning and                  Differences in task demands influence the hemispheric lateralization
      contextual bias. Memory and Cognition 9:225-236.                             and neural correlates of metaphor. Brain Lang 111:114-124.
           Paivio A (1986) Mental representations: A dual-coding approach.             Zhang Y, Brady M, Smith S (2001) Segmentation of brain MR
      New York: Oxford University Press.                                           images through a hidden Markov random field model and the
           Peeters R, Simone L, Nelissen K, Fabbri-Destro M, Vanduffel             expectation-maximization algorithm. IEEE Trans Med Imaging
      W, Rizzolatti G, Orban GA (2009) The representation of tool use in           20:45-57.
      humans and monkeys: common and uniquely human features. J
      Neurosci 29:11523-11539.
           Pulvermuller F (2005) Brain mechanisms linking language and
      action. Nat Rev Neurosci 6:576-582.
           Pynte J, Besson M, Robichon F-H, Poli J (1996) The time-course
      of metaphor comprehension: An event-related potential study. Brain
      and Language 55:293-316.
           Ramayya AG, Glasser MF, Rilling JK (2009) A DTI
      Investigation of Neural Substrates Supporting Tool Use. Cerebral
      cortex (New York, NY : 1991):507-516.
           Raposo A, Moss HE, Stamatakis EA, Tyler LK (2009)
      Modulation of motor and premotor cortices by actions, action words
      and action sentences. Neuropsychologia 47:388-396.
           Rastle K, Harrington J, Coltheart M (2002) 358,534 nonwords:
      The ARC Nonword Database. Quarterly Journal of Experimental
      Psychology Section a-Human Experimental Psychology 55:1339-
      1362.
           Saad ZS, Glen DR, Chen G, Beauchamp MS, Desai R, Cox RW
      (2009) A new method for improving functional-to-structural MRI
      alignment using local Pearson correlation. Neuroimage 44:839-848.
           Saygin AP (2007) Superior temporal and premotor brain areas
      necessary for biological motion perception. Brain 130:2452-2461.
           Saygin AP, McCullough S, Alac M, Emmorey K (2010)
      Modulation of BOLD Response in Motion-sensitive Lateral
      Temporal Cortex by Real and Fictive Motion Sentences. Journal of
      cognitive neuroscience 22:2480-2490.
           Schmidt GL, Seger CA (2009) Neural correlates of metaphor
      processing: the roles of figurativeness, familiarity and difficulty. Brain
      Cogn 71:375-386.
           Seidenberg M, Tanenhaus MK, Leiman JL, Bienkowski M (1982)
      Automatic access of the meanings of ambiguous words in contex:
      Some limiations of knowledge-based processing. Cognitive
      Psychology 14:470-519.
           Sporns O, Honey CJ, Kotter R (2007) Identification and
      classification of hubs in brain networks. PLoS ONE 2:e1049.
           Squire LR, Stark CE, Clark RE (2004) The medial temporal lobe.
      Annu Rev Neurosci 27:279-306.
           Talairach J, Tournoux P (1988) Co-planar Stereotaxic Atlas of
      the Human Brain. New York: Thieme Medical.

Journal of Cognitive Neuroscience                                                                                In press                          12
You can also read