RAAI Summer School 2019 - Motivational Systems in Cognitive Architectures DCA-FEEC-UNICAMP

RAAI
 Summer School 2019

  Motivational Systems in
  Cognitive Architectures

          Ricardo Gudwin
        DCA-FEEC-UNICAMP

         gudwin@unicamp.br
http://www.dca.fee.unicamp.br/~gudwin

   Copyright@2019   DCA-FEEC-UNICAMP

Introduction

                                            Cognitive Architecture
 Intelligent Agents
    Systems capable of sensing
     an environment, creating a
     model of it and acting
     intelligently on it

              Perception
                                  Sensors

Environment
                                            Agent

              Actuation    Actuators

The Problem: The Control of
Intelligent Autonomous Agents

Introduction

 Cognitive Architecture
    General purpose computational architectures providing the basic
     infrastructure to the construction of artificial agents
    Cognitive models of human mind
       • Implementing computational versions of cognitive abilities, such as
         sensing, perception, attention, emotions, memory, decision-making,
         reactive and deliberative behavior, motivations, learning and in some
         cases consciousness, language, meta-cognition and social cognition
    40 years of Cognitive Architectures
       • 4CAPS, 4D/RCS, ACT-R, AIS, Apex, ART, Atlantis, BECCA, biSoar,
         CERA-CRANIUM, Chrest, Clarion, CogNet, CogPrime, CoJACK,
         Copycat, Disciple, DUAL, Emotion Machine, Epic, ERE, FORR, Gat,
         GLAIR, GMU-BICA, Guardian, H-Cogaff, Homer, HTM, Icarus,
         Imprint, Leabra, LIDA, MAX, MECA, NARS, Nexting, Omar,
         Pogamut, Polyscheme, PRODIGY, PRS, Psi-Theory, R-CAST, RALPH-
         MEA, Recommendation Architecture, REM, SAL, Soar, Society of
         Mind, Subsumption architecture, Teton, Theo, Tosca, Ymir

CLARION

LIDA

MECA

SOAR

Introduction

 Kinds of Architectures
    Symbolic (Rule-based)
    Connectionist (Neural Network)
    Hybrid
 Software Implementations
    Framework
    Toolkit
 Cognitive Cycle
    Cycle starting from sensory information and finishing in
     actuations in the environment
 Behavior Generation
    Sequence of processes within the cognitive cycle which ends
     generating the final agent’s behavior at the environment

Peirce’s Modes of Being

 Firstness
    is the mode of being of that which is such as it is, positively and without reference
     to anything else (CP 8.328; 1.295). The idea of First is predominant in the ideas of
     freshness, life, freedom (CP 1.302), novelty, creation, originality, potentiality,
     randomness.
 Secondness
    is the mode of being of that which is such as it is, with respect to a second but
     regardless of any third (CP 8.328; 1.296). The idea of second is predominant in
     the ideas of causation and of static force (CP 1.325), comparison, opposition,
     polarity, differentiation, existence (opposition to everything else).
 Thirdness
    is the mode of being of that which is such as it is, in bringing a first and second
     into relation to each other (CP 8.328;1.297). The idea of third is predominant in
     the ideas of generality, infinity, continuity, diffusion, growth, intelligence (CP
     1.340), meaning, mediation and representation.

Applying Peirce’s Universal
                Categories to Behaviors

 Kinds of Behaviors
    Original Behavior (firstness)
    Reactive Behavior (secondness)
    Motivated Behavior (thirdness)
 Original Behaviors
    Random actions
    Random modifications of already known actions
    Random combinations of already known actions
 Reactive Behaviors
    Depends only on sensory inputs, to what they react
       • Behavior = function(sensory inputs)
    Don’t have internal states
    Are “blind” to what they cause in the world (mechanical reaction)

Motivated Behavior

 Motivated Behavior
    Are not randomly generated behaviors neither pure reactions to
     sensory inputs
    Have a “finality”, a future to be reached, a “goal”
 Complex discussion in Philosophy and Biology
    Aristotle’s Final Cause, Teleology, Teleonomy
 Cybernetics
    Rosenblueth, A., Wiener, N., & Bigelow, J. (1943). Behavior,
     purpose and teleology. Philosophy of science, 10(1), 18-24.
       • Purposeful action: directed to the achievement of a goal – a final
         condition in which the object affected by the action reaches a
         predetermined condition
       • Negative Feedback Loop Control System
    Bratman, M. (1987). Intention, plans, and practical reason.
       • Plans are created before their execution

Motivated Behavior

                                  Thirdness

            Present’s         -               +   Desired Future’s
          Representation                          Representation

Sensory
 Inputs              Distance between Current                 Actuator
                    Present and Desired Future                Outputs

                       Reactive Behavior

                           Environment

Motivational Systems

 Motivational Systems
    Module within a Cognitive Architecture responsible for
     determining the behavior of intelligent agents, based on cognitive
     models of human motivations
 Hull’s Drive Reduction Approach
    Based on the Motivation Theory from Clark Hull
    Identifies the origin of behavior in a set of internal needs,
     ontogenetically inherited, providing motivation in the behavior
     generation

                                             Drive Reduction
                                      (provides needs satisfaction)
   NEEDS              DRIVES

                                                 AUTOMATIC
                  Needs Degree                   BEHAVIOR
                 of Dissatisfaction

Motivational Systems

 Maslow and the Hierarchy of Needs

Motivational Systems

 The Problem in Maslow’s Proposal
    Arguments against the universality of the hierarchy

                                       Source: (Kenrick et.al. 2010)

Motivational Systems

 Maslow’s Considerations to be used
    As the number of needs increase, more difficult it is to find
     actions simultaneously satisfying many needs
    Needs form a hierarchy, where some needs might have priority
     over others
       • Needs with a bigger priority are satisfied first
 Points still requiring investigation
    There might be unique hierarchies depending on the individual ?
     (Free Will)
    Hierarchies might change during existence ?
    Is it possible to create new needs along the creature’s experience
     ?

Motivational Systems

 Impulse Theory (Bausmeister 2016)
    Impulses are particular desires to perform particular behaviors on
     particular occasions
    Contrary to drives, impules have a beginning and an end
 Impulses
    Might imply an evolution of Hull’s standard drive mechanism:
     instead of drive reduction behaviors, this mechanism generates
     corrective impulses, which, in being satisfied, reduce the
     corresponding drive
    Impulses are dependent on drives
    Impulses depend on a “plan” for their satisfaction
    Impulses from different drives compete to each other to be
     selected and executed

Motivational Systems

The Liking – Wanting
                  Dichotomy

 Wanting
    Happens in the present, but is directed to the future
    Specify a possible future to be reached
    Related to the drives/impulses mechanism
 Liking
      Hedonic dimension – pleasure / displeasure
      Pleasure or Pleasures ? One or more dimensions ?
      Uniquely related to present, right now !
      Related in some sense to a kind of emotional mechanism
      Related to the temporal difference on drives (Δdrive, or ∂drive/∂t)
        • Evolution further introduced specialized dimensions
        • Goal: to serve as reinforcers in the drive reduction action learning

Models for Emotions

 Whare are Emotions ?
    How emotions relate to motivations ?
    Big confusion in the literature !!!
 Etymology
    Emovere – ex + movere = to put out
 Origins
    Distinction between active and passive, action and passion
    The concept of “passion” was gradually refined to reach the
     modern concept of emotion, used in psychology
    From “passion” as a mere passivity to something external to the
     mind, the concepts of feeling, pleasure/displeasure as appraisive
     reactions to possible courses of action
    From mere “passivity”, emotions become an action of mind
     guiding rational/intelligent behavior

Models for Emotions

 The Modern Concept of Emotion (Dixon 2012)
    Appears just in XIX century, as an abstraction to different
     concepts as e.g. appetites, passion, affect, feelings
    Created by the Scottish thinker Thomas Brown (1778-1820)
 Main Perspectives
    Darwin: emotions as the result of evolutionary process –
     behavioral expressions constructed during evolution
    William James: emotions as affects in the body, causing feelings
     (Damásio)
    Cognitive (Arnold 1960): Emotions as appraisals
    Social-constructivist: emotions as the cultural byproduct of
     learning social rules, caused by the interaction among human
     beings

Models for Emotions

 The OCC Model (Ortony, Clore, Collins)
    Appraisal / Arousal Dychotomy
    Emotions are inner states with different intensities, related to
     affects, i.e. appraisal reactions to situations considered to be
     good or bad, related to events, agents or environment objects
    The OCC Model specifies 22 emotions, implemented in terms of
     local and global variables, with their values determined from rules
     applied to sensory information
    Emotional parameters:
       • Emotion Triggering Potential
       • Emotion Triggering Threshold
       • Emotion Intensity

Models for Emotions

 Damásio and the Somatic Marker
      Rationality with Emotions
      The need of a body in the process
      Distinction between Emotions and Feelings
      Emotion
        • A body modification due to an external stimuli
    Feeling
        • Cognitive counter-part of an emotion, appearing after the
          consciousness perception of the emotion
    Primary Emotions
        • Experienced since childhood – innate mechanisms
        • Fear, joy, sadness, anger, etc.
    Secondary Emotions
        • Learned along the life-cycle of an individual
        • Jealousy, Guilt, Pride, etc.

Models for Emotions

 Simon and Sloman: Emotions as Alarms
    Process related to the motivational mechanism
    Simon (1967): Emotions as “interruptions”
      • Critical situations might change creature’s goals, causing an
        interruption
      • E.g.: the arousal of a predator, might cause the interruption of an
        animal current behavior and the generation of new goals, in order to
        escape from danger
      • Interruptions might be caused by an imminent danger or a
        remarkable opportunity
    Sloman
      • Emotions as attention filters (AFP:Attention Filter Penetration Theory)
      • Differently from an “interruption”, presuposes a momentaneous
        disruption, with the possibility of returning to its past goal
      • Alarms: triggered by danger or by remarkable opportunities

Models for Emotions

 Cañamero and the Modulation of Motivations
    Emotional System where emotions work as modulators, with the
     definition of synthetic hormones affecting homeostatic processes
     associated to the needs of a creature
    Motivational Mechanism:
      • Modifications in a creature’s body variables determine drives,
        triggering an action aiming at the reduction of these drives (still
        without emotions)
    Emotional Mechanism is independent from the Motivational one,
     but affects it
      • Critical situations trigger the release of hormones which cause a
        modulation in the drives, making some of them more intense while
        others are attenuated.

Emotions: A Unified View

 Independent Motivational and Emotional Systems
    Motivational System based on Drives AND Impulses
    Emotions as a result of an evolutionary process, where the
     internal motivations of a creature are communicated to other
     creatures, by means of their expression in bodily modifications
 Emotions
    Are not necessary for the regular working of the motivational
     system, but are affected by it
    Just appear in the evolutionary scale in animals with a limbic
     system (birds and mammals)
    Strategy to communicate emotional states can be useful in social
     situations, like collaboration, cooperation and competition

Emotions: A Unified View

 Source of confusion between Motivations and Emotions
    As emotions (in our view) are messages representing the
     motivational states, they were misleaded by the own motivational
     states
 And what about the Hedonic States (feelings)
  representing “Liking” ?
    They are used as “rewards” (or reinforcements) in the learning of
     actions required for drive reduction in the motivational system
    Our hypothesis is that there might exist different kinds of
     rewards, associated to the different creature’s needs

Emotions: A Unified View

 Evolution of the Motivational System

Emotions: A Unified View

 Evolution of the Motivational System

Conclusions

 Summary of our Unified View
    Model of “Experience” as SDR (Sensors,Drives, Rewards)
    Experience integrated in the form of percepts embedding SDR
    Motivated actions learning using (reactive + original) actions plus
     rewards (reinforcement learning)
    Use of affordances for the generation of impulses able to satisfy
     the landscape of drives
    Choice among a repertoire of impulses based on a hierarchy of
     needs (unique and dynamic to each creature, differently from
     Maslow)
    Schedule of motivated actions to generate plans
    Emotions as a mechanism of communication of motivational
     states, allowing the appearance of complex social behavior