MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM

 
MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM
MAX: Collaborative Unmanned Air Vehicles
           Recent Progress at UM

               Anouck Girard & Pierre Kabamba
    Baro Hyun, Justin Jackson, Jonathan Las Fargeas, Jinwoo Seok

                  Department of Aerospace Engineering
                        University of Michigan
                         Ann Arbor, Michigan

                         September 2012

ARCLAB (UM)          Collaborative Unmanned Air Vehicles   September 2012   1 / 60
MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM
Introduction

ARCLAB in Numbers

                       Graduated Students:
Current People:
                       4 PhD:
    3 PhD              Justin Jackson, 2012, Llamasoft.
    2 MS               Baro Hyun, 2011, Hyundai Motors.
                       Christopher Orlowski, 2011, US Army, TACOM/TARDEC.
                       Andrew Klesh, 2009, JPL.
Publications:
    10 peer reviewed   6 MS:
    journal articles   Zahid Hasan, 2012, Raytheon Company.
    accepted           Calvin Park, 2012, North American Bancard.
                       Clarence Hanson, 2011, Rockwell Collins.
    41 conference
                       Jonathan White, 2008, US Coast Guard.
    papers accepted
                       John Baker, 2007, Systems Engineering, HDT Robotics.
    2 book chapters    Amir Matlock, 2007, JHU Applied Physics Lab, Ballistic
    published
                       Missile Defense Test and Evaluation Group.

     ARCLAB (UM)        Collaborative Unmanned Air Vehicles   September 2012   2 / 60
MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM
New Members of the ARCLab

Moritz Niendorf

                         Work Experience and Education
                               02/2011 - 07/2012: DLR (German Aerospace
                               Center) - Department for Unmanned Aircraft
                               11/2010: Diploma in Aerospace Engineering -
                               University of Stuttgart, Germany
                               09/2008 - 05/2009: Exchange Student -
                               Aeronautical and Astronautical Engineering -
                               Purdue University

                         Research Interests
                             Mission and path planning for unmanned
                             aircraft under motion constraints.
                               Task assignment for unmanned aircraft
                               considering path planning aspects.

    ARCLAB (UM)              Collaborative Unmanned Air Vehicles   September 2012   3 / 60
MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM
New Members of the ARCLab

Dave Oyler

                      Work Experience and Education
                      Texas A&M University
                            05/2012: B.S. Electrical Engineering
                      NASA Johnson Space Center
                            05/2012-08/2012: Robotic Operations
                            01/2011-08/2011: Robotic Systems Technology
                            01/2010-05/2010: Integrated Communications
                            06/2008-08/2008: Electromagnetic Systems

                      Research Interests
                          Robotic planetary exploration
                            Cooperation of heterogeneous robotic teams

   ARCLAB (UM)              Collaborative Unmanned Air Vehicles   September 2012   4 / 60
MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM
Overview

Mixed-Initiative Nested Classification

Themes
   From ... Sensor, To ... Information
    Trusted highly-autonomous decision-making systems

Objectives
    Improve the classification performance in mixed-initiative system

     ARCLAB (UM)          Collaborative Unmanned Air Vehicles   September 2012   5 / 60
MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM
Overview

Persistent Visitation, Detection, and Capture
Themes
   Coherent change detection for persistent surveillance systems
    Increased operational efficiency and autonomy

Objectives/Results
    Formulation of the Persistent Visitation problem for a single UAV.
    Proof of the existence of periodic paths for single UAVs performing
    persistent visitation.
    Complete algorithm to find minimal cost paths when fuel constraints
    are considered.
    Formulation of the Persistent Visitation, Detection, and Capture
    problem for multiple UAVs.
    Algorithm to generate paths for UAVs that perform persistent
    visitation while attempting to image intruders.

Potential Impacts
     ARCLAB (UM)         Collaborative Unmanned Air Vehicles   September 2012   6 / 60
MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM
Overview

             Mixed-Initiative Nested Classification
                    for n Team Members

      Baro Hyun, Songya Pan, Pierre Kabamba, Anouck Girard
                      Department of Aerospace Engineering
                      University of Michigan, Ann Arbor, MI

                         Annual MACCCS Review

                               September 2012

B. Hyun et al. (UM)               Inverting the ratio         September 2012   7 / 60
MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM
Mixed Initiative Nested Classification

Motivated by military operations

Intelligence, Surveillance, and Reconnaissance missions
                                                                     Objects of interests
                                                                         threat or friend
                                                                     Unmanned aerial vehicles
                                                                     (UAVs)
                                                                         carry a suite of sensors and a
                                                                         communication device
                                                                     Human operators
                                                                         direct the UAVs
                                                                         inspect data and make
                                                                         classification decisions
                                                                     Need high quality classification
                                                                     decisions in the presence of
                                                                     uncertainties

    B. Hyun et al. (UM)                        Inverting the ratio                   September 2012   8 / 60
Mixed Initiative Nested Classification

Motivated by military operations

Intelligence, Surveillance, and Reconnaissance missions
                                                                     Objects of interests
                                                                         threat or friend
                                                                     Unmanned aerial vehicles
                                                                     (UAVs)
                                                                         carry a suite of sensors and a
                                                                         communication device
                                                                     Human operators
                                                                         direct the UAVs
                                                                         inspect data and make
                                                                         classification decisions
                                                                     Need high quality classification
                                                                     decisions in the presence of
                                                                     uncertainties

    B. Hyun et al. (UM)                        Inverting the ratio                   September 2012   8 / 60
Mixed Initiative Nested Classification

Information overflow

   Multiple views from a wide angle camera (Gorgon Stare)

                        Figure:     from Cummings and Bertuccelli, MAX review 10’

   “... man power requirements to deal with these data are burdensome”
   [AF/ST, Report on Tech Horizon ’10]
  B. Hyun et al. (UM)                        Inverting the ratio                    September 2012   9 / 60
Mixed Initiative Nested Classification

Objectives of the research
Global Objective
    Improving the classification performance in mixed-initiative systems

Year 2-3 (2008-2010)
    A mobile classifier taking multiple measurements while seeking
    maximum information
    Discrete Event System (DES) modeling of human operator in
    classification task
    A mobile classifier making sequential decisions while seeking
    minimum risk

Year 4-5 (2010-2012)
    Information-classification performance, classification mechanism by
    thresholding, team classification
    Inverting the human-to-machine ratio
    B. Hyun et al. (UM)                        Inverting the ratio   September 2012   10 / 60
Mixed Initiative Nested Classification

Motivational questions
      How do we leverage the complementary strengths of human/machine
      collaboration in a mixed-initiative system?
      How can we invert the current human-to-machine ratio in the ISR
      mission?

Mixed-initiative system
  1   Classifiers with workload-independent performance (machines)
  2   Classifiers with workload-dependent performance (humans)

  - “First-order” models to capture the features of machines and
    humans, respectively

      B. Hyun et al. (UM)                        Inverting the ratio   September 2012   11 / 60
Mixed Initiative Nested Classification

Motivational questions
      How do we leverage the complementary strengths of human/machine
      collaboration in a mixed-initiative system?
      How can we invert the current human-to-machine ratio in the ISR
      mission?

Mixed-initiative system
  1   Classifiers with workload-independent performance (machines)
  2   Classifiers with workload-dependent performance (humans)

  - “First-order” models to capture the features of machines and
    humans, respectively

      B. Hyun et al. (UM)                        Inverting the ratio   September 2012   11 / 60
Mixed Initiative Nested Classification

Technical relevance to the A.F.

Air Force relevance (Highlights from [Tech. Horizon])
    From ... Sensor, To ... Information
           “The volume of sensor data from current-generation sensors ... has
           become overwhelming, as manpower requirements to deal with these
           data have placed enormous burden on the Air Force.”
           “... systems that can reliably make wide-ranging autonomous decisions
           at cyber speeds to allow reactions in time-critical roles far exceeding
           what humans can possibly achieve.”
    Grand Challenges for Air Force S&T
           Challenge #2: Trusted highly-autonomous decision-making systems
           “... demonstrate technologies that enable current human-intensive
           functions to be replaced, in whole or in part, by more highly
           autonomous decision-making systems, ...”

    B. Hyun et al. (UM)                        Inverting the ratio   September 2012   12 / 60
Mixed Initiative Nested Classification

Literature survey

   Classification
          Theory of classification [Gupta and Leu ’89, Widrow ’63]
          Applications of classification [Jain et al. ’00, Chang et al. ’06]
          Classification with human inputs [Cebron and Berthold ’06, Holsapple
          et al. ’08]
   Statistical decision making
          Hypothesis testing [Lehmann and Romano ’10]
          Bayesian decision theory [Berger ’85]
          Sequential Probability Ratio Test (SPRT) [Wald ’45]
   Human-machine collaboration
          Inverting the ratio [Cummings et al. ’08-’10]
          Adjustable autonomy [Goodrich et al. ’09-’10]

   B. Hyun et al. (UM)                        Inverting the ratio   September 2012   13 / 60
Mixed Initiative Nested Classification

Technical contributions
    We extended our work on mixed-initiative nested thresholding, a
    classification architecture that uses a primary workload-independent
    classifier and a secondary workload-dependent classifier, for a general
    number n of classifiers in the architecture, formally pose the problem,
    and solve it.
    We identified the optimal ratio of mixed-initiative team members, the
    corresponding minimal probability of misclassification, and the
    individual workload applied to the workload-dependent classifier as a
    function of the total workload applied to the architecture.
    We performed a sensitivity analysis of the aforementioned results with
    respect to the peak performance of the workload-dependent classifier.

    B. Hyun et al. (UM)                        Inverting the ratio   September 2012   14 / 60
Mixed Initiative Nested Classification

Recent achievements by numbers (year 5)
    1 accepted and 3 submitted journal papers
    7 accepted or submitted conference papers
    Co-organizer and session chair for an invited session on “information
    collection and decision making” for ACC’12

    B. Hyun et al. (UM)                        Inverting the ratio   September 2012   15 / 60
Theoretical background

What is a classifier?

    A decider D is a deterministic mapping defined on a set of data into
    truth values
                           D : {data} → {T, F }
    A classifier C is a decider with the domain of the mapping being a
    specific realization of a random variable
    The difference between a decider and a classifier is that the latter
    accounts for the randomness of the data being classified

Important parameters
    Processing of the data requires two abilities
       1   recognizing truth out of truth (rate of true positives)
       2   recognizing falsehood out of falsehood (rate of true negatives)
    Characterized by two independent parameters σT and σF

    B. Hyun et al. (UM)                  Inverting the ratio    September 2012   16 / 60
Theoretical background

Theoretical background - Probabilistic modeling

     Let X ∈ {T, F } be the object category variable
     Let Y ∈ {Y1 , Y2 } be the object property variable
The likelihood is modeled by the following conditional probabilities,

                          P (Y = Y2 |X = T ) = σT ,
                          P (Y = Y1 |X = F ) = σF ,
                          P (Y = Y1 |X = T ) = 1 − σT ,
                          P (Y = Y2 |X = F ) = 1 − σF ,                         (1)

where σi ∈ [0.5, 1], i ∈ {T, F }.
     u: proportion of sub-population T among the entire population

    B. Hyun et al. (UM)                  Inverting the ratio   September 2012   17 / 60
Theoretical background

Theoretical background - Maximum likelihood classification
Bayes rule
Provides posterior probability of the object category on the basis the
object property

                                         P (Y = {Y1 , Y2 }|X = T )P (X = T )
   P (X = T |Y = {Y1 , Y2 }) =                                                     (2)
                                                 P (Y = {Y1 , Y2 })

    Let Os ∈ {T, F } be the decision variable

Likelihood-ratio rule
Makes classification decisions by comparing the posterior probability
                                    (X=T |Y ={Y1 ,Y2 })
                            T if PP (X=F
                        (
                                         |Y ={Y1 ,Y2 }) > λ
                  Os =            P (X=T |Y ={Y1 ,Y2 })                            (3)
                            F if P (X=F |Y ={Y1 ,Y2 }) ≤ λ.

where λ ∈ R.
    B. Hyun et al. (UM)                  Inverting the ratio      September 2012   18 / 60
Theoretical background

Theoretical background - Classification performance

Probability of misclassification
    The performance measure is the probability of misclassification, Pm ,

                 Pm = P (Os = T ∧ X = F ) + P (Os = F ∧ X = T )                 (4)

    Pm is the sum of probabilities of two faulty outcomes: the probability
    of false positives and the probability of false negatives

    B. Hyun et al. (UM)                  Inverting the ratio   September 2012   19 / 60
Theoretical background

Thresholding problem
Assumptions
    A continuous measurable property w ∈ R
    Object categories are known a priori
    Distribution of w for each object category is known a priori

                 σF                         σT                                       σF                               σT
                          FN   FP                                                         FN                 FP
                                                      w                                                                       w	
  
       €     F                 €        T                                €   F	
                     €
                                                                                               Unknown	
          T	
  
                                                                             €                    €
            €         €
  Figure: Dichotomous thresholding                                Figure: Trichotomous thresholding
    B. Hyun et al. (UM)                            Inverting the ratio                         September 2012              20 / 60
Theoretical background

Mixed-initiative nested thresholding

                                                   Start                  Prior
                                                                             u

                                          Workload-Independent
                                          Trichotomous Classifier

                                                            T, F     Pm
                                                                                  Yes
                         undecidables   W                         Good?

                                                                     No

                                           Workload-Dependent
                                          Dichotomous Classifier

                                              T, F Pm
                                                   End

   B. Hyun et al. (UM)                      Inverting the ratio                         September 2012   21 / 60
Theoretical background

Mixed-initiative nested thresholding

                                                                                    Start                 Prior
                  "F                             "T                                                          u
                       FN              FP
                                                          !"
     !       !"                 !               #"
             !              $%&%'(%"
                              !                                            Workload-Independent
                                                                           Trichotomous Classifier                            Workload

                                                                                             T, F    Pm
                                                                                                                  Yes
                                                      undecidables        W                     Good?

                                                                                                     No

                                                                            Workload-Dependent
                                                                           Dichotomous Classifier

                                                                               T, F Pm
              "F                                "T                                  End
                       FN      FP
                                                      w
    !    F                     !            T
         !         !
                                                                                                                               Workload

   B. Hyun et al. (UM)                                                        Inverting the ratio                       September 2012    22 / 60
Theoretical background

Problem formulation
Workload
We define a workload variable, W ∈ [0, 1], with 0 indicating idle and 1
                                                  2 2
indicating fully loaded. Let fi (w) = ai e−(w+bi ) /ci with i ∈ {T, F }, then
the workload variable is defined as
                         Z τ2
                    W =       ufT (w) + (1 − u)fF (w)dw.                    (5)
                               τ1

Note that the range of W is [0, 1] for any τ1 and τ2 .
     The region of indecision, i.e., [τ1 , τ2 ], of the primary trichotomous
     classifier determines the workload applied to the secondary classifier.

Optimization problem
                                                2
                                           min Pm ,
                                           τ1 ,τ2

subject to some inequality constraints.
    B. Hyun et al. (UM)                  Inverting the ratio   September 2012   23 / 60
Theoretical background

Comparison of performance
        0             Minimal probability of misclassification vs. classifiability                                        Workload vs. Classifiability
       10                                                                                            0.5

        −1
       10                                                                                           0.45

        −2                                                                                           0.4
       10

                                                                                                    0.35
        −3
       10

                                                                                                     0.3
        −4
 P*m

       10

                                                                                                W
                                                                                                    0.25
        −5
       10
                                                                                                     0.2

                        2
        −6
       10              Pm with τ0 = [mT, mF]
                                                                                                    0.15
                       P1m
                                                                                                                                                         W with τ0 = [mT, mF]
        −7
       10                                                                                            0.1

        −8
       10 −1                 0                       1                          2           3
                                                                                                    0.05 −1           0                  1                     2                  3
         10            10                         10                         10           10           10            10               10                    10                  10
                                                  Cl                                                                                  Cl

(a) The minimal probability of misclassi-                                                                     (b) Workload vs. classifiability
fication vs. classifiability. The blue solid
line indicates the mixed-initiative nested
thresholding while the red dashed line in-
dicates the dichotomous thresholding.

Figure: The minimal probability of misclassification and the workload for the
mixed-initiative nested thresholding as a function of the classifiability
             B. Hyun et al. (UM)                                                     Inverting the ratio                                     September 2012                     24 / 60
Inverting the ratio

Inverting the ratio
                          M1                             M1     M2    ···     M10

            H1      H2    H3    ···        H10                       H1

Figure: Mixed-initiative nested thresholding with more than two team members.
(M denotes a workload-independent classifier and H denotes a
workload-dependent classifier)

Point of interest
Given a workload W provided by a workload-independent classifier (M ),
     What’s the optimal ratio of the mixed-initiative team members?
     What’s the reachable performance?
     What’s the individual workload applied to each workload-dependent
     classifiers (H)?
    B. Hyun et al. (UM)                   Inverting the ratio             September 2012   25 / 60
Inverting the ratio

Setup
                         1    1
    Ratio variable n ∈ { m , m−1 , · · · , 12 , 1, 2, · · · , m}
           the ratio of the number of workload-dependent classifiers to the
           number of workload-independent classifiers in the system with m ∈ N.
           n = 0.1 means a single workload-dependent classifier (human) and 10
           workload-independent classifiers (machines).
    Total workload Wt ∈ [0, ∞)
           the workload applied to the whole secondary layer in the architecture
                                                              Wt
    Individual workload Wn ∈ [0, 1], Wn =                     n
           the workload applied to the individual classifier in the secondary layer
           assume uniform distribution of Wt to the secondary layer

Problem formulation
The objective of the problem is to minimize the probability of
misclassification by choosing the ratio number n, i.e.,
                                         2
                                    min Pm (Wt , n).
                                      n

    B. Hyun et al. (UM)                 Inverting the ratio        September 2012   26 / 60
Inverting the ratio

Analytical results

Theorem
Suppose that Wt is fixed and let n∗ = arg minn Pm 2 (W , n). The optimal
                                                      t
       ∗
ratio n is monotonically increasing with respect to Wt , specifically that
n∗ = 2Wt .

†Proof by the necessary condition for optimality

Corollary

                                    lim Wn∗ = 0.5
                                 Wt →∞

    B. Hyun et al. (UM)               Inverting the ratio   September 2012   27 / 60
Inverting the ratio

                           2                                                                                                                            0.06                                                                                                                      0.7

                                                                                                                                                                                                                                                                              0.65
                                                                                                                                                        0.05

                                                                                                Minimal probability of misclassification (Pm
                                                                                                                                           2
                                                                                                                                             *(n*))
                                                                                                                                                                                                                                                                                  0.6

                                                                                                                                                        0.04

                                                                                                                                                                                                                                                   Individual workload (Wn)
                                                                                                                                                                                                                                                                              0.55
Optimal ratio (n*)

                                                                                                                                                        0.03                                                                                                                      0.5
                           1

                                                                                                                                                                                                                                                                              0.45
                                                                                                                                                        0.02

                                                                                                                                                                                                                                                                                  0.4

                         0.5                                                                                                                            0.01
                                                                                                                                                                                                                                                                              0.35
                        0.33
                        0.25
                         0.2 −1                                                             0
                                                                                                                                                          0
                           10                                                              10                                                             0.1           0.2     0.3   0.4     0.5        0.6     0.7         0.8       0.9     1                                   0.1       0.2    0.3    0.4     0.5        0.6     0.7         0.8       0.9    1
                                                      Total workload (W)                                                                                                                    Total workload (W)                                                                                                   Total workload (W)

(a) Optimal ratio (abscissa (b) Minimal probability of                                                                                                                                                                                                                                  (c) Individual workload
in logarithmic scale)       misclassification
                         10
                                                                                                                                                         0.045                                                                                                                    0.65

                           9
                                                                                                                                               *(n*))

                                                                                                                                                          0.04
                                                                                                                                                                                                                                                                                   0.6
                           8
                                                                                                  Minimal probability of misclassification (Pm
                                                                                                                                             2

                                                                                                                                                         0.035
                           7                                                                                                                                                                                                                                                      0.55

                                                                                                                                                                                                                                                       Individual workload (Wn)
                                                                                                                                                          0.03
   Optimal ratio (n*)

                           6
                                                                                                                                                                                                                                                                                   0.5
                                                                                                                                                         0.025
                           5

                                                                                                                                                          0.02                                                                                                                    0.45
                           4

                                                                                                                                                         0.015
                           3
                                                                                                                                                                                                                                                                                   0.4

                           2                                                                                                                              0.01

                                                                                                                                                                                                                                                                                  0.35
  X: 0.2 1                                                                                                                                               0.005
  Y: 0.3333

                           0      X: 0.3                                                                                                                        0
               X: 0.10            0.5
                                  Y: 0.5   1   1.5    2      2.5     3     3.5   4   4.5    5                                                                       0     0.5     1   1.5    2      2.5     3          3.5         4     4.5   5                                         0    0.5    1     1.5    2      2.5     3          3.5         4    4.5   5
               Y: 0.2                                Total workload (W)                                                                                                                     Total workload (W)                                                                                                   Total workload (W)

                                    (d) Optimal ratio                                           (e) Minimal probability of                                                                                                                                                              (f) Individual workload
                                                                                                misclassification

                                  B. Hyun et al. (UM)                                                                                                                           Inverting the ratio                                                                                                       September 2012                                     28 / 60
Conclusion

Conclusion

Implications
     Guidelines to design a mixed-initiative system that autonomously
     determines the optimal human-to-machine ratio

Relevant publications - available in MACCCS Ctools website
  1   B. Hyun, M. Faied, P. Kabamba, A. Girard, Mixed-Initiative Nested Classification for n Team Members, IEEE
      Conference on Decision and Control, Maui, HI, 2012.
  2   B. Hyun, M. Faied, P. Kabamba, A. Girard, Optimal Multivariate Classification by Linear Thresholding, American
      Control Conference, Montreal, Canada, 2012. (invited paper)
  3   B. Hyun, M. Faied, P. Kabamba, A. Girard, Optimal Classification by Mixed-Initiative Nested Thresholding, IEEE
      Transactions on Systems, Man, and Cybernetics - Part A, 2012, Submitted.
  4   B. Hyun, M. Faied, P. Kabamba, A. Girard, On Minimizing Classification Error by Maximizing Information, IEEE Signal
      Processing Letters, 2012, Submitted.

      B. Hyun et al. (UM)                          Inverting the ratio                        September 2012           29 / 60
Conclusion

Optimal strategies for team classification
Problem
    Given: a number of decision makers, their individual performances
    and prior information.
    Find: the best fusion rules under different decision structures with
    respect to a performance metric.

       A1            B1        B2             B3
                                                   A1               B1   A2              B2

                A2
                                                               A3                   B3
                          A3

               (g) Incremental pairing                  (h) Tournament-like pairing

    B. Hyun et al. (UM)                  Inverting the ratio                  September 2012   30 / 60
Conclusion

                                                  The misclassification of four−team classifier with incremental pairing                                                                    The misclassifaction of Four−team classifier with Touramnet−like Pairing
                         0.35                                                                                                                                           0.35
                                                                                                                                 Fused Result for A2                                                                                                                    Fused Result for B3
                                                                                                                                 Fused Result for A3                                                                                                                    Fused Result for A3
                                                                                                                                 Final Fused Result                                                                                                                     Final Fused Result
                          0.3                                                                                                                                            0.3

                         0.25                                                                                                                                           0.25

                          0.2                                                                                                                                            0.2
            Minimal Pm

                                                                                                                                                           Minimal Pm
                         0.15                                                                                                                                           0.15

                          0.1                                                                                                                                            0.1

                         0.05                                                                                                                                           0.05

                           0                                                                                                                                              0
                                0   0.1     0.2           0.3         0.4          0.5          0.6         0.7            0.8          0.9            1                       0    0.1   0.2        0.3         0.4          0.5         0.6         0.7         0.8          0.9            1
                                                                                    u                                                                                                                                          u

                                          (i) Incremental pairing                                                                                                                  (j) Tournament-like pairing

       We propose a decision structure that exploits a moderator, i.e., an
       entity that exploits Bayesian inference from individual classifiers’
       decisions and makes final decisions based on maximum likelihood
       classification.
       Two pairing schemes, i.e., incremental and tournament-like, are
       proposed and we show that the incremental pairing is the most
       effective decision structure among the proposed ones.
S. Pan, B. Hyun, P. Kabamba, A. Girard, Optimal Fusion Rules in Team Classification under Three Decision Structures,

American Control Conference, Washington, DC, USA, 2013, Submitted.

       B. Hyun et al. (UM)                                                                                                         Inverting the ratio                                                                                                September 2012                              31 / 60
Conclusion

Future work
    Analysis under different performance measures
        - Addressing time-criticality by queueing theory
        - Confidence level
    Kinematic classification (free measurements)
        - Costly kinematic classification (costly measurements)
    Classification with learning
    Strategies for uncertain prior information
    Deceptive strategies

    B. Hyun et al. (UM)            Inverting the ratio            September 2012   32 / 60
Conclusion

                  Automated Classification System
                    for Bone Age X-ray Images

Jinwoo Seok, Baro Hyun, Josephine Kasa-Vubu*, and Anouck Girard
       Department of Aerospace Engineering and Pediatric Endocrinology*
                    University of Michigan, Ann Arbor, MI

                       Annual MACCCS Review

                             September 2012

 J. Seok et al. (UM)       Automated Classification System    September 2012   33 / 60
Introduction

Motivation

                          Importance of Bone Age(BA)
                                  The assessment of growth and pubertal
                                  maturation is central to the practice of
                                  pediatric endocrinology and BA is key
                                  reference
                                  Greulich and Pyle (GP) atlas is a key
                                  clinical indicator in pediatric endocrinology
                                  To determine BA, radiologist compares the
                                  patient’s x-ray to those contained in the
                                  reference atlas and determines which image
                                  in the atlas the patient’s x-ray is closest to
   Hand X-ray Image

   J. Seok et al. (UM)   Automated Classification System      September 2012   34 / 60
Introduction

Literature Review

   There have been attempts at automated BA detection
          CASAS [Tanner ’92]
          Peitka [Pietka et al. ’01]
          BoneXpert [Thodberg et al. ’01 and ’09]
   BoneXpert has been developed recently
          Active Appearance Model (AAM) [Cootes et al. ’01]
          Better performance than previous work [Martin et al. ’09]
          (Root mean square deviation 0.72 years)
   Problems of BoneXpert
          Validating problems
          Clinical Age (CA) and BA relationship is unclear from the publications

   J. Seok et al. (UM)       Automated Classification System   September 2012   35 / 60
Introduction

Original Contributions

                                                                                                                                  More	
  radiographic	
  data	
  
                                                                                                                                                                      i. Create a modified atlas that has
                                              Image	
  morphing	
  

Radiographic	
  data	
  
                                                                                                                                                                         images regularly spaced at
                                                                                                                                                                         three month intervals in the
                                                                                                                                                                         clinically significant ranges
Greulich	
  and	
  Pyle	
  
(1959)	
  

                                                                                                                                 Training	
  
                                                                                                                                                                     ii. Propose a novel Singular Value
                               Thresholding	
  classifier	
  
                                                                                                                                                                         Decomposition (SVD)-based
                                                                      u = 0.5
                                                                                                                                                Bone	
  age?	
  
                                                                                                                                                                         feature extractor to create a
                                      0.04
                                                                                ufT(w), mT = −10, sT = 10

                                     0.035                                      (1−u)fNT(w), mNT = 10, sNT = 15
                                                                                ufT(w)+(1−u)fNT(w)

                                      0.03                                      Optimal Threshold

     Predicted	
                     0.025

     bone	
  age!	
  
                              f(v)

                                      0.02

                                     0.015

                                      0.01

                                     0.005

                                        0
                                       −100   −80   −60   −40   −20     0       20     40      60      80     100
                                                                                                                    Feature	
  extrac=on	
  
                                                                                                                                                                         feature vector out of the
                                                                        v

                                                                                                                                                                         descriptors obtained from SIFT
Schematic overview of the automated classification system                                                                                                            iii. Develop image classifier based
                                                                                                                                                                          on SIFT - SVD

                J. Seok et al. (UM)                                                                                                   Automated Classification System                  September 2012   36 / 60
Technical Section

 Image Feature Extraction

                                                Scale Invariant Feature Transform (SIFT)
                                                       Introduced by David G. Lowe in 1999
                                                       Local-based feature extraction method
                                                       Invariant to scaling and rotation, and
                                                       partially invariant to viewpoint and
                                                       illumination changes
                                                       Algorithm
                                                               Detection of scale-space extrema
                                                               Accurate keypoint localization
                                                               Orientation assignment
100      200    300     400     500       600                  The local image descriptor
      Feature descriptors using VL-SIFT

          J. Seok et al. (UM)                     Automated Classification System      September 2012   37 / 60
Technical Section

Image Feature Extraction

Singular Value Decomposition (SVD)
    Matrix factorization method
    Reduces the size while keeping the characteristics of a matrix
    Given an m × m matrix A, the expression of its SVD is

                                      A = U ΣV T                                       (6)

    where U is an m × m matrix, V is an n × n matrix and Σ is the singular values of matrix
    A which is an m × n non-negative real diagonal matrix.

SIFT - SVD based feature extractor
By applying SVD to the feature descriptors obtained from SIFT, we
produce a novel feature vector for the classifier.

    J. Seok et al. (UM)         Automated Classification System       September 2012   38 / 60
Simulation

Simulation

   Data set
          24 GP female standard images for training: 1 through 27 excluding 13,
          21 and 27, 13 and 27 because of poor image conditions.
          Generated 19 morphing images for validation.
   Classification decision step
          Import images to Matlab
          Apply the SIFT algorithm to get key points and local image descriptors
          Apply SVD to get reduced feature vectors
          Train the neural network
          Validate
          In progress: gathering larger data set for statistical analysis
          Future work: compare Hyun approach to current (neural network)

   J. Seok et al. (UM)       Automated Classification System   September 2012   39 / 60
Simulation

Results

Test result 1, marked with circles
    Classifier works well as most the                                                25

                                                                                              Correct Answer
    answers are closely aligned to the                                               20
                                                                                              Test result1
                                                                                              Test result2

    diagonal line.

                                                       Output (GP standard number)
    Only one result shows radical                                                    15

    misclassification.                                                               10

    Three results showing moderate
                                                                                      5
    errors, and some round-off errors.
Test result 2, marked with crosses                                                    0
                                                                                          0     5            10             15
                                                                                                        Input (GP standard number)
                                                                                                                                     20         25

    Classifier performs less well: There                                                      SIFT - SVD classifier results

    was only one training data per class;
    this is generally not considered
    sufficient to train classifiers. (Proof of
    concept).
    J. Seok et al. (UM)      Automated Classification System                                                   September 2012             40 / 60
Simulation

          Highlights of Other Relevant Research

             Justin Jackson, Eric Sihite, Ricardo Bencatel

                     Annual MACCCS Review

                           September 2012

ARC Lab Team (UM)              Other Research           September 2012   41 / 60
Relevant Research

Highlights of Other Relevant Research

Distributed Task Assignment and Scheduling

VRP Heuristics Comparison

Persistent Flight on Flow Fields

    ARC Lab Team (UM)               Other Research   September 2012   42 / 60
Relevant Research

Task Assignment and Scheduling: Original Contributions

Contributions in two categories
    Centralized minimum-time, precedence-constrained, vehicle routing
    Distributed minimum-time, constrained, task assignment and task
    scheduling

   ARC Lab Team (UM)               Other Research      September 2012   43 / 60
Relevant Research

Centralized Task Assignment and Scheduling

Minimum-time, precedence-constrained vehicle routing
  1   Low complexity algorithm for AFRL-relevant vehicle routing problem
  2   Analysis of algorithm optimality and complexity
  3   Solution quality measurement technique, useful in absence of
      analytical bounds

      Comparison of tabu/2-opt heuristic and optimal tree search method for assignment problems, International Journal of

      Robust and Nonlinear Control, 2011

      A New Measure of Solution Quality for Combinatorial Task Assignment Problems, Conference on Decision and Control,

      2010

      A Combined Tabu Search and 2-opt Heuristic for Multiple Vehicle Routing, Automatic Controls Conference, 2010

      ARC Lab Team (UM)                             Other Research                             September 2012        44 / 60
Relevant Research

Distributed Task Assignment and Scheduling
Minimum-time constrained distributed task assignment and scheduling
  1   Communication-constraints satisfy operational needs
  2   Scheduling constraints express relevant operational constraints
  3   Stochastic Bidding and the OptDNSB Algorithms for assignment and
      scheduling
  4   Correctness, completeness, optimality, complexity characterization
  5   Characterization and utilization of problem separation

      Distributed Constrained Minimum-Time Schedules in Networks of Arbitrary Topology, IEEE Transactions on Robotics,

      2011 (Submitted)

      Communication-Constrained Distributed Assignment on Networks of Arbitrarily Topology, IEEE Transactions on

      Robotics, 2011 (Submitted)

      Communication-Constrained Distributed Assignment, IEEE Conference on Decision and Control, 2011

      Distributed Task Scheduling Subject to Arbitrary Constraints, 18th World Congress of the International Federation of

      Automatic Control (IFAC), 2011

      ARC Lab Team (UM)                              Other Research                             September 2012         45 / 60
Relevant Research

Heuristics Comparison for VRP

  ARC Lab Team (UM)              Other Research   September 2012   46 / 60
Relevant Research

Heuristics Comparison for VRP

  ARC Lab Team (UM)              Other Research   September 2012   47 / 60
Relevant Research

Heuristics Comparison for VRP

  ARC Lab Team (UM)              Other Research   September 2012   48 / 60
Relevant Research

Heuristics Comparison for VRP

   E. Sihite, J. Jackson, A. Girard, VRP Heuristics Comparison, ACC 2013 (Submitted)

  ARC Lab Team (UM)                             Other Research                         September 2012   49 / 60
Relevant Research

Perpetual Flight in Flow Fields

   Extension of UAV endurance
   Inspired by birds behaviors
   Harvest airflow energy

  ARC Lab Team (UM)               Other Research   September 2012   50 / 60
Perpetual Flight in Flow Field

Thermal Soaring
   Models - Chimney & Bubble Thermals
   Observability
   Estimation

         (m) Leaning              Chimney                      (n) Bubble Thermal
         Thermal
  ARC Lab Team (UM)                           Other Research                  September 2012   51 / 60
Perpetual Flight in Flow Field

Thermal Soaring
   Models
   Observability
   Estimation

                                      (o) Trapezoidal model

   Theorem:       The thermal position, velocity and updraft flow field planar parameters are
   locally weakly observable by an aircraft flying trajectories with ϕ̇ 6= γ̇ tan2 (ϕ − γ), as
   long as the trajectory is included in the area defined by r2 ≥ d ≥ r1 . This holds for the
   trapezoidal model.
   The aircraft cannot fly at a constant distance from the thermal
   center.
   The aircraft should be flying around the thermal or turning
  ARC Lab Team (UM)                           Other Research                September 2012       52 / 60
Perpetual Flight in Flow Field

Thermal Soaring

   Models
   Observability
   Estimation - Regularized Adaptive Particle Filter

                                   (p) Estimator initialization

  ARC Lab Team (UM)                           Other Research      September 2012   53 / 60
Perpetual Flight in Flow Field

Thermal Soaring

   Models
   Observability
   Estimation - Regularized Adaptive Particle Filter

                                   (q) Estimator convergence

  ARC Lab Team (UM)                           Other Research   September 2012   54 / 60
Perpetual Flight in Flow Field

Wind Shear Soaring

   Models - Surface, Layer & Ridge Wind Shear
   Estimation

                                     (r) Surface Wind Shear

  ARC Lab Team (UM)                           Other Research   September 2012   55 / 60
Perpetual Flight in Flow Field

Wind Shear Soaring

   Models - Surface, Layer & Ridge Wind Shear
   Estimation

   (s) Layer Wind Shear                                        (t) Ridge Wind Shear

  ARC Lab Team (UM)                           Other Research                   September 2012   56 / 60
Perpetual Flight in Flow Field

Wind Shear Soaring
   Models
   Estimation - Particle Filter

                                   (u) Estimator initialization
  ARC Lab Team (UM)                           Other Research      September 2012   57 / 60
Perpetual Flight in Flow Field

Wind Shear Soaring
   Models
   Estimation - Particle Filter

                               (v) Estimator final convergence
  ARC Lab Team (UM)                           Other Research     September 2012   58 / 60
Perpetual Flight in Flow Field

Formation Flight
   Validation of airflow models
   Collection of spatially distributed samples
   Safe flight at close distances

                                   (w) Formation in a thermal
  ARC Lab Team (UM)                           Other Research    September 2012   59 / 60
Perpetual Flight in Flow Field

Formation Flight
   Validation of airflow models
   Collection of spatially distributed samples
   Safe flight at close distances

                                    (x) Formation in a thermal
  ARC Lab Team (UM)                           Other Research     September 2012   59 / 60
Perpetual Flight in Flow Field

Formation Flight
   Validation of airflow models
   Collection of spatially distributed samples
   Safe flight at close distances

                                    (y) Formation in a thermal
  ARC Lab Team (UM)                           Other Research     September 2012   59 / 60
Perpetual Flight in Flow Field

                                 Thank You!

ARCLAB (UM)                Collaborative Unmanned Air Vehicles   60 / 60
You can also read
NEXT SLIDES ... Cancel