Learning Optimal Decision Trees Using Caching Branch-and-Bound Search

Page created by Adrian Hale
 
CONTINUE READING
Learning Optimal Decision Trees Using Caching Branch-and-Bound Search

                                        Gaël Aglin, Siegfried Nijssen, Pierre Schaus
                                                  {firstname.lastname}@uclouvain.be
                                                         ICTEAM, UCLouvain
                                                       Louvain-la-Neuve, Belgium

                            Abstract                                 • the trees found are accurate while satisfying additional
                                                                       constraints such as on the fairness of the trees: in their
  Several recent publications have studied the use of Mixed            predictions, the trees may favor one group of individuals
  Integer Programming (MIP) for finding an optimal decision            over another.
  tree, that is, the best decision tree under formal require-
  ments on accuracy, fairness or interpretability of the predic-     With the increasing interest in explainable and fair models
  tive model. These publications used MIP to deal with the hard      in machine learning, recent years have witnessed a renewed
  computational challenge of finding such trees. In this paper,      interest in alternative algorithms for learning decision trees
  we introduce a new efficient algorithm, DL8.5, for finding         that can provide such optimality guarantees.
  optimal decision trees, based on the use of itemset mining
                                                                        Most attention has been given in recent years and in
  techniques. We show that this new approach outperforms ear-
  lier approaches with several orders of magnitude, for both nu-     prominent venues to approaches based on mixed integer pro-
  merical and discrete data, and is generic as well. The key idea    gramming (Bertsimas and Dunn 2017; Verwer and Zhang
  underlying this new approach is the use of a cache of itemsets     2019; Aghaei, Azizi, and Vayanos 2019). In these ap-
  in combination with branch-and-bound search; this new type         proaches, a limit is imposed on the depth of the trees that
  of cache also stores results for parts of the search space that    can be learned and a MIP solver is used to find the optimal
  have been traversed partially.                                     tree under well-defined constraints.
                                                                        However, earlier algorithms for finding optimal decision
                                                                     trees under constraints have been studied in the literature,
                        Introduction                                 of which we consider the DL8 algorithm of particular inter-
Decision trees are among the most widely used machine                est (Nijssen and Fromont 2007; 2010). The existence of this
learning models. Their success is due to the fact that they          earlier work does not appear to have been known to the au-
are simple to interpret and that there are efficient algorithms      thors of the more recent MIP-based approaches, and hence,
for learning trees of acceptable quality.                            no comparison with this earlier work was carried out.
   The most well-known algorithms for learning decision                 DL8 is based on a different set of ideas than the MIP-
trees, such as CART (Breiman et al. 1984) and C4.5 (Quin-            based approaches: it treats the paths of a decision tree as
lan 1993), are greedy in nature: they grow the decision tree         itemsets, and uses ideas from the itemset mining literature
top-down, iteratively splitting the data into subsets.               (Agrawal et al. 1996) to search through the space of possible
   While in general these algorithms learn models of good            paths efficiently, performing dynamic programming over the
accuracy, their greedy nature, in combination with the NP-           itemsets to construct an optimal decision tree. Compared to
hardness of the learning problem (Laurent and Rivest 1976),          the MIP-based approaches, which most prominently rely on
implies that the trees that are found are not necessarily opti-      a constraint on depth, DL8 stresses the use of a minimum
mal. As a result, these algorithms do not ensure that:               support constraint to limit the size of the search space. It was
                                                                     shown to support a number of different optimization criteria
• the trees found are the most accurate for a given limit on         and constraints that do not necessarily have to be linear.
  the depth of the tree; as a result, the paths towards deci-           In this paper, we present a number of contributions. We
  sions may be longer and harder to interpret than neces-            will demonstrate that DL8 can also be applied in settings in
  sary;                                                              which MIP-based approaches have been used; we will show
• the trees found are the most accurate for a given lower            that, despite its age, it outperforms the more modern MIP-
  bound on the number of training examples used to deter-            based approaches significantly, and is hence an interesting
  mine class labels in the leaves of the tree;                       starting point for future algorithms.
                                                                        Subsequently, we will present DL8.5, an improved ver-
Copyright c 2020, Association for the Advancement of Artificial      sion of DL8 that outperforms DL8 by orders of magnitude.
Intelligence (www.aaai.org). All rights reserved.                    Compared to DL8, DL8.5 adds a number of novel ideas:
• it uses branch-and-bound search to cut large additional         present paper is easily implemented and understood without
  parts of the search space;                                      relying on CP systems.
• it uses a novel caching approach, in which we store also           Another class of methods for learning optimal decision
  store information for itemsets for which the search space       trees is that based on SAT Solvers (Narodytska et al. 2018;
  has been cut; this allows us to avoid redundant computa-        Bessiere, Hebrard, and O’Sullivan 2009). SAT-based stud-
  tion later on as well;                                          ies, however, focus on a different type of decision tree learn-
                                                                  ing problem than the MIP-based approaches: finding a deci-
• we consider a range of different branching heuristics to        sion tree of limited size that performs 100% accurate predic-
  find good trees more rapidly;                                   tions on training data. These approaches solve this problem
• the algorithm has been made any-time, i.e. it can be            by creating a formula in conjunctive normal form, for which
  stopped at any time to report the best tree it has found        a satisfying assignment would represent a 100% accurate de-
  so far.                                                         cision tree. We believe there is a need for algorithms that
                                                                  minimize the error, and hence we focus on this setting.
In our experiments we focus our attention on traditional de-
                                                                     Most related to this work is the work of Nijssen and
cision tree learning problems with little other constraints, as
                                                                  Fromont (2007; 2010) on DL8, which relies on a link be-
we consider these learning problems to be the hardest. How-
                                                                  tween learning decision trees and itemset mining. Similarly
ever, we will show that DL8.5 remains sufficiently close to
                                                                  to MIP-based approaches, DL8 allows to find optimal deci-
DL8 that the addition of other constraints or optimization
                                                                  sion trees minimizing misclassification error. DL8 does not
criteria is straightforward.
                                                                  require a depth constraint; it does however assume the pres-
   In a recent MIP-based study, significant attention was         ence of a minimum support constraint, that is, a constraint
given to the distinction between binary and numerical             on the minimum number of examples falling in each leaf.
data (Verwer and Zhang 2019). We will show that DL8.5             In the next section we will discuss this approach in more de-
outperforms this method on both types of data.                    tail. This discussion will show that DL8 can easily be used in
   Our implementation is publicly available at https://github.    settings identical to those in which MIP and CP solvers have
com/aglingael/dl8.5 and can easily be used from Python.           been used. Subsequently, we will propose a number of sig-
   This paper is organized as follows. The next section           nificant improvements, allowing the itemset-based approach
presents the state of the art of optimal decision trees induc-    to outperform MIP-based and CP-based approaches.
tion. Then we present the background on which our work
relies, before presenting our approach and our results.
                                                                                         Background
                      Related work                                Itemset mining for decision trees
In our discussion of related work, we will focus our attention    We limit our discussion to the key ideas behind DL8, and
on alternative methods for finding optimal decision trees,        assume that the reader is already familiar with the general
that is, decision trees that achieve the best possible score      concepts behind learning decision trees.
under a given set of constraints.                                    DL8 operates on Boolean data. As running example of
   Most attention has been given in recent years to MIP-          such data, we will use the dataset of Table 1a, which consists
based approaches. Bertsimas and Dunn (2017) developed an          of three Boolean features and eleven examples. The optimal
approach for finding decision trees of a maximum depth K          decision tree for this database can be found in Figure 1a.
that optimize misclassification error. They use K to model           While it may seem a limitation that DL8 only operates on
the problem in a MIP model with a fixed number of vari-           Boolean data, there are straightforward ways to transform
ables; a MIP solver is then used to find the optimal tree.        any tabular database in a Boolean database: for categorical
   Verwer and Zhang (2019) proposed BinOCT, an optimiza-          attributes, we can create a Boolean column for each possible
tion of this approach, focused on how to deal with numeri-        value of that attribute; for numerical attributes, we can create
cal data. To deal with numerical data, decision trees need        Boolean columns for possible thresholds for that attribute.
to identify thresholds that are used to separate examples            DL8 takes an itemset mining perspective on learning deci-
from each other. A MIP model was proposed in which fewer          sion trees. In this perspective, the binary matrix of Table 1a
variables are needed to find high-quality thresholds; conse-      is transformed into the transactional database of Table 1b.
quently, it was shown to work better on numerical data.           Each transaction of the dataset contains an itemset describ-
   A benefit of MIP-based approaches is that it is relatively     ing the presence or absence of each feature in the dataset.
easy from a modeling perspective to add linear constraints        More formally, the database D can be thought of as a col-
or additional linear optimization criteria. Aghaei, Azizi, and    lection D = {(t, I, c) | t ∈ T , I ⊆ I, c ∈ C}, where T
Vayanos (2019) exploit this to formalize a learning problem       represents the transaction or rows identifiers, I is the set of
that also takes into account the fairness of a prediction.        possible items, and C is the set of class labels; within I there
   Verhaeghe et al. (2019) recently proposed a Constraint         are two items (one positive, the other negative) for each orig-
Programming (CP) approach to solve the same problem.              inal Boolean feature, and each itemset I contains either a
It supports a maximum depth constraint and a minimum              positive or a negative item for every feature.
support constraint, but only works for binary classification         Using this representation, every path in a decision tree can
tasks. It also relies on branch-and-bound search and caching,     be mapped to an itemset I ⊆ I, as shown in Figure 1b. For
but uses a less efficient caching strategy. The approach in the   instance, the last path in this tree corresponds to the itemset
A     B        C           class               rowId     items             class         not required; MIP-based approaches have ignored the mini-
    0     1        1             0                   1      ¬a, b, c             0           mum support constraint.
    1     0        1             1                   2      a, ¬b, c             1
    0     0        1             1                   3     ¬a, ¬b, c             1
    0     1        0             0                   4     ¬a, b, ¬c             0            Algorithm 1: DL8(maxdepth, minsup)
    1     0        0             1                   5     a, ¬b, ¬c             1
    0     0        0             0                   6     ¬a, ¬b, ¬c            0           1    struct BestT ree{ tree : T ree; error : f loat }
    0     0        1             0                   7     ¬a, ¬b, c             0           2    cache ← HashSet < Itemset, BestTree >
    1     1        0             1                   8      a, b, ¬c             1           3    (τ, b) ← DL8 − Recurse(∅)
    0     0        0             1                   9     ¬a, ¬b, ¬c            1           4    return τ
    0     0        1             0                   10    ¬a, ¬b, c             0           5    Procedure DL8 − Recurse(I)
    0     0        0             1                   11    ¬a, ¬b, ¬c            1            6        solution ← cache.get(I)
                                                                                              7        if solution was found then
        (a) Binary matrix                             (b) Transactional database
                                                                                              8             return (solution.tree, solution.error)
                           Table 1: Example database                                          9       if leaf error (I) = 0 or |I| = maxdepth then
                                                                                             10            return (make leaf (I), leaf error (I))
                                                                                             11       (τ, b) ← (make leaf (I), leaf error (I))
                       A                                         A                           12       for all attributes i do
                                                                                             13            if |cover (I ∪ {i})| ≥ minsup and
                   1       0                                    a ¬a                                         |cover (I ∪ {¬i})| ≥ minsup then
               1               B                            1         B                      14                 (τ1 , e1 ) ← DL8 − Recurse(I ∪ {¬i})
                                                                                             15                 if e1 ≤ b then
                           1       0                                 b ¬b                    16                       (τ2 , e2 ) ← DL8 − Recurse(I ∪ {i})
                                                                                             17                       if e1 + e2 ≤ b then
                       0               C                         0         C                 18                             (τ, b) ← (make tree(i, τ1 , τ2 ),
                                                                           c ¬c
                                                                                                                             e1 + e2 )
                                   1       0

                               0               1                       0       1             19       cache.store(I, BestT ree(τ, b))
              (a) Common view                               (b) Itemset view                 20       return (τ, b)

Figure 1: Optimal tree corresponding to database of Table 1.
Max depth = 3 and minimum examples per leaf = 1
                                                                                             DL8 Algorithm
                                                                                             A high-level perspective of the DL8 algorithm is given in Al-
{¬a, ¬b, ¬c}. Please note that multiple paths can be mapped                                  gorithm 1. Essentially, the algorithm recursively enumerates
to the same itemset.                                                                         itemsets using the DL8 − Recurse(I) function. The post-
   For every itemset I, we define its cover to be cover (I) =                                condition of this function is that it returns the optimal deci-
{(t0 , I 0 , c0 ) ∈ D | I ⊆ I 0 }: the set of transactions in which                          sion tree for the transactions covered by itemset I, together
the itemset is contained; the class-based support of an item-                                with the quality of that tree. This optimal tree is calculated
set is defined as Sup(I, c) = |{(t0 , I 0 , c0 ) ∈ cover (I) | c0 =                          recursively using the observation that the best decision tree
c}|, and can be used to identify the number of examples for                                  for a set of transactions can be obtained by considering all
a given class c in a leaf. Based on class-based supports, the                                possible ways of partitioning the set of transactions into two,
error of an itemset is defined as:                                                           and determining the best tree for each partition recursively.
                                                                                                Figure 2 illustrates the search space of itemsets for the
    leaf error (I) =| cover (I) | − max{Sup(I, c)}                                     (1)   dataset of Table 1, where all the possible itemsets are repre-
                                                              c∈C
                                                                                             sented. Intuitively, DL8 starts at the top node of this search
Unlike the CP-based approach, our error function is also                                     space, and calculates the optimal decision tree for the root
valid for classification tasks involving more than 2 classes.                                based on its children.
   The canonical decision tree learning problem that we                                         A distinguishing feature of DL8 is its use of a cache.
study in this work can now be defined as follows using item-                                 The idea behind this cache is to store the result of a call to
set mining notation. Given a database D, we wish to identify                                 DL8 − Recurse. Doing so is effective as the same itemset
a collection DT ⊆ I of itemsets such that                                                    can be reached by multiple paths in the search space: item-
• the itemsets in DT represent a decision tree;                                              set ab can be constructed by adding b to itemset a, or by
  P                                                                                          adding a to itemset b. By storing the result, we can reuse the
•    I∈DT leaf error (I) is minimal;                                                         same result for both paths.
• for all I ∈ DT : |I| ≤ maxdepth, where maxdepth is the                                        Note that the optimal decision tree for the root can only be
  maximum depth of the tree;                                                                 calculated after all its children have been considered; hence,
                                                                                             the algorithm will only produce a solution once the entire
• for all I ∈ DT : |cover (I)| ≥ minsup, where minsup is                                     search space of itemsets has been considered.
  a minimum support threshold.                                                                  In our pseudocode, we use the following other func-
As stated earlier, in DL8, the maximum depth constraint is                                   tions. Function make leaf (I) returns a decision tree with
visited node                                                                                                 edge to visited node
                                                                                                                    pruned edge
       pruned node                                                                                                  edge to infrequent node

       infrequent node

                         Figure 2: Complete itemset lattice for introduction database and DL8.5 search execution

one node, representing a leaf that predicts the major-                                  Our approach: DL8.5
ity class for the examples covered by I. The function
make tree(i, τ1 , τ2 ) returns a tree with a test on attribute i,      As identified in the introduction, DL8 has a number of weak-
and subtrees τ1 and τ2 .                                               nesses, which we will address in this section.
   The code illustrates a number of optimizations imple-                  The most prominent of these weaknesses is that the size
mented in DL8:                                                         of the search tree considered by DL8 is unnecessarily large.
                                                                       Reconsider the example of Figure 2, in which DL8’s prun-
Maximum depth pruning In line 9 the search is stopped
                                                                       ing approach does not prune any node except from one in-
   as soon as the itemsets considered are too long;
                                                                       frequent itemset (abc). We will see in this section that a
Minimum support pruning In line 13 an attribute is not                 new type of caching brand-and-bound search can reduce the
   considered if one of its branches has insufficient support;         number of itemsets considered significantly.
   in our running example, the itemset {a, b, c} is not con-              The pseudo-code of our new algorithm, DL8.5, is pre-
   sidered due to this optimization;                                   sented in Algorithm 2. DL8.5 inherits a number of ideas
Purity pruning In line 9 the search is stopped if the error            from DL8, including the use of a cache, the recursive traver-
   for the current itemset is already 0;                               sal of the space of itemsets, and the use of depth and support
Quality bounds In the loop of lines 12–18, the best solu-              constraints to prune the search space.
   tion found among the children is maintained, and used to               The main distinguishing feature of DL8.5 concerns its use
   prune the second branch for an attribute if the first branch        of bounds during the search.
   is already worse than the best solution found so far.                  In DL8.5, the recursive procedure DL8.5 − Recurse has
   We omit a number of optimizations in this pseudo-code               an additional parameter, init ub, which represents an upper-
that can be found in the original publication, in particular,          bound on the quality of the decision trees that the recursive
optimizations that concern the incremental maintenance of              procedure is expected to find. If no sufficiently good tree can
data structures. While we will use most of these optimiza-             be found, the procedure returns a tree of type NO TREE.
tions in our implementation as well, we do not discuss these              Initially, the upper-bound that is used is +∞ (line 3).
in detail here for reasons of simplicity.                              However, as soon as the recursive algorithm has found one
   The most important optimization in DL8 that we do not               decision tree, or has found a better tree than earlier known,
use in this study is the closed itemset mining optimization.           the quality of this decision tree, calculated in line 21, is used
The reason for this choice is that this optimization is hard           as upper-bound for future decision trees and is communi-
to combine with a constraint on the depth of a decision tree.          cated to the children in the search tree (line 25, line 14, 18).
Similarly, while DL8 can be applied to other scoring func-                The upper-bound is used to prune the search space using
tions than error, as long as the scoring function is additive,         a test in line 17; intuitively, as soon as we have traversed one
we prioritize accuracy and the depth constraint here as we             branch for an attribute, and the quality of that branch is al-
focus on solving the same problem as in recent MIP-based               ready worse than accepted by the bound, we do not consider
studies.                                                               the second branch for that attribute.
Algorithm 2: DL8.5(maxdepth, minsup)
1    struct BestT ree {init ub : f loat; tree : T ree;
       error : f loat}
2    cache ← HashSet < Itemset, BestTree >
3    bestSolution ← DL8.5 − Recurse(∅, +∞)
4    return bestSolution.tree
5    Procedure DL8.5 − Recurse(I, init ub)
         if leaf error (I) = 0 or |I| = maxdepth or time-out

                                                                                  ....

                                                                                         ....
 6
           is reached then
 7             return BestT ree(init ub, make leaf (I),
                leaf error (I))
 8       solution ← cache.get(I)
 9       if solution was found and ((solution.tree 6=
           NO TREE) or (init ub ≤ solution.init ub)) then
10             return solution
                                                                                   Figure 3: Example of pruning
11       (τ, b, ub) ← (NO TREE, +∞, init ub)
12       for all attributes i in a well-chosen order do
13            if |cover (I ∪ {i})| ≥ minsup and
                |cover (I ∪ {¬i})| ≥ minsup then                   be interrupted when a time-out is reached, and line 12 offers
14                 sol1 ← DL8.5 − Recurse(I ∪ {¬i}, ub)            the possibility to consider the attributes in a specific heuristic
15                 if sol1 .tree = NO TREE then                    order to discover good trees more rapidly.
16                      continue                                      A number of different heuristics could be considered. In
17               if sol1 .error ≤ ub then                          our experiments, we consider three: the original order of the
18                    sol2 ← DL8.5 − Recurse(I ∪ {i},              attributes in the data, in increasing or in decreasing order of
                        ub − sol1 .error)                          information gain (such as used in C4.5 and CART).
19                    if sol2 .tree = NO TREE then                    Our modifications of DL8 improve drastically the prun-
20                         continue                                ing of the search space. Figure 2 indicates which additional
21                   f eature error ←                              nodes are pruned during the execution of DL8.5 (for an al-
                       sol1 .error + sol2 .error                   phabetic order of the attributes). At the end, 17 nodes are
22                   if f eature error ≤ ub then                   visited instead of 27.
23                         τ ← make tree(i, sol1 .tree,               Figure 3 shows a part of the execution of DL8.5 in more
                            sol2 .tree)
24                         b ← f eature error
                                                                   detail. The initial value of the upper-bound at node φ is +∞
25                         ub ← b − 1                              (line 11). The attribute A provides an error of 2; the upper-
                                                                   bound value is subsequently updated from +∞ to 1 in line
26                   if f eature error = 0 then                    25. In the first branch for attribute C, the new value of the
27                         break
                                                                   upper-bound is passed down recursively (line 14). Notice
                                                                   that the initial value of the upper-bound at node ¬c is 1. At
28       solution ← BestT ree(init ub, τ, b)                       this node, the attribute A is first visited and provides an er-
29       cache.store(I, solution)                                  ror of 1 by summing errors of ¬a¬c and a¬c (line 21). The
30       return solution                                           upper-bound for subsequent attributes is then updated to 0
                                                                   and passed down recursively to the first branch of attribute
                                                                   B. After visiting the first item ¬b¬c the obtained error is 1
                                                                   and greater than the upper-bound of 0. The second item is
   In line 18 we use the quality of the first branch to bound
                                                                   pruned as the condition of line 18 is not satisfied. So, there
the required quality of the second branch further.
                                                                   is no solution by selecting the attribute B, which leads to
   An important modification involves the interaction of the       storing the value NO TREE for this itemset. This error value
bounds with the cache. In DL8.5, we store an itemset also          is represented in Figures 2 and 3 by the character x.
if no solution could be found for the given bound (line 29
                                                                      The reuse of the cache is illustrated for itemset ¬ac (Fig-
is still executed even if the earlier loop did not find a tree).
                                                                   ure 2). The first time we encounter this itemset, we do so
In this case, the special value NO TREE is associated with the
                                                                   coming from the itemset ¬a for an upper-bound of zero; af-
itemset in the cache, and the upper-bound used during the
                                                                   ter the first branch, we observe that no solution can be found
last search is stored.
                                                                   for this bound, and we store NO TREE for this itemset. The
   The benefit of doing so is that at a later moment, we may
                                                                   second time we encounter ¬ac, we do so coming from the
reuse the fact that for a given itemset and bound, no suf-
                                                                   parent c, again with an upper-bound of 0. From the cache
ficiently good decision tree can be found. In particular, in
                                                                   we retrieve the fact that no solution could be found for this
line 9, when the current bound (init ub) is worse than the
                                                                   bound, and we skip attribute A from further consideration.
stored upper-bound for a NO TREE itemset, we return the
NO TREE indicator immediately.
   Other modifications in comparison with DL8 improve the                                       Results
anytime behavior of the algorithm. In line 6 the search can        In our experiments we answer the following questions:
Q1 How does the performance of DL8.5 compare to DL8,                  DL8.5 solved 19 (which answers Q1). The difference in per-
   MIP-based and CP-based approaches on binary data?                  formance is further illustrated in Figure 4, which gives cac-
Q2 What is the impact of different branching heuristics on            tus plots for each algorithm, for different depth constraints.
   the performance of DL8.5?                                          In these plots each point (x, y) indicates the number of in-
                                                                      stances (x) solved within a time limit (y). While for lower
Q3 What is the impact of caching incomplete results in                depth thresholds, BinOCT does find solutions, the perfor-
   DL8.5 (NO TREE itemsets)?                                          mance of all variants of DL8.5 clearly remains superior to
Q4 How does the performance of DL8.5 compare to DL8,                  that of DL8, BinOCT and the CP-based algorithm, obtain-
   MIP-based and CP-based approaches on continuous data?              ing orders of magnitude better performance.
As a representative MIP-based approach, we use BinOCT, as                Comparing the different branching heuristics in DL8.5,
it was shown to be the best performing MIP-based approach             the differences are relatively small; however, for deeper
in a recent study (Verwer and Zhang 2019). The implemen-              trees, a descending order of information gain gives slightly
tations of BinOCT1 , DL8 and the CP-based approach2 used              better results. This confirms the intuition that chosen a split
in our comparison were obtained from their original authors,          with high information gain is a good heuristic (Q2).
and use the CPLEX 12.93 and OscaR 4 solvers. Experiments                 If we disable DL8.5’s ability to cache incomplete results,
were performed on a server with an Intel Xeon E5-2640                 we see a significant degradation in performance. In this vari-
CPU, 128GB of memory, running Red Hat 4.8.5-16.                       ant only 12 instances are solved optimally, instead of 19, for
   To respect the constraint of the CP-based algorithm all            a depth of 4. Hence, this optimization is significant (Q3).
the datasets used in our experiments have binary classes. We             To answer Q4, we repeat these tests on continuous data.
compare our algorithms on 24 binary datasets from CP4IM5 ,            For this, we use the same datasets as Verwer and Zhang
described in the first columns of Table 2.                            (2019). These datasets were obtained from the UCI repos-
   Similar to Verwer and Zhang (2019), we run the different           itory6 and are summarized in the first columns of Table 3.
algorithms for 10 minutes on each dataset and for a maxi-             Before running DL8, the CP-based algorithm and DL8.5, we
mum depth of 2, 3 and 4. All the tests are run with a mini-           binarize these datasets by creating binary features using the
mum support of 1 since this is the setting used in BinOCT.            same approach as the one used by Verwer and Zhang (2019).
   We do not split our datasets in training and test sets since       Note that the number of generated features is very high in
the focus of this work is on comparing the computational              this case. As a result, for most datasets all algorithms reach a
performance of algorithms that should generate decision               time-out for maximum depths of 3 and 4, as was also shown
trees of the same quality. The benefits of optimal decision           by Verwer and Zhang (2019). Hence, we focus on results
trees were discussed in (Bertsimas and Dunn 2017).                    for a depth of 2 in Table 3. Even though BinOCT uses a
   We compare a number of variants of DL8.5. The follow-              specialized technique for solving continuous data, the table
ing table summarizes the abbreviations used.                          shows that DL8.5 outperforms DL8, the CP-based algorithm
   Abbreviation Meaning                                               and BinOCT. Note that the differences between the different
   d.o.             the original order of the attributes in the       variations of DL8.5 are small here, which may not be sur-
                    data is used as branching heuristic               prising given the shallowness of the search tree considered.
   asc              attributes are sorted in increasing value
                    of information gain                                                          Conclusions
   desc             attributes are sorted in decreasing value
                    of information gain                               In this paper we presented the DL8.5 algorithm for learning
   n.p.s.           no partial solutions are stored in the            optimal decision trees. DL8.5 is based on a number of ideas:
                    cache                                             the use of itemsets to represent paths, the use of a cache
                                                                      to store intermediate results (including results for parts of
   Table 2 shows the results for a maximum depth equal to 4,          the search tree that have only been traversed partially), the
as we consider deeper decision trees of more interest. If op-         use of bounds to prune the search space, the ability to use
timality could not be proven within 10 minutes, this is indi-         heuristics during the search, and the ability to return a result
cated using TO; in this case, the objective value of the best         even when a time-out is reached.
tree found so far is shown. Note that we here exploit the                Our experiments demonstrated that DL8.5 outperforms
ability of DL8.5 to produce a result after a time-out. The            existing approaches by orders of magnitude, including ap-
best solutions and best times are marked in bold while a star         proaches presented recently at prominent venues.
(*) is added to mark solutions proven to be optimal.                     In this paper, we focused our experiments on one par-
   BinOCT solved and proved optimality for only 1 instance            ticular setting: learning maximally accurate trees of lim-
within the timeout; the older DL8 algorithm solved 7 in-              ited depth without support constraints. This was motivated
stances and the CP-based algorithm solved 11 instances.               by our desire to compare our new approach with other ap-
   1
     https://github.com/SiccoVerwer/binoct                            proaches. However, we believe DL8.5 can be modified for
   2
     https://bitbucket.org/helene verhaeghe/classificationtree/src/   use in other constraint-based decision tree learning prob-
default/classificationtree/                                           lems, using ideas from DL8 (Nijssen and Fromont 2010).
   3
     https://www.ibm.com/analytics/cplex-optimizer                    Acknowledgements. This work was supported by Bpost.
   4
     https://oscarlib.bitbucket.io
   5                                                                     6
     https://dtai.cs.kuleuven.be/CP4IM/datasets/                             https://archive.ics.uci.edu/ml/index.php
BinOCT                                   DL8                             CP-Based                                                                                                          DL8.5
                              Dataset                            nItems       nTrans                                                                                                                             d.o.                                       asc                                             desc                                 n.p.s.

                                                                                                    time (s)

                                                                                                                                                    time (s)

                                                                                                                                                                                        time (s)

                                                                                                                                                                                                                                   time (s)

                                                                                                                                                                                                                                                                                       time (s)

                                                                                                                                                                                                                                                                                                                              time (s)

                                                                                                                                                                                                                                                                                                                                                          time (s)
                                                                                            obj

                                                                                                                                   obj

                                                                                                                                                                      obj

                                                                                                                                                                                                           obj

                                                                                                                                                                                                                                                      obj

                                                                                                                                                                                                                                                                                                      obj

                                                                                                                                                                                                                                                                                                                                             obj
                              anneal                              186            812       115     TO                          ∞             TO                 91∗         450.69                   91∗            129.24                       91∗            127.45                              91∗       121.87                       91∗        250.64
                              audiology                           296            216         2     TO                          ∞             TO                   1            TO                     1∗           180.84                         1∗            204.19                               1∗        195.73                        1           TO
                              australian-credit                   250            653        82     TO                          ∞             TO                  66            TO                    56∗           566.71                        56∗            586.39                              56∗        593.38                       57           TO
                              breast-wisconsin                    240            683        12     TO                          ∞             TO                   8            TO                     7∗            305.3                         7∗             325.7                               7∗        330.71                        7           TO
                              diabetes                            224             76       170     TO                          ∞             TO                 140            TO                   137∗           553.49                       137∗            562.83                             137∗         565.5                      137           TO
                              german-credit                       224           1000       223     TO                          ∞             TO                 204            TO                   204∗           558.73                        206               TO                              204∗        599.87                      204           TO
                              heart-cleveland                     190            296        39     TO                          ∞             TO                  25            TO                    25∗            124.1                        25∗             130.3                              25∗        132.23                      25∗        214.76
                              hepatitis                           136            137         7     TO                          3∗          66.62                 3∗         109.36                    3∗            13.46                         3∗             14.06                               3∗         14.88                       3∗         27.28
                              hypothyroid                         176           3247        55     TO                          ∞             TO                  53            TO                    53∗            392.22                       53∗           368.95                               53∗        427.34                       53           TO
                              ionosphere                          890            351        27     TO                          ∞             TO                  20            TO                     17               TO                         11               TO                                13           TO                        17           TO
                              kr-vs-kp                            146           3196       193     TO                          ∞             TO                144∗         483.15                  144∗            216.11                      144∗           206.18                              144∗        223.85                     144∗        528.72
                              letter                              448           20000      813     TO                          ∞             TO                 574            TO                    550               TO                        586               TO                               802           TO                       550           TO
                              lymph                               136            148         6     TO                          3∗          56.29                 3∗         112.48                    3∗               8.7                        3∗             11.03                               3∗         8.47                        3∗         25.04
                              mushroom                            238           8124       278     TO                          ∞             TO                  0∗         352.18                    0∗            331.39                          4              TO                                0∗         0.11                          4          TO
                              pendigits                           432           7494       780     TO                          ∞             TO                  38            TO                     32               TO                         26               TO                                14           TO                        32           TO
                              primary-tumor                        62            336        37     TO                         34∗           2.79                34∗           8.96                   34∗              1.48                       34 ∗
                                                                                                                                                                                                                                                                  1.51                              34∗         1.38                       34 ∗
                                                                                                                                                                                                                                                                                                                                                        2.43
                              segment                             470           2310        13     TO                          ∞             TO                  0∗         128.25                    0∗             3.54                         0∗              6.99                               0∗          7.05                       0∗         3.54
                              soybean                             100            630        15     TO                         14∗          41.59                14∗          40.13                   14∗               5.7                       14∗              6.34                              14∗          5.75                      14∗         18.41
                              splice-1                            574            319       574     TO                          ∞             TO                  ∞             TO                    224               TO                        224               TO                               141           TO                       224           TO
                              tic-tac-toe                          54            958       180     TO                        137∗           3.76               137∗           9.17                  137∗             1.43                       137∗              1.54                             137∗          1.55                     137∗          2.12
                              vehicle                             504            846        61     TO                          ∞             TO                  22            TO                     16               TO                         18               TO                                13           TO                        16           TO
                              vote                                 96            435         6     TO                          5∗          29.84                 5∗          44.47                    5∗              5.48                        5∗             5.33                                5∗          5.58                       5∗         12.82
                              yeast                               178           1484       395     TO                          ∞             TO                 366            TO                   366∗           318.87                       366∗             326.2                             366∗        334.15                     366∗        470.88
                              zoo-1                                72            101        0∗    0.52                         0∗           1.11                 0∗            0.2                    0∗             0.01                         0∗             0.01                                0∗         0.01                        0∗         0.01

                                                                        Table 2: Comparison table for binary datasets with max depth = 4

                      max depth = 2                                                                                                           max depth = 3                                                                                                                                        max depth = 4
                                                                                                                                                                                                                                                                            1e+03
                                                                          ● ●
                                                                     ●
                                                                 ● ●
              1e+02                                     ●
                                                             ●
                                                                                                                              1e+02
                                                    ●
                                              ● ●
                                          ●
              1e+01
Runtime (s)

                                                                                                               Runtime (s)

                                                                                                                                                                                                                                                              Runtime (s)
                                  ● ● ●                                                                                       1e+01                                                                                                                                         1e+01
                              ●

              1e+00                                                                                                           1e+00
                                                                                                                                                                                                                                                                                                     ●
                          ●
                                                                                                                                                    ●
              1e−01                                                                                                           1e−01                                                                                                                                         1e−01

              1e−02                                                                                                           1e−02
                      0              5                  10               15           20          25                                        0                        5             10                       15                       20                                                                              5                           10                  15
                                          # instances solved                                                                                                             # instances solved                                                                                                                              # instances solved

                                                                                                       ●                     BinOCT                                  DL8                               DL8.5 d.o                                      DL8.5 n.p.s
                                                                                           Algo
                                                                                                                             CP−based                                DL8.5 asc                         DL8.5 desc

                                                                                Figure 4: Cumulative number of instances solved over time

                                                                                                                     BinOCT                                      DL8                                 CP-Based                                                                                        DL8.5
                              Dataset                            nTrans         nFeat      nItems                                                                                                                                                     d.o.                                            asc                                        desc
                                                                                                                                         time (s)

                                                                                                                                                                             time (s)

                                                                                                                                                                                                                        time (s)

                                                                                                                                                                                                                                                                            time (s)

                                                                                                                                                                                                                                                                                                                   time (s)

                                                                                                                                                                                                                                                                                                                                                          time (s)
                                                                                                                             obj

                                                                                                                                                               obj

                                                                                                                                                                                                     obj

                                                                                                                                                                                                                                                obj

                                                                                                                                                                                                                                                                                                    obj

                                                                                                                                                                                                                                                                                                                                           obj

                              balance-scale                        625           4           32           149∗                        1.2                149∗               0.5                    149∗           0.01                        149∗            0.01                                149∗       0.01                        149∗        0.01
                              banknote                            1372           4          3710           101                        TO                 100∗            363.01                      ∞              TO                        100∗           52.81                                100∗       63.41                       100∗        58.07
                              bank                                4521           51         3380           449                        TO                  448               TO                       ∞              TO                        446∗           253.87                               446∗      223.42                       446∗      222.66
                              biodeg                              1055           41         8356           212                        TO                  212               TO                       ∞              TO                        202∗          341.57                                202∗      365.83                       202∗       370.26
                              car                                 1728           6           28           250∗                       4.09                250∗              0.32                    250∗            0.02                       250∗            0.01                                250∗       0.01                        250∗        0.01
                              IndiansDiabetes                      768           8          1714           171                        TO                 171∗             36.75                      ∞              TO                        171∗            7.43                                171∗        8.46                       171∗         8.72
                              Ionosphere                           351           34         4624            29                        TO                  29∗            423.52                      ∞              TO                         29∗           25.68                                 29∗       33.76                        29∗        33.04
                              iris                                 150           4           52             0∗                       0.02                  0∗              0.05                      0∗           0.01                          0∗            0.01                                  0∗       0.01                          0∗        0.01
                              letter                             20000           16          352           625                        TO                 591∗             5.97                     591∗          392.09                       591∗             8.61                               591∗        8.26                       591∗         8.83
                              messidor                            1151           19         9460           383                        TO                  383               TO                       ∞              TO                        383∗          533.27                                383∗      563.83                       383∗       534.32
                              monk1                                124           6           22            22∗                       0.33                 22∗              0.28                     22∗           0.01                         22∗            0.01                                 22∗       0.01                         22∗        0.01
                              monk2                                169           6           22            57∗                       0.79                 57∗              0.28                     57∗           0.01                         57∗            0.01                                 57∗       0.01                         57∗        0.01
                              monk3                                122           6           22             8∗                       0.31                  8∗              0.28                      8∗           0.01                          8∗            0.01                                  8∗       0.01                          8∗        0.01
                              seismic                             2584           18         2240           166                        TO                 164∗            117.34                      ∞              TO                        164∗            44.3                                164∗       47.36                       164∗        47.17
                              spambase                            4601           57        16012           660                        TO                  900               TO                       ∞              TO                         741              TO                                 845         TO                         586          TO
                              Statlog                             4435           36         3274           460                        TO                  443               TO                       ∞              TO                        443∗           205.14                               443∗      193.87                       443∗       188.9
                              tic-tac-toe                          958           18          36           282∗                       7.52                282∗              0.33                    282∗           0.01                        282∗            0.01                                282∗       0.01                        282∗        0.01
                              wine                                 178           13         1198            6∗                       73.1                  6∗               7.0                      6∗           74.72                         6∗             1.17                                 6∗        1.45                         6∗        1.09

                                                                  Table 3: Comparison table for continuous datasets with max depth = 2
References
Aghaei, S.; Azizi, M. J.; and Vayanos, P. 2019. Learning op-
timal and fair decision trees for non-discriminative decision-
making. arXiv preprint arXiv:1903.10598.
Agrawal, R.; Mannila, H.; Srikant, R.; Toivonen, H.;
Verkamo, A. I.; et al. 1996. Fast discovery of association
rules. Advances in knowledge discovery and data mining
12(1):307–328.
Bertsimas, D., and Dunn, J. 2017. Optimal classification
trees. Machine Learning 106(7):1039–1082.
Bessiere, C.; Hebrard, E.; and O’Sullivan, B. 2009. Min-
imising decision tree size as combinatorial optimisation.
In International Conference on Principles and Practice of
Constraint Programming, 173–187. Springer.
Breiman, L.; Friedman, J.; Olshen, R.; and Stone, C.
1984. Classification and Regression Trees. Monterey, CA:
Wadsworth and Brooks.
Laurent, H., and Rivest, R. L. 1976. Constructing optimal
binary decision trees is NP-complete. Information process-
ing letters 5(1):15–17.
Narodytska, N.; Ignatiev, A.; Pereira, F.; Marques-Silva, J.;
and RAS, I. 2018. Learning optimal decision trees with sat.
In IJCAI, 1362–1368.
Nijssen, S., and Fromont, E. 2007. Mining optimal decision
trees from itemset lattices. In Proceedings of the 13th ACM
SIGKDD international conference on Knowledge discovery
and data mining, 530–539. ACM.
Nijssen, S., and Fromont, E. 2010. Optimal constraint-based
decision tree induction from itemset lattices. Data Mining
and Knowledge Discovery 21(1):9–51.
Quinlan, J. R. 1993. C4.5: Programs for Machine Learn-
ing. San Francisco, CA, USA: Morgan Kaufmann Publish-
ers Inc.
Verhaeghe, H.; Nijssen, S.; Pesant, G.; Quimper, C.-G.; and
Schaus, P. 2019. Learning optimal decision trees using
constraint programming. In The 25th International Confer-
ence on Principles and Practice of Constraint Programming
(CP2019).
Verwer, S., and Zhang, Y. 2019. Learning optimal classi-
fication trees using a binary linear program formulation. In
33rd AAAI Conference on Artificial Intelligence.
You can also read