Locality Relationship Constrained Multi-view Clustering Framework

Page created by Justin Castro
 
CONTINUE READING
Locality Relationship Constrained Multi-view Clustering Framework
IEEE TRANSACTIONS ON MULTIMEDIA, VOL. XXX, NO. XX, 2021                                                                                    1

                                                  Locality Relationship Constrained Multi-view
                                                             Clustering Framework
                                                                                      Xiangzhu Meng, Wei Wei∗ , and Wenzhe Liu

                                            Abstract—In most practical applications, it’s common to utilize          producing satisfied performance by integrating the diversity
                                         multiple features from different views to represent one object.             information among different views. An easy yet effective
                                         Among these works, multi-view subspace-based clustering has                 way is to concatenate all views as one common view, then
                                         gained extensive attention from many researchers, which aims
                                         to provide clustering solutions to multi-view data. However,                utilize traditional single view methods to solve the clustering
arXiv:2107.05073v1 [cs.CV] 11 Jul 2021

                                         most existing methods fail to take full use of the locality                 task. However, this manner not only neglects the specific
                                         geometric structure and similarity relationship among samples               statistical property in each view , but cannot take full use of
                                         under the multi-view scenario. To solve these issues, we propose            the consistency and complementary information from multiple
                                         a novel multi-view learning method with locality relationship               views.
                                         constraint to explore the problem of multi-view clustering, called
                                         Locality Relationship Constrained Multi-view Clustering Frame-                  To solve above issues, plenty of multi-view clustering meth-
                                         work (LRC-MCF). LRC-MCF aims to explore the diversity,                      ods have been widely proposed in the past ten years. A class
                                         geometric, consensus and complementary information among                    of representative multi-view methods [16–20] investigate to
                                         different views, by capturing the locality relationship information         construct one common latent view shared by all views, which
                                         and the common similarity relationships among multiple views.               can integrate the diversity and complementary information
                                         Moreover, LRC-MCF takes sufficient consideration to weights
                                         of different views in finding the common-view locality structure            from different views into the common view and then partition
                                         and straightforwardly produce the final clusters. To effectually            it into the ideal groups. For example, Auto-Weighted Multiple
                                         reduce the redundancy of the learned representations, the low-              Graph Learning (AMGL)[17] method automatically assigns an
                                         rank constraint on the common similarity matrix is considered               ideal weight for each view in integrating multi-view informa-
                                         additionally. To solve the minimization problem of LRC-MCF, an              tion according to the importance of each view. Nevertheless,
                                         Alternating Direction Minimization (ADM) method is provided
                                         to iteratively calculate all variables LRC-MCF. Extensive exper-            they cannot guarantee the complementarity information among
                                         imental results on seven benchmark multi-view datasets validate             different views so that rich multi-view information cannot
                                         the effectiveness of the LRC-MCF method.                                    be further fully utilized. Therefore, another class of multi-
                                           Index Terms—Multi-view clustering, Locality Relationship                  view clustering methods are proposed to further discover
                                         Constraint, Alternating Direction Minimization                              the complementary information among different views. These
                                                                                                                     works based on Canonical Correlation Analysis (CCA) [21]
                                                                                                                     and Hilbert-Schmidt Independence Criterion (HSIC) [22] are
                                                                  I. I NTRODUCTION                                   two classes of representative multi-view clustering methods.
                                            With the rapid development of the information era, one                   The former [23–25] employs CCA to project the pairwise
                                         common object can usually be represented by different view-                 views into the common subspace by maximizing the corre-
                                         points [1, 2]. For example, the document (or paper) can be                  lation between two views, and the latter [26–28] explores
                                         represented by different languages [3, 4]. Generally, multiple              complementary information based on HSIC term by mapping
                                         views can contain more meaningful and comprehensive in-                     variables into a reproducing kernel Hilbert space. Apart from
                                         formation than single view, which provide some knowledge                    these above works, co-training strategy [29, 30] is also con-
                                         information not involved in single view. The fact can show                  sidered as an effective tool to enforce different views to learn
                                         that, properly using multi-view information is beneficial for               from each other. Unfortunately, these methods may produce
                                         improving the practical performance in most situations. How-                unsatisfactory results when the gap of different views is
                                         ever, most traditional methods mainly focus on the single                   varying large. Although such methods have achieved excellent
                                         view, which cannot be directly used to deal with multi-view                 performances in some situations, most of them always focus
                                         data. Therefore, multi-view learning [5–7] methods has been                 on either the intrinsic structure information in each view or
                                         proposed in different tasks, such as classifications [8–10],                geometric structures from other views. To discover the geo-
                                         clustering [11, 12], etc. Notably, multi-view clustering [13–               metric structures information in multi-view data, those works
                                         15] has become an important research field, which aims at                   [31–35] have been developed to learn the intrinsic structure
                                                                                                                     in each view. However, the geometric structures information
                                         X. Meng is with Center for Research on Intelligent Perception and Comput-   in these works usually depend on artificial definition, which
                                         ing, Institute of Automation, Chinese Academy of Sciences, Beijing, China
                                         (xiangzhu.meng@cripac.ia.ac.cn).                                            cannot learn the geometric relation among samples from multi-
                                         W. Wei is with Center for Energy, Environment & Economy Research,           view data. To sum up, multi-view clustering is still an open
                                         Tourism Management School of Zhengzhou University, Zhengzhou, China         yet challenging problem.
                                         (weiwei123@zzu.edu.cn).
                                         W. Liu is with the School of Computer Science and Technology, Dalian            To solve these issues, we propose a novel multi-view
                                         University of Technology, Dalian, China (liuwenzhe@mail.dlut.edu.cn).       learning method, called Locality Relationship Constrained
Locality Relationship Constrained Multi-view Clustering Framework
IEEE TRANSACTIONS ON MULTIMEDIA, VOL. XXX, NO. XX, 2021                                                                                     2

Multi-view Clustering Framework (LRC-MCF), to explore the                                 II. R ELATED WORKS
problem of multi-view clustering. LRC-MCF simultaneously              In this section, we review two classical multi-view learning
explores the diversity, geometric, consensus and complemen-         methods, including Auto-Weighted Multiple Graph Learning,
tary information among different views, which attempts to           and Co-regularized Multi-view Spectral Embedding.
capture the locality relationship information in each view,
and then fuse the locality structure information in all views
                                                                    A. Auto-Weighted Multiple Graph Learning
into one common similarity relationships among multiple
views. In this way, we not only take use of the diversity             Auto-Weighted Multiple Graph Learning (AMGL)[17] is a
and geometric information among multiple views, but enforce         multi-view framework based on the standard spectral learn-
difference different views to learn with each other by the          ing, which can be used for multi-view clustering
                                                                                                                n         and semi-    o
common similarity relationships. Note that there is differ-         supervised classification tasks. Let X v = xv1 , xv2 , . . . , xvN
ent clustering performance for each view, thus LRC-MCF              denote the features set in the vth view. AMGL aims to find
adaptively assign the suitable weights for different views          the common embedding U ∈ RN ×k as follows:
in finding the common-view locality structure and straight-                                  M q
forwardly produce the final clusters. To effectually reduce
                                                                                             X
                                                                                        max        tr(U T Lv U )                     (1)
the redundancy of the learned representations, the low-rank                               U ∈C
                                                                                                 v=1
constraint on the common similarity matrix is additionally                   v
                                                                    where L denotes the normalized graph Laplacian matrix in
considered. To solve the minimization problem of LRC-MCF,
                                                                    the vth view, trace(·) denotes the trace of the matrix, and k is
the optimization algorithm based on the alternating direction
                                                                    the number of clusters. C deontes the different constraints in
minimization strategy is provided to iteratively calculate all
                                                                    the Eq. (1), if C is U T U = I, the framework is used to solve
variables LRC-MCF. Finally, extensive experimental results on
                                                                    the clustering tasks; if C contains the labels information, the
seven benchmark multi-view datasets show the effectiveness
                                                                    framework is also used for the semi-supervised classification
and superiority of the LRC-MCF method.
                                                                    task. Applying the Lagrange Multiplier method to solve the
                                                                    Eq. (1), the adaptive weight mechanism is derived to allocate
                                                                    suitable weights for different views. Therefore, the multi-view
A. Contributions                                                    framework can be transformed as the following problem:
  The major contributions of this paper can be summarized                                        M
                                                                                                 X                T
as follows:                                                                               max          αv tr(U Lv U )                     (2)
                                                                                          U ∈C
                                                                                                 v=1
  •   LRC-MCF simultaneously explores the diversity, geomet-
      ric, consensus and complementary information among            where
                                                                                                q             
      different views, by capturing the locality relationship in-
                                                                                        αv = 1/ 2 tr(U T Lv U )                           (3)
      formation and the common similarity relationships among
      multiple views.
                                                                    According to the Eq. (1), if the vth view is good, then
  •   LRC-MCF takes sufficient consideration to weights of                T
                                                                    tr(U Lv U ) should be small, and thus the learned weight
      different views in finding the common-view locality             v
                                                                    α for the vth view is large. Correspondingly, a bad view will
      structure, and adds the low-rank constraint on the com-
                                                                    be also assigned a small weight. Therefore, AMGL optimizes
      mon similarity matrix to effectually reduce the redun-
                                                                    the weights meaningfully and can obtain better result than the
      dancy of the learned representations.
                                                                    classical combination approach which assigns equal weight to
  •   To solve the minimization problem of LRC-MCF, an
                                                                    all the views.
      Alternating Direction Minimization (ADM) method is
      provided to iteratively calculate all variables LRC-MCF,
      which can convergence in limited iterations.                  B. Co-regularized Multi-view Spectral Clustering
  •   The experimental results on seven benchmark datasets             Co-regularized Multi-view Spectral Clustering (Co-reg)[36]
      demonstrate effectiveness and superiority of LRC-MCF,         is a spectral clustering algorithm under multiple views, which
      which outperforms its counterparts and achieves compa-        achieves this goal by co-regularizing the clustering hypotheses
      rable performance.                                            across views. Co-reg aims to propose a spectral clustering
                                                                    framework for multi-view setting. To achieve this goal, Co-reg
                                                                    works with the cross-view assumption that the true underlying
B. Organization                                                     clustering should assign corresponding points in each view
                                                                    to the same cluster. For the example of two-view case for
   The rest of the paper is organized as follows: in Section II,    the ease of exposition, the cost function for the measure of
we provide briefly some related methods which have attracted        disagreement between two clusters of the learned embedding
extensive attention; in Section III, we describe the construc-      U v and U w in the pth view and the qth view could be defined
tion procedure of LRC-MCF and optimization algorithm for            as follows:
LRC-MCF; in Section IV, extensive experiments on text and                                                                             2
image datasets demonstrate the effectiveness of our proposed                        v     w            KU v                KU w
                                                                              D (U , U ) =                    2       −           2       (4)
approach; in Section V, the conclusion is put forth finally.                                       kKU v kF               kKU w kF    F
Locality Relationship Constrained Multi-view Clustering Framework
IEEE TRANSACTIONS ON MULTIMEDIA, VOL. XXX, NO. XX, 2021                                                                                              3

where KU v and KU w denote the similarity matrix for the                        Inspired by sparse representation method, we attempt to
pth view and the qth view, respectively. For the convenience                 additionally add the sparsity constraint on S v , which encour-
of solving the solution, linear kernel is chosen as the similarity           ages the similarity matrix S v to have more zero elements.
measure. Co-reg builds on the standard spectral clustering by                Usually, L0 or L1 norm could be used to implement the
appealing to the co-regularized framework, which makes the                   sparsity constraint, but this manner might neglect the local
clustering relationships on different views agree with each                  structure in representation space. Based on the fact that the
other. Therefore, combining Eq. (4) with the spectral clustering             sample point could be inflected by its some nearest neighbors,
objectives of all views, we could get the following joint                    we use the local structure information to construct the sparsity
maximization problem for M views:                                            constraint for S v , thus we can obtain the following equation
                                                                             based on locality relationship:
                     M
                     X           T                  X
      min                  tr(U v Lv U v ) + λ              D (U v , V w )                      N X
                                                                                                X N
                                                                                                                                         2
 U 1 ,U 2 ,...,U M
                     v=1                         1≤v6=w≤M                                 min
                                                                                           v
                                                                                                         d(xvi , xvj )Si,j
                                                                                                                       v
                                                                                                                           + λ kS v kF
                                                                                           S
                           T                                                                    i=1 j=1
          s.t.        U v U v = I,1 ≤ v ≤ M                                                                                                        (8)
                                                                       (5)                s.t.si,j ≥ 0, 1T Siv
                                                                                               v
                                                                                                                  =1
                                                                                                 v
                                                                                              Si,j = 0, if xvj    / N (xvi )
                                                                                                                  ∈
where the first term is the spectral clustering objectives, λ
is a the non-negative hyperparameter to trade-off the spectral               where N (xvi ) denotes the K nearest neighbors set of xvi .
clustering objectives and the spectral embedding disagreement
terms across different views. In this way, Co-reg implements                 B. Multi-view information integration
a spectral clustering framework for multi-view setting.
                                                                                For the multi-view setting, a naive way is to incorporate all
                                                                             views directly according to the definition in Eq. (8), which
                     III. T HE P ROPOSED A PPROACH                           can be expressed as follows:
                                                                                             N X
                                                                                           M X N                                  M
A. Locality relationship preserving                                                        X                                      X          2
                                                                                     min
                                                                                      v
                                                                                                         d(xvi , xvj )Si,j
                                                                                                                       v
                                                                                                                             +λ         kS v kF
   Given two samples        andxvi     xvi
                                     in the vth view, the distance                    S
                                                                                          v=1 i=1 j=1                             v=1
                                                                                                                                                   (9)
between two samples is denoted as d(xvi , xvj ), which is usually                             v
                                                                                     s.t.∀v, Si,j ≥ 0, 1T Siv = 1
calculated by Mahalanobis distance. Eulidean distance, L1 -                                v
norm, etc. Intuitively, the distance between samples reflects the                        Si,j = 0, if xvj ∈
                                                                                                          / N (xvi )
relationship between samples, which is of vital importance to                However, this manner is equal to solve the problem for
construct the local geometric structure for the set of samples.              all views separately, which will fail to integrate multi-view
For the vth view, conventional similarity relationship between               features and make favorable use of the complementary infor-
                                          v             d(xv ,xv )
two samples is usually defined as Si,j       = exp (− iσ j ),                mation from multiple views. To solve this issue, we firstly
where σ is a hyper-parameter. However, one major disad-                      make such hypothesis that similarities among the instances in
vantage of this manner is that the hyper-parameter is hard                   each view and the centroid view should be consistent under the
to set in practice due to the noise and outliers in the data.                novel representations. This hypothesis means that all similarity
Therefore, how to learn the suitable similarity matrix is of vital           matrices from multiple views should be consistent with the
importance. Meanwhile, the learnt similarity matrix should be                similarity of the centroid view, which is implemented by
subject to such a condition that a smaller distance between                  aligning the similarities matrix computed from the centroid
two data points corresponds to a large similarity value, and a               view and the vth view. Therefore, we utilize the following
larger distance between two data points corresponds to a small               cost function as a measurement of disagreement between the
similarity value. Toward this end, we model the problem as                   centroid view and the vth view:
follows:                                                                                                                          2
                                                                                                D (S ∗ , S v ) = kS ∗ − S v kF                    (10)
                       N X
                       X N
                                                            2
                 min
                  v
                                 d(xvi , xvj )Si,j
                                               v
                                                   + λ kS v kF               where S ∗ denotes the similarity matrix in the centroid view.
                  S
                      i=1 j=1
                                                                       (6)
                                                                               To integrate rich information among different features, we
                      v
                 s.t.Si,j ≥0                                                 could obtain the following optimization problem by adding up
                                                                             cost function in the Eq. (10) from all views:
where the second term is a regularization term on S v , and
                                                                                                        M
λ is a hyperparameter to balance these two terms. To further                                            X                    2
analyze S v , we add the normalization on S v in the Eq. (6),                                     min
                                                                                                   ∗
                                                                                                               kS ∗ − S v kF
                                                                                                   S
                                                                                                        v=1
i.e. 1T svi = 1, which makes the second term constant. Then,                                             ∗
                                                                                                                                                  (11)
the above problem can be transformed as follows:                                                 s.t.   Si,j   ≥ 0, 1T Si∗ = 1

                       N X
                       X N
                                                             2
                 min
                  v
                                 d(xvi , xvj )Si,j
                                               v
                                                   + λ kS v kF                 In most practical applications, different views cannot play
                 S
                    i,j=1 j=1                                          (7)   the same role in fusing different similarity matrixes. To reflect
                    v
               s.t.Si,j ≥ 0, 1T Siv     =1                                   the importance of different features in integrating multi-view
Locality Relationship Constrained Multi-view Clustering Framework
IEEE TRANSACTIONS ON MULTIMEDIA, VOL. XXX, NO. XX, 2021                                                                                              4

information, we can obtain the following optimization problem                   D. Optimization process
by allocating ideal weights w for all views:                                       Obviously, all the variables in the Eq. (14) are coupled
                                M                                               together, thus solving the multi-view clustering problem to
                                X                    2
                        min          wv kS ∗ − S v kF                           optimize all variable at once is still challenging. In addition,
                        w,S ∗                                                   the constraints are not smooth. To overcome these issues, we
                                v=1
                                                                         (12)
                        s.t.     ∗
                               Si,j ≥    0, 1T Si∗ = 1                          propose a novel algorithm based on the Alternating Direc-
                                                                                tion Minimization strategy. Under the assumption that other
                               w ≥ 0, 1T w = 1
                                v
                                                                                variables have been obtained, we can calculate S ∗ via the
where wv can reflect the importance of the vth view in                          Augmented Lagrange Multiplier scheme. That is to say, one
obtaining the common similarity matrix S ∗ .                                    variable is updated when the other variables are fixed, which
                                                                                inspires us to develop an alternating iterative algorithm to solve
                                                                                problem. The specific updated rules are shown below:
C. Multi-view clustering based on Locality relationship pre-                       Fixing w and S ∗ , update S v : When w and S ∗ are given,
serving                                                                         graph matrix S v in the Eq. (14) is independent. Thus, we can
                                                                                update S v one by one, formulated as follows:
   To simultaneously exploit the diversity and consensus infor-
                                                                                                      N X
                                                                                                        N
mation among different views, we can obtain the multi-view                                            X                               2
                                                                                             min          d(xvi , xvj )Si,j
                                                                                                                        v
                                                                                                                            + λ kS v kF
clustering framework by combining the Eq. (8) and Eq. (12):                                   v
                                                                                              S
                                                                                                      i=1 j=1
                                                                                                                      2                          (15)
                                                                                              + wv kS ∗ − S v kF
                   M X
                     N X
                       N                                 M                                          v
                   X                                     X          2                        s.t. Si,j ≥ 0, 1T Siv = 1
       min                       d(xvi , xvj )Si,j
                                               v
                                                   +λ          kS v kF
                                                                                                   v
   w,S ∗ ,{S v }
                   v=1 i=1 j=1                           v=1                                     Si,j = 0, if xvj ∈
                                                                                                                  / N (xvi )
       M
       X            r                2                                          It’s easy to find that the above equation is independent
   +         (wv ) kS ∗ − S v kF
                                                                                for different i. Thus, we can solve the following problem
       v=1                                                               (13)   separately for each Siv :
         ∗
   s.t. Si,j > 0, 1T Si∗ = 1
                                                                                                        N
       wv > 0, 1T w = 1                                                                                 X                                2
                                                                                                min           d(xvi , xvj )Si,j
                                                                                                                            v
                                                                                                                                + λ kS v kF
       ∀v, svi,j > 0, 1T svi = 1                                                                 vS
                                                                                                        j=1
        v
       Si,j = 0, if xvj ∈
                        / N (xvi )                                                              + wv kSi∗ − Siv k2
                                                                                                                          2                      (16)
                                                                                                     v
                                                                                               s.t. Si,j ≥ 0, 1T Siv = 1
   Our paper aims to solve the final clustering problem, i.e.,
                                                                                                       v
producing the clustering result directly on the unified graph                                         Si,j = 0, if xvj ∈
                                                                                                                       / N (xvi )
matrix S ∗ without an additional clustering algorithm or step.
                                                                                By relaxing the constraint, we can transform the above equa-
So far, the unified graph matrix S ∗ cannot tackle this problem.
                                                                                tion as follows: Siv :
Now we give an efficient and yet simple solution to achieve
this goal by imposing a rank constraint on the graph Laplacian                                          N
                                                                                                        X                                2
matrix L∗ of the unified matrix S ∗ . The works imply that if                                     min
                                                                                                   v
                                                                                                              d(xvi , xvj )Si,j
                                                                                                                            v
                                                                                                                                + λ kSiv k2
                                                                                                  S
rank(L∗ ) = N − c, the corresponding S ∗ is an ideal case                                               j=1
                                                                                                                                                 (17)
                                                                                                                          2
based on which the data points are partitioned into c clusters                                    + wv kSi∗ − Siv k2
directly. Thus, there is no need to run an additional clustering                                     v
                                                                                               s.t. Si,j ≥ 0, 1T Siv = 1
algorithm on the unified graph matrix S ∗ to produce the final
clusters. Motivated by this theorem, we add a rank constraint                   Using Lagrangian multiplier method and KKT condition to
                                                                                                                                    v
to the above problem. Then our final multi-view clustering                      solve it, we can obtain the following solution for Si,j :
model can be formulated as follows:
                                                                                                v
                                                                                                          −d(xvi , xvj ) + 2wv + 2λη
                   M X
                     N X
                       N                                 M                                     Si,j =                                            (18)
                   X                                     X          2                                              2λ + 2wv
       min
       ∗
                                 d(xvi , xvj )Si,j
                                               v
                                                   +λ          kS v kF
   w,S ,{S v }
                   v=1 i=1 j=1                           v=1                    where η is Lagrangian coefficient of the constraint 1T svi = 1.
       M
       X                                                                        Considering the locality constraint that svi,j must be equal to 0
                                 2
   +         wv kS ∗ − S v kF                                                   if xvj ∈/ N (xvi ), we can further solve the η and λ in the above
       v=1                                                                      equation. Finally, we can get the following updated rule to
         ∗
   s.t. Si,j > 0, 1T Si∗ = 1                                             (14)   solve S v :
                                                                                                                                   ∗     ∗
       wv > 0, 1T w = 1                                                                              d(xv    v         v v     v
                                                                                                         i ,xî )−d(xi ,xj )+2w )(Si,j −Si,î )
                                                                                            
                                                                                             K(d(xv ,xv )−2S ∗ )+PK (2wv )S ∗ −d(xv ,xv )) ,
                                                                                            
            v
       ∀v, Si,j > 0, 1T Siv = 1                                                       v
                                                                                   Si,j   =
                                                                                                      i   î      i,î    k=1      i,j        i j

                                                                                                 if xvj ∈ N (xvi )
        v
            = 0, if xvj ∈
                        / N (xvi )
                                                                                            
       Si,j                                                                                 
                                                                                                0, otherwise
       rank(L∗ ) = N − c                                                                                                                          (19)
Locality Relationship Constrained Multi-view Clustering Framework
IEEE TRANSACTIONS ON MULTIMEDIA, VOL. XXX, NO. XX, 2021                                                                                                         5

where î is the index of the (K + 1)th nearest neighbor of xi .                                  Fixing {S v } and S ∗ , update w: When {S v } and S ∗ are
In this way, we can sequentially update the similarity matrix                                 given, the objective function on w can be reduced as follows:
in each view by the above equation.                                                                                     M
   Fixing w and {S v }, update S ∗ : When w and {S v } are
                                                                                                                        X                    2
                                                                                                                  min         wv kS ∗ − S v kF
given, the objective function in the Eq. (14) on S ∗ can be                                                        w
                                                                                                                        v=1
                                                                                                                                                             (26)
reduced as follows:                                                                                               s.t. w > 0, 1T w = 1
                                                                                                                          v

               M
               X             r                     2                                          The solution to w in the above equation is wv = 1 corre-
        min
         ∗
                      (wv ) kS ∗ − S v kF                                                     sponding to the min(D (S ∗ , S v )) over different views, and
         S
               v=1
                                                                                       (20)
                ∗
                                                                                              wv = 0 otherwise. This solution means that only one view is
        s.t.   Si,j    > 0, 1    T
                                     Si∗                     ∗
                                           = 1, rank(L ) = N − c                              finally selected by this method. Therefore, the performance of
                                                                                              this method is equivalent to the one from the best view. This
It is difficult to solve the above equation because the constraint
                                                                                              solution does not meet our objective on exploring the comple-
rank(L∗ ) = N − c is nonlinear. Inspired by the work that
                                                                                              mentary property of multiple views to get a better embedding
L∗ is positive
            Pc semi-definite, the low-rank constraint can be                                  than based on a single view. To avoid this phenomenon, we
achieved i=1 θi = 0, where θi is the ith smallest eigenvalue                                                                                               r
                                                                                              adopt a simple yet effective trick, i.e. we set wv ← (wv )
of L∗ . According to Ky Fan’s theorem [37], we can obtain the
                                                                                              with r ≥ 1. In this way, the novel objective function can be
following equation by combining it with the above equation:
                                                                                              transformed as follows:
               M
               X                                                                                                       M
                             r                     2
                      (wv ) kS ∗ − S v kF + βtr(F L∗ F T )
                                                                                                                       X          r              2
        min
         ∗
                                                                                                               min           (wv ) kS ∗ − S v kF
        S ,F
               v=1
                                                                                       (21)                       w                                          (27)
                                                                                                                       v=1
                ∗
        s.t.   Si,j >    0, 1    T
                                     Si∗   = 1, F F     T
                                                            =I                                                 s.t.     wv ≥ 0, 1T w = 1
To solve the above equation, we iteratively solve S ∗ and F .                                   By using the Lagrange multiplier γ to take the constraint
When F is fixed, the above equation on S ∗ can be transformed                                 1T w = 1 into consideration, we get the following Lagrange
as follows:                                                                                   function:
                                                                                                            M                                 M
               M                                                                                            X           r              2
                                                                                                                                              X
        min
               X
                         v r
                      (w ) kS ∗ −
                                                    2
                                               S v kF   + βtr(F L∗ F T )                        L(w, γ) =         (wv ) kS ∗ − S v kF − γ(           wv − 1) (28)
         ∗
         S
               v=1
                                                                                       (22)                 v=1                               v=1
                ∗
        s.t.   Si,j    > 0, 1T Si∗ = 1                                                        By setting the partial derivatives of L(w, γ) with respect to
                                                                                              w and γ to zeros, w can be calculated as follows:
For solving S ∗ , the above equation can be rewritten as follows:
                                                                                                                                      2 1/(1−r)
                                                                                                                  (kS ∗ − S v kF )
        M X
          N                                                   N                                            wv = P                   1/(1−r)
                                                                                                                                                             (29)
        X
                         v r       ∗            v 2
                                                              X
                                                                      ∗                   2                      M       ∗      v 2
 min
  ∗
                      (w )       (Si,j     −   Si,j )   +β           Si,j (fi∗   −   fj∗ )                       i=1 (kS − S kF )
  si
        v=1 i,j=1                                            i,j=1                                                      2
         ∗
                                                                                              Obviously, kS ∗ − S v kF is a non-negative scala, thus we have
 s.t.   Si,j > 0, 1T Si∗         =1                                                           wv ≥ 0 naturally. According to the above equation, we have
                                                                                       (23)   the following understanding for r in controlling w. When
                                                                                              r → ∞, different wv will be close to each other. When
Let e denote the Eulidian distance matrix of F , i.e. ei,j =
            2                                                                                 r → ∞, only wv = 1 corresponding to the minimum
(fi∗ − fj∗ ) , it’s easy to prove that the above optimal problem
                                                                                              value of D (S ∗ , S v ) over different views, and wv = 0
is equal to the following equation:
                                                                                              otherwise. Therefore, the selection of r should be based on
                        M X
                          N                                             2                     the complementary property of all views.
                        X                                     β
                min                        Si∗ − Siv +           r ei                            To sum up, we can iteratively solve the all variables in Eq.
                 ∗si
                        v=1 i=1
                                                            (wv )       2              (24)   (14) according to the aforementioned descriptions, which ob-
                         ∗
                s.t.    Si,j > 0, 1T Si∗           =1                                         tain a local optimal solution of LRC-MCF. For the convenient
                                                                                              of readers, we summarize the whole optimization process in
The Eq. (24) is a standard quadratic constraint problem, which                                the Algorithm 1. Since problem is not a joint convex problem
can be effectively solved by optimization tools. When S ∗ is                                  of all variables, obtaining a globally optimal solution is still
fixed, the above equation on F can be simplified as follows:                                  an open problem. We solve problem using an alternating
                                                                                              algorithm (as Algorithm 1). As each sub-problem is convex
                                     min tr(F L∗ F T )                                        and we find the optimal solution of each sub-problem, thus
                                      F                                                (25)
                                 s.t.F F T = I                                                the algorithm converges because the objective function reduces
                                                                                              with the increasing of the iteration numbers. In particular,
Obviously, L∗ is a symmetric positive-definite matrix. Based                                  with fixed other variables, one optimal variable can reduce the
on the Ky-Fan theory, F in Eq.(25) has a global optimal                                       value of the objective function. Moreover, we will empirically
solution, which is given as the eigenvectors associated with                                  validate the convergence of our algorithm in the following
the c smallest eigenvalues of L∗ .                                                            experiments.
Locality Relationship Constrained Multi-view Clustering Framework
IEEE TRANSACTIONS ON MULTIMEDIA, VOL. XXX, NO. XX, 2021                                                                                           6

  Algorithm 1: The optimization procedure for LRC-                     A. Datasets
  MCF                                                                     To comprehensively demonstrate the effectiveness of the
   Input: Multi-view features {X v , ∀1 ≤ v ≤ M }, the                 proposed LRC-MCF, we perform all multi-view cluster-
            hyper-parameters α and β, the distance metric              ing methods on seven benchmark multi-view datasets, in-
            function d(·, ·), the number of clustering c.              cluding BBC1 , BBCSport2 , 3Source3 , Cora4 , Handwritten5 ,
 1 for v=1:M do                                                        HW2sources6 , and NGs7 . BBC is a documents dataset con-
 2     Initialize d(xvi , xvj ) according to the distance              sisting of 685 documents, where each document is received
        metric function d(·, ·).                                       from BBC news corpora, corresponding to the sports news
 3     Initialize S v by solving the Eq. (8).                          in five topical areas, and when we treat each topical area
 4     Initialize wv = 1/M.                                            as one view, BBC can be regarded as a dataset with five
 5 end                                                                 views; BBCSport consists of 544 documents received from
              v        v               ∗
 6 Fixing w and S , initialize S by solving the Eq.                    the BBC Sport website, where each document is corresponding
    (24).                                                              to the sports news in five topical areas, and BBCSport is
 7 while all variables not converged do                                also can be regarded as a dataset with five views; 3Source
 8     for v=1:M do                                                    contains 169 news, collected from three well-known news
 9          Fixing wv and S ∗ , update S v by the Eq. (19).            organizations (including BBC, Reuters, and Guardian), where
10          Fixing S v and S ∗ , update wv by the Eq. (29).            each news is manually annotated with one of six labels;
11     end                                                             Cora contains 2708 scientific publications of seven categories,
12     Fixing wv and S v , update S ∗ by solving the Eq.               where each publication document can be described by content
        (24).                                                          and citation, thus Cora could be considered as a two-view
13     Fixing S ∗ , update F by solving the Eq. (25).                  benchmark dataset; Handwritten is a hand-written dataset
14 end                                                                 consisting of 2000 samples, where each sample is represented
   Output: The common similarity relationship matrix                   by six different features; HW2sources is a benchmark dataset
              S∗.                                                      from the UCI repository, including 2000 samples, which is
                                                                       collected two sources including Mnist hand-written digits and
                                                                       USPS hand-written digits; NGs is a dataset consisting of
E. Computational Complexity Analysis                                   500 documents, where each document is pre-proposed by
                                                                       three different methods and manually annotated with one
   To clearly show the efficiency of LRC-MCF, we provide its           of five labels. For convenience, we summarize the statistic
computational complexity analysis in this section. Obviously,          information on these datasets in Table I.
the computational complexity to solve the all variables in
Eq. (14) is mainly dependent on the following five parts:              TABLE I: The summary information of benchmark datasets
the computational complexity to initialize all variables in              Datasets                Samples             Classes            Views
Eq. (14) is O(M N 2 d + M N K), where d is the computa-
                                                                         BBC                        685                 5                 4
tional complexity to the distance metric function d(·, ·); the
                                                                         BBCSport                   544                 5                 2
computational complexity to update S v by the Eq. (19) is
O(M N K); the computational complexity to update S ∗ by                  3Source                    169                 6                 3
solving the Eq. (24) is O(CN ); the computational complexity             Cora                      2708                 7                 2
to update F is O(CN 2 ); the computational complexity to                 Handwritten               2000                 10                6
update wv by the Eq. (29) is O(M N 2 ). Accordingly, the                 HW2sources                2000                 10                2
overall computational complexity of Algorithm 1 is about                 NGs                        500                 5                 3
O(M N 2 d + M N K + T (M N K + CN 2 + CN + M N 2 )),
where T is the iteration times of the alternating optimization
procedure in the Algorithm 1.
                                                                       B. Compared Methods
                                                                         We compare the proposed LRC-MCF with the following
                     IV. E XPERIMENTS                                  baseline methods on the above seven benchmark datasets,
                                                                       including two single-view learning methods, one feature con-
  To evaluate the effectiveness and superiority of the proposed        catenation method, and eight multi-view learning methods.
LRC-MCF, we conduct comprehensive clustering experiments               The detail information of these comparing methods can be
on seven benchmark multi-view datasets in this section. Firstly,       summarized as follows:
we introduce the details of the experiments, including bench-      1   http://mlg.ucd.ie/datasets/segment.html
mark datasets, compared methods, experiment setting, and           2   http://mlg.ucd.ie/datasets/segment.html
                                                                   3
evaluation metrics. Then, we perform all methods on different          http://mlg.ucd.ie/datasets/3sources.html
                                                                   4   3http://lig-membres.imag.fr/grimal/data.html
multi-view datasets to evaluate the performance of LRC-MCF         5   https://archive.ics.uci.edu/ml/datasets/One-hundred+palnt+pecies+leaves+data
and analysis the experimental results. Finally, we empirically         +set
demonstrate the sensitivity analysis and the parameter analysis    6   https://cs.nyu.edu/roweis/data.html
of LRC-MCF.                                                        7   http://lig-membres.imag.fr/grimal/data.html
Locality Relationship Constrained Multi-view Clustering Framework
IEEE TRANSACTIONS ON MULTIMEDIA, VOL. XXX, NO. XX, 2021                                                                               7

  •   Single-view learning methods: Spectral Clustering (SC)          methods for given datasets to obtain the corresponding repre-
      [38] first computes the affinity between each pair of           sentations and then utilize K-means clustering [45] to cluster
      points to construct the similarity matrix, and then make        all samples; for the proposed LRC-MCF, we directly perform
      use of the spectrum (eigenvalues) of the similarity matrix      the clustering process on the common similarity relationship
      to perform dimensionality reduction before clustering           matrix S ∗ solved by the Algorithm 1. Since the clustering [45]
      in fewer dimensions; Graph regularized Non-negative             algorithm is known to be sensitive to initialization, We run all
      Matrix Factorizatio (GNMF) [39] aims to yield a natural         methods repeatedly 10 times and then obtain the mean value
      parts-based representation for the data and jointly dis-        of all indicators for all methods. It’s worthy noting that all
      cover the intrinsic geometrical and discriminating struc-       experiments are performed in the same environment.
      ture of the data space.                                            Given the label information of all samples, three validation
  •   Feature concatenation method: We first concatenates             metrics are adapted to validate the performance of clustering
      the features of all views and then employs the traditional      method, including accuracy (ACC), normalized mutual infor-
      clustering method to obtain the clustering results.             mation (NMI), and purity (PUR). Specifically, ACC is defined
  •   Multi-view learning methods: Canonical Correlation              as follows:
      Analysis (CCA) [21] is utilized to deal with multi-view                                   PN
                                                                                                        δ(yi , map(ŷi ))
      problems by maximizing the cross correlation between                             ACC = i=1                                   (30)
                                                                                                            N
      each pair-views; Multi-view Spectral Embedding (MSE)
      [16] is a multi-view spectral method, which is based            where y i and ŷi are the truth label and cluster label, respec-
      on global coordinate alignment to obtain the common             tively. map(·) is the permutation function that maps cluster
      representations; Multi-view Non-negative Matrix Factor-         label to truth label, and δ(·, ·) is the function that equals one
      ization (MultiNMF) is proposed in work [40] to jointly          if two inputs are equal, and zero otherwise. Meanwhile, NMI
      learn non-negative potential representations from multi-        is formulated as follows:
      view information and obtain the clustering results by
                                                                                              PK PK                   Ni,j
                                                                                                 i=1     j=1 Ni,j log Ni N̂j
      the latent common representations; Co-regularized Multi-                   NMI = qP                                          (31)
                                                                                               K           Ni PK             N̂j
      view Spectral Clustering (Co-reg) is a multi-view spec-                                  i=1 Ni log N      j=1 Nj log N
      tral clustering method proposed in work [36], which
      enforces different view to mutually learn by regular-           where Ni is the number of samples in the ith cluster, N̂j is
      izing different views to be close to each other; Auto-          the number of samples in the jth class, and Ni,j is the number
      weighted multiple graph learning (AMGL) [17] method,            of samples in the intersection between the ith cluster and the
      which automatically assigns an ideal weight for each            jth class. Finally, PUR is expressed as follows:
      view in integrating multi-view information; Multi-view                                       K
                                                                                               1 X
      Dimensionality co-Reduction (MDcR) [41] adopts the                             PUR =          max Ai ∩ Âj                  (32)
      kernel matching constraint based on Hilbert-Schmidt                                      N   1≤j≤K
                                                                                                  i=1
      Independence Criterion to enhance the correlations and
                                                                      where Ai and Âj are two sets with responding to the ith class
      penalizes the disagreement of different views; AASC
                                                                      and the jth cluster, and |·| is the length of set.
      [42] is proposed by aggregating affinity matrices, which
                                                                         Note that the higher values indicate better clustering perfor-
      attempt to make spectral clustering more robust by re-
                                                                      mance for all evaluation metrics. Different metrics can reflect
      lieving the influence of inefficient affinities and unrelated
                                                                      different properties in the multi-view clustering task, thus we
      features; WMSC [43] takes advantage of the spectral
                                                                      summarize all results on these measures to obtain a more
      perturbation characteristics of spectral clustering, uses the
                                                                      comprehensive evaluation.
      maximum gauge angle to estimate the difference between
      the impacts of distinct spectral clustering, and finally
      transforms the weight solution problem into a standard          D. Performance Evaluation
      secondary planning problem; AWP [44] is an extension               In this section, to show the effectiveness of LRC-MCF,
      of spectral rotation for multiview data with Procrustes         extensive clustering experiments have been conducted on
      Average, which takes the clustering capacity differences        seven multi-view datasets (BBC, BBCSport, 3Source, Cora,
      of different views into consideration..                         Handwritten, and NGs), and the experimental results are
                                                                      summarized in Table II, Table III and Table IV, where the bold
                                                                      values represent the superior performance in each table. To
C. Experiment Setting and Evaluation Metrics                          comprehensively show the performance of clustering methods,
   To implement the clustering of the above datasets, we              we additionally add up one new column of MEAN in each
perform all comparing methods and the proposed LRC-MCF                table, which averages the evaluation indexes of all benchmark
on seven multi-view datasets. Specifically, for single-view           datasets for each clustering method.
learning methods, we choose the best ones from the clustering            For BBC and BBCSports datasets, the experimental results
results of different views, i.e. SCbest and GNMFbest ; for            show that LRC-MCF can performs better than comparing
feature concatenation method, we utilize standard Spectral            single-view and multi-view clustering methods in most sit-
Clustering method [38] to deal with the concatenated features;        uations, especially in terms of ACC and PUR. This validates
for multi-view learning methods, we perform all multi-view            the effectiveness of LRC-MCF in the domain of sports news
Locality Relationship Constrained Multi-view Clustering Framework
IEEE TRANSACTIONS ON MULTIMEDIA, VOL. XXX, NO. XX, 2021                                                              8

                            TABLE II: ACC(%) results on seven benchmark datasets for all methods
                        Datasets
                                   BBC     BBCSport   3Source     Cora    Handwritten   HW2sources   NGs     MEAN
             Methods
             SCbest                43.94    50.74         39.64   35.82      92.82        58.50      28.96   50.06
             GNMFbest              32.12    32.12         45.56   31.91      73.80        51.10      23.01   41.37
             SCconcat              47.26    67.22         36.45   32.14      60.03        76.24      35.98   50.76
             CCA                   48.12    38.60         41.01   44.15      73.63        74.51      27.62   49.66
             MSE                   39.26    54.01         44.85   31.69      79.84        73.27      27.55   50.07
             MultiNMF              43.36    50.00         52.66   35.60      88.00        76.75      33.61   54.28
             Co-reg                40.73    61.58         45.56   39.11      91.00        82.15      27.45   55.37
             AMGL                  39.21    55.22         42.31   32.05      83.34        90.38      30.83   53.33
             MDcR                  81.05    80.24         70.53   54.51      61.26        86.56      59.06   70.45
             AASC                  38.39    62.32         36.69   33.09      84.65        83.30      71.00   58.49
             WMSC                  42.92    61.21         42.60   42.25      84.50        83.40      43.00   57.12
             AWP                   37.37    44.85         51.48   32.39      96.70        93.85      23.80   54.34
             LRC-MCF               87.45    80.70         78.70   51.18      88.25        99.45      98.20   83.42

                            TABLE III: NMI(%) results on seven benchmark datasets for all methods
                        Datasets
                                   BBC     BBCSport   3Source     Cora    Handwritten   HW2sources   NGs     MEAN
             Methods
             SCbest                23.04    21.20         33.53   16.35      87.78        59.41      7.39    35.53
             GNMFbest              21.43    19.99         29.39   19.14      74.37        57.49      8.85    30.09
             SCconcat              35.45    52.10         35.72   19.22      61.14        80.57      22.75   43.85
             CCA                   30.93    25.30         35.64   26.74      68.15        60.83      13.35   37.28
             MSE                   26.93    34.71         46.61   18.58      80.13        72.17      13.58   41.82
             MultiNMF              27.57    44.69         43.98   26.28      80.25        68.69      20.37   44.55
             Co-reg                23.36    41.22         44.77   22.50      88.33        85.61      7.66    46.21
             AMGL                  26.90    39.92         45.67   18.77      81.08        88.27      19.05   45.67
             MDcR                  61.47    63.94         64.43   31.30      66.65        77.71      50.14   59.34
             AASC                  20.14    39.11         30.21   16.63      87.48        86.14      48.02   46.81
             WMSC                  22.10    47.12         42.00   27.96      87.44        86.50      24.84   48.28
             AWP                   20.06    23.15         41.53   12.26      92.08        91.00      3.88    40.57
             LRC-MCF               73.62    76.00         70.50   37.85      92.03        98.63      93.92   77.50

                           TABLE IV: PUR(%) results on seven benchmark datasets for all methods
                        Datasets
                                   BBC     BBCSport   3Source     Cora    Handwritten   HW2sources   NGs     MEAN
             Methods
             SCbest                49.64    55.70         56.80   40.73      93.29        62.75      29.44   55.48
             GNMFbest              33.43    32.99         56.80   38.40      75.10        59.10      24.88   45.81
             SCconcat              51.42    68.66         55.38   38.43      61.62        79.33      38.86   56.24
             CCA                   52.63    46.47         54.97   50.37      73.69        74.51      29.33   54.57
             MSE                   45.40    56.58         65.56   37.98      80.71        76.96      28.70   55.98
             MultiNMF              43.65    63.42         60.36   49.74      88.00        76.75      35.99   59.70
             Co-reg                49.20    66.91         62.72   45.27      91.00        86.70      28.68   61.50
             AMGL                  45.40    58.93         64.97   38.30      83.34        91.22      32.96   59.30
             MDcR                  81.05    80.24         76.75   56.19      67.22        86.56      69.30   73.90
             AASC                  46.42    67.10         55.03   40.99      87.10        85.85      71.00   64.78
             WMSC                  48.61    68.01         60.36   47.16      87.10        86.10      46.00   63.33
             AWP                   45.69    51.10         63.31   35.60      96.70        93.85      24.40   58.66
             LRC-MCF               87.45    84.38         82.25   52.22      88.05        99.45      98.20   84.57
Locality Relationship Constrained Multi-view Clustering Framework
IEEE TRANSACTIONS ON MULTIMEDIA, VOL. XXX, NO. XX, 2021                                                                           9

clustering, and shows that LRC-MCF can capture the locality        there exists a wide range for each hyper-parameter in which
relationship information and the common similarity relation-       relatively stable and good results can be readily obtained.
ships among multiple views. Besides, multi-view methods
cannot always obtain better performance than single-view           F. Convergence Analysis
methods, which implies that the diversity and complementary           In this section, an iteration updating scheme is adopted by
information cannot be readily exploited.                           our proposed framework to obtain the optimal solution, thus
   For 3Source and Cora datasets, the experimental results         convergence analysis is conducted to verify the convergence
show that LRC-MCF can obtain the best performance in terms         property of LRC-MCF in detail. We conduct various exper-
of ACC, NMI, and PUR. Besides, most multi-view methods             iments on four datasets including BBCSport, Cora, Hand-
outperform single-view methods, which might imply that the         written and NGs. Fig. 2 plots the curves of objective values
diversity information among multiple views is benefit for          of LRC-MCF with the increase of iterations with respond-
the improvement of clustering performance. However, these          ing to the above four datasets. According to the variation
methods cannot get better than RC-MCF, which might be              trend of objective values of LRC-MCF in these sub-figures
caused by the issue that the geometric and complementary           of Fig.2(a)-Fig.2(d), we can readily find that the objective
information cannot be made full use of.                            value decreases rapidly during the iterations on all the four
   For Handwritten and HW2sources datasets, the experimen-         benchmark datasets. Moreover, it’s not difficult to observe
tal results also show that LRC-MCF can performs better than        that the convergence can be reached within the 10 steps of
comparing single-view and multi-view clustering methods in         iterations. That is to say, LRC-MCF can converge within a
most situations, which can validate the superiority of LRC-        limited number of iterations. It also might imply that LRC-
MCF in the domain of image clustering. It implies that the         MCF can effectually reduce the redundancy of the learned
locality relationship information in image clustering tasks is     representations. However, when facing the complex and large-
the important factor. Thus, how to exploit the common locality     scale datasets, it’s necessary to combine it with deep learning
information among multiple views is of great significance.         technology [46, 47] to construct the distance function. In the
   For NGs dataset, the experimental results show that LRC-        future, we consider how to extend superior deep models into
MCF can obtain the best performance in terms of ACC, NMI,          our work, which can be optimized by the end-to-end manner.
and PUR. This further validates the effectiveness of LRC-
MCF, which shows that LRC-MCF can fully explore the di-
                                                                   G. Discussion
versity, geometric, consensus and complementary information
among different views and take sufficient consideration to            As the experiment results in Tables II - IV on multi-
weights of different views in finding the common-view locality     view clustering tasks, we can clearly find that LRC-MCF
structure.                                                         outperforms other comparing methods in most situations. From
   Finally, the columns of MEAN in Tables II - IV validate the     the above evaluations, it’s readily seen that the embeddings
comprehensive performance of LRC-MCF, the experimental             obtained by our method could be more effective and suitable
results show that LRC-MCF can performs better than com-            for multi-view features. Besides, other multi-view methods
paring clustering methods in terms of ACC, NMI, and PUR.           outperform the other single-view methods in most situations,
It can imply that LRC-MCF not only takes full of the locality      which could show multi-view learning is a valuable research
geometric structure and similarity relationship among samples      field indeed. Compared to single view methods, taking the
under the multi-view scenario, but explores the diversity,         complementary information among different views into con-
consensus and complementary information among different            sideration could achieve more excellent performance, and the
views. Thus, LRC-MCF can obtain the best performance on            key is how to integrate the complementary information among
seven benchmark datasets in most situations.                       different views while preserving its intrinsic characteristic in
                                                                   each view. According to Fig. 1, we find that the number K
                                                                   of nearest neighbors used in our method is relatively smooth
E. Parameter Analysis                                              to the K over the relatively large ranges of values. Note
   The proposed LRC-MCF is based on the locality relation-         that the experimental results of our proposed LRC-MCF on
ships constraint, thus we need to mainly discuss the sensitivity   seven datasets are without fine-tuning, and usage of fine-tuning
analysis on the number K of nearest neighbors used in LRC-         might further improve its performance. For the discussion
MCF in this section. Specifically, we conduct the clustering       of convergence, according to Fig.2, it could be empirically
experiments for LRC-MCF on BBC, 3Source, HW2sources                verified that the proposed LRC-MCF converges once enough
and NGs datasets, then the average value of all evaluation         iterations are completed. That is, LRC-MCF could converge
indexes is used as final criteria. Fig. 1 plots the clustering     within a limited number of iterations.
results in terms of ACC, NMI, and PUR on these four datasets,
under different K in {10, 20, 30, 40, 50, 60, 70, 80, 90,                                V. C ONCLUSION
100, 110, 120, 130}. Through these variation trends of ACC,           In this paper, we propose a novel multi-view learning
NMI, and PUR over these four datasets, it’s easy to find that      method to explore the problem of multi-view clustering,
our method is relatively smooth to the K over the relatively       called Locality Relationship Constrained Multi-view Clus-
large ranges of values, which indicates that the performance is    tering Framework (LRC-MCF). LRC-MCF attempts to cap-
not so sensitive to those hyper-parameters. More importantly,      ture the relationship information among samples under the
Locality Relationship Constrained Multi-view Clustering Framework
IEEE TRANSACTIONS ON MULTIMEDIA, VOL. XXX, NO. XX, 2021                                                                      10

          (a) ACC in BBC dataset                        (b) NMI in BBC dataset                   (c) PUR in BBC dataset

         (d) ACC in 3Source dataset                    (e) NMI in 3Source dataset            (f) PUR in 3Source dataset

       (g) ACC in Handwritten dataset                (h) NMI in Handwritten dataset         (i) PUR in Handwritten dataset

           (j) ACC in NGs dataset                       (k) NMI in NGs dataset                   (l) PUR in NGs dataset

                                      Fig. 1: K parameter analysis of LRC-MCF on four datasets
IEEE TRANSACTIONS ON MULTIMEDIA, VOL. XXX, NO. XX, 2021                                                                        11

                         (a) BBCSport dataset                                           (b) Cora dataset

                        (c) Handwritten dataset                                         (d) NGs dataset

                                     Fig. 2: Objective values of LRC-MCF on four datasets

constraint of locality structure and integrate the relationship                       ACKNOWLEDGMENT
from different views into one common similarity relationships
matrix. Moreover, LRC-MCF provide an adaptive weight                 The authors would like to thank the anonymous review-
allocation mechanism to take sufficient consideration of the      ers for their insightful comments and the suggestions to
importance of different views in finding the common-view          significantly improve the quality of this paper. The work
similarity relationships. To effectually reduce the redundancy    was supported by the National Natural Science Founda-
of the learned representations, the low-rank constraint on        tion of PRChina(72001191), Henan Natural Science Founda-
the common similarity matrix is additionally considered.          tion(202300410442), Henan Philosophy and Social Science
Correspondingly, this paper provides an algorithm based on        Program(2020CZH009)is gratefully acknowledged. We also
the alternating direction minimization strategy to iteratively    would like to thank the anonymous.
calculate all variables of LRC-MCF. Finally, we conduct
extensive experiments on seven benchmark multi-view datasets                              R EFERENCES
to validate the performance of LRC-MCF, and the experi-
mental results show that the proposed LRC-MCF outperforms          [1] C. Zhang, J. Cheng, and Q. Tian, “Multi-view image clas-
its counterparts and achieves comparable performance. In               sification with visual, semantic and view consistency,”
the future, we will incorporate our proposed LRC-MCF and               IEEE Transactions on Image Processing, vol. 29, pp.
graph-based deep models [48] to further explore multi-view             617–627, 2019.
information in those complex situations.                           [2] H. Wang, Y. Yang, and B. Liu, “Gmc: Graph-based multi-
                                                                       view clustering,” IEEE Transactions on Knowledge and
                                                                       Data Engineering, vol. 32, no. 6, pp. 1116–1129, 2019.
                                                                   [3] M. R. Amini, N. Usunier, and C. Goutte, “Learning
                                                                       from multiple partially observed views - an application to
IEEE TRANSACTIONS ON MULTIMEDIA, VOL. XXX, NO. XX, 2021                                                                          12

       multilingual text categorization,” in Advances in Neural      [20] L. Tian, F. Nie, and X. Li, “A unified weight learning
       Information Processing Systems, 2009, pp. 28–36.                   paradigm for multi-view learning,” in Proceedings of
 [4]   G. Bisson and C. Grimal, “Co-clustering of multi-view              Machine Learning Research, vol. 89. PMLR, 2019, pp.
       datasets: a parallelizable approach,” in International Con-        2790–2800.
       ference on Data Mining. IEEE, 2012, pp. 828–833.              [21] D. R. Hardoon, S. Szedmak, and J. Shawe-Taylor,
 [5]   Y. Li, M. Yang, and Z. M. Zhang, “A survey of multi-               “Canonical correlation analysis: An overview with ap-
       view representation learning,” IEEE Transactions on                plication to learning methods,” Neural Computation,
       Knowledge and Data Engineering, vol. 31, no. 10, pp.               vol. 16, no. 12, pp. 2639–2664, 2004.
       1863–1883, 2018.                                              [22] A. Gretton, O. Bousquet, A. Smola, and B. Schölkopf,
 [6]   H. Zhao and Z. Ding, “Multi-view clustering via deep               “Measuring statistical dependence with hilbert-schmidt
       matrix factorization,” in AAAI, 2017, pp. 2921–2927.               norms,” in International conference on algorithmic learn-
 [7]   C. Xu, D. Tao, and C. Xu, “Multi-view intact space                 ing theory. Springer, 2005, pp. 63–77.
       learning,” IEEE Transactions on Pattern Analysis and          [23] J. Rupnik and J. Shawe-Taylor, “Multi-view canonical
       Machine Intelligence, vol. 37, no. 12, pp. 2531–2544,              correlation analysis,” in Conference on Data Mining and
       2015.                                                              Data Warehouses, 2010, pp. 1–4.
 [8]   M. Kan, S. Shan, and X. Chen, “Multi-view deep network        [24] A. Sharma, A. Kumar, H. Daume, and D. W. Jacobs,
       for cross-view classification,” in Proceedings of the IEEE         “Generalized multiview analysis: A discriminative latent
       Conference on Computer Vision and Pattern Recognition,             space,” in 2012 IEEE Conference on Computer Vision
       2016, pp. 4847–4855.                                               and Pattern Recognition. IEEE, 2012, pp. 2160–2167.
 [9]   H. Wang, Y. Yang, B. Liu, and H. Fujita, “A                   [25] M. Kan, S. Shan, H. Zhang, S. Lao, and X. Chen,
       study of graph-based system for multi-view clustering,”            “Multi-view discriminant analysis,” IEEE Transactions
       Knowledge-Based Systems, vol. 163, no. JAN.1, pp.                  on Pattern Analysis and Machine Intelligence, vol. 38,
       1009–1019, 2019.                                                   no. 1, pp. 188–194, 2016.
[10]   H. Wang, Y. Wang, Z. Zhang, X. Fu, L. Zhuo, M. Xu,            [26] Niu, D., Dy, J, Jordan, and a, “Iterative discovery of
       and M. Wang, “Kernelized multiview subspace analysis               multiple alternativeclustering views,” IEEE Transactions
       by self-weighted learning,” IEEE Transactions on Multi-            on Pattern Analysis and Machine Intelligence, vol. 36,
       media, 2020.                                                       no. 7, pp. 1340–1353, 2014.
[11]   Z. Zheng, L. Li, F. Shen, S. H. Tao, and S. Ling, “Binary     [27] X. Cao, C. Zhang, H. Fu, S. Liu, and H. Zhang,
       multi-view clustering,” IEEE Transactions on Pattern               “Diversity-induced multi-view subspace clustering,” in
       Analysis and Machine Intelligence, vol. 41, no. 7, pp.             Computer Vision and Pattern Recognition, 2015, pp.
       1774–1782, 2018.                                                   586–594.
[12]   Y. Yang and H. Wang, “Multi-view clustering: A survey,”       [28] C. Zhang, H. Fu, Q. Hu, P. Zhu, and X. Cao, “Flexible
       Big Data Mining and Analytics, vol. 1, no. 2, pp. 83–107,          multi-view dimensionality co-reduction,” IEEE Transac-
       2018.                                                              tions on Image Processing, vol. 26, no. 2, pp. 648–659,
[13]   L. Fu, P. Lin, A. V. Vasilakos, and S. Wang, “An overview          2016.
       of recent multi-view clustering,” Neurocomputing, vol.        [29] W. Wang and Z. H. Zhou, “A new analysis of co-
       402, pp. 148–161, 2020.                                            training,” in International Conference on International
[14]   Y. Yang and H. Wang, “Multi-view clustering: A survey,”            Conference on Machine Learning, 2010.
       Big Data Mining and Analytics, vol. 1, no. 2, pp. 83–107,     [30] A. Kumar and H. Daumé, “A co-training approach for
       2018.                                                              multi-view spectral clustering,” in Proceedings of the
[15]   G. Chao, S. Sun, and J. Bi, “A survey on multi-view                28th International Conference on Machine Learning,
       clustering,” arXiv preprint arXiv:1712.06246, 2017.                2011, pp. 393–400.
[16]   T. Xia, D. Tao, T. Mei, and Y. Zhang, “Multiview spectral     [31] H. Wang, J. Peng, and X. Fu, “Co-regularized multi-view
       embedding,” IEEE Transactions on Systems, Man, and                 sparse reconstruction embedding for dimension reduc-
       Cybernetics, Part B (Cybernetics), vol. 40, no. 6, pp.             tion,” Neurocomputing, vol. 347, pp. 191–199, 2019.
       1438–1446, 2010.                                              [32] H. Wang, J. Peng, Y. Zhao, and X. Fu, “Multi-path deep
[17]   F. Nie, J. Li, and X. Li, “Parameter-free auto-weighted            cnns for fine-grained car recognition,” IEEE Transactions
       multiple graph learning: A framework for multiview clus-           on Vehicular Technology, vol. 69, no. 10, pp. 10 484–
       tering and semi-supervised classification,” in Proceedings         10 493, 2020.
       of the Twenty-Fifth International Joint Conference on         [33] H. Wang, J. Peng, D. Chen, G. Jiang, T. Zhao, and X. Fu,
       Artificial Intelligence. AAAI Press, 2016, pp. 181–187.            “Attribute-guided feature learning network for vehicle
[18]   F. Nie, G. Cai, J. Li, and X. Li, “Auto-weighted multi-            reidentification,” IEEE MultiMedia, vol. 27, no. 4, pp.
       view learning for image clustering and semi-supervised             112–121, 2020.
       classification,” IEEE Transactions on Image Processing,       [34] H. Wang, J. Peng, G. Jiang, F. Xu, and X. Fu, “Discrim-
       vol. 27, no. 3, pp. 1501–1511, 2017.                               inative feature and dictionary learning with part-aware
[19]   S. Huang, Z. Kang, and Z. Xu, “Self-weighted multi-                model for vehicle re-identification,” Neurocomputing,
       view clustering with soft capped norm,” Knowledge-                 2020.
       Based Systems, vol. 158, no. 15, pp. 1–8, 2018.               [35] Y. Zhu, Y. Xu, F. Yu, S. Wu, and L. Wang,
IEEE TRANSACTIONS ON MULTIMEDIA, VOL. XXX, NO. XX, 2021                                                                                             13

       “Cagnn: Cluster-aware graph neural networks for unsu-                                     Xiangzhu Meng received his B.S. degree from
       pervised graph representation learning,” arXiv preprint                                   Anhui University, in 2015, and the Ph.D. degree
                                                                                                 in Computer Science and Technology from Dalian
       arXiv:2009.01674, 2020.                                                                   University of Technology, in 2021. Now, he is a
[36]   A. Kumar, P. Rai, and H. Daume, “Co-regularized multi-                                    postdoctoral researcher with the Center for Research
       view spectral clustering,” in Advances in Neural Infor-                                   on Intelligent Perception and Computing, Institute of
                                                                                                 Automation, Chinese Academy of Sciences, China.
       mation Processing Systems, 2011, pp. 1413–1421.                                           He regularly publishes papers in prestigious journals,
[37]   R. Bhatia, Matrix analysis. Springer Science & Business                                   including Knowledge-Based Systems, Engineering
       Media, 2013, vol. 169.                                                                    Applications of Artificial Intelligence, Neurocom-
                                                                                                 puting, etc. In addition, he serves as a reviewer for
[38]   V. L. Ulrike, “A tutorial on spectral clustering,” Statistics   ACM Transactions on Multimedia Computing Communications and Applica-
       and Computing, vol. 17, no. 4, pp. 395–416, 2007.               tions. His research interests include multi-modal learning, deep learning, and
[39]   D. Cai, X. He, J. Han, and T. S. Huang, “Graph                  vision-language modeling.
       regularized non-negative matrix factorization for data
       representation,” IEEE Transactions on Pattern Analysis
       and Machine Intelligence, vol. 33, no. 8, pp. 1548–1560,
       2011.
[40]   J. Liu, C. Wang, J. Gao, and J. Han, “Multi-view
       clustering via joint nonnegative matrix factorization,” in
       Proceedings of the 2013 SIAM International Conference
       on Data Mining. SIAM, 2013, pp. 252–260.
[41]   C. Zhang, H. Fu, Q. Hu, P. Zhu, and X. Cao, “Flexible
       multi-view dimensionality co-reduction,” IEEE Transac-
       tions on Image Processing, pp. 648–659, 2016.
[42]   H.-C. Huang, Y.-Y. Chuang, and C.-S. Chen, “Affinity                                      Wei Wei received the B.S. degree from the School
       aggregation for spectral clustering,” in 2012 IEEE Con-                                   of Mathematics and Information Science, Henan
                                                                                                 University, in 2012, and the Ph.D. degree in Institute
       ference on Computer Vision and Pattern Recognition.                                       of Systems Engineering from the Dalian University
       IEEE, 2012, pp. 773–780.                                                                  of Technology, in 2018. He is an associate professor
[43]   L. Zong, X. Zhang, X. Liu, and H. Yu, “Weighted multi-                                    of the Center for Energy, Environment & Econ-
                                                                                                 omy Research, College of Tourism Management,
       view spectral clustering based on spectral perturbation,”                                 Zhengzhou University. His research interests include
       in Proceedings of the AAAI Conference on Artificial                                       energy policy analysis, text mining, machine learn-
       Intelligence, vol. 32, no. 1, 2018.                                                       ing, and artificial intelligence.
[44]   F. Nie, L. Tian, and X. Li, “Multiview clustering via
       adaptively weighted procrustes,” in Proceedings of the
       24th ACM SIGKDD international conference on knowl-
       edge discovery & data mining, 2018, pp. 2022–2030.
[45]   J. A. Hartigan and M. A. Wong, “A k-means clustering
       algorithm,” Applied Statistics, vol. 28, no. 1, 1979.
[46]   Y. Zhu, Y. Xu, F. Yu, Q. Liu, S. Wu, and L. Wang,
       “Deep graph contrastive representation learning,” in
       ICML Workshop on Graph Representation Learning and
       Beyond, 2020.
[47]   ——, “Graph contrastive learning with adaptive aug-
       mentation,” in Proceedings of the Web Conference 2021,
                                                                                                 Wenzhe Liu received her BS degree from Qingdao
       2021, pp. 2069–2080.                                                                      University of Science and Technology, in 2013 and
[48]   Y. Zhu, W. Xu, J. Zhang, Q. Liu, S. Wu, and L. Wang,                                      received MS degree from Liaoning Normal Univer-
       “Deep graph structure learning for robust representations:                                sity . Now she is working towards the PHD degree in
                                                                                                 School of Computer Science and Technology, Dalian
       A survey,” arXiv preprint arXiv:2103.03036, 2021.                                         University of Technology, China. Her research inter-
                                                                                                 ests include mulit-view learning, deep learning and
                                                                                                 data mining.
You can also read