Multiple Random Walkers and Their Application to Image Cosegmentation
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Multiple Random Walkers and Their Application to Image Cosegmentation
Chulwoo Lee Won-Dong Jang Jae-Young Sim Chang-Su Kim
School of Electrical Engineering School of ECE School of Electrical Engineering
Korea University UNIST Korea University
{chulwoo, wdjang}@mcl.korea.ac.kr jysim@unist.ac.kr changsukim@korea.ac.kr
Abstract one another and form their own dominant regions. Eventu-
ally, the power balance among the agents is achieved, and
A graph-based system to simulate the movements and in- their distributions converge. By comparing the stationary
teractions of multiple random walkers (MRW) is proposed distributions, clustering can be achieved. We demonstrate
in this work. In the MRW system, multiple agents traverse that this MRW process can cluster point data and segment
a single graph simultaneously. To achieve desired interac- real images reliably.
tions among those agents, a restart rule can be designed, Moreover, we apply the proposed MRW system to the
which determines the restart distribution of each agent ac- problem of segmenting similar images jointly. Recently, at-
cording to the probability distributions of all agents. In par- tempts to extract common foreground objects from a set of
ticular, we develop the repulsive rule for data clustering. We similar images have been made. This approach, called co-
illustrate that the MRW clustering can segment real images segmentation, was first addressed by Rother et al. [23] and
reliably. Furthermore, we propose a novel image cosegmen- has been researched actively [20, 12, 14, 3, 5, 30, 7, 24, 31].
tation algorithm based on the MRW clustering. Specifically, Compared with segmenting each image independently, it is
the proposed algorithm consists of two steps: inter-image advantageous to delineate similar objects from multiple im-
concurrence computation and intra-image MRW clustering. ages. However, since repeating image features do not al-
Experimental results demonstrate that the proposed algo- ways imply the most important and informative parts of a
rithm provides promising cosegmentation performance. scene, cosegmentation is still a challenging vision problem.
For cosegmentation, we introduce the notion of concur-
rence distribution, which represents the similarity of each
1. Introduction node in an image to foreground objects in the other images.
Then, the MRW clustering is performed by incorporating
A random walk, a process in which a walker moves ran- the concurrence distribution into the repulsive restart rule.
domly from one node to another in a graph, can be used to Experimental results show that the proposed MRW algo-
analyze the underlying data structure of the graph. Many rithm improves the segmentation accuracy significantly, as
properties of a random walk are quantifiable algebraically compared with recent state-of-the-art cosegmentation tech-
based on graph theory [6], and can be solved by optimiza- niques [14, 30, 24, 31], on the iCoseg dataset [3].
tion tools efficiently. Therefore, random walks have been The rest of this paper is organized as follows: Section 2
used in various graph-based learning tasks, including data reviews related work. Section 3 proposes the MRW system
mining [4, 35] and interactive image segmentation [11, 15]. for clustering. Section 4 discusses the characteristics of the
A study in [18, 19] showed that spectral clustering [26] is MRW clustering with image examples. Sections 5 develops
also related to the random walk theory. the MRW cosegmentation algorithm, and Section 6 presents
Whereas the conventional random walk theory describes experimental results. Section 7 concludes this work.
the movements of a single walker (or agent), we propose
a system of multiple random walkers (MRW) to simulate 2. Related Work
multiple agents on a graph simultaneously. Those agents
traverse the graph according to a transition matrix, but they Data Clustering: The task of partitioning data into disjoint
also interact with one another to achieve a desired goal. Our subsets based on the underlying data structure is a primi-
MRW system can support a variety of interactions by em- tive activity to perceive information. The volume of data
ploying different restart rules. In particular, we develop the that people produce and process has grown rapidly, and dis-
repulsive restart rule for data clustering. With this restart covering meaningful information from big data has become
rule, as the random process continues, multiple agents repel an essential subject. Since robust clustering is difficult to
achieve because of noise and outliers in real-world data, var- pose the MRW system to simulate movements and interac-
ious approaches have been attempted [32, 25]. tions of multiple agents on a graph.
Many phenomena in social and biological systems, as
well as engineering problems, can be modelled with graphs,
3.1. Single Random Walker
where nodes represent system elements and edges connect Let G = (V, E) be an undirected, weighted graph. The
the elements to represent their relationships. Graph-based node set V consists of data points xi , i = 1, . . . , N . Edge
clustering hence has drawn much attention, and numerous eij in the edge set E connects xi and xj . Also, let W ∈
algorithms have been proposed [26, 21, 34, 16, 29]. RN ×N be a symmetric matrix, in which the (i, j)th element
wij is the weight of eij , representing the affinity between
Image Cosegmentation: Rother et al. [23] first formulated
xi and xj . Typically, wij is defined as
the cosegmentation problem by incorporating the inter-
image consistency into a Markov random field (MRF) en- ( 2
d (x ,x )
ergy function. Instead of the l1 -norm, Mukherjee et al. [20] exp − σi2 j if eij ∈ E,
wij = (1)
used the l2 -norm to measure the similarity between fore- 0 otherwise,
ground histograms. Hochbaum and Singh [12] developed a
reward model satisfying the submodular condition and thus where d is a dissimilarity function and σ 2 is a scale param-
solved it efficiently using the graph cuts. These early tech- eter.
niques have been proposed to extract almost identical ob- A random walker travels on the graph G. The transition
jects from different backgrounds. probability aij that the walker moves from node j to node
Joulin et al. [14] considered the cosegmentation of fore- i is obtainedPby dividing wij by the degree of node j, i.e.,
ground objects with more appearance variations, by for- aij = wij / k wkj . In other words, the transition matrix
mulating the cosegmentation as a discriminative clustering A = [aij ] is computed by normalizing each column of the
(t) (t)
problem. Batra et al. [3] developed an interactive coseg- affinity matrix W. Let p(t) = [p1 , · · · , pN ]T be a col-
(t)
mentation algorithm, which intelligently suggests to the umn vector, in which pi denotes the probability that the
user where to scribble next. They also made a cosegmen- walker is found at node i at time instance t. We then have
tation dataset with manually-annotated groundtruth, called the temporal recursion
iCoseg, publicly available.
Collins et al. [7] used the random walk algorithm in [11] p(t+1) = Ap(t) . (2)
as a core segmentation tool, and achieved the cosegmenta-
If the graph has a finite number of nodes and is fully con-
tion by enforcing the constraint that foreground histograms
nected, A is irreducible and primitive [13]. Then, the
should match one another. The proposed algorithm is also
walker has a unique stationary distribution π, satisfying
based on a random walker process. However, while [11]
π = Aπ, and π = limt→∞ p(t) regardless of an initial
considers whether a single walker reaches a foreground or
condition p(0) . The stationary distribution π conveys use-
background seed first, the proposed algorithm employs mul-
ful information about the underlying data structure of the
tiple random walkers, who interact one another, and derives
graph, and it is thus exploited in vision applications [11, 10].
segmentation results from their stationary distributions.
RWR [22] is a generalization of the random walk, in
Recently, several cosegmentation algorithms have been
which the walker is compelled to return to specified nodes
proposed to achieve robust correspondence between fore-
with a restart probability ǫ. The RWR recursion is
ground objects. Chang et al. [5] used saliency models to
exclude regions that infrequently appear across images. Vi- p(t+1) = (1 − ǫ)Ap(t) + ǫr, (3)
cente et al. [30] generated object proposals from each im-
age, and used a random forest classifier to score a pair of where r = [r1 , r2 , · · · , rN ]T is the restart distribution with
P
proposals from different images. Rubio et al. [24] devel- i ri = 1 and ri ≥ 0. With probability 1 − ǫ, the walker
oped a region matching method to establish the correspon- moves ordinarily as in (2). On the other hand, with proba-
dences between common objects among images. Wang et bility ǫ, the walker is forced to restart with the distribution
al. [31] used a functional map to represent consistent ap- r. When ri = 1 and rj = 0 for all j 6= i, the stationary
pearance relations between a pair of images, and jointly op- distribution of the RWR represents the affinity of each node
timized the segmentation maps of all images. to the specific node i. This property has been exploited in
image segmentation [15] and data mining [22].
3. Multiple Random Walkers We note that RWR can be interpreted as the ordinary ran-
dom walk as well. The RWR recursion in (3) can be rewrit-
This section introduces the notion of MRW. We first de- ten as
scribe the conventional random walk, which is a Markov
process of a single random walker (or agent). We then pro- p(t+1) = (1 − ǫ)Ap(t) + ǫBp(t) = Ãp(t) , (4)
where B is a square matrix, all columns of which are equal Theorem 1 If the graph is connected and 0 ≤ δ < 1, each
to r, and à = (1 − ǫ)A + ǫB is the equivalent transition agent in the MRW system in (5) and (6) has a stationary
matrix. There exists i such that ri > 0. Thus, bii > 0 and distribution, given by
ãii > 0. It is known in matrix analysis [13] that an irre-
(t)
ducible matrix with at least one positive diagonal elements π k = lim pk . (8)
t→∞
is primitive. Therefore, the RWR recursion yields a unique
stationary distribution regardless of an initial condition, as The term ‘multiple random walks’ was used in [2, 8],
long as the graph is connected (it does not need to be fully but their graph simulations are different from the proposed
connected). MRW system. They assume that multiple particles are in-
dependent of one another, or annihilate one another, or co-
3.2. MRW alesce into one at a meeting node. Then, the expected time
The conventional random walk in (2) or (3) describes for covering all nodes, or annihilating all particles, or co-
the movements of a single agent. In contrast, we consider alescing into a single particle is computed. On the con-
multiple agents who share the same graph and affect the trary, in our system, multiple walkers adapt their move-
movements of one another. ments based on other walkers’ distributions. Then, we ex-
(t)
Suppose there are K agents on a graph. Let pk denote tract useful information from the stationary distributions of
the probability distribution of agent k at time t. Similar to the multiple walkers.
(3), random movements of agent k are governed by
3.3. Repulsive Restart Rule for Clustering
(t+1) (t) (t)
pk = (1 − ǫ)Apk + ǫrk , k = 1, . . . , K. (5) By designing the restart rule φk in (6), we can simulate
a variety of agent interactions to achieve a desired goal. As
Thus, with probability 1 − ǫ, agent k travels on the graph a particular example, we propose the repulsive restart rule
according to the transition matrix A, independently of the for clustering data. For notational simplicity, let us omit the
other agents. However, with probability ǫ, agent k visits superscripts for time instances.
the nodes according to the time-varying restart distribution In the MRW system,
(t) (t) (t)
rk = [rk,1 , · · · , rk,N ]T .
T
We can make the agents interact with one another, by pk = [p(x1 |ωk ), · · · , p(xN |ωk )] (9)
determining the restart distribution as
where p(xi |ωk ) is the probability that agent k is found at
(t) (t−1)
rk = (1 − δ t )rk + δ t φk (P (t) ) (6) node i. According to the Bayes’ rule, the posterior proba-
bility is given by
where the function φk is referred to as the restart rule.
It determines a probability distribution φk from P (t) = p(xi |ωk )p(ωk )
(t) p(ωk |xi ) = P , (10)
{pk }K k=1 , which is the set of the probability distributions l p(xi |ωl )p(ωl )
of all agents at time t.
In (6), δ is a constant within [0, 1], called the cooling which represents the probability that node i is occupied by
factor. In an extreme case of δ = 0, the restart distribu- agent k. The repulsive restart rule sets the ith element of
(t)
tion rk becomes time-invariant, and the MRW recursion φk (P) as
of each agent in (5) is identical with the RWR recursion in
(t) φk,i = α · p(ωk |xi ) · p(xi |ωk ) (11)
(3). In the other extreme case of δ = 1, rk = φk (P (t) )
(t−1)
does not directly depend on the previous distribution rk . where α is a normalizing factor to make φk (P) a probabil-
Suppose that 0 < δ < 1. We have ity distribution. Suppose that agent k is dominant at node i,
i.e., it has a high posterior probability p(ωk |xi ) and a high
(t) (t−1) (t−1)
krk − rk k∞ = δ t kφk (P (t) ) − rk k∞ ≤ δ t . (7) likelihood p(xi |ωk ). Then, it restarts at that node with a
high probability, and tends to become more dominant. This
(s) (t)
Thus, if s ≥ t ≥ T , krk − rk k∞ ≤ δ T /(1 − δ). So each has the effect that a dominant agent at a node repels the
(t) other agents. The repulsive restart rule in (11) can be rewrit-
element in the restart distribution rk is a Cauchy sequence
in terms of time t. Since a Cauchy sequence in R is conver- ten as
(t)
gent, the restart distribution rk converges to a fixed distri- φk (P) = αQk pk (12)
(∞)
bution rk . Therefore, as t approaches infinity, the MRW where Qk is a diagonal matrix whose (i, i)th element is the
recursion in (5) becomes the RWR recursion, and agent k posterior probability p(ωk |xi ).
has a stationary distribution eventually. To summarize we For clustering, we perform the MRW simulation in (5)
have the following convergence theorem. and (6), by employing the restart rule in (12), to obtain the(a) t = 0 (b) t = 1 (c) t = 5 (d) t = 15 (e) t = 30 (f) t = 70 (g) t = 100 (h) t = 200
Figure 1. Double random walkers with the repulsive restart rule. Red and blue walkers move interactively to divide a point set into two
clusters. The top two rows depict the probability distributions of the red and blue walkers, and the bottom row shows the clustering results.
The clustering decides that a point belongs to the red cluster, if the red walker has a higher probability at the point than the blue walker.
stationary distribution π k of each agent k. Then, node i is in (1), by employing the dissimilarity function
assigned a cluster label li based on the maximum a posteri- X
ori (MAP) criterion, d(xi , xj ) = λl dl (xi , xj ). (14)
l
li = arg max p(ωk |xi ). (13) We use five dissimilarities dl of node features, including
k
RGB and LAB super-pixel means, boundary cues, bag-of-
Figure 1 illustrates an MRW process of double random visual-words histograms of RGB and LAB colors [27]. We
walkers with the repulsive restart rule. In Figure 1(a), the average those dissimilarities using empirically determined
top two rows show initial probability distributions of the weights λl . Please refer to the supplemental materials for
red and blue walkers, which are randomly generated. The details. By normalizing each column of the affinity matrix
bottom row is the result of the clustering. The clustering W = [wij ], we obtain the transition matrix A.
result at t = 0 is meaningless. However, as the iteration To achieve bilayer segmentation, we employ double ran-
goes on, each walker repels the other walker, while form- dom walkers, called foreground walker and background
ing a dominant cluster region. Consequently, the probabil- walker, whose probability distributions are denoted by pf
ity distributions of the two walkers converge, respectively, and pb , respectively. These two walkers make interactions
and the power balance between the walkers is achieved. By according to
comparing those probabilities at each node, we obtain the
(t+1) (t) (t)
clustering result in Figure 1(h), which coincides with the pf = (1 − ǫ)Apf + ǫrf
intuitive clustering of the human visual system. pb
(t+1)
=
(t)
(1 − ǫ)Apb + ǫrb
(t)
(15)
4. MRW Clustering – Image Examples with the repulsive restart rule. By Theorem 1, it is guaran-
teed that the probability distributions converge to stationary
This section illustrates how the MRW clustering algo- distributions π f and π b , respectively. However, π f and π b
rithm with the repulsive restart rule can segment real im- (0) (0)
depend on the initial distributions pf and pb . Assuming
ages. With those image examples, we discuss the character- the center prior that foreground objects tend to appear near
istics of the MRW algorithm. (0) (0)
the image center, we initialize pf and pb to be uniformly
distributed at the interior nodes and the boundary nodes, re-
4.1. Methodology
spectively, as shown in Figure 2(a).
Given an input image, we first construct a graph G =
(V, E). The image is initially over-segmented into SLIC 4.2. Examples
super-pixels [1], each of which becomes a node in V . For Figure 2 illustrates the repulsive MRW process in an
the edge set E, we use the edge connection scheme in [33]. image, which is shown in the lower right corner. In Fig-
Specifically, each node is connected not only to its spatial ure 2(a), the top two rows show the initial probability distri-
neighbors but also to the neighbors of the neighbors, and all butions of the foreground and background walkers, based
boundary nodes at the image border are connected to one on the center prior, respectively. The bottom row is the
another. segmentation result, based on the MAP decision in (13).
For each edge eij , we determine the affinity weight wij At early stages, the foreground region is identified around(a) t = 0 (b) t = 3 (c) t = 10 (d) t = 15 (e) t = 20 (f) t = 25 (g) t = 30 (h) t = 50
Figure 2. A repulsive MRW process of foreground and background walkers in an image. The input image is shown in the lower right
corner. At each time instance t, from top to bottom, the probability distributions of the foreground and background walkers, pf and pb ,
and the segmentation result are shown.
the image center due to the initial distributions. However,
as the iteration continues, the foreground walker explores
nearby similar nodes, while competing with the background
walker. The repulsive restart rule facilitates discriminative
clustering. Finally, in Figure 2(h), the probability distribu-
tions converge, and we obtain a satisfactory segmentation
result that extracts the bear faithfully.
Figure 3 shows more examples. For comparison, we
also provide the segmentation results of the normalized cuts
(Ncut) algorithm [26] and the spectral k-means (SKM) al-
gorithm [21]. Both Ncut and SKM are spectral clustering
algorithms, which can be interpreted using the framework
of the conventional random walks [18, 19]. Thus, the pro-
posed MRW is related to Ncut and SKM. However, the
proposed algorithm is different from the spectral cluster-
ing. Note that, using the definition of the transition matrix
in [18, 19], the spectral clustering processes ‘right’ eigen-
vectors of the transition matrix for the clustering. On the
contrary, the proposed algorithm uses the stationary distri-
bution, which is a ‘left’ eigenvector of the transition matrix.
Furthermore, whereas the conventional random walks con-
(a) Input (b) Ncut (c) SKM (d) RWR (e) MRW
sider only a single agent, the proposed algorithm employs
multiple agents and exploits their interactions. Figure 3. Bilayer segmentation results of the proposed MRW clus-
tering in comparison with the conventional approaches: Ncut [26],
In Figure 3, we see that Ncut and SKM tend to divide im- SKM [21], and RWR. In the RWR results, the initial distributions
ages along the strongest edges. In contrast, the proposed al- based on the center prior are used as the fixed restart distributions.
gorithm extracts more meaningful foreground regions. We
note that this is not a fair comparison, since the proposed al-
gorithm assumes the center prior in the initialization of the different labels. Thus, the initial distributions in the pro-
foreground and background distributions. The objective of posed algorithm can be regarded as those scribbles. But,
this comparison is to demonstrate different characteristics whereas [15] assumes that user scribbles are completely
of the proposed algorithm from the spectral clustering, not trustable, the proposed algorithm automatically begins with
to claim superior performance of the proposed algorithm. rough initial guesses. As shown in Figure 2, the rough
The proposed algorithm is fully automatic, but it has guesses at t = 0 are refined according to the interactions of
common components with the interactive segmentation the walkers, as t increases. In other words, while [15] uses
techniques [11, 15]. Especially, in [15], for each cluster fixed restart distributions, the proposed algorithm evolves
label, the RWR recursion is performed by employing user restart distributions adaptively to image contents in a time-
scribbles as the restart distribution. The segmentation is varying manner. Figure 3(d) shows the RWR segmentation
achieved by comparing the stationary distributions for the results when the rough guesses based on the center prior areused as the fixed restart distributions. RWR yields inferior
performance to the proposed algorithm, since the walkers
do not interact and the restart distributions are fixed.
5. Image Cosegmentation (a) Image Iv (b) Image Iu
We propose a novel algorithm for image cosegmentation
based on the MRW system. The input is a set of images I =
{I1 , · · · , IZ }, which contain similar objects. The objective
is to segment those objects jointly from all input images.
(c) pf(v) (d) Auv pf(v) (e) Su Auv pf(v)
5.1. Initialization
Figure 4. Transferring the foreground distribution pf(v) in im-
For each image Iu in I, we construct a graph indepen- age Iv to image Iu . For the purpose of illustration, the manually
dently of the other images, and adopt a foreground walker obtained ground-truth pf(v) is used in this example.
and a background walker. We use the graph construction
scheme in Section 4.1. Let pf(u) and pb(u) denote the prob- In Figure 4(e), we see that this new estimate Su Auv pf(v)
ability distributions of the foreground walker and the back- approximates the object location in Iu more faithfully.
ground walker in Iu , respectively. They are also initialized By integrating the inter-image estimates from all images,
based on the center prior, as described in Section 4.1. we obtain the concurrence distribution cf(u) of the fore-
5.2. Inter-Image Concurrence Computation ground walker in Iu as
1 X
To exploit the correlation among images, we compute cf(u) = Su Auv pf(v) , (17)
the concurrence distribution cf(u) of the foreground walker, Z v
which indicates the similarity of each node in image Iu to
where Z is the number of input images. In other words,
foreground objects in the other images.
cf(u) is the overall inter-image estimate of the object loca-
Let Wuv ∈ RM ×N be a matrix, in which the (i, j)th el- tion in Iu , obtained from all images. We compute the con-
ement wij represents the affinity from node j in image Iv currence distribution cb(u) of the background walker in a
to node i in image Iu . Here, M and N are the numbers of similar manner.
nodes in Iu and Iv , respectively. These inter-image affini-
ties are computed between all nodes in Iv and all nodes in 5.3. Intra-Image MRW Clustering
Iu . We compute each wij using a dissimilarity function,
Next, we perform an MRW clustering process to re-
similar to (14). However, we use features, e.g. SIFT [17]
fine the foreground and background distributions, pf(u) and
and texton [9], which are widely used in object detection
pb(u) , by employing the concurrence distributions cf(u) and
and classification. For more details, please refer to the sup-
cb(u) .
plemental materials. Then, we obtain the transfer matrix
The repulsive restart rule in (12) is effective for cluster-
Auv from Iv to Iu by normalizing each column in Wuv .
ing nodes according to their intra-image feature vectors. On
The transfer matrix Auv represents the correspondences
the other hand, the concurrence distributions cf(u) and cb(u)
from nodes in Iv to nodes in Iu probabilistically. It hence
provide the inter-image estimates of object and background
can transfer the foreground distribution pf(v) in Iv to Iu .
locations, respectively. Thus, to exploit both the intra and
The transferred distribution, Auv pf(v) , can be used as an
inter information, we define the hybrid restart rule for the
inter-image estimate of the object location in Iu . How-
foreground walker at image Iu as
ever, since the correspondences may be inaccurate espe-
cially on homogeneous regions, Auv pf(v) may be concen- φf(u) {pf(u) , pb(u) } = γ α Qf(u) pf(u) + (1 − γ)cf(u)
trated around a few nodes. For example, the foreground (18)
distribution pf(v) in Figure 4(c) is transferred to Auv pf(v) where Qf(u) is a diagonal matrix whose elements are the
in Figure 4(d), but only the lower right part of the object posterior probabilities of the foreground walker and α is a
is overly highlighted. To overcome this limitation, in im- normalizing factor, as in (12). Also, γ is a parameter to
age Iu , we perform the RWR recursion in (3) by employ- controls the balance between the repulsive interaction and
ing Auv pf(v) as the restart distribution r. Notice that the the concurrence preservation. In this work, γ is fixed to
corresponding stationary distribution can be computed as 0.3. We also define the hybrid restart rule φb(u) for the
Su r = Su Auv pf(v) , where background walker in a similar manner.
Using these hybrid restart rules, we perform the iterative
−1
Su = ǫ (I − (1 − ǫ)Au ) . (16) MRW process to obtain the stationary distributions π f(u)Algorithm 1 Image Cosegmentation
Input: Graphs G = {G1 , · · · , GZ } for a set of input images I
1: Initialize P(u) = {pf(u) , pb(u) } for each Iu ⊲ Section 4.1
2: repeat for each image Iu
3: Inter-image concurrence computation ⊲ Section 5.2
4: Intra-image MRW clustering ⊲ Section 5.3
5: Foreground extraction C = {C1 , · · · , CPZ}
6: Computation of the foreground distance u,v df (Cu , Cv )
7: until the foreground distance stops decreasing
8: Pixel-level refinement
Output: Segmentation maps C = {C1 , · · · , CZ }
that foreground objects in images should be similar to one
another. In other words, every foreground node in image Iu
should have a similar node in another image Iv . To quan-
tify this property, let Cu denote the set of the nodes in Iu
that are classified as the foreground. Then, the foreground
distance df (Cu , Cv ) from Iu to Iv is defined as
X
df (Cu , Cv ) = min d(xi , xj ) (19)
(a) Input (b) Initial (c) 1st pass (d) 2nd pass (e) 6th pass j∈Cv
i∈Cu
Figure 5. The evolution of foreground distributions in the multi-
pass clustering. As the passes go on, the Taj Mahal is more clearly where d is the dissimilarity function used to compute the
highlighted. inter-image affinity Wuv . Then, the overall Pforeground dis-
tance among all images is computed as u,v df (Cu , Cv ).
The multi-pass refinement is terminated when the overall
and π b(u) of the foreground and background walkers. Then, distance stops decreasing.
from the stationary distributions, we obtain the posterior Since each image is over-segmented into super-pixels to
probabilities of the foreground and background walkers at reduce the number of graph nodes, the resultant foreground
each node. Finally, we can achieve the bilayer segmentation and background distributions can be further refined at the
of image Iu based on the MAP criterion in (13). pixel level using a bilateral filter [28]. Based on the spatial
distances and the LAB color differences in an input image,
5.4. Multi-Pass Refinement we filter the foreground and background distributions. The
A single execution of the two steps, i.e. (Step 1) inter- segmentation accuracy is improved by about 1% with this
image concurrence computation and (Step 2) intra-image pixel-level refinement. Algorithm 1 summarizes the pro-
MRW clustering, provides satisfactory cosegmentation re- posed cosegmentation algorithm.
sults in most cases. However, multiple executions of the
steps are beneficial in some cases. In such multi-pass re- 6. Experimental Results
finement, the resultant stationary distributions at Step 2 are We test the performance of the proposed MRW coseg-
used as the input for computing the concurrent distributions mentation (MRW-CS) algorithm on the iCoseg dataset [3],
at Step 1 in the next execution. which is composed of 643 images of 38 object classes. For
Figure 5 exemplifies the multi-pass refinement. Fig- each object class, similar images were searched with the
ure 5(b) shows the initial foreground distributions. In the same theme, and sometimes were selected from the same
first pass, they are used to compute the stationary distribu- user’s photo album. This is a realistic scenario for image
tions of the foreground walkers in Figure 5(c). Those sta- cosegmentation, which attempts to improve the segmenta-
tionary distributions highlight not only the Taj Mahal but tion accuracy of an input image by employing similar im-
also less important regions, such as the sky. By feeding ages. A segmentation accuracy is defined as the percentage
these distributions as new initial distributions in the second of correctly labeled pixels.
pass, we obtain the stationary distributions in Figure 5(d), MRW-CS performs the inter-image concurrence compu-
and so on. Note that, as the passes go on, the Taj Mahal is tation to exploit the correlation across images. If this step
more clearly highlighted. is skipped, MRW-CS is reduced to the MRW clustering
We need a stopping criterion for the multi-pass refine- in Section 4, which segments each image independently.
ment. The fundamental assumption of cosegmentation is Thus, we refer to the MRW clustering in Section 4 as theFigure 6. Pairs of segmentation results, obtained by MRW-IS (a) Balloon
(left) and MRW-CS (right).
Table 1. Cosegmentation performance on iCoseg data set.
# of images MRW
Class (used/total) [14] [30] [24] [31] IS CS (b) Baseball
1 Alaskan bear 9/19 74.8 90.0 86.4 90.4 78.0 87.3
2 Balloon 8/24 85.2 90.1 89.0 90.4 64.1 97.7
3 Baseball 8/25 73.0 90.1 90.5 94.2 52.8 97.1
4 Bear 5/5 74.0 95.3 80.4 88.1 75.1 93.7
5 Elephant 7/15 70.1 43.1 75.0 86.7 68.1 93.1
6 Ferrari 11/11 85.0 89.9 84.3 95.6 83.7 91.9
7 Gymnastics 6/6 90.9 91.7 87.1 90.4 90.2 96.1 (c) Kite Panda
8 Kite 8/18 87.0 90.3 89.8 93.9 70.6 95.7
9 Kite panda 7/7 73.2 90.2 78.3 93.1 77.1 96.0
10 Liverpool 9/33 76.4 87.5 82.6 89.4 71.1 88.5
11 Panda 8/25 84.0 92.7 60.0 88.6 85.8 84.8
12 Skating 7/11 82.1 77.5 76.8 78.7 76.8 91.6
13 Statue 10/41 90.6 93.8 91.6 96.8 78.6 94.5
(d) Skating
14 Stonehenge 5/5 56.6 63.3 87.3 92.5 77.3 95.9
15 Stonehenge 2 9/18 86.0 88.8 88.4 87.2 82.6 90.7
16 Taj Mahal 5/5 73.7 91.1 88.7 92.6 70.7 95.2
Average 78.9 85.4 83.5 90.5 75.2 93.1
independent segmentation (MRW-IS). For both CS and IS,
we fix the restart probability ǫ in (5) to 0.2, and the cooling (e) Statue
factor δ in (6) to 0.95. Also, for computing posterior prob-
abilities in (10), the prior probabilities for foreground and
background walkers are set to p(ωf ) = p(ωb ) = 12 .
Figure 6 compares MRW-CS with MRW-IS. IS attempts
(f) Stonehenge
to separate an object from its background, but the defini-
tion of ‘object’ is ambiguous. For instance, given the single Figure 7. Examples of cosegmentation results. Best viewed in
image of ‘Statue,’ it is not clear whether the object should color.
consist of the head only or both the head and body. Thus, IS examples in the supplemental materials.
may fail to delineate desired objects, especially in the cases
of weak boundaries or background clutter. In contrast, CS
7. Conclusions
overcomes the ambiguity, by extracting regions that occur
repeatedly across images. In Figure 6, CS segments out the We proposed the MRW system to simulate the interac-
baseball player, the gymnast, and the statue faithfully. tions of multiple agents on a graph. To achieve desired in-
Table 1 compares the accuracies of the proposed MRW- teractions, a restart rule can be designed to determine the
IS and MRW-CS with those of conventional algorithms. For restart distribution of an agent according to the probability
each class, a number of images are selected for the coseg- distributions of all agents. As a particular example, we de-
mentation. Since [30], the number has been fixed, but the veloped the repulsive rule for data clustering. We discussed
combination of the images is unknown. Hence, for each the characteristics of the MRW clustering with image exam-
class, the cosegmentation is performed with randomly se- ples. Moreover, we proposed an efficient image cosegmen-
lected images. Then, the accuracy is averaged over 20 ran- tation algorithm, composed of two main steps: inter-image
dom selections. Note that CS outperforms the conventional concurrence computation and intra-image MRW clustering.
algorithms in most cases. On average, as compared with Experimental results demonstrated that the proposed algo-
the state-of-the-art in [31], CS improves the accuracy by rithm outperforms the state-of-the art techniques signifi-
about 3%. Figure 7 demonstrates that the proposed MRW- cantly on the iCoseg dataset. Future research issues include
CS yields fine results, even though individual images are the exploration of more applications of the MRW system
challenging. Due to the page limitation, we provide more and the development of other restarting rules.Acknowledgements [17] C. Liu, J. Yuen, and A. Torralba. SIFT flow: Dense corre-
spondence across scenes and its applications. IEEE TPAMI,
This work was supported partly by the Global Frontier R&D 33(5):978–994, May 2011.
Program on Human-centered Interaction for Coexistence funded [18] M. Meila and J. Shi. Learning segmentation by random
by the National Research Foundation of Korea (NRF) grant funded walks. In Adv. Neural Inf. Process. Syst., pages 873–879,
by the Korean Government (MSIP) (2011-0031648), and partly 2001.
by the NRF grant funded by the Korea government (MSIP) (No. [19] M. Meila and J. Shi. A random walks view of spectral seg-
2012-011031). mentation. In Proc. AISTATS, 2001.
[20] L. Mukherjee, V. Singh, and C. Dyer. Half-integrality based
References algorithms for cosegmentation of images. In Proc. IEEE
CVPR, pages 2028–2035, 2009.
[1] R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and [21] A. Y. Ng, M. I. Jordan, and Y. Weiss. On spectral clustering:
S. Susstrunk. SLIC superpixels compared to state-of-the-art Analysis and an algorithm. In Adv. Neural Inf. Process. Syst.,
superpixel methods. IEEE TPAMI, 34(11):2274–2282, Nov. volume 2, pages 849–856, 2002.
2012.
[22] J.-Y. Pan, H.-J. Yang, C. Faloutsos, and P. Duygulu. Au-
[2] N. Alon, C. Avin, M. Koucký, G. Kozma, Z. Lotker, and tomatic multimedia cross-modal correlation discovery. In
M. R. Tuttle. Many random walks are faster than one. In Proc. ACM SIGKDD, pages 653–658, 2004.
Proc. ACM SPAA, pages 110–128, 2008.
[23] C. Rother, T. Minka, A. Blake, and V. Kolmogorov. Coseg-
[3] D. Batra, A. Kowdle, D. Parikh, J. Luo, and T. Chen. iCoseg: mentation of image pairs by histogram matching - incorpo-
Interactive co-segmentation with intelligent scribble guid- rating a global constraint into MRFs. In Proc. IEEE CVPR,
ance. In Proc. IEEE CVPR, pages 3169–3176, 2010. pages 993–1000, 2006.
[4] S. Brin and L. Page. The anatomy of a large-scale hyper- [24] J. Rubio, J. Serrat, A. Lopez, and N. Paragios. Unsupervised
textual web search engine. Computer Networks and ISDN co-segmentation through region matching. In Proc. IEEE
Systems, 30(1):107–117, 1998. CVPR, pages 749–756, 2012.
[5] K.-Y. Chang, T.-L. Liu, and S.-H. Lai. From co-saliency [25] S. E. Schaeffer. Graph clustering. Computer Science Review,
to co-segmentation: An efficient and fully unsupervised en- 1(1):27–64, 2007.
ergy minimization model. In Proc. IEEE CVPR, pages 2129–
[26] J. Shi and J. Malik. Normalized cuts and image segmenta-
2136, 2011.
tion. IEEE TPAMI, 22(8):888–905, 2000.
[6] F. R. K. Chung. Spectral Graph Theory. American Mathe-
[27] J. Sivic and A. Zisserman. Video google: A text retrieval
matical Society, 1997.
approach to object matching in videos. In Proc. IEEE CVPR,
[7] M. Collins, J. Xu, L. Grady, and V. Singh. Random walks pages 1470–1477, 2003.
based multi-image segmentation: Quasiconvexity results and
[28] C. Tomasi and R. Manduchi. Bilateral filtering for gray and
GPU-based solutions. In Proc. IEEE CVPR, pages 1656–
color images. In Proc. IEEE ICCV, pages 839–846, 1998.
1663, 2012.
[29] S. M. van Dongen. Graph Clustering by Flow Simulation.
[8] C. Cooper, A. Frieze, and T. Radzik. Multiple random walks
Ph.D. dissertation, Universiteit Utrecht, 2001.
in random regular graphs. SIAM J. Discrete Mathematics,
[30] S. Vicente, C. Rother, and V. Kolmogorov. Object coseg-
23(4):1738–1761, 2010.
mentation. In Proc. IEEE CVPR, pages 2217–2224, 2011.
[9] T. Deselaers and V. Ferrari. Global and efficient self-
[31] F. Wang, Q. Huang, and L. Guibas. Image co-segmentation
similarity for object classification and detection. In Proc.
via consistent functional maps. In Proc. IEEE ICCV, pages
IEEE CVPR, pages 1633–1640, 2010.
849–856, 2013.
[10] V. Gopalakrishnan, Y. Hu, and D. Rajan. Random walks on
[32] R. Xu and D. Wunsch II. Survey of clustering algorithms.
graphs for salient object detection in images. IEEE Trans.
IEEE Trans. Neural Netw., 16(3):645–678, 2005.
Image Process., 19(12):3232–3242, 2010.
[33] C. Yang, L. Zhang, H. Lu, X. Ruan, and M.-H. Yang.
[11] L. Grady. Random walks for image segmentation. IEEE
Saliency detection via graph-based manifold ranking. In
TPAMI, 28(11):1768–1783, 2006.
Proc. IEEE CVPR, pages 3166–3173, 2013.
[12] D. Hochbaum and V. Singh. An efficient algorithm for co-
[34] L. Zelnik-Manor and P. Perona. Self-tuning spectral clus-
segmentation. In Proc. IEEE ICCV, pages 269–276, 2009.
tering. In Adv. Neural Inf. Process. Syst., pages 1601–1608,
[13] R. A. Horn and C. R. Johnson. Matrix Analysis. Cambridge
2004.
University Press, 1985.
[35] D. Zhou, J. Weston, A. Gretton, O. Bousquet, and
[14] A. Joulin, F. Bach, and J. Ponce. Discriminative cluster-
B. Schölkopf. Ranking on data manifolds. In Adv. Neural
ing for image co-segmentation. In Proc. IEEE CVPR, pages
Inf. Process. Syst., volume 16, pages 169–176. 2004.
1943–1950, 2010.
[15] T. H. Kim, K. M. Lee, and S. U. Lee. Generative image seg-
mentation using random walks with restart. In Proc. ECCV,
pages 264–275, 2008.
[16] Z. Li, J. Liu, S. Chen, and X. Tang. Noise robust spectral
clustering. In Proc. IEEE ICCV, pages 1–8, 2007.You can also read
