DEEP EARTH QUERY: INFORMATION DISCOVERY FROM BIG EARTH OBSERVATION DATA ARCHIVES - PROF. DR. BEGÜM DEMIR REMOTE SENSING IMAGE ANALYSIS (RSIM) ...
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Deep Earth Query: Information Discovery
from Big Earth Observation Data Archives
Prof. Dr. Begüm Demir
Remote Sensing Image Analysis (RSiM) Group, TU Berlin
Introduction: Space Renaissance
Recent Earth Observation (EO) satellite missions have led to a significant growth of EO image
archives, e.g.:
20
Sentinels Archive (PB)
Volume of ESA’s
15
10
5
0
ESA’s Sentinel-1 2014 2015 2016 2017 2018 2019 2020 ESA’s Sentinel-2
(SAR) (Multispectral)
Sentinels: Europe's eyes on Earth
Fort McMurray, Canada-24/06/2016 Brussels, Belgium-08/09/2016
B. Demir 1
Information Discovery: Query by Example
EO image search/retrieval systems aim to explore crucial information from huge EO data
archives.
Burned Forest
Marseille, France-22.09.2016
McMurray, Canada-30.08.2016 Sardinia, Italy-28.07.2016 Sicily, Italy-25.05.2016 Siberia, Russia-08.10.2016
B. Demir 2
Information Discovery: Query by Example
Sweden, Gothenburg
Search and Retrieval from Big EO Data Archives
Bi-Temporal Change-Query
Retrieved Bi-Temporal Images with Similar Change Content
France, Bordeaux Poland, Poznan Finland, Joensuu
Coniferous forest
Coniferous forest Coniferous forest Coniferous forest
Transitional woodland/shrub
Transitional woodland/shrub Transitional woodland/shrub Transitional woodland/shrub
Massive EO
Data Archive
B. Demir 3
Query by Example: Main Blocks
Query Data
Discriminative and Robust EO Data Description
EO Archive
Scalable EO Data Search
Search complexity is O(n) and storage complexity is O(nd), where n is number of images in
the archive and d is the image feature number.
Problem: exhaustive search in huge remote sensing archives is time-demanding.
B. Demir 4
Hashing Methods in Image Retrieval
Hashing Methods in Image
Hash functions
Retrieval
P-th image
in the archive
B. Demir, L. Bruzzone “Hashing based scalable remote sensing image search and retrieval in large archives”, IEEE Transactions on Geoscience and
Remote Sensing, vol. 54, no.2, pp. 892-904, 2016.
B. Demir 5
Kernel-based Hashing Methods
Two popular hashing methods that define hash functions in the kernel space are:
Kernel-based unsupervised locality sensitive hashing.
Kernel-based supervised hashing LS locality sensitive hashing.
Kernel-based methods express the Gaussian random vector as the weighted sum of
m images selected from the archive as:
r r ( j ) X j
m
j 1 nonlinear mapping function
Then the hash function becomes:
m m
hr ( Xi ) sign r ( j ) X j Xi sign r ( j ) K X j , Xi , r 1, 2,..., b
j 1 j 1 kernel
r-th hash
function function
B. Demir, L. Bruzzone, "Hashing Based Scalable Remote Sensing Image Search and Retrieval in Large Archives", IEEE Transactions on
Geoscience and Remote Sensing, vol. 54, no.2, pp. 892-904, 2016.
B. Demir 6
Metric Learning based Deep Hashing(MiLAN)
S. Roy, E. Sangineto, B. Demir, N. Sebe, "Metric-Learning based Deep Hashing Network for Content Based Retrieval of Remote Sensing
Images", IEEE Geoscience and Remote Sensing Letters, accepted for publication, 2020.
B. Demir 7
Metric Learning based Deep Hashing(MiLAN)
The intuition behind the triplet loss: after training, a positive sample is “moved” closer
to the anchor sample than the negative samples of the other classes.
B. Demir 8
MiLAN: Results
t-SNE scatter plot: KSLSH
t-SNE scatter plot: MiLaN
T-SNE: t-distributed stochastic
neighbor embedding
B. Demir 9MiLAN: Results
Figure: (a) The query image, (b) Images retrieved by KSLSH.
and (c) Images retrieved by the MiLaN.
B. Demir 10Image Description: Pre-Trained Models
Use of deep learning models pre-trained on large scale computer vision archives
(e.g., ImageNet)
Problem: Differences on the characteristics of images between computer vision and
remote sensing.
B. Demir 11BigEarthNet: A Large-Scale Benchmark
Archive
Available at http://bigearth.net!
The BigEarthNet is a new large-scale Sentinel-2 benchmark archive, consisting of
590,326 Sentinel-2 image patches.
G. Sumbul, M. Charfuelan, B. Demir, V. Markl, “BigEarthNet: A Large-Scale Benchmark Archive for Remote Sensing Image Understanding", IEEE
International Conference on Geoscience and Remote Sensing Symposium, Yokohama, Japan, 2019.
B. Demir 12BigEarthNet: A Large-Scale Benchmark
Archive
To construct the BigEarthNet, 125 Sentinel-2 tiles (associated to cloud cover percentage
less than 1%) acquired between June 2017 and May 2018 were selected.
Considered tiles are distributed
over the 10 countries of Europe:
• Austria
• Belgium
• Finland
• Ireland
• Kosovo
• Lithuania
• Luxembourg
• Portugal
• Serbia
• Switzerland
All the tiles were atmospherically
corrected.
B. Demir 13BigEarthNet: A Large-Scale Benchmark
Archive
Tiles were divided into 590,326 non-overlapping image patches, each of which has
size of 120x120 pixels in 10-meter resolution.
Each image patch is associated with one or more land-cover class labels provided
from the CORINE Land Cover database of the year 2018 (CLC 2018).
CLC 2018 has been produced by the European Environment Information and
Observation Network of the European Environment Agency.
B. Demir 14BigEarthNet: A Large-Scale Benchmark
Archive
The number of labels associated with each image patch varies between 1 and 12,
whereas 95% of patches have at most 5 multi-labels.
Continuous urban fabric, Non-irrigated arable land, Coniferous forest, Discontinuous urban fabric,
Green urban areas Mixed forest, Mixed forest, Construction sites,
Discontinuous urban fabric. Water bodies, Green urban areas
Transitional woodland/shrub.
Images acquired in different seasons are considered.
Seasons Number of Image Patches
Autumn 154,943
Winter 117,156
Spring 189,276
Summer 128,951
B. Demir 15Cloud, Cloud Shadow and Snow
Identification
Cloud shadow and cloud present within some patches:
Some patches include seasonal snow:
After neglecting them, 519,339 patches remain.
B. Demir 16Pre-Trained Deep Learning Models on
BigEarthNet
Available at http://bigearth.net!
B. Demir 17BigEarthNet: Results
Methods Recall F1 Score F2 Score
Pre-trained Inception-v2 on ImageNet* 43.5 % 0.45 0.44
RGB Bands in Standard CNN 56.8 % 0.57 0.56
All Bands in Standard CNN 62.0 % 0.61 0.62
All Bands in K-Branch CNN 71.5 % 0.67 0.70
* We apply fine-tuning to the pre-trained Inception-v2 architecture.
B. Demir 18Challenging CLC Classes in BigEarthNet
Some CLC classes are challenging to be accurately described by considering only
(single-date) Sentinel-2 images.
Image
Scene
Continuous urban fabric
Industrial or commercial units
Non-irrigated arable land Port Areas
Multi-Labels Construction sites
Airports Industrial or commercial units
Pastures
Industrial or commercial units Green Urban Areas
Road and rail networks and associated land
Estuaries
Image
Scene
Coniferous forest Transitional woodland-shrub
Discontinuous urban fabric Coniferous forest Agro-forestry areas
Multi-Labels Sport and leisure facilities Non-irrigated arable land Rice fields
Port areas Land principally occupied by agriculture, with Olive groves
Road and rail networks and associated land significant areas of natural vegetation Non-irrigated arable land
Coastal lagoons Pastures
B. Demir 19BigEarthNet-19
BigEarthNet-19 Class Labels
Urban fabric
Industrial or commercial units
Arable land
Permanent crops
Pastures
Complex cultivation patterns
Land principally occupied by agriculture, with significant areas of natural vegetation
Agro-forestry areas
Broad-leaved forest
Coniferous forest
Mixed forest
Natural grassland and sparsely vegetated areas
Moors, heathland and sclerophyllous vegetation
Transitional woodland, shrub
Beaches, dunes, sands
Inland wetlands
Coastal wetlands
Inland waters
G. Sumbul, J. Kang, T. Kreuziger, F. Marcelino, H. Costa, P. Benevides, M. Caetano, B. Demir, “BigEarthNet Deep Learning Models with A New
Class-NomenclatureMarine waters
for Remote Sensing Image Understanding”, arXiv:2001.06372, January, 2020.
B. Demir 20Impact of BigEarthNet
More than 9K downloads in a year.
BigEarthNet is included in TensorFlow.
B. Demir 21Concluding Remarks
BigEarthNet makes a significant advancement for the use of deep learning in RS. We
plan to regularly enrich the BigEarthNet by:
Extending it to whole Europe;
Including Sentinel-1 patches;
Including auxiliary data, e.g., DEM;
Including the class-wise appearance percentages in a given patch.
Hashing-based methods are promising for RS retrieval problems due to their capability
on fast and scalable image search and retrieval. We currently work on:
Multi-modal hashing;
Uncertainty sensitive hashing;
Attention guided hashing;
Volunteered geographic information driven hashing.
B. Demir 24Accurate and Fast Discovery of Crucial Information for
Observing Earth from Big EO Archives
http://bigearth.eu/
B. Demir 23You can also read
