LIDAR- AND AR-BASED MONITORING OF EVOLVED BUILDING FAÇADES UPON ZONING CONFLICTS - MDPI
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
sensors
Article
LiDAR- and AR-Based Monitoring of Evolved
Building Façades upon Zoning Conflicts
Naai-Jung Shih * and Yi Chen
Department of Architecture, National Taiwan University of Science and Technology, Taipei 106, Taiwan;
chrisiyi.216@gmail.com
* Correspondence: shihnj@mail.ntust.edu.tw; Tel.: +886-(2)-27376718
Received: 13 September 2020; Accepted: 25 September 2020; Published: 1 October 2020
Abstract: Zoning conflicts have transformed Old Street fabrics in terms of architectural style and
construction material in Lukang, Taiwan. This transformation should be assessed as a contribution to
digital cultural sustainability. The objective of this study was to compare the evolved façades resultant
from the changes made by the development of architectural history and urban planning. A combination
of 3D scan technology and a smartphone augmented reality (AR) app, Augment® , was applied to the
situated comparison with direct interaction on-site. The AR application compared 20 façades in the
laboratory and 18 façades in four different sites using a flexible interface. The comparisons identified
the correlation of evolved façades in real sites in terms of building volumes and components,
pedestrian arcades on store fronts, and previous installations. The situated comparisons were
facilitated in a field study with real-time adjustments to 3D models and analyses of correlations across
details and components. The application of AR was demonstrated to be effective in reinstalling scenes
and differentiating diversified compositions of vocabulary in a remote site.
Keywords: markerless AR; LiDAR; old street heritage; building façade; architecture style; situated
study; smartphone
1. Introduction
The Old Street in Lukang is replete with historical buildings (Figure 1). Lukang had originally
been reconstructed from a harbor before it assumed its current fabric. Its historical identity was
reshaped from a place that could hardly achieve a thorough view of the sky, to a representative icon of
cultural sustainability. Its tangible pattern of development can be traced back over 300 years. The fabric,
which was created from the beginning to the current setting of support, was reshaped in an evolved
manner. The long-term developed urban fabric has transformed the township with the landscape
of the streets being reconstructed and conserved. This evolution, which occurred within each block,
dramatically impacted the construction of building style.
This transformation should be assessed as a fulfillment of cultural sustainability. The assessment
of heritage can benefit from rapid technological advancements in such fields as human computer
interaction, information modeling, construction, situated learning, tourism, and 3D data acquisition.
For example, augmented reality (AR) has been widely and successfully applied in recent years.
AR enables real and virtual information in an actual environment to be combined, registered,
and interacted upon in real-time [1–3]. This interaction is particularly important in tacit knowledge,
which is typically challenging to articulate to others [4]. Similar to case studies of environments prior to
architecture design, AR can present virtual content with the real world around the user [5]. This virtual
content can be created from models originated from the physical world with a high level of realistic
appearance. The integration of three different modeling methodologies (TLS, GPR, 3D modelling) were
Sensors 2020, 20, 5628; doi:10.3390/s20195628 www.mdpi.com/journal/sensors
Sensors 2020, 20, 5628 2 of 26
developed to create an educational app [6]. Users can explore reality with new layers of information
using 2020,
Sensors mobile
20, xAR applications
FOR PEER REVIEWfor a novel interactive and highly dynamic experience [7,8]. 2 of 26
Figure
Figure 1. Street
Street scenes
scenes of
of the
the Old
Old Street
Street in Lukang, Taiwan.
AR applications
This transformation have beenbesuccessfully
should assessed as aimplemented
fulfillment of in a broad
cultural range of disparate
sustainability. The assessmentfields,
including navigation, education, industry, medical practice, and
of heritage can benefit from rapid technological advancements in such fields as human computer landscape architecture [9–13].
Technically, information
interaction, huge sets ofmodeling,
input data have been situated
construction, registered to reconstruct
learning, tourism,accurate
and 3D data 3D city models
acquisition.
for AR applications [14]. Moreover, extended reality and informative
For example, augmented reality (AR) has been widely and successfully applied in recent years. models have been createdAR
for architectural
enables real andheritage
virtual from scans toinbuilding
information an actual information
environment modeling
to be (BIM),
combined, virtual reality (VR),
registered, and
and AR [15].
interacted uponHeritage building
in real-time information
[1–3]. modeling
This interaction (HBIM) embodies
is particularly important complexity in surveying
in tacit knowledge, and
which
preservation [16]. Geological view of relics and landscapes enhances the
is typically challenging to articulate to others [4]. Similar to case studies of environments prior to social awareness of culture
and the protection
architecture design,of AR heritage [17]. virtual content with the real world around the user [5]. This virtual
can present
content can be created from models were
Construction-specific studies madefrom
originated thatthefocused
physical onworldsafetywith with the finer
a high level ofdetails of
realistic
VR/AR environment to attract significant attention [18], to simulate
appearance. The integration of three different modeling methodologies (TLS, GPR, 3D modelling) the environmental context
and spatio-temporal
were developed to create constraints of various processes
an educational app [6]. [19],Userstocan conduct
explore structural
realityanalysis
with new to determine
layers of
behavior under different loading conditions of interactive 3D models
information using mobile AR applications for a novel interactive and highly dynamic experience [20], to setup tasks on a variety
[7,8].
of machine tools basedhave
AR applications on target
beenmarkers [21], and
successfully to contribute
implemented in toa online
broad education for management
range of disparate fields,
and engineering
including [22]. Dimensional
navigation, inconsistencies
education, industry, medical of building
practice,designand amonglandscape participants
architecturein VR[9–13].
seem
to be partially alleviated through AR by adding a real object that is accurately
Technically, huge sets of input data have been registered to reconstruct accurate 3D city models for scaled [23].
AR offers practical
AR applications advantagesextended
[14]. Moreover, because it allowsand
reality educators
informativeto usemodelsreal objects.
haveEducation-related
been created for
studies wereheritage
architectural carried from
out focusing
scans to on STEMinformation
building [24], m-learningmodeling (mobile
(BIM), learning), and traditional
virtual reality (VR), and
e-learning (electronic learning) [25], to enable students to locate
AR [15]. Heritage building information modeling (HBIM) embodies complexity in surveying any object represented in a given
and
space [26], to [16].
preservation achieve a better view
Geological understanding
of relics and of landscapes
cultural aspectsenhances [27], totheenhance laboratoryoflearning
social awareness culture
environments
and the protection[28],ofand to develop
heritage [17]. new learning paradigms in outdoor co-space and implement a
mobile AR system for creative
Construction-specific design
studies werecourses
made that[29].focused
3D moduleson safety can with
be difficult
the finerto details
obtain,of and their
VR/AR
operation should
environment to not be too
attract complicated
significant in use in
attention situated
[18], learning the
to simulate [30]. environmental
The link between real world
context and
objects and digital media should bring a new and beneficial quality for
spatio-temporal constraints of various processes [19], to conduct structural analysis to determine learning [31].
VR and
behavior underARdifferent
have been of great
loading interest to
conditions of tourism
interactive researchers
3D models [32,33].
[20], toTourism
setup tasksexperiences
on a variety can
be machine
of transformed
tools by these
based on technologies
target markers into virtual
[21], and totours and augmented
contribute tourism experiences
to online education for management [34].
Concerning
and tourism,
engineering [22].AR can supplyinconsistencies
Dimensional tourists with detailed
of building and design
visceralamong local information
participants[35–38].
in VR
In addition,
seem the utilities
to be partially of VRthrough
alleviated and ARAR in by
communication
adding a real object and engagement
that is accuratelycan bescaled
understood
[23]. or
improved through
AR offers interactions
practical with because
advantages traditional 3D scenes
it allows or entities.
educators to useTo achieve
real this,
objects. the information
Education-related
model should
studies be extended
were carried to a direct
out focusing onmanipulation of 3D building
STEM [24], m-learning façades
(mobile which possess
learning), a strong
and traditional
relation to the
e-learning past development
(electronic learning) [25],of local urban students
to enable architecture. The exemplification
to locate any object represented of components
in a given can
space [26], to achieve a better understanding of cultural aspects [27], to enhance laboratory learning
environments [28], and to develop new learning paradigms in outdoor co-space and implement a
mobile AR system for creative design courses [29]. 3D modules can be difficult to obtain, and their
operation should not be too complicated in use in situated learning [30]. The link between real world
objects and digital media should bring a new and beneficial quality for learning [31].
Sensors 2020, 20, 5628 3 of 26
not only make tourists knowledgeable in rich and novel ways, but also create a pragmatic pedagogical
environment through AR technology. To promote similar experiences in a wider range of settings,
a representation of past virtual environments can enrich the structure of systems for more vivid
involvement either on the same streets or in remote cities.
Problems, however, can occur with AR under different technology limits. Marker-based AR and
markerless AR constitute the two major AR types [39]. Marker-based AR in outdoor settings is usually
constrained by lighting conditions of the environment, weather, or time of day. To overcome this
challenge, markerless outdoor tracking was developed, which uses images created from different
daylight conditions [40]. Difficulty and aesthetic concerns were involved in arranging and pasting
markers on building surfaces, and markerless AR constitutes a more suitable approach for diversified
scenes [41,42].
A detailed visual representation was proven to be requisite to achieve a connection to reality to
increase comprehension. Light detection and ranging (LiDAR) has been applied to 3D scan technology
to document as-built scenes for construction monitoring [43–45] or to retrieve geometric features
from a distance with a combination of aerial images [46–51]. In contrast to creating 3D models from
the beginning, 3D laser scanning is an effective tool for acquiring important geometric attributes,
such as detailed trees and vegetation [52]. 3D data acquisition should be extended to provide a
higher fidelity experience of the particular subject. Ground scans have been shown to be feasible for
accurate street data retrieval. The combination of LiDAR and AR was proven to be both interesting
and promising [53–55], and should be explored further.
In order to provide an effective situated comparison experience, a smartphone should be considered
as a platform through which to interact with AR. The interface should also work with Cloud support
with flexibility in actual locations or laboratory settings, so that architecture tours and extended field
study experiences can be applied in diverse urban environments.
2. Research Goal
The objective of this study was to provide a support environment to assist situated comparisons
of façades upon zoning conflict and evolution. The evolved façade needs to be assessed with direct
interaction to correlate components and details in reality or in real sites. The comparison and interaction
can be extended from the study conducted at Lukang Township to a remote site as a way to present
and verify research findings.
This study monitored and mapped the profiles of building forms upon zoning changes made
within the same street block in Old Street. Urban zoning control was commonly applied to existing
open spaces, such as streets, to separate commercial areas and form preservation areas. Since 1986,
street blocks near Old Street have been controlled by creating separation of the two types of areas
within the same block. 3D data have to be documented as needed in order to obtain an instantaneous
side-by-side study of the effects of the two types of changes to the same street block. It is noted that
this process can then also be beneficially applied to diverse adaptions of form in various other cities.
In order to make a comparison to a real scene, a higher fidelity of appearance must be provided.
LiDAR and AR-based building form mapping and monitoring are needed to achieve this. In some
cases, a mesh model must also be applied to overlap with the wireframe appearance of data for clear
indications of molding locations.
Sensors 2020, 20, 5628 4 of 26
3. Research Methodology
A situated comparison was carried out in this research. In order to verify the evolved façade,
the proposed process included inspecting restructured space, applying 3D scans and supporting
interaction devices, correlating both sides of the façade in AR, AR-assisted comparisons and scene
reinstallations, and verification. The comparison needs an accurate assessment of configuration and a
user-friendly interface with which to work. The assessment should be made by correctly reconstructed
spaces or objects as a reference. In order to match with a realistic scene in an architecture site,
reconstruction by 3D scans offers an effective approach to capture precise dimensions and correct
relative locations in as-built form. The realistic 3D scan data will facilitate field visits and comparisons
with great visual similarity and dimensional accuracy, instead of a simplified representation of vector
drawings of the original design. The comparison also requires a user-friendly interface to correlate
subjects in the foreground and the background in real time. Indeed, the AR setup establishes an
optimal environment to interact with photorealistic subjects and backgrounds.
We aim to apply a public platform to support these zoning conflict comparisons. Our former
researches developed a customized AR app using ARkit 1.5® to decomposed specific spatial structures
with display options [41]. Although the research purposes were fulfilled, the recycling of the original
codes or design was not optimal. A platform which provides everyday Internet 3D browsing experience
for the young generation would constitute a promising choice. A commercial platform, Augment® [56],
which provides end-to-end AR, was selected as an approach for 3D visualization and communication
in this paper.
In contrast to pre-defined solutions of e-commerce, sales and marketing and platform design,
we intend to utilize it as a study environment for architectural façades by comparing its development
due to zoning changes and geographic differences. This environment should support façade overlay
with/out the real context of background accessed either in the same site or in different cities. An easy
access without a predefined structure may provide another type of flexibility in architectural interaction,
with the least possible development and maintenance costs.
An emphasis is given to the preparation of 3D façade models. A similar concept of scalable model
can also be found in Sketchfab® , which takes the 3D point cloud for the VR display. Augment® was
selected because multiple AR objects can be displayed in the same scene, to be interacted with, within a
single subscriber account. Although the display of each individual object cannot be turned on or off as
needed, the application paradigm can be found in interior design with multiple pieces of furniture
involved in a layout process.
The proposed process was made from restructuring, correlating, 3D data retrieval and conversion,
scene comparison or reinstallation, and finally verification (Figure 2). The flowcharts include, (1) the
stepwise process of situated comparison by issues, subjects, and locations; and (2) the development of
data and related conversion to support the study.
Sensors 2020, 20, 5628 5 of 26
Sensors 2020, 20, x FOR PEER REVIEW 5 of 26
Figure 2.2.Flowchart
Flowchartof augmented reality
of augmented (AR)-assisted
reality situatedsituated
(AR)-assisted comparison and the data
comparison andconversion
the data
process to support
conversion processthe study. the study.
to support
4. The Restructuring Background of Old Street
4. The Restructuring Background of Old Street
Chenggong Road, which was the location of the former Lukang Creek (Figure 3), was originally a
Chenggong Road, which was the location of the former Lukang Creek (Figure 3), was originally
trade distribution center. Cargo, which was unloaded and sold on the riverside, was rearranged and
a trade distribution center. Cargo, which was unloaded and sold on the riverside, was rearranged
repackaged after being moved indoors and sold at store fronts on the other end of buildings facing
and repackaged after being moved indoors and sold at store fronts on the other end of buildings
Putou Street and Yaolin Street. The entire process occurred in a house approximately 4 m wide and
facing Putou Street and Yaolin Street. The entire process occurred in a house approximately 4 m
as much as 40 m deep. The houses located to the west side of Old Street possessed a harbor-facing
wide and as much as 40 m deep. The houses located to the west side of Old Street possessed a
geographical advantage, and the houses located to the east side were generally more inconvenient for
harbor-facing geographical advantage, and the houses located to the east side were generally more
the commercial process. An owner had to bypass Old Street to reach Houche Lane, where the service
inconvenient for the commercial process. An owner had to bypass Old Street to reach Houche Lane,
entrance was located. Houche Lane is located 40 m to the east of Old Street, and faces the rear entrance
where the service entrance was located. Houche Lane is located 40 m to the east of Old Street, and
of buildings on Zhongshan Road.
faces the rear entrance of buildings on Zhongshan Road.
Lukang Old Streets have had a well-preserved Minnan architecture since the Quin Dynasty.
Lukang Old Streets have had a well-preserved Minnan architecture since the Quin Dynasty.
The region defined by the green boundary (Figure 3, middle) is a heritage preservation area, while the
The region defined by the green boundary (Figure 3, middle) is a heritage preservation area, while
commercial area is in the red boundary. The famous Old Street is located at the center of the preservation
the commercial area is in the red boundary. The famous Old Street is located at the center of the
area. In general, the street and houses on it exist under the same zoning classification. The two areas,
preservation area. In general, the street and houses on it exist under the same zoning classification.
however, are now divided by a jagged form that follows the property line. The consistency of the
The two areas, however, are now divided by a jagged form that follows the property line. The
façade in each area stopped in the middle of a block. The lot, which was previously an undivided
consistency of the façade in each area stopped in the middle of a block. The lot, which was
property, was separated irrespective of the original space hierarchy at which approximately the third
previously an undivided property, was separated irrespective of the original space hierarchy at
or fourth hall was located (Figure 4). The traditional layout between the main and rear entrance was
which approximately the third or fourth hall was located (Figure 4). The traditional layout between
no longer connected from the interiors. A new entrance had to be created, or the interiors had to
the main and rear entrance was no longer connected from the interiors. A new entrance had to be
be remodeled from a space next to the service entrance, in order to establish a feasible living space.
created, or the interiors had to be remodeled from a space next to the service entrance, in order to
This type of differentiation was achieved by a long-term development of urban fabric (Figure 4, right).
establish a feasible living space. This type of differentiation was achieved by a long-term
The restructuring of living spaces has occurred since 1986. Since Houche Lane is located next to both
development of urban fabric (Figure 4, right). The restructuring of living spaces has occurred since
the preservation and commercial area, it is necessary to explore how the building style has developed.
1986. Since Houche Lane is located next to both the preservation and commercial area, it is necessary
to explore how the building style has developed.
Sensors 2020, 20, 5628 6 of 26
Sensors2020,
Sensors 2020,20,
20,xxFOR
FORPEER
PEERREVIEW
REVIEW 66of
of26
26
Figure3.3. Site
Figure Siteplan
planof
plan ofthe
of theold
the oldharbor
old harbor[57],
harbor [57],zoning
[57], zoningmap,
zoning map,
map, and
and
and differentiated
differentiated
differentiated street
street
street block
block
block (top,
(top,
(top, from
from
from left
leftleft
to
to right);
to right);
right); cross
cross
cross section
section
section of four
of
of four four streets
streets
streets (bottom).
(bottom).
(bottom).
Figure4.4. The
Figure The section
section before
before and
and after
after zoning
zoning reconfiguration
reconfiguration (left);
reconfiguration (left); the
(left); the different
different space
spaceand
and the
theroof
roof
configurationof
configuration ofspaces
spacesononboth
on bothends
both endsof
ends ofthe
of theblock
the block(right,
block (right,top,
(right, top,and
top, andbottom).
and bottom).
bottom).
Due
Due to the spatial
to the spatialstructure
spatial structurearrangement
structure arrangementused
arrangement usedto
used toto connect
connect
connect the
the the harbor
harbor
harbor front
front
front to the
to the
to the shop
shop shopfront,
front, front,
the
the
the property
property line line
is is linear,
linear, with with
a a
large large depth-to-width
depth-to-width ratio. ratio.
An An
old
property line is linear, with a large depth-to-width ratio. An old property usually occupied an entire old
propertyproperty
usually usually
occupiedoccupied
an an
entire
entire
street street
street block block
block from one
from from
one end oneto
end end
to the
the toother.
the other.
other. Long-term
Long-term
Long-term zoning
zoning zoning development
development
development had reconstructed
had reconstructed
had reconstructed two
two
two different
different
different zones
zones zoneswithin
withinwithin a single
aa single
single street
street
street block,
block,
block, and
and and dramatically
dramatically
dramatically reshapedthe
reshaped
reshaped theurban
the urbanfabric
urban fabricin
fabric in aa 3D3D
configuration.
configuration. The
configuration. The division
The division separated
division separated
separated an an integrated
an integrated traditional
integrated traditional hierarchy
traditional hierarchy with
hierarchy with sufficient
with sufficient space
sufficient space from
space from 3
fromto
433fully
to developed
to 44 fully
fully developed
developed halls to 1–2 to
halls
halls halls
to 1–2
1–2 with insufficient
halls
halls space forspace
with insufficient
with insufficient eitherfor
space part
for of spatial
either
either part structure.
part spatial Moreover,
of spatial
of structure.
structure.
each space each
Moreover,
Moreover, type was
each space
space developed
type was
type with
was developed
developed different
with
with materials,
differentvocabularies,
different styles, and styles,
materials, vocabularies,
materials, vocabularies, spatialand
styles, structures
and spatial
spatial
within the same
structures within
structures block.
within the In
the same fact,
same block.great
block. In contrast was
In fact, established
fact, great
great contrast by
contrast was having a horizontally-developed
was established
established by by having
having aa
structure of traditional architecture
horizontally-developed
horizontally-developed structure of
structure ofnext to a vertically-developed
traditional
traditional architecture next
architecture next to structure of street-facing buildings.
to aa vertically-developed
vertically-developed structure
structure
of The zoning
street-facing differentiation
buildings.
of street-facing buildings. created problems for the truncated part of the traditional architecture.
Specifically,
The zoning
The the dated
zoning layout and differentiation
differentiation
differentiation created problems
created problemsof original
for the
for functions
the truncated
truncated in various
part of
part zones
of theconstrained
the traditional
traditional
the size and Specifically,
architecture.
architecture. flexibility
Specifically, oftheexisting
the datedspaces
dated from
layout and
layout supporting many
and differentiation
differentiation types functions
of original
of original of functions.
functions In order
in various
in various zones
zones to
meet commercial
constrained the
constrained the size and
size anddaily needs,
and flexibility
flexibility of and simultaneously
of existing
existing spaces
spaces from maintain
from supporting the environment,
supporting many many types
types ofsecond building
of functions.
functions. In In
skins
orderwere
order to meet
to created
meet in an additive
commercial
commercial and daily
and dailyor subtractive and manner
needs, and
needs, (Figure maintain
simultaneously
simultaneously 5). The skin
maintain thewas
the accomplished
environment,
environment, second
second by
extending
building skins
building the
skinsmissing
werepart
were createdof an in
created interior
in an space to the
an additive
additive or exterior
or subtractive
subtractive wallsmanner
or open (Figure
manner spaces
(Figurefor laundry
5).
5). The skin
The facilities,
skin was
was
HVAC (heating,
accomplished by
accomplished ventilation,
by extending
extending the and air
the missing conditioning)
missing part part of
of an units,
an interior canopies,
interior space
space to posters,
to the or shop
the exterior signage,
exterior walls
walls or in
or open fixed
open spaces
spacesor
flexible
for laundry
for settings.
laundry FixedHVAC
facilities,
facilities, canopies,
HVAC for example,
(heating,
(heating, wereand
ventilation,
ventilation, usually
and installed withunits,
air conditioning)
air conditioning) an associated
units, canopies,
canopies, platform
posters,for
posters, or
or
booths
shop to sell merchandise.
shop signage,
signage, in fixed
in fixed or All of the
or flexible
flexible commercial
settings.
settings. installations
Fixed canopies,
Fixed canopies, for forindicate
example,
example, a strong purposeinstalled
were usually
were usually of interstitial
installed with
with
space
an betweenplatform
an associated
associated the streetfor
platform and
for indoors.
booths
booths to sell
to sell merchandise.
merchandise. All All ofof the
the commercial
commercial installations
installations indicate
indicate aa
strong purpose
strong purpose of of interstitial
interstitial space
space between
between the the street
street and
and indoors.
indoors.
Sensors 2020, 20, 5628 7 of 26
Sensors
Sensors 2020,
2020, 20,
20, xx FOR
FOR PEER
PEER REVIEW
REVIEW 77 of
of 26
26
Figure
Figure 5.
5. Objects
Objects added
added to
to the
the second
second skin
skin between
between 2019
2019 (top)
(top) and
and 2011
2011 (or
2011 (or 2012)
(or 2012) (bottom).
2012) (bottom).
(bottom).
Additive interstitial
Additive interstitial space
space was
was developed
developed withwith an object-subtraction
an object-subtraction approach of
object-subtraction approach of posters
posters oror
free-standingantennas.
free-standing antennas.Consciousness
antennas. Consciousness
Consciousness and
andresponsibility
and responsibility
responsibilitywere also also
were
were developed
also to rearrange
developed
developed to the layout
to rearrange
rearrange the
the
of booths,
layout
layout of landscape,
of booths,
booths, and personal
landscape,
landscape, and belongings.
and personal
personal The addition
belongings.
belongings. The or
The subtraction
addition
addition or of installations
or subtraction
subtraction of represents
of installations
installations
an adjustment
represents
represents an to the balance
an adjustment
adjustment tobetween
to the cultural
the balance
balance identity
between
between and facilities
cultural
cultural identityfor
identity commercial
and
and facilities activities
facilities for within
for commercial
commercial
activities
the within the same historical street space. Indeed, the street space was
activities within the same historical street space. Indeed, the street space was constructed uponof
same historical street space. Indeed, the street space was constructed upon a constructed
hieratical upon
structure aa
hieratical
hieratical structure
demands and of
of demands
supporting
structure and
facilities
demands and supporting
for 300 years.facilities
supporting facilities for
for 300
300 years.
years.
5.
5. Correlating Both
5. Correlating
Correlating Both Sides
Both Sides of
Sides of the
of the Façade
the Façade
Façade
Since
Since zoning
Since zoning
zoning led led to
led tothe
to thedevelopment
the developmentof
development ofdifferent
of differentevolving
different evolvingfaçades
evolving façadesand
façades andbuilding
and building
building vocabularies,
vocabularies,
vocabularies, aa
acomparison
comparison
comparison mademade across
across zones,
zones, oror even
even across
across cities,
cities, should
should be
be
made across zones, or even across cities, should be able to confirm deviations and able
able to
to confirm
confirm deviations
deviations and
and
achieve
achieve aaa broader
achieve broader understanding
broader understanding
understanding of of the
of the architectural
the architectural transformation.
architectural transformation.
transformation. The The first
The first problem
first problem encountered
problem encountered
encountered
was
was determining
was determining
determining how how
how to to bring
to bring
bring the the façades
the façades together
façades together
together fromfrom two
from two opposite
two opposite sides of
opposite sides
sides of aaa single
of single street
single street block.
street block.
block.
Although
Although two photographs or drawings of elevations
Although two photographs or drawings of elevations can be placed side-by-side or even
two photographs or drawings of can
elevations be placed
can be side-by-side
placed or even
side-by-side overlapped
or even
to highlightto
overlapped
overlapped changes
to highlight
highlight in changes
an off-site
changes inlaboratory
in an
an off-site setting,
off-site laboratory
laboratory 2D-based
setting,verification
setting, 2D-based is made with
2D-based verification
verification is a limited
is made
made with
with
aunderstanding
a limited
limited understandingof depth. of
understanding Otherwise,
of depth. sections have
depth. Otherwise,
Otherwise, sectionsto be
sections brought
have
have to beup
to be as needed
brought
brought up
up as asinneeded
order to
needed incorrelate
in order
order toto
the differences.
correlate
correlate thethe differences.
differences.
To
Tosolve
To solvethe
solve theproblem
the problem
problem ofof correlating
of correlating
correlating 3D3Ddatadata
3D in different
data in locations,
in different
different we proposed
locations,
locations, we
we proposed a solution
proposed for off-site,
aa solution
solution for
for
or even on-site,
off-site,
off-site, or eveninspection.
or even on-site, The approach
on-site, inspection.
inspection. The applies theapplies
The approach
approach AR interface
applies the ARtointerface
the AR bring two
interface to3D
to bringmodels
bring twotogether
two 3D
3D models
modelsfor
an intuitive
together
together forinspection
for an
an intuitive with inspection
intuitive ainspection
beneficial withmanipulation
with aa beneficial of transition,
beneficial scale, and/or
manipulation
manipulation of orientation.
of transition,
transition, In contrast
scale,
scale, and/or
and/or
to superimposition
orientation.
orientation. In
In contrast of real
contrast toscenes
to using a virtual
superimposition
superimposition of 3D model
of real
real scenescreated
scenes using aby
using software
a virtual
virtual 3Dfrom
3D model
model thecreated
beginning,
created by
by
the fidelity
software fromof the
the virtual
beginning, model the should
fidelity provide
of the sufficient
virtual modelinformation
should
software from the beginning, the fidelity of the virtual model should provide sufficient information to
provide identify dissimilarities
sufficient information in
visual
to and
to identify structural
identify dissimilarities details.
dissimilarities in in visual
visual andand structural
structural details.
details.
A
A total
A total
total ofof
of 4040 buildings,
40 buildings,
buildings, i.e., i.e., 20
i.e., 20 on
20 on Old
on Old Street
Old Street and
Street and
and 20 20 on
20 on Houche
on Houche Lane,
Houche Lane, were
Lane, were selected
were selected according
selected according
according to to
to
four
four categories
four categories
categories basedbased
based onon the
on the types
the types of
types of changes
of changes existing
changes existing
existing onon both
on both sides
both sides (Figure
sides (Figure 6).
(Figure 6). Approximately
6). Approximately
Approximately 63% 63% 63%of
new
ofof new
newconstructions
constructions
constructions ononHouche
on Houche
Houche Lane
Lane
Lane are
aretaller
are tallerthan
taller thanthe
than thebuildings
the buildingsthat
buildings thatused
that usedto
used tobe
to belocated
be located within
located within the
within the
the
same
same property
same property boundary,
boundary,while
property boundary, whileanother
while another15%
another 15%of
15% ofplaces
of placeshave
places haveremained
have remainedvacant.
remained vacant.
vacant.
Figure The
Figure 6.6. The four
Thefour volumetric
fourvolumetric
volumetric combinations
combinations
combinations of of
of the
the constructions
constructions
the on
on Old
on Old
constructions Street
Street
Old and
and Houche
and Houche
Street Lane
Lane (left),
Houche Lane
(left),
and the
(left), and
new
and the new
new constructions
theconstructions that
that occurred
that occurred
constructions on
on Houche
on Houche
occurred Lane
Lane between
Lane between
Houche 2019
2019 and
between and
2011
2019 2011
2011 (right).
and(right).(right).
Sensors 2020, 20, 5628 8 of 26
6. Design of the Situated Comparison in AR
The application of AR was carried out based on a combination of appropriate types of 3D
models and background scenes in a remote site. Since the models can be downloaded as required,
the comparison becomes portable, in terms of the database, with the support of a smartphone platform.
The AR application serves the following three purposes:
• Correlating parts at different locations: Geographically-distributed building parts were brought
together side-by-side or overlaid between different contexts to achieve a paired comparison.
• Applying higher fidelity of part appearance: The selected level of fidelity can be a critical factor in
the comparison. The target façades on Old Street were documented using a mid-range 3D laser
scanner for visual and structural details in higher fidelity.
• Situated comparison using a flexible transformation interface: Transition, scaling, and/or rotation
were made possible by adjusting parts for alignment in terms of proportion, length, width,
or depth.
6.1. A Reality-to-Reality Environment and a Wireframe-to-Reality Environment
In order to fulfill the needs of an accurate comparison, the application facilitated an appropriate
manipulation interface and versatile use environment in the following ways:
• A reality-to-reality environment: Two mesh models were located side-by-side to facilitate the
inspection of alignment between moldings or openings across the two models. An overlaid layout
was selected to support the inspection of depth in various levels.
A 3D mesh model over a real scene: A background context-concerned environment was
applied for a meaningful study of development.
Two 3D models on a plain, simplified, or context-free background: Two 3D mesh models
were applied simultaneously with or without the real scene in the background.
A number of 3D models over a real background: This is presented in a way that is similar
to a set of instances to be applied for inspection.
• A wireframe-to-reality environment: Although vector drawings usually have lower fidelity of
details, the enhanced description of an object’s edges provides an advantage when overlaid on
top of the mesh model. The wireframe constituted a set of very thin polygons that can be seen
through to the 3D model behind it.
A wireframe model over a real background.
Two 3D models on a plain, simplified, or context-free background: A 3D mesh model
and a wireframe model were applied simultaneously with or without the real scene
in the background. The latter was represented in a more abstract form for simplicity
of comparison.
A number of wireframe models and 3D models over a real background.
6.2. Remote Comparisons
Comparisons were usually carried out in “remote mode”, in which a 3D model was brought up or
downloaded from the Cloud at a different location.
• The opposite side of the same street block: The model was used to compare the different façade
style developed on the opposite side of the same street block.
• Different street block: The model was placed on a property where the original house
was demolished.
• Different city: The model was attached to a façade located in a different city, where the original
style evolved in a dissimilar manner.
7. 3D Scans and Supporting Interaction Device
In contrast to drawing-based comparisons made by attaining graphics to illustrate differences,
the assessment should be made by higher fidelity 3D digital data which support a real-scene-based
on-site or remote inspection. Moreover, in order to support the on-site study, an accessible device
should2020,
Sensors be 20,
provided
5628 with the support of a 3D display and interaction, in addition to access to Cloud 9 of 26
data. Since AR has been broadly applied in architecture for a combination of 3D models and real
scenes, the assessment of differences can be achieved in an AR platform if a high level of interaction,
•higher
In afidelity
laboratory
of thesetting:
model, Theandmodel was downloaded
domain-specific in a laboratory
transformation setting without
are supported reference to
by a smartphone
any real background
platform using iOS® or Android®.for a background-free context test.
A street 200 m long was scanned (Figure 7). A mid-range Faro Focus 3D S120® laser scanner was
7. 3D Scans and Supporting Interaction Device
applied. In total, 4.97 GB of point cloud data was retrieved and stored in PLY format. The scan of the
In contrast
as-built to drawing-based
street scenes obtained a high comparisons made façades.
fidelity of street by attaining graphics to
The “Poisson illustrate
surface differences,
reconstruction”
the assessment should be made by higher fidelity 3D digital data which support
of the mesh model was created from the point cloud by selecting the “output density as SF” option a real-scene-based
on-site or remote inspection.
in CloudCompare ® (Figure 8,Moreover,
left). The in ordercreation
mesh to support the on-site
process was made study,inanGeomagic
accessibleStudio
device®
should be provided with the support of a 3D display and interaction, in addition
(Figure 2). A “wireframe-like” mesh model (Figure 3, right) was also created based on a field survey to access to Cloud
data.
using Since
AutoDeskAR has3DSbeen
MAXbroadly
® (Figure applied in architecture for a combination of 3D models and real
8, right).
scenes, the assessment
A collection of 20 of differences
paired façadescan onbe achieved
opposite in an
sides of AR
the platform
same streetif ablock
high level
(Figureof interaction,
9, left) was
higher fidelity of the model, and domain-specific transformation are
created by converting the 3D scan data to the mesh models from CloudCompare , Geomagic supported by a smartphone
®
platform
Studio®, using
AutoDeskiOS®3DSor Android
MAX®, or
®.
Meshlab®. The most frequent connection to the models was
A street 200 m long was scanned (Figure ® laser scanner was
through the smartphone app. An iPhone 10® 7). A mid-range
with iOS 12® andFaro Focus 3D
a Realme X50S120
5G® with Android 10®
applied.
were utilized as the smartphones for field comparison with Cloud links available from scan
In total, 4.97 GB of point cloud data was retrieved and stored in PLY format. The of the®
Augment
as-built
(Figure street scenes
9, right). A obtained
subscriptiona high fidelity of
Augment streetwas
® app façades. The as
applied “Poisson surface reconstruction”
a ready-made general-purpose of
the mesh model was created from the point cloud by selecting the “output
platform to alleviate AR software development and maintenance effort. It is an e-commerce density as SF” option in
CloudCompare ® (Figure 8, left). The mesh creation process was made in Geomagic Studio® (Figure 2).
platform for sales, marketing solutions, and design solutions. In this paper, we use it as an
A “wireframe-like”
“end-to-end AR andmesh model (Figure
3D visualization and3,communication
right) was alsoplatform”
created based
[56] ofoncultural
a field survey
artifactsusing
with
AutoDesk 3DS MAX ® (Figure 8, right).
so-called “online shopping experience” replaced by “online situated comparison experience”.
.
®
Figure 7. 3D
Figure 7. 3D scans
scans of
of Old
Old Street
Street shops
shops (top,
(top, bottom left)and
bottom left) andapplied
appliedFaro
FaroFocus
Focus3D
3DS120
S120® scanner
scanner
Sensors 2020, Germany)
(Faro, 20, x FOR PEER REVIEW
(bottom right). 10 of 26
(Faro, Germany) (bottom right).
Figure8.8.Mesh
Figure Meshmodel
modelcreation
creationprocess
processin
inCloudCompare v2.9.1®® and
CloudComparev2.9.1 and AutoDesk
AutoDesk 3DS
3DS MAX 2015®
MAX 2015 ®.
.
Sensors 2020, 20, 5628 10 of 26
Sensors 2020, 20, x FOR PEER REVIEW 10 of 26
A collection of 20 paired façades on opposite sides of the same street block (Figure 9, left) was
created by converting the 3D scan data to the mesh models from CloudCompare® , Geomagic Studio® ,
AutoDesk 3DS MAX® , or Meshlab® . The most frequent connection to the models was through the
smartphone app. An iPhone 10® with iOS 12® and a Realme X50 5G® with Android 10® were utilized
as the smartphones for field comparison with Cloud links available from Augment® (Figure 9, right).
A subscription Augment® app was applied as a ready-made general-purpose platform to alleviate
AR software development and maintenance effort. It is an e-commerce platform for sales, marketing
solutions, and design solutions. In this paper, we use it as an “end-to-end AR and 3D visualization
and communication platform” [56] of cultural artifacts with so-called “online shopping experience”
replaced by “online situated comparison experience”.
Figure 8. Mesh model creation process in CloudCompare v2.9.1® and AutoDesk 3DS MAX 2015®.
Figure 9. A
Figure 9. A collection
collection of
of paired
paired façades
façades on
on the
the opposite
opposite side
side of
of the
the same
same street
street block
block (left) and the
(left) and the
model list in Augment ® (right).
®
model list in Augment (right).
8. AR-Assisted Comparison and Scene Reinstallation
The comparison
comparisonwas wasconducted
conducted in in
thethe following
following fourfour stages:
stages: (1) selection
(1) selection of model
of model pairs;
pairs; (2) rough(2)
rough alignment;
alignment; (3)matching;
(3) detail detail matching;
and (4)and (4) context
context elaboration.
elaboration. Two façades
Two façades were selected
were selected with a
with a similar
similar appearance.
appearance. Rough alignment
Rough alignment wasin
was applied applied
order toinalign
orderthetotwo
align the two
models by models by translation
translation or rotation,
or rotation,
and and made
was usually was usually madeoftothe
to the corner thefaçade.
cornerDetail
of thematching
façade. Detail
applied matching applied
translation translation
and proportional
and proportional
scaling to a certain scaling
dimension to aincertain dimension
order to in order toinshow
show the difference heighttheor difference in height
width of a specific or width
component.
of a specific component.
Adjustments had to be made Adjustments
repetitivelyhad to be made
by selecting repetitively
appropriate by selecting
anchor nodes or appropriate anchor
edges as a reference.
nodes
This is or edges because,
required as a reference. This istwo
for example, required
windows because, for same
with the example,
width twodowindows with the
not necessarily havesame
the
widthheight.
same do not The
necessarily
first three have the same
stages enhancedheight. The first threeofstages
comprehension enhanced
the aspect comprehension
of focus, increased user of
the aspect of
familiarity, and focus, increased
achieved user familiarity,
the desired manipulation. andThe achieved the desired
final stage encouraged manipulation. The final
users to inspect any
stage encouraged
façade that they choseusers to inspect
in the AR app any façade that
by overlaying thethey chose
selected in the
model AR app
on-site withby theoverlaying
same stagesthe to
selected model
elucidate on-site with The
the transformations. the stages
same are stages to elucidate
intuitively performedthe with
transformations.
a few taps andThe stages are
movements of
intuitively
one or two performed
fingers. with a few taps and movements of one or two fingers.
Situated comparisons
comparisons in different cities were made through model adjustments, detail detail checks,
checks,
correspondingcomponents
and corresponding components(Figure (Figure 10).
10). TheThe initial
initial adjustments
adjustments allocated
allocated a façade
a façade modelmodel
in placein
place
by by referring
referring to a of
to a number number of heuristics.
heuristics. The reference Theinformed
referencethe informed
researcher theofresearcher of major
major differences in
differences More
proportion. in proportion.
detailed checksMorewere detailed
madechecks were madepartitions,
to the dimensions, to the dimensions,
or materials.partitions,
Adjustments or
materials. Adjustments
frequently frequently
occurred in switching occurred
between theinreference
switching betweenlevel
of ground the reference of ground
or wall molding, as alevel
way or to
wall molding,
elucidate as a way
the difference to elucidateproportion,
in composition, the difference in composition,
or missing components. proportion, or missing
components.Sensors 2020, 20, 5628 11 of 26
Sensors 2020, 20, x FOR PEER REVIEW 11 of 26
Figure 10. The situated comparison process.
The
The comparisons
comparisonswere wereconducted
conductedatat five sites:
five oneone
sites: laboratory site;site;
laboratory one one
site in Lukang;
site in Lukang;and three
and
sites
threein Taipei.
sites AR made
in Taipei. a greata contribution
AR made in comparing
great contribution the façade
in comparing styles styles
the façade on both onends
bothof a block.
ends of a
The façades on Lukang Old Street facilitated architectural study conducted from
block. The façades on Lukang Old Street facilitated architectural study conducted from different different viewpoints
of historical of
viewpoints culture, building
historical styles,
culture, and scales.
building styles,Aandstepwise
scales.operation
A stepwise andoperation
third-personand photographs
third-person
concerning
photographs on-site operations
concerning areoperations
on-site illustrated are
in Figure 11. The
illustrated in process
Figure 11.was performed
The process wasin several steps
performed
from scanning a QR code, pointing a smartphone camera and moving around
in several steps from scanning a QR code, pointing a smartphone camera and moving around to to determine the ground
plane, tapping
determine the the screenplane,
ground to anchor the 3D
tapping model
the screeninitially, and scaling
to anchor the 3Dormodel
moving the model
initially, andinscaling
referenceor
to the background. Each comparison or visit was followed by discussions
moving the model in reference to the background. Each comparison or visit was followed by which were separated into
three parts: (1)
discussions the preliminary
which were separatedpresentation
into threeof models;
parts: (1)(2)the
thepreliminary
on-site physical adjustments;
presentation and (3) the
of models; (2)
post-visiting group-discussion
the on-site physical adjustments;of findings withpost-visiting
and (3) the screen shotsgroup-discussion
for verification. The last part was
of findings withhelpful
screen
to recall
shots forignored components
verification. The lastor missed
part mentions
was helpful to of adjustments.
recall While mostor
ignored components ofmissed
the field work was
mentions of
conducted by an individual researcher, some of the field work was also performed
adjustments. While most of the field work was conducted by an individual researcher, some of the by two researchers,
which enabled
field work was them to compareby
also performed their
twofindings.
researchers, which enabled them to compare their findings.Sensors 2020, 20, 5628 12 of 26
Sensors 2020, 20, x FOR PEER REVIEW 12 of 26
Figure
Figure 11.
11. The
The screen
screen shot
shot (top),
(top), third-person
third-person photographs
photographs (middle),
(middle), and
and the
the stepwise
stepwise operation
operation
process:
process: launch app, scan QR code of a selected 3D model, detect plane, tap to allocate
launch app, scan QR code of a selected 3D model, detect plane, tap to allocate the
the model,
model,
and
and adjust
adjust the
the model’s
model’s scale,
scale, location,
location, or
or rotation angles (bottom).
rotation angles (bottom).
8.1. Laboratory
8.1. Laboratory Site
Site
The first
The firstsetset
of comparisons,
of comparisons, whichwhich
included the original
included the 40 models40
original with two sets
models of heterogeneous
with two sets of
façades, were made
heterogeneous to the
façades, set of
were wireframe-like
made to the set ofmodels (20 models)
wireframe-like and mesh
models (20 models
models)(20 models)
and mesh
on a plain background in the interiors. Findings were obtained during the interaction
models (20 models) on a plain background in the interiors. Findings were obtained during the between the
two types of models. Even without the real context of a scene presented in the background,
interaction between the two types of models. Even without the real context of a scene presented in different
forms
the and layout compositions
background, different forms were identified
and in the two sets.were
layout compositions The laboratory
identified site wastwo
in the usedsets.
to make
The
a comparison irrespective of background (Figure 12) or prior to the field application.
laboratory site was used to make a comparison irrespective of background (Figure 12) or prior to Using a mesh
model
the fieldand a wireframe-like
application. Using amesh
meshmodel
modelhighlighted each one’s geometric
and a wireframe-like mesh model characteristics.
highlighted each one’s
geometric characteristics.Sensors 2020, 20, 5628 13 of 26
Sensors 2020, 20, x FOR PEER REVIEW 13 of 26
Figure 12. AR comparisons of the booth, material, and composition of façades on the two streets in a
laboratory setting, using a mesh model and a wireframe-like
wireframe-like mesh
mesh model.
model. The top, middle, and two
bottom images represent four cases of comparisons.
8.2. Lukang
8.2. Lukang Site
Site
The second
The second set set of
of comparisons
comparisonswas wasmademadeininLukang
Lukangbybythe thesame
same researcher
researcherwith mesh
with meshmodels
modelson
real scenes under a homogeneous appearance. The former indoor and off-site
on real scenes under a homogeneous appearance. The former indoor and off-site comparison results comparison results were
verified
were in real in
verified sites with
real further
sites withemphases on coherence
further emphases on of neighborhood
coherence heritage relics,
of neighborhood continuity
heritage of
relics,
alley fabrics, reinstallation of facades in demolished sites, and the evolved form
continuity of alley fabrics, reinstallation of facades in demolished sites, and the evolved form of of pedestrian arcades.
The four locations
pedestrian arcades.inTheLukang
four were selected
locations according
in Lukang wereto condition, if it was (1)
selected according to empty; (2) demolished;
condition, if it was (1)
(3) in current form; or (4) located next to a historical artifact. The issues involved
empty; (2) demolished; (3) in current form; or (4) located next to a historical artifact. The consistency of style or
issues
continuity in terms of building components and construction methods.
involved consistency of style or continuity in terms of building components and construction An urban conversation was
restored toAn
methods. theurban
past conditions.
conversation was restored to the past conditions.
The façade on Yaolin
The façade on Yaolin Street was relocated
Street to Houche
was relocated Lane for Lane
to Houche a morefor straightforward side-by-side
a more straightforward
comparison (Figure 13, from top to bottom, left to right):
side-by-side comparison (Figure 13, from top to bottom, left to right):
•• The first comparison
comparison was was conducted
conducted on onbuildings
buildingswith
withaasimilar
similarscale.
scale.For
For example,
example, itsits eave
eave is
is located
located at at approximately
approximately thethe same
same height
height as that
as that of the
of the nextnext building.
building. The The AR simulation
AR simulation of a
of a coherent
coherent building
building component
component created
created a scenea of
scene of continuity
continuity which waswhich wasfound
rarely rarelyinfound
Houche in
Houche
Lane. Lane.
•• The second comparison was carried out next to the Lukang Gate, which is conspicuous among
the modernized constructions. The The inconsistency
inconsistency in in style
style was
was alleviated
alleviated with the traditional
façade installed
installednext
nexttoto
it in
it front of anofempty
in front site. AR
an empty was
site. ARused
wastoused
reconstruct a similar ahistorical
to reconstruct similar
street scene.
historical street scene.
•• thirdcomparison
The third comparisonwas was performed
performed to compare
to compare different
different building
building styles.styles. The side-by-side
The side-by-side layout
layout clearly revealed the difference in construction materials, opening partitions,
clearly revealed the difference in construction materials, opening partitions, and mass volume. and mass
• volume.
The fourth comparison was conducted at an empty site where a former red brick building was
• The fourth comparison
demolished. The site waswasleftconducted
unused and at unconstructed.
an empty site where a former
The same model red brick
was building
inserted was
in order
demolished.
to The siteofwas
restore continuity left unused
the original streetand unconstructed. The same model was inserted in
scene.
order to restore continuity of the original street scene.You can also read
