An ARCore Based User Centric Assistive Navigation System for Visually Impaired People - MDPI

 
An ARCore Based User Centric Assistive Navigation System for Visually Impaired People - MDPI
applied
           sciences
Article
An ARCore Based User Centric Assistive Navigation
System for Visually Impaired People
Xiaochen Zhang 1 , Xiaoyu Yao 1 , Yi Zhu 1,2 and Fei Hu 1, *
 1    Department of Industrial Design, Guangdong University of Technology, Guangzhou 510006, China;
      xzhang@gdut.edu.cn (X.Z.); yaoxiaoyu@mail2.gdut.edu.cn (X.Y.); zhuyi@gdut.edu.cn or
      zhuyi@gatech.edu (Y.Z.)
 2    School of Industrial Design, Georgia Institute of Technology, GA 30332, USA
 *    Correspondence: hufei@gdut.edu.cn
                                                                                                   
 Received: 17 February 2019; Accepted: 6 March 2019; Published: 9 March 2019                       

 Featured Application: The navigation system can be implemented in smartphones. With
 affordable haptic accessories, it helps the visually impaired people to navigate indoor without
 using GPS and wireless beacons. In the meantime, the advanced path planning in the system
 benefits the visually impaired navigation since it minimizes the possibility of collision in
 application. Moreover, the haptic interaction allows a human-centric real-time delivery of motion
 instruction which overcomes the conventional turn-by-turn waypoint finding instructions. Since
 the system prototype has been developed and tested, a commercialized application that helps
 visually impaired people in real life can be expected.

 Abstract: In this work, we propose an assistive navigation system for visually impaired people
 (ANSVIP) that takes advantage of ARCore to acquire robust computer vision-based localization.
 To complete the system, we propose adaptive artificial potential field (AAPF) path planning
 that considers both efficiency and safety. We also propose a dual-channel human–machine
 interaction mechanism, which delivers accurate and continuous directional micro-instruction via a
 haptic interface and macro-long-term planning and situational awareness via audio. Our system
 user-centrically incorporates haptic interfaces to provide fluent and continuous guidance superior
 to the conventional turn-by-turn audio-guiding method; moreover, the continuous guidance makes
 the path under complete control in avoiding obstacles and risky places. The system prototype
 is implemented with full functionality. Unit tests and simulations are conducted to evaluate the
 localization, path planning, and human–machine interactions, and the results show that the proposed
 solutions are superior to those of the present state-of-the-art solutions. Finally, integrated tests are
 carried out with low-vision and blind subjects to verify the proposed system.

 Keywords: assistive technology; ARCore; user centric design; navigation aids; haptic interaction;
 adaptive path planning; visual impairment; SLAM

1. Introduction
     According to statistics presented by the World Health Organization in October 2017, there are more
than 253 million visually impaired people worldwide. Compared to normally sighted people, they
are unable to access sufficient visual clues of the surroundings due to weakness in visual perception.
Consequently, visually impaired people face challenges in numerous aspects of daily life, including
when traveling, learning, entertaining, socializing, and working.
     Visually impaired people have a strong dependency on travel aids. Self-driving vehicles have
achieved SAE (Society of Automotive Engineers) Level 3 long back, which allows the vehicle to make

Appl. Sci. 2019, 9, 989; doi:10.3390/app9050989                                www.mdpi.com/journal/applsci
An ARCore Based User Centric Assistive Navigation System for Visually Impaired People - MDPI
Appl.
Appl. Sci.
      Sci.2019,
           2019,9,
                9,x989
                    FOR PEER REVIEW                                                                                          2 2ofof16
                                                                                                                                    15

decisions autonomously regarding the machine cognition. Autonomous robots and drones have also
been    dispatched
 decisions             for unmanned
              autonomously        regarding  tasks.   Obviously,
                                                 the machine          the advances
                                                                  cognition.             in robotics,
                                                                                Autonomous         robotscomputer
                                                                                                            and drones  vision,
                                                                                                                           have GIS
                                                                                                                                  also
(Geographic
 been dispatched  Information
                        for unmanned System),tasks.andObviously,
                                                         sensors allow        for integrated
                                                                       the advances                smart computer
                                                                                          in robotics,      systems to      perform
                                                                                                                        vision,   GIS
mapping,
 (Geographic  positioning,
                 Information  and    decision-making
                                  System),     and sensors   while
                                                               allowexecuting      in urban
                                                                       for integrated    smart areas.
                                                                                                  systems to perform mapping,
      Human and
 positioning,    beings    have the ability
                       decision-making            to interpret
                                               while    executingtheinsurrounding
                                                                        urban areas.environment using sensory organs.
Over Human
        90% of the    information
                  beings   have the transmitted           to the brain
                                        ability to interpret              is visual, and
                                                                  the surrounding            the brain processes
                                                                                         environment                   images
                                                                                                            using sensory        tens
                                                                                                                              organs.
of  thousands
 Over   90% of theof times   faster than
                      information             texts. This
                                       transmitted           explains
                                                         to the         why
                                                                 brain is       human
                                                                            visual,  andbeings
                                                                                           the brainareprocesses
                                                                                                         called visual   creatures;
                                                                                                                    images    tens of
when    traveling,
 thousands           visually
               of times   fasterimpaired
                                  than texts.  people    have to face
                                                  This explains    whydifficulties
                                                                          human beings  imposed     by their
                                                                                              are called       visual
                                                                                                           visual      impairment
                                                                                                                   creatures;  when
[1].
 traveling, visually impaired people have to face difficulties imposed by their visual impairment [1].
      Traditional
       Traditional assistive solutions
                                   solutions forfor visually
                                                     visually impaired
                                                                impaired people
                                                                             people include
                                                                                       include white
                                                                                                  white canes,
                                                                                                          canes, guide
                                                                                                                   guide dogs,
                                                                                                                          dogs, and
                                                                                                                                  and
volunteers.
 volunteers. However,
                 However, each each ofof these
                                         these solutions
                                                   solutions hashas its
                                                                    its own
                                                                         own restrictions.
                                                                                restrictions. They
                                                                                                 They either
                                                                                                         either work
                                                                                                                 work only
                                                                                                                        only under
                                                                                                                               under
certain
 certain situations,
          situations, with
                         with limited
                                limited capability,
                                           capability, or are expensive in terms of extra manpower.    manpower.
      Modern
       Modern assistive
                  assistivesolutions
                              solutionsfor  forvisually
                                                  visuallyimpaired
                                                             impairedpeople
                                                                          peopleborrow
                                                                                    borrowpowerpowerfromfrommobile
                                                                                                               mobilecomputing,
                                                                                                                        computing,
robotics,    andautonomous
 robotics, and     autonomoustechnology.
                                      technology.   TheyThey    are implemented
                                                           are implemented               in various
                                                                                  in various             forms
                                                                                                forms such        such asterminals,
                                                                                                              as mobile       mobile
terminals,    portable computers,
 portable computers,       wearable sensorwearable      sensor
                                                   stations,  andstations,   and indispensable
                                                                   indispensable       accessories.accessories.
                                                                                                       Most of these Most   of these
                                                                                                                        devices    use
devices
 computer  usevision
                computer     vision orto
                       or GIS/GPS         GIS/GPS
                                             understandto understand      the surroundings,
                                                              the surroundings,        acquire aacquire
                                                                                                    real-timea real-time
                                                                                                                 location,location,
                                                                                                                             and use
and   use turn-by-turn
 turn-by-turn     commands   commands
                                   to guideto  the guide
                                                     user.the   user. However,
                                                            However,      turn-by-turnturn-by-turn
                                                                                            commands    commands       are for
                                                                                                            are difficult   difficult
                                                                                                                                users
for  users to follow.
 to follow.
      In
       In this
           this work,
                 work,we wepropose
                             proposean    anassistive
                                              assistivenavigation
                                                           navigationsystem
                                                                          systemfor forvisually
                                                                                         visuallyimpaired
                                                                                                     impairedpeople
                                                                                                                  people(ANSVIP,
                                                                                                                           (ANSVIP,
see
 see Figure
      Figure 1.)1) using ARCore area learning; we introduce an adaptive             adaptive artificial
                                                                                                  artificial potential
                                                                                                             potential field
                                                                                                                          field path
                                                                                                                                 path
planning
 planning mechanism
             mechanism that   that generates
                                     generates smooth
                                                    smooth and and safe
                                                                    safe paths;
                                                                           paths; wewe design
                                                                                         design aa user-centric
                                                                                                      user-centric dual
                                                                                                                      dual channel
                                                                                                                             channel
interaction
 interaction that
                that uses
                     uses haptic
                            haptic sensors
                                      sensors to todeliver
                                                     deliverreal-time
                                                               real-timetraction
                                                                            tractioninformation
                                                                                       informationto    tothe
                                                                                                           theuser.
                                                                                                                user. To
                                                                                                                       To verify
                                                                                                                          verify the
                                                                                                                                   the
design
 design ofof the
              the system,
                   system, wewe havehave the
                                           the proposed
                                                proposed system
                                                              systemprototype
                                                                        prototypeimplemented
                                                                                       implementedwith    withfull
                                                                                                                 fullfunctionality,
                                                                                                                      functionality,
and
 and have
      have the
             the prototype
                  prototype tested
                                 tested with
                                          with blind
                                                  blind folded and blind subjects.

                              Figure1.
                              Figure   Thecomponents
                                     1.The componentsof
                                                     ofthe
                                                        theproposed
                                                           proposed ANSVIP
                                                                    ANSVIP system.
                                                                           system.

2. Related Works
2. Related Works
2.1. Assistive Navigation System Frameworks
2.1. Assistive Navigation System Frameworks
      Recent advances in sensor technology support the design and integration of portable assistive
navigation. advances
      Recent             in sensoran
               Katz [2] designed   technology  support
                                     assistive device   theaids
                                                      that  design  and integration ofand
                                                                in macro-navigation    portable assistive
                                                                                          micro-obstacle
navigation. Katz [2] designed an assistive device that aids in macro-navigation and micro-obstacle
An ARCore Based User Centric Assistive Navigation System for Visually Impaired People - MDPI
Appl. Sci. 2019, 9, 989                                                                           3 of 15

avoidance. The prototype was on a backpacked laptop employed with a stereo camera and audio
sensors. Zhang [3] proposed a hybrid-assistive system with a laptop with a head-mounted web
camera and a belt-mounted depth camera along with IMU (Inertial Measurement Unit). They used a
robotics-operating system to connect and manage the devices and ultimately help visually impaired
people when roaming indoors. Ahmetovic [4] used a smartphone as the carrier of the system, but a
considerable number of beacons had to be deployed prior to support the system. Furthermore, Bing [5]
used Project Tango Tablet with no extra sensor to implement their proposed system. The system
allowed the on-board depth sensor to support area learning, but the computational power burden
was heavy. Zhu [6] proposed and implemented the ASSIST system on a Project Tango smartphone.
However, due to the presence of more advanced Google ARCore, the Google Project Tango introduced
in 2014 has been depreciated [6] since 2017, and smartphones with the required capability are no
longer available. To the best of our knowledge, the proposed ANSVIP system is the first assistive
human–machine system using an ARCore-supported commercial smartphone.

2.2. Positioning and Tracking
     Most indoor positioning and tracking technologies were borrowed from autonomous robotics and
computer vision. Methods using direct sensing and dead reckoning [7] are no longer qualified options.
Yang [8] proposed a Bluetooth RSSI-based sensing framework to localize users in large public venues.
The particle filter was applied to localize the subject. Jiao [9] used an RGB-D camera to reconstruct a
semantic map to support indoor positioning. They used an artificial neural network on reflectivity to
improve the accuracy of 3D localization. Moreover, Xiao [10,11] and Zhang [3,12] used hybrid sensors
to carry out fast visual odometry and feature-based loop closure in localization, while Zhu [6] and
Bing [5,13] used area learning (a pattern recognition method) to bond subjects with areas of interest.

2.3. Path Planning
     As the most popular path planning in robotics, A* is also extensively used by assistive systems.
Xiao [10,11] and Zhang [3] used A* to connect areas of interest. Bing [5] applied greedy path planning
in the label-intensive semantic map. Meanwhile, Zhao [14] suggested a potential field as a potential
candidate for local planning. Paths in Reference [15,16] were planned on well-labeled maps using
global optimal methods such as Dijkstra [17] and its varieties. Most existing path-planning methods
generate sharp turn-by-turn paths connecting corner or feature anchors. These paths are good for
robots, but bring unbearable experiences to humans.

2.4. Human–Machine Interaction
     Most present systems use audio to deliver turn-by-turn directional instructions [5,11,18].
However, the human brain has its own understanding for positioning, direction, and velocity [19,20],
which is not robot-styled. Some recent works proposed using haptic interfaces in obstacle
avoidance [1,3,6,10,13,21–23]. However, due to the restriction of turn-by-turn path planning, the
haptic interaction is unlikely to be used in a continuous path-following interface in assistive systems.
Fernandes [7] used perceptual 3D audio as a solution; however, the learning was not easy and the
accuracy in real complex scenes needs to be improved. Ahmetovic [15] conducted a data-driven
analysis, which pointed out that turn-by-turn audio instructions have considerable drawbacks due to
latency in interaction and limited information per instruction. Guerreiro [24] stated that turn-by-turn
instructions may confuse visually impaired people’s navigation behaviors, and result in, for example,
deviating from the intended path. These behaviors lead to errors, confusion and longer recovery times
back to the right track. Such behaviors also emphasize that more effective real-time feedback interfaces
are necessary. Ahmetovic [15] studied the factors that cause rotation error in turn-by-turn navigation.
Rotation errors accompanying audio instructions significantly affect user experiences in navigation.
Rector [22] compared the accuracy of three different human guidance interfaces and provided insights
into the design of multimodal feedback mechanisms.
An ARCore Based User Centric Assistive Navigation System for Visually Impaired People - MDPI
Appl. Sci. 2019, 9, 989                                                                                                  4 of 15
Appl. Sci. 2019, 9, x FOR PEER REVIEW                                                                                    4 of 16

3. Design of ANSVIP
3. Design of ANSVIP

3.1. Information
3.1. Information Flow
                 Flow in
                      in System
                         System Design
                                Design
     Most tasks
     Most   tasks in
                   in real
                      real life
                           life are
                                are difficult
                                    difficult to
                                              to accomplish
                                                  accomplish using
                                                                 using aa single
                                                                          single sensor
                                                                                   sensor or
                                                                                           or functional
                                                                                               functional unit.
                                                                                                            unit. Instead,
                                                                                                                    Instead,
they  require  collaboration    (cooperation,    competition,    or  coordination)      from  multiple
they require collaboration (cooperation, competition, or coordination) from multiple functional units    functional    units
                                                                                                                          or
or sensing
sensing       agents
         agents         in intelligent
                 in intelligent  systems systems
                                           to maketothemakemost the   most final
                                                                 favorable    favorable    finalInsynthesis.
                                                                                    synthesis.                   In such a
                                                                                                    such a collaborative
collaborative
context,  each context,
                 functionaleachunit
                                 functional
                                     has its unit
                                               ownhas dutyits and
                                                              owncooperates
                                                                    duty and cooperates         via thechannel,
                                                                                   via the agreed        agreed channel,
                                                                                                                    thereby
thereby   maximizing     the  effectiveness   of shared    resources
maximizing the effectiveness of shared resources to achieve the goal.  to achieve    the  goal.
     Specifically, an
     Specifically,   an assistive
                         assistive navigation
                                    navigation system
                                                  system is is composed
                                                               composed of  of two
                                                                                two parts:
                                                                                      parts: the
                                                                                              the cognitive
                                                                                                  cognitive system
                                                                                                              system andand
the guidance    system.   The   cognitive  system    aims    to understand     the  world,   including
the guidance system. The cognitive system aims to understand the world, including the micro-scale         the micro-scale
surroundingsand
surroundings     andthethemacro-scale
                            macro-scalescene;
                                           scene;thethe  guidance
                                                      guidance       system
                                                                  system   aims aims   to properly
                                                                                  to properly         deliver
                                                                                                 deliver        the micro-
                                                                                                          the micro-scale
scale guidance
guidance          command,
           command,             the macro-scale
                         the macro-scale     plan, plan,
                                                    as wellas as
                                                               well as semantic
                                                                 semantic    scenescene    understanding,
                                                                                     understanding,      to thetouser.
                                                                                                                   the user.
                                                                                                                        The
The  collaboration    of the  two  allows  the  machine     to  understand     the scene,  and
collaboration of the two allows the machine to understand the scene, and then the user acquires  then   the user   acquires
understanding from
understanding     from the
                         the machine,
                              machine, asas shown
                                            shown in  in Figure
                                                         Figure2. 2.

      Figure 2. The
      Figure 2. The information
                    information flow
                                  flow in
                                        in the
                                            the assistive
                                                assistive system:
                                                          system: The
                                                                   The assistive
                                                                       assistive system
                                                                                 system core
                                                                                        core aims
                                                                                             aims to
                                                                                                  to understand
                                                                                                     understand
      the
      the world
          world and
                 and translate
                     translate the
                                the essential
                                    essential understanding
                                              understanding to tothe
                                                                  theuser.
                                                                      user.

      The
      The proposed
            proposed ANSVIP
                          ANSVIP uses uses anan ARCore-supported
                                                  ARCore-supported smartphone
                                                                           smartphone as   as the
                                                                                              the major
                                                                                                    major carrier
                                                                                                            carrier and
                                                                                                                      and uses
                                                                                                                            uses
ARCore-based
ARCore-based SLAM (Simultaneous Localization and Mapping) to track motion so as to create
                    SLAM     (Simultaneous        Localization     and   Mapping)     to  track   motion    so  as to  create aa
scene
scene understanding
        understanding along along with
                                    with mapping.
                                            mapping. The  Thehuman-scale
                                                                 human-scaleunderstanding
                                                                                 understandingofofmotion motion     and
                                                                                                                  and     space
                                                                                                                       space   is
is processed    to  produce     a short    and   safe  path   towards    the  goal.  The
processed to produce a short and safe path towards the goal. The corresponding micro-motioncorresponding       micro-motion
guidance
guidance is is delivered
               delivered to  to the
                                the user
                                     user using
                                            using haptic
                                                     haptic interaction,
                                                             interaction, while
                                                                            while the
                                                                                    the macro-path
                                                                                         macro-path cluesclues are
                                                                                                                 are delivered
                                                                                                                      delivered
using  audio   interaction.
using audio interaction.
      Based
      Based on onthetheinformation
                          information  flow   in aninassistive
                                           flow                  navigation
                                                       an assistive           system,system,
                                                                        navigation      we design we the  ANSVIP
                                                                                                       design    the structure
                                                                                                                       ANSVIP
as follows:
structure as follows:
      Firstly,
      Firstly,the
               thesystem
                     systemshould
                              shouldbe   befully
                                            fullyaware
                                                    awareof ofthe
                                                                the information
                                                                     information related
                                                                                   relatedto to the
                                                                                                 the user’s
                                                                                                     user’s location
                                                                                                              location during
                                                                                                                        during
navigation.
navigation. Unlike GPS-based solutions that are commonly used outdoors, our system has
               Unlike     GPS-based       solutions   that  are   commonly     used    outdoors,    our   system    has to
                                                                                                                         to use
                                                                                                                             use
computer
computer vision-based
             vision-based SLAMSLAM sincesince indoor
                                                indoor GPS
                                                         GPS signals
                                                                signals are
                                                                         are unreliable.
                                                                             unreliable. SLAM
                                                                                           SLAM is  is based
                                                                                                       based on on the
                                                                                                                   the Google
                                                                                                                        Google
ARCore,
ARCore, which
           which integrates
                     integratesthethevision
                                        visionandandinert
                                                      inertsensor
                                                             sensorhardware
                                                                       hardwareto tosupport
                                                                                      supportareaarealearning.
                                                                                                       learning.
      Secondly,
      Secondly, the system should be capable of conveying the abstracted systemic cognition
                   the  system    should     be  capable   of   conveying    the abstracted     systemic     cognition to to the
                                                                                                                             the
user.  Unlike    the  conventional       exclusive    audio    interaction,   we  propose    a
user. Unlike the conventional exclusive audio interaction, we propose a haptic-based cooperativehaptic-based      cooperative
mechanism.
mechanism. This This allows
                        allows usus to
                                     to replace
                                         replace thethe popular
                                                        popular turn-by-turn
                                                                     turn-by-turn guidance
                                                                                     guidance withwith aa more
                                                                                                           more continuous
                                                                                                                   continuous
motion   guidance.
motion guidance.
      The
      The working
           working logiclogic among
                               among the  the ANSVIP
                                               ANSVIP components
                                                          components is    is shown
                                                                              shown in  in Figure
                                                                                           Figure 3.3. Details
                                                                                                        Details ofof the
                                                                                                                     the major
                                                                                                                          major
components      are  discussed    in  the  following
components are discussed in the following subsections:  subsections:
An ARCore Based User Centric Assistive Navigation System for Visually Impaired People - MDPI
Appl.
Appl. Sci. 2019, 9,
      Sci. 2019, 9, 989                                                                                                    5 of
                                                                                                                             of 15
Appl. Sci. 2019, 9, xx FOR
                       FOR PEER
                           PEER REVIEW
                                REVIEW                                                                                   55 of  16
                                                                                                                                16

      Figure 3.
      Figure 3. The
             3. The working
                    working logic
                             logic among
                                   among the
                                   among     the ANSVIP
                                             the ANSVIP components:
                                                 ANSVIP    components: The
                                                           components:     The physical
                                                                               physical components
                                                                                         components and
                                                                                                    and soft
                                                                                                        soft
                                                                                                        soft
      components  are
      components are
      components      shown
                  are shown on
                      shown on  the
                            on the   left
                                the left  hand
                                     left hand side
                                          hand side and
                                               side and  right
                                                    and right  hand
                                                        right hand   side,
                                                               hand side,  respectively.
                                                                    side, respectively.
                                                                           respectively.

3.2.
3.2. Real-Time
3.2. Real-Time Area
     Real-Time Area Learning-Based
               Area Learning-Based Localization
                    Learning-Based Localization
                                   Localization
     The
      The system
      The  systemrelies
           system   relieson
                    relies  onexisting
                            on  existing
                                existingindoor scenario
                                          indoor
                                          indoor  scenario
                                                  scenarioCAD  maps,
                                                             CAD
                                                             CAD      which
                                                                   maps,
                                                                   maps, which
                                                                         whichare available as escape
                                                                                  are available
                                                                                  are available       mapsmaps
                                                                                                 as escape
                                                                                                 as escape   near
                                                                                                            maps
elevators
near       (as requested
near elevators
      elevators  (as         by fire
                 (as requested
                     requested    bydepartments).
                                  by fire            TheThe
                                     fire departments).
                                          departments).    mapmap
                                                          The   we we
                                                              map   useuse
                                                                    we  in this
                                                                       use      study
                                                                            in this
                                                                            in this    is presented
                                                                                    study
                                                                                    study            in in
                                                                                          is presented
                                                                                          is presented  Figure
                                                                                                        in      4.
                                                                                                           Figure
                                                                                                           Figure
We
4.  label
   We     the
       label   area
             the     of
                  area  interest
                        of        on
                           interest  the
                                    on    map
                                        the mapso as
                                                 so  to
                                                    as  allow
                                                       to allowthe
                                                                thesystem
                                                                    system to
                                                                            to understand
                                                                               understand   navigation
                                                                                             navigation
4. We label the area of interest on the map so as to allow the system to understand navigation requests  requests
                                                                                                         requests
and
and to
and  to plan
     to plan the
        plan  the path
              the  path accordingly.
                   path  accordingly.
                         accordingly.

                     Figure
                     Figure 4. The digital
                            4. The digital CAD
                                           CAD map
                                               map before (left) and
                                                   before (left)
                                                          (left) and after
                                                                 and after (right)
                                                                     after (right) being
                                                                           (right) being labeled.
                                                                                   being  labeled.
                                                                                         labeled.

      Google
       Google ARCore
                 ARCore is  is used
                               used to
                               used   to track
                                      to track the
                                         track  the pose
                                                the   pose of
                                                      pose   of the
                                                             of  the system
                                                                 the  system in
                                                                      system    in navigation.
                                                                                in navigation. The
                                                                                   navigation.     The sparse
                                                                                                   The   sparse features
                                                                                                         sparse    features are
                                                                                                                  features   are
                                                                                                                             are
collected
 collected  and
              andstored   in
                     stored  an
                              in area
                                  an  description
                                      area           dataset
                                             description       and
                                                             datasetsubsequently
                                                                         and         used
                                                                                subsequentlyin
 collected and stored in an area description dataset and subsequently used in re-localization. re-localization.
                                                                                                  used    in        Specifically,
                                                                                                              re-localization.
aSpecifically,
   normal-sighted
 Specifically,          person has toperson
                aa normal-sighted
                   normal-sighted        build the
                                       person   has sparse
                                                has   to build
                                                      to build map
                                                                 theof
                                                                 the     the indoor
                                                                      sparse
                                                                      sparse         ofscenario
                                                                                map of
                                                                                map               by running
                                                                                         the indoor
                                                                                         the  indoor   scenariothe
                                                                                                       scenario     by ARCore
                                                                                                                    by running
                                                                                                                       running
SLAM
 the     in
      ARCore advance.
                 SLAM    Then,
                          in     the
                             advance.assistive
                                         Then,  system
                                                 the      is capable
                                                       assistive        to
                                                                   system   re-localize
                                                                             is capable  itself
                                                                                          to    on  the
                                                                                              re-localize
 the ARCore SLAM in advance. Then, the assistive system is capable to re-localize itself on the          pre-built
                                                                                                            itself   map
                                                                                                                    on theafter
                                                                                                                            pre-
                                                                                                                            pre-
entering
 built map  the scenario.
              after  enteringBy  observing
                                the scenario. and
                                                By  recognizing
                                                    observing       the
                                                                   and    labeled
                                                                        recognizingtraceable
                                                                                        the     objects,
                                                                                             labeled      the
                                                                                                      traceable
 built map after entering the scenario. By observing and recognizing the labeled traceable objects, the        system    is
                                                                                                                    objects,able
                                                                                                                             the
to  re-localize
 system is
 system   is able itself
              able to    after  roaming
                    to re-localize
                       re-localize itself  in the
                                    itself after   system
                                           after roaming
                                                  roaming in map.
                                                              in the However,
                                                                  the system
                                                                      system map.  the  mapping
                                                                                 map. However,
                                                                                       However, the  between
                                                                                                     the mappingpoints
                                                                                                          mapping between in the
                                                                                                                       between
system
 points   feature-based
         in  the system     map    and those
                           feature-based    mapon  the
                                                   and  scenario
                                                         those   on CAD
                                                                     the    map   has
                                                                           scenario   to
                                                                                     CAD  be
 points in the system feature-based map and those on the scenario CAD map has to be obtained. obtained.
                                                                                             map   has  to be  obtained.
    We use
    We   use aa singular
                singular value
                         value decomposition
                               decomposition method
                                               method toto find
                                                           find the
                                                                the transportation
                                                                    transportation matrix A .. Two
                                                                                   matrix A    Two
groups  of corresponding  feature point sets are used for finding the  homogeneous  transformation
groups of corresponding feature point sets are used for finding the homogeneous transformation
matrix:
matrix:
An ARCore Based User Centric Assistive Navigation System for Visually Impaired People - MDPI
Appl. Sci. 2019, 9, 989                                                                                            6 of 15

     We use a singular value decomposition method to find the transportation matrix A. Two groups
of corresponding feature point sets are used for finding the homogeneous transformation matrix:
                                                                                                              
                                           "                         #        cos(θ )            − sin(θ ) t x
                                                 R 2×2      t 2×1
                                     A=                                   =  sin(θ )             cos(θ ) ty .       (1)
                                                                                                              
                                                   0           1
                                                                                 0                   0     1

     Let ln = [ xn , yn ] T denote the point sets in the feature map, and let pn = [in , jn ] T denote the
corresponding point set on the scenario CAD map. We use the least squares method to find the rotation
R and translation t as follows:
                                                                               N
                                                ( R, t) ← arg min ∑ k R × pi + t − li k2 .                            (2)
                                                                    R,t       i =1

                               N                N
                          1                1
      Denote l =          N   ∑ li , p =   N   ∑ pi , Equation (2) can be written as
                              i =1             i =1

                                                                               N                    2
                                                      R ← arg min ∑ R × pi − l i                        .             (3)
                                                                    R         i =1

      For M x = b,                                                                          
                                                                        x1      − y1   1 0
                                                           
                                                                       y1       x1    0 1  
                                                           
                                                         M=             ..       ..   .. .. 
                                                                          .        .    . . ,                        (4)
                                                                                             
                                                           
                                                                               −yn
                                                                                            
                                                                       xn             1 0 
                                                                        yn      xn     0 1

                                                         x = [cos(θ ), sin(θ ), t x , ty ] T ,                        (5)

                                                             b = [i1 , j1 , ..., in , jn ] T .                        (6)

      Using SVD to decompose M, we find

                                                        M N ×4 = U N × N S N ×4 V T 4×4 ,                             (7)

where U denotes the feature matrix of MM T , S denotes the diagonal matrix with eigenvalue δi , V is
the eigenvector matrix of M T M, and we have A in (1) calculated by

                                                                                                  −1
                                                      x = (Vdiag(δ1−1 , ..., δ4−1 )U T )               b.             (8)

3.3. Area Learning in ANSVIP
      ARCore is an augmented reality framework for smartphones with Android operating system.
It is an advanced substitute of the deprecated Project Tango. Without the extra depth sensor, an
ARCore-powered cell phone is able to track its pose and build a map of the surroundings in real time.
Besides, ARCore also enhances the area of interest detection by estimating the average illumination
intensity, which helps area segmentation during semantic mapping.
      The smartphone is a remarkable feat of engineering. It integrates a great number of sensors like a
gyroscope, camera, and GPS into a small slab. Specifically, in our work, a HUAWEI P20 with Kirin 970
CPU, Gravity sensor, Ambient light sensor, Proximity sensor, Gyroscope Compass is used.
An ARCore Based User Centric Assistive Navigation System for Visually Impaired People - MDPI
Appl. Sci. 2019, 9, 989                                                                                          7 of 15

3.4. Adaptive Artificial Potential Field-Based Path Planning
     In indoor navigation for the visually impaired, the path planning has to consider both efficiency
and safety. Specifically, our path planning considers the issues that follow.
     1. The path should be planned to be away from obstacles and risks: Where the conventional robot
path planning prefers the shortest path, the assistive system has more in-depth requirements. For
visually impaired users, the path should be away from obstacles and risks such as walls, pillars, and
uneven steps, which may cause falls [25].
     2. The path and guidance shall be updated in real time: Unlike autonomous robot systems, the
assistive system cannot expect visually impaired users to proceed along the planned path accordingly.
When the user deviates from the planned path, there should be a corresponding real-time path
evolution instead of asking the user to return to the planned track.
     3. The mechanism shall be flexible to scale up with new elements: The path-planning algorithm
should be able to easily expand with new elements, such as dynamic obstacle avoidance, functional
unit integration, step warning, and extreme case re-planning.
     4. The path shall be planned in a human-friendly manner: Unlike robots, visually impaired users
are unable to grasp precise turning angles, and thus, it is difficult for them to follow conventional
turn-by-turn paths [15]. Qualitative direction guidance is more suitable. Users prefer continuous
guidance in navigation and a generally smooth plan.
     The artificial potential field path planning is a suitable candidate for the above issues and
challenges, since it has the characteristics of a simple structure, strong practicability, ease of
implementation, and flexibility regarding expansion [14,17].
     Therefore, we propose an adaptive artificial potential field path-planning mechanism for
path generation.
     Specifically, the target (goal) is considered an attractive potential, while walls are repulsive. The
potential fields are the combination of first-order attractive and repulsive potentials:

                                                         U = Uatt + Urep ,                                          (9)

                                             Uatt ( Xcurrent ) = kρ( Xcurrent , Xtarget ),                        (10)
                                                                            
                                                             1             1
                                              η
                                                     ρ( Xcurrent ,Xobs )
                                                                         − ρ0   if ρ( Xcurrent , Xobs ) ≤ ρ0
                          Urep ( Xcurrent ) =                                                                ,    (11)
                                              0                                if ρ( Xcurrent , Xobs ) > ρ0

where η denotes the repulsive factor, ρ denotes a distance function, and ρ0 denotes the effective radius.
A path can be generated along with the potential gradients.
     However, local minimums may block the path or cause redundant travel costs. Thus, we refer to
the local minimum immune A* algorithm path length to control ρ0 and solve this problem.
     ρ0 = ρ0 − ∆ρ until CAAPF > λCA∗ , where λ denotes the control factor, and CAAPF and CA∗
denote the path length of adaptive artificial potential field (AAPF) and A* from the current position,
respectively. A sliding window is used to smooth the path to support and enhance the experience of
motion guidance in human–machine interaction (Figure 5):

                                            1
                               X (i ) =          ( X (i + N ) + X (i + N − 1) + ... + X (i − N )).                (12)
                                          2N + 1

    A case of a smoothed path is shown in Figure 5. Since the plan is discrete, the path (red dotted) is
planned in taxicab style before smoothing. The sliding window described in equation (12) updates
each point on the path by averaging its position with the nearest 2N points’ positions on the path.
Consequently, the path (dark curve) is smoothed after the process.
An ARCore Based User Centric Assistive Navigation System for Visually Impaired People - MDPI
Appl. Sci. 2019, 9, 989                                                                                8 of 15
Appl. Sci. 2019, 9, x FOR PEER REVIEW                                                                           8 of 16

     Figure 5. A5.case
         Figure        of aofsmoothed
                   A case     a smoothedpath
                                         pathby
                                              bysliding  window:before
                                                 sliding window: beforesmoothing
                                                                        smoothing   (red
                                                                                 (red    dotted)
                                                                                      dotted)     versus
                                                                                              versus afterafter
     smoothing  (dark
         smoothing      curve).
                     (dark curve).

     3.5. Dual-Channel Human–Machine Interaction
      A case of a smoothed path is shown in Figure 5. Since the plan is discrete, the path (red dotted)
is plannedThein information
                 taxicab styletransfer
                                beforebetween     the user
                                         smoothing.    The and  system
                                                            sliding      relies on
                                                                     window        human–machine
                                                                                described  in equation interactions
                                                                                                           (12) updates
     (HMIs).  The   HMI   in an assistive navigation  system  has certain unique  characters.
each point on the path by averaging its position with the nearest 2N points’ positions on the First, the HMI   doespath.
     not rely on   visual cognition. Second,  the HMI   is highly
Consequently, the path (dark curve) is smoothed after the process.task-oriented. Third, the different  information
     has distinct delivery requirements in situations of urgency or depending on high accuracy. The most
     popular audio interaction for assistive navigation systems [16,26–28] suffers from the following aspects:
3.5. Dual-Channel Human–Machine Interaction
          Instruction delay: The instruction delivery is not instantaneous, and the latency becomes a
     The information
     bottleneck           transfer
                 when dealing       between
                                  with   urgent the  user and
                                                interaction     systemItrelies
                                                             requests.           onwhen
                                                                           is vital  human–machine          interactions
                                                                                           dealing with urgency
     in navigation.
(HMIs). The HMI in an assistive navigation system has certain unique characters. First, the HMI does
          Limited
not rely on visualinformation:    The amount
                    cognition. Second,          of information
                                            the HMI   is highlyper  message/second
                                                                 task-oriented.        is very
                                                                                   Third,        limited and
                                                                                          the different        tends
                                                                                                            information
     to cause ambiguity,   which  makes   accomplishing   tasks with  multiple  semantics   difficult.
has distinct delivery requirements in situations of urgency or depending on high accuracy. The most
          Vulnerable to interference: The user may not be able to access multiple instructions simultaneously,
popular audio interaction for assistive navigation systems [16,26–28] suffers from the following
     and environmental sounds may cause interference.
aspects:
          Result-oriented instructions: The conventional graphical interaction provides many individual
     Instruction   delay: The instruction delivery is not instantaneous, and the latency becomes a
     small tasks to users, allowing them to choose among different combinations to achieve their
bottleneck
     goals. when
              Audiodealing     with are
                      instructions   urgent   interaction
                                          usually           requests.
                                                   goal-driven         It is vital whenand
                                                                 and result-oriented,      dealing
                                                                                               they with     urgency
                                                                                                       are weak   in in
navigation.
     procedure-oriented interaction tasks.
     Limited
          Thus,information:
                  we design aThe    amount
                                 hybrid        of information
                                           haptic                per message/second
                                                  interaction mechanism      as the major is very   limited
                                                                                             interface        and tends
                                                                                                          to deliver
to cause  ambiguity,
     navigation        which especially
                 instructions,  makes accomplishing
                                            micro-motion tasks   with multiple
                                                           instructions. Audio issemantics      difficult.
                                                                                    used to deliver   less-sensitive
     macro-informative    messages.
     Vulnerable to interference:           The user may not be able to access multiple instructions
          After a path  to the target is determined,
simultaneously, and environmental sounds may          motion
                                                          causeguidance  is generated as shown in Figure 6.
                                                                  interference.
     Result-oriented instructions: The conventional graphical interaction provides many individual
small tasks to users, allowing them to choose among different combinations to achieve their goals.
Audio instructions are usually goal-driven and result-oriented, and they are weak in procedure-
oriented interaction tasks.
     Thus, we design a hybrid haptic interaction mechanism as the major interface to deliver
navigation instructions, especially micro-motion instructions. Audio is used to deliver less-sensitive
macro-informative messages.
     After a path to the target is determined, motion guidance is generated as shown in Figure 6.
Appl.
Appl. Sci. 2019, 9, 989                                                                                      999 of
                                                                                                                 of 15
Appl. Sci.
      Sci. 2019,
           2019, 9,
                  9, xx FOR
                        FOR PEER
                            PEER REVIEW
                                 REVIEW                                                                          of 16
                                                                                                                     16

       Figure 6.
       Figure    The motion
                     motion guidance is
                                     is generated intersecting  the planned  path and the awareness  cycle.
       Figure 6.
              6. The
                 The motion guidance
                            guidance is generated
                                        generated intersecting
                                                   intersecting the
                                                                 the planned
                                                                     planned path
                                                                             path and
                                                                                  and the
                                                                                       the awareness
                                                                                           awareness cycle.
                                                                                                      cycle.

       To
       To deliver the  motion guidance   in real  time via   haptic interaction,
                                                                     interaction, there is
                                                                                        is a numerical solution.
                                                                                                        solution.
       To deliver
           deliver the
                    the motion
                        motion guidance
                               guidance inin real
                                              real time
                                                    time via
                                                         via haptic
                                                             haptic interaction, there
                                                                                  there is aa numerical
                                                                                              numerical solution.
In
In this  work,  we   design haptic gloves  as  shown   in Figure  7.
 In this
     this work,
          work, we
                we design
                     design haptic
                            haptic gloves
                                   gloves asas shown
                                                shown inin Figure
                                                           Figure 7.
                                                                   7.

                                          Figure
                                          Figure 7. The
                                                 7. The
                                          Figure7.      design
                                                    The design of
                                                        design of haptic
                                                               of haptic gloves.
                                                                  hapticgloves.
                                                                         gloves.

      The
      Theleft
          leftglove
               gloveguides
                     guidesthe
                            themotion,
                                motion,andandthe
                                              theright
                                                  rightglove
                                                        glovewarns
                                                              warnsof  obstacles. Using
                                                                    of obstacles. Using the
                                                                                         the middle
                                                                                             middle finger
                                                                                                     finger
as
 as the head front of the subject, the motion directional guidance can be instantaneously delivered
   the head  front of the subject, the motion  directional guidance can  be instantaneously   delivered to
                                                                                                         to
the
 theuser
     useras
          assoon
             soonasasthe motion
                      themotion  plan
                          motionplan   is
                                  planis  made.
                                        ismade.
                                          made.

4.
4. System
   System Prototyping
4. System Prototyping and
          Prototyping and Evaluation
                      and Evaluation
                          Evaluation
4.1.
4.1.  System Prototyping
 4.1. System
      System Prototyping
              Prototyping
       We
       We  use   the HUAWEI        P20 as   the ARCore-supported        smartphone, use       Arduino sensors     to
       We use
            use the
                  the HUAWEI
                       HUAWEI P20   P20 as
                                         as the
                                             the ARCore-supported
                                                 ARCore-supported smartphone,
                                                                         smartphone, useuse Arduino
                                                                                              Arduino sensors
                                                                                                         sensors toto
implement
implement     the haptic    interactive glove,   and use  the BAIDU      open API    for audio   recognition. The
 implement the the haptic
                     haptic interactive
                             interactive glove,
                                          glove, and
                                                 and use
                                                      use the
                                                           the BAIDU
                                                                BAIDU openopen API
                                                                                 API for
                                                                                      for audio
                                                                                          audio recognition.
                                                                                                  recognition. The
                                                                                                                The
application
application   is developed     in Unity3D.  Roberto Lopez   Mendez ARCore        SLAM is applied    as the the
                                                                                                            base for
 application isis developed
                   developed in in Unity3D.
                                   Unity3D. Roberto
                                              Roberto Lopez
                                                       Lopez Mendez
                                                               Mendez ARCore
                                                                          ARCore SLAM
                                                                                    SLAM is is applied
                                                                                               applied as
                                                                                                        as the base
                                                                                                               base
visual
for     odometry      and area   learning.  Bluetooth is used   to connect   the  smartphone    and  the accessory.
 for visual
      visual odometry
              odometry and  and area
                                  area learning.
                                        learning. Bluetooth
                                                   Bluetooth isis used
                                                                   used toto connect
                                                                              connect the
                                                                                       the smartphone
                                                                                             smartphone and and the
                                                                                                                 the
The   ready-to-work
accessory.              human–machine prototype isprototype
                                                         shown in Figure     8. Figure 8.
 accessory. The
             The ready-to-work
                    ready-to-work human–machine
                                     human–machine prototype is     is shown
                                                                       shown in in Figure 8.
Appl. Sci. 2019, 9, 989                                                                                          10 of 15
Appl. Sci. 2019, 9, x FOR PEER REVIEW                                                                           10 of 16
Appl. Sci. 2019, 9, x FOR PEER REVIEW                                                                           10 of 16

                      Figure 8. The implemented ANSVIP prototype with full functionality.
                      Figure 8. The implemented ANSVIP prototype with full functionality.
                                                                           functionality.

4.2.
4.2. Localization
     Localization
4.2. Localization
      To
      To  validate the   localization accuracy   and reliability,  we compare
                                                                       compare the localization      of area  learning
      To validate
          validate the
                    the localization
                         localization accuracy
                                      accuracy and
                                                 and reliability,
                                                      reliability, we
                                                                   we compare the the localization
                                                                                      localization ofof area
                                                                                                        area learning
                                                                                                              learning
with
with   visual odometry     in an indoor  test. Two  subjects  wearing    the system   are asked   to walk   five times
with visual
       visual odometry
              odometry in  in an
                              an indoor
                                 indoor test.  Two subjects
                                         test. Two  subjects wearing
                                                              wearing thethe system
                                                                             system are
                                                                                      are asked
                                                                                          asked toto walk
                                                                                                      walk five
                                                                                                            five times
                                                                                                                 times
along
along   a path in
               in the
                   the corridor,
                       corridor, one  subject with
                                                with area
                                                     area learning
                                                           learning and
                                                                      and the
                                                                           the other
                                                                                other with
                                                                                       with visual
                                                                                            visual odometry
                                                                                                     odometry(VO).
along aa path
          path in the  corridor,  one
                                  one subject
                                       subject with  area  learning   and  the  other with  visual   odometry     (VO).
                                                                                                                 (VO).
The
The  results  on Figure    9 are consistent  with  our expectations:    The  VO   trials suffer from   accumulative
The results
      results on
              on Figure
                  Figure 99 are
                             are consistent
                                 consistent with
                                             with our
                                                   our expectations:
                                                        expectations: The
                                                                        The VOVO trials
                                                                                  trials suffer
                                                                                         suffer from
                                                                                                from accumulative
                                                                                                        accumulative
errors,
errors,  which  cause   localization drifts; meanwhile,    in the area  learning   method,   there  are certain  drifts
errors, which cause localization drifts; meanwhile, in the area learning method, there are
         which  cause   localization drifts; meanwhile,    in the  area learning   method,   there  are certain
                                                                                                         certain drifts
                                                                                                                 drifts
in
in passing corners    but the
                            the system is
                                        is able to
                                                to swiftly correct
                                                            correct the drift
                                                                          drift by recognizing
                                                                                   recognizing learned areas.
                                                                                                            areas.
in passing
   passing corners
             corners but
                       but the system
                                system is able
                                           able to swiftly
                                                   swiftly correct the
                                                                     the drift by
                                                                                by recognizinglearned
                                                                                                  learned areas.

                                                                             Ground Truth
                                                                             Ground
                                                                             VO 1 Truth
                                                                             VO 21
                                                                             VO
                                                                             VO 32
                                                                             VO
                                                                             VO 43
                                                                             VO
                                                                             VO 54
                                                                             VO
                                                                             VO 5Learning
                                                                             Area
                                                                             Area Learning

                              Figure 9. The ground truth and trajectory of test trails.
                              Figure 9. The ground truth and trajectory of test trails.
                              Figure 9. The ground truth and trajectory of test trails.
4.3. Path Planning
4.3. Path Planning
4.3. Path  Planning
      Simulation    comparisons on four different path planning mechanisms are conducted: the adaptive
      Simulation
artificial potential comparisons
                        field (AAPF), on thefour differentartificial
                                                               path planning
                                                                       potentialmechanisms
                                                                                   field withoutare    conducted:     the
      Simulation     comparisons       on fouradaptive
                                                   different path planning        mechanisms        a sliding
                                                                                                  are           window
                                                                                                       conducted:     the
adaptive
(AAPF/S),   artificial  potential
              the artificial       field
                             potential     (AAPF),    the   adaptive   artificial potential   field  without   a sliding
adaptive artificial     potential   fieldfield  without
                                           (AAPF),     thea adaptive
                                                            repulsive artificial
                                                                       force andpotential
                                                                                   sliding window     (AAPF/RS),
                                                                                              field without          and
                                                                                                               a sliding
window
the        (AAPF/S),
     A* path   planning.the    artificial potential field without a repulsive force and sliding window
window      (AAPF/S),     the artificial potential field without a repulsive force and sliding window
(AAPF/RS),     and
      On the map,   the  A*
                      weA*   path
                          setpath  planning.location as the starting position. Then, 100 random destinations
                              the elevator’s
(AAPF/RS),     and the             planning.
      On  the
are generated  map,  we
                  outside set the elevator’s
                                       with alocation     as 25
                                                             the startingcentered
                                                                          position. Then,    100 random    destinations
      On the map,     we setathe
                               circle
                                  elevator’s     radius of
                                                location  as themeters
                                                                 starting position.atThen,
                                                                                        the starting  point,
                                                                                             100 random       as shown
                                                                                                           destinations
are
in   generated
   Figure    10.  outside
                  We   use  a circle
                            the four  with   a radius
                                       candidate        of 25 meters centered
                                                    path-planning      mechanisms at the
                                                                                       to starting
                                                                                           generate point,
                                                                                                     the   as shown
                                                                                                          paths        in
                                                                                                                 for the
are generated outside a circle with a radius of 25 meters centered at the starting point, as shown                     in
Figure  10.
start–target We   use
               pairs.  the  four candidate     path-planning     mechanisms      to generate   the paths  for the  start–
Figure 10. We     use the four candidate path-planning mechanisms to generate the paths for the start–
target pairs.
target pairs.
Appl. Sci. 2019, 9, 989                                                                                              11 of 15
Appl. Sci. 2019, 9, x FOR PEER REVIEW                                                                               11 of 16
Appl. Sci. 2019, 9, x FOR PEER REVIEW                                                                               11 of 16

       Figure 10. The 100 generated destinations (star) and the starting position (pentagram) on the map.
       Figure 10. The 100 generated destinations (star) and the starting position (pentagram) on the map.

      In Figure 11,
      In            11, we compare
                            compare the pathpath lengths generated
                                                              generated by the the four mechanisms.
                                                                                           mechanisms. Obviously,
                                                                                                          Obviously, the the
      In Figure
           Figure 11, wewe compare the the path lengths
                                                   lengths generated by    by the four
                                                                                    four mechanisms. Obviously,          the
path
path   lengths     of AAPF   are  always    lower    than   those   of  AAPF/S    and    AAPF/RS     because   the   sliding
path lengths
       lengths of of AAPF
                      AAPF are
                             are always
                                  always lower
                                            lower than
                                                     thanthose
                                                            thoseofofAAPF/S
                                                                        AAPF/S andand AAPF/RS        because the
                                                                                         AAPF/RS because       the sliding
                                                                                                                     sliding
window curves
window       curvesthethesharp
                          sharpturns
                                 turnsononthe
                                            thepath
                                                  pathinto
                                                        intofilleted
                                                              filleted  turns.  Therefore,    the path  length   is shorter,
window curves the sharp turns on the path into filleted turns. Therefore, the path length shorter,
                                                                      turns.  Therefore,   the  path length  is           as
                                                                                                                 is shorter,
as  expected.
expected.        The
              TheThe
                   pathpath lengths
                         lengths      of
                                  of A*  A*  are   always    the lowest   and   are  the  best among    the four.   Because
as expected.           path lengths  of are  always
                                         A* are        the lowest
                                                  always             and and
                                                            the lowest    are the
                                                                               arebest
                                                                                    the among     the four.
                                                                                          best among        Because
                                                                                                       the four.        A* is
                                                                                                                   Because
A* is using
using   a       a greedy
           greedy          mechanism
                    mechanism    to      to generate
                                    generate    the     theitpath,
                                                     path,    is    it is guaranteed
                                                                 guaranteed    to produceto produce
                                                                                             the      the shortest
                                                                                                  shortest length     length
                                                                                                                    when    a
A* is using a greedy mechanism to generate the path, it is guaranteed to produce the shortest length
when aview
global      global
                is  view is accessible.
                   accessible. However,   However,
                                            it is noted itthat
                                                           is noted
                                                                the    thatlength
                                                                    path     the path   length differences
                                                                                   differences   between    A*between
                                                                                                                and   AAPFA*
when a global view is accessible. However, it is noted that the path length differences between A*
and
is   AAPF
   very        is very limited.
          limited.
and  AAPF      is very limited.

                     120
                     120
                                      AAPF
                                      AAPF
                                      AAPF/S
                     110              AAPF/S
                     110              AAPF/RS
                                      AAPF/RS
                                      A*
                                      A*
                     100
                     100

                       90
                       90

                       80
                       80

                       70
                       70

                       60
                       60

                       50
                       50

                       40
                       40 0      10     20      30      40      50     60      70      80      90     100
                          0      10     20      30      40      50     60      70      80      90     100
                                                           Trial
                                                           Trial
                                 Figure 11. Simulation
                                 Figure 11. Simulation results
                                                       results on
                                                               on path
                                                                  path planning
                                                                       planning cost.
                                                                                cost.
                                 Figure 11. Simulation results on path planning cost.
     In Figure 12, we collect the discrete distances from the path to obstacles along the paths. It is
     In Figure 12, we collect the discrete distances from the path to obstacles along the paths. It is
shownIn that
         Figure 12, and
             AAPF   we collect
                        AAPF/S theproperly
                                   discretedeal
                                            distances from thetopath
                                                with distance        to obstacles
                                                                 obstacles,        along
                                                                            which is     the paths.
                                                                                     consistent with Itour
                                                                                                        is
shown that AAPF and AAPF/S properly deal with distance to obstacles, which is consistent with our
shown   that AAPF   and AAPF/S   properly  deal with distance to obstacles, which is consistent
design: The repulsive forces of obstacles keep the path away. AAPF/RS and A* do not have such   with our
design: The repulsive forces of obstacles keep the path away. AAPF/RS and A* do not have such
design: The repulsive forces of obstacles keep the path away. AAPF/RS and A* do not have such
Appl. Sci. 2019,
Appl. Sci. 2019, 9,
                 9, 989
                    x FOR PEER REVIEW                                                                           12 of
                                                                                                               12  of 16
                                                                                                                      15
Appl. Sci. 2019, 9, x FOR PEER REVIEW                                                                          12 of 16

repulsive  forces,
           forces, and
repulsive forces,      thus,
                       thus, aaa good
                   and thus,     good portion
                                 good portion of
                                      portion of the
                                              of the paths
                                                 the paths is
                                                     paths is  close
                                                            is close to
                                                               close to  obstacles.
                                                                         obstacles. This
                                                                      to obstacles. This is
                                                                                         is not desired
                                                                                            not desired in
                                                                                                desired in
                                                                                                        in
repulsive          and                                                              This is not
assistive navigation
assistive navigation [6,11].
          navigation [6,11].
                     [6,11].
assistive
                           5000
                           5000

                             0
                             0
                           5000
                           5000

                             0
                             0
                           5000
                           5000

                             0
                             0

                           2000
                           2000
                           1000
                           1000
                              0
                              0   0   1   2    3    4    5   6    7    8    9   10   11   12
                                  0   1   2    3    4    5   6    7    8    9   10   11   12

                              Figure
                              Figure 12.
                                     12. Simulation
                                         Simulation results
                                                    results on
                                                            on distances
                                                               distances to
                                                                         to obstacles.
                                                                            obstacles.

     Although the path lengths of A* are a bit shorter than those of AAPF, considering the fact that
     Although the path lengths of A* are a bit shorter than those of AAPF, considering the fact that
subjects in navigation are prone totoexperience risks and  panics if risky  places areare
                                                                                       close to the paths, the
subjects in
          in navigation
             navigation are
                        are prone
                            prone to experience
                                       experience risks
                                                   risks and
                                                         and panics
                                                             panics ifif risky
                                                                         risky places
                                                                               places are close
                                                                                          close to
                                                                                                 to the
                                                                                                    the paths,
                                                                                                        paths,
AAPF
the    outperforms
    AAPF             A*
            outperforms in
                         A*keeping
                            in       the
                               keeping   path
                                         the  safer.
                                             path safer.
the AAPF outperforms A* in keeping the path safer.
4.4.
4.4. Haptic Guidance
            Guidance
4.4. Haptic
     Haptic Guidance
     To
     To  verify the  directional guidance     of the  haptic device,   we carry  out unit   tests of  the haptic
      To verify
         verify the
                 the directional
                      directional guidance
                                   guidance of of the
                                                  the haptic
                                                       haptic device,
                                                              device, we
                                                                       we carry
                                                                          carry out
                                                                                  out unit
                                                                                       unit tests
                                                                                             tests of
                                                                                                   of the
                                                                                                      the haptic
                                                                                                           haptic
guidance
guidance   glove. The   guidance  glove  on  the left hand  and  the Arduino  joystick  to be  controlled  on the
                                                                                                              the
guidance glove. The guidance glove on the left hand and the Arduino joystick to be controlled on
           glove. The   guidance  glove  on  the left hand  and  the Arduino  joystick  to be  controlled  on the
right
right hand are
            are shown in    Figure 13.  A series
                                          series of programmed
                                                     programmed guidance commands
                                                                              commands are stored       and sent
right hand
      hand are shown
                 shown inin Figure
                             Figure 13.
                                    13. A
                                        A series of
                                                  of programmed guidance
                                                                    guidance commands are   are stored
                                                                                                 stored and
                                                                                                        and sent
                                                                                                             sent
to
to the glove  so as to let the subject feel the guidance.   A  blind-folded subject  is told  to use the joystick
to the glove so as to let the subject feel the guidance. A blind-folded subject is told to use the joystick
   the glove  so as to let the subject feel the guidance.   A blind-folded  subject is  told  to use the joystick
to
to depict the  directional instruction  received. The   joystick behaviors  are recorded    every half  second.
to depict
   depict the
          the directional
               directional instruction
                            instruction received.
                                         received. The
                                                    The joystick
                                                         joystick behaviors
                                                                  behaviors are
                                                                             are recorded
                                                                                 recorded every
                                                                                             every half
                                                                                                    half second.
                                                                                                         second.

                   Figure 13. (Left)
                   Figure 13. (Left) Prototype
                                     Prototype of
                                               of haptic glove. (Right)
                                                  haptic glove. (Right) Joystick
                                                                        Joystick for
                                                                                 for test
                                                                                     test purpose.
                                                                                          purpose.

      In Figure 14, the input guidance commands are compared with          with the joystick
                                                                                      joystick records. Obviously,
                                                                                                          Obviously,
      In Figure 14, the input guidance commands are compared with the            the joystick records.
                                                                                               records. Obviously,
there is a latency between the input and records. The latency is caused by three factors: the cognitive
there is a latency between the input and records. The latency is caused by three factors: the cognitive
delay of human haptic sensibility, the delay from understanding                 the guidance to controlling the
delay of human haptic sensibility, the delay from understanding
                                                             understanding the guidance to controlling the
joystick, and the delay between joystick action and recording. The         The average
                                                                                 average delay is is less than
                                                                                                          than 0.4 s, s,
joystick, and the delay between joystick action and recording. The average delay           delay is less
                                                                                                     less than 0.4
                                                                                                                 0.4 s,
which
which   is acceptable   in  most cases. Note that the delays   in later trials are much   less than  those  in  earlier
which is acceptable
           acceptable in in most
                            most cases.
                                 cases. Note that the delays in later trials are much less than those in earlier
trials. One of the reasons for this is that the subject is getting familiar with the haptic          interaction. In
trials. One of the reasons for this is that the subject is getting familiar with the haptic  haptic interaction.
                                                                                                      interaction. In
other words, the subject is capable of efficiently and quickly converting the data perceived by the
other words, the subject is capable of efficiently and quickly converting the data perceived by the
haptic interaction   into their  own perception   after a few  attempts. Thus,     a cooperative cognition     is built
haptic interaction
         interaction into
                       into their
                             their own
                                   own perception
                                        perception after
                                                     after aa few
                                                              few attempts.
                                                                   attempts. Thus,
                                                                                Thus, aa cooperative
                                                                                         cooperative cognition
                                                                                                        cognition is is
between
built      the assistive   system, haptichaptic
                                          interaction, and human       perceptions.
built between
       between the
                 the assistive
                      assistive system,
                                 system, haptic interaction,
                                                interaction, and
                                                               and human
                                                                    human perceptions.
                                                                              perceptions.
Appl. Sci. 2019, 9, 989                                                                                                   13 of 15
       Appl. Sci. 2019, 9, x FOR PEER REVIEW                                                                        13 of 16

                                        Haptic Glove Guidance
                                        Joystick Trail 1
                                        Joystick Trail 2
                                        Joystick Trail 3
                                        Joystick Trail 4

                                        Haptic Glove Guidance
                                        Rocker Trail 1
                                        Rocker Trail 2
                                        Rocker Trail 3
                                        Rocker Trail 4

                                Figure     Haptic
                                       14.14.
                                   Figure     Hapticglove
                                                     glove guidance versusjoystick
                                                           guidance versus  joystick records.
                                                                                   records.

4.5. Integration  TestTest
      4.5. Integration
      To verify   the prototype
           To verify    the prototype system,   we conduct
                                          system,    we conduct target-oriented    navigation
                                                                     target-oriented  navigationtests
                                                                                                   testswith
                                                                                                         withthree
                                                                                                                threelow-vision
                                                                                                                        low-
      vision
subjects   andsubjects  and one
                 one blind         blind subject.
                                 subject.           To evaluate
                                              To evaluate      thethe   human–machine interaction
                                                                     human–machine         interactionthat   occurs
                                                                                                          that        in our
                                                                                                                 occurs     in our
      systems,   we   administrate      navigation    with  two    different interaction  mechanisms.
systems, we administrate navigation with two different interaction mechanisms. One with pure              One    with   pure
audioaudio   instructions
        instructions     [3] [3]
                             andand thethe  otherwith
                                         other     withhaptic
                                                         haptic instructions.
                                                                  instructions. Experience
                                                                                 Experience  surveys   are are
                                                                                                 surveys    collected   after after
                                                                                                                 collected
      the tests. A 5-minute tutorial on the navigation instructions is given prior to the tests, and all of the
the tests. A 5-minute tutorial on the navigation instructions is given prior to the tests, and all of the
      subjects are told that security personnel will interfere before any collision or risk is met. This gives
subjects are told that security personnel will interfere before any collision or risk is met. This gives the
      the users peace of mind.
users peace    of mind.
           After   the test, all four subjects believed they successfully followed the instructions to reach the
      After
      target the  test,
             (5/5);  mostallsubjects
                             four subjects
                                       agreed believed     they successfully
                                                that the instructions    were veryfollowed    the instructions
                                                                                     easy to understand    (4.5/5); to
                                                                                                                     andreach
                                                                                                                           all the
targetsubjects
        (5/5); agreed
                most subjects
                         that theiragreed
                                      cognitionthat   the instructions
                                                  of haptic  instructions were    verya short
                                                                           enhanced     easy while
                                                                                               to understand       (4.5/5);
                                                                                                      after beginning     the and
all subjects   agreed
      experiment        that
                    (5/5).     their cognition
                           Furthermore,             of haptic
                                            all subjects agreed  instructions   enhanced
                                                                   that the haptic           a short
                                                                                   instructions  werewhile     after
                                                                                                       less likely to beginning
                                                                                                                       cause
      hesitation than
the experiment            audio
                     (5/5).       instructions (5/5);
                              Furthermore,              some subjects
                                                  all subjects    agreedbelieved
                                                                           that thethat  theyinstructions
                                                                                      haptic    feel safer than   expected
                                                                                                               were    less likely
      (3.75/5);
to cause         most believed
           hesitation     than audio that instructions
                                           they had a better
                                                           (5/5);experience      with believed
                                                                     some subjects     haptic instructions
                                                                                                   that they than      audio than
                                                                                                                feel safer
      instructions
expected   (3.75/5); in most
                        micro-guidance
                               believed that (4.75/5);
                                                  theyand
                                                        hadall  believed
                                                              a better     that audiowith
                                                                         experience    instructions   were indispensable
                                                                                            haptic instructions       than audio
      as macro-instructions (5/5). Two subjects believed the haptic glove would affect holding objects in
instructions in micro-guidance (4.75/5); and all believed that audio instructions were indispensable as
      daily life and suggested migrating the haptic component to the arm or back of the hand.
macro-instructions (5/5). Two subjects believed the haptic glove would affect holding objects in daily
life and suggested migrating the haptic component to the arm or back of the hand.
Appl. Sci. 2019, 9, 989                                                                                       14 of 15

5. Conclusions
      In this work, we propose a human-centric navigation system to assist people with visual
impairment while travelling indoors. The system takes a commercial smartphone as the carrier
and uses Google ARCore vison SLAM in positioning. Comparing with the conventional visual
odometry-supported travel aids, the system achieves better mapping and tracking. An adaptive
artificial potential field-based path planning has been proposed for the system; it keeps the path
away from the obstacles so as to avoid risk and collision while generating a real-time smooth path.
Finally, a dual-channel human-machine interaction mechanism is introduced in the system. The system
user-centrically incorporates haptic interfaces to provide a fluent and continuous guidance superior to
the conventional turn-by-turn audio-guiding method. The haptic interaction can be carried out via
different candidate devices, but our proposed haptic gloves benefit from an affordable cost and the
convenience to plug and play.
      Evaluation on field tests and simulations shows that the localization and path planning achieves
the expected performance, and as such, the proposed ANSVIP system is welcomed by visually
impaired subjects.

Author Contributions: Conceptualization, X.Z.; Investigation, Y.Z.; Methodology, X.Z. and F.H.; Project
administration, F.H.; Resources, Y.Z.; Software, X.Y.; Writing—original draft, X.Z.
Funding: This work was funded by the Humanity and Social Science Youth foundation of the Ministry of
Education of China, grant number 18YJCZH249, 17YJCZH275.
Acknowledgments: The authors would like to thank Bing Li, Jizhong Xiao and Wei Wang for their insightful
suggestions regarding this research. We thank LetPub for its linguistic assistance during the preparation of
this manuscript.
Conflicts of Interest: The authors declare no conflict of interest.

References
1.    Horton, E.L.; Renganathan, R.; Toth, B.N.; Cohen, A.J.; Bajcsy, A.V.; Bateman, A.; Jennings, M.C.; Khattar, A.;
      Kuo, R.S.; Lee, F.A.; et al. A review of principles in design and usability testing of tactile technology for
      individuals with visual impairments. Assist. Technol. 2017, 29, 28–36. [CrossRef] [PubMed]
2.    Katz, B.F.G.; Kammoun, S.; Parseihian, G.; Gutierrez, O.; Brilhault, A.; Auvray, M.; Truillet, P.; Denis, M.;
      Thorpe, S.; Jouffrais, C. NAVIG: Augmented reality guidance system for the visually impaired. Virtual Reality
      2012, 16, 253–269. [CrossRef]
3.    Zhang, X. A Wearable Indoor Navigation System with Context Based Decision Making for Visually Impaired.
      Int. J. Adv. Robot. Autom. 2016, 1, 1–11. [CrossRef]
4.    Ahmetovic, D.; Gleason, C.; Kitani, K.M.; Takagi, H.; Asakawa, C. NavCog: Turn-by-turn smartphone
      navigation assistant for people with visual impairments or blindness. In Proceedings of the 13th Web for All
      Conference Montreal, Montreal, QC, Canada, 11–13 April 2016; pp. 90–99. [CrossRef]
5.    Bing, L.; Munoz, J.P.; Rong, X.; Chen, Q.; Xiao, J.; Tian, Y.; Arditi, A.; Yousuf, M. Vision-based Mobile Indoor
      Assistive Navigation Aid for Blind People. IEEE Trans. Mobile Comput. 2019, 18, 702–714.
6.    Nair, V.; Budhai, M.; Olmschenk, G.; Seiple, W.H.; Zhu, Z. ASSIST: Personalized Indoor Navigation via
      Multimodal Sensors and High-Level Semantic Information. In Proceedings of the 2018 European Conference
      on Computer Vision, Munich, Germany, 8–14 September 2018; Volume 11134, pp. 128–143. [CrossRef]
7.    Fernandes, H.; Costa, P.; Filipe, V.; Paredes, H.; Barroso, J. A review of assistive spatial orientation and
      navigation technologies for the visually impaired. In Universal Access in the Information Society; Springer:
      Berlin/Heidelberg, Germany, 2017. [CrossRef]
8.    Yang, Z.; Ganz, A. A Sensing Framework for Indoor Spatial Awareness for Blind and Visually Impaired
      Users. IEEE Access 2019, 7, 10343–10352. [CrossRef]
9.    Jiao, J.C.; Yuan, L.B.; Deng, Z.L.; Zhang, C.; Tang, W.H.; Wu, Q.; Jiao, J. A Smart Post-Rectification Algorithm
      Based on an ANN Considering Reflectivity and Distance for Indoor Scenario Reconstruction. IEEE Access
      2018, 6, 58574–58586. [CrossRef]
Appl. Sci. 2019, 9, 989                                                                                          15 of 15

10.   Joseph, S.L.; Xiao, J.Z.; Zhang, X.C.; Chawda, B.; Narang, K.; Rajput, N.; Mehta, S.; Subramaniam, L.V.
      Being Aware of the World: Toward Using Social Media to Support the Blind with Navigation. IEEE Trans.
      Hum.-Mach. Syst. 2015, 45, 399–405. [CrossRef]
11.   Xiao, J.; Joseph, S.L.; Zhang, X.; Li, B.; Li, X.; Zhang, J. An Assistive Navigation Framework for the Visually
      Impaired. IEEE Trans. Hum.-Mach. Syst. 2017, 45, 635–640. [CrossRef]
12.   Zhang, X.; Bing, L.; Joseph, S.L.; Xiao, J.; Yi, S.; Tian, Y.; Munoz, J.P.; Yi, C. A SLAM Based Semantic Indoor
      Navigation System for Visually Impaired Users. In Proceedings of 2015 IEEE International Conference on
      Systems, Man, and Cybernetics, Kowloon, China, 9–12 October 2015.
13.   Bing, L.; Muñoz, J.P.; Rong, X.; Xiao, J.; Tian, Y.; Arditi, A. ISANA: Wearable Context-Aware Indoor Assistive
      Navigation with Obstacle Avoidance for the Blind. In Proceedings of the 2016 European Conference on
      Computer Vision, Amsterdam, The Netherlands, 8–16 October 2016.
14.   Zhao, Y.; Zheng, Z.; Liu, Y. Survey on computational-intelligence-based UAV path planning. Knowl.-Based
      Syst. 2018, 158, 54–64. [CrossRef]
15.   Ahmetovic, D.; Oh, U.; Mascetti, S.; Asakawa, C. Turn Right: Analysis of Rotation Errors in Turn-by-Turn
      Navigation for Individuals with Visual Impairments. In Proceedings of the 20th International ACM Sigaccess
      Conference on Computers and Accessibility, Assets’18, Galway, Ireland, 22–24 October 2018; pp. 333–339.
      [CrossRef]
16.   Balata, J.; Mikovec, Z.; Slavik, P. Landmark-enhanced route itineraries for navigation of blind pedestrians in
      urban environment. J. Multimodal User Interfaces 2018, 12, 181–198. [CrossRef]
17.   Soltani, A.R.; Tawfik, H.; Goulermas, J.Y.; Fernando, T. Path planning in construction sites: Performance
      evaluation of the Dijkstra, A*, and GA search algorithms. Adv. Eng. Inform. 2002, 16, 291–303. [CrossRef]
18.   Sato, D.; Oh, U.; Naito, K.; Takagi, H.; Kitani, K.; Asakawa, C. NavCog3 An Evaluation of a Smartphone-Based
      Blind Indoor Navigation Assistant with Semantic Features in a Large-Scale Environment. In Proceedings of
      the 19th International ACM SIGACCESS Conference on Computers and Accessibility, Baltimore, MD, USA,
      20 October–1 November 2017. [CrossRef]
19.   Epstein, R.A.; Patai, E.Z.; Julian, J.B.; Spiers, H.J. The cognitive map in humans: Spatial navigation and
      beyond. Nat. Neurosci. 2017, 20, 1504–1513. [CrossRef] [PubMed]
20.   Marianne, F.; Sturla, M.; Witter, M.P.; Moser, E.I.; May-Britt, M. Spatial representation in the entorhinal cortex.
      Science 2004, 305, 1258–1264.
21.   Papadopoulos, K.; Koustriava, E.; Koukourikos, P.; Kartasidou, L.; Barouti, M.; Varveris, A.; Misiou, M.;
      Zacharogeorga, T.; Anastasiadis, T. Comparison of three orientation and mobility aids for individuals with
      blindness: Verbal description, audio-tactile map and audio-haptic map. Assist. Technol. 2017, 29, 1–7.
      [CrossRef] [PubMed]
22.   Rector, K.; Bartlett, R.; Mullan, S. Exploring Aural and Haptic Feedback for Visually Impaired People on
      a Track: A Wizard of Oz Study. In Proceedings of the 20th International ACM Sigaccess Conference on
      Computers and Accessibility, Assets’18, Galway, Ireland, 22–24 October 2018. [CrossRef]
23.   Papadopoulos, K.; Koustriava, E.; Koukourikos, P. Orientation and mobility aids for individuals with
      blindness: Verbal description vs. audio-tactile map. Assist. Technol. 2018, 30, 191–200. [CrossRef] [PubMed]
24.   Guerreiro, J.; Ohn-Bar, E.; Ahmetovic, D.; Kitani, K.; Asakawa, C. How Context and User Behavior Affect
      Indoor Navigation Assistance for Blind People. In Proceedings of the 2018 Internet of Accessible Things,
      Lyon, France, 23–25 April 2018. [CrossRef]
25.   Kacorri, H.; Ohn-Bar, E.; Kitani, K.M.; Asakawa, C. Environmental Factors in Indoor Navigation Based on
      Real-World Trajectories of Blind Users. In Proceedings of the 2018 CHI Conference on Human Factors in
      Computing Systems, Montreal, QC, Canada, 21–26 April 2018; pp. 1–12. [CrossRef]
26.   Boerema, S.T.; van Velsen, L.; Vollenbroek-Hutten, M.M.R.; Hermens, H.J. Value-based design for the elderly:
      An application in the field of mobility aids. Assist. Technol. 2017, 29, 76–84. [CrossRef] [PubMed]
27.   Mone, G. Feeling Sounds, Hearing Sights. Commun. ACM 2018, 61, 15–17. [CrossRef]
28.   Martins, L.B.; Lima, F.J. Analysis of Wayfinding Strategies of Blind People Using Tactile Maps. Procedia
      Manuf. 2015, 3, 6020–6027. [CrossRef]

                          © 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access
                          article distributed under the terms and conditions of the Creative Commons Attribution
                          (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
You can also read
Next slide ... Cancel