The mobile future of augmented reality - Qualcomm Technologies, Inc - @qualcomm_tech

Page created by Dustin Sutton
 
CONTINUE READING
The mobile future of augmented reality - Qualcomm Technologies, Inc - @qualcomm_tech
Oct 2018                 @qualcomm_tech

The mobile future of
augmented reality
Qualcomm Technologies, Inc.
The mobile future of augmented reality - Qualcomm Technologies, Inc - @qualcomm_tech
Agenda
    Brief introduction to
1   Augmented Reality

    Evolution of AR from
2   today to the future

3   New technologies for AR
    requirements

                              2
The mobile future of augmented reality - Qualcomm Technologies, Inc - @qualcomm_tech
Augmented reality
will impact every
aspect of our lives
Offering unprecedented experiences
and increased productivity

                                     3
The mobile future of augmented reality - Qualcomm Technologies, Inc - @qualcomm_tech
AR and VR offer distinct experiences
But share similar underlying technologies

Virtual reality                                           Augmented reality
Simulates physical presence in real or imagined worlds,   Superimposes content over the real world such that the
and enables the user to interact in that world            content appears to be part of the real-world scene

                                                                                                                   4
The mobile future of augmented reality - Qualcomm Technologies, Inc - @qualcomm_tech
Evolution of user experience from VR to AR
               Today                                         Soon                                     The future

VR: Mostly 3-DOF, lower resolution videos   VR: Ability to move around through live       XR: Entire scenes, like entertainment
    and games                                   events, with better sense of “presence”       events, can be accessed with your
                                                                                              mobile AR device that are so realistic
                                                                                              and interactive that they’ll be nearly
AR: Pokémon Go, Google Translate,           AR: Still rudimentary, yet more useful            indistinguishable from reality. VR
    Snapchat, and other rudimentary             and immersive, streaming AR services,         becomes an occasionally used “mode”
    AR apps                                     abled to be accessed on the go                within AR

                                                                                                                                       5
The mobile future of augmented reality - Qualcomm Technologies, Inc - @qualcomm_tech
AR will transform the way we live and interact with the world
Offering unprecedented experiences and increased productivity

A plurality of experiences in AR

Play                       Explore                 Communicate                Work                        Thrive

Interactive gaming in      Immersive education     Telepresence               Training and productivity   Health
any environment            and learning
                                                   Shared visualizations      Virtual office, anywhere    Fitness and sports
Movies and shows on a      3D design and art       and personal moments
big screen, anywhere                                                          Collaboration through a     Personalized
                           Travel and navigation   Personalized advertising   shared view                 recommendations
Augmented live concerts,
sports, and other events   Shopping and retail

                                       Helping us navigate our daily lives
                                                                                                                               6
The mobile future of augmented reality - Qualcomm Technologies, Inc - @qualcomm_tech
AR will serve a broad spectrum of roles in daily life
Applicable across ages, genders, and activities

Children               Young                   Families                  Professionals              Fitness Enthusiasts
Playing                Adults Exploring        Communicating             Working                    Thriving

Kids chasing virtual   A young man exploring   Families virtually        Architects collaborating   Group running
characters in more     Rome and seeing         brought together with     on a shared design         with a virtual trainer
interactive and        the Colosseum as        life-like communication   to improve efficiency      to motivate them
immersive games        originally built

                                                                                                                             7
The mobile future of augmented reality - Qualcomm Technologies, Inc - @qualcomm_tech
AR will revolutionize industries and enterprises
Increased productivity, efficiency, and safety
Industrial and manufacturing                     Engineering
• Guided training and remote support             • 3D visualization and CAD
• Improved safety                                • Colleague collaboration and
• Real-time factory diagnostics                    communication

Healthcare                                       Retail
• More efficient patient care                    • Try before you buy: clothes, furniture,
• Diagnosis and treatment assistance               car, real estate shopping, etc.
• Surgical training and visualization            • Navigation to products and
                                                   personalized coupons
Education                                        Marketing and advertising
• Immersive, self-guided,
                                                 • Personalized ads based on
  interactive visual learning
                                                   context
• Any subject, from history and
                                                 • Consumer data – what they like,
  physics to vocational
                                                   what they look at, etc.

Military                                         Emergency response
• Instructional training                         • Police, fire, security response
• In-the-field assistance                        • Potential improvements in safety,
                                                   response time, and saving lives
                                                                                       8
The mobile future of augmented reality - Qualcomm Technologies, Inc - @qualcomm_tech
AR is the next mobile computing platform
Nearly everything we’ve learned for smartphones will be used for AR

                                                           AR glasses

                            Smartphone

   Laptop

                                                                        9
The mobile future of augmented reality - Qualcomm Technologies, Inc - @qualcomm_tech
AR technologies and use cases evolve from mobile
VR usage primarily comes from console/TV/PC, but it’s also moving towards AR

                                                                              Ultimately,
                                                                    this becomes an imperceptible
 Landline
                      PDA                                                 device that replaces
                                                                            nearly all others
                               Handheld

            Camera
                                gaming                 AR glasses

                      Laptop              Smartphone   VR HMD

    TV

             Desktop PC          Gaming consoles
                                                                                           10
AR is here today, but it is still in its infancy
Like smartphones, the AR evolution will take years but has the potential to be huge

        Technology Phase: Infancy
        Market: Mostly early adopter “Prosumers”

                      Technology Phase: Rapid evolution
                      Market: Surging consumer adoption

                                                                               Technology Phase: Maturity
                                                                               Market: Worldwide, ubiquitous use

  AR will follow a similar ~30 year cycle of sleeker designs, with tremendously increasing functionality
                                                                                                                   11
A glimpse into the future
First responder AR glasses

Bone conduction transducers                           Multimode connectivity (4G, 5G, etc.)

Rear camera
                                                      • Many passive and active cameras
                                                        with fisheye and telephoto lenses
                                                      • Optoelectronic night vision and
                                                        thermal imaging sensors

                                                      Ambient light sensors
Directional speakers

Tracking and recording cameras
                                                      Eye tracking cameras
Inertial, haptic, environmental,
and health sensors

                            Ultra-bright LED lights
                                                      New optics and projection
                                                      technologies within a durable,
      Multiple high sensitivity audio microphones     semitransparent display

                                                                                              12
AR will seamlessly merge the real world with virtual objects
Providing an always-on experience that intelligently enhances our lives

            Immersive                                        Intelligent                                    Connected
 The visuals, sounds, and interactions are     It understands the real world, learns personal   An always-on, low power wearable with fast
    so realistic that they are true to life   preferences, and provides security and privacy       wireless cloud connectivity anywhere
                                                                                                                                         13
Immersive
The visuals, sounds, and interactions are
so realistic that they are true to life

                                            14
AR shares requirements similar to VR for immersion
Achieving realistic AR at low power to enable a comfortable, sleek form factor

 Extreme pixel quantity and quality
      Screen is very close to the eyes                                                            High resolution audio
                                                                                                  Up to human hearing capabilities
                     Spherical view             Visual                       Sound
               Look anywhere with a                                                               3D audio
                                                quality                      quality              Realistic 3D, positional, surround
              full 360° spherical view
                                                                                                  audio that is accurate to the real world
                                                             Immersion
              Stereoscopic display
                                                                                                  Crystal clear voice
                   Humans see in 3D
                                                                                                  Clear voice that is enhanced with
                                                                                                  noise cancellation technology

                                                               Intuitive
                                                             interactions
                                Minimal latency                                        Precise motion tracking
                           Minimized system latency                                    Accurate on-device motion tracking
                           to remove perceptible lag

                                                       Natural user interfaces
                                                  Seamlessly interact with VR using
                                                  natural movements, free from wires

                                                                                                                                             15
                               Learn more about our vision for the future of VR: www.qualcomm.com/VR
AR introduces additional requirements for immersion
Seamlessly integrating virtual objects with the real world

       Keeping the world stable
         Seamlessly anchor virtual
          objects to the real world
                                              Visual                      Sound             Realistic Virtual Sounds
                        Occlusion                                         quality           Modifying virtual sounds based
        Showing and hiding virtual
                                              quality
                                                                                            on the real world environment
            objects appropriately                         Immersion
           Common illumination
             Lighting virtual objects
       realistically and dynamically

                                                            Intuitive
                                                          interactions

                                                    Natural user interfaces
                                               Seamlessly interact with VR using
                                               natural movements, free from wires

                                                                                                                             16
                              Learn more about our vision for the future of VR: www.qualcomm.com/VR
Keeping the world stable
In an unstable environment, virtual objects are NOT
seamlessly anchored to the real world

                                                      17
Keeping the world stable
In a stable environment, virtual objects are
seamlessly anchored to the real world

                                               18
Occluding virtual objects correctly
Incorrect occlusion breaks immersion

                                       19
Occluding virtual objects correctly
Correct occlusion accounts for the depth of virtual and real objects

                                                                       20
Occluding virtual objects correctly
Smart occlusion accounts for both object depth and user preferences

                                                                      21
Lighting virtual objects realistically and dynamically
Incorrect lighting poorly represents the position, intensity,
and orientation of all light sources

Poor environment
processing
• Virtual objects look fake
  and out of place
• Static lighting; often incorrect
  for environment
• Solid objects do not look solid
• Materials look physically
  incorrect
• Interactivity is not smooth

                                                                22
Lighting virtual objects realistically and dynamically
Correct lighting considers the position, intensity,
and orientation of all light sources

Proper AR environment
processing
• Virtual objects look real and
  correctly placed
• Dynamic lighting; correct for
  the environment
• Solid objects look solid
• Materials look physically correct
• Interactivity is smooth

Making it possible
• Intelligent, fast interaction between many
  different sensors & rendering systems
• New computer vision and global
  illumination algorithms to dynamically
  render and overlay realistic AR objects

                                                         23
Creating virtual sounds based on the real world
Sound reflections spread and interact with the environment appropriately

Airport
• Limited sound reflections                                        Hotel room
• Significant ambient sound                                        • Significant sound
                                                                     dampening
Conference room
• Enclosed room with
  reflective surfaces                                              Required
• Virtual people should                                            technologies
  sound like they are in the                                       • Environment modeling
  conference room
                                                                   • Noise filtering
                                                                   • Reverberation
                                                                   • Positional audio

                                                                                            24
Interacting naturally with AR
Interactions will become more intuitive and adaptive to personal preferences
                                             Motion and gesture recognition
                            Use CV along with motion sensors, and new types of connected,
                                    haptic devices to help users interact within AR

    Speech recognition and learning                                                Face recognition
    Use of natural language processing,                                            Use of advanced CV to authenticate
      intelligently personalized to user’s                                         and accurately recognize facial
                        voice and lexicon                                          expressions

                                                    Adaptive, multimodal,
             Personalized interfaces                   user interfaces             Eye tracking
            Learn and know a myriad of                                             Use CV to much more accurately
            user preferences based on                                              authenticate, and also track and measure
                      machine learning                                             point of gaze

                                                 Bringing life to objects
                                    Efficient user interfaces for controlling interaction
                                           with IoT devices and cloud services                                                25
Intelligence
Understanding the real world and learning
personal preferences

                                            26
AI technologies are key for AR adoption
Making sense of the world while protecting our privacy and security

         Machine learning                  Security and privacy are critical for AR
           Makes visual, audio,                   Continuous            Utilize combined machine
        and other sensor processing        authentication necessary    learning on biometrics and
              more intelligent              for identity and access          behavioral activity

                                      On-device privacy and security

           Visual Processing             Audio Processing                 Other sensor processing

                                           Artificial intelligence

                                                                                                    27
Intelligent AR will greatly expand our human abilities
By understanding the environment and providing personalized assistance

    Make travel easier               Assist the visually impaired                  Become a pro
 Describe the landmarks around you    Help the visually impaired map their   Make a gourmet meal, fix your car,
     and translate street signs          environment and get around             or perfect your jump shot

                                                                                                                  28
Contextual intelligence to “sense” the world
The fusion of many types of sensors and personal information is required
  Sensor data                                                On-device data           Off-device data

  Environment        Iris scan   Ambient light    Compass        Calendar                Cloud data

    Camera         Microphone    Temperature      Humidity       Messaging                IoT data

                                                                                                                     Sensor fusion
   Gyroscope          Pulse         C-V2X                          Apps

Low power sensing, processing, and connectivity
Efficient, heterogeneous           Sensor fusion and          Integrated, always-on          Low-energy wireless technologies
architectures                      machine learning           data capturing                 (e.g. BT-LE, 5G NR IoT)
                                                                                                                                     29
Visual intelligence to “see” the world around you
Continually monitoring the visual world to intelligently identify

3D depth capture,
interpolation, and                                      29 feet
reconstruction
Using passive + active cameras,
along with advanced CV and                                                       11 feet
machine learning algorithms

Object recognition,
tracking, and registration
                                                                     8 feet
Using CV and machine learning
so that objects in the real and
virtual worlds are properly aligned   Determine the size, direction, and distance of different objects,
with respect to each other            and (sometimes) store the whole 3D scene for various uses

                                                                                                      30
Recognize, track, map amd reconstruct surroundings

Underlying technologies
Object recognition,         Simultaneous localization   Visual inertial   3D reconstruction
tracking and registration   and mapping (SLAM)          odometry (VIO)
                                                                                              31
Lisbon, Portugal

Understand and inform
                                                                                               Temp: 98° F

Recognize the relationship between objects and provide relevant information
          Cable Car
          Cost: €3
          Departing: 10 min                                                                                18th Street
          Destination time: 1 hr
                                                  Café
                                                  4.5 stars
                                                  Serves Caldo Verde

                                                   Taxi                                       Sarah M.
                                                   Cost: €25                                  Last talked: 2 months
                                                   Departing: Now                             Ask about her son, Joe
                                                   Destination time: 15 min
                                                                              Passenger
                                                                              boarding

                                           Perceptual tasks
                          Identify real   Understand             Recognize                Distinguish
                            objects          text                 people                   activities
                                                                                                                         32
Recommend actions
Personalized virtual tour guide

                                  “Catch this trolley to take
                                   your next tour. This one
                                   also has air conditioning
                                   available. Have €3 ready.”

                                                                33
AI to autonomously take actions
Infer context and anticipate needs

                         Autonomously take predictive, necessary actions

     High level concept understanding with learned perception and awareness

         Motion        User identification Ambient speech and     Personal places and    Object detection    Cloud data
      classification    and preferences audio classification    location understanding   and recognition    and services

                                                                                                                           34
Connected
An always-on device with fast wireless
cloud connectivity anywhere

                                         35
AR requires the next level of ubiquitous connectivity
The mass adoption of a highly mobile, intelligent, always-on wearable device

        Providing extreme            Connecting              Enabling cloud
             capacity                everything                computing

                                                                               36
AR drives the need to efficiently grow wireless capacity
High-capacity connectivity to share and consume content 24/7

                      Downlink/
                                         Demanding and generating more data
                      Consume
                                         Visual content, such as videos,images, or graphics
                                         •   Higher resolution, higher frame rate, stereoscopic, High
                                             Dynamic Range (HDR), 360° spherical content

                                         Contextual data from sensors and the cloud
    Uplink/Share
                                         Constant upload and download on an
                                         all-day wearable

                                         Mass adoption—the next mobile computing platform

                                                                                                        37
Gigabit Class LTE is the foundation for AR
wireless capacity
Technology enablers for achieving Gigabit Class LTE

                                                                    Carrier Aggregation
                                                                    Combines multiples LTE carriers for wider bandwidth, e.g. fatter pipe

                                                                    Aggregation across diverse spectrum types
              More bandwidth                                        Makes the best use of spectrum, e.g. FDD/TDD, licensed/unlicensed spectrum
      Better spectral efficiency
Best use of all spectrum assets                                     Advanced MIMO (4x4)
                                                                    Leverages more antennas to increase spectral efficiency

                                                                    Higher-order modulation (256-QAM)
                                                                    Transmits more bits per symbol to increase spectral efficiency

QAM = quadrature amplitude modulation, MIMO = multiple-input and multiple-output                                                             38
Contextual intelligence requires connecting everything
UI augmentation is limited without hyper-connectivity

Creating a                                                              Connecting:
connectivity fabric                                                     • Device to device
for everything
                                                                        • Device to cloud
Securely gathering
contextual data for                                                     • AR glasses to devices
increased intelligence
                                                                        • AR glasses to cloud

Heterogeneous
connectivity for IoT      LTE      Wi-Fi    Bluetooth   GNSS/Location      NFC        Powerline
                                                                                                  39
Cloud computing complements on-device processing
Leveraging ubiquitous connectivity for the best of both worlds

On-device processing                                                              Cloud computing

       Security and                                                                     Boundless cloud data
       user privacy                                                                     and storage
                                           Both are needed
       Low latency for instant                                                          Crowd-sourcing and
       processing                                                                       data aggregation

       Availability in and out of   Personal information    Public information          Big data processing without
       coverage                      and limited context   and in-depth context         power constraints

       Efficient use of network                                                         Reliable, low-latency, and
       bandwidth and processing                                                         global cloud connectivity

                                                                                                                      40
Taking AR experiences to the next level with 5G
Ubiquitous coverage with Gigabit LTE / 5G multi-mode devices

Enjoy AR experiences everywhere                                      Share real-time/interactive experiences
At home, at work, at school, in the car, walking around…             Events, meetings, telepresence…

Extreme throughput             Ultra-low latency            Uniform experience              All while improving energy
multi-gigabits per second      down to 1ms latency          with much more capacity         efficiency and lowering cost
                            Learn more about our vision for the future of mobile networks: www.qualcomm.com/5G
                                                                                                                           41
Power efficiency is essential for AR
The AR headset needs to be comfortable to wear all day

                                                            Constrained mobile
       AR workloads                                        wearable environment
• Compute intensive                                      • Thermally efficient for sleek,
                                                           ultra-light designs
• Complex concurrencies
                                                         • Long battery life for
• Always-on                                                all-day use
• Real-time

                                                                                        42
QTI is uniquely
positioned to support
superior AR experiences
Custom designed SoCs and investments
in the core AR technologies

                                       43
We’re developing foundational technology for AR
Qualcomm Technologies’ investments and the confluence of mobile technologies

Computer                 AI                       Heterogenous            Next-gen
vision                   and security             computing               connectivity
• 6-DOF VIO              • AI for advanced        • Lower power, higher   • Gigabit LTE
                           cognitive processing     perf. AR visual         and Wi-Fi
• SLAM and 3DR
                                                    processing
                         • Local and cloud                                • Pioneering 5G
• Object detection and
                           machine learning       • Advancements in         technologies
  recognition
                                                    always-on sensor
                         • Security and privacy                           • Connectivity
                                                    fusion
                                                                            convergence
                                                  • Next-gen AR audio
                                                                                            44
Snapdragon 845 is designed to meet AR requirements
Utilizing specialized engines across the SoC for efficient processing

                                                                                                                               LPDDR4X                     Qualcomm                Qualcomm®
                                                                                                                               Memory                      Spectra™                Snapdragon™ X20
                                                                                                                                                           280 ISP                 LTE Modem

Augmented reality requires
heterogeneous computing                                                                                                         Display
                                                                                                                                Processor
                                                                                                                                                           Qualcomm®
                                                                                                                                                           Hexagon™
                                                                                                                                                                                   Video Processor
                                                                                                                                                                                   (VPU)
Computer vision, machine learning, image                                                                                        (DPU)                      682 DSP
processing, sensor processing, graphics, video
processing, and cloud connectivity                                                                                             Qualcomm®                                 Qualcomm®
                                                                                                                               Adreno™ 630 GPU                           Kryo™ 385 CPU

                                                                                                                       Entire SoC is used!                                  High-utilization    * Not to scale

Snapdragon, Qualcomm Adreno, Qualcomm Hexagon, Qualcomm Spectra, Qualcomm Aqstic, Qualcomm Secure Processing Unit, and Qualcomm Kryo are products of Qualcomm Technologies, Inc.                                 45
We are also investing in these innovative start-ups
Qualcomm Ventures portfolio

   Professional “light field”   Professional 3D reconstruction   Smartphone AR software for
    cameras and software            cameras and software             “visual marketing”

     Software & hardware          Virtual reality game studio      Wearable mixed reality
     for AR/VR controllers

                                                                                              46
AR is the next mobile
computing platform

AR is here today, but still
in infancy

Advancements are required
to make AR optimally
immersive, intelligent,
and connected

Qualcomm Technologies
will continue to innovate
AR technologies

                              47
Thank you
Follow us on:
For more information, visit us at:
www.qualcomm.com & www.qualcomm.com/blog

Nothing in these materials is an offer to sell any of the   References in this presentation to “Qualcomm” may mean Qualcomm
components or devices referenced herein.                    Incorporated, Qualcomm Technologies, Inc., and/or other subsidiaries
                                                            or business units within the Qualcomm corporate structure, as
©2018 Qualcomm Technologies, Inc. and/or its affiliated
                                                            applicable. Qualcomm Incorporated includes Qualcomm’s licensing
companies. All Rights Reserved.
                                                            business, QTL, and the vast majority of its patent portfolio. Qualcomm
Qualcomm, Snapdragon, Kryo, Qualcomm Spectra, Adreno        Technologies, Inc., a wholly-owned subsidiary of Qualcomm
and Hexagon are trademarks of Qualcomm Incorporated,        Incorporated, operates, along with its subsidiaries, substantially all of
registered in the United States and other countries.        Qualcomm’s engineering, research and development functions, and
Qualcomm Aqstic is a trademark of Qualcomm Incorporated.    substantially all of its product and services businesses, including its
Other products and brand names may be trademarks or         semiconductor business, QCT.
registered trademarks of their respective owners.
Resources
• Websites
    ◦   Augmented reality: https://www.qualcomm.com/AR
    ◦   Virtual reality: https://www.qualcomm.com/VR
    ◦   Immersive experiences: https://www.qualcomm.com/Immersive
    ◦   Developers: https://developer.qualcomm.com
    ◦   Newsletter signup: http://www.qualcomm.com/mobile-computing-newsletter

• Presentations
    ◦ Virtual reality: https://www.qualcomm.com/documents/making-immersive-virtual-reality-possible-mobile
    ◦ Immersive experiences: https://www.qualcomm.com/documents/immersive-experiences-presentation
    ◦ SlideShare: http://www.slideshare.net/qualcommwirelessevolution

• Papers
    ◦ Virtual reality: https://www.qualcomm.com/documents/whitepaper-making-immersive-virtual-reality-possible-mobile
    ◦ Immersive experiences: https://www.qualcomm.com/documents/whitepaper-driving-new-era-immersive-experiences-
      qualcomm

• Videos:
    ◦ Immersive experiences webinar: https://www.qualcomm.com/videos/webinar-new-era-immersive-experiences-whats-
      next
    ◦ Virtual reality webinar: https://www.qualcomm.com/videos/webinar-making-immersive-virtual-reality-possible-mobile
                                                                                                                      49
You can also read