The mobile future of augmented reality - Qualcomm Technologies, Inc - @qualcomm_tech
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Oct 2018 @qualcomm_tech The mobile future of augmented reality Qualcomm Technologies, Inc.
Agenda
Brief introduction to
1 Augmented Reality
Evolution of AR from
2 today to the future
3 New technologies for AR
requirements
2
Augmented reality
will impact every
aspect of our lives
Offering unprecedented experiences
and increased productivity
3
AR and VR offer distinct experiences
But share similar underlying technologies
Virtual reality Augmented reality
Simulates physical presence in real or imagined worlds, Superimposes content over the real world such that the
and enables the user to interact in that world content appears to be part of the real-world scene
4
Evolution of user experience from VR to AR
Today Soon The future
VR: Mostly 3-DOF, lower resolution videos VR: Ability to move around through live XR: Entire scenes, like entertainment
and games events, with better sense of “presence” events, can be accessed with your
mobile AR device that are so realistic
and interactive that they’ll be nearly
AR: Pokémon Go, Google Translate, AR: Still rudimentary, yet more useful indistinguishable from reality. VR
Snapchat, and other rudimentary and immersive, streaming AR services, becomes an occasionally used “mode”
AR apps abled to be accessed on the go within AR
5
AR will transform the way we live and interact with the world
Offering unprecedented experiences and increased productivity
A plurality of experiences in AR
Play Explore Communicate Work Thrive
Interactive gaming in Immersive education Telepresence Training and productivity Health
any environment and learning
Shared visualizations Virtual office, anywhere Fitness and sports
Movies and shows on a 3D design and art and personal moments
big screen, anywhere Collaboration through a Personalized
Travel and navigation Personalized advertising shared view recommendations
Augmented live concerts,
sports, and other events Shopping and retail
Helping us navigate our daily lives
6
AR will serve a broad spectrum of roles in daily life
Applicable across ages, genders, and activities
Children Young Families Professionals Fitness Enthusiasts
Playing Adults Exploring Communicating Working Thriving
Kids chasing virtual A young man exploring Families virtually Architects collaborating Group running
characters in more Rome and seeing brought together with on a shared design with a virtual trainer
interactive and the Colosseum as life-like communication to improve efficiency to motivate them
immersive games originally built
7
AR will revolutionize industries and enterprises
Increased productivity, efficiency, and safety
Industrial and manufacturing Engineering
• Guided training and remote support • 3D visualization and CAD
• Improved safety • Colleague collaboration and
• Real-time factory diagnostics communication
Healthcare Retail
• More efficient patient care • Try before you buy: clothes, furniture,
• Diagnosis and treatment assistance car, real estate shopping, etc.
• Surgical training and visualization • Navigation to products and
personalized coupons
Education Marketing and advertising
• Immersive, self-guided,
• Personalized ads based on
interactive visual learning
context
• Any subject, from history and
• Consumer data – what they like,
physics to vocational
what they look at, etc.
Military Emergency response
• Instructional training • Police, fire, security response
• In-the-field assistance • Potential improvements in safety,
response time, and saving lives
8
AR is the next mobile computing platform
Nearly everything we’ve learned for smartphones will be used for AR
AR glasses
Smartphone
Laptop
9
AR technologies and use cases evolve from mobile
VR usage primarily comes from console/TV/PC, but it’s also moving towards AR
Ultimately,
this becomes an imperceptible
Landline
PDA device that replaces
nearly all others
Handheld
Camera
gaming AR glasses
Laptop Smartphone VR HMD
TV
Desktop PC Gaming consoles
10AR is here today, but it is still in its infancy
Like smartphones, the AR evolution will take years but has the potential to be huge
Technology Phase: Infancy
Market: Mostly early adopter “Prosumers”
Technology Phase: Rapid evolution
Market: Surging consumer adoption
Technology Phase: Maturity
Market: Worldwide, ubiquitous use
AR will follow a similar ~30 year cycle of sleeker designs, with tremendously increasing functionality
11A glimpse into the future
First responder AR glasses
Bone conduction transducers Multimode connectivity (4G, 5G, etc.)
Rear camera
• Many passive and active cameras
with fisheye and telephoto lenses
• Optoelectronic night vision and
thermal imaging sensors
Ambient light sensors
Directional speakers
Tracking and recording cameras
Eye tracking cameras
Inertial, haptic, environmental,
and health sensors
Ultra-bright LED lights
New optics and projection
technologies within a durable,
Multiple high sensitivity audio microphones semitransparent display
12AR will seamlessly merge the real world with virtual objects
Providing an always-on experience that intelligently enhances our lives
Immersive Intelligent Connected
The visuals, sounds, and interactions are It understands the real world, learns personal An always-on, low power wearable with fast
so realistic that they are true to life preferences, and provides security and privacy wireless cloud connectivity anywhere
13Immersive
The visuals, sounds, and interactions are
so realistic that they are true to life
14AR shares requirements similar to VR for immersion
Achieving realistic AR at low power to enable a comfortable, sleek form factor
Extreme pixel quantity and quality
Screen is very close to the eyes High resolution audio
Up to human hearing capabilities
Spherical view Visual Sound
Look anywhere with a 3D audio
quality quality Realistic 3D, positional, surround
full 360° spherical view
audio that is accurate to the real world
Immersion
Stereoscopic display
Crystal clear voice
Humans see in 3D
Clear voice that is enhanced with
noise cancellation technology
Intuitive
interactions
Minimal latency Precise motion tracking
Minimized system latency Accurate on-device motion tracking
to remove perceptible lag
Natural user interfaces
Seamlessly interact with VR using
natural movements, free from wires
15
Learn more about our vision for the future of VR: www.qualcomm.com/VRAR introduces additional requirements for immersion
Seamlessly integrating virtual objects with the real world
Keeping the world stable
Seamlessly anchor virtual
objects to the real world
Visual Sound Realistic Virtual Sounds
Occlusion quality Modifying virtual sounds based
Showing and hiding virtual
quality
on the real world environment
objects appropriately Immersion
Common illumination
Lighting virtual objects
realistically and dynamically
Intuitive
interactions
Natural user interfaces
Seamlessly interact with VR using
natural movements, free from wires
16
Learn more about our vision for the future of VR: www.qualcomm.com/VRKeeping the world stable
In an unstable environment, virtual objects are NOT
seamlessly anchored to the real world
17Keeping the world stable
In a stable environment, virtual objects are
seamlessly anchored to the real world
18Occluding virtual objects correctly
Incorrect occlusion breaks immersion
19Occluding virtual objects correctly
Correct occlusion accounts for the depth of virtual and real objects
20Occluding virtual objects correctly
Smart occlusion accounts for both object depth and user preferences
21Lighting virtual objects realistically and dynamically
Incorrect lighting poorly represents the position, intensity,
and orientation of all light sources
Poor environment
processing
• Virtual objects look fake
and out of place
• Static lighting; often incorrect
for environment
• Solid objects do not look solid
• Materials look physically
incorrect
• Interactivity is not smooth
22Lighting virtual objects realistically and dynamically
Correct lighting considers the position, intensity,
and orientation of all light sources
Proper AR environment
processing
• Virtual objects look real and
correctly placed
• Dynamic lighting; correct for
the environment
• Solid objects look solid
• Materials look physically correct
• Interactivity is smooth
Making it possible
• Intelligent, fast interaction between many
different sensors & rendering systems
• New computer vision and global
illumination algorithms to dynamically
render and overlay realistic AR objects
23Creating virtual sounds based on the real world
Sound reflections spread and interact with the environment appropriately
Airport
• Limited sound reflections Hotel room
• Significant ambient sound • Significant sound
dampening
Conference room
• Enclosed room with
reflective surfaces Required
• Virtual people should technologies
sound like they are in the • Environment modeling
conference room
• Noise filtering
• Reverberation
• Positional audio
24Interacting naturally with AR
Interactions will become more intuitive and adaptive to personal preferences
Motion and gesture recognition
Use CV along with motion sensors, and new types of connected,
haptic devices to help users interact within AR
Speech recognition and learning Face recognition
Use of natural language processing, Use of advanced CV to authenticate
intelligently personalized to user’s and accurately recognize facial
voice and lexicon expressions
Adaptive, multimodal,
Personalized interfaces user interfaces Eye tracking
Learn and know a myriad of Use CV to much more accurately
user preferences based on authenticate, and also track and measure
machine learning point of gaze
Bringing life to objects
Efficient user interfaces for controlling interaction
with IoT devices and cloud services 25Intelligence
Understanding the real world and learning
personal preferences
26AI technologies are key for AR adoption
Making sense of the world while protecting our privacy and security
Machine learning Security and privacy are critical for AR
Makes visual, audio, Continuous Utilize combined machine
and other sensor processing authentication necessary learning on biometrics and
more intelligent for identity and access behavioral activity
On-device privacy and security
Visual Processing Audio Processing Other sensor processing
Artificial intelligence
27Intelligent AR will greatly expand our human abilities
By understanding the environment and providing personalized assistance
Make travel easier Assist the visually impaired Become a pro
Describe the landmarks around you Help the visually impaired map their Make a gourmet meal, fix your car,
and translate street signs environment and get around or perfect your jump shot
28Contextual intelligence to “sense” the world
The fusion of many types of sensors and personal information is required
Sensor data On-device data Off-device data
Environment Iris scan Ambient light Compass Calendar Cloud data
Camera Microphone Temperature Humidity Messaging IoT data
Sensor fusion
Gyroscope Pulse C-V2X Apps
Low power sensing, processing, and connectivity
Efficient, heterogeneous Sensor fusion and Integrated, always-on Low-energy wireless technologies
architectures machine learning data capturing (e.g. BT-LE, 5G NR IoT)
29Visual intelligence to “see” the world around you
Continually monitoring the visual world to intelligently identify
3D depth capture,
interpolation, and 29 feet
reconstruction
Using passive + active cameras,
along with advanced CV and 11 feet
machine learning algorithms
Object recognition,
tracking, and registration
8 feet
Using CV and machine learning
so that objects in the real and
virtual worlds are properly aligned Determine the size, direction, and distance of different objects,
with respect to each other and (sometimes) store the whole 3D scene for various uses
30Recognize, track, map amd reconstruct surroundings
Underlying technologies
Object recognition, Simultaneous localization Visual inertial 3D reconstruction
tracking and registration and mapping (SLAM) odometry (VIO)
31Lisbon, Portugal
Understand and inform
Temp: 98° F
Recognize the relationship between objects and provide relevant information
Cable Car
Cost: €3
Departing: 10 min 18th Street
Destination time: 1 hr
Café
4.5 stars
Serves Caldo Verde
Taxi Sarah M.
Cost: €25 Last talked: 2 months
Departing: Now Ask about her son, Joe
Destination time: 15 min
Passenger
boarding
Perceptual tasks
Identify real Understand Recognize Distinguish
objects text people activities
32Recommend actions
Personalized virtual tour guide
“Catch this trolley to take
your next tour. This one
also has air conditioning
available. Have €3 ready.”
33AI to autonomously take actions
Infer context and anticipate needs
Autonomously take predictive, necessary actions
High level concept understanding with learned perception and awareness
Motion User identification Ambient speech and Personal places and Object detection Cloud data
classification and preferences audio classification location understanding and recognition and services
34Connected
An always-on device with fast wireless
cloud connectivity anywhere
35AR requires the next level of ubiquitous connectivity
The mass adoption of a highly mobile, intelligent, always-on wearable device
Providing extreme Connecting Enabling cloud
capacity everything computing
36AR drives the need to efficiently grow wireless capacity
High-capacity connectivity to share and consume content 24/7
Downlink/
Demanding and generating more data
Consume
Visual content, such as videos,images, or graphics
• Higher resolution, higher frame rate, stereoscopic, High
Dynamic Range (HDR), 360° spherical content
Contextual data from sensors and the cloud
Uplink/Share
Constant upload and download on an
all-day wearable
Mass adoption—the next mobile computing platform
37Gigabit Class LTE is the foundation for AR
wireless capacity
Technology enablers for achieving Gigabit Class LTE
Carrier Aggregation
Combines multiples LTE carriers for wider bandwidth, e.g. fatter pipe
Aggregation across diverse spectrum types
More bandwidth Makes the best use of spectrum, e.g. FDD/TDD, licensed/unlicensed spectrum
Better spectral efficiency
Best use of all spectrum assets Advanced MIMO (4x4)
Leverages more antennas to increase spectral efficiency
Higher-order modulation (256-QAM)
Transmits more bits per symbol to increase spectral efficiency
QAM = quadrature amplitude modulation, MIMO = multiple-input and multiple-output 38Contextual intelligence requires connecting everything
UI augmentation is limited without hyper-connectivity
Creating a Connecting:
connectivity fabric • Device to device
for everything
• Device to cloud
Securely gathering
contextual data for • AR glasses to devices
increased intelligence
• AR glasses to cloud
Heterogeneous
connectivity for IoT LTE Wi-Fi Bluetooth GNSS/Location NFC Powerline
39Cloud computing complements on-device processing
Leveraging ubiquitous connectivity for the best of both worlds
On-device processing Cloud computing
Security and Boundless cloud data
user privacy and storage
Both are needed
Low latency for instant Crowd-sourcing and
processing data aggregation
Availability in and out of Personal information Public information Big data processing without
coverage and limited context and in-depth context power constraints
Efficient use of network Reliable, low-latency, and
bandwidth and processing global cloud connectivity
40Taking AR experiences to the next level with 5G
Ubiquitous coverage with Gigabit LTE / 5G multi-mode devices
Enjoy AR experiences everywhere Share real-time/interactive experiences
At home, at work, at school, in the car, walking around… Events, meetings, telepresence…
Extreme throughput Ultra-low latency Uniform experience All while improving energy
multi-gigabits per second down to 1ms latency with much more capacity efficiency and lowering cost
Learn more about our vision for the future of mobile networks: www.qualcomm.com/5G
41Power efficiency is essential for AR
The AR headset needs to be comfortable to wear all day
Constrained mobile
AR workloads wearable environment
• Compute intensive • Thermally efficient for sleek,
ultra-light designs
• Complex concurrencies
• Long battery life for
• Always-on all-day use
• Real-time
42QTI is uniquely
positioned to support
superior AR experiences
Custom designed SoCs and investments
in the core AR technologies
43We’re developing foundational technology for AR
Qualcomm Technologies’ investments and the confluence of mobile technologies
Computer AI Heterogenous Next-gen
vision and security computing connectivity
• 6-DOF VIO • AI for advanced • Lower power, higher • Gigabit LTE
cognitive processing perf. AR visual and Wi-Fi
• SLAM and 3DR
processing
• Local and cloud • Pioneering 5G
• Object detection and
machine learning • Advancements in technologies
recognition
always-on sensor
• Security and privacy • Connectivity
fusion
convergence
• Next-gen AR audio
44Snapdragon 845 is designed to meet AR requirements
Utilizing specialized engines across the SoC for efficient processing
LPDDR4X Qualcomm Qualcomm®
Memory Spectra™ Snapdragon™ X20
280 ISP LTE Modem
Augmented reality requires
heterogeneous computing Display
Processor
Qualcomm®
Hexagon™
Video Processor
(VPU)
Computer vision, machine learning, image (DPU) 682 DSP
processing, sensor processing, graphics, video
processing, and cloud connectivity Qualcomm® Qualcomm®
Adreno™ 630 GPU Kryo™ 385 CPU
Entire SoC is used! High-utilization * Not to scale
Snapdragon, Qualcomm Adreno, Qualcomm Hexagon, Qualcomm Spectra, Qualcomm Aqstic, Qualcomm Secure Processing Unit, and Qualcomm Kryo are products of Qualcomm Technologies, Inc. 45We are also investing in these innovative start-ups
Qualcomm Ventures portfolio
Professional “light field” Professional 3D reconstruction Smartphone AR software for
cameras and software cameras and software “visual marketing”
Software & hardware Virtual reality game studio Wearable mixed reality
for AR/VR controllers
46AR is the next mobile
computing platform
AR is here today, but still
in infancy
Advancements are required
to make AR optimally
immersive, intelligent,
and connected
Qualcomm Technologies
will continue to innovate
AR technologies
47Thank you
Follow us on:
For more information, visit us at:
www.qualcomm.com & www.qualcomm.com/blog
Nothing in these materials is an offer to sell any of the References in this presentation to “Qualcomm” may mean Qualcomm
components or devices referenced herein. Incorporated, Qualcomm Technologies, Inc., and/or other subsidiaries
or business units within the Qualcomm corporate structure, as
©2018 Qualcomm Technologies, Inc. and/or its affiliated
applicable. Qualcomm Incorporated includes Qualcomm’s licensing
companies. All Rights Reserved.
business, QTL, and the vast majority of its patent portfolio. Qualcomm
Qualcomm, Snapdragon, Kryo, Qualcomm Spectra, Adreno Technologies, Inc., a wholly-owned subsidiary of Qualcomm
and Hexagon are trademarks of Qualcomm Incorporated, Incorporated, operates, along with its subsidiaries, substantially all of
registered in the United States and other countries. Qualcomm’s engineering, research and development functions, and
Qualcomm Aqstic is a trademark of Qualcomm Incorporated. substantially all of its product and services businesses, including its
Other products and brand names may be trademarks or semiconductor business, QCT.
registered trademarks of their respective owners.Resources
• Websites
◦ Augmented reality: https://www.qualcomm.com/AR
◦ Virtual reality: https://www.qualcomm.com/VR
◦ Immersive experiences: https://www.qualcomm.com/Immersive
◦ Developers: https://developer.qualcomm.com
◦ Newsletter signup: http://www.qualcomm.com/mobile-computing-newsletter
• Presentations
◦ Virtual reality: https://www.qualcomm.com/documents/making-immersive-virtual-reality-possible-mobile
◦ Immersive experiences: https://www.qualcomm.com/documents/immersive-experiences-presentation
◦ SlideShare: http://www.slideshare.net/qualcommwirelessevolution
• Papers
◦ Virtual reality: https://www.qualcomm.com/documents/whitepaper-making-immersive-virtual-reality-possible-mobile
◦ Immersive experiences: https://www.qualcomm.com/documents/whitepaper-driving-new-era-immersive-experiences-
qualcomm
• Videos:
◦ Immersive experiences webinar: https://www.qualcomm.com/videos/webinar-new-era-immersive-experiences-whats-
next
◦ Virtual reality webinar: https://www.qualcomm.com/videos/webinar-making-immersive-virtual-reality-possible-mobile
49You can also read
