THE FUTURE OF IN-CABIN EXPERIENCE - Dr. Petronel Bigioi CTO, Products January 2021 - Xilinx
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
THE FUTURE OF
IN-CABIN EXPERIENCE
Dr. Petronel Bigioi
CTO, Products
January 2021
© 2021 Xperi. Confidential. Do not distribute.
XPERI ▪ COMPANY OVERVIEW
40+ OFFICES WORLDWIDE
headquartered in San Jose, CA
$1.5B+ MARKET CAP
public company, trading under XPER
2,000+ EMPLOYEES
1200+ engineers
10,000+ PATENT ASSETS
strong focus on innovation
100B+ DEVICES WORLDWIDE
DTS, HD Radio, Invensas, TiVo
2 © 2021 Xperi. Confidential. Do not distribute.
IN-CABIN
MONITORING
… what is it and why it's important
© 2021 Xperi. Confidential. Do not distribute.
FRONT PASSENGER BACK LEFT PASSENGER PET DETECTED DRIVER
DOG
NAME: RACHEL NAME: BOB
NAME: DAVID
EXPRESSION: SMILING EXPRESSION: SMILING
EXPRESSION: NEUTRAL
BODY POSE: NORMAL BODY POSE: NORMAL
ALERTNESS LEVEL: ATTENTIVE
ACTIVITY: TEXTING SEATBELT: ON
DROWSINESS STATUS: NOT DROWSY
SEATBELT: ON HEART RATE: 64 BPM
ACTIVITY: DRIVING
HEART RATE: 70 BPM RESPIRATION: NORMAL
OBSTRUCTIONS: NO
RESPIRATION: NORMAL SKIN TONE: NORMAL
SEATBELT: ON
SKIN TONE: NORMAL
BODY POSE: HANDS ON WHEEL
HEART RATE: 68 BPM
RESPIRATION: NORMAL
SKIN TONE: NORMAL
UNDER THE INFLUENCE: NO
BACK RIGHT PASSENGER
CHILD DETECTED
CHILD SEAT: PRESENT
HEART RATE: 76 BPM
RESPIRATION: NORMAL
SKIN TONE: NORMAL
OBJECT DETECTED
PHONE
HOLISTIC IN-CABIN MONITORING
4 © 2021 Xperi. Confidential. Do not distribute.
DMS TRANSITION TO ICM ▪ NEW EXPERIENCES ENABLER
Full
Automation
High ▪ Continuous health & safety monitoring
Automation
▪ Personalized experience
▪ Authentication
Conditional
Automation ▪ Human machine interface
▪ Social and work Interaction
▪ Next Gen Entertainment
Partial
Automation 2021 DMS transition to ICM
Driver
Assistance
No 2018 Must have DMS 2030
Automation
0 1 2 3 4 5
2000 2005 2010 2015 2020 2025 2030 2035 2040
Human Driver Automated Driving System
Monitors Driving Environment Monitors Driving Environment
5 © 2021 Xperi. Confidential. Do not distribute.
ICM ▪ POSSIBILITIES AND COMBINATIONS
SKIN HEALTH
BEAUTY THERMAL
OCCLUSIONS HDR & LTM
RGB
OCCLUSIONS
BODY POSE
NIR
HEAD POSE MICROGESTURES EIS
GAZE THERMAL+NIR
HEAD POSE
GAZE
ACTIVITY GESTURES RGB & NIR
EYE ANALYTICS ACTIVITY GESTURES
EVT
FACE ID BREATHING
IRIS ID PRESENCE
FACE ID
PRESENCE
FACE CLASSIF BODY SIZE
DMS FACE CLASSIF BODY SIZE
EXPRESSIONS BODY CLASSIF
EXPRESSIONS ENVIRONMENT BODY CLASSIF
OBJ DET ICM
BACKGROUND
OBJ DET
BIOMETRICS
6 © 2021 Xperi. Confidential. Do not distribute.
ICM ▪ QUESTIONS
MONITORING FEATURES SOUND
PD, HP, Gaze (what precision)? How many speakers? What
Identity (how strong)? Fatigue (how configuration? What types?
many levels?), Biometrics (what
type)? Health monitoring (what)?
PROCESSING
EXPERIENCE FEATURES
Centralized vs Decentralized?
HMI, Video conferencing, TOPS? Memory? ISPs?
Beautification, Background
replacement, personalized
sweet spot sound;
SAFETY
ENTERTAINMENT FEAT.
ASIL Levels? Redundancy?
Movies (back seat or main For which systems/features?
windshield), active noise
cancellation
SENSORS
IMAGING SYSTEM
What sensors? How
How many cameras? What type? to fuse? What for?
What FOV? What resolution?
7 © 2021 Xperi. Confidential. Do not distribute.
TOTAL FELXIBILITY ▪ IS REQUIRED
In-cabin monitoring, and personalized experience use-cases are diverse,
and need to be powered by ultimate flexible platform (XILINX) with built in domain expertise (XPERI)
8 © 2021 Xperi. Confidential. Do not distribute.
OCCUPANCY
MONITORING
The first step to ICM.
Solution presentation.
© 2021 Xperi. Confidential. Do not distribute.
OMS ▪ IS DESIGNED FOR
OCCUPANTS SAFETY
37,133 47% 14,955 2,549
lost lives wore no estimated saved lives that could
in accidents1 seat belt1 lives by seat belt1 have been saved1
CONTINOUS MONITORING FOR HMI AND HEALTH
Understand where and what occupants do at any moment in time
BETER EXPERIENCE
Identify who the occupants are and tailor comfort settings such as
seat position and entertainment content.
10 US figures for 2017, source: 1NHTSA. https://www.nhtsa.gov/risky-driving/seat-belts ©
© 2021
2021 Xperi.
Xperi. Confidential.
Confidential.Do
Do not
notdistribute.
distribute.OMS ▪ SOLUTION WORKFLOW
Adjust
Parameters
Tailored
ISP, Env.
DCE & Analytics
LTM
OMS Camera Hand
(RGB/RGB-IR) Face Detection Body Detection Detection
(160-220 degree FOV)
OMS Face ROI Body ROI
Xilinx specific Hand ROI
Reliability
Driver A
Male
Caucasian
Happy
Face Direction & Face Feature Emotions & Face Hand
Continuous Driver ID Classification Body Pose
Position Points Skeleton
Direction Face Features Emotions, Gender
Identification Body Points Finger Points
Ethnicity, Age
11 © 2021 Xperi. Confidential. Do not distribute.12 © 2021 Xperi. Confidential. Do not distribute.
IN-CABIN
MONITORING
FUTURE
Vision
© 2021 Xperi. Confidential. Do not distribute.ICM ▪ LONG TERM VISION
NO AUTOMATION HANDS ON HANDS OFF EYES OFF MIND OFF FULL AUTOMATION
✓ DMS FOR DROWSINESS OR ✓ DMS FOR SEMI-AUTO ✓ IN-CABIN MONITORING FOR
INATTENTION DRIVING SAFETY AND PERSONALIZED
✓ HD-RADIO ✓ DTS AUTOSTAGE EXPERIENCE
DTS AUTOSTAGE HD RADIO IMAX ENHANCED
WE BELIEVE THAT
ALL THE VEHICLES OF TOMORROW
WILL HAVE IN-CABIN SENSING, IMAGING AND
PERSONALIZED ENTERTAINMENT AT THEIR CORE
14 © 2021 Xperi. Confidential. Do not distribute.ENABLING
TECHNOLOGIES
AND INFRASTRUCTURE
We have many technologies & infrastructure
to significantly contribute to the future of
automotive in cabin experience
© 2021 Xperi. Confidential. Do not distribute.ENABLING ▪ TECHNOLOGIES
MULTI-SPECTRAL SENSING IMAGE ENHANCEMENT BIOMETRICS
Visible + NIR + LWIR Analytics EIS with RS correction, LTM, Production ready biometrics all the way from
portrait enhancement, De-fogger 2D FR on visible or NIR, 3D-FR with
and de-warping iTOF/SL/Stereo to IRIS recognition
DEPTH SENSING TIME EVENT Dual Camera Sensing
Neural network based Analytics based on time event sensors to WFOV RGB-IR + AF PT NFOV NIR tightly
stereo depth engine capture information never seen before connected to provide better features for ICM
(DMS + OMS)
16 © 2021 Xperi. Confidential. Do not distribute.TESTING ▪ INFRASTRUCTURE
IMAGE DATABASE MANAGEMENT
Storage and versioning of video data and associated
metadata (annotations, acquisition information)
TESTING FRAMEWORK
Fully automated testing of algorithm KPI’s
Tests execution distributed across the Xperi
intranet. Automatic generation of reports.
TEST IMAGE DATABASE HIGHLIGHTS
Over 50 million annotated frames, keeps increasing
each year. Combination of data acquired in the wild
and in controlled environment. Multiple light
conditions, camera positions, thermal noise, etc.
17 © 2021 Xperi. Confidential. Do not distribute.INFRASTRUCTURE ▪ COMPUTER VISION BASED
ACQUISITION – 3D IMAGES AUTO MARKING – 3D MODEL
MESH
TEXTURE (vis)
TEXTURE (nir)
3D CUSTOM
SETUP (VISIBLE «& NIR») ▪ PHOTOGRAMMETRY ▪ 3D MODEL MODEL BACKGROUNDS
GENERATING MARKED 2D RENDERS
MARKED 2D RENDERING
SAMPLES LENS/SENSOR/ISP MODEL
3D CUSTOM
ACCESSORIES LIGHTING
18 © 2021 Xperi. Confidential. Do not distribute.EXAMPLES ▪ OF SYNTHETIC DATA GENERATION
DMS VIEWPOINT OMS ROOFTOP VIEWPOINT
Example of synthetic data generation for specific camera lens type and positions
Enables very fast algorithm training, validation and testing & last mile optimizations (for target camera)
19 ©
© 2021 Xperi. Confidential.
2021 Xperi. Confidential. Do
Do not
not distribute.
distribute.IN CAR ▪ ACQUISITION SETUP
In-car driving scenarios strong value comes from
ensuring ground truth real testing data is available.
20 ©
© 2021 Xperi. Confidential.
2021 Xperi. Confidential. Do
Do not
not distribute.
distribute.MORE ▪ INFRASTRUCTURE
21 ©
© 2021 Xperi. Confidential.
2021 Xperi. Confidential. Do
Do not
not distribute.
distribute.CONCLUSIONS
XPERI is your right partner for smart
Cars & best in cabin experience
© 2021 Xperi. Confidential. Do not distribute.FUTURE ▪ HIGHLIGHTS
▪ Intelligent sensing is required by next
generation In-cabin experience, but the exact
method is not yet set-in stone …
▪ Flexible and reusable architecture for many
use-cases where intelligent sensors are
required is a must …
▪ XPERI has proven & accepted algorithms
by two major car OEMs as ‘best in the world’;
all thanks to its novel data preparation,
testing and validation methods as well as
delivery mechanisms (standard SoCs and/or
FPGAs);
23 © 2021 Xperi. Confidential. Do not distribute.FUTURE ▪ HIGHLIGHTS
▪ XPERI provides the intelligence forward to
reach required power efficiency, latency, and
performance for automotive in cabin sensing; Future
in-cabin sensing and entertainment will bring multiple
sensors and cameras opportunities in the car with
multiple challenges to SoC providers related to signal
processing, processing power, power consumption and
latency
▪ Dedicated and proven hard-cores coupled with
specific proven algorithms for ICM from XPERI can
enhance XILINX future parts flexibility & adoption
▪ Xilinx can be the driving force and trusted partner for
XPERI’s innovative solutions in automotive
24 © 2021 Xperi. Confidential. Do not distribute.TRUSTED PARTNER ▪ FOR ICM AND ENTERTAINMENT
SIGNAL (PRE)PROCESSING AUTOMOTIVE PROVEN
On the fly input signal processing for Shipping DMS, HD Radio &
optimum analytics Connected radio technologies
ANALYTICS,
ENHANCEMENT & R&D INFRASTRUCTURE
STRONG BIOMETRICS
Testing and computer vision
Core proven facial, people infrastructure for effective
analytics, recognition, scene training, validation and testing
understanding and enhancement our algorithms
technologies
CORE ENTERTAINMENT
HARDWARE IP CORES
DTS & IMAX ENHANCED for
Dedicated IP cores for ultimate living room
extreme performance and experience on wheels
programmable inference IP
Cores for flexibility
DBI® DIRECT BOND PACKAGING & MINIATURI.
INTERCONNECT
High performance, high-
3D ASIC Enabler for massive ultra- integration low-cost intelligent
low power AI on the Edge camera modules & sensors
25 © 2021 Xperi. Confidential. Do not distribute.THANK YOU
© 2021 Xperi. Confidential. Do not distribute.You can also read
