Building the Metaverse - One Standard at a Time Web3D Conference - Khronos Group
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Building the Metaverse
One Standard at a Time
Neil Trevett | Khronos President
NVIDIA Vice President Developer Ecosystem
ntrevett@nvidia.com | @neilt3d
Web3D Conference
Poznań, Poland, June 22nd 2018
© Copyright Khronos™ Group 2018 - Page 1
Khronos Mission
Software
Silicon
Khronos is an International Industry Consortium creating royalty-free, open standards
to enable software to access hardware acceleration for
3D Graphics, Virtual and Augmented Reality, Parallel Computing,
Neural Networks and Vision Processing
© Copyright Khronos™ Group 2018 - Page 2
Looking Back at 2011’s Web3D Keynote
The Goal
The Browser IS the m etaverse
What is the Real-world AR through Web pages not apps
final push Web standards interoperability overcom es
needed? platform fragm entation
What we didn’t foresee in 2011
1. The rise of Machine Learning
2. Re-em ergence of
platform fragm entation
3. Layered ecosystem s and com piler
technology to com bat fragm entation
Trevett Web3D Conference Keynote, Paris June 2011
© Copyright Khronos™ Group 2018 - Page 3
Vital Elements for XR in the Browser
Efficient 3D Cross-platform Sensor
Asset Delivery and UI Handling
Fast, low-latency Compute, Vision and
3D Graphics Inferencing acceleration
All mobile and desktop have GPUs as offload processors
© Copyright Khronos™ Group 2018 - Page 4
Ecosystem = API + File Format
Run-time APIs File Formats
3D Graphics
Heterogenous Compute
(Parallel Processing)
VR and AR
Vision and Inferencing
© Copyright Khronos™ Group 2018 - Page 5
glTF – Cross-Platform 3D Asset Transmission
All glTF spec developm ent
on open GitHub:
https://github.com/KhronosGroup/glTF
O p e n G L T ra n sm issio n F o rm a t glTF 1.0 – Primarily for WebGL glTF 2.0 - Physically Based Rendering
Efficient transmission of 3D Uses GLSL for materials Metallic-Roughness and Specular-Glossiness Materials
scenes and assets Released December 2015 Rendering API independence
Released @ Web3D 2017
© Copyright Khronos™ Group 2018 - Page 6
glTF Momentum 3D Content
Creation Tools 1 5 0 K glT F M o d e ls in C re ativ e C o m m o n s!
Apps and Engines
Discover
S o n y 3 D C re a to r
Create Experience
Oculus
Modo Paint 3D
gltf-vscode Drive Demand
Users
Collada2gltf FBX2glTF Windows Mixed
Reality Home
glTF-validator glTF-asset-generator
© Copyright Khronos™ Group 2018 - Page 7
Consistent glTF Rendering on Multiple Engines
glT F c o n sid e rin g
o p tio n a l, sta n d a rd iz e d
e n v iro n m e n t a n d ligh tin g
gu id e lin e s fo r c o n siste n t
d e fa u lt a p p e a ra n c e
b e tw e e n v ie w e rs
https://github.com/KhronosGroup/glTF-Sample-Models/pull/154
© Copyright Khronos™ Group 2018 - Page 8
glTF Recent Highlights
• Microsoft makes glTF files as usable as JPGs in Windows 10
- Office , Remix 3D, Paint 3D, Simplygon, Mixed Reality Portal and Viewer
• Facebook supports drag and drop for glTF models to your feed
- Driving exporter/tool demand: FBX2glTF, Modo, …
• Sketchfab repository has over 150K glTF models
- Under free, Creative Commons license
• Adobe Dimension is adopting glTF for delivery of 3D marketing assets
- glTF publishing service for interactive marketing materials
• Mozilla integrating glTF into A-FRAME Mesh
- Anchors for glTF objects in AR scenes Compression
• Google Draco Mesh Compression Ratios
- Extension is shipping and being
used by tools and engines
© Copyright Khronos™ Group 2018 - Page 9
glTF Extension and Roadmap Discussions
• glTF is extensible
- Use extensions for specialist functions or testing new functionality
• Some functionality under discussion
- Advanced lighting and shadows glTF Ecosystem m eeting at SIGGRAPH 2018
- Environmental and HDR lighting Let Neil know if you would like an invite!
- Compressed Texture Transmission
with efficient expansion into GPU formats
- Advanced PBR – NVIDIA MDL?
- Subdivision surfaces - Pixar OpenSubdiv?
- Point clouds with Draco compression
- Draco compression of animation data
- Geometry streaming etc. etc.
Any delivery format must balance functionality
versus speed of processing and simplicity of
implementing reliable importers/exporters
https://gltf.sketchfab.com/models/damagedHelmet.zip
© Copyright Khronos™ Group 2018 - Page 10USDZ
At WWDC 2018 Apple announced their own 3D delivery format – USDZ
Pixar USD is a m ovie authoring interchange form at – open sourced in 2016
USDZ packages USD files in a non-com pressed ZIP container
Early days – not clear if Apple will subset USDZ for their platform - no spec yet
USDZ
Com pact Delivery Form at Authoring Interchange Form at
Multi-com pany Governance Delivery subset controlled by Apple?
Open standard for use on any platform Apple platform only?
Widespread open source tools ecosystem ?
Competition is healthy
Awaiting more details to be announced by Apple
Conversion between glTF and USDZ is certainly possible
© Copyright Khronos™ Group 2018 - Page 11Vital Elements for XR in the Browser
Efficient 3D Cross-platform Sensor
Asset Delivery and UI Handling
Fast, low-latency Compute, Vision and
3D Graphics Inferencing acceleration
All mobile and desktop have GPUs as offload processors
© Copyright Khronos™ Group 2018 - Page 12WebGL Evolution
D e sk to p G ra p h ic s
Te xtu re s: N PO T, 3 D , D e p th , A rrays, In t/flo at E x p lic it G ra p h ic s
O b je cts: Q u e ry, Syn c, Sam p le rs G e n e ral Pu rp o se C o m p u te
M o b ile G ra p h ic s Se am le ss C u b e m ap s, In te ge r ve rte x attrib u te s M u lti-th re ad e d R e n d e rin g
Pro gram m ab le M u ltip le R e n d e r Targe ts, In stan ce d re n d e rin g Lo w -late n cy re n d e rin g
V e rte x an d Fragm e n t Tran sfo rm fe e d b ack, U n ifo rm b lo cks
sh ad e rs V e rte x array o b je cts, G LSL ES 3 .0 sh ad e rs
Apple does not ship
OpenGL ES 3.1 -
ending the era of
universal native 3D APIs
2007 2012 2014 2016
OpenGL ES 2.0 OpenGL ES 3.0 OpenGL ES 3.1
Compute Shaders
4 years
2011 5 years
WebGL 1.0 March 2017
WebGL 2.0
Pervasive OpenGL ES 2.0 Conformance Testing is vital for
OpenGL and OpenGL ES ships on
every desktop and mobile OS Significant ongoing work on multiple Cross-Platform Reliability
3D on the Web is enabled! WebGL 2.0 conformance tests are very thorough
performance enhancements and getting all 10x more tests than WebGL 1.0 tests
browsers to pass all the complete test suite
© Copyright Khronos™ Group 2018 - Page 13WebGL Momentum – WebGL 2.0 is Here!
93.26% Globally
http://caniuse.com/#feat=webgl
62.85% Globally
WebGL 2.0 brings Desktop-class
graphics to the Web
The time to create a new class of
Web-based 3D Apps is now!
© Copyright Khronos™ Group 2018 - Page 14Pervasive Vulkan
Major GPU Companies supporting Vulkan for Desktop and Mobile Platforms
http://vulkan.gpuinfo.org/
Platforms
Consoles
Desktop Mobile Media Players
( A n d r o id 7 . 0 + )
Virtual Reality
Cloud Services Embedded
Game Engines
C ro te a m
S e rio u s E n g in e
© Copyright Khronos™ Group 2018 - Page 15Vulkan and New Generation GPU APIs
Non-proprietary, royalty-free open standard ‘By the industry for the industry’
Portable across multiple platforms - desktop and mobile
Modern architecture | Low overhead| Multi-thread friendly
EXPLICIT GPU access for EFFICIENT, LOW-LATENCY,
PREDICTABLE performance
Vulkan Porting Tools
Vulkan is available on Android 7.0+
© Copyright Khronos™ Group 2018 - Page 16Bringing Vulkan Apps to Apple Platforms
Dota 2 running on Mac up to Applications
50% faster than native OpenGL
Open source SDK to build,
Efficiently map a subset of Vulkan over Metal run, and debug applications
Very little functionality not supported today: on macOS including
validation layer support
T ria n gle fa n s, S e p a ra te ste n c il re fe re n c e m a sk s
V u lk a n E v e n ts, A llo c a tio n c a llb a c k s
Vulkan
S o m e te x tu re -sp e c ific sw iz z le s macOS SDK
SPIRV-Cross macOS / iOS Beta release – but working to pass
Convert SPIR-V shaders to Run-time all applicable conformance tests
platform source formats Maps Vulkan to Metal Previously a paid product
MoltenVK for macOS and iOS Now released into OPEN SOURCE
For macOS 10.11, iOS 9.0 and up Completely free to use - no fees
or royalties – including
commercial applications
© Copyright Khronos™ Group 2018 - Page 17Valve - Vulkan Dota 2 on macOS
Vulkan delivering up to
50% perform ance increase
over native OpenGL
© Copyright Khronos™ Group 2018 - Page 18SPIR-V Ecosystem
Khronos defined cross-API IR
•
GLSL HLSL
• Native graphics and parallel com pute
T h ird p a rty k e rn e l a n d • Stable specification to com plem ent
sh a d e r la n gu a ge s glslang DXC LLVM Open Source Project
MSL
HLSL
SPIRV-Cross OpenCL C OpenCL C++ SYCL
Front-end Front-end Front-end
GLSL
SPIR-V (Dis)Assembler
SPIR-V Validator LLVM
LLVM to SPIR-V
Bi-directional K h ro n o s c o o p e ra tin g w ith
Translators
SPIRV-opt | SPIRV-remap C la n g/L L V M C o m m u n ity
Optimization K h ro n o s-h o ste d
Tools O p e n S o u rc e P ro je c ts
https://github.com/KhronosGroup/SPIRV-Tools
IH V D riv e r
R u n tim e s
© Copyright Khronos™ Group 2018 - Page 19Bringing OpenCL Compute to Vulkan
• Experimental Clspv Compiler from Google, Adobe and Codeplay
- Compiles OpenCL C to Vulkan’s SPIR-V execution environment
Successfully tested on over 200K lines of Adobe OpenCL C production code
- Open source - tracks top-of-tree LLVM and clang, not a fork
Layering and cross-
compilation is making Vulkan OpenCL C OpenCL
a Universally Portable API for Source Host Code
Graphics AND Compute
Prototype open Clspv Run-time Possible
source project Compiler API Translator future
h ttp s://gith u b .c o m /go o gle /c lsp v
project –
if interest
Runtime
Increasing deployment options for
OpenCL kernel developers
e.g. Vulkan is a supported API on Android
© Copyright Khronos™ Group 2018 - Page 20WebGL Future Don’t panic!
Deprecation means the APIs will
be removed in the future - still GPUWeb
available right now Community Group
WebGL 2.0 could be layered
over Metal Makes sense that work on nexgen
Web stack takes place at W3C
But currently considering
designing a fourth native GPU API
– without significant input from
the GPU hardware community L
Could cause further industry
fragmentation and be a barrier to
adoption L L
No universally available 3D API
since Apple did not ship
Watching with interest to see if
OpenGL ES 3.1 GPUWeb becomes a formal
working group
That situation just got worse at
Apple WWDC June 2018 – OpenGL May use SPIR-V for shader code
and OpenGL ES deprecated in
Dean is Apple’s WebGL lead
favor of Apple’s Metal API
© Copyright Khronos™ Group 2018 - Page 21Vital Elements for XR in the Browser
Efficient 3D Cross-platform Sensor
Asset Delivery and UI Handling
Fast, low-latency Compute, Vision and
3D Graphics Inferencing acceleration
All mobile and desktop have GPUs as offload processors
© Copyright Khronos™ Group 2018 - Page 22Neural Network Workflow
Applications
Using Embedded
Vision and Inferencing
WinML
Framework
Specific
Formats
NeuralNet
Neural NetTraining
Training
Neural
Neural Net Training
Frameworks
Net Training Trained Vision and Neural Net
Frameworks
Frameworks
Compilation and
A u t h o r in g in t e r c h a n g e Frameworks Networks Optimization
Inferencing Runtimes
E m b e d d e d D e p lo y m e n t
CPU
Desktop and GPU
CPU
CPU GPU
Cloud Hardware FPGA DSP
Diverse Inferencing
Custom
cuDNN MIOpen clDNN TPU Acceleration Hardware Hardware
Training = Desktop / Cloud Deployment on Embedded Devices
© Copyright Khronos™ Group 2018 - Page 23NNVM - Open Compiler for AI Inferencing
P a u l G . A lle n S c h o o l o f C o m p u t e r S c ie n c e & E n g in e e r in g , U n iv e r s it y o f W a s h in g t o n
h ttp :/ / w w w .tv m la n g.org/ 2 0 1 7 / 0 8 / 1 7 / tv m -re le a se -a n n ou n c e m e n t.h tm l
1 .Im p o rt T ra in e d
N e tw o rk D e sc rip tio n
2 . G ra p h -le v e l
O p tim iz a tio n s
3 . D e c o m p o se in to
p rim itiv e in stru c tio n s
h ttp s:/ / gith u b .c om / d m lc / n n v m / b lob / m a ste r/ tu toria ls/ from _ m xn e t_ to_ w e b gl.p y
4 . E m it p ro gra m s
e x e c u ta b le b y ru n -tim e s
S P IR -V IR fo r p a ra lle l a c c e le ra to rs L L V M IR fo r C P U s
B a c k e n d in d e v e lo p m e n t
TensorFlow over WebGL demo:
TensorFlow.js powered implementation of the tSNE algorithm for high-dimensional data analysis
https://nicola17.github.io/tfjs-tsne-demo/
© Copyright Khronos™ Group 2018 - Page 24Platform Neural Network Stacks
Consistent Fundam ental Steps
1 . Im p o rt tra in e d N N m o d e l file
2 . B u ild o p tim iz e d v e rsio n o f gra p h
3 . A c c e le ra te o n G P U o r o th e r p ro c e sso r
u sin g a v a ila b le lo w -le v e l A P I
Core ML Model
Microsoft Windows Google Android Apple MacOS and iOS
Windows Machine Learning (WinML) Neural Network API (NNAPI) CoreML
h t t p s : / / d o c s . m ic r o s o f t . c o m / e n - u s / w in d o w s / u w p / m a c h in e - le a r n in g / h t t p s : / / d e v e lo p e r . a n d r o id . c o m / n d k / g u id e s / n e u r a ln e t w o r k s / h t t p s : / / d e v e lo p e r . a p p le . c o m / d o c u m e n t a t io n / c o r e m l
© Copyright Khronos™ Group 2018 - Page 25NNEF Ecosystem
NNEF = Neural Network
Live
T e n so rF lo w Exchange Format
a n d C a ffe
V1.0 Provisional Released for
Imminent E x p o rte rs
industry feedback before
finalization
T e n so rF lo w
S y n ta x
a n d C a ffe 2
P a rse r/
Im p o rte r /
E x p o rte rs Files V a lid a to r
Comparing Neural Network
Exchange Industry Initiatives
G o o gle OpenVX
NNAPI In ge stio n &
C o n v e rto r E x e c u tio n
D e fin e d Sp e c ific a tio n O p e n So u rc e P ro je c t
N N E F o p e n so u rc e p ro je c ts h o ste d o n
K h ro n o s N N E F G itH u b re p o sito ry
Sta b ility fo r h a rd w a re d e p lo ym e n t So ftw a re sta c k fle xib ility
A p a c h e 2 .0 lic e n se
https://github.com/KhronosGroup/NNEF-Tools
M u lti-c o m p a n y G o ve rn a n c e In itia te d b y F a c e b o o k
F le xib le P re c isio n 3 2 -b it F lo a tin g P o in t o n ly
F la t a n d C o m p o u n d O p s F la t O p s O n ly
© Copyright Khronos™ Group 2018 - Page 26Vital Elements for XR in the Browser
Efficient 3D Cross-platform Sensor
Asset Delivery and UI Handling
Fast, low-latency Compute, Vision and
3D Graphics Inferencing acceleration
All mobile and desktop have GPUs as offload processors
© Copyright Khronos™ Group 2018 - Page 27OpenXR – Solving VR Fragmentation
VR VR VR VR VR VR VR VR
App App App App App App App App
1 2 3 4 1 2 3 4
Proprietary
Engine Proprietary
Engine
Application Interface
Device Layer
VR VR VR VR VR VR VR VR VR VR
Device Device Device Device Device Device Device Device Device Device
1 2 3 4 5 1 2 3 4 5
Before OpenXR After OpenXR
VR Market Wide interoperabilityof
Fragmentation VR apps and devices
© Copyright Khronos™ Group 2018 - Page 28Companies Publicly Supporting OpenXR
OpenXR is a collaborative design
1) For cross-platform XR portability – VR in V1.0, then add AR
2) Integrating many lessons from proprietary ‘first-generation’ API designs
© Copyright Khronos™ Group 2018 - Page 29Input and Haptics
Controll
erCorp
• Input uses abstracted Input Actions
- E.g. “Move,” “Jump,” “Teleport” /user/hand/left/input/trigger/click
(/devices/ControllerCorp/fancy_controller/
• Many advantages input/trigger/click)
- Existing content can easily use
new devices OpenXR Runtime
- Mix-and-match multiple input .../input/button_a/click Explode
.../input/trigger/click Teleport
sources to create a unified UI .../input/grip/value SpawnKittens
- Easy optional feature support (e.g.
…
eye and body tracking)
- Future-proofing for innovation in
input devices and form factors XrAction: “Teleport”
© Copyright Khronos™ Group 2018 - Page 30OpenXR Viewport Configurations
• Applications can:
- Query for runtime supported Viewport Configurations
- Applications can then set the Viewport Configurations that they plan to use
- Select and change their active configuration over the lifetime of the session
Camera Passthrough AR Stereoscopic VR / AR Projection CAVE
Photo Credit: Dave Pape
O n e V ie w p o rt T w o V ie w p o rts (o n e p e r e ye ) T w e lve V ie w p o rts (six p e r e ye )
/viewport_configuration/ar_mono/magic_window /viewport_configuration/vr/hmd /viewport_configuration/vr_cube/cave_vr
© Copyright Khronos™ Group 2018 - Page 31OpenXR Development Progress
Call for Participation / Exploratory Group Formation
Fall F2F, October 2016: Korea
Statement of Work / Working Group Formation
Winter F2F, January 2017: Vancouver
Specification Work
Spring F2F, April 2017: Amsterdam
M u c h m o re d e ta ile d sp e c ific a tio n o v e rv ie w
Specification Work
Interim F2F, July 2017: Washington a n d G D C se ssio n v id e o s:
https://www.khronos.org/developers/library/2018-gdc
Defining the MVP
Fall F2F, September 2017: Chicago
Resolving Implementation Blockers
Winterim F2F, November 2017: Washington
Raising Implementation Issues
Winter F2F, January 2018: Taipei
First Public Information!
Implementations Underway!
Present Day GDC, March 2018
Spec Finalization – incorporating
Coming Soon Provisional Release
implementation experience
Conformance Testing and Implementation
Ratification and Release
© Copyright Khronos™ Group 2018 - Page 32Mobile Augmented Reality Libraries
Encapsulated Vision-based
Functionality
Also leveraging motion sensors
ARKit
Pose Tracking Yes Yes 3rd party AR Libraries
typically use
Plane Detection and Tracking Yes Yes
ARKit/ARCore if
Image Recognition and Tracking Yes Yes available or
implement own
Ambient light level and temperature Yes Yes tracking if not
Link to Neural Net-based Object Detection Yes Yes
Access to Point Cloud Yes Yes
Camera Intrinsics Yes Yes
V1.0 is VR focused.
Multi-user and persistent cloud-based anchors Yes Yes Subsequent
Face tracking (iPhone X) Yes No versions will
extend to cross-
Light probe (using a tracked face) Yes No platform AR
OS Availability iOS Android and iOS
© Copyright Khronos™ Group 2018 - Page 33Layered XR Ecosystems with GPU Acceleration
Native XR Apps Web XR Apps
Vision and
WebXR Inferencing
Vision and 3D Engines libraries and
Inferencing run-times
libraries and System-
run-times exposed AR
Close ongoing 3D Engines
Capabilities collaboration between
WebXR and OpenXR
Khronos providing the
foundation for 3D and
XR in the Web
© Copyright Khronos™ Group 2018 - Page 34Looking Back at 2011’s Web3D Keynote
WebXR
With WebXR and OpenXR, portable,
accelerated XR in the Web is NOT
going to take another 7 years J
It will be here much sooner!
Trevett Web3D Conference Keynote, Paris June 2011
© Copyright Khronos™ Group 2018 - Page 35Please Get Involved!
• These slides and information on all Khronos standards at:
- www.khronos.org
• Khronos is driving to new levels of community engagement
- Specifications and resources increasingly open for industry input and feedback
• Any company or organization is welcome to join Khronos
- For a voice and a vote in any of these standards
• If joining is not possible – ask about a Khronos Advisory Panel
- Free of charge – enables design reviews, requirements and contributions
• Khronos is always open to liaison projects with the 3D Web community
- The industry benefits from everyone working more closely together
• Neil Trevett
- ntrevett@nvidia.com
- @neilt3d
© Copyright Khronos™ Group 2018 - Page 36You can also read
