Bringing Computer Vision to the Court - Adrià Arbués-Sangüesa Universitat Pompeu Fabra Invited talk at CVSports 2021
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Bringing Computer Vision
to the Court
Adrià Arbués-Sangüesa
Universitat Pompeu Fabra
Invited talk at CVSports 2021
Brief CV (Curriculum Vitae, not Computer Vision)
• PhD Fellow at Universitat Pompeu Fabra (Barcelona), defense in
September 2021 (hopefully!).
• Academical contributions: tracking & body-orientation & applications
@arbues6 1
Brief CV (Curriculum Vitae, not Computer Vision)
• PhD Fellow at Universitat Pompeu Fabra (Barcelona), defense in
September 2021 (hopefully!).
• Academical contributions: tracking & body-orientation & applications
@arbues6 2
Brief CV (Curriculum Vitae, not Computer Vision)
• PhD Fellow at Universitat Pompeu Fabra (Barcelona), defense in
September 2021 (hopefully!).
• Academical contributions: tracking & body-orientation & applications
[1] Single-Camera Basketball Tracker through Pose and Semantic Feature Fusion
[2] Multi-Person Tracking by Multi-Scale Detection in Basketball Scenarios
[3] Always Look on the Bright Side of the Field: Merging Pose and
Contextual Data to Estimate Orientation of Soccer Players
[4] Using Player's Body-Orientation to Model Pass Feasibility in Soccer
+ paper extension under review!
[5] Learning Football Body-Orientation as a Matter of Classification
Research team: Adrián Martín (postdoc), Coloma Ballester and Gloria Haro (supervisors)
@arbues6 3
Brief CV (Curriculum Vitae, not Computer Vision)
• PhD Fellow at Universitat Pompeu Fabra (Barcelona), defense in
September 2021 (hopefully!).
• Academical contributions: tracking & body-orientation & applications
• When models are training… I am balling!
@arbues6 4
Brief CV (Curriculum Vitae, not Computer Vision)
• PhD Fellow at Universitat Pompeu Fabra (Barcelona), defense in
September 2021 (hopefully!).
• Academical contributions: tracking & body-orientation & applications
• When models are training… I am balling coaching!
C.B. Cornellà (1st team - 3rd division) F.C. Barcelona (u15)
@arbues6 5
Brief CV (Curriculum Vitae, not Computer Vision)
• PhD Fellow at Universitat Pompeu Fabra (Barcelona), defense in
September 2021 (hopefully!).
• Academical contributions: tracking & body-orientation & applications
• When models are training… I am coaching & disseminating!
ImproveSports Stats Course Beyond the 4 Factors
@arbues6 6
In this talk…
Da Da Mo Co Im
ta ta d mmu plem
Ga Pro els
the ce &M nic en
rin ssi e at tat
g ng trics io n ion
7
But first… Why?
Build a competitive advantage
through analytics
Da Da Mo Co Im
ta ta d mmu plem
Ga Pro els
the ce &M nic en
rin ssi e at tat
g ng trics io n ion
7
Competitive Advantage
GM’s / Board of directors
Hire the best possible
players / coaches
Shared Goal:
Coaching Staff Winning as many
Build team chemistry games as possible
Design a playbook
Exhaustive scouting
Players
Stay healthy
Work hard & listen
Data-driven processes 8Competitive Advantage
GM’s / Board of directors
Hire the best possible
players / coaches
Coaching Staff
Design a playbook
Exhaustive scouting
How? Endless hours of film sessions.
Automated tasks through data-driven processes.
9In this talk…
Da Da Mo Co Im
ta ta d mmu plem
Ga Pro els
the ce &M nic en
rin ssi e at tat
g ng trics io n ion
10What kind of stats do we have?
Da Da Mo Co Im
ta ta d mmu plem
Ga Pro els
the ce &M nic en
rin ssi e at tat
g ng trics io n ion
11Basketball analytics: coarse-to-fine approach
Who produce data?
Source: Basketball England
12Basketball analytics: box score
March 28th 1990
Chicago Bulls 117 - Cleveland Cavaliers 113
e s s A A b b
t t M M A e e T L V
i n u
Po
in G
2F
G G
3F
G TM T R R S ST
TO PF
M 2F 3F F F O D A
Michael Jordan 50 69 21 31 2 6 21 23 7 11 6 4 2 5
Scottie Pippen 41 7 3 10 0 0 1 2 2 6 7 5 8 5
Horace Grant 40 16 7 14 0 0 2 4 2 3 4 1 2 4
Bill Cartwright 39 9 3 7 0 0 3 4 1 4 1 0 3 5
John Paxon 23 2 1 4 0 0 0 0 0 0 1 0 2 6
B.J. Armstrong 29 6 2 7 0 0 2 2 0 1 3 2 1 5
Stacey King 17 1 0 4 0 0 1 2 1 0 1 2 0 2
Will Perdue 12 0 0 0 0 0 0 0 0 4 0 0 0 3
Charles Davis 11 5 1 2 1 1 0 0 1 1 1 0 1 2
Clifford Lett 3 2 1 1 0 0 0 0 0 0 0 0 0 2
TEAM 265 117 39 80 3 7 30 37 14 30 24 14 19 39
13Basketball analytics: box score
March 28th 1990
Chicago Bulls 117 - Cleveland Cavaliers 113
Other data sources have to be exploited!
14Basketball analytics: play-by-play
Temporal domain
Chicago Bulls End of Game || 117-113 Cleveland Cavaliers
M. Jordan - 2FG made 117-113 (Ov1 51:32)
S. Pippen - AST (7)
End of Reg. || 105-105
S. King - FT missed 40-37 (2Q - 12:15) C. Ehlo - DReb (2)
S. King - FT made
S. King - Drawn Foul 39-37 (2Q - 12:15) L. Nance - Committed Foul
H. Grant - OUT 27-26 (2Q - 12:15)
S. King - IN
End of Q1 || 27-26
M. Jordan - 2FG made 2-0 (1Q - 00:23)
S. Pippen - AST (1)
M. Jordan - IN 0-0 (1Q - 00:00)
Game Tip-off (1Q - 00:00) 15Basketball analytics: shot charts
Spatial domain
Single Game Whole Career
69 points - (21/31 FG2, 2/6 FG3) Michael Jordan 30.1 points - (10.9/21.2 FG2, 0.5/1.7 FG3)
Accuracy (percentage)
Below Above
Made shot Missed shot Average Average
16Basketball analytics: coarse-to-fine approach
Box score + play-by-play + shot charts… Is it enough?
[Utah] Cross-screen between Hornacek and Malone
Malone - turnover
[Chicago] No defensive switch Jordan - steal
[Utah] Low-post play for Malone Jordan - 2 FG made
[Chicago] Double team from weak side (baseline)
Jordan steals the ball — Malone loses it
[Chicago] Jordan calls an isolation play
[Chicago] Jordan dribbles to his right, crosses over, and scores
a 2-point-open shot pull-up step-back with 5 seconds left.
[Utah] Byron Russell was the assigned defender, and could not
handle Jordan’s crossover.
17Basketball analytics: coarse-to-fine approach
Box score + play-by-play + shot charts… Is it enough, Zeljko?
We are missing:
- Types of shots (playtype)
- Types of shots (difficulty)
- Team strategy
- Defensive metrics
- Defensive assignments
Source: GettyImages
Solution? Time-consuming video-analysis shifts, or… Tracking!
18Where does Computer Vision fall in this context?
Da Da Mo Co Im
ta ta d mmu plem
Ga Pro els
the ce &M nic en
rin ssi e at tat
g ng trics io n ion
19Computer Vision in Sports - 2D Tracking
Source: STATS
20Computer Vision in Sports - 2D Tracking
Challenge - ball-screen play
1
1
3 3
2 4
2 5 4
5
21Computer Vision in Sports - 2D Tracking
Challenge - ball-screen play
1
1
3 3
2 4
2 5 4
5
21Computer Vision in Sports - 2D Tracking
Challenge - ball-screen play
1
1 5
5
3 3
2 4
2 4
21Computer Vision in Sports - 2D Tracking
Challenge - ball-screen play
1 1 1
1 5 1 1 5
5
5 5 5
Show / Hedge Trap Switch
Decision-making process: defense
22Computer Vision in Sports - 2D Tracking
Challenge - ball-screen play
1 1 1
5
5
5 5 5
5
Split Reject Snake
Decision-making process: ball handler
23Computer Vision in Sports - 2D Tracking
Challenge - ball-screen play
1 1 1
5
5 5
2
Roll / Short Roll Pop Spanish PnR
Decision-making process: screener
24Computer Vision in Sports - 2D Tracking
Challenge - ball-screen play
At least MOTA > 0.95 is required!
25Computer Vision in Sports - 2D Tracking
• Complex set-ups, or… Provide automatic CV-based tracking data
through creative low-cost solutions!
Multimodal and multiview distillation for real-time player detection on a football field
[Cioppa, Deliège, et al.]
26Computer Vision in Sports - 2D Tracking
• Complex set-ups, or… Provide automatic CV-based tracking data
through creative low-cost solutions!
Self-Supervised Small Soccer Player Detection and Tracking [Hurault et al.]
27Computer Vision in Sports - 2D Tracking
28Computer Vision in Sports - 2D Tracking
GM
Coaching Staff
Data Science Department 28Computer Vision in Sports - 2D Tracking
Source: Seth Partnow 29Computer Vision in Sports - 2D Tracking
NBA scientific departments are not CV-based (still)
Goal: create novel CV metrics a priori
Source: Seth Partnow 29What metrics are we supposed to extract?
Da Da Mo Co Im
ta ta d mmu plem
Ga Pro els
the ce &M nic en
rin ssi e at tat
g ng trics io n ion
30Metrics: Guidelines (not only CV)
[Credits - Justin Jacobs]
Model implicit features that cannot be estimated with the physics of
the game. Learn from the output - valuable insights.
Actionable
Prediction Interpretability
Intelligence
Sacrifice accuracy for interpretability.
31Metrics: on top of CV - Predict
• Train models from tracking data to obtain meaningful insights
EPV Models [Cervone et al. 2014 - basketball, Fernández et al. - soccer 2019]
32Metrics: on top of CV - Predict
• Train models from tracking data to obtain meaningful insights
Offense
Real
Defense
Ghosting
Defense
Bhostgusters - What if scenarios / Ghosting [Seidl et al., 2018]
33Metrics: on top of CV - Enrich & Analyze
• Add more layers the 2D information to re-define metrics:
- A player is at (X, Y) and he/she is…
34Metrics: on top of CV - Enrich & Analyze
• Add more layers the 2D information to re-define metrics:
- A player is at (X, Y) and he/she is jumping / passing / screening…
Proper definition of
open / contested shots
Body-language / Effort
Automatic playtype
Summarization
Action Recognition [Francia et al., 2018] 34Metrics: on top of CV - Enrich & Analyze
• Add more layers the 2D information to re-define metrics:
- A player is at (X, Y) and he/she’s facing towards xº (+ pose data)
Field-of-view estimation
Specific angle stats:
- Passing / screening
Left / right deviations
Model refinement
a posteriori
Body Orientation [Arbués-Sangüesa et al., 2020-2021] 35Metrics: on top of CV - Enrich & Analyze
• Add more layers the 2D information to re-define metrics:
- A player is at (X, Y) and he/she’s facing towards xº (+ pose data)
Corners Position Homography (Im-2D) Smoothing +
WPose WBall
Shoulder-Hip
Coarse Validation
SupRes OpenPose
Face
Pose Orientation Ball-based Orientation
Always Look on the Bright Side of the Field: Merging Pose and
Contextual Data to Estimate Orientation of Soccer Players [ICIP 2021] 36Metrics: on top of CV - Enrich & Analyze
• Add more layers the 2D information to re-define metrics:
- A player is at (X, Y) and he/she’s facing towards xº (+ pose data)
Learning Football Body-Orientation as a Matter of Classification [AISA 2021] 37Metrics: on top of CV - Enrich & Analyze
• Add more layers the 2D information to re-define metrics:
- A player is at (X, Y) and he/she’s facing towards xº (+ pose data)
OrientSonar Maps Reaction Maps On-Field Maps
360º 360º
Reception Event
Orientation
Orientation
Receiver
0º Pass Event 360º 0º Goal 360º
Orientation Orientation
Low Volume High Volume Low Accuracy High Accuracy
Orientation-based Maps 38Metrics: on top of CV - Enrich & Analyze
• Add more layers the 2D information to re-define metrics:
- A player is at (X, Y) and he/she’s facing towards xº (+ pose data)
4 3 4
3
2 2
5 5
1 1
Using Player's Body-Orientation to Model
Pass Feasibility in Soccer [CVSports20’, 2n Place Award] 39Metrics: on top of CV - Enrich & Analyze
• Add more layers the 2D information to re-define metrics:
- A player is at (X, Y) and he/she’s facing towards xº (+ pose data)
(...)
Stay tuned! 40I have stats… Now what?
Da Da Mo Co Im
ta ta d mmu plem
Ga Pro els
the ce &M nic en
rin ssi e at tat
g ng trics io n ion
41Communication
Each communication channel is unique
ar ke t GM
r m
ay e
p l
d ,
at e s
re l t ure
d - d p ic
g e t s a n
d l i g h
Bu High
Coaching Staff
Detailed game facets
(l) Assistant Coaches
Pac
e, in (m) Head Coach
jury
Ess
prev
enti (r) S&C
ent on
ial
info
Players
42Communication
Complement
each other
Be a valuable resource, Learn how to ask the Acquire the coaching
not a coaching thread appropriate questions staff’s lingo
43Communication: visualization
DPIPM
2010 Elite Rim Protectors
+ Valuable Offensive Assets
+4 12
10 13 2012
2013
+3
2010 2014 All-Around Offensive Players
2007
04
+ Agressive/Smart Defenders
+2 2004
13 14
11
2013 18
2011
+1 2018
Shane Larkin
Otherworldly Scorer
+0 Disclaimer: “only” 25 games
OPIPM 44
+2 +4 +6 2020 +8Communication
[Credits - Seth Partnow]
• Being better is not being right at everything, it is being less wrong.
• Put data in the language of sport — hybrid profiles.
• Properties of language:
Arbitrariness
Almost all statistics are arbitrary (not the scoreboard)
We use representative abstractions of game concepts (e.g. rebound)
46Communication
[Credits - Seth Partnow]
• Being better is not being right at everything, it is being less wrong.
• Put data in the language of sport — hybrid profiles.
• Properties of language:
Cultural
Transmission
Naming stats: objective & basic, clear calculation / concepts.
Avoid: unnecessary jargon / opaque terms, snake oil.
47In this talk…
Da Da Mo Co Im
ta ta d mmu plem
Ga Pro els
the ce &M nic en
rin ssi e at tat
g ng trics io n ion
48Implementation
49Acknowledgements
Gloria Haro
Coloma Ballester
Adrián Martín
Martí Casals
Seth Partnow
Todd Whitehead
Justin Jacobs
Andrew Patton
Mike Beuoy
Nathan Walker
Xavi Pardina
Juan Llaneza
Diego Ocampo
Javier Fernández
Raúl Peláez 50Thanks for the attention Adrià Arbués Sangüesa arbues6@gmail.com @arbues6 Free agent from fall 2021!
You can also read
