MRP 2020: Cross-Framework and Cross-Lingual Meaning Representation Parsing - Daniel Hershcovich
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
MRP 2020:
Cross-Framework and Cross-Lingual
Meaning Representation Parsing
http://mrp.nlpl.eu
Stephan Oepen♣ , Omri Abend♠ , Lasha Abzianidze♥ , Johan Bos♦ ,
Jan Hajič◦ , Daniel Hershcovich? , Bin Li• , Tim O’Gorman ,
Nianwen Xue∗ , and Daniel Zeman◦
♣ University of Oslo, Department of Informatics
♠ The Hebrew University of Jerusalem, School of Computer Science and Engineering
♥ Utrecht University, Logic, Language, and Information
♦ University of Groningen, Center for Language and Cognition
◦ Charles University, Prague, Institute of Formal and Applied Linguistics
? University of Copenhagen, Department of Computer Science
• Nanjing Normal University, School of Chinese Language and Literature
University of Massachusetts at Amherst, College of Information and Computer Sciences
∗ Brandeis University, Department of Computer Science
mrp-organizers@nlpl.eu10,000-Meter Perspective: Parsing into Semantic Graphs
A similar technique is almost impossible to apply to other crops.
210,000-Meter Perspective: Parsing into Semantic Graphs
almost
domain
possible-01
polarity -
ARG1
resemble-01 apply-02 other
ARG1 ARG1 ARG2 domain
technique crop
210,000-Meter Perspective: Parsing into Semantic Graphs
A F D F P A U
⟨20:22⟩ ⟨41:43⟩ ⟨44:49⟩ ⟨65:66⟩
F E E C R E C
⟨0:1⟩ C ⟨23:29⟩ ⟨30:40⟩ ⟨50:52⟩ ⟨53:58⟩ ⟨59:65⟩
S A
⟨2:9⟩ ⟨10:19⟩
210,000-Meter Perspective: Parsing into Semantic Graphs
PRED
be
⟨20:22⟩
sentmod enunc
sempos v
frame en-v#ev-w218f2
ACT PAT
apply
possible
⟨41:43⟩ ⟨44:49⟩
⟨30:40⟩
sempos v
sempos adj.denot
frame en-v#ev-w119f2
ADDR PAT BEN EXT
crop technique almost
#Benef
⟨50:52⟩ ⟨59:64⟩ ⟨0:1⟩ ⟨10:19⟩ ACT ⟨23:29⟩
sempos x
sempos n.denot sempos n.denot sempos adv.denot.grad.neg
RSTR RSTR coref.gram
other similar
#Gen
⟨53:58⟩ ⟨2:9⟩
sempos x
sempos adj.denot sempos adj.denot
210,000-Meter Perspective: Parsing into Semantic Graphs
in ATTRIBUTION PRESUPPOSITION
impossible.a.01
in in in in
Time Topic Degree in apply.v.01
Theme in
in time.n.08 proposition.n.01 almost.r.01 Theme Goal in
EQU technique.n.01 crop.n.01
"now" Attribute NEQ
similar.a.01 crop.n.01
210,000-Meter Perspective: Parsing into Semantic Graphs
_almost_a_1
⟨23:29⟩
ARG1
_impossible_a_for
comp
⟨30:40⟩
⟨2:9⟩
TENSE pres
ARG1 ARG1
_similar_a_to _a_q _apply_v_to udef_q _other_a_1
⟨2:9⟩ ⟨0:1⟩ ⟨44:49⟩ ⟨53:100⟩ ⟨53:58⟩
ARG1 BV ARG2 ARG3 BV ARG1
_technique_n_1 _crop_n_1
⟨10:19⟩ ⟨59:65⟩
NUM sg NUM pl
2Why Graph-Based Meaning Representation?
I saw Joe’s dog, which was running in the garden.
The dog was chasing a cat.
3Why Graph-Based Meaning Representation?
I saw Joe’s dog, which was running in the garden.
The dog was chasing a cat.
semantic parsing
see-01
ARG0 ARG1
I dog chase-01
poss ARG0 ARG0 ARG1
joe run-02 dog cat
location
garden
3Why Graph-Based Meaning Representation?
I saw Joe’s dog, which was running in the garden.
The dog was chasing a cat.
semantic parsing
see-01
ARG0 ARG1
I dog chase-01
poss ARG0 ARG0 ARG1
joe run-02 dog cat
location
garden
merge
see-01 chase-01
ARG0 ARG1 ARG0 ARG1
I dog cat
poss ARG0
joe run-02
location
garden
3Why Graph-Based Meaning Representation?
I saw Joe’s dog, which was running in the garden.
The dog was chasing a cat.
semantic parsing
see-01
ARG0 ARG1
I dog chase-01
poss ARG0 ARG0 ARG1
joe run-02 dog cat
location
garden
merge
see-01 chase-01
chase-01
ARG0 ARG1 ARG0 ARG1
ARG0 location ARG1
I dog cat
cat
poss ARG0
summarize dog garden
joe run-02 poss
location
joe
garden
3Why Graph-Based Meaning Representation?
I saw Joe’s dog, which was running in the garden.
The dog was chasing a cat.
semantic parsing
see-01
ARG0 ARG1
I dog chase-01
poss ARG0 ARG0 ARG1
joe run-02 dog cat
location
garden
Joe’s dog was chasing a cat in the garden.
merge surface realisation
see-01 chase-01
chase-01
ARG0 ARG1 ARG0 ARG1
ARG0 location ARG1
I dog cat
cat
poss ARG0
summarize dog garden
joe run-02 poss
location
joe
garden
3Why Graph-Based Meaning Representation?
I saw Joe’s dog, which was running in the garden.
The dog was chasing a cat.
semantic parsing
see-01
ARG0 ARG1
I dog chase-01
poss ARG0 ARG0 ARG1
joe run-02 dog cat
location
garden
Joe’s dog was chasing a cat in the garden.
merge surface realisation
see-01 chase-01
chase-01
ARG0 ARG1 ARG0 ARG1
ARG0 location ARG1
I dog cat
cat
poss ARG0
summarize dog garden
joe run-02 poss
location
joe
garden
Hardy & Vlachos (2018): 2+ ROUGE points over strong encoder–decoder.
3Syntactic Trees vs. Semantic Graphs
root
p
nmod prd pmod
nmod sbj amod amod im adv nmod
A similar technique is almost impossible to apply to other crops .
4Syntactic Trees vs. Semantic Graphs
root
p
nmod prd pmod
nmod sbj amod amod im adv nmod
A similar technique is almost impossible to apply to other crops .
top
BV ARG2 ARG3
ARG1 ARG1 ARG1 ARG1
A similar technique almost impossible apply other crops
4Syntactic Trees vs. Semantic Graphs
root
p
nmod prd pmod
nmod sbj amod amod im adv nmod
A similar technique is almost impossible to apply to other crops .
top
BV ARG2 ARG3
ARG1 ARG1 ARG1 ARG1
A similar technique almost impossible apply other crops
∃x : technique’(x) ∧ similar’(x); ∃y : crop’(y) ∧ other’(y)
→ almost’(¬possible’(apply’(e, x, y)))
4Syntactic Trees vs. Semantic Graphs
root
p
nmod prd pmod
nmod sbj amod amod im adv nmod
A similar technique is almost impossible to apply to other crops .
top
BV ARG2 ARG3
ARG1 ARG1 ARG1 ARG1
A similar technique almost impossible apply other crops
∃x : technique’(x) ∧ similar’(x); ∃y : crop’(y) ∧ other’(y)
→ almost’(¬possible’(apply’(e, x, y)))
4Syntactic Trees vs. Semantic Graphs
root
p
nmod prd pmod
nmod sbj amod amod im adv nmod
A similar technique is almost impossible to apply to other crops .
top
BV ARG2 ARG3
ARG1 ARG1 ARG1 ARG1
A similar technique almost impossible apply other crops
∃x : technique’(x) ∧ similar’(x); ∃y : crop’(y) ∧ other’(y)
→ almost’(¬possible’(apply’(e, x, y)))
4Syntactic Trees vs. Semantic Graphs
root
p
nmod prd pmod
nmod sbj amod amod im adv nmod
A similar technique is almost impossible to apply to other crops .
top
BV ARG2 ARG3
ARG1 ARG1 ARG1 ARG1
A similar technique almost impossible apply other crops
∃x : technique’(x) ∧ similar’(x); ∃y : crop’(y) ∧ other’(y)
→ almost’(¬possible’(apply’(e, x, y)))
4Syntactic Trees vs. Semantic Graphs
root
p
nmod prd pmod
nmod sbj amod amod im adv nmod
A similar technique is almost impossible to apply to other crops .
top
BV ARG2 ARG3
ARG1 ARG1 ARG1 ARG1
A similar technique almost impossible apply other crops
∃x : technique’(x) ∧ similar’(x); ∃y : crop’(y) ∧ other’(y)
→ almost’(¬possible’(apply’(e, x, y)))
Different Desiderata and Levels of Abstraction
I Grammaticality (e.g. subject–verb agreement) vs. relational structure.
4Semi-Formally: Trees vs. Graphs
Structural Wellformedness Conditions on Trees
I Unique root, connected, single parent, free of cycles; maybe projective;
→ all nodes (but the root) reachable by unique directed path from root.
5Semi-Formally: Trees vs. Graphs
Structural Wellformedness Conditions on Trees
I Unique root, connected, single parent, free of cycles; maybe projective;
→ all nodes (but the root) reachable by unique directed path from root.
top
BV ARG2 ARG3
ARG1 ARG1 ARG1 ARG1
A similar technique is almost impossible to apply to other crops .
5Semi-Formally: Trees vs. Graphs
Structural Wellformedness Conditions on Trees
I Unique root, connected, single parent, free of cycles; maybe projective;
→ all nodes (but the root) reachable by unique directed path from root.
top
BV ARG2 ARG3
ARG1 ARG1 ARG1 ARG1
A similar technique is almost impossible to apply to other crops .
Beyond Trees: General Graphs
I Argument sharing: nodes with multiple incoming edges (in-degree > 1);
5Semi-Formally: Trees vs. Graphs
Structural Wellformedness Conditions on Trees
I Unique root, connected, single parent, free of cycles; maybe projective;
→ all nodes (but the root) reachable by unique directed path from root.
top
BV ARG2 ARG3
ARG1 ARG1 ARG1 ARG1
A similar technique is almost impossible to apply to other crops .
Beyond Trees: General Graphs
I Argument sharing: nodes with multiple incoming edges (in-degree > 1);
I some surface tokens do not contribute (as nodes; many function words);
5Semi-Formally: Trees vs. Graphs
Structural Wellformedness Conditions on Trees
I Unique root, connected, single parent, free of cycles; maybe projective;
→ all nodes (but the root) reachable by unique directed path from root.
top
BV ARG2 ARG3
ARG1 ARG1 ARG1 ARG1
A similar technique is almost impossible to apply to other crops .
Beyond Trees: General Graphs
I Argument sharing: nodes with multiple incoming edges (in-degree > 1);
I some surface tokens do not contribute (as nodes; many function words);
I (structurally) multi-rooted: more than one node with zero in-degree;
5Semi-Formally: Trees vs. Graphs
Structural Wellformedness Conditions on Trees
I Unique root, connected, single parent, free of cycles; maybe projective;
→ all nodes (but the root) reachable by unique directed path from root.
top
BV ARG2 ARG3
ARG1 ARG1 ARG1 ARG1
A similar technique is almost impossible to apply to other crops .
Beyond Trees: General Graphs
I Argument sharing: nodes with multiple incoming edges (in-degree > 1);
I some surface tokens do not contribute (as nodes; many function words);
I (structurally) multi-rooted: more than one node with zero in-degree;
→ massive growth in modeling and algorithmic complexity (NP-complete).
5High-Level Goals of the Shared Task
Cross-Framework Comparability and Interoperability
I Vast, complex landscape of representing natural language meaning;
I diverse linguistic traditions, modeling assumptions, levels of ambition;
→ clarify concepts and terminology; unify representations and evaluation.
6High-Level Goals of the Shared Task
Cross-Framework Comparability and Interoperability
I Vast, complex landscape of representing natural language meaning;
I diverse linguistic traditions, modeling assumptions, levels of ambition;
→ clarify concepts and terminology; unify representations and evaluation.
Parsing into Graph-Structured Representations
I Cottage industry of parsers with output structures beyond rooted trees;
I distinct techniques, e.g. based on transitions, composition, ‘translation’;
→ evaluate across frameworks; learning from complementary knowledge.
6High-Level Goals of the Shared Task
Cross-Framework Comparability and Interoperability
I Vast, complex landscape of representing natural language meaning;
I diverse linguistic traditions, modeling assumptions, levels of ambition;
→ clarify concepts and terminology; unify representations and evaluation.
Parsing into Graph-Structured Representations
I Cottage industry of parsers with output structures beyond rooted trees;
I distinct techniques, e.g. based on transitions, composition, ‘translation’;
→ evaluate across frameworks; learning from complementary knowledge.
Two Distinct Tracks in MRP 2020
I Cross-Framework Perspective: Seek commonality and complementarity.
I Cross-Lingual Perspective: In-framework transfer to another language.
6Graph Structure vs. Node (or Edge) Decorations
Zero-Arity Predicates vs. Constants
I Nodes and edges can be labeled (e.g. by relation and role identifiers);
7Graph Structure vs. Node (or Edge) Decorations
Zero-Arity Predicates vs. Constants
I Nodes and edges can be labeled (e.g. by relation and role identifiers);
I labels can be internally structured: node properties and edge attributes;
I properties (and attributes) are non-recursive attribute–value matrices;
I node (and edge) label is merely a distinguished property (or attribute);
7Graph Structure vs. Node (or Edge) Decorations
Zero-Arity Predicates vs. Constants
I Nodes and edges can be labeled (e.g. by relation and role identifiers);
I labels can be internally structured: node properties and edge attributes;
I properties (and attributes) are non-recursive attribute–value matrices;
I node (and edge) label is merely a distinguished property (or attribute);
I distinction is not commonly discussed, but used by many frameworks.
person
person
name age
name age
name temporal-quantity
OP1 Pierre name temporal-quantity
OP2 Vinken QUANT 61
OP1 OP2
QUANT unit
unit
Pierre Vinken 61 year
year
Pierre Vinken is 61 years old.
7Anchoring in the Surface String
Relating Pieces of Meaning to the Linguistic Signal
I Intuitively, sub-structures of meaning relate to sub-parts of the input;
I semantic frameworks vary in how much weight to put on this relation;
8Anchoring in the Surface String
Relating Pieces of Meaning to the Linguistic Signal
I Intuitively, sub-structures of meaning relate to sub-parts of the input;
I semantic frameworks vary in how much weight to put on this relation;
I anchoring of graph elements in sub-strings of the underlying utterance;
I can be part of semantic annotations or not; can take different forms;
8Anchoring in the Surface String
Relating Pieces of Meaning to the Linguistic Signal
I Intuitively, sub-structures of meaning relate to sub-parts of the input;
I semantic frameworks vary in how much weight to put on this relation;
I anchoring of graph elements in sub-strings of the underlying utterance;
I can be part of semantic annotations or not; can take different forms;
I hierarchy of anchoring types: Flavor (0)–(2); bilexical graphs strictest;
Name Example Type of Anchoring
(0) bilexical DM,
hh
( (h(PSD
h(
(
h nodes are sub-set of surface tokens
(1) anchored EDS, PTG, UCCA free node–sub-string correspondences
(2) unanchored AMR, DRG no explicit sub-string correspondences
8Anchoring in the Surface String
Relating Pieces of Meaning to the Linguistic Signal
I Intuitively, sub-structures of meaning relate to sub-parts of the input;
I semantic frameworks vary in how much weight to put on this relation;
I anchoring of graph elements in sub-strings of the underlying utterance;
I can be part of semantic annotations or not; can take different forms;
I hierarchy of anchoring types: Flavor (0)–(2); bilexical graphs strictest;
I anchoring central in parsing, explicit or latent; aka ‘alignment’ for AMR;
I relevant to at least some downstream tasks; should impact evaluation.
Name Example Type of Anchoring
(0) bilexical DM,
hh
( (h(PSD
h(
(
h nodes are sub-set of surface tokens
(1) anchored EDS, PTG, UCCA free node–sub-string correspondences
(2) unanchored AMR, DRG no explicit sub-string correspondences
8A Selection of Semantic Graphbanks
Selection Criteria
I ‘Full-sentence’ semantics: all content-bearing units receive annotations;
I natively graph-based: meaning representation through (directed) graphs;
I large-scale, gold-standard annotations and parsers are publicly available;
9A Selection of Semantic Graphbanks
Selection Criteria
I ‘Full-sentence’ semantics: all content-bearing units receive annotations;
I natively graph-based: meaning representation through (directed) graphs;
I large-scale, gold-standard annotations and parsers are publicly available;
→ five distinct frameworks, bi-lexical to unanchored; sadly, English only;
9A Selection of Semantic Graphbanks
Selection Criteria
I ‘Full-sentence’ semantics: all content-bearing units receive annotations;
I natively graph-based: meaning representation through (directed) graphs;
I large-scale, gold-standard annotations and parsers are publicly available;
→ five distinct frameworks, bi-lexical to unanchored; sadly, English only;
9A Selection of Semantic Graphbanks
Selection Criteria
I ‘Full-sentence’ semantics: all content-bearing units receive annotations;
I natively graph-based: meaning representation through (directed) graphs;
I large-scale, gold-standard annotations and parsers are publicly available;
→ five distinct frameworks, bi-lexical to unanchored; sadly, English only;
→ new in MRP 2020: one additional language for four of the frameworks.
9A Selection of Semantic Graphbanks
Selection Criteria
I ‘Full-sentence’ semantics: all content-bearing units receive annotations;
I natively graph-based: meaning representation through (directed) graphs;
I large-scale, gold-standard annotations and parsers are publicly available;
→ five distinct frameworks, bi-lexical to unanchored; sadly, English only;
→ new in MRP 2020: one additional language for four of the frameworks.
(With Apologies to) Non-Graph or Non-Meaning Banks
I PropBank (Palmer et al., 2005), Framenet (Baker et al., 1998), . . . ;
I Universal Decompositional Semantics (White et al., 2016);
I Enhanced Universal Dependencies (Schuster & Manning, 2016);
I ...
9(1) Elementary Dependency Structures (EDS)
Simplification of Underspecified Logical Forms (Oepen & Lønning, 2006)
I Converted from LinGO Redwoods Treebank (Flickinger et al., 2017);
I decomposition or construction meaning; anchors: arbitrary sub-strings.
_almost_a_1
⟨23:29⟩
ARG1
comp _impossible_a_for
⟨2:9⟩ ⟨30:40⟩
ARG1 ARG1
_similar_a_to _a_q _apply_v_to udef_q _other_a_1
⟨2:9⟩ ⟨0:1⟩ ⟨44:49⟩ ⟨53:100⟩ ⟨53:58⟩
ARG1 BV ARG2 ARG3 BV ARG1
_technique_n_1 _crop_n_1
⟨10:19⟩ ⟨59:65⟩
10(1) Prague Tectogrammatical Graphs (PTG)
Simplification of FGD Tectogrammatical ‘Trees’ (Zeman & Hajič, 2020)
I Prague (Czech–English) Dependency Treebanks (Hajič et al., 2012);
I unanchored nodes for unexpressed arguments, e.g. #Benef and #Gen.
PRED
be
⟨20:22⟩
ACT PAT
apply possible
⟨41:43⟩ ⟨44:49⟩ ⟨30:40⟩
ADDR PAT BEN EXT
crop technique almost
ACT #Benef
⟨50:52⟩ ⟨59:64⟩ ⟨0:1⟩ ⟨10:19⟩ ⟨23:29⟩
RSTR RSTR coref.gram
other similar
#Gen
⟨53:58⟩ ⟨2:9⟩
11(1) Universal Conceptual Cognitive Annotation (UCCA)
Multi-Layered Design (Abend & Rappoport, 2013); Foundational Layer
I Tree backbone: semantic ‘constituents’ are scenes (‘clauses’) and units;
A F D F P A U
⟨20:22⟩ ⟨41:43⟩ ⟨44:49⟩ ⟨65:66⟩
F E E C R E C
⟨0:1⟩ C ⟨23:29⟩ ⟨30:40⟩ ⟨50:52⟩ ⟨53:58⟩ ⟨59:65⟩
S A
⟨2:9⟩ ⟨10:19⟩
A similar technique is almost impossible to apply to other crops.
12(1) Universal Conceptual Cognitive Annotation (UCCA)
Multi-Layered Design (Abend & Rappoport, 2013); Foundational Layer
I Tree backbone: semantic ‘constituents’ are scenes (‘clauses’) and units;
I scenes (Process or State): pArticipants and aDverbials (plus F and U);
A F D F P A U
⟨20:22⟩ ⟨41:43⟩ ⟨44:49⟩ ⟨65:66⟩
F E E C R E C
⟨0:1⟩ C ⟨23:29⟩ ⟨30:40⟩ ⟨50:52⟩ ⟨53:58⟩ ⟨59:65⟩
S A
⟨2:9⟩ ⟨10:19⟩
A similar technique is almost impossible to apply to other crops.
12(1) Universal Conceptual Cognitive Annotation (UCCA)
Multi-Layered Design (Abend & Rappoport, 2013); Foundational Layer
I Tree backbone: semantic ‘constituents’ are scenes (‘clauses’) and units;
I scenes (Process or State): pArticipants and aDverbials (plus F and U);
I complex units distinguish Center and Elaborator(s); allow remote edges.
A F D F P A U
⟨20:22⟩ ⟨41:43⟩ ⟨44:49⟩ ⟨65:66⟩
F E E C R E C
⟨0:1⟩ C ⟨23:29⟩ ⟨30:40⟩ ⟨50:52⟩ ⟨53:58⟩ ⟨59:65⟩
S A
⟨2:9⟩ ⟨10:19⟩
A similar technique is almost impossible to apply to other crops.
12(2) Abstract Meaning Representation (AMR)
Banarescu et al. (2013)
possible-01
polarity -
I Abstractly (if not linguistically)
similar to EDS, but unanchored;
mod (domain) ARG1
I verbal senses from PropBank++ ;
almost apply-02
I negation as node-local property;
ARG1 ARG2
I tree-like annotation: inversed
technique crop edges normalized for evaluation;
(ARG1)-of mod (domain) I originally designed for (S)MT;
various NLU applications to date.
resemble-01 other
A similar technique is almost impossible to apply to other crops.
13(2) Discourse Representation Graphs (DRG)
Graph Encoding of DRS ‘Nested Boxes’ (Kamp & Reyle, 1993)
I From Groningen Parallel Meaning Bank (Abzianidze et al., 2017);
I explicit encoding of scope (boxes and in edges), using reified roles.
in ATTRIBUTION PRESUPPOSITION
impossible.a.01
in in in in
Time Topic Degree in apply.v.01
Theme in
in time.n.08 proposition.n.01 almost.r.01 Theme Goal in
EQU technique.n.01 crop.n.01
"now" Attribute NEQ
similar.a.01 crop.n.01
14Cross-Framework Training and Evaluation Data
EDS PTG UCCA AMR DRG
Flavor 1 1 1 2 2
Text Type newspaper newspaper mixed mixed mixed
train
Sentences 37,192 42,024 6,872 57,885 6,606
Tokens 861,831 1,026,033 171,838 1,049,083 44,692
Text Type mixed mixed mixed mixed mixed
validate
Sentences 3,302 1,664 1,585 3,560 885
Tokens 65,564 40,770 25,982 61,722 5,541
Text Type mixed newspaper mixed mixed mixed
test
Sentences 4,040 2,507 600 2,457 898
Tokens 68,280 59,191 18,633 49,760 5,991
15Cross-Framework Training and Evaluation Data
EDS PTG UCCA AMR DRG
Flavor 1 1 1 2 2
Text Type newspaper newspaper mixed mixed mixed
train
Sentences 37,192 42,024 6,872 57,885 6,606
Tokens 861,831 1,026,033 171,838 1,049,083 44,692
Text Type mixed mixed mixed mixed mixed
validate
Sentences 3,302 1,664 1,585 3,560 885
Tokens 65,564 40,770 25,982 61,722 5,541
Text Type mixed newspaper mixed mixed mixed
test
Sentences 4,040 2,507 600 2,457 898
Tokens 68,280 59,191 18,633 49,760 5,991
15Cross-Framework Training and Evaluation Data
EDS PTG UCCA AMR DRG
Flavor 1 1 1 2 2
Text Type newspaper newspaper mixed mixed mixed
train
Sentences 37,192 42,024 6,872 57,885 6,606
Tokens 861,831 1,026,033 171,838 1,049,083 44,692
Text Type mixed mixed mixed mixed mixed
validate
Sentences 3,302 1,664 1,585 3,560 885
Tokens 65,564 40,770 25,982 61,722 5,541
Text Type mixed newspaper mixed mixed mixed
test
Sentences 4,040 2,507 600 2,457 898
Tokens 68,280 59,191 18,633 49,760 5,991
I Validation split is MRP 2019 evaluation data; allowed for fine-tuning;
15Cross-Framework Training and Evaluation Data
EDS PTG UCCA AMR DRG
Flavor 1 1 1 2 2
Text Type newspaper newspaper mixed mixed mixed
train
Sentences 37,192 42,024 6,872 57,885 6,606
Tokens 861,831 1,026,033 171,838 1,049,083 44,692
Text Type mixed mixed mixed mixed mixed
validate
Sentences 3,302 1,664 1,585 3,560 885
Tokens 65,564 40,770 25,982 61,722 5,541
Text Type mixed newspaper mixed mixed mixed
test
Sentences 4,040 2,507 600 2,457 898
Tokens 68,280 59,191 18,633 49,760 5,991
I Validation split is MRP 2019 evaluation data; allowed for fine-tuning;
I linguistics: smallish WSJ sample in all frameworks publicly available;
I evaluation: subset of 100 sentences from The Little Prince is public.
15Cross-Lingual Training and Evaluation Data
PTG UCCA AMR DRG
Language Czech German Chinese German
Flavor 1 1 1 2
Text Type newspaper mixed mixed mixed
train
Sentences 43,955 4,125 18,365 1,575
Tokens 740,466 95,634 428,054 9,088
Text Type newspaper mixed mixed mixed
test
Sentences 5,476 444 1,713 403
Tokens 92,643 10,585 39,228 2,384
16Cross-Lingual Training and Evaluation Data
PTG UCCA AMR DRG
Language Czech German Chinese German
Flavor 1 1 1 2
Text Type newspaper mixed mixed mixed
train
Sentences 43,955 4,125 18,365 1,575
Tokens 740,466 95,634 428,054 9,088
Text Type newspaper mixed mixed mixed
test
Sentences 5,476 444 1,713 403
Tokens 92,643 10,585 39,228 2,384
16Cross-Lingual Training and Evaluation Data
PTG UCCA AMR DRG
Language Czech German Chinese German
Flavor 1 1 1 2
Text Type newspaper mixed mixed mixed
train
Sentences 43,955 4,125 18,365 1,575
Tokens 740,466 95,634 428,054 9,088
Text Type newspaper mixed mixed mixed
test
Sentences 5,476 444 1,713 403
Tokens 92,643 10,585 39,228 2,384
16Cross-Lingual Training and Evaluation Data
PTG UCCA AMR DRG
Language Czech German Chinese German
Flavor 1 1 1 2
Text Type newspaper mixed mixed mixed
train
Sentences 43,955 4,125 18,365 1,575
Tokens 740,466 95,634 428,054 9,088
Text Type newspaper mixed mixed mixed
test
Sentences 5,476 444 1,713 403
Tokens 92,643 10,585 39,228 2,384
I Gold-standard graphs for one additional language in four frameworks;
16Cross-Lingual Training and Evaluation Data
PTG UCCA AMR DRG
Language Czech German Chinese German
Flavor 1 1 1 2
Text Type newspaper mixed mixed mixed
train
Sentences 43,955 4,125 18,365 1,575
Tokens 740,466 95,634 428,054 9,088
Text Type newspaper mixed mixed mixed
test
Sentences 5,476 444 1,713 403
Tokens 92,643 10,585 39,228 2,384
I Gold-standard graphs for one additional language in four frameworks;
I ‘low-resource’ training setting for two frameworks: UCCA and DRG;
? explor opportunities for cross-lingual transfer learning (in-framework).
16Cross-Framework Evaluation: MRP Graph Similarity
I Break down graphs into types of information: per-type and overall F1 ;
Different Types of Semantic Graph ‘Atoms’
_retire_v_1 proper_q
⟨7:14⟩ ⟨0:6⟩ EDS PTG UCCA AMR DRG
Top Nodes 3 3 3 3 3
ARG1 BV
Labeled Edges 3 3 3 3 (3)
Node Labels 3 3 7 3 3
named
Node Properties 3 3 7 3 7
CARG Pierre
⟨0:6⟩ Node Anchoring 3 (3) (3) 7 7
Edge Attributes 7 3 3 7 7
Pierre retired.
17Cross-Framework Evaluation: MRP Graph Similarity
I Break down graphs into types of information: per-type and overall F1 ;
I tops
Different Types of Semantic Graph ‘Atoms’
_retire_v_1 proper_q
⟨7:14⟩ ⟨0:6⟩ EDS PTG UCCA AMR DRG
Top Nodes 3 3 3 3 3
ARG1 BV
Labeled Edges 3 3 3 3 (3)
Node Labels 3 3 7 3 3
named
Node Properties 3 3 7 3 7
CARG Pierre
⟨0:6⟩ Node Anchoring 3 (3) (3) 7 7
Edge Attributes 7 3 3 7 7
Pierre retired.
17Cross-Framework Evaluation: MRP Graph Similarity
I Break down graphs into types of information: per-type and overall F1 ;
I tops and (labeled) edges;
Different Types of Semantic Graph ‘Atoms’
_retire_v_1 proper_q
⟨7:14⟩ ⟨0:6⟩ EDS PTG UCCA AMR DRG
Top Nodes 3 3 3 3 3
ARG1 BV
Labeled Edges 3 3 3 3 (3)
Node Labels 3 3 7 3 3
named
Node Properties 3 3 7 3 7
CARG Pierre
⟨0:6⟩ Node Anchoring 3 (3) (3) 7 7
Edge Attributes 7 3 3 7 7
Pierre retired.
17Cross-Framework Evaluation: MRP Graph Similarity
I Break down graphs into types of information: per-type and overall F1 ;
I tops and (labeled) edges; labels,
Different Types of Semantic Graph ‘Atoms’
_retire_v_1 proper_q
⟨7:14⟩ ⟨0:6⟩ EDS PTG UCCA AMR DRG
Top Nodes 3 3 3 3 3
ARG1 BV
Labeled Edges 3 3 3 3 (3)
Node Labels 3 3 7 3 3
named
Node Properties 3 3 7 3 7
CARG Pierre
⟨0:6⟩ Node Anchoring 3 (3) (3) 7 7
Edge Attributes 7 3 3 7 7
Pierre retired.
17Cross-Framework Evaluation: MRP Graph Similarity
I Break down graphs into types of information: per-type and overall F1 ;
I tops and (labeled) edges; labels, properties,
Different Types of Semantic Graph ‘Atoms’
_retire_v_1 proper_q
⟨7:14⟩ ⟨0:6⟩ EDS PTG UCCA AMR DRG
Top Nodes 3 3 3 3 3
ARG1 BV
Labeled Edges 3 3 3 3 (3)
Node Labels 3 3 7 3 3
named
Node Properties 3 3 7 3 7
CARG Pierre
⟨0:6⟩ Node Anchoring 3 (3) (3) 7 7
Edge Attributes 7 3 3 7 7
Pierre retired.
17Cross-Framework Evaluation: MRP Graph Similarity
I Break down graphs into types of information: per-type and overall F1 ;
I tops and (labeled) edges; labels, properties, anchors,
Different Types of Semantic Graph ‘Atoms’
_retire_v_1 proper_q
⟨7:14⟩ ⟨0:6⟩ EDS PTG UCCA AMR DRG
Top Nodes 3 3 3 3 3
ARG1 BV
Labeled Edges 3 3 3 3 (3)
Node Labels 3 3 7 3 3
named
Node Properties 3 3 7 3 7
CARG Pierre
⟨0:6⟩ Node Anchoring 3 (3) (3) 7 7
Edge Attributes 7 3 3 7 7
Pierre retired.
17Cross-Framework Evaluation: MRP Graph Similarity
I Break down graphs into types of information: per-type and overall F1 ;
I tops and (labeled) edges; labels, properties, anchors, and attributes;
Different Types of Semantic Graph ‘Atoms’
_retire_v_1 proper_q
⟨7:14⟩ ⟨0:6⟩ EDS PTG UCCA AMR DRG
Top Nodes 3 3 3 3 3
ARG1 BV
Labeled Edges 3 3 3 3 (3)
Node Labels 3 3 7 3 3
named
Node Properties 3 3 7 3 7
CARG Pierre
⟨0:6⟩ Node Anchoring 3 (3) (3) 7 7
Edge Attributes 7 3 3 7 7
Pierre retired.
17Cross-Framework Evaluation: MRP Graph Similarity
I Break down graphs into types of information: per-type and overall F1 ;
I tops and (labeled) edges; labels, properties, anchors, and attributes;
I requires node–node correspondences; search for overall maximum score;
I maximum common edge subgraph isomorphism (MCES) is NP-hard;
Different Types of Semantic Graph ‘Atoms’
_retire_v_1 proper_q
⟨7:14⟩ ⟨0:6⟩ EDS PTG UCCA AMR DRG
Top Nodes 3 3 3 3 3
ARG1 BV
Labeled Edges 3 3 3 3 (3)
Node Labels 3 3 7 3 3
named
Node Properties 3 3 7 3 7
CARG Pierre
⟨0:6⟩ Node Anchoring 3 (3) (3) 7 7
Edge Attributes 7 3 3 7 7
Pierre retired.
17Cross-Framework Evaluation: MRP Graph Similarity
I Break down graphs into types of information: per-type and overall F1 ;
I tops and (labeled) edges; labels, properties, anchors, and attributes;
I requires node–node correspondences; search for overall maximum score;
I maximum common edge subgraph isomorphism (MCES) is NP-hard;
→ smart initialization, scheduling, and pruning yield strong approximation.
Different Types of Semantic Graph ‘Atoms’
_retire_v_1 proper_q
⟨7:14⟩ ⟨0:6⟩ EDS PTG UCCA AMR DRG
Top Nodes 3 3 3 3 3
ARG1 BV
Labeled Edges 3 3 3 3 (3)
Node Labels 3 3 7 3 3
named
Node Properties 3 3 7 3 7
CARG Pierre
⟨0:6⟩ Node Anchoring 3 (3) (3) 7 7
Edge Attributes 7 3 3 7 7
Pierre retired.
17Graphbank Statistics (Kuhlmann & Oepen, 2016)
EDS PTG UCCA AMR−1 DRG
(02) Average Tokens per Graph 22.17 24.42 25.01 18.12 6.77
(03) Average Nodes per Token 1.26 0.74 1.33 0.64 2.09
(04) Distinct Edge Labels 10 72 15 101 16
(05) Percentage of top nodes 0.99 1.27 1.66 3.77 3.40
proportions
(06) Percentage of node labels 29.02 21.61 – 43.91 39.81
(07) Percentage of node properties 12.54 26.22 – 7.63 –
(08) Percentage of node anchors 29.02 19.63 38.80 – –
(09) Percentage of (labeled) edges 28.43 26.10 56.88 44.69 56.79
(10) Percentage of edge attributes – 5.17 2.66 – –
(11) %g Rooted Trees 0.09 22.63 28.19 22.05 0.35
(12) %g Treewidth One 68.60 22.67 34.17 49.91 0.35
(13) Average Treewidth 1.317 2.067 1.691 1.561 2.131
treeness
(14) Maximal Treewidth 3 7 4 5 5
(15) Average Edge Density 1.015 1.177 1.055 1.092 1.265
(16) %n Reentrant 32.77 16.23 4.90 19.89 25.92
(17) %g Cyclic 0.27 33.97 0.00 0.38 0.27
(18) %g Not Connected 1.90 0.00 0.00 0.00 0.00
(19) %g Multi-Rooted 99.93 0.00 0.00 71.64 32.32
18Graphbank Statistics (Kuhlmann & Oepen, 2016)
EDS PTG UCCA AMR−1 DRG
(02) Average Tokens per Graph 22.17 24.42 25.01 18.12 6.77
(03) Average Nodes per Token 1.26 0.74 1.33 0.64 2.09
(04) Distinct Edge Labels 10 72 15 101 16
(05) Percentage of top nodes 0.99 1.27 1.66 3.77 3.40
proportions
(06) Percentage of node labels 29.02 21.61 – 43.91 39.81
(07) Percentage of node properties 12.54 26.22 – 7.63 –
(08) Percentage of node anchors 29.02 19.63 38.80 – –
(09) Percentage of (labeled) edges 28.43 26.10 56.88 44.69 56.79
(10) Percentage of edge attributes – 5.17 2.66 – –
(11) %g Rooted Trees 0.09 22.63 28.19 22.05 0.35
(12) %g Treewidth One 68.60 22.67 34.17 49.91 0.35
(13) Average Treewidth 1.317 2.067 1.691 1.561 2.131
treeness
(14) Maximal Treewidth 3 7 4 5 5
(15) Average Edge Density 1.015 1.177 1.055 1.092 1.265
(16) %n Reentrant 32.77 16.23 4.90 19.89 25.92
(17) %g Cyclic 0.27 33.97 0.00 0.38 0.27
(18) %g Not Connected 1.90 0.00 0.00 0.00 0.00
(19) %g Multi-Rooted 99.93 0.00 0.00 71.64 32.32
18Graphbank Statistics (Kuhlmann & Oepen, 2016)
EDS PTG UCCA AMR−1 DRG
(02) Average Tokens per Graph 22.17 24.42 25.01 18.12 6.77
(03) Average Nodes per Token 1.26 0.74 1.33 0.64 2.09
(04) Distinct Edge Labels 10 72 15 101 16
(05) Percentage of top nodes 0.99 1.27 1.66 3.77 3.40
proportions
(06) Percentage of node labels 29.02 21.61 – 43.91 39.81
(07) Percentage of node properties 12.54 26.22 – 7.63 –
(08) Percentage of node anchors 29.02 19.63 38.80 – –
(09) Percentage of (labeled) edges 28.43 26.10 56.88 44.69 56.79
(10) Percentage of edge attributes – 5.17 2.66 – –
(11) %g Rooted Trees 0.09 22.63 28.19 22.05 0.35
(12) %g Treewidth One 68.60 22.67 34.17 49.91 0.35
(13) Average Treewidth 1.317 2.067 1.691 1.561 2.131
treeness
(14) Maximal Treewidth 3 7 4 5 5
(15) Average Edge Density 1.015 1.177 1.055 1.092 1.265
(16) %n Reentrant 32.77 16.23 4.90 19.89 25.92
(17) %g Cyclic 0.27 33.97 0.00 0.38 0.27
(18) %g Not Connected 1.90 0.00 0.00 0.00 0.00
(19) %g Multi-Rooted 99.93 0.00 0.00 71.64 32.32
18Graphbank Statistics (Kuhlmann & Oepen, 2016)
EDS PTG UCCA AMR−1 DRG
(02) Average Tokens per Graph 22.17 24.42 25.01 18.12 6.77
(03) Average Nodes per Token 1.26 0.74 1.33 0.64 2.09
(04) Distinct Edge Labels 10 72 15 101 16
(05) Percentage of top nodes 0.99 1.27 1.66 3.77 3.40
proportions
(06) Percentage of node labels 29.02 21.61 – 43.91 39.81
(07) Percentage of node properties 12.54 26.22 – 7.63 –
(08) Percentage of node anchors 29.02 19.63 38.80 – –
(09) Percentage of (labeled) edges 28.43 26.10 56.88 44.69 56.79
(10) Percentage of edge attributes – 5.17 2.66 – –
(11) %g Rooted Trees 0.09 22.63 28.19 22.05 0.35
(12) %g Treewidth One 68.60 22.67 34.17 49.91 0.35
(13) Average Treewidth 1.317 2.067 1.691 1.561 2.131
treeness
(14) Maximal Treewidth 3 7 4 5 5
(15) Average Edge Density 1.015 1.177 1.055 1.092 1.265
(16) %n Reentrant 32.77 16.23 4.90 19.89 25.92
(17) %g Cyclic 0.27 33.97 0.00 0.38 0.27
(18) %g Not Connected 1.90 0.00 0.00 0.00 0.00
(19) %g Multi-Rooted 99.93 0.00 0.00 71.64 32.32
18High-Level Overview of Submissions Cross-Framework mCross-Lingual Teams AMR DRG EDS PTG UCCA Hitachi m m m m m ÚFAL m m m m m HIT-SCIR m m m m m HUJI-KU m m m m ISCAS TJU-BLCU m m m JBNU ÚFAL m m m m ERG 19
Score Distribution
1 Cross-Framework (Full)
m Cross-Lingual m
m m
m
m
0.8 m m
m
m
m m
m
m m
m
All MRP F1
0.6 m
m
m
m
0.4
m
m
0.2 m m
0
Overall AMR DRG EDS PTG UCCA
20Score Distribution: Zoom In
1
Cross-Framework (Full)
0.95 m Cross-Lingual
m
m
m
0.9
m
All MRP F1
m
0.85
m
0.8 m m
m
m m
0.75
m
Overall AMR DRG EDS PTG UCCA
21HIT-SCIR 0.81
Overall
Hitachi + ÚFAL 0.86
HIT-SCIR 0.70
ÚFAL 0.80
AMR
Hitachi 0.82
HIT-SCIR 0.89
Hitachi 0.93
DRG
ÚFAL 0.94
HIT-SCIR 0.87
ÚFAL 0.93
EDS
Hitachi 0.94
Cross-Framework Track: Full Evaluation
HIT-SCIR 0.84
ÚFAL 0.88
PTG
Hitachi 0.89
Hitachi + HIT-SCIR 0.75
UCCA
ÚFAL 0.76
22HIT-SCIR 0.69
Overall
Hitachi + ÚFAL 0.85
HIT-SCIR 0.49
ÚFAL 0.78
AMR
Hitachi 0.80
HIT-SCIR 0.68
ÚFAL 0.90
DRG
Hitachi 0.93
EDS
mCross-Lingual Track: Full Evaluation
HIT-SCIR 0.78
Hitachi 0.87
PTG
ÚFAL 0.91
Hitachi 0.79
HIT-SCIR 0.80
UCCA
ÚFAL 0.81
23Medal Ceremony! Cross-Framework mCross-Lingual Teams AMR DRG EDS PTG UCCA Hitachi m m m m m ÚFAL m m m m m HIT-SCIR m m m m m HUJI-KU m m m m ISCAS TJU-BLCU m m m JBNU ÚFAL m m m m ERG 24
HIT-SCIR 0.80
Overall
Hitachi + ÚFAL 0.85
HIT-SCIR 0.70
ÚFAL 0.78
AMR
Hitachi 0.79
DRG
HIT-SCIR 0.89
ÚFAL 0.95
EDS
Hitachi 0.97
Cross-Framework Track: The Little Prince
HIT-SCIR 0.78
Hitachi 0.82
PTG
ÚFAL 0.83
HIT-SCIR 0.82
UCCA
Hitachi + ÚFAL 0.83
25HIT-SCIR 0.70 ÚFAL MRPipe 0.71
ÚFAL 0.78 Amazon 0.71
AMR
AMR
Hitachi 0.79 Saarland 0.73
MRP 2020:
MRP 2019:
Saarland 0.95
SJTU–NICT 0.95
DM
DRG
HIT-SCIR 0.95
HIT-SCIR 0.89 SJTU–NICT 0.91
ÚFAL 0.95 Saarland 0.92
EDS
EDS
Hitachi 0.97 SUDA–Alibaba 0.93
State of the Art: The Little Prince
HIT-SCIR 0.78 Saarland 0.88
Hitachi 0.82 Hitachi 0.88
PSD
PTG
ÚFAL 0.83 SJTU–NICT 0.89
HIT-SCIR 0.82 Saarland 0.76
SUDA–Alibaba 0.82
UCCA
UCCA
Hitachi + ÚFAL 0.83 HIT-SCIR 0.83
26Interim Conclusions & Outlook
Lessons Learned (from Two Consecutive Shared Tasks)
I Good community interest: 180 subscribers; 19 data licenses (via LDC);
I technical barriers and ‘competitive selection’: 6 + 2 teams submitted;
27Interim Conclusions & Outlook
Lessons Learned (from Two Consecutive Shared Tasks)
I Good community interest: 180 subscribers; 19 data licenses (via LDC);
I technical barriers and ‘competitive selection’: 6 + 2 teams submitted;
→ advanced state of the art on four frameworks (but possibly not AMR);
27Interim Conclusions & Outlook
Lessons Learned (from Two Consecutive Shared Tasks)
I Good community interest: 180 subscribers; 19 data licenses (via LDC);
I technical barriers and ‘competitive selection’: 6 + 2 teams submitted;
→ advanced state of the art on four frameworks (but possibly not AMR);
→ greatly increased cross-framework uniformity; but limited (M)TL so far.
27Interim Conclusions & Outlook
Lessons Learned (from Two Consecutive Shared Tasks)
I Good community interest: 180 subscribers; 19 data licenses (via LDC);
I technical barriers and ‘competitive selection’: 6 + 2 teams submitted;
→ advanced state of the art on four frameworks (but possibly not AMR);
→ greatly increased cross-framework uniformity; but limited (M)TL so far.
Outlook: Beyond MRP 2020
I High-quality, robust meaning representation parsers generally available;
I MRP 2020 data, metrics, submissions, and scores as stable benchmark;
27Interim Conclusions & Outlook
Lessons Learned (from Two Consecutive Shared Tasks)
I Good community interest: 180 subscribers; 19 data licenses (via LDC);
I technical barriers and ‘competitive selection’: 6 + 2 teams submitted;
→ advanced state of the art on four frameworks (but possibly not AMR);
→ greatly increased cross-framework uniformity; but limited (M)TL so far.
Outlook: Beyond MRP 2020
I High-quality, robust meaning representation parsers generally available;
I MRP 2020 data, metrics, submissions, and scores as stable benchmark;
? post-mortem contrastive analysis of architectures (Buljan et al., 2020);
27Interim Conclusions & Outlook
Lessons Learned (from Two Consecutive Shared Tasks)
I Good community interest: 180 subscribers; 19 data licenses (via LDC);
I technical barriers and ‘competitive selection’: 6 + 2 teams submitted;
→ advanced state of the art on four frameworks (but possibly not AMR);
→ greatly increased cross-framework uniformity; but limited (M)TL so far.
Outlook: Beyond MRP 2020
I High-quality, robust meaning representation parsers generally available;
I MRP 2020 data, metrics, submissions, and scores as stable benchmark;
? post-mortem contrastive analysis of architectures (Buljan et al., 2020);
? increased focus on evaluation metrics: score ‘larger pieces’; SEMBLEU;
27Interim Conclusions & Outlook
Lessons Learned (from Two Consecutive Shared Tasks)
I Good community interest: 180 subscribers; 19 data licenses (via LDC);
I technical barriers and ‘competitive selection’: 6 + 2 teams submitted;
→ advanced state of the art on four frameworks (but possibly not AMR);
→ greatly increased cross-framework uniformity; but limited (M)TL so far.
Outlook: Beyond MRP 2020
I High-quality, robust meaning representation parsers generally available;
I MRP 2020 data, metrics, submissions, and scores as stable benchmark;
? post-mortem contrastive analysis of architectures (Buljan et al., 2020);
? increased focus on evaluation metrics: score ‘larger pieces’; SEMBLEU;
→ open discussion with 2020 participants towards the end of this session.
27Acknowledgments
Jayeol Chun, Dotan Dvir, Dan Flickinger,
Jiří Mírovský, Anna Nedoluzhko, Sebastian Schuster,
Milan Straka, and Zdeňka Urešová
Linguistic Data Consortium,
Nordic Language Processing Laboratory
28References I
Omri Abend & Ari Rappoport. 2013. Universal Conceptual Cognitive
Annotation (UCCA). In Proceedings of the 51th Meeting of the
Association for Computational Linguistics, pages 228 – 238, Sofia,
Bulgaria.
Lasha Abzianidze, Johannes Bjerva, Kilian Evang, Hessel Haagsma, Rik
van Noord, Pierre Ludmann, Duc-Duy Nguyen, & Johan Bos. 2017. The
Parallel Meaning Bank. Towards a multilingual corpus of translations
annotated with compositional meaning representations. In Proceedings
of the 15th Meeting of the European Chapter of the Association for
Computational Linguistics, pages 242–247, Valencia, Spain.
Collin F. Baker, Charles J. Fillmore, & John B. Lowe. 1998. The Berkeley
FrameNet project. In Proceedings of the 17th International Conference
on Computational Linguistics and the 36th Meeting of the Association
for Computational Linguistics, pages 86 – 90, Stroudsburg, PA, USA.
29References II
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt,
Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, &
Nathan Schneider. 2013. Abstract Meaning Representation for
sembanking. In Proceedings of the 7th Linguistic Annotation Workshop
and Interoperability with Discourse, pages 178 – 186, Sofia, Bulgaria.
Maja Buljan, Joakim Nivre, Stephan Oepen, & Lilja Øvrelid. 2020. A tale
of three parsers. Towards diagnostic evaluation for meaning
representation parsing. In Proceedings of the 12th Language Resources
and Evaluation Conference, pages 1902–1909, Marseille, France.
European Language Resources Association.
Dan Flickinger, Stephan Oepen, & Emily M. Bender. 2017. Sustainable
development and refinement of complex linguistic annotations at scale.
In Nacy Ide & James Pustejovsky, editors, Handbook of Linguistic
Annotation, pages 353 – 377. Springer, Dordrecht, The Netherlands.
30References III
Jan Hajič, Eva Hajičová, Jarmila Panevová, Petr Sgall, Ondřej Bojar, Silvie
Cinková, Eva Fučíková, Marie Mikulová, Petr Pajas, Jan Popelka, Jiří
Semecký, Jana Šindlerová, Jan Štěpánek, Josef Toman, Zdeňka
Urešová, & Zdeněk Žabokrtský. 2012. Announcing Prague
Czech-English Dependency Treebank 2.0. In Proceedings of the 8th
International Conference on Language Resources and Evaluation, pages
3153 – 3160, Istanbul, Turkey.
Hardy & Andreas Vlachos. 2018. Guided neural language generation for
abstractive summarization using abstract meaning representation. In
Proceedings of the 2018 Conference on Empirical Methods in Natural
Language Processing, Brussels, Belgium.
Hans Kamp & Uwe Reyle. 1993. From Discourse to Logic. An Introduction
to Modeltheoretic Semantics of Natural Language, Formal Logic and
DRT. Kluwer, Dordrecht, The Netherlands.
Marco Kuhlmann & Stephan Oepen. 2016. Towards a catalogue of
linguistic graph banks. Computational Linguistics, 42(4):819 – 827.
31References IV
Stephan Oepen & Jan Tore Lønning. 2006. Discriminant-based MRS
banking. In Proceedings of the 5th International Conference on
Language Resources and Evaluation, pages 1250 – 1255, Genoa, Italy.
Martha Palmer, Dan Gildea, & Paul Kingsbury. 2005. The Proposition
Bank. A corpus annotated with semantic roles. Computational
Linguistics, 31(1):71 – 106.
Sebastian Schuster & Christopher D. Manning. 2016. Enhanced English
Universal Dependencies. An improved representation for natural
language understanding tasks. In Proceedings of the 10th International
Conference on Language Resources and Evaluation, Portorož, Slovenia.
Aaron Steven White, Drew Reisinger, Keisuke Sakaguchi, Tim Vieira,
Sheng Zhang, Rachel Rudinger, Kyle Rawlins, & Benjamin Van Durme.
2016. Universal Decompositional Semantics on Universal Dependencies.
In Proceedings of the 2016 Conference on Empirical Methods in Natural
Language Processing, pages 1713 – 1723, Austin, TX, USA.
32References V
Daniel Zeman & Jan Hajič. 2020. FGD at MRP 2020: Prague
Tectogrammatical Graphs. In Proceedings of the CoNLL 2020 Shared
Task: Cross-Framework Meaning Representation Parsing, pages 33 – 39,
Online.
33You can also read
