Jabberwocky Parsing: Dependency Parsing with Lexical Noise - ScholarWorks@UMass Amherst
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Proceedings of the Society for Computation in Linguistics Volume 2 Article 13 2019 Jabberwocky Parsing: Dependency Parsing with Lexical Noise Jungo Kasai University of Washington, jkasai@cs.washington.edu Robert Frank Yale University, robert.frank@yale.edu Follow this and additional works at: https://scholarworks.umass.edu/scil Part of the Computational Linguistics Commons Recommended Citation Kasai, Jungo and Frank, Robert (2019) "Jabberwocky Parsing: Dependency Parsing with Lexical Noise," Proceedings of the Society for Computation in Linguistics: Vol. 2 , Article 13. DOI: https://doi.org/10.7275/h12q-k754 Available at: https://scholarworks.umass.edu/scil/vol2/iss1/13 This Paper is brought to you for free and open access by ScholarWorks@UMass Amherst. It has been accepted for inclusion in Proceedings of the Society for Computation in Linguistics by an authorized editor of ScholarWorks@UMass Amherst. For more information, please contact scholarworks@library.umass.edu.
Jabberwocky Parsing: Dependency Parsing with Lexical Noise Jungo Kasai ⇤ Robert Frank University of Washington Yale University jkasai@cs.washington.edu robert.frank@yale.edu Abstract Lee (2018)). These models exploit lexical infor- mation in a way that avoids some of the data spar- Parsing models have long benefited from the sity issues, by making use of distributed represen- use of lexical information, and indeed current tations (i.e., word embeddings) that support gen- state-of-the art neural network models for de- eralization across different words. pendency parsing achieve substantial improve- While humans certainly make use of lexical ments by benefiting from distributed represen- tations of lexical information. At the same information in sentence processing (MacDonald time, humans can easily parse sentences with et al., 1994; Trueswell and Tanenhaus, 1994), it unknown or even novel words, as in Lewis is also clear that we are able to analyze sentences Carroll’s poem Jabberwocky. In this paper, we in the absence of known words. This can be seen carry out jabberwocky parsing experiments, most readily by our ability to understand Lewis exploring how robust a state-of-the-art neu- Carroll’s poem, Jabberwocky (Carroll, 1883), in ral network parser is to the absence of lex- which open class items are replaced by non-words. ical information. We find that current pars- ing models, at least under usual training reg- imens, are in fact overly dependent on lexi- Twas brillig, and the slithy toves cal information, and perform badly in the jab- Did gyre and gimble in the wabe; berwocky context. We also demonstrate that All mimsy were the borogoves, the technique of word dropout drastically im- And the mome raths outgrabe proves parsing robustness in this setting, and also leads to significant improvements in out- Work in neurolinguistics and psycholinguistics of-domain parsing. has demonstrated the human capacity for unlexi- calized parsing experimentally, showing that hu- 1 Introduction mans can analyze syntactic structure even in pres- ence of pseudo-words (Stromswold et al., 1996; Since the earliest days of statistical parsing, lexi- Friederici et al., 2000; Kharkwal, 2014). cal information has played a major role (Collins, 1996, 1999; Charniak, 2000). While some of The word embeddings used by current lexical- the performance gains that had been derived from ized parsers are of no help in sentences with nonce lexicalization can be gotten in other ways (Klein words. Yet, it is at present unknown the degree and Manning, 2003), thereby avoiding increases to which these parsers are dependent on the in- in model complexity and problems in data sparsity formation contained in these embeddings. Pars- (Fong and Berwick, 2008), recent neural network ing evaluation on such nonce sentences is, there- models of parsing across a range of formalisms fore, critical to bridge the gap between cognitive continue to use lexical information to guide pars- models and data-driven machine learning models ing decisions (constituent parsing Dyer et al. in sentence processing. Moreover, understanding (2016)); dependency parsing: Chen and Manning the degree to which parsers are dependent upon (2014); Kiperwasser and Goldberg (2016); Dozat lexical information is also of practical importance. and Manning (2017); CCG parsing: Ambati et al. It is advantageous for a syntactic parser to gen- (2016); TAG parsing: Kasai et al. (2018); Shi and eralize well across different domains. Yet, heavy reliance upon lexical information could have detri- ⇤ Work done at Yale University. mental effects on out-of-domain parsing because 113 Proceedings of the Society for Computation in Linguistics (SCiL) 2019, pages 113-123. New York City, New York, January 3-6, 2019
lexical input will carry genre-specific information (Gildea, 2001). In this paper, we investigate the contribution of lexical information (via distributed lexical repre- sentations) by focusing on a state-of-the-art graph- based dependency parsing model (Dozat and Man- ning, 2017) in a series of controlled experiments. Concretely, we simulate jabberwocky parsing by adding noise to the representation of words in the input and observe how parsing performance varies. We test two types of noise: one in which words are replaced with an out-of-vocabulary word without a lexical representation, and a sec- ond in which words are replaced with others (with associated lexical representations) that match in their Penn TreeBank (PTB)-style fine-grained part of speech. The second approach is similar to the Figure 1: Biaffine parsing architecture. W and p denote method that Gulordava et al. (2018) propose to as- the word and POS embeddings. sess syntactic generalization in LSTM language models. model, and assess its ability to generalize over lex- In both cases, we find that the performance of ical noise. the state-of-the-art graph parser dramatically suf- fers from the noise. In fact, we show that the Input Representations The input for each word performance of a lexicalized graph-based parser is the concatenation of a 100-dimensional embed- is substantialy worse than an unlexicalized graph- ding of the word and a 25-dimensional embedding based parser in the presence of lexical noise, even of the PTB part of speech (POS). We initialize when the lexical content of frequent or function all word embeddings to be a zero vector and the words is preserved. This dependence on lexical out-of-vocabulary word is also mapped to a zero information presents a severe challenge when ap- vector in testing. The POS embeddings are ran- plying the parser to a different domain or hetero- domly initialized. We do not use any pretrained geneous data, and we will demonstrate that indeed word embeddings throughout our experiments in parsers trained on the PTB WSJ corpus achieve order to encourage the model to find abstractions much lower performance on the Brown corpus. over the POS embeddings. Importantly, PTB POS On the positive side, we find that word dropout categories also encode morphological features that (Iyyer et al., 2015), applied more aggressively should be accessible in jabberwocky situations. than is commonly done (Kiperwasser and Gold- We also conducted experiments by taking as input berg, 2016; de Lhoneux et al., 2017; Nguyen et al., words, universal POS tags, and character CNNs 2017; Ji et al., 2017; Dozat and Manning, 2017; (Ma and Hovy, 2016). We observed similar pat- Bhat et al., 2017; Peng et al., 2017, 2018), reme- terns throughout the experiments. While those ap- dies the susceptibility to lexical noise. Further- proaches can more easily scale to other languages, more, our results show that models trained on the one concern is that the character CNNs can en- PTB WSJ corpus with word dropout significantly code the identity of short words along side their outperform those trained without word dropout in morphological properties, and therefore would not parsing the out-of-domain Brown corpus, confirm- achieve a pure jabberwocky situation. For this ing the practical significance of jabberwocky pars- reason, we only present results using fine-grained ing experiments. POS. 2 Parsing Models Biaffine Parser Figure 1 shows our biaffine parsing architecture. Following Dozat and Here we focus ourselves on a graph-based parser Manning (2017) and Kiperwasser and Goldberg with deep biaffine attention (Dozat and Man- (2016), we use BiLSTMs to obtain features for ning, 2017), a state-of-the-art graph-based parsing each word in a sentence. We first perform unla- 114
beled arc-factored scoring using the final output use BiLSTMs layers with 400 units each. Input, vectors from the BiLSTMs, and then label the re- layer-to-layer, and recurrent dropout rates are all sulting arcs. Specifically, suppose that we score 0.33. The depth of all MLPs is 1, and the MLPs for edges coming into the ith word in a sentence i.e. unlabeled attachment and those for labeling con- assigning scores to the potential parents of the ith tain 500 (darc ) and 100 (drel ) units respectively. word. Denote the final output vector from the BiL- We train this model with the Adam algorithm to STM for the kth word by hk and suppose that hk minimize the sum of the cross-entropy losses from is d-dimensional. Then, we produce two vectors head predictions (si from Eq. 1) and label predic- from two separate multilayer perceptrons (MLPs) tions (`i from Eq. 2) with ` = 0.001 and batch with the ReLU activation: size 100 (Kingma and Ba, 2015). After each train- arc-dep ing epoch, we test the parser on the dev set. When hk = MLP(arc-dep) (hk ) labeled attachment score (LAS) does not improve harc-head k = MLP(arc-head) (hk ) on five consecutive epochs, training ends. arc-dep where hk and harc-head k are darc -dimensional 3 Dropout as Regularization vectors that represent the kth word as a dependent Dropout regularizes neural networks by randomly and a head respectively. Now, suppose the kth row setting units to zero with some probability dur- of matrix H (arc-head) is harc-head k . Then, the proba- ing training (Srivastava et al., 2014). In addi- bility distribution si over the potential heads of the tion to usual dropout, we consider applying word ith word is computed by dropout, a variant of dropout that targets entire si = softmax(H (arc-head) W (arc) hi arc-dep words and therefore entire rows in the lexical em- (1) bedding matrix (Iyyer et al., 2015). The intuition +H (arc-head) b(arc) ) we follow here is that a trained network will be where W (arc) 2 Rdarc ⇥darc and b(arc) 2 Rdarc . less dependent on lexical information, and more In training, we simply take the greedy maximum successful in a jabberwocky context, if lexical in- probability to predict the parent of each word. In formation is less reliably present during training. the testing phase, we use the heuristics formulated We consider a number of ways of word dropout. by Dozat and Manning (2017) to ensure that the Uniform Word Dropout Iyyer et al. (2015) resulting parse is single-rooted and acyclic. introduced the regularization technique of word Given the head prediction of each word in the dropout in which lexical items are replaced by sentence, we assign labeling scores using vectors the “unknown” word with some fixed probability obtained from two additional MLP with ReLU. p and demonstrated that it improves performance For the kth word, we obtain: for the task of text classification. Replacing words rel-dep with the out-of-vocabulary word exposes the net- hk = MLP(rel-dep) (hk ) works to out-of-vocabulary words that only occur hrel-head k = MLP(rel-head) (hk ) in testing. In our experiments, we will use word rel-dep dropout rates of 0.2, 0.4, 0.6, and 0.8. where hk , hrel-head k 2 Rdrel . Let pi be the in- dex of the predicted head of the ith word, and r be Frequency-based Word Dropout Dropping the number of dependency relations in the dataset. words with the same probability across the vo- Then, the probability distribution `i over the possi- cabulary might not behave as an ideal regularizer. ble dependency relations of the arc pointing from The network’s dependence on frequent words or the pi th word to the ith word is calculated by: function words is less likely to lead to overfitting (rel-dep) on the training data or corpus-specific properties, `i = softmax(hTpi(rel-head) U (rel) hi as the distribution of such words is less variable (2) +W (rel) (h(rel-head) + h(rel-head) pi ) + b(rel) ) across different corpora. To avoid penalizing the i networks for utilizing lexical information (in the where U (rel) 2 Rdrel ⇥drel ⇥r , W (rel) 2 Rr⇥drel , and form of word embeddings) for frequent words, b(rel) 2 Rr . Kiperwasser and Goldberg (2016) propose that We generally follow the hyperparameters cho- word dropout should be applied to a word with a sen in Dozat and Manning (2017). Specifically, we probability inversely proportional to the word’s 115
frequency. Specifically, they drop out each word POS preserved, and the latter zeroes outs the em- w that appears #(w) times in the training data beddings for words (i.e. replacing words with an with probability: out-of-vocabulary word). In either case, we keep ↵ POS input to the parsers intact. pw = (3) #(w) + ↵ Colorless Green Experiments Gulordava et al. Kiperwasser and Goldberg (2016) set ↵ = 0.25, (2018) propose a framework to evaluate the gen- which leads to relatively little word dropout. In eralization ability of LSTM language models that our WSJ training data, ↵ = 0.25 yields an ex- abstracts away from idiosyncratic properties of pected word dropout rate of 0.009 in training, an words or collocational information. In particu- order of magnitude less than commonly used rates lar, they generate nonce sentences by randomly in uniform word dropout. We experimented with replacing words in the original sentences while ↵ = 0.25, 1, 40, 352, 2536 where the last three preserving part-of-speech and morphological fea- values yield expected word dropout rates of 0.2, tures. This can be thought of as a computational 0.4, and 0.6 (the uniform dropout rates we con- approach to producing sentences that are “gram- sider). In fact, we will confirm that ↵ needs to be matical” yet meaningless, exemplified by the fa- much larger to significantly improve robustness to mous example “colorless green ideas sleep furi- lexical noise. ously” (Chomsky, 1957). Concretely, for each PTB POS category, we pick the 50 most frequent Open Class Word Dropout The frequency- words of that category in the training set and re- based word dropout scheme punishes the model place each word w in the test set by a word uni- less for relying upon frequent words in the train- formly drawn from the 50 most frequent words ing data. However, some words may occur fre- for w’s POS category. We consider three situa- quently in the training data because of corpus- tions: 1) full colorless green experiments where specific properties of the data. For instance, in all words are replaced by random words, 2) top the PTB WSJ training data, the word “company” 100 colorless green experiments where all words is the 40th most frequent word. If our aim is to but the 100 most frequent words are replaced by construct a parser that can perform well in dif- random words, and 3) open class colorless green ferent domains or across heterogeneous data, the experiments where the input word is replaced by networks should not depend upon such corpus- a random word if and only if the word is an open specific word senses. Hence, we propose to apply class word.1 word dropout only on open class (non-function) words with a certain probability. We experimented Jabberwocky Experiments One potential with open class word dropout rates of 0.38 and shortcoming with the approach above is that it 0.75 (where open class words are zeroed out 38% produces sentences which might violate con- or 75% of the time), corresponding to the expected straints that are imposed by specific lexical items, overall dropout rates of 0.2 and 0.4 respectively. but which are not represented by the POS cate- To identify open class words in the data we used gory. For instance, this approach could generate the following criteria. We consider a word as an a sentence like “it stays the shuttle” in which the open class word if and only if: 1) the gold UPOS is intransitive verb “stay” takes an object (Gulordava “NOUN”, “PROPN”, “NUM”, “ADJ”, or “ADV”, et al., 2018).2 Such a violation of argument or 2) the gold UPOS is “VERB” and the the gold structure constraints could mislead parsers (as XPOS (PTB POS) is not “MD” and the lemma is well as language models studied in Gulordava not “’be”, “have”, or “do”. et al. (2018)) and we will show that is indeed the 4 Experiments 1 We use the same criteria for open class words as in the We test trained parsers on input that contains open class word dropout. 2 This shortcoming might be overcome by using lexical two types of lexical noise, designed to as- resources like PropBank (Palmer et al., 2005) or NomBank sess their ability to abstract away from idiosyn- (Meyers et al., 2004) to guide word substitutions. In this pa- cratic/collocational properties of lexical items: 1) per, we do not do this, follow Gulordava et al.’s approach for the creation of colorless green sentences. We instead use the colorless green noise and 2) jabberwocky noise. jabberwocky manipulation to avoid creating sentences that The former randomly exchanges words with PTB violate selectional constraints. 116
case.3 To address this issue, we also experiment it is available, and may well perform less well with jabberwocky noise, in which input word when it is not. In fact, in jabberwocky and col- vectors are zeroed out. This noise is equivalent orless green settings, the absence of lexical infor- to replacing words with an out-of-vocabulary mation may lead to an underdetermination of the word by construction. Because fine-grained POS parse by the POS or word sequence, so that there information is retained in the input to the parser, is no non-arbitrary “gold standard” parse. As a the parser is still able to benefit from the kind of result, simply observing a performance drop of a morphological information present in Carroll’s parser in the face of lexical noise does not help poem. We again consider three situations 1) to establish an appropriate baseline with respect to full jabberwocky experiments where all word how well a parser can be expected to perform in embeddings are zeroed out, 2) top 100 jabber- a lexically noisy setting. We propose three base- wocky experiments where word embeddings for line parsers: 1) an unlexicalized parser where the all but the most frequent 100 words are zeroed network input is only POS tags, 2) a “top 100” out, and 3) open class jabberwocky experiments parser where the network input is only POS tags where the input word vector is zeroed out if and and lexical information for the 100 most frequent only if the word is an open class word. Open words and 3) a “function word” parser where the class jabberwocky experiments are the closest to network input is only POS tags and lexical infor- the situation when humans read Lewis Carroll’s mation for function words. Each baseline parser Jabberwocky.4 can be thought of as specialized to the correspond- ing colorless green and jabberwocky experiments. Out-of-domain experiments We also explore a For example, the unlexicalized parser gives us an practical aspect of our experiments with lexical upper bound for full colorless green and jabber- noise. We apply our parsers that are trained on wocky experiments because the parser is ideally the WSJ corpus to the Brown corpus and observe adapted to the unlexicalized situation, as it has no how parsers with various configurations perform.5 dependence on lexical information. Prior work showed that parsers trained on WSJ yield degraded performance on the Brown cor- Experimental Setup We use Universal Depen- pus (Gildea, 2001) despite the fact that the aver- dency representations obtained from converting age sentence length is shorter in the Brown cor- the Penn Treebank (Marcus et al., 1993) using pus (23.85 tokens for WSJ; 20.56 for Brown). We Stanford CoreNLP (ver. 3.8.0) (Manning et al., show that robustness to lexical noise improves out- 2014). We follow the standard data split: sections of-domain parsing. 2-21, 22, and 23 for training, dev, and test sets respectively. For the out-of-domain experiments, Baseline Parsers Lexical information is clearly we converted the Brown corpus in PTB again us- useful for certain parsing decisions, such as PP- ing Stanford CoreNLP into Universal Dependency attachment. As a result, a lexicalized parser representations.6 clearly should make use of such information when We only use gold POS tags in training for sim- 3 Prior work in psycholinguistics argued that verbs can plicity,7 but we conduct experiments with both in fact be used in novel argument structure constructions, gold and predicted POS tags. Experiments with and assigned coherent interpretations on the fly (Johnson and Goldberg, 2013). Our colorless green parsing experiments gold POS tags allow us to isolate the effect of lex- can be interpreted as a simulation for such situations. ical noise from POS tagging errors, while those 4 An anonymous reviewer notes that because of its greater with predicted POS tags simulate more practi- complexity, human performance on a jabberwocky version of the WSJ corpus may not be at the level we find when cal situations where POS input is not fully reli- reading the sentences of Lewis Carroll’s poem or in the psy- able. Somewhat surprisingly, however, we find cholinguistic work that has explored human ability to process that relative performance patterns do not change jabberwocky-like sentences. We leave it for future work to explore whether human performance in such complex cases even when using predicted POS tags. All pre- is indeed qualitatively different, and also whether the pattern 6 of results changes if we restrict our focus to a syntactically We exclude the domains of CL and CP in the Brown cor- simpler corpus, given a suitable notion of simplicity. pus because the Stanford CoreNLP converter encountered an 5 error. We initially intended to apply our trained parsers to the 7 Universal Dependency corpus (Nivre et al., 2015) as well for One could achieve better results by training a parser on out-of-domain experiments, but we found annotation incon- predicted POS tags obtained from jackknife training, but im- sistency and the problem of conversion from phrase structures proving normal parsing performance is not the focus of our to universal dependencies. We leave this problem for future. work. 117
dicted POS tags are obtained from a BiLSTM POS Gold Predicted Model UAS LAS UAS LAS tagger with character CNNs, trained on the same No Dropout 93.60.2 92.30.2 92.70.1 90.60.1 training data (sections 2-21) with hyperparameters Unlexicalized 88.00.1 85.40.1 87.10.1 83.80.1 from Ma and Hovy (2016) and word embeddings Top 100 92.50.1 90.80.1 91.70.1 89.20.1 Function 90.70.4 88.10.6 90.00.3 86.80.5 initialized with GloVe vectors (Pennington et al., Uniform 0.2 93.90.1 92.60.1 93.00.1 90.90.2 2014). We train 5 parsing models for each train- Uniform 0.4 94.00.1 92.50.1 93.00.1 90.80.1 ing configuration with 5 different random initial- Uniform 0.6 93.70.1 92.20.1 92.70.1 90.50.1 Uniform 0.8 93.00.1 91.40.1 92.10.1 89.70.2 izations and report the mean and standard devia- Freq 0.25 93.70.1 92.40.1 92.90.1 90.80.1 tion.8 We use the CoNLL 2017 official script for Freq 1 93.90.1 92.60.1 93.00.1 91.00.1 evaluation (Zeman et al., 2017). Freq 40 94.00.2 92.60.2 93.00.2 90.90.2 Freq 352 93.60.1 92.20.1 92.70.1 90.50.1 Freq 2536 92.90.1 91.40.1 92.00.1 89.70.1 5 Results and Discussions Open Cl 0.38 93.90.1 92.50.2 93.00.1 90.90.2 Open Cl 0.75 93.50.1 92.10.1 92.70.1 90.50.1 Normal Parsing Results Table 1 shows normal Table 1: Normal Parsing Results on the Dev Set. The parsing results on the dev set. In both gold and subscripts indicate the standard deviations. predicted POS experiments, we see a significant discrepancy between the performance in rows 2-4 and the rest, suggesting that lexicalization of a de- pendency parser greatly contributes to parsing per- formance; having access to the most frequent 100 Full Experiments Table 2 shows results for words (row 3) or the function words (row 4) recov- full colorless green and jabberwocky experiments. ers part of the performance drop from unlexical- The models without word dropout yield extremely ization (row 2), but the LAS differences from com- poor performance both in colorless and jabber- plete lexicalization (row 1, row 5 and below) are wocky settings, suggesting that a graph-based still significant. For each of the three word dropout parsing model learns to rely heavily on word in- schemes in gold POS experiments, we see a com- formation if word dropout is not performed. Here, mon pattern: performance improves up to a certain unlike the normal parsing results, we see mono- degree of word dropout (Uniform 0.2, Frquency- tone increasing performance as word dropout is based 1-40, Open Class 0.38), and it drops after more aggressively applied, and the performance as word dropout becomes more aggressive. This rises more dramatically. In particular, with uni- suggests that word dropout also involves the bias- form word dropout rate 0.2, full jabberwocky per- variance trade-off. Although performance gener- formance increases by more than 40 LAS points, ally degrades with predicted POS tags, the pat- suggesting the importance of the parser’s expo- terns of relative performance still hold. Again, sure to unknown words to abstract away from lex- for each of the three dropout schemes, there is ical information. Frequency-based word dropout a certain point in the spectrum of word dropout needs to be performed more aggressively (↵ 40) intensity that achieves the best performance, and than has previously been done for dependency such points are almost the same both in the models parsing (Kiperwasser and Goldberg, 2016; Dozat trained with gold and predicted POS tags. This is and Manning, 2017) in order to achieve robust- a little surprising because a higher word dropout ness to full lexical noise similar to that obtained rate encourages the model to rely more on POS with uniform word dropout with p > 0.2. Open input, and noisy POS information from the POS class word dropout does not bring any benefit tagger can work against the model. Indeed, we ob- to parsers in the full jabberwocky and colorless served this parallelism between experiments with green settings. This is probably because parsers gold and predicted POS tags consistently through- trained with open class word dropout has consis- out the colorless green and jabberwocky experi- tent access to function words, and omitting the lex- ments, and therefore we only report results with ical representations of the function words is very gold POS tags for the rest of the colorless green harmful to such parsers. Interestingly, in some and jabberwocky experiments for simplicity. of the cases, colorless green outperforms jabber- wocky performance, perhaps because noisy word 8 Our code is available at https://github.com/ information, even with argument constraint viola- jungokasai/graph_parser for easy replication. tions, is better than no word information. 118
Gold Predicted Colorless Jabberwocky Model UAS LAS UAS LAS Model UAS LAS UAS LAS No Dropout 89.70.1 87.50.1 88.80.1 85.70.1 No Dropout 62.60.2 56.30.1 51.92.3 39.11.9 Unlexicalized 83.00.1 79.30.2 81.30.1 76.70.1 Unlexicalized 88.00.1 85.40.1 88.00.1 85.40.2 Top 100 89.40.1 86.80.1 88.50.1 84.90.1 Top 100 71.70.4 67.10.3 72.70.3 68.20.5 Function 88.30.3 84.80.7 87.40.3 83.10.7 Function 69.20.9 62.10.7 58.83.0 39.83.4 Uniform 0.2 90.00.1 87.80.2 89.10.1 86.00.2 Uniform 0.2 74.00.2 69.10.2 85.70.3 82.70.3 Uniform 0.4 90.10.1 87.90.1 89.30.1 86.10.1 Uniform 0.4 76.90.3 72.30.2 87.10.1 84.30.1 Uniform 0.6 90.10.1 87.70.1 89.10.1 85.80.1 Uniform 0.6 79.20.2 75.00.2 87.70.1 85.00.1 Uniform 0.8 89.40.1 86.80.1 88.50.1 85.00.1 Uniform 0.8 82.00.3 78.50.3 88.00.1 85.40.1 Freq 0.25 89.90.2 87.70.2 89.10.2 86.00.2 Freq 0.25 62.90.2 56.40.1 55.01.4 43.42.5 Freq 1 90.20.2 88.10.2 89.30.1 86.30.1 Freq 1 63.60.4 57.10.1 60.11.7 48.83.6 Freq 40 90.70.2 88.50.3 89.80.2 86.70.2 Freq 40 67.50.7 61.60.6 76.41.0 72.01.2 Freq 352 90.30.1 88.00.1 89.30.1 86.10.1 Freq 352 74.50.5 69.70.5 82.90.4 79.50.6 Freq 2536 89.30.2 86.80.2 88.40.1 85.00.2 Freq 2536 82.60.2 78.80.3 86.50.4 85.40.2 Open Cl 0.38 90.30.2 88.10.2 89.40.1 86.20.2 Open Cl 0.38 65.00.5 58.80.4 53.72.6 36.83.6 Open Cl 0.75 90.30.1 88.00.1 87.40.3 86.10.1 Open Cl 0.75 66.70.2 60.50.3 53.81.0 34.01.3 Table 5: Brown CF Results. Table 2: Full Colorless Green and Jabberwocky Exper- iments on the Dev Set. Top 100 Experiments Table 3 shows the results of top 100 colorless green and jabberwocky ex- Colorless Jabberwocky periments. Performance by parsers trained with- Model UAS LAS UAS LAS out word dropout is substantially better than what No Dropout 85.50.1 82.70.1 86.40.8 83.40.9 is found in the full colorless green or jabber- Unlexicalized 88.00.1 85.40.1 88.00.1 85.40.2 wocky settings. However, the performance is still Top 100 92.50.1 90.80.1 92.50.1 90.80.1 Function 88.70.4 85.40.7 90.80.3 88.00.6 much lower than the unlexicalized parsers (2.7 Uniform 0.2 87.50.2 84.90.2 90.20.6 88.20.6 LAS points for colorless green and 2.0 LAS points Uniform 0.4 88.50.2 86.00.2 90.80.4 88.90.4 for jabberwocky), meaning that the parser without Uniform 0.6 89.20.2 86.80.1 91.00.3 89.10.2 Uniform 0.8 89.70.1 87.40.2 90.60.2 88.60.2 word dropout has significant dependence on less Freq 0.25 85.80.2 83.00.2 87.80.6 85.00.6 frequent words. On the other hand, parsers trained Freq 1 86.10.1 83.30.1 88.90.4 86.30.4 Freq 40 88.10.2 85.50.1 90.90.2 88.70.4 with a high enough word dropout rate outper- Freq 352 89.70.1 87.40.1 91.90.2 90.00.2 form the unlexicalized parser (e.g., uniform 0.4, Freq 2536 90.70.2 88.60.3 91.30.2 89.30.3 frequency-based ↵ = 40, and open class 0.38). Open Cl 0.38 88.60.3 86.20.3 90.60.2 88.30.2 Open Cl 0.75 89.60.1 87.40.2 90.80.3 88.00.6 Frequency-based word dropout is very effective. Recall that ↵ = 352, 2536 correspond to the ex- Table 3: Top 100 Colorless Green and Jabberwocky pected word dropout rates of 0.4 and 0.6. The two Experiments on the Dev Set. configurations yield better results than uniform 0.4 and 0.6. Colorless Jabberwocky Open Class Experiments Table 4 gives the re- Model UAS LAS UAS LAS sults of open class colorless green and jabber- No Dropout 84.10.3 81.30.2 84.90.6 82.10.6 wocky experiments. We see similar patterns to the Unlexicalized 88.00.1 85.40.1 88.00.1 85.40.2 Top 100 90.50.2 88.40.2 91.80.1 89.90.1 top 100 jabberwocky experiments except that open Function 90.70.4 88.10.6 90.70.4 88.10.6 class word dropout performs better and frequency- Uniform 0.2 87.40.2 84.80.2 89.70.7 87.80.7 based word dropout performs worse, which natu- Uniform 0.4 88.30.2 85.90.3 90.60.4 88.70.4 Uniform 0.6 89.20.2 86.80.2 90.90.3 89.00.3 rally follows from the way we train the parsers. Uniform 0.8 89.90.2 87.70.1 90.50.3 88.60.2 Freq 0.25 84.70.2 82.00.2 86.70.5 84.30.6 Out-of-domain Parsing Table 5 reports pars- Freq 1 85.20.4 82.40.4 88.10.6 85.90.6 ing performance on the CF domain in the Brown Freq 40 87.70.2 85.20.2 91.20.3 89.20.3 corpus.9 POS tagging accuracy was 96.4%, rela- Freq 352 89.50.2 87.30.1 92.00.1 90.20.1 Freq 2536 90.70.2 88.70.2 91.70.2 89.90.1 tively low as compared to in-domain performance Open Cl 0.38 89.00.2 86.70.3 92.10.2 90.30.2 in WSJ. Gildea (2001) demonstrated that remov- Open Cl 0.75 90.70.1 88.40.2 92.40.1 90.70.1 ing lexical information from lexicalized PCFG Table 4: Open Class Colorless Green and Jabberwocky does not deteriorate out-of-domain parsing perfor- Experiments on the Dev Set. 9 We found similar patterns in relative performance when applied to the other domains of the Brown corpus. 119
Colorless Jabberwocky relation Pre Rec F1 Prec Rec F1 50 nsubj 87.8 86.2 87.0 91.0 90.0 90.5 dobj 79.2 81.4 80.3 83.0 90.7 86.7 Accuracy (%) iobj 29.4 27.8 28.6 71.4 55.6 62.5 csubj 30.4 46.7 36.8 33.3 60.0 42.9 ccomp 75.3 75.3 75.3 77.2 78.2 77.7 45 xcomp 69.0 69.2 69.1 78.7 64.3 70.8 others 83.6 83.6 83.6 86.7 86.6 86.6 Table 6: Performance Breakdown by Dependency Re- 40 lation. 0.0 0.2 0.4 0.6 0.8 Figure 2: POS Prediction Results from the Word Em- mance in the Brown corpus. In contrast, our re- bedding Induced in Varying Uniform Dropout Rates. sults show that lexicalization of the neural network dependency parsers via distributed representations facilitates parsing performance. Most of the gains from lexicalization stem from the 100 most fre- the word embeddings themselves might also rep- quent words (row 3) and function words (row 4), resent POS information. It might instead be the but we get a significant improvement by perform- case that what word dropout does is merely dis- ing relatively aggressive word dropout (↵ = 40). couraging the model from constructing grammati- This confirms the practical significance of jabber- cal abstractions in the word embeddings, depend- wocky parsing experiments. ing instead on the information present in the POS embeddings. 6 Analysis of Results To test this possibility, we attempt to use a sim- Colorless Green vs. Jabberwocky Earlier we ple softmax regression on top of the induced word hypothesized that colorless green lexical noise embeddings to detect the presence of grammati- could mislead the parser by violating argument cal category information. Concretely, we find the structure constraints. And in fact, we saw in the set of words that occur 100 times or more in the previous section that jabberwocky parsing results training data whose unigram probability for the are generally better than colorless green parsing most frequent POS tag is greater than 0.5. In results. In order to further test our hypothesis, this way, we obtain a set of pairs of a word and we provide a breakdown of parsing performance its most frequent POS tag, and we then randomly broken down by dependency type (Nivre et al., split the set to run 5-fold cross validation. Figure 2 2015). Table 6 provides the open class perfor- shows POS prediction accuracy over varying uni- mance breakdown for a parser trained with uni- form word dropout rates. These results do not sup- form word dropout rate 0.2. We observe that in the port the idea that word dropout reduces the repre- colorless green situation, the parser particularly sentation of grammatical category information in suffers in “iobj” and “dobj”, consistent with what word embedding. On the contrary, word dropout we would expect for parsing sentences with argu- rates of 0.2 through 0.6 leads to word embeddings ment structure violations. For example, the col- that better represent POS. orless green scheme does not prevent us from re- placing a ditransitive verb with a non-ditransitive verb. This result validates our hypothesis, and the colorless green scheme is limited as a framework Parser Weight Sizes What then is the effect of to purely assess parser’s generalization ability. word dropout on the network? Figure 3 shows L2 norms of the trained POS embeddings (46 by 25) Trained Word Embeddings The logic of our and the input/output gate weights for POS input in jabberwocky experiments and the role of word the first BiLSTM layer (400 by 25) across vary- dropout involves the assumption that the parser ing uniform word dropout rates. We can observe will succeed in the absence of lexical information that L2 norm for each becomes larger as the word if it depends more heavily on the grammatical cat- dropout rate increases. This suggests that what egory information present in the POS embeddings. word dropout does is to encourage the model to However, this fails to allow for the possibility that make greater use of POS information. 120
30 Michael Collins. 1999. . Head-Driven Statistical Mod- Embeddings Input Gate els for Natural Language Parsing. Ph.D. thesis, Output Gate 25 University of Pennsylvania. L2 Norm Timothy Dozat and Christopher Manning. 2017. Deep 20 biaffine attention for neural dependency parsing. In ICLR. 15 Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. 2016. Recurrent neural network 0.0 0.2 0.4 0.6 0.8 grammars. In NAACL. Sandiway Fong and Robert C. Berwick. 2008. Tree- Figure 3: POS Embeddings and Weight Sizes over Uni- bank parsing and knowledge of language : A cogni- form Word Dropout Rates. tive perspective. In CogSci. Angela D. Friederici, Mia Viktoria Meyer, and D. Yves 7 Conclusion and Future Work von Cramon. 2000. Auditory language comprehen- sion: an event-related fmri study on the processing We conducted extensive analysis on the robust- of syntactic and lexical information. Brain and lan- ness of a state-of-the-art graph-based dependency guage, 75 3:289–300. parser against lexical noise. Our experiments showed that parsers trained under usual regimens Daniel Gildea. 2001. Corpus variation and parser per- formance. In EMNLP. performed poorly in the face of lexical noise. However, we demonstrated that the technique of Kristina Gulordava, Piotr Bojanowski, Edouard Grave, word dropout, when applied aggressively, reme- Tal Linzen, and Marco Baroni. 2018. Colorless dies this problem without sacrificing parsing per- green recurrent networks dream hierarchically. In NAACL. Association for Computational Linguistics. formance. Word dropout is commonly used in lit- erature, but our results provide further guidance Mohit Iyyer, Varun Manjunatha, Jordan L. Boyd- about how it should be used in the future. In future Graber, and Hal Daumé. 2015. Deep unordered work, we would like to compare our results with composition rivals syntactic methods for text clas- different parsing architectures such as transition- sification. In ACL. based parsers. Tao Ji, Yuanbin Wu, and Man Lan. 2017. A fast and lightweight system for multilingual dependency parsing. In CoNLL Shared Task. References Matt A. Johnson and Adele E. Goldberg. 2013. Ev- Bharat Ram Ambati, Tejaswini Deoskar, and Mark idence for automatic accessing of constructional Steedman. 2016. Shift-reduce ccg parsing using meaning: Jabberwocky sentences prime associ- neural network models. In NAACL. ated verbs. Language and Cognitive Processes, 28(10):14391452. Riyaz Ahmad Bhat, Irshad Ahmad Bhat, and Dipti Misra Sharma. 2017. Leveraging newswire Jungo Kasai, Robert Frank, Pauli Xu, William Merrill, treebanks for parsing conversational data with argu- and Owen Rambow. 2018. End-to-end graph-based ment scrambling. In IWPT. TAG parsing with neural networks. In NAACL. As- sociation for Computational Linguistics. Lewis Carroll. 1883. Through the Looking-Glass. Macmillan and Co., New York. Gaurav Kharkwal. 2014. Taming the Jabberwocky: Eugene Charniak. 2000. A maximum-entropy-inspired Examining Sentence Processing with Novel Words. parser. In ANLP. Ph.D. thesis, Rutgers University. Danqi Chen and Christopher Manning. 2014. A fast Diederik P. Kingma and Jimmy Lei Ba. 2015. ADAM: and accurate dependency parser using neural net- A Method for Stochastic Optimization. In ICLR. works. In EMNLP, pages 740–750, Doha, Qatar. Association for Computational Linguistics. Eliyahu Kiperwasser and Yoav Goldberg. 2016. Sim- ple and accurate dependency parsing using bidirec- Noam Chomsky. 1957. Syntactic Structures. Mouton tional LSTM feature representations. TACL, 4:313– & Co., Berlin, Germany. 327. Michael Collins. 1996. A new statistical parser based Dan Klein and Christopher D. Manning. 2003. Accu- on bigram lexical dependencies. In ACL. rate unlexicalized parsing. In ACL. 121
Miryam de Lhoneux, Yan Shao, Ali Basirat, Eliyahu Szántó, Takaaki Tanaka, Reut Tsarfaty, Sumire Ue- Kiperwasser, Sara Stymne, Yoav Goldberg, and matsu, Larraitz Uria, Viktor Varga, Veronika Vincze, Joakim Nivre. 2017. From raw text to universal de- Zdeněk Žabokrtský, Daniel Zeman, and Hanzhi pendencies - look, no tags! In CoNLL Shared Task. Zhu. 2015. Universal dependencies 1.2. LIN- DAT/CLARIN digital library at the Institute of For- Xuezhe Ma and Eduard Hovy. 2016. End-to-end mal and Applied Linguistics, Charles University. sequence labeling via bi-directional LSTM-CNNs- CRF. In ACL, pages 1064–1074, Berlin, Germany. Martha Palmer, Dan Gildea, and Paul Kingsbury. 2005. Association for Computational Linguistics. The proposition bank: A corpus annotated with se- mantic roles. Computational Linguistics, 31(1). Maryellen C. MacDonald, Neal J. Pearlmutter, and Mark S. Seidenberg. 1994. Lexical nature of syn- Hao Peng, Sam Thomson, and Noah A. Smith. 2017. tactic ambiguity resolution. Psychological Review, Deep multitask learning for semantic dependency 101:676–703. parsing. In ACL. Christopher D. Manning, Mihai Surdeanu, John Bauer, Hao Peng, Sam Thomson, Swabha Swayamdipta, and Jenny Finkel, Steven J. Bethard, and David Mc- Noah A. Smith. 2018. Learning joint semantic Closky. 2014. The Stanford CoreNLP natural lan- parsers from disjoint data. In NAACL. guage processing toolkit. In Association for Compu- tational Linguistics (ACL) System Demonstrations. Jeffrey Pennington, Richard Socher, and Christo- pher D. Manning. 2014. Glove: Global vectors for Mitchell Marcus, Beatrice Santorini, and Mary Ann word representation. In EMNLP. Marcinkiewicz. 1993. Building a large annotated Tianze Shi and Lillian Lee. 2018. Valency-augmented corpus of english: The Penn Treebank. Computa- dependency parsing. In EMNLP. Association for tional Linguistics, 19(2):313–330. Computational Linguistics. Adam Meyers, Ruth Reeves, Catherine Macleod, Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Rachel Szekely, Veronika Zielinska, Brian Young, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. and Ralph Grishman. 2004. The NomBank project: Dropout: A simple way to prevent neural networks An interim report. In Proceedings of the Workshop from overfitting. Journal of Machine Learning Re- on Frontiers in Corpus Annotation at HLT-NAACL. search, 15:1929–1958. Dat Quoc Nguyen, Mark Dras, and Mark Johnson. Karen Stromswold, David O. Caplan, Nathaniel M. 2017. A novel neural network model for joint pos Alpert, and Scott L. Rauch. 1996. Localization of tagging and graph-based dependency parsing. In syntactic comprehension by positron emission to- CoNLL Shared Task. mography. Brain and language, 52 3:452–73. Joakim Nivre, Željko Agić, Maria Jesus Aranzabe, John C. Trueswell and Michael K. Tanenhaus. 1994. Masayuki Asahara, Aitziber Atutxa, Miguel Balles- Toward a lexicalist framework for constraint-based teros, John Bauer, Kepa Bengoetxea, Riyaz Ah- syntactic ambiguity resolution. In Charles Clifton, mad Bhat, Cristina Bosco, Sam Bowman, Giuseppe Jr., Lyn Frazier, and Keith Rayner, editors, Per- G. A. Celano, Miriam Connor, Marie-Catherine spectives on Sentence Processing, pages 155–179. de Marneffe, Arantza Diaz de Ilarraza, Kaja Do- Lawrence Erlbaum Associates, Hillsdale, NJ. brovoljc, Timothy Dozat, Tomaž Erjavec, Richárd Farkas, Jennifer Foster, Daniel Galbraith, Filip Daniel Zeman, Martin Popel, Milan Straka, Jan Ha- Ginter, Iakes Goenaga, Koldo Gojenola, Yoav jic, Joakim Nivre, Filip Ginter, Juhani Luotolahti, Goldberg, Berta Gonzales, Bruno Guillaume, Jan Sampo Pyysalo, Slav Petrov, Martin Potthast, Fran- Hajič, Dag Haug, Radu Ion, Elena Irimia, An- cis Tyers, Elena Badmaeva, Memduh Gokirmak, ders Johannsen, Hiroshi Kanayama, Jenna Kan- Anna Nedoluzhko, Silvie Cinkova, Jan Hajic jr., erva, Simon Krek, Veronika Laippala, Alessan- Jaroslava Hlavacova, Václava Kettnerová, Zdenka dro Lenci, Nikola Ljubešić, Teresa Lynn, Christo- Uresova, Jenna Kanerva, Stina Ojala, Anna Mis- pher Manning, Cătălina Mărănduc, David Mareček, silä, Christopher D. Manning, Sebastian Schuster, Héctor Martı́nez Alonso, Jan Mašek, Yuji Mat- Siva Reddy, Dima Taji, Nizar Habash, Herman Le- sumoto, Ryan McDonald, Anna Missilä, Verginica ung, Marie-Catherine de Marneffe, Manuela San- Mititelu, Yusuke Miyao, Simonetta Montemagni, guinetti, Maria Simi, Hiroshi Kanayama, Valeria de- Shunsuke Mori, Hanna Nurmi, Petya Osenova, Lilja Paiva, Kira Droganova, Héctor Martı́nez Alonso, Øvrelid, Elena Pascual, Marco Passarotti, Cenel- Çağr Çöltekin, Umut Sulubacak, Hans Uszkor- Augusto Perez, Slav Petrov, Jussi Piitulainen, Bar- eit, Vivien Macketanz, Aljoscha Burchardt, Kim bara Plank, Martin Popel, Prokopis Prokopidis, Harris, Katrin Marheinecke, Georg Rehm, Tolga Sampo Pyysalo, Loganathan Ramasamy, Rudolf Kayadelen, Mohammed Attia, Ali Elkahky, Zhuoran Rosa, Shadi Saleh, Sebastian Schuster, Wolfgang Yu, Emily Pitler, Saran Lertpradit, Michael Mandl, Seeker, Mojgan Seraji, Natalia Silveira, Maria Simi, Jesse Kirchner, Hector Fernandez Alcalde, Jana Str- Radu Simionescu, Katalin Simkó, Kiril Simov, nadová, Esha Banerjee, Ruli Manurung, Antonio Aaron Smith, Jan Štěpánek, Alane Suhr, Zsolt Stella, Atsuko Shimada, Sookyoung Kwak, Gustavo 122
Mendonca, Tatiana Lando, Rattima Nitisaroj, and Josie Li. 2017. CoNLL 2017 shared task: Mul- tilingual parsing from raw text to universal depen- dencies. In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Univer- sal Dependencies, pages 1–19, Vancouver, Canada. Association for Computational Linguistics. 123
You can also read