Scaling the Tower of Babel Fish: An Analysis of the Machine Translation of Legal Information
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Scaling the Tower of Babel Fish:
An Analysis of the Machine Translation of Legal
Information*
Sarah Yates**
Ms. Yates evaluates the accuracy of Babel Fish in translating legal informa-
tion by comparing Babel Fish translations of law-oriented texts in Spanish and
German to professional translations. Most contained severe errors that altered
the meaning. She concludes that Babel Fish is not appropriate for most uses
in law libraries.
¶1 In late 1997, Babel Fish (http://babelfish.altavista.com) debuted on the AltaVista
Web site.1 Babel Fish is a “machine translation” program that allows Internet users
to translate blocks of text or entire Web pages automatically. What was a novelty in
the late 1990s is now widely used—by law librarians, among others.2 This article
presents the results of a study that I conducted to evaluate Babel Fish’s accuracy
in translating texts of interest to law librarians and law library users.
¶2 It should go without saying that computers cannot translate as well as
trained professionals. As Scott Bennett explains, “Anyone who expects [Babel
Fish] to match a professional translator’s skill is asking far more than the system
was intended to do.”3 On Babel Fish’s FAQ page, AltaVista advises users, “Expect
Babel Fish to allow you to grasp the general intent of the original, not to produce
a polished translation.”4
¶3 The purpose of this article is not to prove that Babel Fish is less accurate
than professional translation, which is probably not even in question. Instead,
I attempt to determine whether Babel Fish succeeds according to its own stan-
dards—that is, whether it provides translations accurate enough for users in the
law library community to “grasp the general intent of the original.”
¶4 Babel Fish was selected as the basis for this study because, in addition to
being the first machine translation system freely available on the Web, today it
* © Sarah Yates, 2006.
** Foreign Law and Rare Book Cataloger, University of Minnesota Law Library, Minneapolis,
Minnesota. The author wishes to thank Connie Lenz and Suzanne Thorpe for their helpful comments,
and especially Mary Rumsey for her help at every stage of the writing of this article.
1. Babelfish, http://www.infotektur.com/demos/babelfish/en.html (last visited Apr. 5, 2006).
2. See Sarah Yates, I Need This in English, AALL SPECTRUM, Apr. 2005, at 8. I interviewed several
foreign, comparative, and international law librarians and found that even those most skeptical about
the quality of Babel Fish and other Web-based translators use them occasionally.
3. Winfield Scott Bennett, Taking the Babble out of Babel Fish, LANGUAGE INT’L, June 2000, at 20, 20.
4. AltaVista, Babel Fish FAQ, Why Are Some Translations Lesser Quality? http://www.altavista.com/
help/babelfish/babel_faq#3 (last visited Apr. 3, 2006).
481
98n3_2p.indd 481 7/19/2006 12:08:11 PM482 Law Library Journal [Vol. 98:3
is one of the most heavily used and arguably the most well known.5 It uses tech-
nology designed by Systran, widely considered one of the premier companies
in the machine translation industry.6 Systran’s technology is available elsewhere
on the Web, including Google’s language tools7 and on its own Web site (www
.Systransoft.com).
¶5 I examined translations of portions of Mexico’s and Germany’s civil codes,
as well as press releases from the foreign ministries of both countries, and found
that the overall translation quality was very poor, even when measured against the
“general intent” standard. These poor results lead me to recommend only very
limited use of Babel Fish and similar programs in law libraries.
¶6 This article also contains some background information for readers who
may not be familiar with machine translation or legal translation. It begins with
a discussion of translation in general, which is necessary for understanding the
special issues of machine translation and legal translation.
Translation in General
¶7 Anyone who has attempted to translate a document by simply looking up each
word in a bilingual dictionary realizes that this method does not work. The translator
must understand the language at the morphological, lexical, and syntactical levels;
some would argue that understanding at the cultural level is equally indispensable.8
¶8 The morphological level is the level of morphemes, which are the smallest
meaning-bearing units of a language.9 In vastly simplified terms, this means word
roots and affixes. A conjugated verb such as walks is made up of two morphemes:
walk, which conveys the “main” meaning, and –s, which conveys the meaning
“third person singular, present tense.” Most dictionaries are useless to anyone
without a basic knowledge of how morphemes form words in a given language,
because they only list the basic forms of words. Consider how large a dictionary
would have to be to contain every inflected form of every English root verb: not
just walk, for example, but also walked, walks, and walking. The problem would be
much worse in more heavily inflected languages, where, for example, there might
be six or more affixes for a single tense. And that is not even considering nouns or
any other parts of speech.
¶9 Lexical understanding is necessary for obvious reasons: one needs to
understand what words mean to translate them accurately. This can be accom-
5. Bennett, supra note 3, at 20. Bennett quotes the Systran site as reporting that Babel Fish was used a
million times a day, as of 2000.
6. Roy Balleste, Could You Translate This Site for Me? Rating the Usefulness of Translation Software
Programs for Law Library Web Sites, AALL SPECTRUM, July 2001, at 4, 5; Kenneth Fink,
Translations-To-Go: Software for a Changing World, SEARCHER, Oct. 2001, at 33, 33.
7. Mary Flanagan & Steve McClure, SYSTRAN and the Reinvention of MT, IDC BULL. 26459 (Jan.
2002), http://www.systransoft.com/IDC/26459.html.
8. EDWIN GENTZLER, CONTEMPORARY TRANSLATION THEORIES 75, 193–95 (2d rev. ed. 2001).
9. MONA BAKER, IN OTHER WORDS: A COURSEBOOK ON TRANSLATION 11 (1992).
98n3_2p.indd 482 7/19/2006 12:08:11 PM2006-27] Scaling the Tower of Babel Fish 483
plished largely through the use of dictionaries, both bilingual and monolingual.
Homonyms present a problem for lexical understanding, although this problem
can usually be solved by examining the context in which the questionable word
appears. Suppose a translator comes across a word like bear. The meaning of bear
depends on whether it appears in the sentence “I can’t bear the pain!” or in “Does
a bear picnic in the woods?” Sometimes, however, even the whole sentence is not
enough context. In the command, “Go to the bank and wait for me,” does bank
refer to a place where money is kept, or the shore of a river?
¶10 Syntax is how words are put together into larger units. As Baker puts it,
“Syntax covers the grammatical structure of groups, clauses, and sentences.”10
No two languages have exactly the same syntactic structure. A simple example
is the placement of adjectives: after the noun (usually) in languages like French
and Spanish, but before the noun in languages like English and German. A more
complex example is the placement of verbs in German sentences compared to
their placement in English sentences. In translation, it is important to structure
sentences in a way that is syntactically correct in the target language (the language
translated into), not just to convey the proper meaning, but also to ensure read-
ability of the sentence.
¶11 Syntax and morphology have sometimes overlapping roles in form-
ing meaningful linguistic structures. In the English sentence, “The girl kissed
the boy,” we understand that the girl is the subject of the sentence (the kisser)
because it precedes the verb, and the boy is the direct object (the recipient of
the kiss) because it follows the verb. However, in some languages the positions
of the subject and the direct object are interchangeable because their grammati-
cal functions are marked morphologically instead. The above sentence could be
translated into German as either “Das Mädchen küsste den Jungen” or “Den
Jungen küsste das Mädchen.” In either sentence, it is clear that the boy is the
recipient of the kiss because den Jungen is in the accusative case. Differences
in the morphological systems of two languages can have an impact on syntax in
translation, and vice versa.
¶12 The cultural dimension of translation is strongly related to the mean-
ings of words and phrases. Cultural differences often make translation difficult
because they can result in a lack of equivalent terms among languages. One of
the most troubling culture-related difficulties arises when terms refer to cul-
ture-specific concepts, which have no lexical equivalents in languages spoken
where the concept is unknown. Baker mentions the English word privacy as
an example of “a very ‘English’ concept which is rarely understood by people
from other cultures.”11
10. Id. at 83.
11. Id. at 21.
98n3_2p.indd 483 7/19/2006 12:08:11 PM484 Law Library Journal [Vol. 98:3
Machine Translation
¶13 Machine translation (MT) is the term for using a computer to translate from
one natural language to another. Machine translation is nothing new—the first
MT programs originated in the mid-1940s and were among the first nonnumeri-
cal applications of computers.12 There are three methods of machine translation:
direct, a modified version of which is used by Systran’s Babel Fish;13 and trans-
fer and interlingua, which are not discussed here. Direct translation is the oldest
method and, some would say, the most primitive.14 In an article for Wired, Steve
Silberman writes:
While it may seem like the forerunner of a new species of Web sites, the Babel Fish is
more like the coelacanth discovered in an Indonesian market in 1997: a prehistoric survi-
vor, the descendant of a race of MT dinosaurs who once roamed the Earth, mighty with
the ambitions of the first generation of AI theorists and computer linguists.15
¶14 In direct translation systems, some linguistic analysis is performed at three
of the four levels mentioned above. The analysis is most detailed at the morpho-
logical and lexical levels; syntactic analysis is minimal. Cultural analysis, not
surprisingly, is nonexistent.
¶15 The first analysis performed in machine translation is morphological:
breaking up individual words into roots and affixes.16 Basic morphological analy-
sis is necessary to keep the bilingual dictionary reasonably small. It is much more
efficient for the dictionary to recognize the verb walk and the third person singular
present tense ending –s as distinct units than to include separate entries for walk
and walks, not to mention walked and walking. Irregular forms, such as run and
ran, do require separate entries in the dictionary; while humans can easily recog-
nize these as forms of the same verb, computers cannot make the same association
without being explicitly programmed to do so.
¶16 After the morphological analysis has been completed, the main work of
the translation is completed by simply replacing words (or their components) with
corresponding terms in the target language.17 This means that the MT system’s
bilingual dictionary is of the utmost importance. It also underscores some of the
shortcomings of the direct translation method.
¶17 The first and most obvious is the problem of homonyms or, more spe-
cifically, homographs. Homographs are two or more words that are spelled alike,
regardless of their pronunciation, such as tear (the byproduct of crying) and tear
12. W. JOHN HUTCHINS, MACHINE TRANSLATION: PAST, PRESENT, FUTURE 16 (1986). The early origin of
machine translation is the reason for its old-fashioned-sounding name.
13. Paul A. Watters & Malti Patel, Semantic Processing Performance of Internet Machine Translation
Systems, 9 INTERNET RES. 153, 154 (1999).
14. W. JOHN HUTCHINS & HAROLD L. SOMERS, AN INTRODUCTION TO MACHINE TRANSLATION 72 (1992).
15. Steve Silberman, Talking to Strangers, WIRED, May 2000, at 224, 226.
16. HUTCHINS & SOMERS, supra note 14, at 72.
17. Id.
98n3_2p.indd 484 7/19/2006 12:08:11 PM2006-27] Scaling the Tower of Babel Fish 485
(rip). Since no semantic analysis is performed in pure direct translation systems,18
the program cannot know which of two or more possible corresponding words
in the target language is appropriate in a given context. In fact, without semantic
analysis, there can be no understanding of context at all.
¶18 Syntax presents a problem as well. Systran and other direct translation sys-
tems do perform some rudimentary reordering of sentences in the target language
to make the output more understandable.19 However, one of the most jarring prob-
lems with machine-translated sentences, especially complex ones such as those
in legal texts, is that the syntax is sometimes so strange as to make the sentences
nearly incomprehensible.20
¶19 Hutchins and Somers sum up the direct translation method as follows:
The severe limitations of this approach should be obvious. It can be characterized as
‘word-for-word’ translation with some local word-order adjustment. It gave the kind of
translation quality that might be expected from someone with a very cheap bilingual dic-
tionary and only the most rudimentary knowledge of the grammar of the target language:
frequent mistranslations at the lexical level and largely inappropriate syntax structures
which mirrored too closely those of the source language.21
¶20 One more note about the translation method used by Systran is neces-
sary. MT systems can be either multilingual, meaning that they translate among
three or more languages, or bilingual, meaning that they translate between two
languages. While Systran offers translation to and from several languages, each
“language pair” is designed as a separate program, so it is really a bilingual
system, or a series of bilingual systems. In fact, the system for each language
pair comprises two separate unilateral bilingual dictionaries and translation pro-
grams.22 The English to German translation program, for example, is separate
from the German to English program. This explains some unexpected quirks,
such as words that can be translated from one language to the other but not vice
versa. For example, the English sentence “My son eats broccoli” is translated
correctly into German as “Mein Sohn ißt Brokkoli”; however, the German sen-
tence is translated into English as “My son eats Brokkoli.” The word Brokkoli is
apparently not in the German-English dictionary, even though broccoli is in the
English-German dictionary.
18. Id.
19. Id.
20. For example, the translation of one sentence in this study, paragraph 627 of the German civil code,
is: “With a service relationship, which is not employer-employee relationship in the sense § 622, the
notice is also without the condition designated in § 626 permissible, if to the service the obligated
one, without standing in a continuing service relationship with firm purchases, services of higher kind
to carry out has, which due to special confidence to be transferred to tend.”
21. HUTCHINS & SOMERS, supra note 14, at 72.
22. Id. at 70.
98n3_2p.indd 485 7/19/2006 12:08:11 PM486 Law Library Journal [Vol. 98:3
Special Problems in Legal Translation
¶21 Legal language has many peculiarities that make it particularly difficult to
translate. Peter M. Tiersma writes that legal language “diverges in many ways
from ordinary speech, far more than the technical languages of most other profes-
sions.”23
¶22 Tiersma discusses many of the characteristics of legal English that set it
apart from ordinary language. Many of these are specific to English, but others are
more nearly universal in legal language. For example, his comment that “long and
complex sentences with unusual word order and other odd features make legal lan-
guage convoluted, cumbersome, and hard to comprehend”24 applies to the excerpts
from the Mexican and German civil codes used in this study equally well as it does
to the texts of comparable laws in English.
¶23 In his discussion of the legal codes of Mexico, Francisco Avalos explains
that “[c]odes in the civil law tradition have been written through the years on the
assumption that using a rational scholarly process, rules and laws can be formu-
lated to apply to most all situations that may arise. As a result, codes tend to be
very detailed and vast in size.”25 The German language in particular has a reputa-
tion for being complex and greatly detailed, especially in legal contexts. Howard
Fisher writes:
Germany has one of the most scientific legal systems anywhere in the world. For a per-
son with English as his or her mother tongue to seek to understand that system is not
easy, even with a legal education. Why is this so? The principal reason lies in the fact
that, for historical reasons, the methods and sources of German and English law have
developed differently. Another reason is the German language and its seemingly endless
supply of formal terminology. Certainly, legal and everyday language are not the same
anywhere, but where can this be more so than in Germany?26
¶24 Additional differences between legal and ordinary language that are not
specific to English include differences in the degree of precision,27 legal archa-
isms,28 new terminology for new areas of law,29 and technical terms.30
¶25 Perhaps the largest obstacle to successful translation of legal texts is the lack
of one-to-one correspondence between concepts—much less terms—in different
languages. Writing about the study of comparative law, René David comments:
The absence of an exact correspondence between legal concepts and categories in dif-
ferent legal systems is one of the greatest difficulties encountered in comparative legal
23. PETER M. TIERSMA, LEGAL LANGUAGE 49 (1999).
24. Id. at 69.
25. FRANCISCO A. AVALOS, THE MEXICAN LEGAL SYSTEM 12 (2000).
26. HOWARD D. FISHER, GERMAN LEGAL SYSTEM AND LEGAL LANGUAGE: A GENERAL SURVEY TOGETHER
WITH NOTES AND A GERMAN VOCABULARY, at vii (1996).
27. TIERSMA, supra note 23, at 71–85.
28. Id. at 87–95.
29. Id. at 98–99.
30. Id. at 108–09.
98n3_2p.indd 486 7/19/2006 12:08:12 PM2006-27] Scaling the Tower of Babel Fish 487
analysis. . . . [T]he reality must be faced that legal science has developed independently
within each legal family, and that those categories and concepts which appear so ele-
mentary, so much a part of the natural order of things, to a jurist of one family may be
wholly strange to another.31
Obviously, this difficulty has serious implications for legal translation. Susan
Šarčević writes:
Unlike medicine, chemistry, computer science, and other disciplines of the exact sci-
ences, law remains first and foremost a national phenomenon. Each national or municipal
law, as it is called, constitutes an independent legal system with its own terminological
apparatus and underlying conceptual structure, its own rules of classification, sources of
law, method ological approaches, and socio-economic principles. . . . Due to differences
in historical and cultural development, the elements of the source legal system cannot be
simply transposed into the target legal system. As a result, the main challenge to the legal
translator is the incongruency of legal systems.32
¶26 Can machine translation meet this challenge? Specifically, can Babel Fish,
an all-purpose, basic machine translation system with no subject-specific function-
ality, deal with the incongruency of legal systems? Consider another comment by
Šarčević on the nature of legal translation:
Despite the continued emphasis on preserving the letter of the law in legal translation, the
basic unit of translation is not the word but the text. Since a text derives its meaning from
one or more legal systems, legal translation is essentially a process of translating legal
systems. Accordingly, it follows that, if legal translation is to be effective, the so-called
search for equivalents cannot be reduced to a process of matching up ‘equivalents.’33
¶27 Machine translation works by matching up equivalents—not even equiva-
lent concepts, but mostly just words and morphemes. Clearly Babel Fish cannot
provide legal translation of a caliber anywhere near that discussed by Šarčević. To
evaluate Babel Fish according to such a standard would be fruitless.
¶28 Perhaps a more modest criterion can be established. Alexander Fraser
Tytler, an early translation theorist, proposes three laws of translation:
I. That the Translation should give a complete transcript of the ideas of the original
work.
II. That the style and manner of writing should be of the same character with that of the
original.
III. That the Translation should have all the ease of original composition.34
31. RENÉ DAVID & JOHN E.C. BRIERLY, MAJOR LEGAL SYSTEMS IN THE WORLD TODAY: AN INTRODUCTION
TO THE COMPARATIVE STUDY OF LAW 16 (3d ed. 1985). David apparently understands the importance
of good translation—John E.C. Brierly, the translator of this book, which was originally issued in
French, is credited as a coauthor.
32. SUSAN ŠARČEVIĆ, NEW APPROACH TO LEGAL TRANSLATION 13 (1997).
33. Id. at 229.
34. ALEXANDER FRASER TYTLER, ESSAY ON THE PRINCIPLES OF TRANSLATION 16 (Jeffrey F. Huntsman ed.,
John Benjamins B.V. 1978) (1791).
98n3_2p.indd 487 7/19/2006 12:08:12 PM488 Law Library Journal [Vol. 98:3
Tytler’s second and third laws cannot be expected to apply to machine transla-
tion. John Joseph argues that they should not be applied to legal translation either.
Joseph writes that Tytler’s “First Law assumes such overwhelming importance
that all else is swallowed up in its wake; and yet the nature of law is such that the
requirements of Tytler’s First Law are impossible to fulfill.”35
¶29 Whether Tytler’s first law can ever be fulfilled in legal translation is a ques-
tion beyond the scope of this study. Instead, this study will attempt to determine
whether Babel Fish succeeds in a scaled-down version of the first law. Systran,
the company responsible for Babel Fish, reports that “[t]he SYSTRAN technology
powering most of the Internet portals is a generic translation service that is based
on the largest possible dictionaries. Its aim is to provide a ‘gisting level’ translation
service, such as looking at a foreign page and being able to understand the main
lines of the page.”36 The question addressed in this study, then, is whether Babel
Fish accurately conveys the gist of law-oriented texts.
Evaluation of Machine Translation
¶30 Like most new technological developments, machine translation has had its
share of naïvely overenthusiastic supporters. Émile Delavenay concludes his 1960
treatise on machine translation with the prediction that “translating machines will
soon take their place beside gramophone records and colour reproductions in the
first rank of modern techniques for the spread of culture and of science.”37 More
recent was Bill Clinton’s claim in his 2000 State of the Union speech that “[s]oon
researchers will bring us devices that can translate foreign languages as fast as you
can talk. . . .”38
¶31 Machine translation has its share of detractors as well. Paul F. Wood, a
professional translator, writes: “The combined power of the world’s computer
industry has . . . had five decades to come up with an acceptable solution . . . and it
hasn’t. The Alta Vista Babelfish Translator on the Internet, for example . . . is sim-
ply dreadful.”39 Law librarian Ruth Balkin evaluates Babel Fish in a 1999 article
for the journal Database. Her assessment is best summed up in a single sentence:
“I was amazed at just how bad it was.”40
35. John E. Joseph, Indeterminancy, Translation and the Law, in TRANSLATION AND THE LAW 13, 17
(Marshall Morris ed., 1995).
36. Systran Info. & Translation Tech., Standard Machine Translation vs. Customized Translation
Services, http://www.Systransoft.com/Technology/Customization.html (last visited Apr. 3, 2006).
37. ÉMILE DELAVENAY, AN INTRODUCTION TO MACHINE TRANSLATION 117 (Katharine M. Delavenay &
Émile Delavenay trans., 1960).
38. William J. Clinton, Address Before a Joint Session of the Congress on the State of the Union, 1 PUB.
PAPERS 129, 139 (Jan. 27, 2000).
39. Paul F. Wood, What Do Translators Do? And What Machines Can Not, LINGUIST, Oct. 2000, available
at http://language.home.sprynet.com/lingdex/pwood1.htm#t8.
40. Ruth Balkin, Babel Fish: AltaVista’s Automatic Translation Program, DATABASE, Apr./May 1999, at
56, 56.
98n3_2p.indd 488 7/19/2006 12:08:12 PM2006-27] Scaling the Tower of Babel Fish 489
¶32 Although Balkin is a law librarian, her article on Babel Fish does not spe-
cifically address issues related to law librarianship. A small amount has been writ-
ten about Babel Fish in law library literature. In a 2001 article in AALL Spectrum,
Roy Balleste rates several MT systems, including Systran, which receives his
highest rating.41 In another AALL Spectrum article, I discuss the use of Babel Fish
and other Web-based translators among law librarians.42 Like Balkin’s article,
neither of these articles includes a detailed analysis of Babel Fish or any other MT
system.
¶33 Some systematic evaluations of Babel Fish have been published, although
these have not focused on the translation of legal texts. Perhaps the most well-
known is Paul A.Watters and Malti Patel’s 1999 study.43 Watters and Patel translated
four common English proverbs from English into the five other languages avail-
able on Babel Fish at the time of the study (French, German, Italian, Portuguese,
and Spanish), then back into English again. Of the twenty double translations
performed, only two were translated successfully according to the authors’ crite-
rion—that the final result had nearly the same meaning as the original.
¶34 The Watters and Patel study has some shortcomings, aside from the fact
that it was performed several years ago.44 For one thing, it uses an English
target language English approach. This makes the evaluation easy for the
researchers because all they have to do is compare the final translation with the
original text. However, it does not replicate an actual use of machine translation,
since there are few, if any, practical reasons to translate text into a foreign lan-
guage and then back into one’s own language. Furthermore, this approach tends
to magnify translation problems because translating the same text twice doubles
the opportunity for errors. Not only that, but an error made during the first transla-
tion is only exacerbated with subsequent translations. Also, because the authors
translated proverbs, their results do not necessarily apply to translation of more
literal texts.
¶35 There are many methods for evaluating machine translation. Nico Weber
distinguishes between application-oriented evaluation, which focuses on per-
formance, usability, and cost-effectiveness; and theoretical evaluation, which is
concerned more with purely linguistic criteria.45 This article employs an applica-
tion-oriented evaluation, based solely on performance. Since the system being
41. Balleste, supra note 6, at 5.
42. Yates, supra note 2.
43. Watters & Patel, supra note 13.
44. Babel Fish has changed in the meantime. The most obvious change is that there are more language
pairs available now. Also, the sentences translated by Watters and Patel in 1999 are not translated the
same way today. For example, the final result of the translation of “A stitch in time saves nine” into
Italian and then back to English was “A point to time saves nine” in Watters and Patel’s study, but
now the same translations result in “A point to time conserve nine.”
45. Nico Weber, Machine Translation, Evaluation, and Translation Quality Assessment, in MACHINE
TRANSLATION: THEORY, APPLICATIONS, AND EVALUATION: AN ASSESSMENT OF THE STATE-OF-THE-ART
47, 61 (Nico Weber ed., 1998).
98n3_2p.indd 489 7/19/2006 12:08:12 PM490 Law Library Journal [Vol. 98:3
evaluated entails no cost other than the staff time spent using it, there is no analysis
of cost-effectiveness. Babel Fish has a very straightforward interface and is no
more prone to service disruptions than any other Web site, so usability issues are
not evaluated either.
¶36 The various methods of application-oriented evaluation include “glass
box” (if it is known how the system works) versus “black box” (if it is not),
“micro evaluation” (if only one MT system is evaluated) versus “macro evalua-
tion” (if two or more systems are compared), and “positive” (performance-based)
versus “negative” (error-based) evaluation.46 This article relies on a black box,
micro evaluation. Only the basic workings of machine translation in general and
Systran’s system in particular, as discussed above, are relevant to the needs of most
librarians and library patrons. Babel Fish is the only system evaluated; it is not
compared with other available MT systems. While this evaluation is error-based,
it shares many characteristics with “positive” performance assessment because of
its emphasis on understandability (i.e., intelligibility and comprehensibility) and
fidelity (i.e., correctness and accuracy).47
¶37 This evaluation of Babel Fish is user-oriented. The emphasis is on deter-
mining the system’s success or failure based on the needs of a specific user group,
in this case law librarians and law library patrons. Rita Nübel outlines several
criteria for user-oriented MT evaluation: translation quality (i.e., what’s the pur-
pose of the translation?), financial budget, time budget, availability of tools and
resources at the user’s disposal, and characteristics of the text(s) to be translated.
Some text characteristics that should be taken into consideration include the for-
mat and layout, length, genre, structure and length of sentences, and whether the
text will be pre-edited (i.e., rendered into a more MT-friendly style before the
translation), postedited (i.e., rendered into a more reader-friendly style after the
translation), or both.48
¶38 The simplest of Nübel’s criteria to address for this study are financial and
time budget. The greatest advantage of Babel Fish and other machine translation
programs on the Web is that they are free and quick. Even if one assumes a user
with no money and very little time, Babel Fish meets the user’s needs in these
two respects.
¶39 The availability of tools and resources at the user’s disposal is more dif-
ficult to assess, since this study is not considering a specific user but rather the
diverse group of law librarians and law library users. Librarians, of course, do have
access to a wide range of resources, but these vary by library. Also, users might
not know about all the resources available through their law libraries. Because of
46. Id. at 64.
47. Id. at 67.
48. Rita Nübel, MT Evaluation in Research and Industry: Two Case Studies, in MACHINE TRANSLATION:
THEORY, APPLICATIONS, AND EVALUATION: AN ASSESSMENT OF THE STATE-OF-THE-ART, supra note 45,
at 85, 106–07.
98n3_2p.indd 490 7/19/2006 12:08:12 PM2006-27] Scaling the Tower of Babel Fish 491
the uncertainty in this area, no tools other than Babel Fish itself are considered in
this study.
¶40 Characteristics of the text have already been touched on generally in the
section on legal language.49 Some specific characteristics of the actual texts cho-
sen for this study are also worth noting. The texts are taken from the Mexican
and German civil codes and press releases from the Mexican and German foreign
ministries. The civil code excerpts are not only classic examples of “legalese,” but
they are also examples of old-fashioned language. The Mexican code was writ-
ten in 1928; the German code in 1896. Both are marked by formal and complex
sentence structure and old-fashioned vocabulary. The press releases, on the other
hand, are very current and contain no archaisms. The German press release is writ-
ten in a relatively plain and straightforward style, but the Mexican release contains
lengthy and complex sentences.
¶41 The purpose of the translation is an extremely important consideration,
particularly when translating legal materials. Because of the precision required in
legal language, an extremely high quality standard is necessary when translated
texts will be used in actual legal practice. This study does not evaluate Babel Fish
according to such a high standard; rather, it evaluates the program’s ability to
express the general meaning of a text. For the purposes of this study, a “correct”
translation is one that conveys the same general meaning as a professional transla-
tion of the text; that is, the machine translation has the same meaning in an ordi-
nary context, but not necessarily in a strict legal sense. Issues of style and fluency
are only important if they affect the comprehensibility of the text.
Method
¶42 I translated twenty sentences with Babel Fish; of these, ten were Spanish
to English and ten were German to English.50 Spanish and German were cho-
sen because they are foreign languages frequently encountered by American
law librarians. Also, they belong to different language families (Romance and
Germanic), which increases the variety of linguistic characteristics in the sample.
The selection of one of Babel Fish’s non-Indo-European languages (Chinese,
Japanese, or Korean) would have provided even greater linguistic diversity, but
these were rejected because of the practical complications of working with non-
Roman scripts. The predominant use of online translation programs in American
law libraries is foreign to English rather than vice versa,51 so no translations from
English into another language were performed.
49. See supra ¶¶ 21–26.
50. For complete results, including Babel Fish and professional translations of the sample texts and analy-
sis of the errors, see Sarah Yates, Complete Translations and Analysis of Errors (2005), http://www
.law.umn.edu/uploads/images/2288/translations.pdf.
51. Yates, supra note 2. Not one librarian interviewed for this article mentioned using online translators
to translate from English into another language.
98n3_2p.indd 491 7/19/2006 12:08:12 PM492 Law Library Journal [Vol. 98:3
¶43 The texts selected for translation were five sentences each from the
Mexican and German civil codes52 and five sentences each from recent press
releases from the Mexican and German foreign ministries.53 In selecting the
excerpts from the civil codes, articles were chosen at random using a random
number generator. When those articles consisted of more than one sentence, I
translated only the first sentence. In real life, an isolated sentence from a lengthy
article of a country’s civil code could hardly be considered a text in itself.
However, Babel Fish does not take context into account beyond a single sentence
(one could argue that it rarely takes context into account at all),54 so attempting
to compare longer passages would not improve the analysis.
¶44 In the case of the press releases, I chose one release from each foreign
ministry’s Web site that was of potential interest to the American legal community.
The press release had to have an official English translation available from its Web
site.55 Only the first five sentences of the German press release are translated and
evaluated in this study.56 The German release was translated using Babel Fish’s Web
page translation bar. I attempted to use this method for the Mexican press release
as well; however, Babel Fish failed to recognize the diacritics, so any word with
an accented letter was not translated. This problem did not occur when sentences
were individually copied and pasted into the block text translation box, which is
the method that I ultimately used. It is important to note, however, that many users
might simply give up after Babel Fish failed to translate the entire Web page.
¶45 In addition to their relevance for legal researchers, one reason for select-
ing the civil codes and foreign ministry press releases was that professional
English translations of the texts were available for comparison. The translations
of the codes used in this study are Eckstein and Trujillo’s Mexican Civil Code57
and Goren’s The German Civil Code.58 The translations of the press releases are
52. Both the original Spanish text and the professional English translation of articles 164, 507, 1778,
2226, and 2340 of the Mexican code are taken from MEXICAN CIVIL CODE (Abraham Eckstein &
Enrique Zepeda Trujillo trans., 1996). The German text of paragraphs 270, 627, 854, 1979, and 2279
of the German civil code is taken from OTHMAR JAUERNIG, BÜRGERLICHES GESETZBUCH: MIT GESETZ
ZUR REGELUNG DES RECHTS DER ALLGEMEINEN GESHÄFTSBEDINGUNGEN 7 (rev. ed.1994).
53. Secretaría de Relaciones Exteriores (México), Comunicado de prensa no. 012, El Gobierno de
Mexico Presentó un Documento de “Amigo de la Corte” (Amicus Curiae) ante la Suprema Corte
de Estados Unidos en Apoyo al Caso del Mexicano Ernesto Medellín, Sentenciado a Muerte en
Texas (Jan. 26, 2005), http://www.sre.gob.mx/comunicados/comunicados/2005/enero/b_012.htm;
Deutschland. Auswärtiges Amt., Bundesminister Fischer zu Parlamentswahlen im Irak (Jan. 31,
2005), http://www.auswaertiges-amt.de/www/de/ausgabe_archiv?archiv_id=6746.
54. Watters & Patel, supra note 13, at 154.
55. Several press releases on the German site were rejected because the English versions were not
similar enough to the original German releases. For example, the English-language press release
Federal Minister Fischer Takes Part in a Forum in Spain on the European Constitutional Treaty (Jan.
31, 2005), available at http://www.auswaertiges-amt.de/www/en/ausgabe_archiv?archiv_id=6751,
appears at first glance to be a translation of the German-language release Bundesminister Fischer
nimmt in Spanien an Diskussionsrunde über den Europäischen Verfassungsvertrag teil (Jan. 28,
2005), http://www.auswaertiges-amt.de/www/de/ausgabe_archiv?archiv_id=6740, but the English is
in the past tense, whereas the German is in the future tense.
56. The entire Mexican press release is translated because it is only five sentences long.
57. MEXICAN CIVIL CODE, supra note 52.
58. GERMAN CIVIL CODE (as amended to January 1, 1992) (Simon L. Goren trans., rev. ed. 1994).
98n3_2p.indd 492 7/19/2006 12:08:13 PM2006-27] Scaling the Tower of Babel Fish 493
official English versions found on the foreign ministries’ Web sites.59 Babel Fish’s
translations are compared to the professional translations.
¶46 Because of the lack of one-to-one correspondences among languages,
there is rarely only one correct translation of a given text. Therefore, while the
professional translations used in this study are certainly correct, Babel Fish is not
considered to have made an error in cases where its translation is semantically
equivalent to the professional one, even if it is not identical.
¶47 Errors are divided into two categories: lexical errors (i.e., individual mis-
translated words) and structural errors. In a few cases, short phrases fall under
the lexical category, when the phrase functions as a word-like unit. For example,
in its translation of article 507 of the Mexican civil code, Babel Fish translates
Ministerio Público as Public Ministry rather than attorney general; this is consid-
ered a lexical error although it is not a single word.
¶48 The structural error category is something of a catchall; any error that is
not lexical is considered structural. Since only errors that affect comprehension of
the text are counted, some inelegant English structures are allowed. A few single-
word errors are considered structural rather than lexical if the word is virtually
meaningless when considered in isolation from its function in a given sentence.
For example, the Babel Fish translation of “für ihren . . . Willen” (from the third
sentence of the German press release) as “for its will,” rather than “for their will”
as in the professional translation, is considered a structural error because the mean-
ing of either its or their is so closely tied to the structure of the sentence.
¶49 A three-point scale is used to rate the errors, with one being minor, two
being moderate, and three being severe. Ratings of three are reserved for errors
that change or obscure a sentence’s meaning completely;60 therefore, any sentence
with even one level-three error is considered a failed translation. Conversely, any
sentence without at least one level-three error is considered successful, regardless
of the number of minor or moderate errors.
¶50 Some of the phrases that are considered structural errors actually contain
more than one error. In some cases it is very difficult to pinpoint exactly where Babel
Fish went wrong, and isolating the individual errors and comparing them to the
59. Press Release Number 012, The Mexican Government Submits an Amicus Curiae (Friend of the
Court) Brief to the U.S. Supreme Court to Support the Case of Ernesto Medellin, a Mexican Citizen
Sentenced to Death in Texas (Jan. 26, 2005), http://portal.sre.gob.mx/usa/index.php?option=news&
task=viewarticle&sid=116 Federal Minister Fischer on the Parliamentary Elections in Iraq (Jan. 31,
2005), available at http://www.auswaertiges-amt.de/www/en/ausgabe_archiv?archiv_id=6753.
60. Two types of errors are automatically rated severe (level three). The first is the failure to translate a
word at all, as in the translation of the Spanish phrase (from article 507 of the Mexican civil code) “El
Ministerio Público y los parientes del pupilo” as “The Public Ministry and the relatives of the pupilo.”
While some untranslated words might be correctly guessed by English speakers, many others might
not be; in this case, the word pupilo is translated not as pupil but as ward in the professional transla-
tion. The other automatic level-three rating is assigned to proper names that Babel Fish translates as
if they were regular words, such as the translation of the name José Ernesto Medellín Rojas (from the
first sentence of the Mexican press release) as Jose Ernesto Red Medellín.
98n3_2p.indd 493 7/19/2006 12:08:13 PM494 Law Library Journal [Vol. 98:3
corresponding parts in the professional translations would require detailed linguistic
analysis beyond the scope of this study. An example is the beginning of Babel Fish’s
translation of article 1979 of the German civil code: “The creditors of an estate must
let the correction of a liability of an estate by inheriting be considered as to be paid
. . . ,” which Goren translates as “The discharge of an obligation of the estate by the
heir is deemed, as against the creditors of the estate, to have been made. . . .” Among
the problems with the Babel Fish translation of this phrase are: (1) “the creditors” is
the subject rather than the object of a prepositional phrase; (2) it is unclear what “by
inheriting” modifies; and (3) “be considered as to be paid” is awkward. But this selec-
tion does not lend itself to being divided into three smaller sections that still corre-
spond with exactly three smaller selections of the professional translation.
¶51 Sometimes lexical errors occur within structural errors. In these cases, the
structural errors are rated as if the words that are counted as errors were correct. For
example, Babel Fish’s translation of article 1778 of the Mexican code contains the
phrase “will stoop of the common bottom”; the corresponding phrase in the profes-
sional translation is “shall be paid from the common fund.” Although the Babel Fish
phrase is meaningless, this is only a level-two structural error. Stoop instead of paid
and bottom instead of fund are counted as level-three lexical errors, so the phrase is
evaluated as if it read “will pay of the common fund.” Note that it is not evaluated as if
it read “will be paid of the common fund,” because the difference between the active
(stoop, or its stand-in, pay) and the passive (be paid) is a structural error separate from
the difference in meaning between stoop (in any form) and pay (in any form).
¶52 The error rating system is subjective, but necessary: a weighted system
of error rating provides a more useful result than does a mere tally of errors. A
translation with several minor errors usually conveys the original meaning more
accurately than a translation with just one severe error. To reduce the level of
subjectivity, a librarian who was also a practicing attorney reviewed the ratings to
ensure their reasonableness.61
Results
¶53 Of the twenty sentences tested, fifteen (75%) must be considered failed trans-
lations because they contain at least one severe error (see figure 1). This in itself
is a dismally high failure rate, especially considering the relatively low standards
set for the study. It is even more alarming that only four of these fifteen sentences
fail because of a single severe error; the remainder contain anywhere from two to
seven severe errors (see figure 2). All of the failed translations also contain minor
or moderate errors, or both. Of the four translations with only one severe error, the
“best” is the fifth sentence of the German press release,62 which contains no mod-
61. Many thanks to Mary Rumsey, foreign, comparative, and international law librarian at the University
of Minnesota Law Library.
62. “After the formation of a government it will be priority task to prepare a condition for the whole Iraq.”
The professional translation is: “Once a government has been formed, the prime task will be the draft-
ing of a constitution for all Iraq.” For a list and ratings of the errors in this and all other sentences in
the study, see Yates, supra note 50.
98n3_2p.indd 494 7/19/2006 12:08:13 PM2006-27] Scaling the Tower of Babel Fish 495
erate errors and four minor ones. Of the two with seven severe errors, the worst is
the second sentence of the Mexican press release,63 which contains six moderate
errors and two minor ones.
¶54 Turning to the five successful translations, none is error-free. Two sen-
tences come close, however; both the translation of paragraph 854 of the German
civil code64 and that of the second sentence of the German press release65 contain
only one minor error and no moderate errors. The first sentence of the German
press release66 (actually a sentence fragment rather than a sentence, not only in
both the translations but also in the original) contains one minor and one moderate
error, making it reasonably accurate given the limitations of machine translation.
But the remaining successful translations, of article 2226 of the Mexican civil
code67 and of the fourth sentence of the Mexican press release,68 contain multiple
errors and would not be considered good translations under any but the least strin-
gent standards.
¶55 Babel Fish handles the German sentences in this study markedly better
than the Spanish ones. The Spanish translations contain roughly twice as many
moderate and severe errors as the German ones; the number of minor errors in
the Spanish and German translations is nearly equal (see figure 3). The significant
63. “In its presentation before the Supreme Court of the United States, Mexico expressed its endorsement
to the done arguments to be worth by the defense of Mr. Medellín, in the sense that the cuts of that
country must give to fulfillment to the failure of the Court the International in the case known like
Oats (Mexico versus the United States), that ordered the revision and reconsideración of its sentence
to the light of the violation to its consular rights.” The professional translation is: “In its presentation
to the U.S. Supreme Court, Mexico expressed its support for the arguments given by Mr. Medellín’s
lawyers, that the U.S. courts must comply with the ruling of the International Court of Justice in the
Avena case (Mexico vs. the United States) that called for a review and reconsideration of his sentence
given that his consular rights were violated.”
64. “The possession of a thing is acquired by the acquisition of the actual force over the thing.” The pro-
fessional translation is: “Possession of a thing is acquired by obtaining actual power of control over
the thing.”
65. “The execution of the elections in the Iraq means an important step toward democratization of the
country.” The professional translation is: “The elections in Iraq mark an important step in the democ-
ratization of the country.” The presence of the word execution in the Babel Fish translation is not an
error, even though no corresponding word appears in the professional translation, because the original
German sentence does contain the word Durchführung (execution). Also, while “the Iraq” is not cor-
rect usage in English, the definite article does not impede comprehension, so it is not considered an
error.
66. “Federal Minister of Foreign Affairs Joschka Fischer today (31,01.) to the elections of yesterday
in the Iraq.” The professional translation is: “Statement of 31 January by Federal Foreign Minister
Joschka Fischer on Sunday’s elections in Iraq:” Yesterday is not an error compared to Sunday’s; it is
actually a closer translation of the word gestrigen, which appears in the original text.
67. “The absolute invalidity as a rule does not prevent that the act produces provisionally its effects,
which will be destroyed retroactively when the invalidity is pronounced by the judge.” The pro-
fessional translation is: “Absolute nullity, as a general rule, does not prevent an act from having
provisional consequences, which can be retroactively abolished upon an adjudication of nullity by a
judge.”
68. “According to the procedural calendar of the case, the Supreme American Court it will listen to the
arguments of the parts at the end of the month March and a decision could be emitted in the month of
July.” The professional translation is: “The U.S Supreme Court will hear arguments from both parties
towards the end of March. It could issue a decision in July.”
98n3_2p.indd 495 7/19/2006 12:08:13 PM496 Law Library Journal [Vol. 98:3
Figure 1
Failed vs. Successful Translations
Figure 2
Number of Severe Errors in Failed Translations
Figure 3
Average Number of Errors per Sentence, by Language
98n3_2p.indd 496 7/19/2006 12:08:14 PM2006-27] Scaling the Tower of Babel Fish 497
difference in Babel Fish’s performance in translating the two languages should
serve as a warning against extrapolating too much about Babel Fish’s accuracy
translating other languages based on the results of this study.
¶56 I expected to find more errors—and more severe ones—in the civil codes
than in the foreign ministry press releases. The codes are older, and I expected
them to contain many outdated words and structures that Babel Fish is not pro-
grammed to deal with. Also, since the codes are actual laws, I expected them to be
full of formal legalisms equally difficult for Babel Fish. Surprisingly, though, there
is virtually no difference between the number and severity of errors found in the
codes and those found in the press releases, at least when the Spanish and German
results are considered together (see figure 4.1).
Figure 4.1
Average Number of Errors per Sentence, by Text Type
¶57 Among the German sentences, there are close to the same number of minor
and moderate errors in the code and the press release, but the number of severe
errors in the code is more than three times greater than the number in the press
release (see figure 4.2). This is not surprising, for the reasons mentioned earlier.
What is surprising is that in the Spanish translations, more errors of every level
occur with the press release than with the code (see figure 4.3). The most remark-
able difference is in the number of severe errors per sentence—2.4 for the civil
code versus 4.4 for the press release. The reason for this is unclear, but it does
illustrate the dangers in making assumptions about how Babel Fish will perform
given a certain type of text.
98n3_2p.indd 497 7/19/2006 12:08:14 PM498 Law Library Journal [Vol. 98:3
Figure 4.2
Average Number of Errors per Sentence, by Text Type—German Only
Figure 4.3
Average Number of Errors per Sentence, by Text Type—Spanish Only
Conclusion
¶58 Babel Fish’s performance in this study was poor, but it was not uniformly
poor. Babel Fish translated the German texts less poorly than the Spanish ones.
Furthermore, the civil code excerpts were translated more accurately than the press
releases in Spanish, but less accurately in German. These discrepancies suggest
that further research—with different languages and possibly different types of
texts—is needed to more broadly determine Babel Fish’s relative strengths and
weaknesses.
¶59 In the meantime, what are law librarians to do when dealing with foreign-
language legal information? There are three alternatives to Web-based machine
98n3_2p.indd 498 7/19/2006 12:08:14 PM2006-27] Scaling the Tower of Babel Fish 499
translation. The first alternative, one that should be pursued before even consid-
ering Babel Fish or another similar program, is to try to find the information in
English.69 The obvious drawback to this approach is that a great deal of foreign
legal information is not available in English. Translations of primary legal sources
are particularly problematic: they often do not exist, and when they do, they may
not be authoritative or current.70 Any professional translation—even if not authori-
tative—is preferable to a Babel Fish translation, although even the best translation
of a law that is no longer in effect may be completely useless. Secondary sources
are somewhat easier to find in English71 and may be acceptable for many research
needs.
¶60 Another alternative to Web translation is amateur human translation. Law
librarians cannot be expected to translate foreign texts for patrons, but they might
persuade patrons with some foreign-language skills that the patron’s own transla-
tion is likely to be better than Babel Fish’s. Similarly, law librarians doing their
own research might try their hand at translating a text themselves if they know
some of the language.
¶61 This is probably the most problem-fraught of the alternatives to free
machine translation. The most obvious disadvantage is that it is not even an option
for researchers who do not know a given language. Even in cases where it might
be possible, translation of legal materials is a difficult and time-intensive task even
for experienced translators, and all the more so for people with limited translat-
ing experience or a shaky grasp of the language in question. To recommend this
approach to a patron, a law librarian would need to be confident that the patron’s
language expertise is strong enough to translate the given text—something a librar-
ian is not likely to know.
¶62 The third alternative is to pay for a translation. In addition to professional
“human translation,” fee-based machine translation services are available. This
study did not evaluate any such services,72 but it is reasonable to expect that they
are better than free programs. Even without evaluating the fee-based services, it
is safe to say that machine translation is always inferior to translation performed
by a trained professional, for the reasons outlined in the introduction. However,
machine translation has some undeniable advantages, including its speed and its
relative cost-effectiveness, even for the fee-based services. For example, SDL
International offers both professional and machine translation through its Web site
(www.freetranslation.com). A one-year machine translation subscription for up to
fifty translations a day costs $39.95, versus $150 for a professional German-to-
69. See Mary Rumsey, Strangers in a Strange Land: How to Answer Common Foreign Law Research
Questions, AALL SPECTRUM, July 2004, at 16, for an excellent discussion on how to locate foreign
legal resources in English.
70. THOMAS H. REYNOLDS & ARTURO A. FLORES, 1 FOREIGN LAW: CURRENT SOURCES OF CODES AND
BASIC LEGISLATION IN JURISDICTIONS OF THE WORLD 13–19 (1989–).
71. See Rumsey, supra note 69.
72. But see Balleste, supra note 6 (rating both free and fee-based systems).
98n3_2p.indd 499 7/19/2006 12:08:14 PMYou can also read
