Music Information Retrieval Research and Its Context at the University of Waikato
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Music Information Retrieval Research and Its Context at
the University of Waikato
David Bainbridge
Department of Computer Science, University of Waikato, Hamilton, New Zealand.
E-mail: d.bainbridge@cs.waikato.ac.nz
This article describes the digital music library work at The structure of this article is as follows. First, we detail
the University of Waikato, New Zealand. At the heart of the software workbench that is at the heart of our research
the project is a music information retrieval workbench
for evaluating algorithms and performing experiments
efforts. Designed with flexibility in mind, this software tool
used in conjunction with four datasets of symbolic no- allows us to experiment with different algorithms, compar-
tation ranging from contemporary to classical pieces. ing and contrasting particular aspects: recall and precision,
The outcome of this experimentation is woven together computational cost, memory use, and so on. Next, we de-
with strands from our larger digital library project to scribe the datasets we work with, which are based around
form the Web-based music digital library MELDEX (short
for melody index). An overview of the workbench soft-
symbolic notation reflecting our interest in this form of
ware architecture is given along with a description of music information retrieval. We conclude by describing our
how this fits the larger digital library design, followed by Web-based digital music library service MELDEX (short
several examples of MELDEX in use. for melody index), which takes advantage of both our ex-
perimentation and digital library components developed
elsewhere in the larger project.
Introduction
Music information retrieval (MIR) research at the Uni- A Music Information Retrieval Workbench
versity of Waikato, New Zealand, is an integral part of a Figure 1 gives an overview of the workbench, designed
digital music library project, which in turn is part of the to integrate with the parent group’s broader digital library
larger New Zealand Digital Library (http://www.nzdl.org) software architecture. Using an object-oriented paradigm,
project (Witten, McNab, Boddie, & Bainbridge, 2000). The the software was originally written using a mixture of C⫹⫹
music work covers a broad range of topics—from content and Perl; an updated version in Java is in progress. The
and query acquisition, through retrieval algorithms, to doc- workbench is divided into two phases: assimilation and
ument presentation and software scalability—topics that runtime. Controlled by a configuration file, the assimilation
mirror the key stages of the main digital library project. phase is responsible for collating files together to form
At the core of the music group is a mixture of faculty and indexes and/or databases. The runtime phase is where ex-
graduate students—all active musicians—with expertise perimentation is carried out, guided by user input. While
that includes signal processing, information retrieval, soft- assimilation is typically performed once for a given set of
ware architecture, and human computer interaction. The experiments, the runtime phase is executed many times to
precise boundary between this group and its parent digital garner results.
library project is intentionally indistinct to encourage over- Importing takes care of the multitude of different file
lap and serendipity between researchers. To take one gen- formats associated with music data, converting input files
eral example: given the context of the music information into a canonical format that is both general and expressive
retrieval work as a digital music library, interface issues are enough for our needs, yet straightforward to parse. After
an important consideration, and it has been a natural process considering a range of possible formats—and factoring in
for staff skilled in human computer interaction to become requirements from the larger digital library project—we
involved in the computer music work. selected the XML version of Guido (Hoos, Renz, & Gorg,
2001). For each format handled by the workbench, a new
object is inherited from the Import base class, and its
© 2004 Wiley Periodicals, Inc. ● Published online 25 May 2004 in Wiley member functions overridden as appropriate. A mixture of
InterScience (www.interscience.wiley.com). DOI: 10.1002/asi.20062 hand-tailored code and, when available, existing open
JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY, 55(12):1092–1099, 2004forth. The output of a search is held as a result set from
which statistics are extracted, graphs plotted, and tables
generated. The particular operations performed by the
workbench in the runtime phase are controlled by the user
who can interact directly with the workbench through its
command line syntax or develop scripts that batch process
the desired experiment.
The group supports open source projects, and there are
plans to release the workbench under the GNU public
license. It is hoped that this will help foster a stronger sense
of community within the music information retrieval field
where other groups will add their retrieval algorithms to the
workbench, thereby allowing a more comprehensive com-
parison of the relative strengths and weaknesses of the
algorithms being developed.
Datasets
FIG. 1. A workbench for symbolic music information retrieval. There are four main datasets used for experimentation.
We call them Folksongs, Fakebook, Midimini, and Midi-
source utilities are used to perform this. For example, the max. The first two are formed from symbolic data that have
MIDI import module is based around the MIDI to Guido been either entered by hand or generated automatically by
conversion utility (available at http://www.salieri.org/ processing scanned sheet music using optical music recog-
guido). nition software. The other two, as reflected by their names,
Filtering and expanding are optional steps. Conceptually, are derived from MIDI files and are, therefore, only pseudo-
the former reduces the stream of musical data passing symbolic from the point of view of music notation (because
through it while the latter increases it. From an implemen- this format captures note-on and note-off events, not true
tation standpoint, however, this distinction is insignificant abstract notation) but interesting to work with nonetheless.
and both categories inherit from the same base class that is The Folksong dataset contains 9,354 monophonic tunes
set up to manipulate (add, remove, or modify) a stream of and is a selected combination of the Essen database (Schaf-
musical data. The two entities are represented separately in frath, 1992) and a snapshot of the Digital Tradition (Green-
the workbench because it is useful to be able to differentiate haus, 1996) taken in 1996. Some editing of the data was
between them during the configuration phase. An example performed when the two sets were put together. In the case
of filtering is retaining only MIDI tracks that are monopho- of duplication, repeated tunes were initially identified by
nic, which is useful when the searching algorithm on trial title, then checked musically for co-incidence, and only one
only works on this type of notation. An example of expand- version retained for the merged set. Alternative tunes for the
ing is generating a tune in different keys, which is useful for same song, however, were kept; for instance, there are two
algorithms that only work in an absolute key. distinctly different versions of the mining song Clementine
A more interesting example of filtering is RepairMotif, and likewise for Auld Lang Syne (the latter came as a
which detects repeated fragments of tune. Central to the surprise for the author, despite being raised in Scotland!).
module is the “re-pair” algorithm (Larsson & Moffat, 1999) Because the initial workbench did not support triplets, songs
from which the filter takes its name. This is a rule-based with this metric were culled from the dataset.
character comparison algorithm that, if coded carefully, A fake book is an informal collection of often-requested
detects repeated substrings in linear time and space. To tunes typically collated by a band performing over several
work in a music context, a preprocessing step within Re- years. The sheet music used typically includes the basic tune
pairMotif maps notes to characters, and a postprocessing (verse and chorus), the lyrics, and some guitar chords. The
step studies the grammar rules produced to pick the most information is sufficient for a band to give a rendition of a
frequent and/or longest substrings and then reverses the requested song, sometimes “faking it” if they have never
mapping for the chosen strings. heard the piece before.
To perform searching using the workbench, there is a One such collection was scanned and processed by Can-
choice among three types of algorithms: state-based match- tor, a research-based optical music recognition system ini-
ing (Wu & Manber, 1992), dynamic programming (Mon- tially developed at the University of Canterbury (Bainbridge
geau & Sankoff, 1990), and text-based information retrieval & Bell, 1996; Bainbridge, 1997) and also used in a research
(Witten, Moffat, & Bell, 1999b), each with optional argu- context at Waikato. A customized configuration was used
ments that modify their behaviors, for example, matching that identified song title information in addition to the
only at the start rather than at any position within a tune, notation located on the staff. No attempt was made to
using a pitch contour rather than exact intervals, and so recognize guitar chord information, although this would
JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—October 2004 1093have made an interesting project in its own right. Textual amples of this include assigning bibliographical information
information such as title, composer, and lyrics, however, to audio recordings and detecting hyperlinks that point
was entered by hand. The end result was 1,235 monophonic outside of the collection (useful when your digital library
tunes richly augmented with textual metadata. collection is shipped out as a CD-ROM).
Based on a random sample of 20 images, the number of Because importing requirements can differ greatly from
edits necessary to correct the score was nine edits per 100 one collection to another, even if they use the same data
notes to fix durational mistakes, and one edit per 100 notes format, a collection has a configuration file that lists the
for pitch. It is estimated that it would take 12–14 working plugins needed and supplies the specific transformations
days to correct the complete collection, although this has not required as options to these plugins. All these abilities tie in
been done. Given that many music-content search algo- well with the music information retrieval workbench: its
rithms employ approximate matching techniques to allow assimilation phase already incorporates the notion of a
for discrepancies between how a user remembers a tune configuration file, and the filtering and expanding compo-
fragment and how it is scored in the database, the uncor- nents drop into place as plugin options.
rected Fakebook dataset makes an interesting test case for Less apparent between the diagrams, but equally well
how such algorithms perform on an error-prone repository. aligned in terms of integration, is the workbench’s runtime
Despite being sourced from the same format, the two system, which maps to the collection server in the digital
MIDI datasets have strikingly different characteristics. Mi- library design. Although not shown in Figure 2, inside the
dimini is a collection of 1,216 files from a single Web site collection server component are Search and Result objects,
where the maintainer had organized the tunes into a hierar- among other things. These two objects perform the same
chy. Broadly classified as pop, rock, and classical, file function as their workbench counterparts, and through in-
names encode the name of the tune and in the case of artists heritance a new collection music server can be readily
with several songs, a further subdirectory is used to encode constituted that, by accessing the workbench, overrides only
this also. the parts specific to supporting music content. The other
In contrast, the Midimax dataset was gathered from the
objects to the collection server remain unchanged, function-
Web at large. A utility was written that spawned a query to
ing as before providing generic services such as user au-
a search engine searching for hyperlinks that ended in .mid.
thentication.
The located files were downloaded using file names that
The role of the receptionist component in the digital
encoded their URLs. The data was then cleaned up, remov-
library system is to provide an interface between the digital
ing zero length files, identifying 404 errors (Web parlance
library’s content and a user wishing to access it. By decou-
for file not found), and detecting exact (binary) duplicates.
pling user interface issues from content management, a
The end result was 99,000 MIDI files.
flexible and versatile environment is formed. For example,
one receptionist might be set up to be multilingual and
Digital Library Software Architecture interact through HTML pages, whereas another—still ac-
Figure 2 shows the general software architecture devised cessing the same basic content—might provide a graphic
for the main digital library project (Witten et al., 1999a). querying environment through a Java client. In addition to
Because documents are represented as abstract entities, the this, communication between the receptionist and collection
design is flexible enough to handle diverse forms of media, server is through a protocol, enabling distributed configu-
as demonstrated by example collections built from text, rations of the digital library. In response to a user’s query,
images, video, and audio documents (Bainbridge et al., a receptionist might contact two collection servers running
2001). To realize a digital music library, the music infor- on different computers and present the result, seamlessly
mation retrieval workbench was embedded into the general merged, to the user. This ability is used in the digital music
digital library design. This was straightforward as the digital library to support multiple indexes to large collections that
library architecture foreshadowed the design of the work- are divided into smaller sections and distributed to multiple
bench and, consequently, the workbench was developed computers.
with this embedding step in mind. For now, our interest is in how the receptionist compo-
The workbench assimilation phase corresponds to the nent fits in with the workbench, and again the answer is
lower half of Figure 2 used to build a collection that is smoothly due to forward planning. Whereas in the work-
searchable and browsable. The term “import” in Figure 2 bench instructions come directly from the user, in the digital
carries a broader meaning in the general design due to the library they arrive via the receptionist. For the workbench to
diverse assortment of data types handled. Using a system of be used to support musical queries, therefore, all the recep-
“plugins”—modules of code tailored to parse particular tionist need do is translate these requests into meaningful
formats—source documents in potentially different formats instruction for the workbench (aided by the specially inher-
are imported into the collection and transformed into a ited music collection server) and to format the result into a
homogeneous XML format from which the necessary in- form the digital library is expecting. The object-oriented
dexes and databases can be formed. The transformation design of the digital library architecture is such that once
process also includes the ability to augment the data. Ex- more, implementation is a matter of inheriting a new ob-
1094 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—October 2004FIG. 2. Digital library software architecture.
ject—this time a specialized music receptionist—that over- interactive panel. Through mouse clicks, a user can expand
rides specific member functions. and contract phrases, and play the tunes containing the
Used in this way, it is true that the full power of the selected phrase. A particularly nice feature of the hierarchy
workbench is not utilized by the receptionist, but it is is that it naturally groups together songs by band name if
equally true that the receptionist supplies extra services, this has been used in its URL.
such as browsing titles A–Z, through other non-workbench
related components.
Raw Audio Queries
Figure 3 shows a user entering a sung fragment of audio
MELDEX: Example Services and Abilities
through the Java applet interface to MELDEX: the user is
We now present a selected tour of services and abilities singing a phrase from Abide With Me. Although not shown,
of the resulting digital music library system, MELDEX, through the digital library’s preference page the user has
with emphasis on the music-specific features. We start with elected to use the dynamic programming algorithm for
searching by music content using a sung raw audio query, matching, compare interval pitches, but ignore durations.
then move on to combined text and music queries, this time From the query page itself, the user can choose between
illustrating a symbolic form of music query entry through a matching anywhere in the tune or only at the start (con-
graphically presented piano keyboard. Also important in a trolled through a pull-down menu in the upper half of the
digital music library is delivery of a music document, and interface). If so desired, the user can supply a textual query
we present a Java applet designed to enrich the playback as well.
experience. Other features of the digital library not specific The buttons in the bottom portion of the interface control
to music content, such as hierarchical phrases and key- the acquisition of the music query. Not only can the user
words, are described elsewhere (Bainbridge et al., 2001; record a query by singing directly (as is shown), the user can
Witten, Bainbridge, & Boddie, 2001; Paynter & Witten, also load in a query saved earlier and see the result of
2001). transcribing the audio as music notation before deciding to
Here is one brief text-based example. In the Midimax submit it as a search.
collection, an interactive phrase hierarchical of the collec- Figure 4 shows the resulting matches that are found by
tion’s URLs makes for a compelling environment to browse activating the “Begin Search” button. At the top of the page
around. The user starts by entering a single word, and then are navigational aids that access particular services within
all the root phrases containing that word appear in an the library: home page, preferences, titles sorted alphabeti-
JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—October 2004 1095FIG. 3. Using the MELDEX-in applet for raw audio input.
cally A–Z. Next is a short excerpt of music notation dis- Enriching Playback
playing the computer’s transcription of the raw audio query.
To enrich the audio delivery experience of a tune re-
There is also a hyperlinked speaker icon that plays the
trieved by a melodic query, a playback applet has been
transcription back as audio should a user wish to hear it.
developed. If, instead of clicking on one of the four icons to
Then comes a ranked list of the matching documents. the left of a matching title, the user clicks on the thumbnail
In this particular collection (based on the Fakebook data- visualization, the playback applet is activated. Figure 5
set), for each matching tune MELDEX lists four icons for shows one such example. At the top, a larger version of the
retrieving the document in various forms: the first is the visualization is shown, and a set of control buttons similar
scanned original; the second (for the sake of interest) is the to a tape or CD player is provided below that lets the user
computer-generated music notation based on the error- play, skip to the next matching segment of tune, pause and
prone optical music recognition process; the third delivers stop the tune. As the tune is played, a time-based progress
an audio rendition of the piece; and the fourth is the sup- bar located between the enlarged visualization and control
plementary text that was entered by hand. Following the panel is used to proportionally show where in the tune the
icons is the title of the tune, a thumbnail visualization that playback currently is. Additional controls vary the tempo
displays where in the tune and how well the query matched, and volume.
the database name (superfluous in this context, but is more
meaningful when there are subcollections such as pop, rock,
Text and Symbolic Entry Music Queries
and classical), and its overall matching score. Further down
the page, but not shown in Figure 4, is the familiar next link Figure 6 shows a combined text and music query being
to view subsequent pages (and where appropriate previous entered, where an alternative form of music query entry
pages). is demonstrated. The user has entered the term
1096 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—October 2004FIG. 4. Result of audio query.
FIG. 5. Enriching tune retrieval playback through an interactive applet.
“Beethoven” in the text box in the upper half of the possibility of transcription errors in the entered query, the
screen, and is in the process of tapping out a music query note durations are controlled precisely through the cen-
on the graphical rendition of a piano keyboard. Since the tral part of the interface. Employing an HCI principle
purpose of this form of music input is to eliminate the known as equal opportunity, the user is free to click on
JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—October 2004 1097FIG. 6. Text and symbolic music query.
the music notation showing the duration they want, move the possibility of filtering or expanding the input stream of
the slider to the required duration, or enter the duration music data. Both are desirable abilities given the different
directly in the text box provided, whichever makes the characteristics and needs of the datasets.
most sense to them at any given instance. All three One filter, for example, RepairFilter, identifies repeated
methods are linked, and changing any one automatically fragments of tune using a linear space and time algorithm.
updates the other two. While undoubtedly a crude method with little to no music
The remaining controls provide the same abilities as theory involved, in practice we have found it effective on
those in Figure 3: pull-down menus to control how much of MIDI file– based collections. In a large collection with mil-
a tune is searched for a match and the like; and button lions upon millions of notes, it is a useful way of reducing
controls to record, save, and load queries. the index to prominent parts of a tune. Furthermore, by
starting a query with one of the identified motifs, tunes that
are most likely variations of the same tune are also found.
Conclusions
This is how MELDEX provides its “find more tunes like
In this article, we have described the context to the music this” service.
information retrieval work at the University of Waikato. The distributed ability of the general digital library ar-
Motivated by a digital music library project, a workbench chitecture is also exploited by MELDEX for large-scale
has been developed that not only provides an environment collections, such as ones based around the Midimax dataset.
for experimentation, but is engineered to integrate smoothly In addition to basing matching on motifs, parallel indexes
with the group’s wider digital library software architecture, can be distributed around several computers, and even a
forming a key component in the services offered. single index can be divided into constituent parts for distri-
Using this design, the concept of plugins provides a bution. The receptionist component of the design is respon-
flexible approach for importing music files in a range of sible for issuing the query to the various collection servers
formats, and the ability for plugins to take options allows for and for gathering in and merging the results.
1098 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—October 2004Human computer interaction is also an important aspect Greenhaus, D. (1996). About the digital tradition. Retrieved April 23,
of the work, which benefits greatly from the expertise of the 2004, from http://www.deltablues.com/
Hoos, H., Renz, K., & Gorg, M. (2001). GUIDO/MIR: An experimental
wider digital library project. Above we presented two applets musical information retrieval system based on GUIDO music notation.
developed in Java that provide better user interface support In J.S. Downie & D. Bainbridge (Eds.), Proceedings of the Second
than is available using HTML alone. Using the user’s own Annual International Symposium on Music Information Retrieval (IS-
computer resources, the first applet provides a rich environ- MIR 2001) (pp. 41–50). Bloomington, IN: Indiana University.
ment for constructing a music query with the ability to play- Larsson, N., & Moffat, A. (1999). Offline dictionary-based compression.
Proceedings of the IEEE Data Compression Conference (pp. 296 –305),
back, save, record, and show the user the query transcribed Snowbird, Utah.
before the more costly operation of searching is activated on Mongeau, M., & Sankoff, D. (1990). Comparison of musical sequences.
the digital music library server. The second applet enriches the Computers and the Humanities, 24, 161–175.
playback experience of the matching tunes. It displays a visu- Paynter, G., & Witten, I. (2001). A combined phrase and thesaurus browser
alization of where in the tune matches occurred and how close for large document collections. European Conference on Digital Librar-
ies (pp. 25–36), Darmstadt, Germany.
each of these matches was to the original query. It also plays Schaffrath, H. (1992). The ESAC databases and MAPPET software. Com-
the tune and provides the ability to skip ahead to the next puting in Musicology, 8, 66.
section of tune that matched the query. Witten, I., Bainbridge, D., & Boddie, S. (2001). Power to the people:
End-user building of digital library collections. Joint ACM and IEEE
Conference on Digital Libraries (pp. 94 –103), Roanoke, Virginia.
References Witten, I., McNab, R., Boddie, S., & Bainbridge, D. (2000). Greenstone: A
comprehensive open-source digital library software system. Proceedings
Bainbridge, D. (1997). Extensible optical music recognition. Unpublished of the Fifth ACM Conference on Digital Libraries (pp. 113–121), San
doctoral dissertation, Department of Computer Science, University of Antonio, Texas.
Canterbury, Canterbury, New Zealand. Witten, I., McNab, R., Jones, J., Apperley, M., Bainbridge, D., & Cun-
Bainbridge, D., & Bell, T. (1996). An extensible optical music recognition ningham, S. (1999a). Managing complexity in a distributed digital
system. Proceedings of the Nineteenth Australasian Computer Science library. IEEE Computer, 32(2), 74 –79.
Conference (pp. 308 –317), Melbourne, Australia. Witten, I., Moffat, A., & Bell, T. (1999b). Managing gigabytes: Compress-
Bainbridge, D., Buchanan, G., McPherson, J., Jones, S., Mahoui, A., & ing and indexing documents and images. San Francisco, CA: Morgan
Witten, I. (2001). Greenstone: A platform for distributed digital library Kaufmann.
applications. European Conference of Digital Libraries (pp. 137–148), Wu, S., & Manber, U. (1992). Fast text searching allowing errors. Com-
Darmstadt, Germany. munications of the ACM, 35(10), 83–91.
JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—October 2004 1099You can also read
