A Usability Survey at the University of Mississippi Libraries for the Improvement of the Library Home Page
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
A Usability Survey at the University of Mississippi
Libraries for the Improvement of the Library
Home Page
by Elizabeth Stephan, Daisy T. Cheng, and Lauren M. Young
Available online 27 January 2006
A usability survey was conducted at the INTRODUCTION
University of Mississippi Libraries as part of the The University of Mississippi Libraries has maintained a Web
ongoing assessment of the library and its site since 1994. Its earliest version was based on a static
format that provided access to the library catalog, a subject
services. By setting criteria to measure the guide, a personnel directory, and library hours. The pages
success of the survey, librarians at UM were were poorly coordinated, the navigation not intuitive. A
able to assess if the library home page usability survey was administrated to a group of high school
students in the summer of 2003 with the goal of collecting
successfully met the goals and mission statement
user feedback on the site. The results indicated that improve-
of the library. ment was necessary.
In July 2003, a Web Redesign Task Force was formed to
overhaul the Web site in order to provide a more dynamic
and logical structure that would provide easier navigation
and understanding. The task force was under the time
constraint of having the new pages revamped by the
beginning of the fall semester. The new library Web site
was launched in September 2003 without further site
analysis. Most users seemed to like the new site much
better than the old one. A few public service staff reported
that some of the users were confused and were unable to
find information. As a fundamental part of the University of
Mississippi Libraries’ mission is to ‘‘increase access to
information and communication on campus,’’1 and as the
Internet has become the predominant medium for the dissem-
ination of library resources, it was essential that the adequacy of
the new Web site be assessed, specifically the new home page. A
Usability Survey Committee was formed to conduct a survey to
assess the library home page as a part of the ongoing
assessment of the library and its services. Certain criteria were
established to determine whether the home page had met our
Elizabeth Stephan is Business Reference goals of increasing access to information. Our benchmark for
Librarian/Assistant Professor, success was for at least 75% of students to be able to
The University of Mississippi, MS 38677, USA successfully complete each task. If the criteria for success
bestephan@olemiss.eduN; were not met, the results would be used to improve the design
Daisy T. Cheng is Head of Cataloging/Assistant Professor, of the page.
The University of Mississippi, MS 38677, USA The committee consisted of six faculty and staff members
bdtcheng@olemiss.eduN; and began meeting in January 2004. As indicated by Nicole
Lauren M. Young is Instructor/Outreach Services Librarian, Campbell, a usability survey is ‘‘a method that tests how a
Rowland Medical Library, user interacts with a system. The participant is given a list of
The University of Mississippi Medical Center, pre-defined tasks to accomplish using the system and asked
Jackson, MS 39216-4505, USA to dthink out loudT about their thoughts, reactions, and
blmyoung@rowland.umsmed.eduN. feelings.’’2 The committee decided to adopt this method as
The Journal of Academic Librarianship, Volume 32, Number 1, pages 35–51 January 2006 35
an assessment tool in order to improve the library home Much of the literature emphasizes the amount of
page. preparatory work required in the creation of a usability
In February 2004, a usability survey was administered to survey. In the National Institute of Standards and Technology
a group of twelve undergraduate students. Each student was Virtual Library redesign project in 2001, librarians consulted
asked to complete eight tasks as administrators observed and with focus groups to get verbal feedback on the site’s
documented their actions. The tasks included both simple strengths and weaknesses, as well as consulting with a
and more complex searches and included using the library professional in the creation of their usability test design.8
home page, the library catalog, and databases from outside Continually recording usage statistics for the site and noting
vendors. the most used pages will help to identify areas on which to
concentrate.9 From this information, librarians can derive
LITERATURE REVIEW specific, answerable questions for the survey that will
There was not an abundance of literature discussing usability illuminate gaps in test takers’ knowledge.10 Scripted recruit-
surveys as a form of assessment. We analyzed the existing ment and test dialogue ensures consistent testing,11 as does
body of literature from two angles. We selected a few practicing test administration on an individual before
articles that discussed facets of successful library Web sites performing the actual test.12
and then turned to literature discussing usability surveys, When dealing with the actual testers, a contrived cross-
their use, design, implementation, and effectiveness. The section of the user population is encouraged, as are incentives
articles chosen were for the most part limited to academic and a preconceived strategy for testing locations and struc-
library settings for more direct application to our survey ture.13 The body of literature indicates that shorter (half-hour)
planned for the University of Mississippi Libraries. tests with a few specific questions are preferable to longer
As a foundation, select resources were consulted that tests.14 According to Jakob Nielsen, libraries engaging in
highlighted elements necessary for library Web sites to be usability testing will achieve the best results when testing a
deemed successful. In one such article, Sandra Shropshire small number of participants multiple times in order to reach
outlines her efforts to identify primary concerns that four the target fifteen test-user goal.15
medium-size academic libraries had regarding their Web Making sure that survey participants are comfortable is
sites. Shropshire found many concerns to be echoed at all always a concern when conducting a usability survey. Several
four institutions, including a lack of enough staff to create usability survey case studies that offer suggestions on this point
and maintain an effective site and to answer patron are featured in Campbell’s Usability Assessment of Library-
feedback; the need for a library Intranet; the need for a Related Web Sites. In a survey performed at the University of
designated back-up person in the absence of the systems Nevada-Las Vegas, student workers were used to administer the
librarian; the navigation of the when/why/how of redesign; survey as a way to reduce intimidation.16 A survey performed at
and the integration of the OPAC with the rest of the site. Arizona State University was administered in an empty office
Without these core elements in place, a library will not be decorated with posters, plants, and items to help ensure privacy
able to meet users’ needs efficiently and effectively.3 and a relaxed atmosphere.17
Leo Robert Klein reminds academic librarians that Web The literature indicates that across the board, testers
sites are meant to serve the patrons’ information needs, not employed the ‘‘think-aloud’’ method and acted as observers,
the librarians’. He contends that library users are busy, non- recording qualitative measurements such as the test taker’s
selective, and far more interested in finding their resource level of satisfaction with his answer and quantitative
than in learning how to navigate a convoluted library Web measurements such as the pathway taken to arrive at the
site. He challenges librarians to embrace elements found in correct Web page and the amount of time taken to do so.
popular sites such as Google, as these are the sites with When Susan Feldman of Datasearch, an information system
which their patrons seem to be most comfortable.4 Nicole Web design company, performs tests on prototype sites for
Campbell looks closely at this issue as well and urges clients, she videotapes the sessions for later review.18
librarians to make sure that all of the hard work that goes Fourteen students observed while doing personal research
into library Web site design and maintenance leads to a at Roger Williams University were tape-recorded while
product that the patron can actually use.5 performing tasks in the think-aloud, observation/interview
Before starting a usability survey, one must always ask: method.19
What exactly is usability? All academic libraries actively The literature we reviewed makes several points very clear:
seek to have a streamlined, intuitive Web site for their There are differing opinions on what made a Web site ‘‘usable’’;
patrons. Susan McMullen notes that distance learning different opinions exist about how a survey should be
programs and users’ remote access preferences heighten the administered; and the length of studies varied from institution
need for self-explanatory Web sites, as there is sometimes to institution. All of the above literature helped us develop and
not an opportunity provided for formal instruction. Some administer a usability survey that fit our needs and goals.
sites still prove to be difficult for new users and experienced
users alike, failing to lead users through the pages to the GOALS AND OBJECTIVES
information sought, and turning them off from the library The goal of the usability survey at the University of Mississippi
Web site all together.6 Characteristics such as these, contend Libraries was to improve access to library resources through
Elaina Norlin and CM Winters, are what make a site un- library home page by examining the way the undergraduate
usable. In order to ascertain if a library site is usable, the used the home page. The objectives were as follows:
back-door approach of finding what does not work, achieved
with a usability survey, proves to be far more fruitful than 1. to find out if students could find the basic library sources,
lauding one’s self on what does work.7 for example a book, a journal article, a newspaper article,
36 The Journal of Academic Librarianship
and personal account information, from the library home fifteen undergraduate students. We wanted to have a cross-
page; and section of grade levels and looked at recruiting students
through mass e-mails or simply asking for participants from
2. to see if 75% of the participants successfully completed students in the library. In the end, we did neither because
various tasks using the most efficient number of clicks. of time constraints. We had to complete the entire survey in
five weeks. This did not allow us the time we would have
needed for recruitment. Instead, we decided to recruit the
METHODS students from an existing library course. With the cooper-
Survey Design ation of the instructor, students from the class of
Introduction to Library Resources and Electronic Research
The survey administered to a group of high school students (EDLS 101) were selected for the survey. At this point in
the previous summer was adopted as a starting point. Two of
the semester they had not covered the use of the library
the tasks – locating a copy of Rolling Stone magazine and
catalog or other library resources. There were fifteen
locating an article about Internet retailing in EbscoHost – were students enrolled in the class, representing every grade
taken directly from the previous survey. The remaining content
level. Only twelve students attended class the day of the
was modified based on input from reference librarians who
survey. We discussed recruiting three more students to raise
deal directly with students and who had observed common
our number to fifteen. After looking at our results, we
problems with the home page.
decided that we had sufficient information to assess the
It has been noted that for a one-hour session, the survey
library’s home page.
should be limited to seven to ten tasks.20 Because we were
Since we used students from a scheduled class, we
working within a 50-minute time frame and knew that we
conducted our entire survey in a single fifty-minute period
needed to allow time for students and administrators to get
instead of administering the survey over several weeks.
settled, we chose to limit our survey to eight questions.
While this may not have been the ideal situation, it worked
Before it was administered to students, the survey was with our schedule. We were able to separate the students so
tested on a non-library user and adjustments were made as
that they could not hear or observe others taking the survey.
needed. We found that some of our language was too heavy
Students did not know before attending class that day that
on ‘‘library ease,’’ meaning that only librarians would fully they would be taking the survey, and each student was
understand the directions. When we asked our non-library
given four points extra credit for participating. The
user to locate a copy of The Catcher in the Rye, we realized
University of Mississippi requires any researchers using
we had not noted in the task that it was a book. As we
human participants to file for permission through the
specifically wanted the users to find a book, that detail was
Institutional Review Board (IRB). The IRB required that
added. We also discussed whether we were telling students
we have each student sign a release form and be given the
how to complete the task by using language used on the
choice to not take the survey. The instructor of the class
existing home page. If we asked students to find a subject
was a member of the committee and did not take part in the
guide, was that going to skew the results since the link was
administration of the survey as she felt the students might
clearly labeled ‘‘Subject Guides?’’
feel uncomfortable or that they were being graded if she
We decided that in some cases we wanted to use terms used was involved.
on the existing home page because we wanted to know if
Each student was assigned to an administrator, and each
students understood what that term meant. On the other side of
administrator/student pair was assigned to a computer.
this issue, we asked students to find an article in Ebsco. The
Students were given a number after they signed the release.
library’s Ebsco databases are labeled by their names, ‘‘Aca-
When they walked out of the library classroom they matched
demic Search Premier (Ebsco),’’ ‘‘Business Source Premier,’’
their number with the number each administrator was given.
etc. But both students and faculty refer to ‘‘Academic Search’’
Administrators knew what their number was and to which
simply as ‘‘Ebsco.’’ We wanted to know if students understood
computer they were assigned, which helped to streamline the
how to get to the databases page and if they understood that
process. We used office computers in the library’s Technical
when they were looking for ‘‘Ebsco,’’ they were really looking
Services Department as well as one additional office
for ‘‘Academic Search Premier.’’ computer and two computers in our computer lab. Each
Administrators student was read a statement explaining the purpose of the
Fifteen library faculty and staff members were recruited to survey and was reassured that they were not being tested
administer the survey. Before the survey was conducted, (see Appendix B).
administrators were required to attend a training session. Not It is not easy to make a student feel comfortable in a
everyone was familiar with the purpose of a usability survey, cubicle of a busy office, but we did what we could. Our
much less how to administer a survey. To achieve uniformity of administrators allowed us to use their cubicles and we were
survey results, administrators were trained on how to give the able to find some empty areas in the same office. We placed
survey, what to say, and what not to say (see Appendix A). Also students in cubicles that would not be adjacent to one
discussed were each of the questions and what type of com- another so they could not hear the other administrators or
ments and behaviors they were to look for and what to record. students. We asked that personal items (pictures, notes, etc.)
be removed from computer monitors and asked that the
Students computer screens be set to the University of Mississippi
We debated how to recruit students. Going by Jakob home page, the starting point of our survey.
Nielsen’s rule that with fifteen students you will learn what We assigned three students to two other locations. Two
you need, we decided to limit our survey participants to students were assigned to our bibliographic instruction room.
January 2006 37
Figure 1
Web Experiences
Figure 2
The University of Mississippi Home Page (www.olemiss.edu). The Arrows Point to the Links to the
Library’s Home Page. The University Libraries Link (1) Was the One Most Students Used; The Libraries and
Museums Link (2) was Used Less, Although this Link Took the User Directly to the Home Page While the
University Libraries Link Went to an Intermediary Page. The UM Home Page has Since Been Redesigned
38 The Journal of Academic Librarianship
We placed them far apart and chose administrators with quieter had completed the task; average when a student thought
voices in order not to disrupt each other. One student was he/she had completed the task but was not positive; and
placed in a private office with one administrator. This student low when a student did not think he/she had completed
took the longest to complete the survey. Whether this was the task.
related to his location or not was inconclusive.
Qualitative Metrics
Measurement
Qualitative measures are based on the administrator’s
A student’s response to each question was scored by a observation and both the administrator’s and student’s com-
number of quantitative and qualitative measures that included ments. These included the following:
the number of clicks, the time it took to complete the task, as
well as the administrator’s observation of the student’s action o Signs of indecision: the administrator wrote down any
and his/her comments. indecision or hesitation observed in student.
Quantitative Metrics o Indications of frustration: the administrator recorded a
o Number of clicks to reach destination: a click was counted if student’s frustration when he/she started to grumble or
it took a student to a different page. If a student went back to mumble to himself or herself about the difficulty of the
the previous page, that was counted as a click. If a student task.
clicked a checkbox on a page (i.e., ‘‘full-text’’ box in
EbscoHost), that did not count as a click. o User comments: the student’s oral observations, such as
‘‘the catalog page looks a lot like the home page,’’ or
o Time required to complete the task: the amount of time it comments about what he/she would like to see, were
took to complete each task. recorded.
o Completed the task (yes/no): if a student said that he/she o Observer comments: the administrator’s own observations
could not reach the page, the task was not complete, even if and comments.
they had indeed reached the page and did not realize that SURVEY RESULT
they had done so.
Demographic Questions and Results
o Satisfaction level (high/average/low): administrators were Before the survey began administrators asked students a
asked to rate high when a student was certain that he/she number of questions ranging from their class and major to how
Figure 3
Students Were Confused By the Multiple Listings for The Catcher in the Rye, But Seeing the Author’s Name
Listed in the Second Record Helped Some Students
January 2006 39
Figure 4
The Students Who Found a Record for Rolling Stone in Journal Finder (TD Net) Misinterpreted the
Record. They Thought the Library Only had it in Electronic Format
Figure 5
When Looking for a Copy of Rolling Stone, Most Students Did a Title Search in the Catalog. Many
Were Confused by the Results List. Not Seeing a Listing for Rolling Stone Magazine Lead Some to Think
we did not Have a Current Copy in the Library
40 The Journal of Academic Librarianship
comfortable they were with using the library Web page. This results and what students felt their online research com-
information was used to help us understand our survey petencies were.
participants. The average ratings that upper and lower classmen gave for
using the library Web page and ‘‘other’’ Web pages (Google,
What Class are You?
Yahoo, etc.) were similar. The lowest score given for searching
Our goal was to have an even representation of under- the library Web page was 1; the highest, 8. The lowest for
graduate class levels. More seniors participated than any other ‘‘other’’ was 2, the highest, 10 (Fig. 1).
class level, but as a whole there was an even representation of
undergraduates. Survey Findings
Find the University Library Home Page
What is Your Major?
While we have little influence over the design of the
As with the class level, we had hoped to get an even University of Mississippi’s home page (www.olemiss.edu)
representation of majors. In this survey business majors made (Fig. 2), we wanted to know if students knew how to get to
up half of our participants. Approximately 25% of UM students the library home page; if a student cannot even locate the library
are in the School of Business Administration, so our sample did home page it is not possible for us to provide any online
not accurately reflect the make-up of UM majors. services.
Twelve out of twelve (100%) students completed this task;
On a Scale From 1 to 10 (10 Being the Highest), How Much
of these, ten were noted as being highly satisfied and two were
Experience Do You Have Using the Library Web Page? and at average satisfaction. No students were noted as having low
On a Scale From 1 to 10, How Much Experience Do You Have
satisfaction because all completed the task. The average
Using a Web Site, Such as Google, to Find Information?
number of clicks used to complete the task was 1. It took the
The purpose of these two questions was to get a feel for survey creators one click. The least number of clicks needed to
how the students ranked themselves in using both the complete the task was one; the most was eight.
library Web resources and Internet search engines such as Despite all of the students completing the task, not all of
Google. This provided context when looking at survey them found the library page easily. There were two links to two
Figure 6
The Two Links Where Students Could Find the Library Hours (1 and 2). Before Finding the Correct Link,
Some Students Looked Under Library Quick Links (3)
January 2006 41
Figure 7
Nine Out of Twelve (75%) Students Were Able to Successfully Find the Course Reserves Page
different pages on the UM’s home page: the ‘‘University ately clicked on the ‘‘University Libraries’’ and commented
Libraries’’ link went directly to the library site and the that she did that because ‘‘she knew it was the library Web
‘‘Libraries & Museums’’ link went to an intermediary page page.’’
with separate links to both the library page and the Museum Eight out of twelve went through the intermediary page.
page. Three out of twelve students reached the library page One student did not realize the intermediary page was not the
directly through the ‘‘University Libraries’’ link. One student library page. The administrator said, ‘‘Please go to the library
began to do a keyword search on UM’s home page before he home page’’ when it became apparent the student did not
saw the ‘‘University Libraries’’ link. Another student immedi- realize that he was not at the library home page.
Figure 8
Students had Problems with the Professor’s Name. Several Did Not Read the Instructions and Typed
‘‘E. Smith’’ Instead of ‘‘Smith, E.’’
42 The Journal of Academic Librarianship
One student clicked on the ‘‘University and Libraries’’ took the survey creators four clicks to complete the same
page and got stuck. He clicked back forth between the task.
UM’s home page and the ‘‘Libraries & Museums’’ page All of the students found the library catalog from the library
several times before turning to Google. He found the home page. Two students hesitated at the home page before
library site by doing a Google search for the ‘‘Ole Miss clicking on the ‘‘Catalog’’ link. The two unable to complete the
Library.’’ task got as far as the title list in the ‘‘Catalog’’ (Fig. 3). When
Because of the two different links and paths to the library they got to the title list, they did not know which record was for
home page, we were not surprised by the results. After the the book.
survey was completed and the results tabulated, we did ask the The Catalog proved confusing even to those who success-
university webmaster to make both library links go directly to fully completed the task. Like those who did not complete the
the library home page. task, those who did were confused by the title page but
located the correct record. One student initially searched by
Does the Library Have a Copy of the Book, The Catcher in
author but quickly realized his mistake when he did not get
the Rye? any results. He was able to correct his initial search and
Locating items in the library catalog was key to the successfully locate the record. The ‘‘Library Search Engine’’
assessment of our services. We wanted to know if students (MetaFind page) was used by one student, but when it locked
knew where to go if they needed to look up a book, and if they up he went to the ‘‘Catalog’’ and conducted a successful title
got there, if they could find one. search.
Ten out of twelve (83%) were able to successfully
complete this task, two out of twelve (17%) were not. Nine Does the Library Have a Current Copy of Rolling Stone
were highly satisfied with their results, two had an Magazine?
average satisfaction (meaning they thought they found This proved to be the most difficult task. Three out of
it but were not sure), and one was rated at low satis- twelve (25%) students were able to successfully complete
faction. The average number of clicks used in this task this task; nine (75%) were unsuccessful. Only one had a
was 3.6 including those who did not complete the task. It high satisfaction rate with two average, and nine marked as
Figure 9
Twelve Out of Twelve (100%) Students Were Able to Locate a Subject Guide by Using One of
the Two Subject Links on The Library Home Page
January 2006 43
low. It took the students an average of five clicks, that stopped. At this point he commented, ‘‘either I went to the
includes the number of students who did not complete the wrong place or they do not have it.’’
task. It took the survey creators two clicks through ‘‘Journal One student skipped the ‘‘Catalog’’ and went directly to
Finder’’ (TDNet page) and four through the Library ‘‘Journal Finder,’’ where she was able to find a record for
‘‘Catalog.’’ Rolling Stone. When she saw the TDNet record she said, ‘‘I
Students went different directions with this task. For their guess it’s like EBSCO, not here in the library’’ (Fig. 4) (TDNet
initial search, six went to the ‘‘Catalog,’’ one went directly to records indicate whether we have a periodical in print, online, or
the ‘‘Library Search Engine,’’ one went to ‘‘Journal Finder,’’ both). She either failed to see that it was available in print or did
and two went to the ‘‘Articles and Journals’’ page. We do not not understand how to read the record.
know the path taken by two students. The problems encountered with the ‘‘Catalog’’ were
Nine out of twelve students were not able to complete this similar to their problems with The Catcher in the Rye task.
task, six went directly to the ‘‘Catalog,’’ and two went other They were confused by the title list (Fig. 5). Many students
directions. Of the six who searched in the Catalog, two were did a keyword search instead of a journal title search. One
successful in locating a current copy of Rolling Stone; four student did a journal title search and was still unsuccessful.
were not. Of the four unable to locate the magazine, two did He scrolled up and down the results page while mumbling
not try another route. ‘‘Rolling Stone magazine, Rolling Stone magazine. . ..’’ The
The two who did try a second route went to the ‘‘Articles & first item listed was the record for Rolling Stone magazine;
Journals’’ page where one tried ‘‘Journal Finder’’ but was we can speculate that because it did not read the Rolling
unsuccessful, and the other student tried the ‘‘Library Search Stone magazine,’’ he did not click on it. This reaction to the
Engine’’ without any success. He returned to the ‘‘Catalog’’ title list was common.
and did a title search but only found Rolling Stone books. At Three out of twelve students were successful in completing
this point, he stopped. this task. Three went to the ‘‘Catalog’’ immediately; one
One student went directly to the ‘‘Article Quick Search’’ successfully found the record. The second student did a keyword
(MetaFind) on the ‘‘Articles & Journals’’ page. Unable to search and could not locate the record. The student returned to
locate the article, he went to the ‘‘Databases’’ page where he the library home page and went to the ‘‘Articles and Journals’’
Figure 10
Eleven Out of Twelve (92%) Participants Were Able to Locate Their Library Account Information From the
Home Page. One Student Returned to the University of Mississippi Home Page and Looked Under Registration
44 The Journal of Academic Librarianshippage without any success. After returning to the library home had an average satisfaction, and no students were marked at
page a third time the student clicked on ‘‘Journal Finder’’ and low satisfaction. There were two links to the library
found the record. Another student tried the ‘‘Library Search ‘‘Hours’’ page on the library home page: one in the main
Engine,’’ and when that did not work he found the record in the navigation bar at the top of the page and one in the blue
‘‘Catalog.’’ side box.
The different and numerous avenues students took to One problem we had with this task was that seven of the
complete this task demonstrated their determination to find twelve administrators did not note which ‘‘Hours’’ link the
what they needed. They knew what the ‘‘Catalog’’ was and students chose. We do know that three of the twelve chose the
how to find it but did not always know how to use it. It was ‘‘Hours’’ link in the blue side box while only two chose the
also evidenced that the ‘‘Articles and Journals’’ link was link in the top navigation bar. Three students showed some
confusing. Students immediately went there but were not sure hesitation at the beginning of the task. Of those three, two
what to do when they got there. Or, they tried ‘‘Library Search initially went to the ‘‘Library Quick Links’’ drop-down box
Engine’’ but it would not work (we did find out later that this before seeing one of the links on the library home page (Fig. 6).
might have been due to some technical issues we were not When the student noticed the ‘‘Hours’’ link he commented that
aware of at the time of the survey). We initially assumed he ‘‘should’ve seen that [link].’’ One student, when asked
students would use the ‘‘Catalog’’ first and then ‘‘Journal ‘‘What are the hours of the Library?,’’ turned around and
Finder.’’ While they did use the ‘‘Catalog,’’ few of them responded, ‘‘Seven to midnight.’’ Then he realized he needed
noticed or tried ‘‘Journal Finder.’’ to locate the hours on the home page. He immediately went to
the link in the navigation bar.
What are the Hours of the Library? One general observation was that the most current hours
This was one of the easiest tasks for our students with were found at the bottom of the ‘‘Hours’’ page, such as special
twelve out of twelve (100%) students completing the task. holiday hours or normal hours resumed. A reverse chrono-
Eleven students were noted as having high satisfaction, one logical order that places the most recent update at the top of the
Figure 11
When Looking for a Full-text Article, Students Were Not Sure Where to Go First, Some Went Directly to
Databases (1), Others Went to Articles and Journals (2). Those Unsure of Where to Go Clicked on the
Library Quick Links Box (3). Some Students Used the Library Search Engine (4) But it Would Lock Up
After They Entered Their Search
January 2006 45page was determined to be clearer, and this was fixed shortly same six, three commented that they did not know course
after the survey was completed. reserves could be looked up through the library home page.
Six out of the twelve were unable to complete the task.
Does Prof. E. Smith Have Anything on Reserve? Three of these six were the students mentioned above who
This question was designed to see if students knew how to located the ‘‘Course Reserves’’ page but searched under ‘‘E.
locate course reserves from the library home page through the Smith.’’ The remaining three were unable to locate the ‘‘Course
‘‘Course Reserves’’ link. Most faculty tell their students they Reserves’’ page. Two looked under ‘‘Contact a person’’ and
have put an item on ‘‘reserve.’’ We wanted to know if students one said he did not know how to do that and did not make an
knew to go to the ‘‘Course Reserves’’ link on the library home attempt.
page when looking for items on reserve.
Six out of twelve (50%) students were able to complete the Suppose You are Taking a Class on a Subject Unfamiliar to
task; six were not. Three were highly satisfied with their You; Find a Subject Guide Relating to That Topic
results, four were average, while five had low satisfaction with Twelve out of twelve (100%) of the survey participants were
their results. This task took students an average of 2.3 clicks; it able to complete this task. There was very little if any hesitation
took the creators two clicks. shown by students. It took students an average of 1.6 clicks to
The results are misleading. Nine out of twelve were able to get complete the task; it took the creators one. Several students
to the ‘‘Course Reserves’’ search page from the library home commented that it was ‘‘easy’’ or that this task was an ‘‘easy
page (Fig. 7). When searching for reserves by a professor’s name, one.’’ Each student was able to locate the ‘‘Subject Guides’’
the user is instructed to search by professor’s last name. Of the page and then chose a subject and clicked on the link to a
nine students who reached the ‘‘Course Reserves’’ page, three subject guide.
entered ‘‘E. Smith’’ instead of ‘‘Smith’’ or ‘‘Smith, E.’’ Of the six One student showed some hesitation after he was read the
students who did complete the task, two did an initial search for task. He looked over the home page and said, ‘‘I’m going to
‘‘E. Smith’’ but realized their mistake, returned to the ‘‘Course say dSubject Guides.T’’ Another student had no problem
Reserves’’ page, and searched under ‘‘Smith’’ (Fig. 8). Of those reaching the ‘‘Subject Guides’’ page. He chose the chemistry
Figure 12
Students Were Confused By the List of Databases. Many Tried to Click on the Basic Search or
Advanced Search Tabs at the Top of the Page But They Were Grayed Out. As a Result
of the Usability Survey, This Page is Being Phased Out of Use at UM
46 The Journal of Academic Librarianshipsubject guide. When he reached the page for chemistry subject as average. Three did not have a satisfaction level marked for
guide he hesitated. Judging from his body language, he seemed this task, but those three did complete the task. It took both the
unsure. He scrolled up and down the page, noting the list of students and creators an average of one click to complete this
databases and Internet sites. When he reached the bottom and task.
scrolled back to the top, he seemed satisfied he had completed Eleven out of twelve of survey participants were able to
the task. locate the link and the page where their name and barcode are
The library home page had two ‘‘Subject Guides’’ links: one entered in order to retrieve their library account information
on the main navigation bar and one in the main set of links in (we did not ask them to enter their personal information).
the center part of the page (Fig. 9). None of the administrators Students commented that it was an ‘‘easy one’’ and said they
noted the path the students took. But by looking at the average knew to go to ‘‘My Library Account’’ (Fig. 10). Of the students
number (1.6 clicks) of clicks it took students to find the subject able to complete the task, only one showed any sign of
guide, it was evident that they went through either of the two hesitation. This hesitation was caused when the student
links on the library home page. confused the catalog page with the home page. When he
This was one task where we questioned our terminology. By returned to the home page, he was able to easily locate the ‘‘My
using the term ‘‘subject guide’’ in the task, we were leading Library Account’’ link.
them directly to the ‘‘Subject Guides’’ link on the home page. One student was unable to locate his library account. When
This question was loosely based on a question used in the the task was presented to him, he commented that he ‘‘did not
usability survey conducted in 2003 and the same terminology know’’ and that he had ‘‘no clue’’ how to find the information.
was used. After some debate, the decision was made to use the He returned to the University of Mississippi home page and
term ‘‘subject guide’’ considering that it was not only the term looked under the Registration link.
used on the home page (which was out target of assessment) This is another case where we may have found out more if
but also the term most likely to be used by both teaching and we had worded the task differently. We know that a large
library faculty. majority of our students know how to find their library account.
Does the same majority know that this is where they go to see
How Do You Look at Your Library Account? if they have a book overdue or to renew a book? We can only
A patron’s library account allows them to see what items speculate that because they knew how to locate their library
they have checked out, renew books, etc. Being able to find account that they know what it is for.
this information will allow users to access their library
information easier and faster. While it is not a new feature Locate a Full-text Article in Ebscohost About Internet
the University of Mississippi Libraries, we did not know if Retailing
students knew it was available or if they knew how to find their This proved to be a difficult task. Five out of eleven students
information. (45%) completed the task; six out of eleven (55%) did not. One
Eleven out of twelve (92%) were able to complete this task; student was classified as a ‘‘yes and no’’ because Ebsco timed
one out of twelve (8%) were not. Eight were marked as being out before he could get to the search page; therefore, the
highly satisfied, one had low satisfaction, and none were rated percentages for this task are based on eleven surveys instead of
Table 1
A Task was Considered Success if 75% of the Participants Completed the Task
Number Task Complete Incomplete
Q1 Find the University Library Web page. 2 out of 12 (100%) 0%
Q2 Does the library have a copy 10 out of 12 (83%) 2 out of 12 (17%)
of the book The Catcher in the
Rye by J.D. Salinger?
Q3 Does the library have a current 3 out of 12 (25%) 9 out of 12 (75%)
copy of Rolling Stone magazine?
Q4 What are the hours of the library? 12 out of 12 (100%) 0%
Q5 Does Professor E. Smith have any 6 out of 12 (50%) 6 out of 12 (50%)
course reserves?
Q6 Suppose you are taking a class on 12 out of 12 (100%) 0%
a subject unfamiliar to you, find a
subject guide relating to that topic.
Q7 How do you look at your library account? 11 out of 12 (92%) 1 out of 12 (8%)
Q8 Locate a full-text article in EbscoHost 5 out of 11 (45%) 6 out of 11 (55%)
about Internet retailing.
This table shows which tasks met our criteria.
January 2006 47twelve. The satisfaction levels, however, did not match up with One student never even tried. When asked to look for an
the level of completion. Three out of eleven were noted to be article in EbscoHost he asked, ‘‘What’s that?’’
highly satisfied with the results, two out of eleven were The EbscoHost Web page also proved confusing to those
average, and six out of eleven were low. It took students an students who were able to complete the task. One student in
average of 4.1 clicks to complete the task; it took the creators particular clicked all over the page including the grayed-out
of the survey three (Fig. 11). Basic Search tab. She continued to click back and forth
Of the students who did not complete this task, many between the alphabetical databases page and EbscoHost
experienced similar problems. Two students were confused by Web. After reading the definitions of the Ebsco databases,
the EbscoHost Web page they had to pass through to get to she chose the Newspaper Source where she found an article.
Academic Search Premier (or any other Ebsco database). The The ‘‘Library Search Engine’’ was another popular choice.
list of databases was in no particular order and was confusing Some students were successful and some were not. One student
to students. Students that got stuck here tried to click on the went to the ‘‘Library Search Engine’’ and searched for
Basic Search tab but it was grayed out. After scrolling up and ‘‘EbscoHost.’’ After not getting any results, he searched for
down the page, clicking on grayed-out tabs, and reading ‘‘Internet retailing.’’ He clicked on the fist result with ‘‘Ebsco’’
definition tabs, one student finally commented that he had ‘‘no in the URL and was taken to an article in EbscoHost.
idea’’ how to locate an article. The other student stuck on this When designing this question we chose to ask students to
page said he was ‘‘looking for some place to type dInternet search in ‘‘EbscoHost’’ because most of their professors tell
retailingT’’ but could not find one (Fig. 12). them to find articles in ‘‘Ebsco’’ as opposed to Academic
Another student bypassed the library home page and went Search Elite, Newspaper Source, etc. Students come to the
directly to www.ebscohost.com. Because of the library’s reference desk and ask for articles from Ebsco. In addition
subscription, the site recognized him as a subscriber; he was to finding out if the library home page gave them enough
able to search using ‘‘Internet’’ and ‘‘retailing’’ as keywords. direction to get to a database, we wanted to know if they
He got sixty-six results but did not know how to access the full- associated any specific database with ‘‘EbscoHost.’’ We
text articles. found that most knew what EbscoHost was because several
Figure 13
The Redesigned Library Home Page
48 The Journal of Academic Librarianshiptold us they had used it before. It was the library site in APPENDIX A. INSTRUCTIONS FOR ADMINISTRATOR
combination with the EbscoHost Web page that caused
Instructions for Administrator
confusion. The library is in the process of phasing out the
EbscoHost Web page because of what we learned from the Be at Lyceum Circ Desk at 2:05
usability survey. Bring
A watch
CONCLUSION Form
Because the usability survey was administered for our biennial Number
assessment, we assigned some criteria in order to measure our Pen/pencil
success. For assessment purposes, the library home page would Book, folder, clipboard
meet our goals of making library resources accessible if at least
75% or nine out of twelve participants would be able to Computer
successfully complete each task. The rate of completion ran It should be set up and ready to go when you get there. A
from three out of twelve (25%) to all (100%) students, with member of the committee will come around and check
five of the eight tasks completed by at least ten out of twelve computers before you arrive.
(83%) participants. Table 1 shows which tasks met our
requirements. Going by our criteria, we found users were able Survey Begins
to successfully complete the directional tasks (Q1, Q4, Q6, and Read Script to your student.
Q7) with at least eleven out of twelve students (92%) being Time started: Time after you read script and before first
able to complete these tasks. The more complex tasks (Q2, Q3, question asked
Q5, and Q8) proved to be more difficult. When students did not Time ended (on last page): Time last task completed
know where else to go, they were drawn to the ‘‘Library Quick
Links’’ drop-down box and to the blue side box. We already While students are completing each task they will ask a lot
knew students liked the blue box but we had always dismissed of questions: ‘‘Is this right?’’ ‘‘Is this what you want?’’ ‘‘Did I
the drop-down box until we saw how much students used it. In spell this correctly?’’ You can’t answer their questions. Some
general, there was a lot of confusion between the library home possible responses to these questions: ‘‘Do you think it is
page and main ‘‘Catalog’’ page. On several occasions, students correct?’’ or read the question again. They may also say ‘‘Is
would stop at the ‘‘Catalog’’ page not knowing they were not at this what you wanted’’ when they get to the desired page. If
the library home page. possible restate the question. Hopefully, they will realize they
Some issues raised by the usability survey have been resolved are at the correct page. If after your prodding they do not
through Web design. We redesigned the library home page giving realize they have completed the task tell them to move on to the
more prominence to the ‘‘Databases’’ and ‘‘Subject Guides.’’ We next question.
reordered the items in the blue sidebar box, and we plan on
Quantitative Metrics
redesigning the page header to make the ‘‘Home’’ and ‘‘Inter-
Library Loan’’ links more visible. We eliminated the ‘‘Articles & Number of clicks to reach destination:
Journals’’ page and have begun working on a series of pages The number of clicks listed after each question is how it was
devoted to how to use the ‘‘Catalog’’ and ‘‘Databases’’ pages (Fig. reached by one librarian. One click is considered a click that
13). At our request, the secondary ‘‘Library & Museums’’ page has takes them to a different page. If they go back to the previous
been eliminated. However, the UM home page has since been page, that is a click. If the click a check-box on a page (i.e.,
redesigned and an intermediary page between www.olemiss.edu ‘‘full-text’’ box in Ebsco), that does not count as a click.
and the library home page has been added.
Some issues will have to be addressed by instruction—both Complete task: yes ____ no ____
in the classroom and at the reference desk. The survey showed If they get to the desired page they completed the task. If
us where we need to concentrate our instruction and in which they tell you they can’t get to the page then they did not
areas students are having problems. ‘‘Journal Finder’’ is useful complete the task. Let them decide. If it is obvious they ca not
but many students do not understand what it is for or how to figure it out, say ‘‘let’s move on to the next question’’
read the records. In conjunction with other committees, it was
decided to purchase the MARC records for journals we have in Satisfaction level: High ___ Average ____ Low ____
electronic format. When students search for a journal in the Was the student satisfied with the result.
catalog, they will get a record for both the print and electronic High: They completed the task and they know they
versions of a journal. ‘‘Journal Finder’’ will be phased out. completed the task.
Some areas of the ‘‘Catalog’’ are causing confusion. We have Average: They think they completed the task but they are
very little control over the design of the ‘‘Catalog,’’ but we not positive.
know what areas to concentrate on during instruction sessions. Low: They did not complete the task or the completed the
A usability survey is a lengthy and time-consuming project. task and did not know it.
Like some of the studies done by other universities, ours was
rushed due to the deadline. But the information we learned from Qualitative Metrics
observing our users using the library resources was invaluable. Signs of indecision:
Looking back, there are things our committee knows now that Use this to write down any indecision or hesitation you
we should have done differently, but all in all we feel it was a might see. If they point towards ‘‘Journal Finder’’ while
successful and worthwhile endeavor. Plans are in the works to looking for Rolling Stone and wonder if they should go there,
conduct a follow-up survey. make note of that here.
January 2006 49Indications of frustration: User comments:
When the student starts grumbling or mumbling to him- or Observer comments:
herself about how difficult something is, indicate that here. Please Return to the Library Home page
What did they say? When did they get frustrated? Please wait one moment
User comments:
We are looking for student observations: ‘‘The catalog page 2. Does the library have a copy of the book, The Catcher in the
looks a lot like the home page.’’ Or if the student makes a Rye by J.D. Salinger? (Four clicks from main library page)
comment about what they would like to see, put it here.
3. Does the library have a current copy of Rolling Stone
Observer comments: magazine? (Two clicks through Journal Finder; four clicks
Use this area to write any comments you have while through catalog)
observing the student (i.e., for task #3 you might write: ‘‘Was
confused by word dmagazineT’’). 4. What are the hours of the library? (One click from main
While these are your notes please understand others will be library page)
reading them. We do not expect full sentences but they do have
to make sense. 5. Does Professor E. Smith have any course reserves? (Two
clicks from main library page)
APPENDIX B. SURVEY FORM
6. Suppose you are taking a class on a subject unfamiliar to
TIME STARTED: _____________________________ you (Chemistry, Telecommunications, Civil Engineering,
Administrator’s Name_____________________________ Women’s Studies, etc.), find a subject guide relating to that
topic. (One click from main library page)
Script
Hello. My name is _____. I will be working with you in 7. How do you look at your library account? (One click to
today. Thank you for participating.
My library page) [They do not need to enter their personal
data. We just want to know if they know to go to ‘‘My
The library has used its Web site to provide information for
Library.’’]
several years and we want to know if our Web site is useful to
undergraduate students. In order to find this out, we want to
8. Locate a full-text article in EbscoHost about Internet
watch how you and other undergraduates use the Web site. I will
retailing. (Six clicks: four to get to Ebsco, two in Ebsco.)
be asking you to do a set number of tasks. I want to emphasize
that you are not being tested, and you are not being graded.
While you are doing these tasks please tell me why you are Thank you for your time.
doing what you are doing. If you click on a link, tell me why. It TIME ENDED:_______________________
is okay if you cannot complete a task or find some information;
there is no wrong answer. If you cannot complete a task, tell me NOTES AND REFERENCES
why and we will move on to the next item.
I will be taking notes on what you say and how you 1. Assessment Record for Department/Unit of University Libraries,
complete the tasks. Again, I am not testing you. Try your best 2001–2003, Form B: Expanded Statement of Institutional Purpose
Linkage, submitted 10 October 2003.
to ignore me and everyone else in the room. While I am asking 2. Nicole Campbell, ‘‘Usability Methods,’’ in Usability Assessment of
you to dthink out loudT as you are completing these task I Library-Related Web Sites: Methods and Case Studies, edited by
cannot answer any questions you may have. Do you have any Nicole Campbell (Chicago: Library and Information Technology
questions before we begin? Association, 2001), p. 2.
Before we get started I have a few general questions to ask: 3. Sandra Shropshire, ‘‘Beyond the Design and Evaluation of Library
What is your class rank? Web Sites: An Analysis and Four Case Studies,’’ Journal of
Class: Freshman Sophomore Junior Senior Other Academic Librarianship 29 (March 2003): 95 – 101.
What is your major? ________________________ 4. Leo Robert Klein, ‘‘The Expert User Is Dead,’’ Library Journal
On a scale from 1 to 10 with ‘‘1’’ being no experience to 128 (Fall 2003): 36.
‘‘10’’ being lots of experience, how much experience do you 5. Nicole Campbell, ‘‘Introduction,’’ in Usability Assessment of
Library-Related Web Sites: Methods and Case Studies, edited by
have using the Library Web page? Nicole Campbell (Chicago: Library and Information Technology
On a sale from 1 to 10, how much experience do you have Association, 2001), p. v.
using a Web site such as Google to find information? 6. Susan McMullen, ‘‘Usability Testing in a Library Web Site
Redesign Project,’’ Reference Services Review 29 (February
1. Find the University Library Web page. (One click from UM 2001): 7 – 22.
Web page) 7. Elaina Norlin & C.M. Winters, Usability Testing for Library Web
Quantitative metrics Sites: A Hands-On Guide (Chicago: American Library Associa-
tion, 2002).
Number of clicks to reach destination: 8. Susan Makar, ‘‘Earning the Stamp of Approval,’’ Computers in
Complete task: yes ____ no ____ Libraries 23 (January 2003): 16 – 21.
Satisfaction level: High ____ Average ____ Low ____ 9. Marshall Breeding, ‘‘Library Web Site Analysis,’’ Library
Qualitative metrics Technology Reports 38 (May/June 2002): 22 – 35.
Signs of indecision: 10. David King, ‘‘The Mom-and-Pop Shop Approach to Usability
Indications of frustration: Studies,’’ Computers in Libraries 23 (January 2003): 12 – 15.
50 The Journal of Academic Librarianship11. Barbara J. Cockrell & Elaine Anderson Jayne, ‘‘How Do I Find an of Nevada, Las Vegas Libraries,’’ in Usability Assessment of
Article? Insights from a Web Usability Study,’’ Journal of Library-Related Web Sites: Methods and Case Studies, edited by
Academic Librarianship 28 (May/June 2002): 122 – 132. Nicole Campbell (Chicago: Library and Information Technology
12. Breeding, ‘‘Library Web Site Analysis.’’ Association, 2001), p. 110.
13. Janet Crum, Dolores Judkins & Laura Zeigen, ‘‘A Tale of Two 17. Kathleen Collins & José Aguinaga, ‘‘Learning as We Go: Arizona
Needs: Usability Testing and Library Orientation,’’ Computers in State University West Library’s Usability Experience,’’ in Usabil-
Libraries 23 (January 2003): 22 – 24. ity Assessment of Library-Related Web Sites: Methods and Case
14. Crum, Judkins, & Zeigen, ‘‘A Tale of Two Needs;’’ Makar, Studies, edited by Nicole Campbell (Chicago: Library and
‘‘Earning the Stamp of Approval.’’ Information Technology Association, 2001), p. 20.
15. Jakob Nielsen, ‘‘Why You Only Need to Test With 5 Users,’’ 18. Susan Feldman, ‘‘The Key to Online Catalogs that Work? Testing:
Useit.com: Alertbox (March 19, 2000), http://useit.com/alertbox/ One, Two, Three,’’ Computers in Libraries 19 (May 1999): 16 – 19.
20000319.html (accessed December 31, 2004). 19. McMullen, ‘‘Usability Testing in a Library Web Site Redesign
16. Jennifer Church, Jeanne Brown & Diane VanderPol, ‘‘Walking the Project.’’
Web: Usability Testing of Navigational Pathways at the University 20. Norlin & Winters, Usability Testing for Library Web Sites, p. 32.
January 2006 51You can also read
