A Usability Survey at the University of Mississippi Libraries for the Improvement of the Library Home Page

 
A Usability Survey at the University of Mississippi Libraries for the Improvement of the Library Home Page
A Usability Survey at the University of Mississippi
               Libraries for the Improvement of the Library
               Home Page
               by Elizabeth Stephan, Daisy T. Cheng, and Lauren M. Young
               Available online 27 January 2006

        A usability survey was conducted at the                                                   INTRODUCTION
University of Mississippi Libraries as part of the                        The University of Mississippi Libraries has maintained a Web
       ongoing assessment of the library and its                          site since 1994. Its earliest version was based on a static
                                                                          format that provided access to the library catalog, a subject
     services. By setting criteria to measure the                         guide, a personnel directory, and library hours. The pages
   success of the survey, librarians at UM were                           were poorly coordinated, the navigation not intuitive. A
         able to assess if the library home page                          usability survey was administrated to a group of high school
                                                                          students in the summer of 2003 with the goal of collecting
successfully met the goals and mission statement
                                                                          user feedback on the site. The results indicated that improve-
                                    of the library.                       ment was necessary.
                                                                              In July 2003, a Web Redesign Task Force was formed to
                                                                          overhaul the Web site in order to provide a more dynamic
                                                                          and logical structure that would provide easier navigation
                                                                          and understanding. The task force was under the time
                                                                          constraint of having the new pages revamped by the
                                                                          beginning of the fall semester. The new library Web site
                                                                          was launched in September 2003 without further site
                                                                          analysis. Most users seemed to like the new site much
                                                                          better than the old one. A few public service staff reported
                                                                          that some of the users were confused and were unable to
                                                                          find information. As a fundamental part of the University of
                                                                          Mississippi Libraries’ mission is to ‘‘increase access to
                                                                          information and communication on campus,’’1 and as the
                                                                          Internet has become the predominant medium for the dissem-
                                                                          ination of library resources, it was essential that the adequacy of
                                                                          the new Web site be assessed, specifically the new home page. A
                                                                          Usability Survey Committee was formed to conduct a survey to
                                                                          assess the library home page as a part of the ongoing
                                                                          assessment of the library and its services. Certain criteria were
                                                                          established to determine whether the home page had met our
                    Elizabeth Stephan is Business Reference               goals of increasing access to information. Our benchmark for
                                 Librarian/Assistant Professor,           success was for at least 75% of students to be able to
               The University of Mississippi, MS 38677, USA               successfully complete each task. If the criteria for success
                                    bestephan@olemiss.eduN;               were not met, the results would be used to improve the design
   Daisy T. Cheng is Head of Cataloging/Assistant Professor,              of the page.
               The University of Mississippi, MS 38677, USA                   The committee consisted of six faculty and staff members
                                     bdtcheng@olemiss.eduN;               and began meeting in January 2004. As indicated by Nicole
  Lauren M. Young is Instructor/Outreach Services Librarian,              Campbell, a usability survey is ‘‘a method that tests how a
                                     Rowland Medical Library,             user interacts with a system. The participant is given a list of
               The University of Mississippi Medical Center,              pre-defined tasks to accomplish using the system and asked
                                Jackson, MS 39216-4505, USA               to dthink out loudT about their thoughts, reactions, and
                          blmyoung@rowland.umsmed.eduN.                   feelings.’’2 The committee decided to adopt this method as

The Journal of Academic Librarianship, Volume 32, Number 1, pages 35–51                                                      January 2006 35
A Usability Survey at the University of Mississippi Libraries for the Improvement of the Library Home Page
an assessment tool in order to improve the library home               Much of the literature emphasizes the amount of
page.                                                             preparatory work required in the creation of a usability
   In February 2004, a usability survey was administered to       survey. In the National Institute of Standards and Technology
a group of twelve undergraduate students. Each student was        Virtual Library redesign project in 2001, librarians consulted
asked to complete eight tasks as administrators observed and      with focus groups to get verbal feedback on the site’s
documented their actions. The tasks included both simple          strengths and weaknesses, as well as consulting with a
and more complex searches and included using the library          professional in the creation of their usability test design.8
home page, the library catalog, and databases from outside        Continually recording usage statistics for the site and noting
vendors.                                                          the most used pages will help to identify areas on which to
                                                                  concentrate.9 From this information, librarians can derive
                     LITERATURE REVIEW                            specific, answerable questions for the survey that will
There was not an abundance of literature discussing usability     illuminate gaps in test takers’ knowledge.10 Scripted recruit-
surveys as a form of assessment. We analyzed the existing         ment and test dialogue ensures consistent testing,11 as does
body of literature from two angles. We selected a few             practicing test administration on an individual before
articles that discussed facets of successful library Web sites    performing the actual test.12
and then turned to literature discussing usability surveys,           When dealing with the actual testers, a contrived cross-
their use, design, implementation, and effectiveness. The         section of the user population is encouraged, as are incentives
articles chosen were for the most part limited to academic        and a preconceived strategy for testing locations and struc-
library settings for more direct application to our survey        ture.13 The body of literature indicates that shorter (half-hour)
planned for the University of Mississippi Libraries.              tests with a few specific questions are preferable to longer
    As a foundation, select resources were consulted that         tests.14 According to Jakob Nielsen, libraries engaging in
highlighted elements necessary for library Web sites to be        usability testing will achieve the best results when testing a
deemed successful. In one such article, Sandra Shropshire         small number of participants multiple times in order to reach
outlines her efforts to identify primary concerns that four       the target fifteen test-user goal.15
medium-size academic libraries had regarding their Web                Making sure that survey participants are comfortable is
sites. Shropshire found many concerns to be echoed at all         always a concern when conducting a usability survey. Several
four institutions, including a lack of enough staff to create     usability survey case studies that offer suggestions on this point
and maintain an effective site and to answer patron               are featured in Campbell’s Usability Assessment of Library-
feedback; the need for a library Intranet; the need for a         Related Web Sites. In a survey performed at the University of
designated back-up person in the absence of the systems           Nevada-Las Vegas, student workers were used to administer the
librarian; the navigation of the when/why/how of redesign;        survey as a way to reduce intimidation.16 A survey performed at
and the integration of the OPAC with the rest of the site.        Arizona State University was administered in an empty office
Without these core elements in place, a library will not be       decorated with posters, plants, and items to help ensure privacy
able to meet users’ needs efficiently and effectively.3           and a relaxed atmosphere.17
    Leo Robert Klein reminds academic librarians that Web             The literature indicates that across the board, testers
sites are meant to serve the patrons’ information needs, not      employed the ‘‘think-aloud’’ method and acted as observers,
the librarians’. He contends that library users are busy, non-    recording qualitative measurements such as the test taker’s
selective, and far more interested in finding their resource      level of satisfaction with his answer and quantitative
than in learning how to navigate a convoluted library Web         measurements such as the pathway taken to arrive at the
site. He challenges librarians to embrace elements found in       correct Web page and the amount of time taken to do so.
popular sites such as Google, as these are the sites with         When Susan Feldman of Datasearch, an information system
which their patrons seem to be most comfortable.4 Nicole          Web design company, performs tests on prototype sites for
Campbell looks closely at this issue as well and urges            clients, she videotapes the sessions for later review.18
librarians to make sure that all of the hard work that goes       Fourteen students observed while doing personal research
into library Web site design and maintenance leads to a           at Roger Williams University were tape-recorded while
product that the patron can actually use.5                        performing tasks in the think-aloud, observation/interview
    Before starting a usability survey, one must always ask:      method.19
What exactly is usability? All academic libraries actively            The literature we reviewed makes several points very clear:
seek to have a streamlined, intuitive Web site for their          There are differing opinions on what made a Web site ‘‘usable’’;
patrons. Susan McMullen notes that distance learning              different opinions exist about how a survey should be
programs and users’ remote access preferences heighten the        administered; and the length of studies varied from institution
need for self-explanatory Web sites, as there is sometimes        to institution. All of the above literature helped us develop and
not an opportunity provided for formal instruction. Some          administer a usability survey that fit our needs and goals.
sites still prove to be difficult for new users and experienced
users alike, failing to lead users through the pages to the                         GOALS    AND   OBJECTIVES
information sought, and turning them off from the library         The goal of the usability survey at the University of Mississippi
Web site all together.6 Characteristics such as these, contend    Libraries was to improve access to library resources through
Elaina Norlin and CM Winters, are what make a site un-            library home page by examining the way the undergraduate
usable. In order to ascertain if a library site is usable, the    used the home page. The objectives were as follows:
back-door approach of finding what does not work, achieved
with a usability survey, proves to be far more fruitful than      1. to find out if students could find the basic library sources,
lauding one’s self on what does work.7                               for example a book, a journal article, a newspaper article,

36 The Journal of Academic Librarianship
A Usability Survey at the University of Mississippi Libraries for the Improvement of the Library Home Page
and personal account information, from the library home         fifteen undergraduate students. We wanted to have a cross-
   page; and                                                       section of grade levels and looked at recruiting students
                                                                   through mass e-mails or simply asking for participants from
2. to see if 75% of the participants successfully completed        students in the library. In the end, we did neither because
   various tasks using the most efficient number of clicks.        of time constraints. We had to complete the entire survey in
                                                                   five weeks. This did not allow us the time we would have
                                                                   needed for recruitment. Instead, we decided to recruit the
                          METHODS                                  students from an existing library course. With the cooper-
                        Survey Design                              ation of the instructor, students from the class of
                                                                   Introduction to Library Resources and Electronic Research
   The survey administered to a group of high school students      (EDLS 101) were selected for the survey. At this point in
the previous summer was adopted as a starting point. Two of
                                                                   the semester they had not covered the use of the library
the tasks – locating a copy of Rolling Stone magazine and
                                                                   catalog or other library resources. There were fifteen
locating an article about Internet retailing in EbscoHost – were   students enrolled in the class, representing every grade
taken directly from the previous survey. The remaining content
                                                                   level. Only twelve students attended class the day of the
was modified based on input from reference librarians who
                                                                   survey. We discussed recruiting three more students to raise
deal directly with students and who had observed common
                                                                   our number to fifteen. After looking at our results, we
problems with the home page.
                                                                   decided that we had sufficient information to assess the
   It has been noted that for a one-hour session, the survey
                                                                   library’s home page.
should be limited to seven to ten tasks.20 Because we were
                                                                       Since we used students from a scheduled class, we
working within a 50-minute time frame and knew that we
                                                                   conducted our entire survey in a single fifty-minute period
needed to allow time for students and administrators to get
                                                                   instead of administering the survey over several weeks.
settled, we chose to limit our survey to eight questions.
                                                                   While this may not have been the ideal situation, it worked
   Before it was administered to students, the survey was          with our schedule. We were able to separate the students so
tested on a non-library user and adjustments were made as
                                                                   that they could not hear or observe others taking the survey.
needed. We found that some of our language was too heavy
                                                                   Students did not know before attending class that day that
on ‘‘library ease,’’ meaning that only librarians would fully      they would be taking the survey, and each student was
understand the directions. When we asked our non-library
                                                                   given four points extra credit for participating. The
user to locate a copy of The Catcher in the Rye, we realized
                                                                   University of Mississippi requires any researchers using
we had not noted in the task that it was a book. As we
                                                                   human participants to file for permission through the
specifically wanted the users to find a book, that detail was
                                                                   Institutional Review Board (IRB). The IRB required that
added. We also discussed whether we were telling students
                                                                   we have each student sign a release form and be given the
how to complete the task by using language used on the
                                                                   choice to not take the survey. The instructor of the class
existing home page. If we asked students to find a subject
                                                                   was a member of the committee and did not take part in the
guide, was that going to skew the results since the link was
                                                                   administration of the survey as she felt the students might
clearly labeled ‘‘Subject Guides?’’
                                                                   feel uncomfortable or that they were being graded if she
   We decided that in some cases we wanted to use terms used       was involved.
on the existing home page because we wanted to know if
                                                                       Each student was assigned to an administrator, and each
students understood what that term meant. On the other side of
                                                                   administrator/student pair was assigned to a computer.
this issue, we asked students to find an article in Ebsco. The
                                                                   Students were given a number after they signed the release.
library’s Ebsco databases are labeled by their names, ‘‘Aca-
                                                                   When they walked out of the library classroom they matched
demic Search Premier (Ebsco),’’ ‘‘Business Source Premier,’’
                                                                   their number with the number each administrator was given.
etc. But both students and faculty refer to ‘‘Academic Search’’
                                                                   Administrators knew what their number was and to which
simply as ‘‘Ebsco.’’ We wanted to know if students understood
                                                                   computer they were assigned, which helped to streamline the
how to get to the databases page and if they understood that
                                                                   process. We used office computers in the library’s Technical
when they were looking for ‘‘Ebsco,’’ they were really looking
                                                                   Services Department as well as one additional office
for ‘‘Academic Search Premier.’’                                   computer and two computers in our computer lab. Each
                        Administrators                             student was read a statement explaining the purpose of the
   Fifteen library faculty and staff members were recruited to     survey and was reassured that they were not being tested
administer the survey. Before the survey was conducted,            (see Appendix B).
administrators were required to attend a training session. Not         It is not easy to make a student feel comfortable in a
everyone was familiar with the purpose of a usability survey,      cubicle of a busy office, but we did what we could. Our
much less how to administer a survey. To achieve uniformity of     administrators allowed us to use their cubicles and we were
survey results, administrators were trained on how to give the     able to find some empty areas in the same office. We placed
survey, what to say, and what not to say (see Appendix A). Also    students in cubicles that would not be adjacent to one
discussed were each of the questions and what type of com-         another so they could not hear the other administrators or
ments and behaviors they were to look for and what to record.      students. We asked that personal items (pictures, notes, etc.)
                                                                   be removed from computer monitors and asked that the
                            Students                               computer screens be set to the University of Mississippi
   We debated how to recruit students. Going by Jakob              home page, the starting point of our survey.
Nielsen’s rule that with fifteen students you will learn what          We assigned three students to two other locations. Two
you need, we decided to limit our survey participants to           students were assigned to our bibliographic instruction room.

                                                                                                                  January 2006 37
A Usability Survey at the University of Mississippi Libraries for the Improvement of the Library Home Page
Figure 1
                                           Web Experiences

                                                 Figure 2
     The University of Mississippi Home Page (www.olemiss.edu). The Arrows Point to the Links to the
Library’s Home Page. The University Libraries Link (1) Was the One Most Students Used; The Libraries and
  Museums Link (2) was Used Less, Although this Link Took the User Directly to the Home Page While the
  University Libraries Link Went to an Intermediary Page. The UM Home Page has Since Been Redesigned

38 The Journal of Academic Librarianship
A Usability Survey at the University of Mississippi Libraries for the Improvement of the Library Home Page
We placed them far apart and chose administrators with quieter          had completed the task; average when a student thought
voices in order not to disrupt each other. One student was              he/she had completed the task but was not positive; and
placed in a private office with one administrator. This student         low when a student did not think he/she had completed
took the longest to complete the survey. Whether this was               the task.
related to his location or not was inconclusive.
                                                                                         Qualitative Metrics
                         Measurement
                                                                        Qualitative measures are based on the administrator’s
   A student’s response to each question was scored by a             observation and both the administrator’s and student’s com-
number of quantitative and qualitative measures that included        ments. These included the following:
the number of clicks, the time it took to complete the task, as
well as the administrator’s observation of the student’s action      o Signs of indecision: the administrator wrote down any
and his/her comments.                                                  indecision or hesitation observed in student.
                    Quantitative Metrics                             o Indications of frustration: the administrator recorded a
o Number of clicks to reach destination: a click was counted if        student’s frustration when he/she started to grumble or
  it took a student to a different page. If a student went back to     mumble to himself or herself about the difficulty of the
  the previous page, that was counted as a click. If a student         task.
  clicked a checkbox on a page (i.e., ‘‘full-text’’ box in
  EbscoHost), that did not count as a click.                         o User comments: the student’s oral observations, such as
                                                                       ‘‘the catalog page looks a lot like the home page,’’ or
o Time required to complete the task: the amount of time it            comments about what he/she would like to see, were
  took to complete each task.                                          recorded.

o Completed the task (yes/no): if a student said that he/she         o Observer comments: the administrator’s own observations
  could not reach the page, the task was not complete, even if         and comments.
  they had indeed reached the page and did not realize that                               SURVEY RESULT
  they had done so.
                                                                                Demographic Questions and Results
o Satisfaction level (high/average/low): administrators were           Before the survey began administrators asked students a
  asked to rate high when a student was certain that he/she          number of questions ranging from their class and major to how

                                                 Figure 3
 Students Were Confused By the Multiple Listings for The Catcher in the Rye, But Seeing the Author’s Name
                           Listed in the Second Record Helped Some Students

                                                                                                                   January 2006 39
A Usability Survey at the University of Mississippi Libraries for the Improvement of the Library Home Page
Figure 4
      The Students Who Found a Record for Rolling Stone in Journal Finder (TD Net) Misinterpreted the
                    Record. They Thought the Library Only had it in Electronic Format

                                                  Figure 5
     When Looking for a Copy of Rolling Stone, Most Students Did a Title Search in the Catalog. Many
   Were Confused by the Results List. Not Seeing a Listing for Rolling Stone Magazine Lead Some to Think
                             we did not Have a Current Copy in the Library

40 The Journal of Academic Librarianship
A Usability Survey at the University of Mississippi Libraries for the Improvement of the Library Home Page
comfortable they were with using the library Web page. This       results and what students felt their online research com-
information was used to help us understand our survey             petencies were.
participants.                                                        The average ratings that upper and lower classmen gave for
                                                                  using the library Web page and ‘‘other’’ Web pages (Google,
                    What Class are You?
                                                                  Yahoo, etc.) were similar. The lowest score given for searching
   Our goal was to have an even representation of under-          the library Web page was 1; the highest, 8. The lowest for
graduate class levels. More seniors participated than any other   ‘‘other’’ was 2, the highest, 10 (Fig. 1).
class level, but as a whole there was an even representation of
undergraduates.                                                                           Survey Findings
                                                                     Find the University Library Home Page
                   What is Your Major?
                                                                     While we have little influence over the design of the
   As with the class level, we had hoped to get an even           University of Mississippi’s home page (www.olemiss.edu)
representation of majors. In this survey business majors made     (Fig. 2), we wanted to know if students knew how to get to
up half of our participants. Approximately 25% of UM students     the library home page; if a student cannot even locate the library
are in the School of Business Administration, so our sample did   home page it is not possible for us to provide any online
not accurately reflect the make-up of UM majors.                  services.
                                                                     Twelve out of twelve (100%) students completed this task;
On a Scale From 1 to 10 (10 Being the Highest), How Much
                                                                  of these, ten were noted as being highly satisfied and two were
Experience Do You Have Using the Library Web Page? and            at average satisfaction. No students were noted as having low
On a Scale From 1 to 10, How Much Experience Do You Have
                                                                  satisfaction because all completed the task. The average
 Using a Web Site, Such as Google, to Find Information?
                                                                  number of clicks used to complete the task was 1. It took the
   The purpose of these two questions was to get a feel for       survey creators one click. The least number of clicks needed to
how the students ranked themselves in using both the              complete the task was one; the most was eight.
library Web resources and Internet search engines such as            Despite all of the students completing the task, not all of
Google. This provided context when looking at survey              them found the library page easily. There were two links to two

                                               Figure 6
  The Two Links Where Students Could Find the Library Hours (1 and 2). Before Finding the Correct Link,
                         Some Students Looked Under Library Quick Links (3)

                                                                                                                    January 2006 41
A Usability Survey at the University of Mississippi Libraries for the Improvement of the Library Home Page
Figure 7
           Nine Out of Twelve (75%) Students Were Able to Successfully Find the Course Reserves Page

different pages on the UM’s home page: the ‘‘University           ately clicked on the ‘‘University Libraries’’ and commented
Libraries’’ link went directly to the library site and the        that she did that because ‘‘she knew it was the library Web
‘‘Libraries & Museums’’ link went to an intermediary page         page.’’
with separate links to both the library page and the Museum          Eight out of twelve went through the intermediary page.
page. Three out of twelve students reached the library page       One student did not realize the intermediary page was not the
directly through the ‘‘University Libraries’’ link. One student   library page. The administrator said, ‘‘Please go to the library
began to do a keyword search on UM’s home page before he          home page’’ when it became apparent the student did not
saw the ‘‘University Libraries’’ link. Another student immedi-    realize that he was not at the library home page.

                                                   Figure 8
      Students had Problems with the Professor’s Name. Several Did Not Read the Instructions and Typed
                                     ‘‘E. Smith’’ Instead of ‘‘Smith, E.’’

42 The Journal of Academic Librarianship
A Usability Survey at the University of Mississippi Libraries for the Improvement of the Library Home Page
One student clicked on the ‘‘University and Libraries’’       took the survey creators four clicks to complete the same
page and got stuck. He clicked back forth between the            task.
UM’s home page and the ‘‘Libraries & Museums’’ page                 All of the students found the library catalog from the library
several times before turning to Google. He found the             home page. Two students hesitated at the home page before
library site by doing a Google search for the ‘‘Ole Miss         clicking on the ‘‘Catalog’’ link. The two unable to complete the
Library.’’                                                       task got as far as the title list in the ‘‘Catalog’’ (Fig. 3). When
   Because of the two different links and paths to the library   they got to the title list, they did not know which record was for
home page, we were not surprised by the results. After the       the book.
survey was completed and the results tabulated, we did ask the      The Catalog proved confusing even to those who success-
university webmaster to make both library links go directly to   fully completed the task. Like those who did not complete the
the library home page.                                           task, those who did were confused by the title page but
                                                                 located the correct record. One student initially searched by
   Does the Library Have a Copy of the Book, The Catcher in
                                                                 author but quickly realized his mistake when he did not get
   the Rye?                                                      any results. He was able to correct his initial search and
   Locating items in the library catalog was key to the          successfully locate the record. The ‘‘Library Search Engine’’
assessment of our services. We wanted to know if students        (MetaFind page) was used by one student, but when it locked
knew where to go if they needed to look up a book, and if they   up he went to the ‘‘Catalog’’ and conducted a successful title
got there, if they could find one.                               search.
   Ten out of twelve (83%) were able to successfully
complete this task, two out of twelve (17%) were not. Nine          Does the Library Have a Current Copy of Rolling Stone
were highly satisfied with their results, two had an                Magazine?
average satisfaction (meaning they thought they found               This proved to be the most difficult task. Three out of
it but were not sure), and one was rated at low satis-           twelve (25%) students were able to successfully complete
faction. The average number of clicks used in this task          this task; nine (75%) were unsuccessful. Only one had a
was 3.6 including those who did not complete the task. It        high satisfaction rate with two average, and nine marked as

                                                 Figure 9
         Twelve Out of Twelve (100%) Students Were Able to Locate a Subject Guide by Using One of
                             the Two Subject Links on The Library Home Page

                                                                                                                    January 2006 43
A Usability Survey at the University of Mississippi Libraries for the Improvement of the Library Home Page
low. It took the students an average of five clicks, that           stopped. At this point he commented, ‘‘either I went to the
includes the number of students who did not complete the            wrong place or they do not have it.’’
task. It took the survey creators two clicks through ‘‘Journal          One student skipped the ‘‘Catalog’’ and went directly to
Finder’’ (TDNet page) and four through the Library                  ‘‘Journal Finder,’’ where she was able to find a record for
‘‘Catalog.’’                                                        Rolling Stone. When she saw the TDNet record she said, ‘‘I
    Students went different directions with this task. For their    guess it’s like EBSCO, not here in the library’’ (Fig. 4) (TDNet
initial search, six went to the ‘‘Catalog,’’ one went directly to   records indicate whether we have a periodical in print, online, or
the ‘‘Library Search Engine,’’ one went to ‘‘Journal Finder,’’      both). She either failed to see that it was available in print or did
and two went to the ‘‘Articles and Journals’’ page. We do not       not understand how to read the record.
know the path taken by two students.                                    The problems encountered with the ‘‘Catalog’’ were
    Nine out of twelve students were not able to complete this      similar to their problems with The Catcher in the Rye task.
task, six went directly to the ‘‘Catalog,’’ and two went other      They were confused by the title list (Fig. 5). Many students
directions. Of the six who searched in the Catalog, two were        did a keyword search instead of a journal title search. One
successful in locating a current copy of Rolling Stone; four        student did a journal title search and was still unsuccessful.
were not. Of the four unable to locate the magazine, two did        He scrolled up and down the results page while mumbling
not try another route.                                              ‘‘Rolling Stone magazine, Rolling Stone magazine. . ..’’ The
    The two who did try a second route went to the ‘‘Articles &     first item listed was the record for Rolling Stone magazine;
Journals’’ page where one tried ‘‘Journal Finder’’ but was          we can speculate that because it did not read the Rolling
unsuccessful, and the other student tried the ‘‘Library Search      Stone magazine,’’ he did not click on it. This reaction to the
Engine’’ without any success. He returned to the ‘‘Catalog’’        title list was common.
and did a title search but only found Rolling Stone books. At           Three out of twelve students were successful in completing
this point, he stopped.                                             this task. Three went to the ‘‘Catalog’’ immediately; one
    One student went directly to the ‘‘Article Quick Search’’       successfully found the record. The second student did a keyword
(MetaFind) on the ‘‘Articles & Journals’’ page. Unable to           search and could not locate the record. The student returned to
locate the article, he went to the ‘‘Databases’’ page where he      the library home page and went to the ‘‘Articles and Journals’’

                                               Figure 10
 Eleven Out of Twelve (92%) Participants Were Able to Locate Their Library Account Information From the
Home Page. One Student Returned to the University of Mississippi Home Page and Looked Under Registration

44 The Journal of Academic Librarianship
page without any success. After returning to the library home     had an average satisfaction, and no students were marked at
page a third time the student clicked on ‘‘Journal Finder’’ and   low satisfaction. There were two links to the library
found the record. Another student tried the ‘‘Library Search      ‘‘Hours’’ page on the library home page: one in the main
Engine,’’ and when that did not work he found the record in the   navigation bar at the top of the page and one in the blue
‘‘Catalog.’’                                                      side box.
   The different and numerous avenues students took to               One problem we had with this task was that seven of the
complete this task demonstrated their determination to find       twelve administrators did not note which ‘‘Hours’’ link the
what they needed. They knew what the ‘‘Catalog’’ was and          students chose. We do know that three of the twelve chose the
how to find it but did not always know how to use it. It was      ‘‘Hours’’ link in the blue side box while only two chose the
also evidenced that the ‘‘Articles and Journals’’ link was        link in the top navigation bar. Three students showed some
confusing. Students immediately went there but were not sure      hesitation at the beginning of the task. Of those three, two
what to do when they got there. Or, they tried ‘‘Library Search   initially went to the ‘‘Library Quick Links’’ drop-down box
Engine’’ but it would not work (we did find out later that this   before seeing one of the links on the library home page (Fig. 6).
might have been due to some technical issues we were not          When the student noticed the ‘‘Hours’’ link he commented that
aware of at the time of the survey). We initially assumed         he ‘‘should’ve seen that [link].’’ One student, when asked
students would use the ‘‘Catalog’’ first and then ‘‘Journal       ‘‘What are the hours of the Library?,’’ turned around and
Finder.’’ While they did use the ‘‘Catalog,’’ few of them         responded, ‘‘Seven to midnight.’’ Then he realized he needed
noticed or tried ‘‘Journal Finder.’’                              to locate the hours on the home page. He immediately went to
                                                                  the link in the navigation bar.
   What are the Hours of the Library?                                One general observation was that the most current hours
   This was one of the easiest tasks for our students with        were found at the bottom of the ‘‘Hours’’ page, such as special
twelve out of twelve (100%) students completing the task.         holiday hours or normal hours resumed. A reverse chrono-
Eleven students were noted as having high satisfaction, one       logical order that places the most recent update at the top of the

                                                  Figure 11
   When Looking for a Full-text Article, Students Were Not Sure Where to Go First, Some Went Directly to
    Databases (1), Others Went to Articles and Journals (2). Those Unsure of Where to Go Clicked on the
   Library Quick Links Box (3). Some Students Used the Library Search Engine (4) But it Would Lock Up
                                      After They Entered Their Search

                                                                                                                    January 2006 45
page was determined to be clearer, and this was fixed shortly           same six, three commented that they did not know course
after the survey was completed.                                         reserves could be looked up through the library home page.
                                                                           Six out of the twelve were unable to complete the task.
   Does Prof. E. Smith Have Anything on Reserve?                        Three of these six were the students mentioned above who
   This question was designed to see if students knew how to            located the ‘‘Course Reserves’’ page but searched under ‘‘E.
locate course reserves from the library home page through the           Smith.’’ The remaining three were unable to locate the ‘‘Course
‘‘Course Reserves’’ link. Most faculty tell their students they         Reserves’’ page. Two looked under ‘‘Contact a person’’ and
have put an item on ‘‘reserve.’’ We wanted to know if students          one said he did not know how to do that and did not make an
knew to go to the ‘‘Course Reserves’’ link on the library home          attempt.
page when looking for items on reserve.
   Six out of twelve (50%) students were able to complete the              Suppose You are Taking a Class on a Subject Unfamiliar to
task; six were not. Three were highly satisfied with their                 You; Find a Subject Guide Relating to That Topic
results, four were average, while five had low satisfaction with           Twelve out of twelve (100%) of the survey participants were
their results. This task took students an average of 2.3 clicks; it     able to complete this task. There was very little if any hesitation
took the creators two clicks.                                           shown by students. It took students an average of 1.6 clicks to
   The results are misleading. Nine out of twelve were able to get      complete the task; it took the creators one. Several students
to the ‘‘Course Reserves’’ search page from the library home            commented that it was ‘‘easy’’ or that this task was an ‘‘easy
page (Fig. 7). When searching for reserves by a professor’s name,       one.’’ Each student was able to locate the ‘‘Subject Guides’’
the user is instructed to search by professor’s last name. Of the       page and then chose a subject and clicked on the link to a
nine students who reached the ‘‘Course Reserves’’ page, three           subject guide.
entered ‘‘E. Smith’’ instead of ‘‘Smith’’ or ‘‘Smith, E.’’ Of the six      One student showed some hesitation after he was read the
students who did complete the task, two did an initial search for       task. He looked over the home page and said, ‘‘I’m going to
‘‘E. Smith’’ but realized their mistake, returned to the ‘‘Course       say dSubject Guides.T’’ Another student had no problem
Reserves’’ page, and searched under ‘‘Smith’’ (Fig. 8). Of those        reaching the ‘‘Subject Guides’’ page. He chose the chemistry

                                                   Figure 12
          Students Were Confused By the List of Databases. Many Tried to Click on the Basic Search or
             Advanced Search Tabs at the Top of the Page But They Were Grayed Out. As a Result
                     of the Usability Survey, This Page is Being Phased Out of Use at UM

46 The Journal of Academic Librarianship
subject guide. When he reached the page for chemistry subject                as average. Three did not have a satisfaction level marked for
guide he hesitated. Judging from his body language, he seemed                this task, but those three did complete the task. It took both the
unsure. He scrolled up and down the page, noting the list of                 students and creators an average of one click to complete this
databases and Internet sites. When he reached the bottom and                 task.
scrolled back to the top, he seemed satisfied he had completed                   Eleven out of twelve of survey participants were able to
the task.                                                                    locate the link and the page where their name and barcode are
   The library home page had two ‘‘Subject Guides’’ links: one               entered in order to retrieve their library account information
on the main navigation bar and one in the main set of links in               (we did not ask them to enter their personal information).
the center part of the page (Fig. 9). None of the administrators             Students commented that it was an ‘‘easy one’’ and said they
noted the path the students took. But by looking at the average              knew to go to ‘‘My Library Account’’ (Fig. 10). Of the students
number (1.6 clicks) of clicks it took students to find the subject           able to complete the task, only one showed any sign of
guide, it was evident that they went through either of the two               hesitation. This hesitation was caused when the student
links on the library home page.                                              confused the catalog page with the home page. When he
   This was one task where we questioned our terminology. By                 returned to the home page, he was able to easily locate the ‘‘My
using the term ‘‘subject guide’’ in the task, we were leading                Library Account’’ link.
them directly to the ‘‘Subject Guides’’ link on the home page.                   One student was unable to locate his library account. When
This question was loosely based on a question used in the                    the task was presented to him, he commented that he ‘‘did not
usability survey conducted in 2003 and the same terminology                  know’’ and that he had ‘‘no clue’’ how to find the information.
was used. After some debate, the decision was made to use the                He returned to the University of Mississippi home page and
term ‘‘subject guide’’ considering that it was not only the term             looked under the Registration link.
used on the home page (which was out target of assessment)                       This is another case where we may have found out more if
but also the term most likely to be used by both teaching and                we had worded the task differently. We know that a large
library faculty.                                                             majority of our students know how to find their library account.
                                                                             Does the same majority know that this is where they go to see
   How Do You Look at Your Library Account?                                  if they have a book overdue or to renew a book? We can only
   A patron’s library account allows them to see what items                  speculate that because they knew how to locate their library
they have checked out, renew books, etc. Being able to find                  account that they know what it is for.
this information will allow users to access their library
information easier and faster. While it is not a new feature                     Locate a Full-text Article in Ebscohost About Internet
the University of Mississippi Libraries, we did not know if                      Retailing
students knew it was available or if they knew how to find their                This proved to be a difficult task. Five out of eleven students
information.                                                                 (45%) completed the task; six out of eleven (55%) did not. One
   Eleven out of twelve (92%) were able to complete this task;               student was classified as a ‘‘yes and no’’ because Ebsco timed
one out of twelve (8%) were not. Eight were marked as being                  out before he could get to the search page; therefore, the
highly satisfied, one had low satisfaction, and none were rated              percentages for this task are based on eleven surveys instead of

                                                            Table 1
                          A Task was Considered Success if 75% of the Participants Completed the Task
Number                                               Task                                    Complete                           Incomplete
Q1                                Find the University Library Web page.                  2 out of 12 (100%)                         0%
Q2                                Does the library have a copy                          10 out of 12 (83%)                   2 out of 12 (17%)
                                  of the book The Catcher in the
                                  Rye by J.D. Salinger?
Q3                                Does the library have a current                        3 out of 12 (25%)                   9 out of 12 (75%)
                                  copy of Rolling Stone magazine?
Q4                                What are the hours of the library?                    12 out of 12 (100%)                         0%
Q5                                Does Professor E. Smith have any                       6 out of 12 (50%)                   6 out of 12 (50%)
                                  course reserves?
Q6                                Suppose you are taking a class on                     12 out of 12 (100%)                         0%
                                  a subject unfamiliar to you, find a
                                  subject guide relating to that topic.
Q7                                How do you look at your library account?              11 out of 12 (92%)                   1 out of 12 (8%)
Q8                                Locate a full-text article in EbscoHost                5 out of 11 (45%)                   6 out of 11 (55%)
                                  about Internet retailing.
This table shows which tasks met our criteria.

                                                                                                                               January 2006 47
twelve. The satisfaction levels, however, did not match up with         One student never even tried. When asked to look for an
the level of completion. Three out of eleven were noted to be       article in EbscoHost he asked, ‘‘What’s that?’’
highly satisfied with the results, two out of eleven were               The EbscoHost Web page also proved confusing to those
average, and six out of eleven were low. It took students an        students who were able to complete the task. One student in
average of 4.1 clicks to complete the task; it took the creators    particular clicked all over the page including the grayed-out
of the survey three (Fig. 11).                                      Basic Search tab. She continued to click back and forth
    Of the students who did not complete this task, many            between the alphabetical databases page and EbscoHost
experienced similar problems. Two students were confused by         Web. After reading the definitions of the Ebsco databases,
the EbscoHost Web page they had to pass through to get to           she chose the Newspaper Source where she found an article.
Academic Search Premier (or any other Ebsco database). The              The ‘‘Library Search Engine’’ was another popular choice.
list of databases was in no particular order and was confusing      Some students were successful and some were not. One student
to students. Students that got stuck here tried to click on the     went to the ‘‘Library Search Engine’’ and searched for
Basic Search tab but it was grayed out. After scrolling up and      ‘‘EbscoHost.’’ After not getting any results, he searched for
down the page, clicking on grayed-out tabs, and reading             ‘‘Internet retailing.’’ He clicked on the fist result with ‘‘Ebsco’’
definition tabs, one student finally commented that he had ‘‘no     in the URL and was taken to an article in EbscoHost.
idea’’ how to locate an article. The other student stuck on this        When designing this question we chose to ask students to
page said he was ‘‘looking for some place to type dInternet         search in ‘‘EbscoHost’’ because most of their professors tell
retailingT’’ but could not find one (Fig. 12).                      them to find articles in ‘‘Ebsco’’ as opposed to Academic
    Another student bypassed the library home page and went         Search Elite, Newspaper Source, etc. Students come to the
directly to www.ebscohost.com. Because of the library’s             reference desk and ask for articles from Ebsco. In addition
subscription, the site recognized him as a subscriber; he was       to finding out if the library home page gave them enough
able to search using ‘‘Internet’’ and ‘‘retailing’’ as keywords.    direction to get to a database, we wanted to know if they
He got sixty-six results but did not know how to access the full-   associated any specific database with ‘‘EbscoHost.’’ We
text articles.                                                      found that most knew what EbscoHost was because several

                                                         Figure 13
                                             The Redesigned Library Home Page

48 The Journal of Academic Librarianship
told us they had used it before. It was the library site in              APPENDIX A. INSTRUCTIONS           FOR   ADMINISTRATOR
combination with the EbscoHost Web page that caused
                                                                                    Instructions for Administrator
confusion. The library is in the process of phasing out the
EbscoHost Web page because of what we learned from the                   Be at Lyceum Circ Desk at 2:05
usability survey.                                                        Bring
                                                                         A watch
                         CONCLUSION                                      Form
Because the usability survey was administered for our biennial           Number
assessment, we assigned some criteria in order to measure our            Pen/pencil
success. For assessment purposes, the library home page would            Book, folder, clipboard
meet our goals of making library resources accessible if at least
75% or nine out of twelve participants would be able to                 Computer
successfully complete each task. The rate of completion ran             It should be set up and ready to go when you get there. A
from three out of twelve (25%) to all (100%) students, with           member of the committee will come around and check
five of the eight tasks completed by at least ten out of twelve       computers before you arrive.
(83%) participants. Table 1 shows which tasks met our
requirements. Going by our criteria, we found users were able           Survey Begins
to successfully complete the directional tasks (Q1, Q4, Q6, and         Read Script to your student.
Q7) with at least eleven out of twelve students (92%) being             Time started: Time after you read script and before first
able to complete these tasks. The more complex tasks (Q2, Q3,         question asked
Q5, and Q8) proved to be more difficult. When students did not          Time ended (on last page): Time last task completed
know where else to go, they were drawn to the ‘‘Library Quick
Links’’ drop-down box and to the blue side box. We already               While students are completing each task they will ask a lot
knew students liked the blue box but we had always dismissed          of questions: ‘‘Is this right?’’ ‘‘Is this what you want?’’ ‘‘Did I
the drop-down box until we saw how much students used it. In          spell this correctly?’’ You can’t answer their questions. Some
general, there was a lot of confusion between the library home        possible responses to these questions: ‘‘Do you think it is
page and main ‘‘Catalog’’ page. On several occasions, students        correct?’’ or read the question again. They may also say ‘‘Is
would stop at the ‘‘Catalog’’ page not knowing they were not at       this what you wanted’’ when they get to the desired page. If
the library home page.                                                possible restate the question. Hopefully, they will realize they
   Some issues raised by the usability survey have been resolved      are at the correct page. If after your prodding they do not
through Web design. We redesigned the library home page giving        realize they have completed the task tell them to move on to the
more prominence to the ‘‘Databases’’ and ‘‘Subject Guides.’’ We       next question.
reordered the items in the blue sidebar box, and we plan on
                                                                                            Quantitative Metrics
redesigning the page header to make the ‘‘Home’’ and ‘‘Inter-
Library Loan’’ links more visible. We eliminated the ‘‘Articles &         Number of clicks to reach destination:
Journals’’ page and have begun working on a series of pages               The number of clicks listed after each question is how it was
devoted to how to use the ‘‘Catalog’’ and ‘‘Databases’’ pages (Fig.   reached by one librarian. One click is considered a click that
13). At our request, the secondary ‘‘Library & Museums’’ page has     takes them to a different page. If they go back to the previous
been eliminated. However, the UM home page has since been             page, that is a click. If the click a check-box on a page (i.e.,
redesigned and an intermediary page between www.olemiss.edu           ‘‘full-text’’ box in Ebsco), that does not count as a click.
and the library home page has been added.
   Some issues will have to be addressed by instruction—both             Complete task: yes ____ no ____
in the classroom and at the reference desk. The survey showed            If they get to the desired page they completed the task. If
us where we need to concentrate our instruction and in which          they tell you they can’t get to the page then they did not
areas students are having problems. ‘‘Journal Finder’’ is useful      complete the task. Let them decide. If it is obvious they ca not
but many students do not understand what it is for or how to          figure it out, say ‘‘let’s move on to the next question’’
read the records. In conjunction with other committees, it was
decided to purchase the MARC records for journals we have in             Satisfaction level: High ___ Average ____ Low ____
electronic format. When students search for a journal in the             Was the student satisfied with the result.
catalog, they will get a record for both the print and electronic        High: They completed the task and they know they
versions of a journal. ‘‘Journal Finder’’ will be phased out.         completed the task.
Some areas of the ‘‘Catalog’’ are causing confusion. We have             Average: They think they completed the task but they are
very little control over the design of the ‘‘Catalog,’’ but we        not positive.
know what areas to concentrate on during instruction sessions.           Low: They did not complete the task or the completed the
   A usability survey is a lengthy and time-consuming project.        task and did not know it.
Like some of the studies done by other universities, ours was
rushed due to the deadline. But the information we learned from                             Qualitative Metrics
observing our users using the library resources was invaluable.          Signs of indecision:
Looking back, there are things our committee knows now that              Use this to write down any indecision or hesitation you
we should have done differently, but all in all we feel it was a      might see. If they point towards ‘‘Journal Finder’’ while
successful and worthwhile endeavor. Plans are in the works to         looking for Rolling Stone and wonder if they should go there,
conduct a follow-up survey.                                           make note of that here.

                                                                                                                         January 2006 49
Indications of frustration:                                         User comments:
   When the student starts grumbling or mumbling to him- or            Observer comments:
herself about how difficult something is, indicate that here.          Please Return to the Library Home page
What did they say? When did they get frustrated?                       Please wait one moment
   User comments:
   We are looking for student observations: ‘‘The catalog page      2. Does the library have a copy of the book, The Catcher in the
looks a lot like the home page.’’ Or if the student makes a            Rye by J.D. Salinger? (Four clicks from main library page)
comment about what they would like to see, put it here.
                                                                    3. Does the library have a current copy of Rolling Stone
   Observer comments:                                                  magazine? (Two clicks through Journal Finder; four clicks
   Use this area to write any comments you have while                  through catalog)
observing the student (i.e., for task #3 you might write: ‘‘Was
confused by word dmagazineT’’).                                     4. What are the hours of the library? (One click from main
   While these are your notes please understand others will be         library page)
reading them. We do not expect full sentences but they do have
to make sense.                                                      5. Does Professor E. Smith have any course reserves? (Two
                                                                       clicks from main library page)
                 APPENDIX B. SURVEY FORM
                                                                    6. Suppose you are taking a class on a subject unfamiliar to
TIME STARTED: _____________________________                            you (Chemistry, Telecommunications, Civil Engineering,
Administrator’s Name_____________________________                      Women’s Studies, etc.), find a subject guide relating to that
                                                                       topic. (One click from main library page)
   Script
   Hello. My name is _____. I will be working with you in           7. How do you look at your library account? (One click to
today. Thank you for participating.
                                                                       My library page) [They do not need to enter their personal
                                                                       data. We just want to know if they know to go to ‘‘My
    The library has used its Web site to provide information for
                                                                       Library.’’]
several years and we want to know if our Web site is useful to
undergraduate students. In order to find this out, we want to
                                                                    8. Locate a full-text article in EbscoHost about Internet
watch how you and other undergraduates use the Web site. I will
                                                                       retailing. (Six clicks: four to get to Ebsco, two in Ebsco.)
be asking you to do a set number of tasks. I want to emphasize
that you are not being tested, and you are not being graded.
    While you are doing these tasks please tell me why you are         Thank you for your time.
doing what you are doing. If you click on a link, tell me why. It      TIME ENDED:_______________________
is okay if you cannot complete a task or find some information;
there is no wrong answer. If you cannot complete a task, tell me                       NOTES AND REFERENCES
why and we will move on to the next item.
    I will be taking notes on what you say and how you               1. Assessment Record for Department/Unit of University Libraries,
complete the tasks. Again, I am not testing you. Try your best          2001–2003, Form B: Expanded Statement of Institutional Purpose
                                                                        Linkage, submitted 10 October 2003.
to ignore me and everyone else in the room. While I am asking        2. Nicole Campbell, ‘‘Usability Methods,’’ in Usability Assessment of
you to dthink out loudT as you are completing these task I              Library-Related Web Sites: Methods and Case Studies, edited by
cannot answer any questions you may have. Do you have any               Nicole Campbell (Chicago: Library and Information Technology
questions before we begin?                                              Association, 2001), p. 2.
    Before we get started I have a few general questions to ask:     3. Sandra Shropshire, ‘‘Beyond the Design and Evaluation of Library
    What is your class rank?                                            Web Sites: An Analysis and Four Case Studies,’’ Journal of
    Class: Freshman Sophomore Junior Senior Other                       Academic Librarianship 29 (March 2003): 95 – 101.
    What is your major? ________________________                     4. Leo Robert Klein, ‘‘The Expert User Is Dead,’’ Library Journal
    On a scale from 1 to 10 with ‘‘1’’ being no experience to           128 (Fall 2003): 36.
‘‘10’’ being lots of experience, how much experience do you          5. Nicole Campbell, ‘‘Introduction,’’ in Usability Assessment of
                                                                        Library-Related Web Sites: Methods and Case Studies, edited by
have using the Library Web page?                                        Nicole Campbell (Chicago: Library and Information Technology
    On a sale from 1 to 10, how much experience do you have             Association, 2001), p. v.
using a Web site such as Google to find information?                 6. Susan McMullen, ‘‘Usability Testing in a Library Web Site
                                                                        Redesign Project,’’ Reference Services Review 29 (February
1. Find the University Library Web page. (One click from UM             2001): 7 – 22.
   Web page)                                                         7. Elaina Norlin & C.M. Winters, Usability Testing for Library Web
   Quantitative metrics                                                 Sites: A Hands-On Guide (Chicago: American Library Associa-
                                                                        tion, 2002).
   Number of clicks to reach destination:                            8. Susan Makar, ‘‘Earning the Stamp of Approval,’’ Computers in
   Complete task: yes ____ no ____                                      Libraries 23 (January 2003): 16 – 21.
   Satisfaction level: High ____ Average ____ Low ____               9. Marshall Breeding, ‘‘Library Web Site Analysis,’’ Library
   Qualitative metrics                                                  Technology Reports 38 (May/June 2002): 22 – 35.
   Signs of indecision:                                             10. David King, ‘‘The Mom-and-Pop Shop Approach to Usability
   Indications of frustration:                                          Studies,’’ Computers in Libraries 23 (January 2003): 12 – 15.

50 The Journal of Academic Librarianship
11. Barbara J. Cockrell & Elaine Anderson Jayne, ‘‘How Do I Find an         of Nevada, Las Vegas Libraries,’’ in Usability Assessment of
    Article? Insights from a Web Usability Study,’’ Journal of              Library-Related Web Sites: Methods and Case Studies, edited by
    Academic Librarianship 28 (May/June 2002): 122 – 132.                   Nicole Campbell (Chicago: Library and Information Technology
12. Breeding, ‘‘Library Web Site Analysis.’’                                Association, 2001), p. 110.
13. Janet Crum, Dolores Judkins & Laura Zeigen, ‘‘A Tale of Two         17. Kathleen Collins & José Aguinaga, ‘‘Learning as We Go: Arizona
    Needs: Usability Testing and Library Orientation,’’ Computers in        State University West Library’s Usability Experience,’’ in Usabil-
    Libraries 23 (January 2003): 22 – 24.                                   ity Assessment of Library-Related Web Sites: Methods and Case
14. Crum, Judkins, & Zeigen, ‘‘A Tale of Two Needs;’’ Makar,                Studies, edited by Nicole Campbell (Chicago: Library and
    ‘‘Earning the Stamp of Approval.’’                                      Information Technology Association, 2001), p. 20.
15. Jakob Nielsen, ‘‘Why You Only Need to Test With 5 Users,’’          18. Susan Feldman, ‘‘The Key to Online Catalogs that Work? Testing:
    Useit.com: Alertbox (March 19, 2000), http://useit.com/alertbox/        One, Two, Three,’’ Computers in Libraries 19 (May 1999): 16 – 19.
    20000319.html (accessed December 31, 2004).                         19. McMullen, ‘‘Usability Testing in a Library Web Site Redesign
16. Jennifer Church, Jeanne Brown & Diane VanderPol, ‘‘Walking the          Project.’’
    Web: Usability Testing of Navigational Pathways at the University   20. Norlin & Winters, Usability Testing for Library Web Sites, p. 32.

                                                                                                                             January 2006 51
You can also read
NEXT SLIDES ... Cancel