Beta.govt.nz - User research (Round 3) - WEB TOOLKIT New Zealand Government 5/23/2014

 
5/23/2014           beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit

  New Zealand Government

  WEB TOOLKIT

  beta.govt.nz — User research (Round 3)
  About this report
  This is a report completed for the beta.govt.nz site as it was in early 2014. Since this research was
  completed we’ve already made some changes to the beta site, so some of the recommendations in this
  report have already been addressed.

  The bulk of this report was prepared by Optimal Experience who conducted the research on our behalf.
  At various places in the report you will see comment boxes titled “What we think” or “What we’re
  doing”. These comments are the views of the project team not Optimal Experience and have been added
  to the report.

  Contents
       1. Executive Summary
               Overview
                     User Testing Process
                     User Findings Summary
                     Key Findings and Recommendations
       2. Participants
                 Characteristics
                     Internet Usage
                     NZ Government Websites and newzealand.govt.nz Usage
                     Website Perceptions
       3. Scores and Ratings
                     SUS Scores
                     Task Scores
       4. Task Related Findings
               Task 1 – Low Income & Housing Support
                     Task 2 – Jury Duty
                     Task 3 – Consumer Rights

https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/                               1/50
5/23/2014           beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit

                     Task 4 – Contact Phone Number
                     Task 5 – Name Change
                     Task 6 – School Holidays
                     Task 7 – Travel Overseas
                     Task 8A – Complain about Government Agency
                     Task 8B – Minister of Government Agency
       5. General Findings
               Website Purpose and Scope
                     Visual Design
                     Navigation
                     Site Search
                     Homepage
                     Signpost Page – Level 4
                     Government A-Z page (aka Contact Government)
                     Agency Page
                     Footer
                     Topics page (aka Browse govt.nz)

  Executive Summary

  Overview

  Between February and April 2014, Optimal Experience undertook six sessions of user testing with 30
  participants in six different locations throughout New Zealand. A beta version of the govt.nz website
  was tested.

  This project followed earlier rounds of user testing in October 2013 for the same website.

  User Testing Process

  Objectives of the Study

  The objectives of this project were to find out:

            What search terms do users use in Google and how does landing, following a Google search
            result, on a deep content page effect the users’ performance?
            When and why do users use the local site search, what search terms do they use and what
            problems do they encounter?
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/                               2/50
5/23/2014           beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit

            What pathways do users take when approaching a task?
            Do users use the ‘Topics’ (aka ‘Browse.govt.nz’) and ‘Contact government’ (aka ‘Government
            A-Z’) sections and do they understand how they work?
            Can users easily find information about Departments and Agencies?
            Do users have problems with the different information hubs on the site?
            Can users find contact details easily?
            Do users notice when content links to other agency websites and are they comfortable with that?
            What do participants make of the ‘no frills’ design?

  Study Approach

  The user tests were carried out in six different locations over a 3-month period:

            Auckland on Monday, 17th February
            Manukau on Wednesday, 19th February
            Porirua on Thursday, 6th March
            Christchurch on Thursday, 13th March
            Dunedin on Thursday, 3rd April
            Nelson on Tuesday, 8th April

  Participants were given several tasks to complete using the beta.govt.nz website. Most participants
  completed all nine tasks while some, due to time constraints completed less tasks.

  Some tasks were slightly altered between the rounds of testing to test different aspects of a task, but
  remained essentially the same.

  There were also some formatting and layout changes to the website itself during this timeframe.

  Participants were asked to start some tasks with Google, but otherwise were given the choice between
  browsing and using the local site search.

  As part of the testing session, participants were also given two exercises in order to capture their
  perceptions of NZ government sites in general, and the beta.govt.nz site in particular.

  Usability Findings Summary

  The user testing revealed 89 findings which were given a usability rating which indicates the severity of
  the finding and to which extend this finding will impact on the user experience.

            22 positive findings,
            2 severe issues that require immediate attention because they prevent users from completing their
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/                               3/50
5/23/2014           beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit

             tasks, appear frequently or in crucial situations,
             25 major issues that make it difficult to use the site and should be fixed as soon as possible,
             16 minor issues that should be fixed if resources are available because they can cause
             irritation/confusion or occurs rarely,
             24 observations which are not problematic but important to note as well.

  Each minor, major or severe issue is accompanied by a recommendation, how this issue could be
  solved.

  Key Findings and Recommendations

  Liked Website purpose.

  Most participants understood the purpose of the site and liked that it was a one-stop shop for
  government information. Most participants said that they would like to use it.

            “The mother ship of all government departments.”

            —30-49 year old (Asian) from Christchurch

            “Definitely use it – would go on and learn things even though I didn’t need to use it.”

            —30-49 year old (NZ European) from Auckland

  Govt.nz website better than other government websites.

  Some participants commented that the govt.nz website was easy to use and made them realise how bad
  some other government websites were.

            “That was fairly easy, it’s all straightforward. Most government websites are intimidating.”

            —Female (NZ European) from Auckland

  Content easy to understand.

  Participants found information easy to understand and apprehend.

            “It’s right there. Bullet points, short sentences – done!”

            —20-29 year old (Pacific Islander) from Manukau

  Would start with Google and take it from there.

  Most participants said that they would start looking for information by using Google.
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/                               4/50
5/23/2014           beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit

            “Anything I want to know I just Google it, grandchildren tell me to.”

            —50-65 year old (Māori) from Manukau

  Younger participants less patient.

  Older participants seemed more willing to explore and try different avenues when they couldn’t find
  information straight away.

  Task success not reflected in SUS.

  A high task success rate did not necessarily result in a high System Usability Score (SUS). For example
  Christchurch had the highest task success rate (93%) but the lowest SUS (63.5%).

  Prefer direct contact for help.

  Most participants said that they would call somebody or the department in question if they weren’t able
  to find the information online. Participants who got stuck, usually looked for a number to call or a way
  to contact the responsible department or agency.

            “It’s sometimes easier to call; it saves you clicking. … It’s a habit.”

            —30-49 year old (Pacific Islander) from Porirua

  Too plain.

  Most participants found the website plain and unattractive. Though this didn’t seem to be a problem for
  most participants; when asked what they would change about the site, a quarter of
  participants said that it should be more appealing.

            “Would make it a bit more appealing for the eye – not saying put flowers. It’s not attractive
            ay. I’d rather have effective and efficient rather than attractive.”

            —50-65 year old (Pacific Islander) from Auckland

            “I wouldn’t enjoy looking at that, there’s no pictures, nothing that screams excitement.”

            —Under 20 year old (Asian) from Porirua

  Recommendation

  Consider making the website visually more appealing to improve ‘perceived usability’.

  What we’re doing

https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/                               5/50
5/23/2014           beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit

  We’re already working on the next iteration of the look and feel for Govt.nz which should be live by
  June this year.

  Search often unhelpful.

  Multiple search terms and misspellings usually resulted in a ‘No results’ message or in irrelevant search
  results.

            “This doesn’t help me at all.”

            —50-65 year old (NZ European) from Dunedin

  Recommendations

  Make sure the search handles multiple search term queries well and if returning no results, provide
  suggestions on how to improve the search results.

  Cater for common misspellings.

  What we’re doing

  We were still working on fixes and enhancements to the site’s search engine when this research was
  undertaken. We’ve already made some changes to the way the search engine works, it should now
  handle multiple search terms better than it has previously. We’re also setting up synonyms for common
  terms and phrases based on actual search keywords used by people visiting beta.govt.nz

  Government A-Z needs improvement.

  Participants often overlooked the ‘Government A-Z’ section on the homepage. On the ‘Government A-
  Z’ page they often overlooked the filter and had to scroll the list up and down in order to find the
  department or agency they were looking for. Participants tried to but weren’t able to filter by subjects.

  Recommendations

  Consider moving the ‘Government A-Z’ section above the fold to the top part of the homepage.

  Consider enabling users to filter the ‘Government A-Z’ page by subject and not only by department
  name.

  What we’re doing

  The next version of the homepage moves some of the elements around and should make both the
  Government A-Z and the Topics page easier to find. We’re going to track which pathways users follow
  to measure if the changes have had a positive impact.
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/                               6/50
5/23/2014           beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit

  Browsing too tedious.

  Some participants found the website tedious to navigate and felt that it had too many levels.

            “A maze of levels.”

            —50-65 year old (NZ European) from Nelson

  Recommendation

  Enable users to complete tasks more quickly, e.g. by providing quicker access to the footer, making
  search more reliable and making the ‘Topics’ page layout another layout option for the homepage.

  Missing content by not scrolling.

  Some participants missed important information on the Homepage like ‘Government A-Z’ or on other
  pages because they didn’t scroll.

            “I didn’t scroll down. Usually there is a bunch of un-useful information at the bottom, it
            would need to be at the top. I didn’t even see that, I scrolled past it like 3 times.”

            —30-49 year old (NZ European) from Nelson

  Recommendations

  Consider including page elements that stretch over more than one page length.

  Consider removing horizontal lines that reach across the whole page width.

  Participants

  Characteristics

  For this round of user testing we tested 30 participants in total in six different locations, i.e. Auckland,
  Manukau, Porirua, Christchurch, Dunedin and Nelson.

https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/                               7/50
5/23/2014           beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit

  There was an even split between genders and a good spread across the different age groups.

  Based on the most recent census data from Statistics New Zealand, we selected participants from
  different ethnic backgrounds including NZ European, Pacific Islander, Māori and Asian.

https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/                               8/50
5/23/2014           beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit

  Internet Usage

  There was an even spread across different levels of internet usage by age group and internet usage by
  ethnicity.

https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/                               9/50
5/23/2014           beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit

  When asked what things they used the internet for, most participants responded

            Facebook (73%),
            Doing research (50%),
            Emails (37%),
            TradeMe (33%),
            And watching or downloading videos, movies or TV (33%).

  Almost every participant said that they would use Google to find information online.

  NZ Government Websites and newzealand.govt.nz Usage

  Most participants had used NZ government websites before (93%) but only a third of participants had
  heard about newzealand.govt.nz

  Most thought that the website would have something to do with NZ government or government
  departments.

            “I imagine it’s an overall website that covers all the government departments.”

            —50-65 year old (NZ European) from Nelson

https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/                               10/50
5/23/2014           beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit

  The NZ government websites most mentioned by participants were:

            Inland Revenue (43%),
            Work and Income (30%),
            Local Council websites (30%),
            Department of Internal Affairs (23%),
            New Zealand Transport Agency (17%),
            StudyLink (17%),
            Library websites (17%),
            And public transport websites (17%).

  Most participants rarely visited NZ government websites, i.e. once a month or less and quite often it
  depended on the participant’s needs or the time of year.

            “They are not compelling, you only use it because you have to.”

            —50-65 year old (NZ European) from Dunedin

  Most participants said that they would use Google to find government information and services (80%),
  some said they would prefer to talk to somebody in person either face-to-face (17%) or over the phone
  (13%).

            “I would Google it and hope that something comes up.”

            —20-29 year old (NZ European) from Christchurch

  Three quarters of the participants said that they would definitely use the site or use it when they needed
  to.

            “Probably go on tonight and use it, found it really interesting. Lot of things I didn’t know
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/                               11/50
5/23/2014           beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit

            about.”

            —20-29 year old (Pacific Islander) from Manukau

            “I will certainly put it in my favourites.”

            —30-49 year old (Asian) from Christchurch

            “I don’t need it I guess, haven’t really got any problems at the moment.”

            —Under 20 year old (Asian) from Dunedin

            “I’m going to bookmark this site and use it all the time.”

            —50-65 year old (Pacific Islander) from Porirua

            “I actually like it, it’s really good.”

            —50-65 year old (NZ European) from Dunedin.

  Only 10% said that they wouldn’t use it.

            “I wasn’t overly impressed – definitely room for improvement.”

            —50-65 year old (Pacific Islander) from Auckland

            “Not well integrated, information too much all over the place”

            —Under 20 year old (Asian) from Porirua

            “I have to read too much, there was lots of information to decipher first. You would have to
            get use to the scrolling and knowing the footer and what not.”

            —30-49 year old (Asian) from Christchurch”

  Website Perceptions

  In order to gain an insight of users perceptions of government websites in general and the govt.nz
  website in particular, we asked participants, in two separate exercises, to sort a list of 35
  adjectives into two categories; one that included the ones that applied to government websites or the
  govt.nz website and one for the adjectives that didn’t apply.

  NZ Government websites

  The words that participants most commonly associated with NZ Government websites were
  Professional (93%) and Helpful (89%).

https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/                               12/50
5/23/2014           beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit

  Also high in the rankings were Usable (79%), Organised (79%), Reliable (78%), Convenient (78%),
  Secure (78%), Trustworthy (75%), and Ordinary (71%).

  There was also high agreement that NZ Government websites were NOT Fun (96%), NOT Stimulating
  (89%), NOT Incomprehensible (85%), NOT Patronising (81%), and NOT Inconsistent (79%).

  Beta.govt.nz

  The words that participants most commonly associated with beta.govt.nz were Reliable (74%),
  Organised (70%) and Trustworthy (70%).

  Also high in the rankings were Convenient (68%), Helpful (68%) and Friendly (68%).

  There was also high agreement that beta.govt.nz was NOT Impersonal (70%), NOT Slow (67%), NOT
  Hard to Use (61%), and NOT Intimidating (60%).

  Main perception differences

  There were some interesting differences between how participants perceived NZ government websites
  in general and beta.govt.nz in particular:

  Data collected about participants’ perception of
  NZ Government and newzealand.govt.nz websites
            Perception                   NZ Government websites beta.govt.nz

   Professional                                           93%                              53%

   Fun                                                     0%                              44%

   Impersonal                                             56%                              15%

   NOT Compelling                                         64%                              28%

   NOT Impressive                                         69%                              27%

   NOT Incomprehensible                                   85%                              42%

   NOT Stimulating                                        89%                              33%

  Scores and Ratings

  SUS Scores

  The overall Usability (SUS) score for the govt.nz website was 70.3%, which is higher than

https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/                               13/50
5/23/2014           beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit

  average.

  A score of 70.3% indicates that people agreed that the website is easy to use. For example, a score of
  50% would mean that respondents neither agreed nor disagreed that the website was usable and a score
  of 100% would mean that everyone strongly agreed that the website is usable.

  A score of 68% is considered average. In the book Measuring the User Experience: Collecting,
  Analyzing, and Presenting Usability Metrics 50 SUS scores were analysed from published usability
  studies that determined that the mean score was 66% and the median as 69%.

  SUS by Location

  The SUS varied widely between the different locations with the lowest score in Christchurch (63.5%)
  and the highest score in Dunedin (77.5%).

  It is important to note that there is only a modest correlation between the SUS and the users’ task
  performance. For example in this test, Christchurch had the highest average task success rate (93%) but
  the lowest SUS (63.5%) while Manukau had the lowest average task success rate (52%) and a higher
  SUS (70%).

  SUS by Ethnicity

  The SUS also varied widely between people with different ethnicity with participants of Asian
  background giving the lowest score (59.5%) and participants of NZ European background giving the
  highest score (78.3%).

https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/                               14/50
5/23/2014           beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit

  SUS by Age Group

  Looking at SUS by age group, the website scored the highest usability score with 30-49 year olds
  (73.5%), while under 20 year old gave it the lowest score (65%).

  SUS by Participants

  The SUS score also varied widely between participants with 30% being the lowest score and 97.5%
  being the highest score given.

  More than two thirds of participants gave the website a higher score than 70%.

https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/                               15/50
5/23/2014           beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit

  Task Scores

  For all tasks the following measurements were recorded:

  Task success rate: Shows how many percent of participants managed to complete the set task
  successfully within the set parameters.

  Task directness: Indicates how directly participants completed a task by showing how many percent of
  participants managed to complete the tasks through the shortest and most straightforward route. For
  example, 80% directness means 80% of participants managed to complete the task through the most
  direct route.

  Time taken: Shows how long it took participants to complete a task. The times included in this report
  represent the average time taken across all participants who completed a task.

  Overall Scores

  Overall, Task 3 – Consumer Rights was the best performing task with the highest task success as well as
  directness. Only one out of 24 participants who attempted this task, failed to complete it.

  Task 8A – Complaining about a government department was the worst performing task. Only 10 out of
  26 participants who attempted to complete this task managed to complete it successfully.

  The tasks with the highest success rates (Task 2, Task 3, Task 5 and Task 6) were also the ones
  completed the fastest while the least successful task (Task 7) was also the slowest to be completed.

https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/                               16/50
5/23/2014           beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit

  Task Success by Location

  Comparing the task success rates between the different locations and with the overall success score
  shows that Manukau performed the worst and Christchurch the best.

  Task Success by Ethnicity

  ‘NZ European’ participants completed the most tasks (80%) and ‘Pacific Islander’ participants the least
  (66%).

  Task Success by Age Group

  Overall participants over 50 years old (79%), between 20 and 29 year old (75%) and between 30-49
  year old (74%) were more successful in completing tasks than participants under 20 years old (61%).

  Task related findings

  Task 1 – Low Income & Housing Support

  Participants were given the following task: “Imagine you’re not working as many hours as you used to
  and you’re now having difficulty paying your rent. What would you do? Where would you look for
  help?”
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/                               17/50
5/23/2014           beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit

  Overall Task 1 received the following scores:

             Task success rate: 87%
             Task directness: 57%
             Average time taken: 1m 43 sec

  1 – Most common path for Task 1 (Observation)

  More than half of the participants who attempted this task selected the ‘Help paying rent and housing
  costs’ in the search results or on the ‘Money, benefits and tax’ page. Here most participants selected the
  ‘Check what benefits you might get’ link to go to the Work and Income website.

            “I suppose it’s like an invitation to receive help, it’s telling me what I might get.”

            —30-49 year old (Pacific Islander) from Porirua

  2 – Google search for Task 1 (Observation)

  Most participants who used Google for this task used the search term ‘Work and Income’ or ‘rent’.

  A third of those selected the ‘Don’t have enough income’ search result option because it was a link to
  the Work and Income site or because it seemed the most fitting one for the participant’s situation.

  Three said they wouldn’t select the ‘Budgeting and managing your money’ option because it wouldn’t
  help them when they needed help immediately.

            “Because it [Don’t have enough income] says what I don’t have.”

            —Under 20 year old (Asian) from Dunedin

  3 – Help from the government (Observation)

  A third of participants said they would first talk to their Landlord, ask their family for help, get another
  job or talk to their bank before seeking help from the government.

  Some even said they wouldn’t go to any government department because of pride or because they
  needed to provided too much private information to get help.

  Most thought that Work and Income would be the right government agency to help with this problem;
  with a third of the participants wanting to go straight to the Work and Income website. Though

https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/                               18/50
5/23/2014           beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit

  participants who were students mentioned that they would go to StudyLink first.

            “They will want to know everything about you.”

            —50-65 year old (Pacific Islander) from Porirua

            “I’d stop here. I’d do everything I could do before I had to give them my information.”

            —50-65 year old (NZ European) from Dunedin

  4 – Homepage category for Task 1 (Positive)

  All participants who tried to find the answer to this task by going through the homepage either went to
  ‘Money, benefits and tax’ or ‘Work and jobs’.

            “That’s more relevant to me.” 30-49 year old (Pacific Islander) from Porirua

            “I was gonna go for a search but I’m thinking this [Money, benefits and tax] might help.”” 50-
            65 year old (NZ European) from Nelson”

  5 – Couldn’t find information on external site (Major)

  Participants who didn’t complete this task usually had trouble finding the information they required.

  Most said that they would look for a phone number to call the responsible department.

            “I feel a bit stuck with that.”

            —30-49 year old (NZ European) from Auckland

            “Would be helpful to see something like ‘are you struggling to pay rent, buy food’. Something
            that acknowledges that you’re in the right place, could be so stressed in that situation and
            panic.”

            —30-49 year old (NZ European) from Auckland

            “I’d probably have to ring someone up – it’s a bit complicated.”

            —30-49 year old (NZ European) from Auckland

  Recommendation

  Consider providing enough context and information on the site before offering a link to an external
  website, so users don’t feel the need to go to
  another site.

https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/                               19/50
5/23/2014           beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit

  Task 2 – Jury Duty

  Participants were given the following task: “You’ve been selected to be on a jury and you’d like to try to
  get out of it. Find out if you can do that and why.”

  Overall Task 2 received the following scores:

            Task success rate: 93%
            Task directness: 73%
            Average time taken: 1m 22 sec

  6 – Most common path for Task 2 (Observation)

  Most participants selected the ‘Crime, law and justice’ category on the homepage and then the ‘Jury
  Service’ option in the ‘Popular pages’ section.

  7 – Search for Task 2 (Observation)

  Most participants who used Google or the site search for this task used the search term ‘jury service’ or
  ‘get out of a jury’.

  Most participants that used Google went straight to the Ministry of Justice website.

  8 – Homepage category for Task 2 (Positive)

  All participants who tried to find the answer to this task by going through the homepage went to ‘Crime,
  law and justice’.

  9 – Information easy to understand (Positive)

  Participants commented that the information was easy to understand.

            “It’s just simple how it’s worded.”

            —20-29 year old (Pacific Islander) from Manukau

  10 – Missed information because didn’t scroll (Major)

  Some participants missed the ‘Read ‘If you can’t do jury service’ section or the ‘Apply to be excused
  from jury service’ link because they didn’t scroll all the way to the bottom of the ‘Jury Service’ page.

            “Won’t blame the site, I didn’t read further down.”
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/                               20/50
5/23/2014           beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit

            —50-65 (Pacific Islander) from Auckland

            “Would have been easier if I took my time, scrolled to the bottom of the page – instead of just
            read the top bits.”

            —20-29 year old (Pacific Islander) from Manukau

  Recommendation

  Consider displaying a page index or page summary at the top of the page with links to key sections.

  What we think

  We’ve seen this work pretty well on hub pages that show ‘popular pages’ a featured section and links to
  other sub categories. This is probably worth trying. We’ll set it up so that the template inserts the links
  on the page automatically to save our editors from having to create and change links as content changes.
  For this particular page we’ve also split it into 2 different pages. The old version of the page was trying
  to address 2 completely different user needs on a single page – being on a jury, and getting out of jury
  service.

  11 – Information not applicable to user’s situation (Minor)

  Two participants were unsure if they could be excused from jury service because they were studying
  and not working.

  Recommendation

  Consider including information or references that specifically relate to students.

  What we’re doing

  We will put this content back through our review and editing process, information for students might not
  be the only things that’s missing from the page.

  12 – Popular pages too specific (Observation)

  Four participants didn’t select ‘Jury Service’ from ‘Popular pages’ on the ‘Crime, Law and Justice’ page
  but the ‘Courts and Jury service’ link further down, because the ‘Jury Service’ link looked too specific.

            “Maybe that popular pages kinda threw me a bit. …Not sure why, sometimes you think some
            of these sites the simplest one isn’t the one.”

            —50-65 year old (NZ European) from Nelson

https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/                               21/50
5/23/2014           beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit

  Task 3 – Consumer Rights

  Participants were given the following task: “You bought a computer from Harvey Smith 18 months ago,
  but it now keeps crashing. You take it back to the shop but they tell you it’s not covered since it’s more
  than a year old and want to charge you $800 to fix it. You don’t think that’s fair, is there anything you
  can
  do?”

  Overall Task 3 received the following scores:

             Task success rate: 97%
             Task directness:83%
             Average time taken: 1m 46 sec

  13 – Most common path for Task 3 (Observation)

  Most participants selected the ‘Consumer rights’ category on the homepage and then the ‘Your rights
  even if you don’t have a warranty’ option in the ‘Popular pages’
  section or the ‘Consumer disputes and complaints’ option and then the ‘Your rights even if you don’t
  have a warranty’ link.

  Some also selected the ‘Complain about faulty goods’ or the ‘Your rights when making a customer
  complaint’ options.

  14 – Homepage category for Task 3 (Positive)

  Most participants who tried to find the answer to this task went straight to ‘Consumer rights’.

  Some selected ‘Consumer rights’ after going to a different section, i.e. ‘Internet, Media and
  Communication’, ‘Government and politics’, ‘Crime, Law and Justice’, but realised that they were in
  the wrong place, after looking at the page.

  15 – Information easy to understand (Positive)

  Most participants found this task very easy to complete and the content easy to understand.

            “This is easier, a lot easier.”

            —50-65 year old (Māori) from Manukau

            “Good, it gave you all the avenues you could follow. And it didn’t take long to find that.”

https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/                               22/50
5/23/2014           beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit

            —30-49 year old (Māori) from Nelson

  16 – Options too similar (Minor)

  Some participants were unsure which option on the ‘Consumer disputes and complaints’ page was the
  right one to choose because they sounded quite similar, e.g. ‘Complain about faulty goods’, ‘Your rights
  even if you don’t have a warranty’ and ‘Your rights when making a consumer complaint’.

            “I’m playing a guessing game?”

            —Under 20 year old (Asian) in Porirua

  Recommendation

  Consider if all pages are necessary or if they could be combined.

  Task 4 – Contact Phone Number

  Participants were given the following task: “You want to contact the [government department]. Find a
  contact phone.”

  Most participants were asked to find the phone number for the Ministry of Fisheries. Other government
  departments included Work and Income and Ministry of Justice.

  Overall Task 4 received the following scores:

            Task success rate:83%
            Task directness: 25%
            Average time taken: 1m 57 sec

  17 – Unsure about department responsibilities (Observation)

  Some participants who were asked to find a contact phone number to enquire about fish catch limits
  were unsure who was responsible for those.

  Some thought the Department of Conservation was responsible and went to the ‘Environment and
  climate’ section.

  18 – Contact information in footer (Positive)

  Some participants found the contact information through the footer, either by clicking on the ‘Contact
  government’, ‘Government A-Z’ or the agency name.
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/                               23/50
5/23/2014           beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit

  19 – Unfamiliar with new ministry structure (Observation)

  Half of the participants didn’t know that Ministry of Fisheries is now part of Ministry for Primary
  Industries.

  20 – Helpful synopsis (Positive)

  Almost all participants who used the site search to find the Ministry of Fisheries selected ‘Ministry for
  Primary Industries’ from the search results list.

  Though some participants were slightly confused when the search didn’t return ‘Ministry of Fisheries’
  as a result the synopsis underneath the heading helped them to understand that the ‘Ministry for Primary
  Industries’ search result was the right one.

            “Oh! Found it. Oh okay, now it looks like MPI is covering Fisheries.”

            —50-65 year old (Pacific Islander) from Porirua

  21 – Right panel obvious (Positive)

  Participants who found the agency page saw the phone number in the right panel.

  22 – ‘Hidden’ contact details (Major)

  A third of participants who attempted this task selected a category on the homepage that seemed related
  to the agency they were looking for, e.g. ‘Community, arts and
  sport’ or ‘Environment and climate’ for ‘Ministry of fisheries’, ‘Crime, law and justice’ for Ministry of
  Justice or ‘Money, benefits and tax’ for Work and Income.

  Most gave up on the second or third level (Information hub Level 2 and 3) because they couldn’t see
  any contact information on these pages.

            “I’m lost now.”

            —30-49 year old (NZ European) from Auckland

  Recommendation

  Consider providing prominent links to subject-related contact information on every page, e.g. in the right
  panel.

  23 – Site search for Task 4 (Observation)

  Most participants who attempted this task by using the site search used the name of the agency, e.g.
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/                               24/50
5/23/2014           beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit

  ‘Ministry of fisheries’ or the subject about which they wanted to contact the agency, e.g. ‘fishing limits’.

  Task 5 – Name Change

  Participants were given the following task: “Imagine that a female friend of yours has changed her
  name/has got married and has changed her name and
  now needs to change the details on her driver license. Find out how to do this.”

  Overall Task 5 received the following scores:

             Task success rate:97%
             Task directness: 64%
             Average time taken: 1m 13 sec

  24 – Most common path for Task 5 (Positive)

  Almost two thirds of participants who attempted this task selected ‘Driving and transport’ on the
  homepage and then the ‘Driver licences’ and ‘Change your name on your driver licence’ links. All
  participants who followed this path found the right information.

            “That was easier than the last few – saw driver licence on the homepage.”

            —20-29 year old (Pacific Islander) from Manukau

  25 – Browsing by context (Observation)

  Almost half of the participants selected the ‘Births, deaths and marriages’ category from the homepage
  because they saw changing your name in the context of marriage.

  26 – Obvious link on the ‘Change name’ page (Positive)

  Most participants who went to the ‘Change name’ page saw and selected the ‘Change name on driver’s
  license’ link.

            “I thought that was quite easy. I don’t usually find things that easily.”

            —20-29 year old (NZ European) from Christchurch

            “That was quick. I had a legalised name change in my mind, but I took the easy route. I’ll
            change my name to Phil Dotcom; I’ll earn millions.”

            —50-65 year old (NZ European) from Christchurch
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/                               25/50
5/23/2014           beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit

  27 – Information easy to understand (Positive)

  Most participants found this task very easy to complete and the content easy to understand.

            “It clearly says what you need to change the name.”

            —20-29 year old (Pacific Islander) from Manukau

            “Simple. Had to be simple or I wouldn’t have got there.”

            —50-65 year old (NZ European) from Dunedin

  28 – Confusing terminology (Minor)

  Three participants were confused by the term ‘replace’ in the ‘application for a replacement driver
  licence’ link.

  They didn’t seem to see the heading ‘If you’re changing your name, you have to replace your licence’
  because they were concentrating
  on the link.

  Recommendation

  Consider providing an explanation close to the ‘application for a replacement driver licence’ link that
  explains why users need to replace the driver licence.

  Task 6 – School Holidays

  Participants were given the following task: “You’ve just seen a really good holiday deal online for 5
  days on the Gold Coast. You want to book it as it will be gone soon but need to check if your kids will
  be in school. Where would you find this information?” or “Find out when the next school holidays
  start.”

  Overall Task 6 received the following scores:

            Task success rate:91%
            Task directness: 77%
            Average time taken: 1m 14 sec

  29 – Most common path for Task 6 (Positive)

  Three quarters of all participants who attempted this task selected the ‘Education and training’ category
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/                               26/50
5/23/2014           beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit

  on the homepage and the ‘School terms 2014’ option either in the ‘Popular pages’ or the ‘Schools and
  colleges’ section straight away.

            “Really easy, couple of clicks away and right there. Might even have it set as my
            homepage/bookmark. Something I’d refer to quite a bit.”

            —Female (NZ European) from Auckland

            “Oh, that was quick. That was just so easy.”

            —50-65 year old (Pacific Islander) from Porirua

            “So far I can tell you it’s very, very easy, and the pages complimented each other.”

            —50-65 year old (NZ European) from Christchurch”

  30 – Synopsis not applicable to user’s situation (Minor)

  Three participants hesitated to select the ‘Education and training’ category on the homepage because
  they synopsis underneath the heading didn’t cover their individual
  situation, e.g. kids in primary school.

  Recommendation

  Consider providing a more detailed synopsis on mouse over.

  What we think

  This is a real challenge. We have to balance the need to present enough information to users to help
  them choose the pathways through the site, but we need to do it in the most accessible way. Showing
  additional information in a tooltip that displays on mouse over won’t work for users who rely on
  keyboards or other devices to navigate the web, and there will be inconsistencies in the way assistive
  technologies interpret and present the information. We think the first place to start is to review the
  summaries under each category and see if we can improve them. Feedback from users here is really
  welcome.

  Task 7 – Travel Overseas

  Participants were given the following task: “(You’re getting ready to go on Holiday to the Gold Coast
  with your family.) You receive accommodation supplement (and childcare subsidy). You’re only going
  away for a week. Do you have to tell anyone in Government you’re going?”

https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/                               27/50
5/23/2014           beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit

  Overall Task 7 received the following scores:

            Task success rate: 33%
            Task directness: 21%
            Average time taken: 1m 42 sec

  31 – Correct path for task 7 ignored (Major)

  Only a quarter of participants who attempted this task went to the ‘Before you travel’ page in the
  ‘Passport, travel and tourism’ section.

  Some participants said that they wouldn’t go to ‘Passports, travel and tourism’ because the synopsis
  didn’t say anything about benefits.

            “I’m going to look in the benefits [section] – not thinking holidaying and passports. It should
            have been under the benefits section.”

            —20-29 year old (Pacific Islander) from Manukau

            “Yeaaah, it’s a bit out of left field. Na, I wouldn’t have thought it would have to do with the
            benefit.”” 50-65 year old (NZ European) from Nelson

  Recommendation

  Provide a link to the ‘Before you go’ page from the ‘Money, benefits and tax’ section.

  What we’re doing

  The information people needed was actually already in the ‘Money, benefits and tax’ section but we
  think it might not have been prominent enough. We’ll put some of the pages in this section back through
  our content review process and see if we can improve them. We might need to split some pages where
  they are currently trying to address too many user needs in one place.

  32 – Found required information (Observation)

  Most participants who went to the ‘Before you travel’ page found the information about benefits.

  33 – Search by benefit type unsuccessful (Major)

  Most participants were unsure where to get started and used the site search searching by benefit type,
  e.g. accommodation supplement but didn’t get any or only irrelevant results.

  Recommendation

https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/                               28/50
5/23/2014           beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit

  Improve search.

  What we think

  We’ve kept the content on the beta site pretty thin, and this has meant we don’t actually list the specific
  benefits and payments people may be able to get from agencies like Work and Income. Instead, we
  linked users to the ‘Find out what you might get’ tool but this means we’ve ignored that group of users
  that has some familiarity with what benefits are available, or at least what they used to be called. We’ll
  take another look at the content affected by this and make some improvements.

  34 – Concentrating on the benefits aspect (Major)

  Participants who approached this task from a benefit and not a travel point of view, usually didn’t
  manage to complete this task successfully.

  A third of participants selected the ‘Money, benefits and tax’ category on the homepage.

  One participant who used the search term ‘travel overseas on benefit’ didn’t select the ‘Before you
  travel’ search result but went to ‘Benefits from Work and Income’ page.

            “I was focusing on benefits, not in terms of travel.”

            —50-65 (NZ European) from Dunedin

  Recommendation

  Ensure that users can complete this task when approaching it from the ‘benefit’ angle.

  Consider including a link to the ‘Before you travel’ page in the ‘Money, benefits and tax’ section.

  35 – Task not completed or abandoned (Observation)

  Four participants gave up on this task.

  Most participants who didn’t complete the task said they would call the responsible department, e.g.
  Work and Income.

            “It’s just too hard – if I was at home I’d cry, or call someone.”

            —20-29 year old (Pacific Islander) from Manukau

            “Usually I jump on the phone and use their 0800 number.”

            —50-65 year old (Pacific Islander) from Auckland

https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/                               29/50
5/23/2014           beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit

  36 – Misleading homepage category (Minor)

  Two participants selected ‘Work and Jobs’ on the homepage because they were looking for information
  from Work and Income and thought Work
  and Income would be under there.

  Recommendation

  Consider renaming this category to ‘Jobs and work’.

  What we think

  Having seen users work through this task, it’s now really easy to see why some people would confuse
  ‘Work and Income’ the department, with ‘Work and jobs’ the topic. We will make this change before
  the beta period finishes and the site goes fully live.

  Task 8A – Complain about Government Agency

  Participants were given the following task: “You have been dealing with [government agency]. It’s
  been a nightmare and you’re really unhappy. You already complained to the responsible government
  agency/department without success. Find out if there is another way you can complain about this
  agency/department or an impartial party to complain to.”

  Government agencies we asked participants to complain about included IRD, ACC, StudyLink and
  Work and Income.

  Overall Task 8A received the following scores:

            Task success rate: 34%
            Task directness: 10%
            Average time taken: 3m 01 sec

  37 – Most common path for Task 8A unsuccessful (Major)

  More than half of the participants expected to find the answer to this task under ‘Consumer rights’ but
  couldn’t find it there.

  Some participants went to this section because it included the term ‘complaints’ in the synopsis under
  the heading.

  Recommendation

https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/                               30/50
5/23/2014           beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit

  Consider providing a link to the ‘Complain about a government agency’ page in this section.

  38 – Expected complaint option in the footer (Major)

  Some participants looked for complaint section in the footer, because that it’s were it was on most
  government pages.

  Recommendation

  Consider providing a link to the ‘Complain about a government agency’ page in the footer.

  39 – ‘Consumer rights’ page provided good overview (Positive)

  Most of the participants who went to the ‘Consumer rights’ section realised that it wasn’t the right
  section, after viewing the content of the ‘Consumer rights’ page.

  40 – Relevant search results (Positive)

  In general, participants who used search used the term ‘complaint’ in combination with the name of the
  agency they wanted to complain about.

  For most, the search returned results including ‘Complain to the ombudsman’ and ‘Complain about a
  government agency’.

  41 – Didn’t know about the ombudsman (Observation)

  Half of the participants were unfamiliar with the term ‘ombudsman’ and didn’t know who it was.

  Even some of the participants who knew about the ombudsman didn’t associate complaining about a
  government department with this role straight away.

            “Complain to the who?”

            —20-29 year old (Pacific Islander) from Manukau

            “I don’t know, is that a person or a department?”

            —20-29 year old (NZ European) from Christchurch”

  42 – No clear starting point on the homepage (Major)

  Most participants were unsure where to get started on the homepage and were focused on the agency or
  subject they wanted to complain about rather than the complaints process itself.

https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/                               31/50
5/23/2014           beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit

  Some selected categories which seemed to relate to the agency they wanted to complain about, e.g.
  ‘Money, tax and benefits’ for IRD, ‘Education and training’ for StudyLink, ‘Passport, travel and
  tourism’ for DIA and from there to the agency page.

  Some selected ‘Crime, law and justice’ on the homepage and then ‘Legal rights’ on the ‘Crime, law and
  justice’ page.

            “Can’t see anything immediately on the homepage about complaints”

            —30-49 year old (NZ European) from Auckland

  Recommendation

  Consider including a link to the ‘Complain about a government department’ section at the bottom of
  each department page.

  What we think

  We think we can make information about making a complaint easier to find, but we can also help users
  by making it easier to find the information about getting help when dealing with government. Adding
  something to each organisation’s page in the ‘Government A-Z’ section is something we’ve added to
  our backlog of work to do.

  43 – ‘Parliament and politics’ heading not working (Minor)

  Nobody selected the ‘Parliament and politics’ heading to find the complaints section.

  Some participants selected the ‘Government and politics’ on the homepage and then on of the
  ‘Complaint’ links on the ‘Government and politics’ page.

            “Doesn’t make sense to me, or to the common person, it would be more about policy, I’d be
            looking for advice, complaints.”

            —30-49 year old (NZ European) from Nelson

  Recommendation

  Consider renaming the ‘Parliament and politics’ section to ‘Government and politics’ or move the
  ‘Complain about a government department’ link to the ‘Consumer
  rights’ section.

  44 – Task abandoned (Observation)

  Almost a quarter of participants gave up on this task.
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/                               32/50
5/23/2014           beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit

  Most said they would go to the department website to find out how to complain about the department.

            “Damn.”

            —20-29 year old (Pacific Islander) from Auckland

            “Don’t know what it would be under. Would probably just Google it.”

            —20-29 year old (Pacific Islander) from Manukau

            “That was kind of complicated, found that complicated.”

            —20-29 year old (Pacific Islander) from Manukau

            “I’m lost.”

            —20-29 year old (Pacific Islander) from Manukau

  Task 8B – Minister of Government Agency

  Participants were given the following task: “Find out who the Minister is who is responsible for
  [government agency/department].”

  Government agencies we asked participants to find out about included IRD, StudyLink, Work and
  Income, Ministry of Education, Ministry of Fisheries and Ministry of Justice.

  Overall Task 8A received the following scores:

            Task success rate:51%
            Task directness: 10%
            Average time taken: 2m 13 sec

  45 – Most common path for Task 8B unsuccessful (Major)

  Half of the participants selected the ‘Government and politics’ or ‘Parliament and politics’ category on
  the homepage.

  Most then selected the ‘Parliament’ and then the ‘Members of Parliament’ option and ended up on the
  Parliament website where they didn’t find the information they were looking for

            “Don’t think this is the right way.”

            —30-49 year old (Asian) from Christchurch

https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/                               33/50
You can also read
NEXT SLIDES ... Cancel