Beta.govt.nz - User research (Round 3) - WEB TOOLKIT New Zealand Government 5/23/2014
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
5/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit
New Zealand Government
WEB TOOLKIT
beta.govt.nz — User research (Round 3)
About this report
This is a report completed for the beta.govt.nz site as it was in early 2014. Since this research was
completed we’ve already made some changes to the beta site, so some of the recommendations in this
report have already been addressed.
The bulk of this report was prepared by Optimal Experience who conducted the research on our behalf.
At various places in the report you will see comment boxes titled “What we think” or “What we’re
doing”. These comments are the views of the project team not Optimal Experience and have been added
to the report.
Contents
1. Executive Summary
Overview
User Testing Process
User Findings Summary
Key Findings and Recommendations
2. Participants
Characteristics
Internet Usage
NZ Government Websites and newzealand.govt.nz Usage
Website Perceptions
3. Scores and Ratings
SUS Scores
Task Scores
4. Task Related Findings
Task 1 – Low Income & Housing Support
Task 2 – Jury Duty
Task 3 – Consumer Rights
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/ 1/505/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit
Task 4 – Contact Phone Number
Task 5 – Name Change
Task 6 – School Holidays
Task 7 – Travel Overseas
Task 8A – Complain about Government Agency
Task 8B – Minister of Government Agency
5. General Findings
Website Purpose and Scope
Visual Design
Navigation
Site Search
Homepage
Signpost Page – Level 4
Government A-Z page (aka Contact Government)
Agency Page
Footer
Topics page (aka Browse govt.nz)
Executive Summary
Overview
Between February and April 2014, Optimal Experience undertook six sessions of user testing with 30
participants in six different locations throughout New Zealand. A beta version of the govt.nz website
was tested.
This project followed earlier rounds of user testing in October 2013 for the same website.
User Testing Process
Objectives of the Study
The objectives of this project were to find out:
What search terms do users use in Google and how does landing, following a Google search
result, on a deep content page effect the users’ performance?
When and why do users use the local site search, what search terms do they use and what
problems do they encounter?
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/ 2/505/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit
What pathways do users take when approaching a task?
Do users use the ‘Topics’ (aka ‘Browse.govt.nz’) and ‘Contact government’ (aka ‘Government
A-Z’) sections and do they understand how they work?
Can users easily find information about Departments and Agencies?
Do users have problems with the different information hubs on the site?
Can users find contact details easily?
Do users notice when content links to other agency websites and are they comfortable with that?
What do participants make of the ‘no frills’ design?
Study Approach
The user tests were carried out in six different locations over a 3-month period:
Auckland on Monday, 17th February
Manukau on Wednesday, 19th February
Porirua on Thursday, 6th March
Christchurch on Thursday, 13th March
Dunedin on Thursday, 3rd April
Nelson on Tuesday, 8th April
Participants were given several tasks to complete using the beta.govt.nz website. Most participants
completed all nine tasks while some, due to time constraints completed less tasks.
Some tasks were slightly altered between the rounds of testing to test different aspects of a task, but
remained essentially the same.
There were also some formatting and layout changes to the website itself during this timeframe.
Participants were asked to start some tasks with Google, but otherwise were given the choice between
browsing and using the local site search.
As part of the testing session, participants were also given two exercises in order to capture their
perceptions of NZ government sites in general, and the beta.govt.nz site in particular.
Usability Findings Summary
The user testing revealed 89 findings which were given a usability rating which indicates the severity of
the finding and to which extend this finding will impact on the user experience.
22 positive findings,
2 severe issues that require immediate attention because they prevent users from completing their
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/ 3/505/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit
tasks, appear frequently or in crucial situations,
25 major issues that make it difficult to use the site and should be fixed as soon as possible,
16 minor issues that should be fixed if resources are available because they can cause
irritation/confusion or occurs rarely,
24 observations which are not problematic but important to note as well.
Each minor, major or severe issue is accompanied by a recommendation, how this issue could be
solved.
Key Findings and Recommendations
Liked Website purpose.
Most participants understood the purpose of the site and liked that it was a one-stop shop for
government information. Most participants said that they would like to use it.
“The mother ship of all government departments.”
—30-49 year old (Asian) from Christchurch
“Definitely use it – would go on and learn things even though I didn’t need to use it.”
—30-49 year old (NZ European) from Auckland
Govt.nz website better than other government websites.
Some participants commented that the govt.nz website was easy to use and made them realise how bad
some other government websites were.
“That was fairly easy, it’s all straightforward. Most government websites are intimidating.”
—Female (NZ European) from Auckland
Content easy to understand.
Participants found information easy to understand and apprehend.
“It’s right there. Bullet points, short sentences – done!”
—20-29 year old (Pacific Islander) from Manukau
Would start with Google and take it from there.
Most participants said that they would start looking for information by using Google.
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/ 4/505/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit
“Anything I want to know I just Google it, grandchildren tell me to.”
—50-65 year old (Māori) from Manukau
Younger participants less patient.
Older participants seemed more willing to explore and try different avenues when they couldn’t find
information straight away.
Task success not reflected in SUS.
A high task success rate did not necessarily result in a high System Usability Score (SUS). For example
Christchurch had the highest task success rate (93%) but the lowest SUS (63.5%).
Prefer direct contact for help.
Most participants said that they would call somebody or the department in question if they weren’t able
to find the information online. Participants who got stuck, usually looked for a number to call or a way
to contact the responsible department or agency.
“It’s sometimes easier to call; it saves you clicking. … It’s a habit.”
—30-49 year old (Pacific Islander) from Porirua
Too plain.
Most participants found the website plain and unattractive. Though this didn’t seem to be a problem for
most participants; when asked what they would change about the site, a quarter of
participants said that it should be more appealing.
“Would make it a bit more appealing for the eye – not saying put flowers. It’s not attractive
ay. I’d rather have effective and efficient rather than attractive.”
—50-65 year old (Pacific Islander) from Auckland
“I wouldn’t enjoy looking at that, there’s no pictures, nothing that screams excitement.”
—Under 20 year old (Asian) from Porirua
Recommendation
Consider making the website visually more appealing to improve ‘perceived usability’.
What we’re doing
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/ 5/505/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit
We’re already working on the next iteration of the look and feel for Govt.nz which should be live by
June this year.
Search often unhelpful.
Multiple search terms and misspellings usually resulted in a ‘No results’ message or in irrelevant search
results.
“This doesn’t help me at all.”
—50-65 year old (NZ European) from Dunedin
Recommendations
Make sure the search handles multiple search term queries well and if returning no results, provide
suggestions on how to improve the search results.
Cater for common misspellings.
What we’re doing
We were still working on fixes and enhancements to the site’s search engine when this research was
undertaken. We’ve already made some changes to the way the search engine works, it should now
handle multiple search terms better than it has previously. We’re also setting up synonyms for common
terms and phrases based on actual search keywords used by people visiting beta.govt.nz
Government A-Z needs improvement.
Participants often overlooked the ‘Government A-Z’ section on the homepage. On the ‘Government A-
Z’ page they often overlooked the filter and had to scroll the list up and down in order to find the
department or agency they were looking for. Participants tried to but weren’t able to filter by subjects.
Recommendations
Consider moving the ‘Government A-Z’ section above the fold to the top part of the homepage.
Consider enabling users to filter the ‘Government A-Z’ page by subject and not only by department
name.
What we’re doing
The next version of the homepage moves some of the elements around and should make both the
Government A-Z and the Topics page easier to find. We’re going to track which pathways users follow
to measure if the changes have had a positive impact.
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/ 6/505/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit
Browsing too tedious.
Some participants found the website tedious to navigate and felt that it had too many levels.
“A maze of levels.”
—50-65 year old (NZ European) from Nelson
Recommendation
Enable users to complete tasks more quickly, e.g. by providing quicker access to the footer, making
search more reliable and making the ‘Topics’ page layout another layout option for the homepage.
Missing content by not scrolling.
Some participants missed important information on the Homepage like ‘Government A-Z’ or on other
pages because they didn’t scroll.
“I didn’t scroll down. Usually there is a bunch of un-useful information at the bottom, it
would need to be at the top. I didn’t even see that, I scrolled past it like 3 times.”
—30-49 year old (NZ European) from Nelson
Recommendations
Consider including page elements that stretch over more than one page length.
Consider removing horizontal lines that reach across the whole page width.
Participants
Characteristics
For this round of user testing we tested 30 participants in total in six different locations, i.e. Auckland,
Manukau, Porirua, Christchurch, Dunedin and Nelson.
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/ 7/505/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit There was an even split between genders and a good spread across the different age groups. Based on the most recent census data from Statistics New Zealand, we selected participants from different ethnic backgrounds including NZ European, Pacific Islander, Māori and Asian. https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/ 8/50
5/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit Internet Usage There was an even spread across different levels of internet usage by age group and internet usage by ethnicity. https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/ 9/50
5/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit
When asked what things they used the internet for, most participants responded
Facebook (73%),
Doing research (50%),
Emails (37%),
TradeMe (33%),
And watching or downloading videos, movies or TV (33%).
Almost every participant said that they would use Google to find information online.
NZ Government Websites and newzealand.govt.nz Usage
Most participants had used NZ government websites before (93%) but only a third of participants had
heard about newzealand.govt.nz
Most thought that the website would have something to do with NZ government or government
departments.
“I imagine it’s an overall website that covers all the government departments.”
—50-65 year old (NZ European) from Nelson
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/ 10/505/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit
The NZ government websites most mentioned by participants were:
Inland Revenue (43%),
Work and Income (30%),
Local Council websites (30%),
Department of Internal Affairs (23%),
New Zealand Transport Agency (17%),
StudyLink (17%),
Library websites (17%),
And public transport websites (17%).
Most participants rarely visited NZ government websites, i.e. once a month or less and quite often it
depended on the participant’s needs or the time of year.
“They are not compelling, you only use it because you have to.”
—50-65 year old (NZ European) from Dunedin
Most participants said that they would use Google to find government information and services (80%),
some said they would prefer to talk to somebody in person either face-to-face (17%) or over the phone
(13%).
“I would Google it and hope that something comes up.”
—20-29 year old (NZ European) from Christchurch
Three quarters of the participants said that they would definitely use the site or use it when they needed
to.
“Probably go on tonight and use it, found it really interesting. Lot of things I didn’t know
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/ 11/505/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit
about.”
—20-29 year old (Pacific Islander) from Manukau
“I will certainly put it in my favourites.”
—30-49 year old (Asian) from Christchurch
“I don’t need it I guess, haven’t really got any problems at the moment.”
—Under 20 year old (Asian) from Dunedin
“I’m going to bookmark this site and use it all the time.”
—50-65 year old (Pacific Islander) from Porirua
“I actually like it, it’s really good.”
—50-65 year old (NZ European) from Dunedin.
Only 10% said that they wouldn’t use it.
“I wasn’t overly impressed – definitely room for improvement.”
—50-65 year old (Pacific Islander) from Auckland
“Not well integrated, information too much all over the place”
—Under 20 year old (Asian) from Porirua
“I have to read too much, there was lots of information to decipher first. You would have to
get use to the scrolling and knowing the footer and what not.”
—30-49 year old (Asian) from Christchurch”
Website Perceptions
In order to gain an insight of users perceptions of government websites in general and the govt.nz
website in particular, we asked participants, in two separate exercises, to sort a list of 35
adjectives into two categories; one that included the ones that applied to government websites or the
govt.nz website and one for the adjectives that didn’t apply.
NZ Government websites
The words that participants most commonly associated with NZ Government websites were
Professional (93%) and Helpful (89%).
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/ 12/505/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit
Also high in the rankings were Usable (79%), Organised (79%), Reliable (78%), Convenient (78%),
Secure (78%), Trustworthy (75%), and Ordinary (71%).
There was also high agreement that NZ Government websites were NOT Fun (96%), NOT Stimulating
(89%), NOT Incomprehensible (85%), NOT Patronising (81%), and NOT Inconsistent (79%).
Beta.govt.nz
The words that participants most commonly associated with beta.govt.nz were Reliable (74%),
Organised (70%) and Trustworthy (70%).
Also high in the rankings were Convenient (68%), Helpful (68%) and Friendly (68%).
There was also high agreement that beta.govt.nz was NOT Impersonal (70%), NOT Slow (67%), NOT
Hard to Use (61%), and NOT Intimidating (60%).
Main perception differences
There were some interesting differences between how participants perceived NZ government websites
in general and beta.govt.nz in particular:
Data collected about participants’ perception of
NZ Government and newzealand.govt.nz websites
Perception NZ Government websites beta.govt.nz
Professional 93% 53%
Fun 0% 44%
Impersonal 56% 15%
NOT Compelling 64% 28%
NOT Impressive 69% 27%
NOT Incomprehensible 85% 42%
NOT Stimulating 89% 33%
Scores and Ratings
SUS Scores
The overall Usability (SUS) score for the govt.nz website was 70.3%, which is higher than
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/ 13/505/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit average. A score of 70.3% indicates that people agreed that the website is easy to use. For example, a score of 50% would mean that respondents neither agreed nor disagreed that the website was usable and a score of 100% would mean that everyone strongly agreed that the website is usable. A score of 68% is considered average. In the book Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics 50 SUS scores were analysed from published usability studies that determined that the mean score was 66% and the median as 69%. SUS by Location The SUS varied widely between the different locations with the lowest score in Christchurch (63.5%) and the highest score in Dunedin (77.5%). It is important to note that there is only a modest correlation between the SUS and the users’ task performance. For example in this test, Christchurch had the highest average task success rate (93%) but the lowest SUS (63.5%) while Manukau had the lowest average task success rate (52%) and a higher SUS (70%). SUS by Ethnicity The SUS also varied widely between people with different ethnicity with participants of Asian background giving the lowest score (59.5%) and participants of NZ European background giving the highest score (78.3%). https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/ 14/50
5/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit SUS by Age Group Looking at SUS by age group, the website scored the highest usability score with 30-49 year olds (73.5%), while under 20 year old gave it the lowest score (65%). SUS by Participants The SUS score also varied widely between participants with 30% being the lowest score and 97.5% being the highest score given. More than two thirds of participants gave the website a higher score than 70%. https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/ 15/50
5/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit Task Scores For all tasks the following measurements were recorded: Task success rate: Shows how many percent of participants managed to complete the set task successfully within the set parameters. Task directness: Indicates how directly participants completed a task by showing how many percent of participants managed to complete the tasks through the shortest and most straightforward route. For example, 80% directness means 80% of participants managed to complete the task through the most direct route. Time taken: Shows how long it took participants to complete a task. The times included in this report represent the average time taken across all participants who completed a task. Overall Scores Overall, Task 3 – Consumer Rights was the best performing task with the highest task success as well as directness. Only one out of 24 participants who attempted this task, failed to complete it. Task 8A – Complaining about a government department was the worst performing task. Only 10 out of 26 participants who attempted to complete this task managed to complete it successfully. The tasks with the highest success rates (Task 2, Task 3, Task 5 and Task 6) were also the ones completed the fastest while the least successful task (Task 7) was also the slowest to be completed. https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/ 16/50
5/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit Task Success by Location Comparing the task success rates between the different locations and with the overall success score shows that Manukau performed the worst and Christchurch the best. Task Success by Ethnicity ‘NZ European’ participants completed the most tasks (80%) and ‘Pacific Islander’ participants the least (66%). Task Success by Age Group Overall participants over 50 years old (79%), between 20 and 29 year old (75%) and between 30-49 year old (74%) were more successful in completing tasks than participants under 20 years old (61%). Task related findings Task 1 – Low Income & Housing Support Participants were given the following task: “Imagine you’re not working as many hours as you used to and you’re now having difficulty paying your rent. What would you do? Where would you look for help?” https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/ 17/50
5/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit
Overall Task 1 received the following scores:
Task success rate: 87%
Task directness: 57%
Average time taken: 1m 43 sec
1 – Most common path for Task 1 (Observation)
More than half of the participants who attempted this task selected the ‘Help paying rent and housing
costs’ in the search results or on the ‘Money, benefits and tax’ page. Here most participants selected the
‘Check what benefits you might get’ link to go to the Work and Income website.
“I suppose it’s like an invitation to receive help, it’s telling me what I might get.”
—30-49 year old (Pacific Islander) from Porirua
2 – Google search for Task 1 (Observation)
Most participants who used Google for this task used the search term ‘Work and Income’ or ‘rent’.
A third of those selected the ‘Don’t have enough income’ search result option because it was a link to
the Work and Income site or because it seemed the most fitting one for the participant’s situation.
Three said they wouldn’t select the ‘Budgeting and managing your money’ option because it wouldn’t
help them when they needed help immediately.
“Because it [Don’t have enough income] says what I don’t have.”
—Under 20 year old (Asian) from Dunedin
3 – Help from the government (Observation)
A third of participants said they would first talk to their Landlord, ask their family for help, get another
job or talk to their bank before seeking help from the government.
Some even said they wouldn’t go to any government department because of pride or because they
needed to provided too much private information to get help.
Most thought that Work and Income would be the right government agency to help with this problem;
with a third of the participants wanting to go straight to the Work and Income website. Though
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/ 18/505/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit
participants who were students mentioned that they would go to StudyLink first.
“They will want to know everything about you.”
—50-65 year old (Pacific Islander) from Porirua
“I’d stop here. I’d do everything I could do before I had to give them my information.”
—50-65 year old (NZ European) from Dunedin
4 – Homepage category for Task 1 (Positive)
All participants who tried to find the answer to this task by going through the homepage either went to
‘Money, benefits and tax’ or ‘Work and jobs’.
“That’s more relevant to me.” 30-49 year old (Pacific Islander) from Porirua
“I was gonna go for a search but I’m thinking this [Money, benefits and tax] might help.”” 50-
65 year old (NZ European) from Nelson”
5 – Couldn’t find information on external site (Major)
Participants who didn’t complete this task usually had trouble finding the information they required.
Most said that they would look for a phone number to call the responsible department.
“I feel a bit stuck with that.”
—30-49 year old (NZ European) from Auckland
“Would be helpful to see something like ‘are you struggling to pay rent, buy food’. Something
that acknowledges that you’re in the right place, could be so stressed in that situation and
panic.”
—30-49 year old (NZ European) from Auckland
“I’d probably have to ring someone up – it’s a bit complicated.”
—30-49 year old (NZ European) from Auckland
Recommendation
Consider providing enough context and information on the site before offering a link to an external
website, so users don’t feel the need to go to
another site.
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/ 19/505/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit
Task 2 – Jury Duty
Participants were given the following task: “You’ve been selected to be on a jury and you’d like to try to
get out of it. Find out if you can do that and why.”
Overall Task 2 received the following scores:
Task success rate: 93%
Task directness: 73%
Average time taken: 1m 22 sec
6 – Most common path for Task 2 (Observation)
Most participants selected the ‘Crime, law and justice’ category on the homepage and then the ‘Jury
Service’ option in the ‘Popular pages’ section.
7 – Search for Task 2 (Observation)
Most participants who used Google or the site search for this task used the search term ‘jury service’ or
‘get out of a jury’.
Most participants that used Google went straight to the Ministry of Justice website.
8 – Homepage category for Task 2 (Positive)
All participants who tried to find the answer to this task by going through the homepage went to ‘Crime,
law and justice’.
9 – Information easy to understand (Positive)
Participants commented that the information was easy to understand.
“It’s just simple how it’s worded.”
—20-29 year old (Pacific Islander) from Manukau
10 – Missed information because didn’t scroll (Major)
Some participants missed the ‘Read ‘If you can’t do jury service’ section or the ‘Apply to be excused
from jury service’ link because they didn’t scroll all the way to the bottom of the ‘Jury Service’ page.
“Won’t blame the site, I didn’t read further down.”
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/ 20/505/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit
—50-65 (Pacific Islander) from Auckland
“Would have been easier if I took my time, scrolled to the bottom of the page – instead of just
read the top bits.”
—20-29 year old (Pacific Islander) from Manukau
Recommendation
Consider displaying a page index or page summary at the top of the page with links to key sections.
What we think
We’ve seen this work pretty well on hub pages that show ‘popular pages’ a featured section and links to
other sub categories. This is probably worth trying. We’ll set it up so that the template inserts the links
on the page automatically to save our editors from having to create and change links as content changes.
For this particular page we’ve also split it into 2 different pages. The old version of the page was trying
to address 2 completely different user needs on a single page – being on a jury, and getting out of jury
service.
11 – Information not applicable to user’s situation (Minor)
Two participants were unsure if they could be excused from jury service because they were studying
and not working.
Recommendation
Consider including information or references that specifically relate to students.
What we’re doing
We will put this content back through our review and editing process, information for students might not
be the only things that’s missing from the page.
12 – Popular pages too specific (Observation)
Four participants didn’t select ‘Jury Service’ from ‘Popular pages’ on the ‘Crime, Law and Justice’ page
but the ‘Courts and Jury service’ link further down, because the ‘Jury Service’ link looked too specific.
“Maybe that popular pages kinda threw me a bit. …Not sure why, sometimes you think some
of these sites the simplest one isn’t the one.”
—50-65 year old (NZ European) from Nelson
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/ 21/505/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit
Task 3 – Consumer Rights
Participants were given the following task: “You bought a computer from Harvey Smith 18 months ago,
but it now keeps crashing. You take it back to the shop but they tell you it’s not covered since it’s more
than a year old and want to charge you $800 to fix it. You don’t think that’s fair, is there anything you
can
do?”
Overall Task 3 received the following scores:
Task success rate: 97%
Task directness:83%
Average time taken: 1m 46 sec
13 – Most common path for Task 3 (Observation)
Most participants selected the ‘Consumer rights’ category on the homepage and then the ‘Your rights
even if you don’t have a warranty’ option in the ‘Popular pages’
section or the ‘Consumer disputes and complaints’ option and then the ‘Your rights even if you don’t
have a warranty’ link.
Some also selected the ‘Complain about faulty goods’ or the ‘Your rights when making a customer
complaint’ options.
14 – Homepage category for Task 3 (Positive)
Most participants who tried to find the answer to this task went straight to ‘Consumer rights’.
Some selected ‘Consumer rights’ after going to a different section, i.e. ‘Internet, Media and
Communication’, ‘Government and politics’, ‘Crime, Law and Justice’, but realised that they were in
the wrong place, after looking at the page.
15 – Information easy to understand (Positive)
Most participants found this task very easy to complete and the content easy to understand.
“This is easier, a lot easier.”
—50-65 year old (Māori) from Manukau
“Good, it gave you all the avenues you could follow. And it didn’t take long to find that.”
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/ 22/505/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit
—30-49 year old (Māori) from Nelson
16 – Options too similar (Minor)
Some participants were unsure which option on the ‘Consumer disputes and complaints’ page was the
right one to choose because they sounded quite similar, e.g. ‘Complain about faulty goods’, ‘Your rights
even if you don’t have a warranty’ and ‘Your rights when making a consumer complaint’.
“I’m playing a guessing game?”
—Under 20 year old (Asian) in Porirua
Recommendation
Consider if all pages are necessary or if they could be combined.
Task 4 – Contact Phone Number
Participants were given the following task: “You want to contact the [government department]. Find a
contact phone.”
Most participants were asked to find the phone number for the Ministry of Fisheries. Other government
departments included Work and Income and Ministry of Justice.
Overall Task 4 received the following scores:
Task success rate:83%
Task directness: 25%
Average time taken: 1m 57 sec
17 – Unsure about department responsibilities (Observation)
Some participants who were asked to find a contact phone number to enquire about fish catch limits
were unsure who was responsible for those.
Some thought the Department of Conservation was responsible and went to the ‘Environment and
climate’ section.
18 – Contact information in footer (Positive)
Some participants found the contact information through the footer, either by clicking on the ‘Contact
government’, ‘Government A-Z’ or the agency name.
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/ 23/505/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit
19 – Unfamiliar with new ministry structure (Observation)
Half of the participants didn’t know that Ministry of Fisheries is now part of Ministry for Primary
Industries.
20 – Helpful synopsis (Positive)
Almost all participants who used the site search to find the Ministry of Fisheries selected ‘Ministry for
Primary Industries’ from the search results list.
Though some participants were slightly confused when the search didn’t return ‘Ministry of Fisheries’
as a result the synopsis underneath the heading helped them to understand that the ‘Ministry for Primary
Industries’ search result was the right one.
“Oh! Found it. Oh okay, now it looks like MPI is covering Fisheries.”
—50-65 year old (Pacific Islander) from Porirua
21 – Right panel obvious (Positive)
Participants who found the agency page saw the phone number in the right panel.
22 – ‘Hidden’ contact details (Major)
A third of participants who attempted this task selected a category on the homepage that seemed related
to the agency they were looking for, e.g. ‘Community, arts and
sport’ or ‘Environment and climate’ for ‘Ministry of fisheries’, ‘Crime, law and justice’ for Ministry of
Justice or ‘Money, benefits and tax’ for Work and Income.
Most gave up on the second or third level (Information hub Level 2 and 3) because they couldn’t see
any contact information on these pages.
“I’m lost now.”
—30-49 year old (NZ European) from Auckland
Recommendation
Consider providing prominent links to subject-related contact information on every page, e.g. in the right
panel.
23 – Site search for Task 4 (Observation)
Most participants who attempted this task by using the site search used the name of the agency, e.g.
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/ 24/505/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit
‘Ministry of fisheries’ or the subject about which they wanted to contact the agency, e.g. ‘fishing limits’.
Task 5 – Name Change
Participants were given the following task: “Imagine that a female friend of yours has changed her
name/has got married and has changed her name and
now needs to change the details on her driver license. Find out how to do this.”
Overall Task 5 received the following scores:
Task success rate:97%
Task directness: 64%
Average time taken: 1m 13 sec
24 – Most common path for Task 5 (Positive)
Almost two thirds of participants who attempted this task selected ‘Driving and transport’ on the
homepage and then the ‘Driver licences’ and ‘Change your name on your driver licence’ links. All
participants who followed this path found the right information.
“That was easier than the last few – saw driver licence on the homepage.”
—20-29 year old (Pacific Islander) from Manukau
25 – Browsing by context (Observation)
Almost half of the participants selected the ‘Births, deaths and marriages’ category from the homepage
because they saw changing your name in the context of marriage.
26 – Obvious link on the ‘Change name’ page (Positive)
Most participants who went to the ‘Change name’ page saw and selected the ‘Change name on driver’s
license’ link.
“I thought that was quite easy. I don’t usually find things that easily.”
—20-29 year old (NZ European) from Christchurch
“That was quick. I had a legalised name change in my mind, but I took the easy route. I’ll
change my name to Phil Dotcom; I’ll earn millions.”
—50-65 year old (NZ European) from Christchurch
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/ 25/505/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit
27 – Information easy to understand (Positive)
Most participants found this task very easy to complete and the content easy to understand.
“It clearly says what you need to change the name.”
—20-29 year old (Pacific Islander) from Manukau
“Simple. Had to be simple or I wouldn’t have got there.”
—50-65 year old (NZ European) from Dunedin
28 – Confusing terminology (Minor)
Three participants were confused by the term ‘replace’ in the ‘application for a replacement driver
licence’ link.
They didn’t seem to see the heading ‘If you’re changing your name, you have to replace your licence’
because they were concentrating
on the link.
Recommendation
Consider providing an explanation close to the ‘application for a replacement driver licence’ link that
explains why users need to replace the driver licence.
Task 6 – School Holidays
Participants were given the following task: “You’ve just seen a really good holiday deal online for 5
days on the Gold Coast. You want to book it as it will be gone soon but need to check if your kids will
be in school. Where would you find this information?” or “Find out when the next school holidays
start.”
Overall Task 6 received the following scores:
Task success rate:91%
Task directness: 77%
Average time taken: 1m 14 sec
29 – Most common path for Task 6 (Positive)
Three quarters of all participants who attempted this task selected the ‘Education and training’ category
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/ 26/505/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit
on the homepage and the ‘School terms 2014’ option either in the ‘Popular pages’ or the ‘Schools and
colleges’ section straight away.
“Really easy, couple of clicks away and right there. Might even have it set as my
homepage/bookmark. Something I’d refer to quite a bit.”
—Female (NZ European) from Auckland
“Oh, that was quick. That was just so easy.”
—50-65 year old (Pacific Islander) from Porirua
“So far I can tell you it’s very, very easy, and the pages complimented each other.”
—50-65 year old (NZ European) from Christchurch”
30 – Synopsis not applicable to user’s situation (Minor)
Three participants hesitated to select the ‘Education and training’ category on the homepage because
they synopsis underneath the heading didn’t cover their individual
situation, e.g. kids in primary school.
Recommendation
Consider providing a more detailed synopsis on mouse over.
What we think
This is a real challenge. We have to balance the need to present enough information to users to help
them choose the pathways through the site, but we need to do it in the most accessible way. Showing
additional information in a tooltip that displays on mouse over won’t work for users who rely on
keyboards or other devices to navigate the web, and there will be inconsistencies in the way assistive
technologies interpret and present the information. We think the first place to start is to review the
summaries under each category and see if we can improve them. Feedback from users here is really
welcome.
Task 7 – Travel Overseas
Participants were given the following task: “(You’re getting ready to go on Holiday to the Gold Coast
with your family.) You receive accommodation supplement (and childcare subsidy). You’re only going
away for a week. Do you have to tell anyone in Government you’re going?”
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/ 27/505/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit
Overall Task 7 received the following scores:
Task success rate: 33%
Task directness: 21%
Average time taken: 1m 42 sec
31 – Correct path for task 7 ignored (Major)
Only a quarter of participants who attempted this task went to the ‘Before you travel’ page in the
‘Passport, travel and tourism’ section.
Some participants said that they wouldn’t go to ‘Passports, travel and tourism’ because the synopsis
didn’t say anything about benefits.
“I’m going to look in the benefits [section] – not thinking holidaying and passports. It should
have been under the benefits section.”
—20-29 year old (Pacific Islander) from Manukau
“Yeaaah, it’s a bit out of left field. Na, I wouldn’t have thought it would have to do with the
benefit.”” 50-65 year old (NZ European) from Nelson
Recommendation
Provide a link to the ‘Before you go’ page from the ‘Money, benefits and tax’ section.
What we’re doing
The information people needed was actually already in the ‘Money, benefits and tax’ section but we
think it might not have been prominent enough. We’ll put some of the pages in this section back through
our content review process and see if we can improve them. We might need to split some pages where
they are currently trying to address too many user needs in one place.
32 – Found required information (Observation)
Most participants who went to the ‘Before you travel’ page found the information about benefits.
33 – Search by benefit type unsuccessful (Major)
Most participants were unsure where to get started and used the site search searching by benefit type,
e.g. accommodation supplement but didn’t get any or only irrelevant results.
Recommendation
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/ 28/505/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit
Improve search.
What we think
We’ve kept the content on the beta site pretty thin, and this has meant we don’t actually list the specific
benefits and payments people may be able to get from agencies like Work and Income. Instead, we
linked users to the ‘Find out what you might get’ tool but this means we’ve ignored that group of users
that has some familiarity with what benefits are available, or at least what they used to be called. We’ll
take another look at the content affected by this and make some improvements.
34 – Concentrating on the benefits aspect (Major)
Participants who approached this task from a benefit and not a travel point of view, usually didn’t
manage to complete this task successfully.
A third of participants selected the ‘Money, benefits and tax’ category on the homepage.
One participant who used the search term ‘travel overseas on benefit’ didn’t select the ‘Before you
travel’ search result but went to ‘Benefits from Work and Income’ page.
“I was focusing on benefits, not in terms of travel.”
—50-65 (NZ European) from Dunedin
Recommendation
Ensure that users can complete this task when approaching it from the ‘benefit’ angle.
Consider including a link to the ‘Before you travel’ page in the ‘Money, benefits and tax’ section.
35 – Task not completed or abandoned (Observation)
Four participants gave up on this task.
Most participants who didn’t complete the task said they would call the responsible department, e.g.
Work and Income.
“It’s just too hard – if I was at home I’d cry, or call someone.”
—20-29 year old (Pacific Islander) from Manukau
“Usually I jump on the phone and use their 0800 number.”
—50-65 year old (Pacific Islander) from Auckland
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/ 29/505/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit
36 – Misleading homepage category (Minor)
Two participants selected ‘Work and Jobs’ on the homepage because they were looking for information
from Work and Income and thought Work
and Income would be under there.
Recommendation
Consider renaming this category to ‘Jobs and work’.
What we think
Having seen users work through this task, it’s now really easy to see why some people would confuse
‘Work and Income’ the department, with ‘Work and jobs’ the topic. We will make this change before
the beta period finishes and the site goes fully live.
Task 8A – Complain about Government Agency
Participants were given the following task: “You have been dealing with [government agency]. It’s
been a nightmare and you’re really unhappy. You already complained to the responsible government
agency/department without success. Find out if there is another way you can complain about this
agency/department or an impartial party to complain to.”
Government agencies we asked participants to complain about included IRD, ACC, StudyLink and
Work and Income.
Overall Task 8A received the following scores:
Task success rate: 34%
Task directness: 10%
Average time taken: 3m 01 sec
37 – Most common path for Task 8A unsuccessful (Major)
More than half of the participants expected to find the answer to this task under ‘Consumer rights’ but
couldn’t find it there.
Some participants went to this section because it included the term ‘complaints’ in the synopsis under
the heading.
Recommendation
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/ 30/505/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit
Consider providing a link to the ‘Complain about a government agency’ page in this section.
38 – Expected complaint option in the footer (Major)
Some participants looked for complaint section in the footer, because that it’s were it was on most
government pages.
Recommendation
Consider providing a link to the ‘Complain about a government agency’ page in the footer.
39 – ‘Consumer rights’ page provided good overview (Positive)
Most of the participants who went to the ‘Consumer rights’ section realised that it wasn’t the right
section, after viewing the content of the ‘Consumer rights’ page.
40 – Relevant search results (Positive)
In general, participants who used search used the term ‘complaint’ in combination with the name of the
agency they wanted to complain about.
For most, the search returned results including ‘Complain to the ombudsman’ and ‘Complain about a
government agency’.
41 – Didn’t know about the ombudsman (Observation)
Half of the participants were unfamiliar with the term ‘ombudsman’ and didn’t know who it was.
Even some of the participants who knew about the ombudsman didn’t associate complaining about a
government department with this role straight away.
“Complain to the who?”
—20-29 year old (Pacific Islander) from Manukau
“I don’t know, is that a person or a department?”
—20-29 year old (NZ European) from Christchurch”
42 – No clear starting point on the homepage (Major)
Most participants were unsure where to get started on the homepage and were focused on the agency or
subject they wanted to complain about rather than the complaints process itself.
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/ 31/505/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit
Some selected categories which seemed to relate to the agency they wanted to complain about, e.g.
‘Money, tax and benefits’ for IRD, ‘Education and training’ for StudyLink, ‘Passport, travel and
tourism’ for DIA and from there to the agency page.
Some selected ‘Crime, law and justice’ on the homepage and then ‘Legal rights’ on the ‘Crime, law and
justice’ page.
“Can’t see anything immediately on the homepage about complaints”
—30-49 year old (NZ European) from Auckland
Recommendation
Consider including a link to the ‘Complain about a government department’ section at the bottom of
each department page.
What we think
We think we can make information about making a complaint easier to find, but we can also help users
by making it easier to find the information about getting help when dealing with government. Adding
something to each organisation’s page in the ‘Government A-Z’ section is something we’ve added to
our backlog of work to do.
43 – ‘Parliament and politics’ heading not working (Minor)
Nobody selected the ‘Parliament and politics’ heading to find the complaints section.
Some participants selected the ‘Government and politics’ on the homepage and then on of the
‘Complaint’ links on the ‘Government and politics’ page.
“Doesn’t make sense to me, or to the common person, it would be more about policy, I’d be
looking for advice, complaints.”
—30-49 year old (NZ European) from Nelson
Recommendation
Consider renaming the ‘Parliament and politics’ section to ‘Government and politics’ or move the
‘Complain about a government department’ link to the ‘Consumer
rights’ section.
44 – Task abandoned (Observation)
Almost a quarter of participants gave up on this task.
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/ 32/505/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit
Most said they would go to the department website to find out how to complain about the department.
“Damn.”
—20-29 year old (Pacific Islander) from Auckland
“Don’t know what it would be under. Would probably just Google it.”
—20-29 year old (Pacific Islander) from Manukau
“That was kind of complicated, found that complicated.”
—20-29 year old (Pacific Islander) from Manukau
“I’m lost.”
—20-29 year old (Pacific Islander) from Manukau
Task 8B – Minister of Government Agency
Participants were given the following task: “Find out who the Minister is who is responsible for
[government agency/department].”
Government agencies we asked participants to find out about included IRD, StudyLink, Work and
Income, Ministry of Education, Ministry of Fisheries and Ministry of Justice.
Overall Task 8A received the following scores:
Task success rate:51%
Task directness: 10%
Average time taken: 2m 13 sec
45 – Most common path for Task 8B unsuccessful (Major)
Half of the participants selected the ‘Government and politics’ or ‘Parliament and politics’ category on
the homepage.
Most then selected the ‘Parliament’ and then the ‘Members of Parliament’ option and ended up on the
Parliament website where they didn’t find the information they were looking for
“Don’t think this is the right way.”
—30-49 year old (Asian) from Christchurch
https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and-reports/beta-govt-nz-user-research-round-3/ 33/50You can also read
