Beta.govt.nz - User research (Round 3)

Beta.govt.nz - User research (Round 3)

5/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and -reports/beta-govt-nz-user-research-round-3/ 1/50 New Zealand Government WEB TOOLKIT beta.govt.nz — User research (Round 3) About this report This is a report completed for the beta.govt.nz site as it was in early 2014. Since this research was completed we’ve already made some changes to the beta site, so some of the recommendations in this report have already been addressed.

The bulk of this report was prepared by Optimal Experience who conducted the research on our behalf.

At various places in the report you will see comment boxes titled “What we think” or “What we’re doing”. These comments are the views of the project team not Optimal Experience and have been added to the report. Contents 1. Executive Summary Overview User Testing Process User Findings Summary Key Findings and Recommendations 2. Participants Characteristics Internet Usage NZ Government Websites and newzealand.govt.nz Usage Website Perceptions 3. Scores and Ratings SUS Scores Task Scores 4. Task Related Findings Task 1 – Low Income & Housing Support Task 2 – Jury Duty Task 3 – Consumer Rights

5/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and -reports/beta-govt-nz-user-research-round-3/ 2/50 Task 4 – Contact Phone Number Task 5 – Name Change Task 6 – School Holidays Task 7 – Travel Overseas Task 8A – Complain about Government Agency Task 8B – Minister of Government Agency 5. General Findings Website Purpose and Scope Visual Design Navigation Site Search Homepage Signpost Page – Level 4 Government A-Z page (aka Contact Government) Agency Page Footer Topics page (aka Browse govt.nz) Executive Summary Overview Between February and April 2014, Optimal Experience undertook six sessions of user testing with 30 participants in six different locations throughout New Zealand.

A beta version of the govt.nz website was tested.

This project followed earlier rounds of user testing in October 2013 for the same website. User Testing Process Objectives of the Study The objectives of this project were to find out: What search terms do users use in Google and how does landing, following a Google search result, on a deep content page effect the users’ performance? When and why do users use the local site search, what search terms do they use and what problems do they encounter?

5/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and -reports/beta-govt-nz-user-research-round-3/ 3/50 What pathways do users take when approaching a task? Do users use the ‘Topics’ (aka ‘Browse.govt.nz’) and ‘Contact government’ (aka ‘Government A-Z’) sections and do they understand how they work?

Can users easily find information about Departments and Agencies? Do users have problems with the different information hubs on the site? Can users find contact details easily? Do users notice when content links to other agency websites and are they comfortable with that? What do participants make of the ‘no frills’ design? Study Approach The user tests were carried out in six different locations over a 3-month period: Auckland on Monday, 17th February Manukau on Wednesday, 19th February Porirua on Thursday, 6th March Christchurch on Thursday, 13th March Dunedin on Thursday, 3rd April Nelson on Tuesday, 8th April Participants were given several tasks to complete using the beta.govt.nz website.

Most participants completed all nine tasks while some, due to time constraints completed less tasks. Some tasks were slightly altered between the rounds of testing to test different aspects of a task, but remained essentially the same.

There were also some formatting and layout changes to the website itself during this timeframe. Participants were asked to start some tasks with Google, but otherwise were given the choice between browsing and using the local site search. As part of the testing session, participants were also given two exercises in order to capture their perceptions of NZ government sites in general, and the beta.govt.nz site in particular. Usability Findings Summary The user testing revealed 89 findings which were given a usability rating which indicates the severity of the finding and to which extend this finding will impact on the user experience.

22 positive findings, 2 severe issues that require immediate attention because they prevent users from completing their

5/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and -reports/beta-govt-nz-user-research-round-3/ 4/50 tasks, appear frequently or in crucial situations, 25 major issues that make it difficult to use the site and should be fixed as soon as possible, 16 minor issues that should be fixed if resources are available because they can cause irritation/confusion or occurs rarely, 24 observations which are not problematic but important to note as well.

Each minor, major or severe issue is accompanied by a recommendation, how this issue could be solved.

Key Findings and Recommendations Liked Website purpose. Most participants understood the purpose of the site and liked that it was a one-stop shop for government information. Most participants said that they would like to use it. “The mother ship of all government departments.” —30-49 year old (Asian) from Christchurch “Definitely use it – would go on and learn things even though I didn’t need to use it.” —30-49 year old (NZ European) from Auckland Govt.nz website better than other government websites. Some participants commented that the govt.nz website was easy to use and made them realise how bad some other government websites were.

That was fairly easy, it’s all straightforward. Most government websites are intimidating.” —Female (NZ European) from Auckland Content easy to understand. Participants found information easy to understand and apprehend. “It’s right there. Bullet points, short sentences – done!” —20-29 year old (Pacific Islander) from Manukau Would start with Google and take it from there. Most participants said that they would start looking for information by using Google.

5/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and -reports/beta-govt-nz-user-research-round-3/ 5/50 “Anything I want to know I just Google it, grandchildren tell me to.” —50-65 year old (Māori) from Manukau Younger participants less patient.

Older participants seemed more willing to explore and try different avenues when they couldn’t find information straight away.

Task success not reflected in SUS. A high task success rate did not necessarily result in a high System Usability Score (SUS). For example Christchurch had the highest task success rate (93%) but the lowest SUS (63.5%). Prefer direct contact for help. Most participants said that they would call somebody or the department in question if they weren’t able to find the information online. Participants who got stuck, usually looked for a number to call or a way to contact the responsible department or agency.

It’s sometimes easier to call; it saves you clicking. ... It’s a habit.” —30-49 year old (Pacific Islander) from Porirua Too plain.

Most participants found the website plain and unattractive. Though this didn’t seem to be a problem for most participants; when asked what they would change about the site, a quarter of participants said that it should be more appealing. “Would make it a bit more appealing for the eye – not saying put flowers. It’s not attractive ay. I’d rather have effective and efficient rather than attractive.” —50-65 year old (Pacific Islander) from Auckland “I wouldn’t enjoy looking at that, there’s no pictures, nothing that screams excitement.” —Under 20 year old (Asian) from Porirua Recommendation Consider making the website visually more appealing to improve ‘perceived usability’.

What we’re doing

5/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and -reports/beta-govt-nz-user-research-round-3/ 6/50 We’re already working on the next iteration of the look and feel for Govt.nz which should be live by June this year. Search often unhelpful. Multiple search terms and misspellings usually resulted in a ‘No results’ message or in irrelevant search results.

This doesn’t help me at all.” —50-65 year old (NZ European) from Dunedin Recommendations Make sure the search handles multiple search term queries well and if returning no results, provide suggestions on how to improve the search results.

Cater for common misspellings. What we’re doing We were still working on fixes and enhancements to the site’s search engine when this research was undertaken. We’ve already made some changes to the way the search engine works, it should now handle multiple search terms better than it has previously. We’re also setting up synonyms for common terms and phrases based on actual search keywords used by people visiting beta.govt.nz Government A-Z needs improvement.

Participants often overlooked the ‘Government A-Z’ section on the homepage. On the ‘Government AZ’ page they often overlooked the filter and had to scroll the list up and down in order to find the department or agency they were looking for. Participants tried to but weren’t able to filter by subjects. Recommendations Consider moving the ‘Government A-Z’ section above the fold to the top part of the homepage. Consider enabling users to filter the ‘Government A-Z’ page by subject and not only by department name.

What we’re doing The next version of the homepage moves some of the elements around and should make both the Government A-Z and the Topics page easier to find.

We’re going to track which pathways users follow to measure if the changes have had a positive impact.

5/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and -reports/beta-govt-nz-user-research-round-3/ 7/50 Browsing too tedious. Some participants found the website tedious to navigate and felt that it had too many levels. “A maze of levels.” —50-65 year old (NZ European) from Nelson Recommendation Enable users to complete tasks more quickly, e.g. by providing quicker access to the footer, making search more reliable and making the ‘Topics’ page layout another layout option for the homepage.

Missing content by not scrolling.

Some participants missed important information on the Homepage like ‘Government A-Z’ or on other pages because they didn’t scroll. “I didn’t scroll down. Usually there is a bunch of un-useful information at the bottom, it would need to be at the top. I didn’t even see that, I scrolled past it like 3 times.” —30-49 year old (NZ European) from Nelson Recommendations Consider including page elements that stretch over more than one page length. Consider removing horizontal lines that reach across the whole page width. Participants Characteristics For this round of user testing we tested 30 participants in total in six different locations, i.e.

Auckland, Manukau, Porirua, Christchurch, Dunedin and Nelson.

5/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and -reports/beta-govt-nz-user-research-round-3/ 8/50 There was an even split between genders and a good spread across the different age groups. Based on the most recent census data from Statistics New Zealand, we selected participants from different ethnic backgrounds including NZ European, Pacific Islander, Māori and Asian.

5/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and -reports/beta-govt-nz-user-research-round-3/ 9/50 Internet Usage There was an even spread across different levels of internet usage by age group and internet usage by ethnicity.

5/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and -reports/beta-govt-nz-user-research-round-3/ 10/50 When asked what things they used the internet for, most participants responded Facebook (73%), Doing research (50%), Emails (37%), TradeMe (33%), And watching or downloading videos, movies or TV (33%). Almost every participant said that they would use Google to find information online. NZ Government Websites and newzealand.govt.nz Usage Most participants had used NZ government websites before (93%) but only a third of participants had heard about newzealand.govt.nz Most thought that the website would have something to do with NZ government or government departments.

I imagine it’s an overall website that covers all the government departments.” —50-65 year old (NZ European) from Nelson

5/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and -reports/beta-govt-nz-user-research-round-3/ 11/50 The NZ government websites most mentioned by participants were: Inland Revenue (43%), Work and Income (30%), Local Council websites (30%), Department of Internal Affairs (23%), New Zealand Transport Agency (17%), StudyLink (17%), Library websites (17%), And public transport websites (17%).

Most participants rarely visited NZ government websites, i.e. once a month or less and quite often it depended on the participant’s needs or the time of year.

They are not compelling, you only use it because you have to.” —50-65 year old (NZ European) from Dunedin Most participants said that they would use Google to find government information and services (80%), some said they would prefer to talk to somebody in person either face-to-face (17%) or over the phone (13%). “I would Google it and hope that something comes up.” —20-29 year old (NZ European) from Christchurch Three quarters of the participants said that they would definitely use the site or use it when they needed to.

Probably go on tonight and use it, found it really interesting.

Lot of things I didn’t know

5/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and -reports/beta-govt-nz-user-research-round-3/ 12/50 about.” —20-29 year old (Pacific Islander) from Manukau “I will certainly put it in my favourites.” —30-49 year old (Asian) from Christchurch “I don’t need it I guess, haven’t really got any problems at the moment.” —Under 20 year old (Asian) from Dunedin “I’m going to bookmark this site and use it all the time.” —50-65 year old (Pacific Islander) from Porirua “I actually like it, it’s really good.” —50-65 year old (NZ European) from Dunedin.

Only 10% said that they wouldn’t use it. “I wasn’t overly impressed – definitely room for improvement.” —50-65 year old (Pacific Islander) from Auckland “Not well integrated, information too much all over the place” —Under 20 year old (Asian) from Porirua “I have to read too much, there was lots of information to decipher first. You would have to get use to the scrolling and knowing the footer and what not.” —30-49 year old (Asian) from Christchurch” Website Perceptions In order to gain an insight of users perceptions of government websites in general and the govt.nz website in particular, we asked participants, in two separate exercises, to sort a list of 35 adjectives into two categories; one that included the ones that applied to government websites or the govt.nz website and one for the adjectives that didn’t apply.

NZ Government websites The words that participants most commonly associated with NZ Government websites were Professional (93%) and Helpful (89%).

5/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and -reports/beta-govt-nz-user-research-round-3/ 13/50 Also high in the rankings were Usable (79%), Organised (79%), Reliable (78%), Convenient (78%), Secure (78%), Trustworthy (75%), and Ordinary (71%).

There was also high agreement that NZ Government websites were NOT Fun (96%), NOT Stimulating (89%), NOT Incomprehensible (85%), NOT Patronising (81%), and NOT Inconsistent (79%). Beta.govt.nz The words that participants most commonly associated with beta.govt.nz were Reliable (74%), Organised (70%) and Trustworthy (70%).

Also high in the rankings were Convenient (68%), Helpful (68%) and Friendly (68%). There was also high agreement that beta.govt.nz was NOT Impersonal (70%), NOT Slow (67%), NOT Hard to Use (61%), and NOT Intimidating (60%). Main perception differences There were some interesting differences between how participants perceived NZ government websites in general and beta.govt.nz in particular: Data collected about participants’ perception of NZ Government and newzealand.govt.nz websites Perception NZ Government websites beta.govt.nz Professional 93% 53% Fun 0% 44% Impersonal 56% 15% NOT Compelling 64% 28% NOT Impressive 69% 27% NOT Incomprehensible 85% 42% NOT Stimulating 89% 33% Scores and Ratings SUS Scores The overall Usability (SUS) score for the govt.nz website was 70.3%, which is higher than

5/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and -reports/beta-govt-nz-user-research-round-3/ 14/50 average. A score of 70.3% indicates that people agreed that the website is easy to use. For example, a score of 50% would mean that respondents neither agreed nor disagreed that the website was usable and a score of 100% would mean that everyone strongly agreed that the website is usable. A score of 68% is considered average. In the book Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics 50 SUS scores were analysed from published usability studies that determined that the mean score was 66% and the median as 69%.

SUS by Location The SUS varied widely between the different locations with the lowest score in Christchurch (63.5%) and the highest score in Dunedin (77.5%).

It is important to note that there is only a modest correlation between the SUS and the users’ task performance. For example in this test, Christchurch had the highest average task success rate (93%) but the lowest SUS (63.5%) while Manukau had the lowest average task success rate (52%) and a higher SUS (70%). SUS by Ethnicity The SUS also varied widely between people with different ethnicity with participants of Asian background giving the lowest score (59.5%) and participants of NZ European background giving the highest score (78.3%).

5/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and -reports/beta-govt-nz-user-research-round-3/ 15/50 SUS by Age Group Looking at SUS by age group, the website scored the highest usability score with 30-49 year olds (73.5%), while under 20 year old gave it the lowest score (65%).

SUS by Participants The SUS score also varied widely between participants with 30% being the lowest score and 97.5% being the highest score given.

More than two thirds of participants gave the website a higher score than 70%.

5/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and -reports/beta-govt-nz-user-research-round-3/ 16/50 Task Scores For all tasks the following measurements were recorded: Task success rate: Shows how many percent of participants managed to complete the set task successfully within the set parameters. Task directness: Indicates how directly participants completed a task by showing how many percent of participants managed to complete the tasks through the shortest and most straightforward route.

For example, 80% directness means 80% of participants managed to complete the task through the most direct route.

Time taken: Shows how long it took participants to complete a task. The times included in this report represent the average time taken across all participants who completed a task. Overall Scores Overall, Task 3 – Consumer Rights was the best performing task with the highest task success as well as directness. Only one out of 24 participants who attempted this task, failed to complete it. Task 8A – Complaining about a government department was the worst performing task. Only 10 out of 26 participants who attempted to complete this task managed to complete it successfully. The tasks with the highest success rates (Task 2, Task 3, Task 5 and Task 6) were also the ones completed the fastest while the least successful task (Task 7) was also the slowest to be completed.

5/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and -reports/beta-govt-nz-user-research-round-3/ 17/50 Task Success by Location Comparing the task success rates between the different locations and with the overall success score shows that Manukau performed the worst and Christchurch the best. Task Success by Ethnicity ‘NZ European’ participants completed the most tasks (80%) and ‘Pacific Islander’ participants the least (66%).

Task Success by Age Group Overall participants over 50 years old (79%), between 20 and 29 year old (75%) and between 30-49 year old (74%) were more successful in completing tasks than participants under 20 years old (61%).

Task related findings Task 1 – Low Income & Housing Support Participants were given the following task: “Imagine you’re not working as many hours as you used to and you’re now having difficulty paying your rent. What would you do? Where would you look for help?”

5/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and -reports/beta-govt-nz-user-research-round-3/ 18/50 Overall Task 1 received the following scores: Task success rate: 87% Task directness: 57% Average time taken: 1m 43 sec 1 – Most common path for Task 1 (Observation) More than half of the participants who attempted this task selected the ‘Help paying rent and housing costs’ in the search results or on the ‘Money, benefits and tax’ page.

Here most participants selected the ‘Check what benefits you might get’ link to go to the Work and Income website. “I suppose it’s like an invitation to receive help, it’s telling me what I might get.” —30-49 year old (Pacific Islander) from Porirua 2 – Google search for Task 1 (Observation) Most participants who used Google for this task used the search term ‘Work and Income’ or ‘rent’. A third of those selected the ‘Don’t have enough income’ search result option because it was a link to the Work and Income site or because it seemed the most fitting one for the participant’s situation. Three said they wouldn’t select the ‘Budgeting and managing your money’ option because it wouldn’t help them when they needed help immediately.

Because it [Don’t have enough income] says what I don’t have.” —Under 20 year old (Asian) from Dunedin 3 – Help from the government (Observation) A third of participants said they would first talk to their Landlord, ask their family for help, get another job or talk to their bank before seeking help from the government. Some even said they wouldn’t go to any government department because of pride or because they needed to provided too much private information to get help.

Most thought that Work and Income would be the right government agency to help with this problem; with a third of the participants wanting to go straight to the Work and Income website.

Though

5/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and -reports/beta-govt-nz-user-research-round-3/ 19/50 participants who were students mentioned that they would go to StudyLink first. “They will want to know everything about you.” —50-65 year old (Pacific Islander) from Porirua “I’d stop here. I’d do everything I could do before I had to give them my information.” —50-65 year old (NZ European) from Dunedin 4 – Homepage category for Task 1 (Positive) All participants who tried to find the answer to this task by going through the homepage either went to ‘Money, benefits and tax’ or ‘Work and jobs’.

That’s more relevant to me.” 30-49 year old (Pacific Islander) from Porirua “I was gonna go for a search but I’m thinking this [Money, benefits and tax] might help.”” 50- 65 year old (NZ European) from Nelson” 5 – Couldn’t find information on external site (Major) Participants who didn’t complete this task usually had trouble finding the information they required. Most said that they would look for a phone number to call the responsible department. “I feel a bit stuck with that.” —30-49 year old (NZ European) from Auckland “Would be helpful to see something like ‘are you struggling to pay rent, buy food’.

Something that acknowledges that you’re in the right place, could be so stressed in that situation and panic.” —30-49 year old (NZ European) from Auckland “I’d probably have to ring someone up – it’s a bit complicated.” —30-49 year old (NZ European) from Auckland Recommendation Consider providing enough context and information on the site before offering a link to an external website, so users don’t feel the need to go to another site.

5/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and -reports/beta-govt-nz-user-research-round-3/ 20/50 Task 2 – Jury Duty Participants were given the following task: “You’ve been selected to be on a jury and you’d like to try to get out of it. Find out if you can do that and why.” Overall Task 2 received the following scores: Task success rate: 93% Task directness: 73% Average time taken: 1m 22 sec 6 – Most common path for Task 2 (Observation) Most participants selected the ‘Crime, law and justice’ category on the homepage and then the ‘Jury Service’ option in the ‘Popular pages’ section.

7 – Search for Task 2 (Observation) Most participants who used Google or the site search for this task used the search term ‘jury service’ or ‘get out of a jury’. Most participants that used Google went straight to the Ministry of Justice website. 8 – Homepage category for Task 2 (Positive) All participants who tried to find the answer to this task by going through the homepage went to ‘Crime, law and justice’. 9 – Information easy to understand (Positive) Participants commented that the information was easy to understand. “It’s just simple how it’s worded.” —20-29 year old (Pacific Islander) from Manukau 10 – Missed information because didn’t scroll (Major) Some participants missed the ‘Read ‘If you can’t do jury service’ section or the ‘Apply to be excused from jury service’ link because they didn’t scroll all the way to the bottom of the ‘Jury Service’ page.

Won’t blame the site, I didn’t read further down.”

5/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and -reports/beta-govt-nz-user-research-round-3/ 21/50 —50-65 (Pacific Islander) from Auckland “Would have been easier if I took my time, scrolled to the bottom of the page – instead of just read the top bits.” —20-29 year old (Pacific Islander) from Manukau Recommendation Consider displaying a page index or page summary at the top of the page with links to key sections. What we think We’ve seen this work pretty well on hub pages that show ‘popular pages’ a featured section and links to other sub categories.

This is probably worth trying. We’ll set it up so that the template inserts the links on the page automatically to save our editors from having to create and change links as content changes. For this particular page we’ve also split it into 2 different pages. The old version of the page was trying to address 2 completely different user needs on a single page – being on a jury, and getting out of jury service.

11 – Information not applicable to user’s situation (Minor) Two participants were unsure if they could be excused from jury service because they were studying and not working. Recommendation Consider including information or references that specifically relate to students. What we’re doing We will put this content back through our review and editing process, information for students might not be the only things that’s missing from the page. 12 – Popular pages too specific (Observation) Four participants didn’t select ‘Jury Service’ from ‘Popular pages’ on the ‘Crime, Law and Justice’ page but the ‘Courts and Jury service’ link further down, because the ‘Jury Service’ link looked too specific.

Maybe that popular pages kinda threw me a bit. ...Not sure why, sometimes you think some of these sites the simplest one isn’t the one.” —50-65 year old (NZ European) from Nelson

5/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and -reports/beta-govt-nz-user-research-round-3/ 22/50 Task 3 – Consumer Rights Participants were given the following task: “You bought a computer from Harvey Smith 18 months ago, but it now keeps crashing. You take it back to the shop but they tell you it’s not covered since it’s more than a year old and want to charge you $800 to fix it. You don’t think that’s fair, is there anything you can do?” Overall Task 3 received the following scores: Task success rate: 97% Task directness:83% Average time taken: 1m 46 sec 13 – Most common path for Task 3 (Observation) Most participants selected the ‘Consumer rights’ category on the homepage and then the ‘Your rights even if you don’t have a warranty’ option in the ‘Popular pages’ section or the ‘Consumer disputes and complaints’ option and then the ‘Your rights even if you don’t have a warranty’ link.

Some also selected the ‘Complain about faulty goods’ or the ‘Your rights when making a customer complaint’ options. 14 – Homepage category for Task 3 (Positive) Most participants who tried to find the answer to this task went straight to ‘Consumer rights’. Some selected ‘Consumer rights’ after going to a different section, i.e. ‘Internet, Media and Communication’, ‘Government and politics’, ‘Crime, Law and Justice’, but realised that they were in the wrong place, after looking at the page.

15 – Information easy to understand (Positive) Most participants found this task very easy to complete and the content easy to understand.

This is easier, a lot easier.” —50-65 year old (Māori) from Manukau “Good, it gave you all the avenues you could follow. And it didn’t take long to find that.”

5/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and -reports/beta-govt-nz-user-research-round-3/ 23/50 —30-49 year old (Māori) from Nelson 16 – Options too similar (Minor) Some participants were unsure which option on the ‘Consumer disputes and complaints’ page was the right one to choose because they sounded quite similar, e.g. ‘Complain about faulty goods’, ‘Your rights even if you don’t have a warranty’ and ‘Your rights when making a consumer complaint’.

I’m playing a guessing game?” —Under 20 year old (Asian) in Porirua Recommendation Consider if all pages are necessary or if they could be combined. Task 4 – Contact Phone Number Participants were given the following task: “You want to contact the [government department]. Find a contact phone.” Most participants were asked to find the phone number for the Ministry of Fisheries. Other government departments included Work and Income and Ministry of Justice.

Overall Task 4 received the following scores: Task success rate:83% Task directness: 25% Average time taken: 1m 57 sec 17 – Unsure about department responsibilities (Observation) Some participants who were asked to find a contact phone number to enquire about fish catch limits were unsure who was responsible for those. Some thought the Department of Conservation was responsible and went to the ‘Environment and climate’ section. 18 – Contact information in footer (Positive) Some participants found the contact information through the footer, either by clicking on the ‘Contact government’, ‘Government A-Z’ or the agency name.

5/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and -reports/beta-govt-nz-user-research-round-3/ 24/50 19 – Unfamiliar with new ministry structure (Observation) Half of the participants didn’t know that Ministry of Fisheries is now part of Ministry for Primary Industries. 20 – Helpful synopsis (Positive) Almost all participants who used the site search to find the Ministry of Fisheries selected ‘Ministry for Primary Industries’ from the search results list.

Though some participants were slightly confused when the search didn’t return ‘Ministry of Fisheries’ as a result the synopsis underneath the heading helped them to understand that the ‘Ministry for Primary Industries’ search result was the right one. “Oh! Found it. Oh okay, now it looks like MPI is covering Fisheries.” —50-65 year old (Pacific Islander) from Porirua 21 – Right panel obvious (Positive) Participants who found the agency page saw the phone number in the right panel. 22 – ‘Hidden’ contact details (Major) A third of participants who attempted this task selected a category on the homepage that seemed related to the agency they were looking for, e.g.

Community, arts and sport’ or ‘Environment and climate’ for ‘Ministry of fisheries’, ‘Crime, law and justice’ for Ministry of Justice or ‘Money, benefits and tax’ for Work and Income.

Most gave up on the second or third level (Information hub Level 2 and 3) because they couldn’t see any contact information on these pages. “I’m lost now.” —30-49 year old (NZ European) from Auckland Recommendation Consider providing prominent links to subject-related contact information on every page, e.g. in the right panel. 23 – Site search for Task 4 (Observation) Most participants who attempted this task by using the site search used the name of the agency, e.g.

5/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and -reports/beta-govt-nz-user-research-round-3/ 25/50 ‘Ministry of fisheries’ or the subject about which they wanted to contact the agency, e.g.

fishing limits’. Task 5 – Name Change Participants were given the following task: “Imagine that a female friend of yours has changed her name/has got married and has changed her name and now needs to change the details on her driver license. Find out how to do this.” Overall Task 5 received the following scores: Task success rate:97% Task directness: 64% Average time taken: 1m 13 sec 24 – Most common path for Task 5 (Positive) Almost two thirds of participants who attempted this task selected ‘Driving and transport’ on the homepage and then the ‘Driver licences’ and ‘Change your name on your driver licence’ links.

All participants who followed this path found the right information.

That was easier than the last few – saw driver licence on the homepage.” —20-29 year old (Pacific Islander) from Manukau 25 – Browsing by context (Observation) Almost half of the participants selected the ‘Births, deaths and marriages’ category from the homepage because they saw changing your name in the context of marriage. 26 – Obvious link on the ‘Change name’ page (Positive) Most participants who went to the ‘Change name’ page saw and selected the ‘Change name on driver’s license’ link.

I thought that was quite easy. I don’t usually find things that easily.” —20-29 year old (NZ European) from Christchurch “That was quick.

I had a legalised name change in my mind, but I took the easy route. I’ll change my name to Phil Dotcom; I’ll earn millions.” —50-65 year old (NZ European) from Christchurch

5/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and -reports/beta-govt-nz-user-research-round-3/ 26/50 27 – Information easy to understand (Positive) Most participants found this task very easy to complete and the content easy to understand. “It clearly says what you need to change the name.” —20-29 year old (Pacific Islander) from Manukau “Simple. Had to be simple or I wouldn’t have got there.” —50-65 year old (NZ European) from Dunedin 28 – Confusing terminology (Minor) Three participants were confused by the term ‘replace’ in the ‘application for a replacement driver licence’ link.

They didn’t seem to see the heading ‘If you’re changing your name, you have to replace your licence’ because they were concentrating on the link. Recommendation Consider providing an explanation close to the ‘application for a replacement driver licence’ link that explains why users need to replace the driver licence. Task 6 – School Holidays Participants were given the following task: “You’ve just seen a really good holiday deal online for 5 days on the Gold Coast. You want to book it as it will be gone soon but need to check if your kids will be in school. Where would you find this information?” or “Find out when the next school holidays start.” Overall Task 6 received the following scores: Task success rate:91% Task directness: 77% Average time taken: 1m 14 sec 29 – Most common path for Task 6 (Positive) Three quarters of all participants who attempted this task selected the ‘Education and training’ category

5/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and -reports/beta-govt-nz-user-research-round-3/ 27/50 on the homepage and the ‘School terms 2014’ option either in the ‘Popular pages’ or the ‘Schools and colleges’ section straight away. “Really easy, couple of clicks away and right there. Might even have it set as my homepage/bookmark. Something I’d refer to quite a bit.” —Female (NZ European) from Auckland “Oh, that was quick. That was just so easy.” —50-65 year old (Pacific Islander) from Porirua “So far I can tell you it’s very, very easy, and the pages complimented each other.” —50-65 year old (NZ European) from Christchurch” 30 – Synopsis not applicable to user’s situation (Minor) Three participants hesitated to select the ‘Education and training’ category on the homepage because they synopsis underneath the heading didn’t cover their individual situation, e.g.

kids in primary school.

Recommendation Consider providing a more detailed synopsis on mouse over. What we think This is a real challenge. We have to balance the need to present enough information to users to help them choose the pathways through the site, but we need to do it in the most accessible way. Showing additional information in a tooltip that displays on mouse over won’t work for users who rely on keyboards or other devices to navigate the web, and there will be inconsistencies in the way assistive technologies interpret and present the information. We think the first place to start is to review the summaries under each category and see if we can improve them.

Feedback from users here is really welcome.

Task 7 – Travel Overseas Participants were given the following task: “(You’re getting ready to go on Holiday to the Gold Coast with your family.) You receive accommodation supplement (and childcare subsidy). You’re only going away for a week. Do you have to tell anyone in Government you’re going?”

5/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and -reports/beta-govt-nz-user-research-round-3/ 28/50 Overall Task 7 received the following scores: Task success rate: 33% Task directness: 21% Average time taken: 1m 42 sec 31 – Correct path for task 7 ignored (Major) Only a quarter of participants who attempted this task went to the ‘Before you travel’ page in the ‘Passport, travel and tourism’ section.

Some participants said that they wouldn’t go to ‘Passports, travel and tourism’ because the synopsis didn’t say anything about benefits. “I’m going to look in the benefits [section] – not thinking holidaying and passports. It should have been under the benefits section.” —20-29 year old (Pacific Islander) from Manukau “Yeaaah, it’s a bit out of left field. Na, I wouldn’t have thought it would have to do with the benefit.”” 50-65 year old (NZ European) from Nelson Recommendation Provide a link to the ‘Before you go’ page from the ‘Money, benefits and tax’ section. What we’re doing The information people needed was actually already in the ‘Money, benefits and tax’ section but we think it might not have been prominent enough.

We’ll put some of the pages in this section back through our content review process and see if we can improve them. We might need to split some pages where they are currently trying to address too many user needs in one place. 32 – Found required information (Observation) Most participants who went to the ‘Before you travel’ page found the information about benefits. 33 – Search by benefit type unsuccessful (Major) Most participants were unsure where to get started and used the site search searching by benefit type, e.g. accommodation supplement but didn’t get any or only irrelevant results.

Recommendation

5/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and -reports/beta-govt-nz-user-research-round-3/ 29/50 Improve search. What we think We’ve kept the content on the beta site pretty thin, and this has meant we don’t actually list the specific benefits and payments people may be able to get from agencies like Work and Income. Instead, we linked users to the ‘Find out what you might get’ tool but this means we’ve ignored that group of users that has some familiarity with what benefits are available, or at least what they used to be called.

We’ll take another look at the content affected by this and make some improvements.

34 – Concentrating on the benefits aspect (Major) Participants who approached this task from a benefit and not a travel point of view, usually didn’t manage to complete this task successfully. A third of participants selected the ‘Money, benefits and tax’ category on the homepage. One participant who used the search term ‘travel overseas on benefit’ didn’t select the ‘Before you travel’ search result but went to ‘Benefits from Work and Income’ page. “I was focusing on benefits, not in terms of travel.” —50-65 (NZ European) from Dunedin Recommendation Ensure that users can complete this task when approaching it from the ‘benefit’ angle.

Consider including a link to the ‘Before you travel’ page in the ‘Money, benefits and tax’ section. 35 – Task not completed or abandoned (Observation) Four participants gave up on this task.

Most participants who didn’t complete the task said they would call the responsible department, e.g. Work and Income. “It’s just too hard – if I was at home I’d cry, or call someone.” —20-29 year old (Pacific Islander) from Manukau “Usually I jump on the phone and use their 0800 number.” —50-65 year old (Pacific Islander) from Auckland

5/23/2014 beta.govt.nz — User research (Round 3) | Case studies and reports | Design and development | New Zealand Government Web Toolkit https://webtoolkit.govt.nz/guidance/design-and-development/case-studies-and -reports/beta-govt-nz-user-research-round-3/ 30/50 36 – Misleading homepage category (Minor) Two participants selected ‘Work and Jobs’ on the homepage because they were looking for information from Work and Income and thought Work and Income would be under there.

Recommendation Consider renaming this category to ‘Jobs and work’. What we think Having seen users work through this task, it’s now really easy to see why some people would confuse ‘Work and Income’ the department, with ‘Work and jobs’ the topic. We will make this change before the beta period finishes and the site goes fully live.

Task 8A – Complain about Government Agency Participants were given the following task: “You have been dealing with [government agency]. It’s been a nightmare and you’re really unhappy. You already complained to the responsible government agency/department without success. Find out if there is another way you can complain about this agency/department or an impartial party to complain to.” Government agencies we asked participants to complain about included IRD, ACC, StudyLink and Work and Income.

Overall Task 8A received the following scores: Task success rate: 34% Task directness: 10% Average time taken: 3m 01 sec 37 – Most common path for Task 8A unsuccessful (Major) More than half of the participants expected to find the answer to this task under ‘Consumer rights’ but couldn’t find it there.

Some participants went to this section because it included the term ‘complaints’ in the synopsis under the heading. Recommendation

You can also read
Next part ... Cancel