Comparative Evaluation: Anime Streaming Websites Crunchyroll and Funimation By Sarah Letham
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Comparative Evaluation: Anime Streaming Websites
Crunchyroll and Funimation
By
Sarah Letham
in Partial Fulfillment of the Requirements for the Degree of
Masters of Science in Human Factors in Information Design
at
Bentley University
For course HF751
Instructor: Bill AlbertExecutive Summary
Method:
A comparative evaluation was performed between anime streaming websites Crunchyroll and Funimation.
The two primary areas of interest were the usability of the video streaming service navigation and the usability
of account signup. Users performed six tasks, four of which were oriented towards the video streaming
navigation, and two of which were oriented towards account signup:
1. Find the latest episode of a show that is on the front page which can be viewed completely for free.
2. Navigate to the full episode list of the front page show.
3. Go to the first episode of a show that is not on the front page.
4. Navigate to second episode of the previous task’s show’s first episode page.
5. Go to where you would sign up for an account.
6. Where would you go to find out what paid account benefits are?
Users were asked a variety of demographic questions to determine their streaming habits, anime viewing
habits, and general commitment to being an anime fan. The sample contained a mix of non-fans, novice fans,
moderate fans, and ardent fans, with a slight tendency towards lesser fan commitment.
Most users had experience with general streaming sites Netflix and Hulu, and 52% of participants watch anime
via online streaming. A third of users use Crunchyroll, half of whom have an account, while about ten percent
use Funimation. None had a Funimation account.
Results:
Crunchyroll’s primary advantages are in episode navigation, facilitating faster task times, and overall
usability/ease of use as measured by SUS.
Crunchyroll is at a severe disadvantage in teaching users about account benefits, and has generally poor
performance indicating to users the latest episode that can be watched without an account.
Crunchyroll had good performance but still has minor disadvantages in regards to account signup and helping
users find shows that are not promoted on the front page.
2
Page
Sarah Letham (818) 667-5316 sarah.blass.letham@gmail.comBackground and Methods
This study is a comparative evaluation of anime streaming websites Crunchyroll.com and Funimation.com.
Crunchyroll and Funimation are industry leaders in providing streaming anime content, particularly in offering
subtitled versions of episodes on the same day they air in Japan, a service that sets them apart from
competitors like Netflix.
The purpose of the study was to compare the usability of the two streaming sites on key aspects of streaming
content and account creation. The two primary research questions were:
1. How usable is the navigation for the video streaming service on each site?
Video streaming services are the primary value proposition of the site, so it is important that these are
usable. I also sought to compare the usability of navigating the streaming of promoted, front-page
shows and the usability of other non-promoted library titles.
2. How usable is the account signup page for each site, and do users understand the benefits of the paid
account?
Paid accounts are a primary source of revenue on these sites, so it’s important that the account signup
be usable and that users understand what they are getting when they sign up. Difficulty signing up
leads to abandonment, and mismatches between expectations and actual services leads to
cancellations.
I also aimed to answer whether users found the sites easy or difficult to use overall. After each task, the user
rated the ease of use on a seven-point Likert scale. At the end of the study, the user was given a ten-question
SUS questionnaire, modified for ease of use to be all positive.
This study was a between subjects design in which participants completed six tasks on either the Crunchyroll
site or Funimation site. The six tasks were identical between the sites, and shown in random order:
1. Find the latest episode of a show that is on the front page which can be viewed completely for free.
This task was chosen because typically streaming of episodes on the same day they air in Japan is
offered only as a premium service. After a period of time (typically about a week) the episode
becomes available for free. I wanted to test and see if new potential users could find the latest
episode that they are able to watch without first signing up.
2. Navigate to the full episode list of the front page show.
This task was chosen for its relevance to any user who isn’t caught up to the latest episode of a
show displayed on the front page of the website. The ability to navigate to whatever episode the
viewer is currently caught up to is important to the viewing experience.
3. Go to the first episode of a show that is not on the front page.
The front page has an extremely limited selection of the streaming website’s library, so another
vital service for a streaming site is facilitating navigation to titles that aren’t on the front page.
4. Navigate to second episode of the previous task’s show’s first episode page.
Navigation between episodes is as important as navigation between shows. This task was chosen
to test inter-episode navigation.
3
5. Go to where you would sign up for an account and enter information. You do not need to submit any
Page
information or actually sign up.
Sarah Letham (818) 667-5316 sarah.blass.letham@gmail.comThis task is to test the findability of the signup page. Account signup is a necessary part of
conversion.
6. Where would you go to find out what paid account benefits are?
This task and its accompanying pre- and post- test questionnaire were utilized to test how well
Crunchyroll and Funimation communicate their respective account value propositions.
The front page show for Task 1 of each site was chosen for its likelihood of remaining on the front page
throughout the test. As such, long-running popular shows were chosen. For Crunchyroll, Naruto Shippuden
was used, while for Funimation, Fairy Tail was chosen. However, it was unanticipated that Funimation would
completely rotate its front page shows on a regular basis even with the long-running Fairy Tail airing. As a
result, the front page show for Funimation had to be changed three times, since it was more important for the
basic task structure to stay intact than for the show to stay consistent. Five users completed task 1 on
Funimation for Fairy Tail, five completed for Puchim@s, and one completed for Jormungand. This is also
included in the “Limitations” section later in the report.
The other title was chosen to be a classic that anime fans would likely be familiar with, and which also would
not likely be removed from the library during the test. Unlike the front page show for Funimation, neither of
the non-front-page shows had to be changed during the test. Crunchyroll’s program for task 3 and 4 was
Puella Magi Madoka Magica, and Funimation’s was Spice and Wolf.
The following metrics were used to evaluate the data:
Task success
o Binary – success/failure
o Analysis of failure types, such as aborting task vs incorrect completion
Time on task in seconds
General ease of use
o 7-point Likert scale rating ease of use after each question
o 10-question SUS at the end
Qualitative information about users’ knowledge and expectations about accounts/ the site, as well as
anime habits
Qualitative analysis of how people navigate through site
o Do they typically employ the search bar? Go through episode lists? Navigate through show
titles?
Participants were a collection of friends, family, classmates, and friends-of-friends. Recruiting was done
through e-mail and Facebook contacts. The subjects’ familiarity with anime ranged from complete novices to
hardcore fans. As an incentive, a raffle for small anime-related prizes was done.
Task success data was collected from 22 participants, 11 using Crunchyroll and 11 using Funimation. Due to a
technical error all other data was lost for one participant. Therefore, outside of task success, all data was
drawn from 21 participants.
4
Page
Sarah Letham (818) 667-5316 sarah.blass.letham@gmail.comUser Demographics:
The user group was a mix of non-fans of anime, novice fans, moderate fans, and a few serious fans.
Demographic questions were asked to determine baseline viewing habits of users, particularly their familiarity
with video streaming and with anime. They were first asked what streaming sites they used, including
Crunchyroll, Funimation, and a sample of competitor sites. Participants were then asked to state which of
these sites they had an account for.
Users were asked how often they watch anime, whether they stream anime, and how they watch anime most
often. Users were also asked how many titles they own. The answer tiers, chosen based on anecdotal data,
represent four groups: those on the sidelines who just buy the a few classics, moderate fans who own a small
collection of favorites, serious fans who have a collection but don’t buy a lot of titles, and hardcore purchasers
who regularly purchase new content.
The following is a breakdown of their anime and streaming habits:
Streaming Sites - Usage and Accounts (in Percent)
100
90
90
Percent of Users Polled
80
71
70 62
60
Select all sites that
50 43 43 you use
40
33 29 Select all sites you
30
19 19 have an account for
20 10 14 10
10 0 0 0 0
0
Streaming Site
Most users had experience with general streaming sites Netflix and Hulu, with slightly less than half using
Amazon Video. 33% of users use Crunchyroll and a little over half have an account. Ten percent of
participants use Funimation’s website but none have an account. 19% of users have no streaming site account.
Another question found that 52% of participants watch anime via online streaming (see below).
5
Page
Sarah Letham (818) 667-5316 sarah.blass.letham@gmail.comResponses to: "How do you typically watch anime?"
100
90
Percent of Uses Polled
80
70
60 52%
50 43%
40 29%
30 19%
20 14% 14%
10
0
Participants were asked how often they watched anime and how many titles they own in order to establish
whether our sample consisted primarily of novice, moderate, and hardcore anime fans.
Responses to: "How often do you watch anime?"
100
90
80
Percent of Users Polled
70
60
50
38%
40
29%
30
19%
20 10%
10 5%
0
Never Almost Sometimes Often All the Time
Never
Responses to these questions indicated that the sample was mixed. Those who responded that they never
watch anime were counted as non-fans, those who watch almost never were counted as novice fans, those
who watch “sometimes” were counted as moderate fans, and those who watch “often” or “all the time” were
counted as hardcore fans. The sample had a good mix with a slight tendency towards lesser fan commitment.
6
Page
Sarah Letham (818) 667-5316 sarah.blass.letham@gmail.comSimilar results were found when we asked users how many titles they owned. More than half owned fewer
than five titles, indicating that they are likely non-fans or novice fans. The remaining participants were divided
between those who owned more titles.
Responses to: "How many anime series
titles do you own?"
More
than 20
14%
11-20
14%
Fewer than 5
57%
5-10
15%
Findings
The following is a summary of all results:
Task success (binary success and failure)
o Overall, the two sites had similar task completion rates
o Compared to Funimation, Crunchyroll is at a disadvantage in two places: teaching users about
paid account benefits, and helping users find shows that aren’t being promoted on the front
page
o Performance was very good on both sites for account signup and finding full episode lists.
o Both sites performed poorly on task time for finding the latest free episode of a show
promoted on the front page
Analysis of failure types
o Four of six Funimation tasks included timing out among non-successes, compared to one for
Crunchyroll
o Only Crunchyroll tasks had non-successes caused by abandonment
o On both sites, the majority of non-successes are subject errors (false-positive successes)
Time on task in seconds
o In all four video streaming navigation tasks, subjects using Funimation required significantly
longer times to complete tasks than those using Crunchyroll
o Funimation had the same success as Crunchyroll for Account Signup, but was significantly
7
faster on this one task
Page
Sarah Letham (818) 667-5316 sarah.blass.letham@gmail.como Crunchyroll clearly has an advantage in facilitating task speed in the navigation of episodes,
but Funimation’s performance is equal or superior on account signup and finding account
information
Click paths and page views
o 100% of people on Funimation’s website clicked the front page “Join” or “Subscribe” button
for Task 5 (Account Signup)
o Users in Task 6 followed a nearly identical ratio of paths between websites, which was
facilitated by Funimation’s design but not Crunchyroll’s
Users’ knowledge and expectations about paid account benefits
oCrunchyroll’s most well-known post-task feature was High Definition Streaming (70%)
oIncreased Mobile Access jumped from 10% to 40% (the largest increase) between pre and post
test
o Knowledge of store discounts increased from zero responses to 20%
General ease of use, as measured by a 7-point Likert scale rating ease of use after each question and a
10-question SUS questionnaire given at the end of the test
o Crunchyroll’s SUS score is 64.75
o This score places Crunchyroll at just below the 50th percentile for SUS
o Crunchyroll’s SUS score is much higher than Funimation’s score, which is just under 25 th
percentile
o Funimation’s account signup had the highest individual task rating
Task Success:
Rate of Task Success in Percent
Percent of Users who Successfully Completed Task
(Bars represent 90% Confidence Interval)
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Find the Latest Find Full Find Episode 1 - Find Episode 2 Account Signup Find Account
Free Episode - Episode List - Non-Front Page from Episode 1 Benefits Info
Front Front
8
Page
Crunchyroll Funimation
Sarah Letham (818) 667-5316 sarah.blass.letham@gmail.comOverall results of the comparison show that the two sites had similar task completion rates. Five of the six tasks did not have significantly different results, with alpha at 0.1. However, two tasks had nearly significant results, with p values under 0.2. Task 6 (Find Account Benefits Information) was the only task with a truly significant difference, p
Failure Types:
The superior task success rate of Funimation that was described above does not tell the whole story. It is clear
from the chart that Crunchyroll has many more instances of non-successes caused by “Error,” indicating that
the user thought they were successful even though they had not actually completed the task.
Type of Non-Success by Task
Error Abandon Timeout
33% 25% 29%
40% 33%
56%
100% 100% 33% 100% 100% 100% 100%
67% 75% 71%
60%
44%
33%
Crunchyroll did not have any non-successes due to timing out with the exception of Task 4 (Navigate to
Episode 2 from Episode 1 of a non-front-page show). Meanwhile, four out of six Funimation tasks included
timing out among the non-successes, typically at an equal or higher rate than Crunchyroll’s Task 4. However,
only Crunchyroll tasks had non-successes caused by abandonment.
Statistically significant differences were found for Task 1 and Task 6 when comparing the makeup of failure
types between Funimation and Crunchyroll (Task 1 p=.04, Task 6 p=.03). The other tasks had more similar
rates of non-success causes, especially “Error” non-successes, which made up the overall majority of non-
successes. There was a clear effect overall of non-success type (pTime on Task:
Mean Task Time (in seconds)
180
160
140
Task Time in Seconds
120
100
80
60
40
20
0
Significant differences were found in mean task time between Crunchyroll and Funimation for Task 1 (Find the
Latest Free Episode of a Front Page Show) (pThese data support the earlier suggestions that Crunchyroll should focus its energies on helping users navigate
front page episodes and increasing access to information about account benefits.
Click Paths and Page Views:
Videos of participant performance were examined to find out where users go first in trying to navigate their
task. The majority of users clicked on a link somewhere on the front page (typically above the fold) such as
“Shows” or an episode title. The next most common option was to utilize the search bar. The other options
were divided between a catch-all Other category (used to denote clicking on, for example, DVD ads or
navigating out of the site, or going down a path that did not fit into any other category) and between links
from the bottom navigation bar.
First Click in Task by Percent of Users
100%
90% 18% 13% 18%
25% 27% 29% 25% 25% 27%
80%
Percent of Users
50% 45%
70% 38% 13%
60% 38%
50% 43% 9% 100%
40% 82% 73% 82% 75% 73%
30% 63%
20% 50% 50% 45%
38% 29%
10%
0%
Front Page Link* Search Bar Bottom Bar Other
It is noteworthy that 100% of people on Funimation’s website clicked the front page “Join” or “Subscribe”
button for Task 5 (account signup), which likely contributed to its extremely low task time and high success
rate.
It is also noteworthy that users in Task 6 followed a nearly identical ratio of paths between websites. Since
Funimation had a significantly higher task success rate on this task, it is clear that Funimation’s design is
congruent with the task paths chosen by users, while Crunchyroll’s design is not. Crunchyroll should update its
design so that paths to account information are easier to find with intuitive paths.
12
Page
Sarah Letham (818) 667-5316 sarah.blass.letham@gmail.comMean Unique Page Views and Clicks
(Bars represent 90% Confidence Interval)
Mean Unique Pageviews Mean Number of Clicks
18
16
14
12
10
Mean Views
8
6
4
2
0
The average number of unique page views and average number of clicks were examined to find differences
between the sites, especially to be compared with time on task. However, few significant or differences were
found among these data. The lowest scores, on Crunchyroll for finding the full episode list of a front page
show and finding the first episode of a non-front page show, coincide perfectly with the highest proportions of
using the search bar on Crunchyroll. Use of the search bar on Funimation did not decrease the number of
clicks nearly as much, which is unsurprising considering that Funimation’s search function involves navigating
to a separate search page rather than using a drop-down selection like on Crunchyroll.
Account Benefit Understanding:
Task 6 was used to determine what users knew about paid account benefits. For each site, users were
presented with a pre-test and post-test. The list given was the same for both shows.
Crunchyroll’s questionnaire contained six actual account benefits (No Commercials, High Definition Streaming,
See Episodes Earlier, Store Discounts, Access to more shows/titles, Increased Mobile Access) and four false
account benefits (Watch Dubbed Episodes, Watch Uncut Mature Shows, Episode Upload Notification, Episode
Downloads). Ten users were asked to identify the paid account benefits of Crunchyroll.
Funimation’s questionnaire contained eight actual account benefits and two false account benefits. Actual
account benefits were No Commercials, High Definition Streaming, See Episodes Earlier, Store Discounts,
Access to more shows/titles, Increased Mobile Access, Watch Dubbed Episodes, and Watch Uncut Mature
13
Shows. Episode Upload Notification and Episode Downloads are still false account benefits for Funimation.
Eleven users were asked to identify the paid account benefits of Funimation.
Page
Sarah Letham (818) 667-5316 sarah.blass.letham@gmail.comCrunchyroll:
Crunchyroll Pre-Test results
In the pre-test before finding the information page, half of users identified high definition streaming and
seeing episodes earlier as account benefits. 40% of users answered that no commercials and greater access to
shows were account benefits. The “Other” category results consisted of comments that the user was
unfamiliar with the site and so did not know the answers.
One user mentioned increased mobile access. No users were aware that Crunchyroll offers store discounts as
an account benefit.
Episode Downloads was the most common false positive, with 30% of users identifying this service as an
account feature. Episode upload notification and watching dubbed episodes were picked by one participant
each.
14
Page
Crunchyroll Post-Test results
Sarah Letham (818) 667-5316 sarah.blass.letham@gmail.comPerformance was overall better on the post-test questionnaire, which users took after completing the task. It
is noteworthy that only 40% of users successfully completed task 6, so it is unsurprising that many users’
accuracy did not improve.
The most well-known feature after the post-test questionnaire was High Definition Streaming, which was
identified by 70% of the participants, followed by Access to More Shows/Titles, which was identified by 60%.
No Commercials increased from 40% to 50%, and Increased Mobile Access jumped from 10% to 40% (the
largest increase). Customers learned about store discounts, increasing from zero responses to 20%.
False positives also increased, indicating that the results are not entirely due to the information from the task.
Watch Uncut Mature Shows increased from 0% to 10%, Episode Upload Notification increased from 10% to
20%. However, Episode Downloads did decrease from 30% to 20%, and Watch Dubbed Episodes remained at
10%, so the increase was not across the board.
Oddly, the actual account feature See Episodes Earlier decreased in response rate from 50% to 40%, but this
may be a result of the term used. In the “Other” comments one user mentioned Simulcast as another benefit,
which was intended by the researcher to be covered by “See episodes earlier.” Had the word “simulcast” been
used, it is possible that more users would have recognized the feature.
Additional answers in the “Other” category indicate that users did not have enough time to read and commit
to memory the necessary information for the questionnaire. This is discussed further in the “Limitations”
section below.
Funimation:
Funimation Pre-Test Results
The feature with the highest response rate was No Commercials, with 64%. This is followed by Access to More
Shows/Titles, with 45% response rate. Third place is tied at 27% between See Episodes Earlier, Store Discounts,
15
and the false positive answer Episode Upload Notification.
Page
Sarah Letham (818) 667-5316 sarah.blass.letham@gmail.comOnly 18% of users identified High Definition Streaming as an account benefit, the same number as thoe who
erroneously identified Episode Downloads as a feature. Only 9% of users identified Watch Dubbed Episodes,
Watch Uncut Mature Shows, and Increased Mobile Access.
Funimation Post-Test Results
Overall performance was improved by completing Task 6, even more than for Crunchyroll. No Commercials
held its place with 64% response rate, while Access to More Shows/Titles increased from 45% to 55%. High
Definition Streaming had the greatest increase, going from 18% to 45% response rate. See Episodes Earlier
also increased, from 27% to 36%. Watch Dubbed Episodes, Watch Uncut Mature Shows, and Increased Mobile
Access all increased from 9% to 18%.
The only account feature for which response rate decreased was Store Discounts, which decreased from 27%
to 9%.
The false positives for Episode Upload Navigation decreased from 27% to zero. Episode Downloads stayed the
same, with 18% response rate. There was no increase in false positives.
While improvement differed between Funimation and Crunchyroll, aspects of the task implementation may
have influenced the results in such a way that prevents drawing conclusions from these differences. The
Limitations section discusses these possible influences in further detail.
Post Task Ratings and Overall SUS:
At the conclusion of the tasks, participants were presented with a ten question SUS questionnaire measuring
overall satisfaction with the site. A SUS questionnaire with all positive wording was used in order to prevent
16
confusion on the part of the participant.
Page
Sarah Letham (818) 667-5316 sarah.blass.letham@gmail.comThe SUS score for Crunchyroll, found by averaging scores for each participant, was 64.75. This score places
Crunchyroll at just below the 50th percentile for SUS, indicating below average performance. It should be
noted that if one possible outlier score of 15 is removed, the mean score increases to 70.28, which is a little
over 50th percentile.
The SUS score for Funimation was 54.77. This score places Funimation at just below the 25 th percentile for SUS,
indicating poor performance. If one possible outlier score of 12.5 is removed, the mean score increases to 59,
which is over the 25th percentile but still far under the 50th.
Crunchyroll’s SUS score is quite a bit higher than Funimation’s score, but Crunchyroll should still aim to
improve its SUS score so that customer satisfaction better compares with other websites.
Post-task ratings measured on a seven-point Likert scale were also compared. Satisfaction ratings were
measured for those who were successful, those who were unsuccessful, and then the two ratings were
combined:
Ease of Use Rating by Task and Performance
6
Mean Ease of Use Rating (0-6)
5
Mean Ease Rating
4 (Successful)
3 Mean Ease Rating
(Unsuccessful)
2
Mean Ease Rating (All)
1
0
Unsurprisingly, Funimation’s account signup had the highest rating. Interestingly, successful users rated Task 2
on Crunchyroll very highly, but the 9% of unsuccessful users gave Task 2 the lowest rating, zero. This is not a
major concern since 91% of users were successful on this task and they rated the task as very easy, but it is
worth noting cases like Task 2 where unsuccessful users found the task very hard and successful participants
found it very easy. In most other cases, ratings were similar across the board.
17
Page
Sarah Letham (818) 667-5316 sarah.blass.letham@gmail.comLimitations and Possible Extensions
Due to changes in the Funimation front page, the title chosen for the first and second task on
Funimation had to be rotated during the test in order to keep the task consistent. The following shows
were used:
o 1 complete: Jormungand
o 5 completes: Puchim@s
o 5 completes: Fairy Tail
Future studies could avert this by timing the study to have one show throughout, changing the task so that it
did not involve the front page show, or working directly with Funimation to make sure one show remains on
the site throughout the test.
Crunchyroll title Puella Magi Madoka Magica is more complex than Spice and Wolf, so it is easier to
utilize the search function for Funimation’s title than Crunchyroll’s title. However, the task with title
was visible at all times during the test, so it would be possible for users hoping to utilize the search
function to copy and paste Puella Magi Madoka Magica. It is still possible that the longer title was
more difficult to work with, so future studies should use titles of similar length, word familiarity, and
word complexity in order to eliminate this potential confounding factor.
The Task 6 questionnaire was identical for both shows, meaning that there were four false options for
Crunchyroll and two false options for Funimation. As such, the Funimation test had an unfair
advantage against false positives. As a result, the questionnaire performance for Task 6 was not
directly compared between sites.
Use of non-randomly chosen convenience samples was also a limitation, as was the limited number of
participants. Future studies should use a larger sample of randomly selected representative
participants.
For task 6, not enough time was left for users to read the page. In addition, it was not made clear
enough that users needed to review the information on the site. The reason is that for all other tasks,
immediate rerouting to the questionnaire is the sign that the task is complete, and I didn’t want
participants to navigate away from the correct page assuming it was incorrect. In future studies,
subjects should be given enough time to read the page, and they should be clearly instructed about
the learning aspect and that this task’s success notification would be different.
It is the researcher’s first time using User Zoom, so there was a certain amount of trial and error in the
setting up of the test and in the analysis of results. Some analysis of results was made more difficult
due to the setup of UserZoom; for example, the tasks were easily divided into two groups, but
questionnaire results were not. The details of the study plan, setup, and implementation might be
better planned and executed by a researcher more familiar with User Zoom.
18
Page
Sarah Letham (818) 667-5316 sarah.blass.letham@gmail.com Besides task success results, data was lost for one participant in the Crunchyroll Group who did not
officially finish the final question in User Zoom. As a result, that user was not counted by UserZoom as
a completed participant, so the results were discarded.
Additional opportunities for future research exist in studying why users are more likely to spend time
attempting to finish Funimation tasks but more likely to abandon Crunchyroll tasks (or erroneously
mark them as successful).
Conclusions
Crunchyroll’s strongest areas are:
Episode navigation, including inter-episode navigation and finding full episode lists
Facilitating faster task times across tasks, particularly those involving navigation
Overall usability and ease of use as rated by SUS when compared to Funimation
Crunchyroll’s weakest areas are:
Informing users about account benefits
o Users in Task 6 followed a nearly identical navigation strategy regardless of website. This
navigation strategy was facilitated by Funimation’s design but not Crunchyroll’s.
o Users did not know at all about store discounts being a benefit of Crunchyroll’s paid account, and
on the Task 6 post-task questionnaire the responses increased only to 20%. These scores fell
below the otherwise typical 40-70% range of users knowing about the account benefits.
Helping users identify the most recent episode that can be watched without an account
o It is not clear whether this is due to users not being able to distinguish what episodes are
restricted to premium viewers, due to being unable to find the latest episodes in general, or if
there is another cause. Further investigation is warranted.
Other areas to improve:
These are strengths of the competition where Crunchyroll is relatively less effective:
Account signup
o 100% of people on Funimation’s website clicked the front page “Join” or “Subscribe” button for
Account signup. For comparison, 63% of users clicked a front page direct link to sign up for an
account.
Facilitating navigation to shows that are not promoted on the front page
o Recall that 91% of Funimation users successfully located the first episode of the non-front-page
target program, while only 60% of Crunchyroll users could locate the episode, p=0.13
Additional Recommendations for Crunchyroll:
19
Try to further improve usability and ease of use in order to better compare to other websites.
Page
Crunchyroll’s score is higher than Funimation’s, but is still lower than 50% of the websites tested.
Sarah Letham (818) 667-5316 sarah.blass.letham@gmail.comCrunchyroll should aim to not only beat the SUS scores of their direct competition, but even peripheral
competitors such as general streaming sites like Netflix. If Crunchyroll can aim high enough to take its
navigation to the next level and provide a seamless user experience, the result should be increases in both
conversion and positive word of mouth.
Look into why users were more likely to abandon tasks on Crunchyroll or erroneously mark them
successful, but more likely to keep at a task for a longer period of time on Funimation. The investigation
into this difference may lead to other fruitful avenues for improving Crunchyroll’s usability.
20
Page
Sarah Letham (818) 667-5316 sarah.blass.letham@gmail.comYou can also read
