MOBILE VIDEO GAME QUALITY ASSURANCE IN A STARTUP COMPANY DURING A CLOSED BETA TEST PHASE
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
MOBILE
VIDEO
GAME
QUALITY
ASSURANCE
IN
A
STARTUP
COMPANY
DURING
A
CLOSED
BETA
TEST
PHASE:
Case
Grey
Area
Jonas
Aaltio
Bachelor’s
Thesis
October
2013
Degree
Programme
in
Music
and
Media
Management
School
of
Business
and
Services
Management
DESCRIPTION
Author(s)
Type
of
publication
Date
AALTIO,
Jonas
Juhani
Bachelor´s
Thesis
20.10.2013
Pages
Language
61
English
Confidentiality
Permission
for
web
Parts
3-‐6
for
the
duration
of
publication
5
years
(
)
Title
MOBILE
VIDEO
GAMES
QUALITY
IN
A
STARTUP
COMPANY
DURING
A
CLOSED
BETA
TEST
PHASE:
Case
Grey
Area
Degree
Programme
Degree
Programme
in
Music
and
Media
Management
Tutor(s)
KIVIAHO,
Niko
LUCK,
Heidi
Assigned
by
Grey
Area
Ltd.
Abstract
Quality
assurance
in
mobile
video
games
development
has
a
crucial
role.
It
improves
the
product
in
all
frontiers
and
serves
as
one
of
the
final
steps
before
introducing
the
product
to
an
audience.
A
working
quality
assurance
will
offer
benefits
to
both
customer
and
developer
company.
Startup
companies
have
a
unique
approach
to
quality
assurance
and
Grey
Area
with
its
product
development
approach
is
among
so
called
lean
startup
companies.
When
a
startup
company
enters
a
closed
beta
test
phase,
the
quality
assurance
process
is
affected
along
with
other
product
development
processes.
The
thesis
introduces
mobile
video
game
quality
assurance
in
two
separate
phases
of
product
development:
in
product
development
phase
and
in
a
closed
beta
test
phase.
The
research
task
was
approached
a
set
of
research
objectives:
Grey
Area’s
quality
assurance
and
its
development
during
a
closed
beta
test
phase,
Grey
Area’s
quality
assurance
during
a
product
development
phase
and
a
literature
overview
into
the
general
mobile
video
game
industry.
The
study
was
conducted
by
using
a
case
study
research
approach.
The
case
study
approach
was
chosen
because
it
can
prove
very
effective
when
focusing
on
a
single
entity
observed
during
a
limited
period
of
time.
The
applied
case
study
research
methods
and
tools
included
mainly
qualitative
but
also
quantitative
approaches.
The
results
of
the
research
revealed
subtle
direction
changes
in
the
quality
assurance
process
when
Grey
Area’s
two
product
development
phases
were
observed.
The
results
also
provide
insight
into
a
startup
company’s
quality
assurance
and
product
development
approaches
in
the
mobile
video
game
industry.
Keywords
Quality
assurance,
mobile
video
game
development,
closed
beta
testing,
startup
company,
lean
methods
Miscellaneous
The
thesis
was
created
under
a
confidentiality
agreement.
Therefore
any
Grey
Area’s
confidential
documents
are
not
included
as
attachments
and
any
confidential
figures
are
not
mentioned.
KUVAILULEHTI
Tekijä(t)
Julkaisun
laji
Päivämäärä
AALTIO,
Jonas
Juhani
Opinnäytetyö
20.10.2013
Sivumäärä
Julkaisun
kieli
61
Englanti
Luottamuksellisuus
Verkkojulkaisulupa
Osat
3-‐6
viiden
(5)
vuoden
myönnetty
ajan
(
)
Työn
nimi
MOBIILIVIDEOPELIEN
LAADUNVALVONTA
STARTUP
YRITYKSESSÄ
SULJETUN
BETATESTAUSVAIHEEN
AIKANA:
Tapaus
Grey
Area
Koulutusohjelma
Degree
Programme
in
Music
and
Media
Management
Työn
ohjaaja(t)
KIVIAHO,
Niko
LUCK,
Heidi
Toimeksiantaja(t)
Grey
Area
Oy
Tiivistelmä
Laadunvalvonta
on
elintärkeä
osa
mobiilipelien
kehitystä.
Se
parantaa
tuotetta
kaikilla
osa-‐alueilla
ja
toimii
viimeisenä
askeleena
ennen
tuotteen
esittelyä
yleisölle.
Toimiva
laadunvalvonta
hyödyttää
sekä
asiakasta,
että
kehittäjäyritystä.
Startup
yritykset
lähestyvät
laadunvalvontaa
omaleimaisella
tavalla
ja
Grey
Area
on
tuotekehityksessään
osa
niin
sanottuja
lean
startup-‐yrityksiä.
Kun
startup-‐
yritys
siirtyy
suljettuun
beetatestausvaiheeseen,
heijastuvat
muutokset
yleisen
tuotekehityksen
ohella
myös
laadunvalvontaprosessiin.
Tutkimus
perehdyttää
lukijan
mobiilipelien
laadunvalvontaan
kahdessa
eri
tuotekehitysvaiheessa:
tavanomaisessa
tuotekehitysvaiheessa,
sekä
suljetussa
beetatestausvaiheessa.
Tutkimusta
lähestytään
joukolla
tutkimustavoitteita:
Grey
Arean
laadunvalvonta
ja
sen
kehittyminen
betatausvaiheen
aikana,
Grey
Arean
laadunvalvonta
tuotekehitysvaiheen
aikana
ja
kirjallinen
katsaus
mobiilipelien
laadunvalvontaan
yleisellä
tasolla.
Tutkimus
suoritettiin
tapaustutkimuksena.
Tapaustutkimus
valittiin,
koska
se
mahdollistaa
itsenäisen
kokonaisuuden
tehokkaan
tutkimisen
rajoitettuna
ajanjaksona.
Käytetyt
tutkimusmetodit
ja
-‐
työkalut
sisältävät
pääasiassa
laadullisia,
mutta
myös
määrällisiä
tutkimusmenetelmiä.
Tutkimuksen
tulokset
paljastavat
hienovaraisia
muutoksia
laadunvalvontaprosessissa
kun
Grey
Arean
kahta
eri
tuotekehitysvaihetta
tarkastellaan.
Tulokset
myös
avaavat
startup-‐yrityksen
laadunvalvontaa,
sekä
tuotekehityslähestymistapaa
mobiilipelialalla.
Avainsanat
(asiasanat)
Laadunvalvonta,
mobiilipelit,
mobiilipelien
kehitys,
suljettu
beetatestaus
,
startup
yritys
Muut
tiedot
Opinnäytetyö
tehtiin
salassapitosopimuksen
alaisena.
Täten
mitään
Grey
Arean
luottamuksellisia
dokumentteja
ei
ole
liitettynä
tutkimukseen,
eikä
luottamuksellisia
lukuja
ole
esitelty.
4
Contents
1 Introduction ................................................................................................. 5
2 Literature review ......................................................................................... 8
2.1 Brief history of video game quality assurance .............................................8
2.2 Current mobile video game quality assurance definition ............................9
2.3 Mobile video game quality assurance elements .........................................10
2.4 Quality assurance as part of the product development cycle and the
concept of rapid development ............................................................................13
2.5 Mobile video game developer startup companies ......................................16
3 Method ....................................................................................................... 21
3.1 Overview of the research method and its reliability ...................................21
3.2 Setting up the case study ..............................................................................22
3.3 Implementation ...............................................................................................27
4 Results ....................................................................................................... 46
5 Discussion ................................................................................................. 50
6 Conclusion ................................................................................................. 54
References .................................................................................................... 565 1 Introduction The mobile video game industry is acknowledged for extensive competition and is growing towards an estimated value of $ 18.3 billion in 2016 (Haggerty, 2012). In order for a mobile video game to be successful it must have a comprehensive number of users adopting the product or a very high percentage of users returning to use the product during a certain period of time (retention percentage). When competition is extensive, high quality products offer better chances of success, and this is where quality assurance becomes truly important. Quality assurance in the mobile video game industry provides the opportunity to develop truly high quality products. It is a process that has one single goal: reassuring that all elements of the end product function perfectly in a predetermined fashion. Generally, in mobile video game developer companies, quality assurance processes consist of various elements that are operating in full force when the product is nearing its completion. However, the mobile video game industry includes companies that differentiate their development approach by implementing quality assurance in the earlier stages of the product development cycle. Among these companies is a startup company called Grey Area. Grey Area is a mobile video game developer from Helsinki, Finland, operating in the mobile video game industry. The company was founded in 2008 and it is considered a medium-sized startup company. Currently, Grey Area has released two mobile video game for iOS (Apple’s mobile operating system) devices. The company is currently working on its next product that has not yet a planned release date. The author worked as QA (quality assurance) Lead within Grey Area between December 2012 and September 2013. Grey Area’s approach to the product development cycle includes applying quality assurance already in the early stages such as the product development phase. This creates a situation where traditional video game quality assurance processes are situated differently. This, in turn, creates unusual quality
6 assurance process approaches. Generally when a mobile video game company is closing into a beta test phase, the product is nearing its completion, shifting quality assurance into a direction that is straightforward and widely applied. However, when it comes to Grey Area, a mobile video game developer startup company, the shift from the product development phase into the closed beta test phase can create quality assurance processes that are not at all traditional. The main objective of this thesis was to evaluate how Grey Area’s quality assurance process evolved during a closed beta test phase taking place between the 4th April and the 16th May 2013. A secondary aim was to define the state of Grey Area’s quality assurance process during a product development phase. An additional objective was to study general mobile video game quality assurance on a theoretical basis. To meet these objectives, the thesis addresses three main research questions: 1. Based on a literature review what are the essential elements and processes that define mobile video game quality assurance? 2. What are the defining quality assurance processes in Grey Area during a product development phase and how do they differ from the generally used mobile video game QA processes? 3. Did Grey Area’s quality assurance processes evolve during the closed beta test phase and how were the end results evaluated when compared with the general mobile video game industry? To reach a deeper understanding, Grey Area quality assurance processes were studied by implementing a case study in two phases: during the product development phase and during the closed beta test phase. The study on both areas was conducted to examine if these processes evolved when a closed beta test phase in the product development cycle was entered. This thesis starts with a literature review to describe the general mobile video game quality assurance process and Grey Area as a startup company. The
7 second part of this thesis describes the research method by providing an overview of the study and its reliability and by describing how the case study research was set up and implemented. From this chapter the thesis moves on to the first implementation part where Grey Area’s specific quality assurance processes are revealed. In the second implementation phase of this thesis, Grey Area’s specific quality assurance processes during a closed beta test phase are examined. To conclude this thesis, the results from both implementation phases are defined after which they are discussed together with the literature review using the predefined research questions.
8
2 Literature review
2.1 Brief history of video game quality assurance
In order to provide a more specific quality assurance definition, a brief
historical review is in order so as to shed light on the overall process of video
game quality assurance. Video game quality assurance has been
implemented since the beginning of the whole industry. During the early days
of the industry, developers and manufacturers lacked the necessary tools to
update their software and hardware because there were no networks through
which the updates could be distributed (Jarman, 2010). Assuring a product’s
quality became crucial because dysfunctional products would not sell. Nolan
Bushnell, the Cofounder of Atari who is behind the commercial success of
Pong, one of the earliest video game in history describes game testing in an
interview by Morgan Ramsay.
And the only thing that slowed us down was testing, because every one
that would come off the line would have minor little problems. ... So, the
testers would find those problems (Ramsay, 2012, p. 29).
In 1972, when Pong was released, mainly the same person who developed
the game was in charge of quality assurance. Occasionally, as in the case of
Nolan Bushnell and Pong, additional testers were brought in if production
rates rose to unmanageable amounts. However, game testing was not a
profession at this time because it was not yet required. The evolution of the
video game quality assurance profession began along with Nintendo’s
Famicom system released in 1983 (later Nintendo Entertainment System,
NES). (Levy & Novak, 2010, pp. 8-9) Famicom was the dawn of the
development of more sophisticated and complex game that required
specialized staff for game testing purposes. These people were called game
testers (Levy & Novak, 2010, p. 9). From the following chapter it becomes
clear that the modern video game quality assurance both in general and in
mobile video game operates on a diverse area. 9
2.2 Current mobile video game quality assurance definition
Quality Assurance (QA) is defined in Oxford Dictionary 2013 as follows
Quality Assurance: the maintenance of a desired level of quality in a
service or product, especially by means of attention to every stage of
the process of delivery or production (Oxford Dictionary, 2013).
Current industry professionals are among the best to describe video game
quality assurance and play testing, which fall under the main techniques that
the QA personnel use. Brandon Sheffield (Sheffield, 2012), an editor in
Gamasutra, states that quality assurance is one of the most important aspects
of game development. Play testing is discussed among others by Luis Levy
and Jeannie Novak in their book Game Development Essentials: Game QA
and Testing (2010) as follows
Testing and quality assurance (QA) are roughly the same disciplined.
However, QA is used to describe the broad surface testing done inside
a publisher while testing cover the entire spectrum of techniques used
to find, investigate and defeat bugs (p. 4).
Melissa A. Federoff defines play testing in her thesis Heuristic and Usability
Guidelines for the Creation and Evaluation of Fun in Video Game (2002) “Play
testing is defined as in-house, formal observation of temporary consumer
testers” (p. 20). Later on, after observing video game development, she
concludes
However, while listening to a conversation between Jerod (producer)
and a tester, I learned that they (testers) actually analyze all aspects of
the game including game interface, game mechanics, and game play
(p. 40).
Based on these definitions it is clear that quality assurance is an important
aspect of functioning game development and, as such, it should be held in
high priority when considering the main ingredients that define a successful
video game development process. Although mobile video game are their own
separate branch of the general video game industry, they are in reality video10 games for mobile devices. Therefore, the same rules and definitions of quality assurance are applicable. As a conclusion: The current mobile video game quality assurance is defined to include a broader surface of monitoring and evaluating the general quality of a product, but it also consist the techniques of testing specific product elements. (Federoff, 2002 p. 40; Levy & Novak, 2010, p. 4) A fully functioning mobile video game quality assurance will provide an end result of high quality that functions perfectly in every scenario. However, this raises the question of how can this end goal can be reached. This is be discussed in the following chapter by introducing the essential elements of applied mobile video game quality assurance. 2.3 Mobile video game quality assurance elements Resolving defects, also known as “bugs” The crucial mobile video game QA elements, when considering the topic of this thesis, should be examined in three main categories. The first of these is the role of “bug hunting”, and the second the most commonly used QA tools and techniques. The final category is the execution of QA in the different states of the product development cycle. The main responsibility of a person operating in mobile video game quality assurance is to identify defects in the product. Defects, more commonly known in the industry as “bugs”, impair the users’ gaming experience, and due to this they must be identified and revised. In mobile video game, defects can generally be identified in the code that is created by game developers. (Jarman, 2010, p. 5) These defects can be relatively minimal such as a brief texture flicker in an animated object in the game world or potential critical deficiencies that cause the product to “hard crash” shutting down the game and completely ruining the user experience (Sheffield, 2009).
11 At the core of identifying bugs, conceivably the most frequently used technique is repetition in comparing inputs and outputs as described by Jimmy Thang, Luis Levy and Jeannie Novak: “You get an area of the game, that's your area, and you test everything about that one area for months on end” (Thang, 2012). When you’re testing video game, you often play the same game level over and over again for months on end. “You have blisters on your hands and a headache from looking at a tiny screen 10 hours a day” (Levy & Novak, 2010, p. 28). A tester is given information on what should be the correct output of a scenario when performing a specific input (Jarman, 2010, p. 6). If the end result of this simplified process is analyzed as incorrect, a bug is often identified. Repeating a single element in countless scenarios leads to unexpected results that frequently provide means to identify more bugs and potential relations between multiple bugs. When a bug is identified the same process is replicated in order to prove that the cause-effect hypotheses and the bug are valid. After a bug has been identified an accurate reproduce pattern is reported to the developer who is responsible for creating the game element so that the bug can be revised (Levy & Novak, 2010, p. 48). The holy grail of video game QA is that every single detail in every conceivable scenario of the product is tested. However, this is often unfeasible for reasons of constrained resources in product development. These constrains can be minimized by managing a correct set of quality assurance techniques and tools.
12
Techniques and tools
Quality assurance professionals seem to favor two main testing techniques:
Ad-Hoc testing and the usage of test cases (Wilson, 2009). Ad-Hoc testing is
considered more fluid, random and broad. This is a technique where the tester
has no limitations on execution and is free to test any scenario imaginable.
The main benefit in this technique is that it provides means to identify large
numbers of defects during a test session. It is up to the tester to explore the
system. (Chhabra, 2012, pp. 66-67) Test cases are strict, disciplined and
more feature specific. In test case testing a single game element is thoroughly
tested by repeating it in as many scenarios as the tester can imagine. The
main benefit of using test cases is that common actions or elements are
tested in a large variety of ways in every area of the product (Wilson, 2009).
When it comes to quality assurance tools, there are specific tool sets in use
for evaluating specific mobile game elements. These tools are mainly
electronic and used with computers but also traditional pens and papers are
used for documentation. Electronic tools include:
• Documentation, project tracking and management sofware: Software
that is used specifically for managing diverse projects include Jira
(www.atlassian.com), Trello (trello.com), Dropbox (www.dropbox.com)
and Google Drive (drive.google.com). The more traditional quality
assurance documentation software includes all “office software” such
as Microsoft’s Word, Excel, PowerPoint and Notepad, or similar.
• Bug databases and defect revise process trackers: Every person who
has work experience in mobile video game quality assurance knows
that the number of identified bugs can be extensive. In order to manage
the process of identifying and revising defects, databases and process
trackers are widely used. These include Bugzilla (www.bugzilla.org),
FogBugz (www.fogcreek.com), Jira and customized databases that are
created by the company for the company.13 Taking part regularly in a company’s internal meetings or monitoring when testers from outside the company are brought in are common tools as well. (The Road to QA Excellence, 2012, p. 14) Especially in modern mobile video game testing, device testing becomes crucial. There are numerous devices on which a company’s products can be played with, which places a significant emphasis on testing different devices with different user interfaces. An example of this is Pocket Gems, a San Francisco based startup that develop games for mobile devices that operate on user interfaces developed by Apple and Google. Their quality assurance testing is a twenty-four-seven process that involves teams both in the U.S. and abroad. They distribute the workload of compatibility testing with 40 mobile devices to offshore teams outside of the main developer team. (Cutler, 2012) 2.4 Quality assurance as part of the product development cycle and the concept of rapid development Time is frequently considered as the main reason for an insufficient end result from a quality assurance perspective. Testers themselves often consider the testing stage to be rushed or insufficient once it ends (Wilson, 2009). This is contradictory because quality assurance started early provides means to decrease the overall use of resources in the product development cycle, which is examined in the following chapter. Video game quality assurance has different characteristics based on the state of development. Production testing happens during game development, which makes the quality assurance process more flexible as testing is done simultaneously as the game is being made. However, quality assurance has a vital role in this phase, because resolving problems as soon as they appear is far less expensive at this state than it would be later on in the product development cycle (see Figure 1) (Vasquez, 2009). Early problems will
14 interfere with features that are considered complete if they are left untreated (Fristrom, 2003). Normally the main QA process starts when the game reaches the beta testing phase and it is in a more polished state (Levy & Novak, 2010, p. 53). Figure 1. Prevention over detection (Vasquez, 2009) Melissa A. Federoff describes a situation in which she was visiting a video game developer company that was in a production testing state when examined from a quality assurance perspective. She describes the evaluation process as ”informal”. This was due to the fact that the ongoing state of flux created a situation where nothing could be locked down and thoroughly tested as a fixed feature and the developers circled around each other commenting on each others work on the go. (Federoff, 2002, pp. 32-33) In order to improve the flow of the quality assurance process throughout the product development cycle, rapid development is commonly used. Rapid development stresses the importance of creating daily versions of the product that are distributed internally or externally. These daily versions are generally created and managed by product managers and appropriated QA persons test
15
them. If complications are found they are reported, ensuring that defects will
not collect into extensive numbers eventually ceasing the development
completely (Fristrom, 2003).
As a conclusion, based on a literature review, the generally used mobile video
game quality assurance essentials and processes include
• Testing every aspect of the product in order to find issues that are not
functioning so that they can be revised (Thang, 2012)
• Quality assurance during the product development phase is far more
adjustable than during the beta test phase. However, it is not to be
considered as unimportant (Fristrom, 2003)
• Two main mobile video game quality assurance approaches can be
defined: Ad-Hoc testing, which is considered more fluid, random and
broad and usage of test cases that are strict, disciplined and more
feature specific (Wilson, 2009)
• Quality assurance professionals analyze all aspects of the game
including game interface, game mechanics and game play (Federoff,
2002, p. 40)
• The mobile game industry provides additional challenge as a result of a
large range of supported devices (Cutler, 2012).
• The quality assurance process evolves during the development cycle of
a product (Levy & Novak, 2010, pp. 52-53)
• During a beta test phase the main quality assurance process begins
and the main objectives are to find bugs, replicate them and report
them (Levy & Novak, 2010, p. 49)16 2.5 Mobile video game developer startup companies Quality assurance in Grey Area has its unique features. This uniqueness is based on a startup company mentality that has been adopted also by Grey Area. Before defining Grey Area’s quality assurance in the further parts of the study, startup companies should be studied in order to provide a better understanding of the foundations that Grey Area’s quality assurance is based on. In other words, what it means to be a startup company in the mobile game industry that is in a constant state of flux. A large part of surviving in the industry is based on company’s unique company culture. Therefore introducing Grey Area’s company culture is befitting to the research objectives of this thesis. However, before introducing Grey Area’s definition of their unique company culture, the main elements of mobile game developers in general should be introduced. These are characteristics that generally every company in the industry has and they provide a starting point to conceive Grey Area as a mobile video game developer. A visionary game designer Jane McGonigal states in her book Reality is Broken (2011) that game developers have understood that game become hits and make money in direct proportion to how much satisfaction they provide and how much positive emotion they provoke (pp. 37). According to McGonical (2011), game developers are happiness creators who increasingly provide more powerful and mobile systems that satisfy the craving of people looking for optimistic engagement and emotional rewards (pp. 37-38). These characteristics are built deeply into the core of video game creation and it is often the main ingredient in defining video game developer company culture. Video game companies have a strong historical relationship to computer science and engineering (Wong, 2011, pp. 8-9). In addition to this, the video game industry is considered to be a creative industry, which shares the characteristics of more traditional creative industries such as design and performing arts (BOP Consulting, 2010, p. 40). Creative industries rely on making the most of people’s talent, creativity and passion. In addition, video game developers share the same considerable uncertainty about the likely
17 demand for creative products, the frequently collective nature of creative production and the need to develop and maintain creative teams with diverse skills. The members of these teams often also possess diverse interests and expectations about the final product (Flew, 2002, p. 7). Before going deeper, it must be taken into consideration that Grey Area is a startup company that shares unique features with other startup companies. There are numerous definitions of startup companies, but the most befitting to Grey Area is provided by a book called The Lean Startup (2011) by Eric Ries. Grey Area has adopted Ries’s approach as the core of its business. This approach can most suitably be examined when defining the elements forming Grey Area’s company culture on which its quality assurance is also based. Lean startups are companies that aim for the validation of their products via continuous hypotheses and their experiments (Ries, 2011, pp. 56-72). The outcome of this experimental approach is to build sustainable business based on learning. This leads “lean” startups to apply a build-measure-learn feedback loop as the core of their business (Figure 2).
18 Figure 2. Build-Measure-Learn feedback loop (Ries, 2011, p. 75) The build-measure-learn feedback loop is used to recognize whether it is wiser for a startup to continue in the direction that it is heading to or make to the strategic decision to make a pivot: a structured course correction designed to test a new fundamental hypothesis about the product, strategy, and engine of growth (Ries, 2011, pp. 77, 149). A successful lean startup has been able to minimize the total time through the build-measure-learn feedback loop thus decreasing the amount of wasted time and money (Ries, 2011, p. 78). In The Lean Startup, Ries defines numerous techniques on how to decrease the overall amount of time by advancing through the build-measure-learn feedback loop. However, all these are not discussed in greater detail. Instead the main focus is on the elements that are the most suitable when considering the topic of this thesis. Among all of the techniques one technique highlighted by Ries is the most appropriate, because this specific technique is emphasized in the second research phase of this thesis. The technique in
19
question is the process of measuring which serves as one of the core building
blocks of the build-measure-learn feedback loop. In order to measure, a
startup will use metrics. As defined by the Oxford Dictionary 2013, metrics are
“a method of measuring something, or the results obtained from this”. Ries
defines three main metrics areas that a startup must adopt in order to practice
successful measurement, the three A’s of metrics: actionable, accessible and
auditable (Ries, 2011, p. 143).
• Actionable: A demonstration of cause and effect in order for people to
learn from their actions
• Accessible: Build a metrics system that is understandable and that
preferably can be accessed by all of the company’s employees
• Auditable: Data must be able to be tested with real customers in order
to prove that it is auditable. Whenever possible, reports should be
drawn directly from the master data, rather than from an intermediate
system. This reduces opportunities for error. (Ries, 2011, pp. 143-147) 20
When a company is able to use metrics that follow the rule of three A’s,
measurement provides valid data that can be used to minimize the amount of
time used to progress through the build-measure-learn loop. Grey Area has
adopted this method, and it is a crucial ingredient when it comes to quality
assurance during the beta test phase. After discussing the defining elements
that are built into the core of many mobile video game developer companies’
culture and introducing the lean startup model that Grey Area has adopted, it
is befitting to introduce Grey Area’s company culture.
Grey Area values and culture based on Mikko Hämäläinen’s, one of the
founders of Grey Area, presentation “The 10 things we live by”:
1. Adaptive 6. Passion
2. Adoptive 7. Customer
3. Integrity 8. Best argument wins
4. Input, not consensus 9. Courage
5. Hard work 10. Humble and open.
(Hämäläinen, 2012)56 References About Bugzilla, Information provided by Bugzilla, Accessed on 02.04.2013, http://www.bugzilla.org/about/ About Dropbox, Information provided by Dropbox, Accessed on 02.04.2013, https://www.dropbox.com/about About FogBugz, Information provided by FogBugz, Accessed on 02.04.2013, http://www.fogcreek.com/FogBUGZ/ About Github, Information provided by Github, Accessed on 02.04.2013, https://github.com/ About Google Drive, Information provided by Google, Accessed on 02.04.2013, https://drive.google.com/start About Jira, Information provided by Atlassian, Accessed on 02.04.2013, http://www.atlassian.com/software/jira/overview About Testflight, Information provided by Testflight, Accessed on 02.04.2013, https://testflightapp.com/ About Trello, Information provided by Trello, Accessed on 02.04.2013, https://trello.com/ About Unity 3D, Information provided by Unity, Accessed on 02.04.2013, http://unity3d.com/
57 About Xcode, Information provided by Apple Developer, Accessed on 02.04.2013, https://developer.apple.com/technologies/tools/ Baltaretu, C. (2013) Discussions with Supercell, Email message of 06.04.2013. Receiver J. Aaltio. 06.04.2013. Divide & Conquer methodology recapitulation for thesis writer. Accessed on 13.04.2013 Bayer, J., Gacek, C., Muthig, D., Widen, T. (2000) PuLSE-I: Deriving Instances from a Product Line Infrastructure, City University London, London, Referred to on 04.04.2013 BOP Consulting (2010) Mapping the Creative Industries: A Toolkit, The British Council, London, Referred to on 06.04.2013 Bosser, Anne-Gwenn (2004) Massively Multi-player Games : Matching Game Design with Technical Design, National University of Singapore, Singapore, Referred to on 04.04.2013 Brown, R., Maire, F., Nantes, A. (2008) A Framework for the Semi-Automatic Testing of Video Games, Queensland University of Technology, Queensland, Referred to on 04.04.2013 Burger-Helmchen, T., Guittard, C. (2007) Are users the next entrepreneurs ? A case study on the video game industry, Faculté des sciences économiques et de gestation, Strasbourg, Referred to on 04.04.2013
58 Chhabra, N. (2012) Introduction To Adhoc Testing, International Journal of Scientific & Technology Research August 2012, p. 66-67, Referred to on 21.04.2013 Cutler, Kim-Mai (2012) How Do Top Android Developers QA Test Their Apps?, TechCrunch, 02.06.2012, Accessed 02.04.2013. http://techcrunch.com/2012/06/02/android-qa-testing-quality-assurance/ Erola, H. (2012) Usability in video games. Usability testing as part of video game development, Bachelor’s Thesis, Kajaani University of Applied Sciences, Kajaani, Referred to on 20.04.2013 Federoff, Melissa A. (2002) Heuristic and Usability Guidelines for the Creation and Evaluation of Fun in Video Games, Master’s Thesis, Department of Flew, T. (2002) Beyond ad hocery: Defining Creative Industries, Queensland University of Technology, Queensland, Referred to on 06.04.2013 Fristrom, J. (2003) Production Testing and Bug Tracking, Gamasutra 14.07.2003, Accessed 04.03.2013. http://www.gamasutra.com/view/feature/131238/production_testing_and_bug_ tracking.php Haggerty, M. (2012) The State of Mobile Game Development, Gamesindustry.biz, 28.11.2012, Accessed 03.06.2013. http://www.gamesindustry.biz/articles/2012-11-28-the-state-of-mobile-game- development
59 Hämäläinen, M. (2012) PowerPoint Presentation, This is presentation to our Freedom & Responsibility Culture, Helsinki, Helsinki, Finland, Accessed on 06.11.2012 Hopson, J. Hullett, K., Nagappan, N., Schuh, E., (2011) Data Analytics for Game Development (NIER Track), ICSE, Honolulu, Referred to on 04.04.2013 Jarman, A. (2010) Testing and Maintenance in the Video Game Industry Today, Liberty University, Lynchburg, Referred to on 21.04.2013 Levy, L., Lovak J. (2010) Game Development Essentials: Game QA and Testing, Delmar Cengage Learning, Delmar McGonical, J. (2011) Reality Is Broken, The Penguin Press, New York McKenzie, H. (2013) Now pulling in $1M a day, Supercell reflects on a banner year, Pandodaily, 04.01.2013, Accessed 02.04.2013. http://pandodaily.com/2013/01/04/now-pulling-in-1m-a-day-supercell-reflects- on-a-banner-year/ Meredith, J. (1998) Building operations management theory through case and field research, Journal of Operations Management 16, p. 441-454, 29.05.2013 Oxford Dictionary (2013) Quality Assurance definition, 01.05.2013, Accessed 15.04.2013. http://oxforddictionaries.com/definition/english/quality- assurance?q=quality+assurance Ramsay, M. (2012) Gamers at Work: Stories Behind the Games People Play, Apress, New York
60 Ries, E. (2011) The Lean Startup, Penguin Group, London Rose, M. (2013) Clash Of Clans' 5 Keys To Success, Gamasutra, 28.01.2013, Accessed 02.04.2013. http://www.gamasutra.com/view/news/185406/ Santos, Michael P. (2012) The 24 Hour Knowledge Factory in Video Game Development, The University of Arizona, Arizona, Referred to on 02.04.2013 Sheffield, B. (2009) Dirty Coding Tricks, Gamasutra, 20.08.2009, Accessed 21.04.2013. http://www.gamasutra.com/view/feature/4111/dirty_coding_tricks.php Telecommunications of Indiana University, Indiana, Referred to on 30.03.2013 and 04.06.2013 Thang, J. (2012) The Tough Life of a Games Tester, IGN, 29.03.2012, 02.04.2013. http://www.ign.com/articles/2012/03/29/the-tough-life-of-a-games- tester The Road to QA Excellence (2012) The Road to QA Excellence, PractiTest, Referred to on 02.04.2013 Thomas, D., Orland, K., Steinberg, S. (2007) The Videogames Style Guide and Reference Manual, Power Play Publishing, Lilburn Thomas, R. Murray (2003) Blending Qualitative & Quantitative Research Methods in Theses and Dissertations, Corwin Press Inc., Santa Barbara
61 Vasquez, M. (2009) PowerPoint Presentation: The art and science of quality assurance in the video game industry, Vancouver, British Columbia, Canada 12-13.05.2009, Accessed on 13.04.2013, original figure http://www.isixsigma.com/industries/software-it/software-defect-prevention- nutshell/ Wilson, D. (2009) Quality Quality Assurance: A Methodology for Wide- Spectrum Game Testing, Gamasutra, 28.04.2009, Accessed 02.04.2013. http://www.gamasutra.com/view/feature/132398/quality_quality_assurance_a_ .php Wong, Hinlan P. (2011) A Study of the Vide Game Industry in U.S Metropolitan Areas Using Occupational Analysis, University of Massachusetts Amherst, Massachusetts, Referred to on 06.04.2013
You can also read
