Testing Applications for the Real World - Can the crowd deliver "better, faster, cheaper" testing for a software-driven world?

Testing Applications for the Real World - Can the crowd deliver "better, faster, cheaper" testing for a software-driven world?
Testing Applications for the Real World …
Can the crowd deliver “better, faster, cheaper” testing
for a software-driven world?

November . 2012
Testing Applications for the Real World - Can the crowd deliver "better, faster, cheaper" testing for a software-driven world?
Testing Your Applications for the Real World...

 Table of Contents
   1.   Background & Introduction                                                                                    3

   2.   The Changing World of Business, IT and Applications                                                          4

   3.   The Market Response: New Ways to Perform …Enter the crowd                                                    5

   4.   Solutions Defined:  How Does Crowdtesting Work?                                                              7

   5.   Features & Benefits:  When and How Is Crowdtesting Useful?                                                   8

   6.   Applicability: When Is Crowdtesting Appropriate (and when is it not!)                                       11

   7.   Current Users:  Who Is Crowdtesting Today?                                                                  12

   8.   Challenges in Using The Crowd                                                                               14

   9.   The Future of Innovation From The Crowd                                                                     16

   10. About PASS and massolution                                                                                   17

   1.   IT and Business Are Changing                                                                                 3

   2.   Enterprise Crowdsourcing for Applications Testing                                                            6

   3.   The passbrains Process for Crowdtesting                                                                      7

   4.   Sample Test Results from passbrains                                                                          9

   5.   Crowdtesting ROI for Market Facing Apps                                                                     11

   6.   The Crowd Comes From Many Directions                                                                        13

Testing Applications for the Real World - Can the crowd deliver "better, faster, cheaper" testing for a software-driven world?
Testing Your Applications for the Real World...

 Background & Introduction

        “Global firms need to deliver continuous access, any time and anywhere, often to thousands
        of employees and sometimes millions of customers, over different types of devices, and are
        struggling with the challenges…” 1

        “ … the ability to effectively test the quality of software applications has never been more
        critical to an organization’s reputation and operations.” 2
        										-- ComputerWeekly.com

 When it comes to software applications testing, statements such as these speak directly to how crowd-
 sourcing can deliver value and provide a new model for getting access to talent and resources where
 companies use the scale, expertise, and diversity of the public domain to complete business-critical tasks.
 Our goal is to ask and help answer questions about using the crowd for software testing as a critical part of
 a company’s overall application deployment, with an emphasis on challenges facing the buyer, whether you
 are a small, high-growth company or a Global 1000 enterprise.

 The market is in its early stages, which calls for more clarity and education on everyone’s part, so that both
 buyers and sellers can determine if and how using the crowd fits into their future applications testing.  Thus,
 we’ve included multiple views from buyers, analysts, and providers to assess whether crowdsourcing can
 replace some current testing methods or add capabilities previously unavailable through either in-house or
 traditional outsourcing approaches.

 Written with the help of applications developers, software testers, and executives responsible for application
 quality, PASS Technologies AG has sponsored this white paper through its passbrains division in collaboration
 with massolution, a research and advisory firm, specializing in the design and implementation of enterprise
 crowdsourcing and crowdfunding models. We hope you will join us for the conversation about Crowdsourced
 Software Testing, or in short “crowdtesting” at www.crowdsourcing.org.

 1        ‘Few Firms Test Security of Mobile Apps….” ComputerWeekly.com, Sep 2012
 2        “Weighing Up The Options for Software Testing,” ComputerWeekly.com, Aug 2012.

www.massolution.com                                                                                                                         3
Testing Applications for the Real World - Can the crowd deliver "better, faster, cheaper" testing for a software-driven world?
Testing Your Applications for the Real World...

 The Changing World of Business, IT and Applications

 Today’s world is changing faster than ever. Trends in business and information technology are inextricably
 linked for Global 1000 companies, governments, and consumers alike. One data point: the number of smart
 phone applications is approaching 1 Million, while the number of mobile devices has surpassed the number
 of desktop and laptop computers combined.3 Product life cycles have shrunk from years, to months, to
 weeks in some cases, with all that implies for multiple and frequent software releases as companies struggle
 to just to keep up with consumers. IT decisions and control have migrated from centralized to distributed
 systems, and users are increasingly in charge (see Figure 1).

 Figure 1 .  IT and Business Are Changing … It’s All About the User

     Old World                                                            New World
     • Corporate-driven IT                                                • Consumer-driven IT
     • Centralized systems and apps                                       • Distributed computing (The Cloud)
     • Command-and-Control                                                • Open source
     • HW and SW “included” & OEM-driven                                  • More HW & SW than ever … hundreds of thousands
     • Sourcing, like IT, stays close to home                               of smart phone apps
                                                                          • Sourcing, like IT, also becomes more open & distributed

 In many cases, the physical world has moved online, from content to commerce, and applications are the life-
 blood of many businesses, especially those that aspire to reach the mass markets through the latest media.
 Companies are doing everything they can to refine, differentiate and maximize the end-customer experience,
 which is as likely today to occur at the end of an application versus a phone call or in-person. Applications,
 like other tools, must operate “to spec” (functionality), be secure yet easy to use, and work properly across
 other systems (hardware, software, networks). Moreover, companies must be able to do all this at a reason-

 3        “Infographic: Smartphone App Numbers,” WebPro News, Jun 2011.

www.massolution.com                                                                                                                           4
Testing Applications for the Real World - Can the crowd deliver "better, faster, cheaper" testing for a software-driven world?
Testing Your Applications for the Real World...

 able cost. As a portion of applications spending, testing comprises 20% to 30% of total costs, and “functional
 testing” typically makes up 80% of testing spend.4 Accordingly, F1000 companies have increased their use of
 third parties to get the job done.

 Globally companies invest more than $50 billion per year on applications testing and quality assurance,5
 while research firms such as IDC and Forrester indicate the market will more spend nearly $20B of that
 figure on external providers, with an increase in outsourcing of 15% annually over the next several years.6
 Drivers behind decisions to outsource applications testing include lower costs, better speed/time to market,
 and lack of in-house resources, tools and automation to ensure software works in the real world.

 The Market Response: New Ways to Perform …Enter the crowd

 With all the innovations to date - more tools, more automation, more outsourcing, the Cloud, etc. – many
 of the age-old business problems of effective (quality) and efficient (cost) software testing still exist:  cost
 and time over runs, insufficient domain knowledge, tester attrition, and fragmented testing processes.  As
 markets move faster than ever, there is now more testing, more often, and the quality of apps suffers as
 organizations try to keep up. For example, one global payment processing provider found that, while its
 software engine could process millions of individual payments without a glitch, customers who wanted to
 make simultaneous and sequential payments from the same account to multiple payees experienced a
 software failure, which is to say a “serious customer service issue.”

 Crowdtesting aims to raise the bar for achievable levels of quality, flexibility, speed and cost for testing
 software. The crowd comes in various shapes and sizes, but, as Figure 2 shows, crowdtesting for large enter-
 prises refers to managed services delivered over public networks through highly-skilled, uniquely qualified
 professionals who test consumer-facing applications, from corporate websites, to transaction processing
 engines, to mobile consumer life-style apps. Contrary to the populist hype around “the crowd”, getting
 access to your own group of dedicated testers involves a professional relationship with the testing company
 who can help execute on commercial testing needs.

 4        “Independent Testing Services -‘It’s Time to Innovate’,” Everest Research Institute, May 2011.
 5        “Top six trends to drive market for software testing in 2012,” TechJournal, citing Pierre Audoin Consultants, Nov 2011.
 6        Ibid.

www.massolution.com                                                                                                                                                   5
Testing Applications for the Real World - Can the crowd deliver "better, faster, cheaper" testing for a software-driven world?
Testing Your Applications for the Real World...

 Figure 2 . Enterprise Crowdsourcing Model for Software Testing

 • Led by certified program managers
 • Single point contact
 • Managed services approach offers ‘crowd management’
                                                                                                                 CLIENT LEAD

 • Testers are screened and proven before deployment
 • Tester rating system enables skill-level matching
 • Many are full-time employees in lead IT shops
                                                                                                       SENIOR QA SUPERVISORS

 • Same processes and quality on every project
 • Reports generated in usable/actionable formats
 • Enterprise users can re-use favorite testers                                                                10-150 TESTERS

 Online apps, broadly defined:

      Company Website                           Mobile Payments                      Gaming & Entertainment    Lifestyle & Social Media

 Issues of privacy, security, and quality (addressed below) are all legitimate concerns for this new form of
 software testing, but even without the crowd such problems exist. For example, more than half the software
 commonly used by businesses fails to meet acceptable levels of security, according to a study of almost 3,000
 applications.7   And the problem only worsens for mobile applications.  The real question is ‘How can crowd-
 testing actually help with these issues?’

 Today, crowdtesting is becoming a viable testing option, able to operate effectively in enterprise environments,
 where 47% of QA professionals indicate they know what crowdtesting is and how it works, and more than
 one-third believe using the crowd is a trend that will change the landscape of enterprise application testing,

 7        “Crowd open security testing to all developer, says Veracode,” re-printed on Cowdsourcing.org, Jan 2011.

www.massolution.com                                                                                                                                               6
Testing Your Applications for the Real World...

 according to a study by Forte.8 Global market
 research firm IDC notes that “traditional                                         “I think as an industry, we need to make crowd-
 outsourcers will face increasing competition                                      testing part of the normal testing process. At the
 with evolving cloud {and} crowdsourcing                                           same time you need to be clear on your objectives,
 models for testing applications,” and that                                        and use the crowd in a such a way that it is both
 some “enterprises now prefer a crowd-                                             simple and detailed in terms of bug tracking, so that
 sourced marketplace … to get access to                                            companies can integrate it into their overall testing
 global talent and diverse set of skills.”9                                        process. Some people have the impression that
                                                                                   quality is an issue with crowdtesting, but I think they
                                                                                   have not spent enough time looking at the crowd.
 How does Crowdtesting work?                                                       In my opinion, if you have the right provider, it
                                                                                   makes sense to invest in crowdtesting.”
 For software that is consumer-oriented
 and multi-platform, companies of all sizes                                        Bodo Vahldieck, Senior Manager
 and sectors are now exploring how to test                                         Quality Management Localization Services
 their applications using the crowd. As one                                        Autodesk – a global leader in 3D design,
 industry analyst and blogger notes, “… the                                        engineering and entertainment software
 crowd makes it possible to validate that appli-
 cations perform to a given standard across
 locations” {because} “each individual tester

 figure 3 . Passbrains Process for Applications Testing

        PREPARATION                         INITIATION                         EXECUTION                         EVALUATION                 ACCEPTANCE
                                                                                                                                             & WRAP UP

     Client requirements            Test Cases Design                  Test Tasks                         Test Case Reports              Test Report
     analysis                                                          Execution                          Evaluation                     Delivery to
                                    Test Configurations
     Test plan design               Definition                         Test Case                          Test Result Metrics
                                                                       Reporting                                                         Customer’s
     Test resources                 Test Tasks                                                            Test Results Report
     determination                  Assignments                        Progress                           Design
     • No. of Testers                                                  Monitoring &
                                    passbrains PM-                                                        Testers’
     • Skills                                                          Support                                                           Internal Project
                                    System Set-up                                                         classification
     • Configurations                                                                                                                    Review
                                                                       Results                            refinement
                                    Project Briefing
     Proposal to client                                                Monitoring                                                        Testers’ Feedback
                                    for Testers                                                           Testers’
     (fixed price and                                                  Test Tasks-                        remuneration
     duration)                      Testers’ NDA
                                                                       Reassignments /
     Contract with client                                              Recruitment of
                                    Testers’ Task                      additional Testers
     passbrains Project             Acceptance                         (only if necessary)
     Testers recruitment

 8        Efficacy & Applicability of Crowdtesting, Forte Consulting Group, Mar 2012.
 9        “Top six trends to drive market for software testing in 2012,” TechJournal, citing Pierre Audoin Consultants, Nov 2011.

www.massolution.com                                                                                                                                                   7
Testing Your Applications for the Real World...

 knows how the application must behave in his/her locale.”10 Sourcing software testing through the crowd
 offers access to an on-demand, distributed and scalable workforce delivering on a pay-for-performance
 basis. Figure 3 shows the passbrains process for an enterprise crowdtesting environment.

 Large corporate users should expect “the same or better” in terms of the testing robustness, quality, and
 detailed processes suitable for enterprise testing. It’s true that not all crowd testers are the same but
 companies can identify and solve critical issues by using enterprise-capable crowdtesters. Accordingly,
 the right enterprise crowdtesting solutions should offer …

 • testing-as-a-service for mobile, web and enterprise apps
 • “on-demand” test projects with selected, qualified test experts, with the opportunity to select and re-use
   the best testers, based on customer experience
 • guaranteed coverage of all major devices, computers, operating systems and browser versions
 • exploratory, scripted and automated testing for functionality, compatibility, security, load/performance,
   localization and end-customer usability
 • the ability to test across all product development 
phases, from early prototype and pre-release beta
   testing to continuous post-release testing
 • professional project management and quality assurance

 When (and how) is crowdtesting useful?

 The features and benefits of professional crowdtesting services include:

 •  Industrialized Testing
    (also referred to as “real world testing”)                                    “Testing is expensive and takes time and resources.
                                                                                  Crowdtesting certain apps is not only more cost
 The irony of the evolution of software devel-                                    effective but does things that companies cannot
 opment is that no longer are corporate/                                          reproduce in a lab environment, which helps find
 industrial software labs able to mimic and                                       more bugs as well as issues that no one even thinks
 test for all the possible permutations in the                                    about internally. I think that while there are a few
 end use of an application. In part, they lack                                    pioneers out there now, more and more companies
 the numbers (5 to 10 dedicated testers versus                                    will decide to try crowdtesting because they will find
 a crowd of 150 - 200), while in other cases                                      that it generates real added value to their testing.”
 they cannot replicate the literally thousands
 of IT configurations that applications will face                                 Vladimir Vlasic, Software Testing Manager
 across multiple devices, operating systems,
                                                                                  Mobiliar, leading Swiss pensions and insurance
 versions, and configurations.

 For example, in one passbrains testing
 program for a Windows-based software

 10       “Crowdtesting – Applicability & Benefits,” HCL ERS Blogs, Kunal Banerjee, Feb 2012.

www.massolution.com                                                                                                                                      8
Testing Your Applications for the Real World...

 product, there were multiple software versions, processors (32 and 64 bit) and 9 languages, including Arabic,
 Chinese, English, and Russian, resulting in more than 60 different configurations and over 1,000 permutations
 in just the desktop environment alone. This “ability to test under real-world conditions” is the #1 reason IT
 shops turn to the crowd, as “crowdtesters normally find many more defects in an application compared to
 traditional outsourcing.”11

 Figure 4 . Sample Test Result from passbrains



         # of Bugs Found




                                 Day 1   Day 2    Day 3         Day 4   Day 5           Day 6          Day 7

                                                 Total Active Testers   Total Tickets

 11        Ibid.

www.massolution.com                                                                                                            9
Testing Your Applications for the Real World...

 •  Speed to market

 There are two types of speed that matter – the speed of getting something started and the speed of getting
 something done! In an Information Week article, General Motors revealed how it has recently moved away
 from traditional forms of outsourcing for applications development and testing, “so its teams can be more
 nimble and creative.”12 The CIO wanted more “business-changing ideas from IT” and he wanted them faster.
 He noted with traditional forms of outsourcing, however, that “When the business says ‘go’ then that means
 we start working on a contract, we don’t start working on a project.” For companies that want speed and
 flexibility, crowdtesting may offer a solution.

 Regarding the rate of completion, enterprise buyers cited “quick-turnaround time” as the number two reason
 why they look to crowdtesting.13   Typically, the completion rate is significantly faster, often in proportion
 to the greater number of allocated test specialists. Whereas a typical enterprise applications test cycle
 might require 2 to 3 weeks, a crowdtest solution might take only 2 to 3 days. Moreover, typically, crowdtest
 resources are always “on call” so that the question “When can you start testing?” typically results in the
 answer companies most want to hear … “now”. This is especially critical when testing and release cycles
 once measured in months are now more likely to occur in weeks or even days.

 •  “Glocalization”

 Crowdtesting is not the next evolution of outsourcing, so much as it is a specific strategy to address the
 emerging challenges of applications in today’s world, namely how to track and test new applications across
 thousands of configurations at market speed.  In fact, where outsourcing led to offshoring, and the migration
 of applications testing further away from the client, crowdtesting actually reverses this trend, by enabling
 companies to localize testing around the ultimate end user, but on a global basis.

 Crowdtest platform providers can source qualified testers for specific markets since commerce may be
 global but buyers and their opinions, habits and values are often very local. Such coverage should include all
 the possible virtual environments in which the applications are expected to work (operating system, device,
 network, etc.), as well as all physical environments and demographic attributes – language, age, gender,
 location, etc.

 •  Cost Savings

 In passbrains’ experience, test execution through a crowdsourced platform typically costs 50% to 70% less than
 traditional in-house solutions. According to analysts at Everest, “crowd labor is {about} 60% cheaper than other
 models.”14 For those areas where the crowd makes sense – e.g., mobile, web and desktop software built for
 B2C applications – the cost savings must also make sense. At the same, “buyer beware” applies to crowdtesting
 just as it does to any other product or service, and the adage ‘you get what you pay for’ applies as well.  Value
 and price should align.

 12       “General Motors Will Slash Outsourcing in IT Overhaul,” InformationWeek, Jul 2012.
 13       Independent Testing Services -‘It’s Time to Innovate’, ...
 14       “Every Crowd Has A Silver Lining,” Everest Group Research, 2011.

www.massolution.com                                                                                                                              10
Testing Your Applications for the Real World...

 One key factor in lower costs through crowdtesting is the variable, “pay-as-you-go” model, something tradi-
 tional forms of outsourcing have always aspired to but failed to achieve as they end up building the same
 infrastructure they sought to help others avoid. Another key factor is that companies using crowdtesting
 typically pay for valid bugs reported and other deliverables, as defined by the customer.   Based on research
 between passbrains and massolution, we find that the typical cost of an accepted and validated bug is 3 to 4
 times higher in-house than with crowdtesting.15

 Moreover, key applications that drive sales and customer interactions cannot afford bugs, which can not only
 affect short-term sales but also damage a company’s brand and reputation as well. In one case, a global
 e-retailer used crowdtesting to discover bugs that their in-house team did not find and which would have
 cost the company $150,000 had the bugs gone undetected.16 Figure 5 below outlines the high-level features
 and benefits of a crowdtesting approach versus either in-house or traditional outsourcing models

 Figure 5:  Crowdtesting ROI for Market-Facing Apps

                                     IN-HOUSE                                                            CROWDSOURCING

      COST                                 “X”                       Savings of 20%+                       Savings of 50%+

                             Focus on all apps with                   Large % of ERP                  Focus on external apps
      SCOPE                    limited resources                  with ‘flexible’ resources          with unlimited resources

      SPEED                             Months                             Weeks                                  Days

                                                                                                    PRO: unlimited innovation
                          PRO: fewer IP issues                  PRO: services focus
      QUALITY                                                                                       CON: potential security
                          CON: fewer brains                     CON: flexibility a myth

                                              For market-facing applications aimed at customers or employees,
                                                crowdtesting can deliver more bugs, faster and at lower cost.

 When is crowdtesting appropriate (and when is it not!)

 So how does crowdtesting fit into an overall enterprise sourcing strategy and when is it appropriate to use
 the crowd?  There are several key variables companies should consider when thinking about where and how
 best to use crowdtesting for their software applications.

 15       Based on actual customer experiences at passbrains.
 16       Ibid.

www.massolution.com                                                                                                                         11
Testing Your Applications for the Real World...

 Typically, crowdtesting works best when …

 •  the Internet is ‘operating environment’ and connection mechanism
 •  the application server is publically hosted, able to accept global requests
 •  there is a need and capacity to handle large volumes of simultaneous users
 •  the client computing device is open, with minimal requirements – PC, mobile
 •  there is some level of application stability, pre- or post-release17

 There are certain things that enterprises can only test through the crowd – e.g., multiple languages or “dirty
 configurations” that reflect actual users’ IT set up.  Crowdtesting is also appropriate for multiple testing
 methods – e.g., waterfall, agile – and lends itself well to blackbox testing, obviously, for applications outside
 the firewall.  However, the crowd is not a panacea.  Companies cannot count on the crowd to test all the
 exceptions and adverse conditions driven from the web- or application-server side, especially if conditions
 there change before or after testing.

 Nonetheless, companies do use private crowds for white-box testing. For example, working with enterprises
 that manage sensitive customer records and data, passbrains has “anonymized” data sets or used dummy
 data in the past. In fact, inside the larger trend of crowdsourcing and applications testing, traditional
 outsourcing providers are attempting to develop dedicated, private crowds to increase the utilization rates
 of their own testing staff. In short, there is a time and place for companies to apply crowdtesting as a critical
 part of their overall testing strategy, based upon the factors above.

 Who is crowdtesting today?

 Crowdtesting is on a trajectory similar to most new markets for IT and professional services: providers of
 crowdtesting services are out ahead of their buyers in many cases, while many companies still seek to answer
 the question, ‘How can the crowd help me with my applications testing?’  Today roughly 80% of enterprise
 crowdtesting occurs in North American and European markets, primarily among large multi-national companies
 with multiple and multi-dimensional markets where the crowd can help.  Leading industries mimic those who
 have lead the way for other types of sourcing (traditional, outsourcing, offshoring) and include financial
 services, telecom, publishing, health and pharmaceuticals, high-tech, retail, and travel. Again, the crowd
 works best wherever companies must push application-based services to the end consumer.

 Interestingly for smaller, high-growth companies, crowdtesting also offers a head start and accelerated
 testing capability, especially for companies with short product and service life cycles, which require faster
 time to market and more frequent application releases or updates – e.g., any iPhone banking app! Every
 business is, in some sense, becoming a technology company, which is why crowdtesting is so much more
 useful and accepted today than even a few years ago. As for the digital-age enterprises, who represent
 the current and next-generation stock board leaders, companies such as Amazon, eBay, Google, Groupon,
 Microsoft, and others are all testing their applications with the crowd.

 17       “Crowdtesting – Applicability & Benefits,” HCL ERS Blogs, Kunal Banerjee, Feb 2012.

www.massolution.com                                                                                                                               12
Testing Your Applications for the Real World...

 Massolution’s CsFactor™ enterprise crowdsourcing framework for identifying crowdsourcing providers and
 tasks identifies over a hundred enterprise-grade providers and 65 enterprise applications of crowdsourcing,
 including applications testing. Figure 6 depicts today’s “supplier market” for crowdtesting services, based
 upon the various attributes and differentiators among current providers, which are broadly defined and
 range in their focus and competence in areas such as testing coverage, depth of expertise, volume of testers,
 geographic coverage and enterprise focus. It is important to remember that crowdsourcing is not as much a
 type of supplier as it is a testing approach.

 Figure 6: The Crowd Comes From Many Directions

                                                                                          Managed Services
                                                                                          for Crowdsourcing

                                                                           Legacy IT Providers
        Improved Time                                                        & Outsourcers
                          TARGET OPPORTUNITIES

        to Market

        Lower Cost                                                                          “The Crowd” At Large /
                                                                                           Independent Contractors
        Higher % of
        Critical Bugs                                        Large Enterprises

                                                 LOW                                                                      HIGH
                                                                                PERCEIVED RISKS

                                                        Protection of IP        Security of             Concerns over
                                                                                Customer Data           quality control

www.massolution.com                                                                                                                            13
Testing Your Applications for the Real World...

 Challenges in using the crowd

 As with all forms of sourcing, there can be issues around the relative comfort buyers have with alternative
 ways to work, and there are barriers and concerns around various methods of applications testing. For
 traditional outsourcers, companies have both internal and external worries, including:18

 •  lack of in-house testing, which makes it hard for outside providers to step in
 •  an internal inability to compare costs to shop other options
 •  a lack of test automation tools and expertise on the part of the provider
 •  provider’s industry knowledge of the business and related applications

 Companies assessing crowdtesting may have some of these same concerns, while others are not as relevant
 (e.g., deep vertical or functional expertise may be less relevant for B2C apps). There are some concerns that
 are more particular to crowdtesting, such as the novelty of the market, issues around quality of processes
 and workers, and worries over data privacy, IP protection and security.

 Novelty - ‘It’s a new market, has the service proven itself?’

 Crowdsourcing is a new model for getting work done, and crowdtesting applications is a part of this trend.
 As this paper shows, numerous buyers and sellers of crowdtesting are already active in the market.  History
 tells us there will be a surge in users (whether it makes sense for all of them or not!), as the lemming effect
 takes hold.  Later, as the market matures, industries and business will take a more deliberate but confident
 eye towards using the crowd. As one industry analyst notes, “The question is not whether or you should use
 crowdtesting, but only where and when.”19

 Quality - ‘How do I know what I’m getting?  Am I getting the right crowd?’

 This is perhaps the most important and valid concern when considering where and how to use crowdtesting,
 especially given that not all crowds perform the same. Companies interested in using the crowd should
 focus on the quality of the provider’s platform, processes and systems … all of the things that will enable
 them to build and manage a chosen crowd of individual testers. Then, they should focus on the quality of
 those individuals – certifications, real-life experience, and the ability to complete tasks to high standards
 (i.e., find bugs that your in-house team could not).

 Key questions to assess the quality of a potential crowdtesting provider include:

 1. About the testing platform and program management

 •  How long / involved are your testing engagements (partner versus vendor)?
 •  How do you break down component activities for crowd testers?

 18       Based in part on Independent Testing Services -‘It’s Time to Innovate’, ...
 19       “Crowdtesting – Applicability & Benefits,” HCL ERS Blogs, Kunal Banerjee, Feb 2012.

www.massolution.com                                                                                                                               14
Testing Your Applications for the Real World...

 •  How do you capture and organize accumulated knowledge across projects?
 •  How do you document your processes and how mature are they?
 •  What are your escalation procedures?
 •  What does a typical testing report look like – bugs found, severity, cost?
 •  How much time and money have you invested in your testing management system and platform (“years”
    and “millions” puts you at the ‘enterprise level’)?

 2. About the testers

 •  Where, how (and how quickly) do you source your testers?
 •  What security and background procedures do you follow?
 •  What is the profile of your testers – education, location, languages?
 •  How many of your testers operate at what level – e.g. % junior versus senior?
 •  How do you recognize and reward your testers?
 •  Where else are your testers employed (many will have other jobs)?

 Given that apps testing is already a very prescribed and procedural activity, providers of crowdtesting
 services should apply the same rigor and discipline to managing crowdtesters. For example, at passbrains,
 in-house program managers source, interview, background check, and ‘beta test’ individual testers, all before
 customers ever see or use them through their platform. Passbrains also engages their testers through
 internal forums to send out and seek advice, so that not only are they tapping into unique testing skills for
 specific environments and configurations, but they are also collectively learning how to go about testing in
 the most efficient and effective way possible.

 Another aspect of quality control is ‘testing fatigue’. Just as a handful of internal testers repeat the same
 processes, fail to see new things or simply lose focus and interest, the same can be said of the crowd, though
 it takes longer and is easier to avoid. In the case of passbrains, the company structures and layers their
 testing resources to introduce new testers after 36 – 64 hours (a typical period when most crowdtesting
 occurs and easy-to-find bugs emerge), and then calibrates testers’ compensation to incent and reward those
 who stay with the testing process.

 Security – ‘How do I protect my company’s IP and/or customer data?’

 Companies using the crowd should demand the infrastructure that provides the best possible security – e.g.,
 passbrains replicates “same as” systems and intranet-based testing environments as needed, depending on
 the nature of the testing, including devoting dedicated servers specific to clients.  However, given the nature
 of crowdtesting, companies are often less worried about their IP and the guts of their apps (which testers
 cannot see), than about the issue of privacy and customer data.

 For example, if companies working with passbrains have privacy concerns, the company can anonymize the
 data used in tested applications. Security, especially mobile security, is a huge issue whether companies
 are using in-house or external resources. The gaping hole that was the entrée to many companies IP and

www.massolution.com                                                                                                           15
Testing Your Applications for the Real World...

 internal systems was once “the web” (or the website, to be more accurate). It has now switched to the
 mobile world. Computer Weekly states that, “Organisations are struggling to manage the challenges of the
 mobile era, with less than a third formally testing their mobile applications,”20  and “a mere 18% of 1,500
 CIOs, IT directors and QA professionals {saying} their organisations were focusing on security,” according to

 The Future of Innovation From The Crowd

 The environment and circumstances for crowd testing will continue to evolve, but the real key is  ‘flexible
 innovation’. With hundreds of thousands of applications, changing platforms and devices, and endless
 releases, companies are really in a constant state of beta testing their software in real-time with customers.
 That’s a brave new world, and very different than the past.

 According to PASS CEO and inventor of the passbrains platform, Dieter Speidel, “It’s really about harnessing
 change, about innovation, which means you have to look beyond the crowd … it’s not so much a model but
 a mindset.” This means that, while outsourced crowdtesting is certainly one way to get things done, as we’ve
 discussed in this paper, it’s also a path to new ideas for how to use the crowd.

 For example, your company might …

 • engage a provider for a fully outsourced crowdtesting program
 • use the crowdsourced pool of experts for consulting your employees in their day-to-day tasks and
 • use crowd platforms to identify and recruit specific talent
 • partner with a provider to build your own crowd, managed by someone else
 • license a crowdtesting platform to build and manage your own crowd
 • use the crowd to assess your in-house testing tools and capabilities
 • engage yet other 3rd parties to help navigate the crowd – sourcing strategies, vendor management, and
   governance for crowd-based solution

 These are all possibilities depending on your company’s unique circumstances and goals. Flexibility is the
 aspect of innovation that matters most … in how companies can identify, build and manage the crowd …
 in how they can set up and run projects compared to traditional methods by tapping into vast resources,
 quickly … and realizing that the crowd is not just about ‘doing the job’ but also about creating ideas for how
 they can do the job better.

 20       “Automate testing for agile quality,” ComputerWeekly.com, Jun 2012.
 21       “Building software security testing skills for managers,” TechTarget, Feb 2012.

www.massolution.com                                                                                                                           16
Testing Your Applications for the Real World...

 Enter the crowd.

 When it comes to innovation, the truth is that if companies are doing it the same as they did a year ago,
 they are probably not as current as they think. Crowdtesting is a way to maintain and reinforce what you
 value – quality, reliability, usability. Today companies can blend traditional and crowdtesting approaches to
 maximize their ROI, especially for businesses with public-facing applications, which are difficult to test in a
 lab. Again, the question is not whether or you should use crowdtesting, but rather where and when. Maybe
 making your applications better means letting them outside the four walls of your business in new ways you
 have not tried before?


 About PASS Group

 Building on long-term success in quality engineering and process excellence, the PASS Group is a Swiss-
 headquartered global IT and professional services provider with over two decades of experience in providing
 applications development and precision testing services. We help our customers, from small, high-growth
 start-up’s to Global 1000 companies, embrace “the crowd” through our passbrains platform and crowdsourced
 testing services as a way to close the speed versus quality gap in in a digital world where applications require
 more support, and testing, than ever before.

 For more information, please visit www.passbrains.com and www.pass.ch

 About massolution

 Massolution is a unique research and advisory firm specializing in the crowdsourcing and crowdfunding
 industry. Massolution works with leading organizations to deliver crowdsourcing and crowdfunding business
 models that access an on-demand, scalable workforce to deliver improved business performance and to
 drive product and service innovation and enhanced levels of customer engagement. Our team has experience
 working in large enterprise environments designing, implementing, and managing crowdsourcing initiatives.
 Massolution also operates the industry website Crowdsourcing.org.

 More information at www.massolution.com or www.crowdsourcing.org or via contact@crowdsourcing.org.

www.massolution.com                                                                                                           17
You can also read
NEXT SLIDES ... Cancel