Validating Ajax Applications Using a Delay-Based Mutation Technique

Validating Ajax Applications Using a Delay-Based Mutation Technique
Validating Ajax Applications Using a Delay-Based Mutation

                                                                                               ∗                                ∗
                Yuta Maezawa, Kazuki Nishiura, Shinichi Honiden                                       Hironori Washizaki
                                            The University of Tokyo                                      Waseda University
                                                Tokyo, Japan                                               Tokyo, Japan
                         {maezawa, k-nishiura, honiden}                         

ABSTRACT                                                                          side and asynchronously retrieve data necessary to dynam-
The challenge of validating Asynchronous JavaScript and                           ically update Web page content without page transitions,
XML (Ajax) applications lies in actual errors exposed in a                        and this is crucial to attracting Web users [3]. Moreover,
user environment. Several studies have proposed effective                          the number of users exceeded 2.4 billion as of June 2012 [4],
and efficient testing techniques to identify executable faults.                     making Ajax applications essential platforms for daily life.
However, the applications might have faults that are not ex-                         The challenge to validating Ajax applications arises from
ecuted during testing, but might cause actual errors in a user                    their event-driven, asynchronous, and dynamic features. De-
environment. Although we have investigated static methods                         spite concerted efforts by developers, these features mean
for finding “potential faults” that seem to cause actual er-                      that not all possible behaviors of running applications can
rors if executed, developers need to confirm whether or not                       be predicted, and these difficult-to-predict behaviors may
the potential faults are actually executable. Herein, we pro-                     be error-prone. To find faults in Ajax applications, a state-
pose a mutation-based testing method implemented in a tool                        based approach may be more effective than navigation-model-
called JSPreventer. Even if the potential faults are not easily                   based, code-coverage, or black-box ones [5]. Several state-of-
executable in a given environment, our method mutates the                         the-art studies on state-based testing [6, 7, 8, 9] have effec-
applications until they are executable using two delay-based                      tively and efficiently found executable faults using the state
mutation operators to manipulate the timing of the appli-                         space of Ajax applications. However, the applications might
cations handling interactions. Thus, JSPreventer provides                         have faults that are not executed during testing if the faults
executable evidences of the not-easily-executable faults for                      cause actual errors only when specific conditions are met,
developers, if it reveals actual errors by testing the mutated                    such as unexpected user operations, Web browser behaviors,
applications. We applied our method to real-world applica-                        and network delays. Since it would be unrealistic to test the
tions and found actual errors that developers could debug                         applications under all possible conditions, these faults are
to improve their reliability. Therefore, JSPreventer can help                     not easily detectable even if developers use the state-of-the-
developers validate reliable real-world Ajax applications.                        art testing techniques. Consequently, users might encounter
                                                                                  erroneous behaviors in the applications when a user envi-
                                                                                  ronment meets the specific conditions.
Categories and Subject Descriptors                                                   In our previous studies [10, 11], we have investigated static
D.2.5 [Software Engineering]: [Testing and Debugging]                             methods for identifying Ajax application behaviors that seem
                                                                                  to cause errors when specific conditions are met (e.g., un-
Keywords                                                                          able to login using a correct username and password because
                                                                                  of improper use of asynchronous communications). Figure
Ajax; JavaScript; Web application testing; Software valida-
                                                                                  1 overviews our tool, called JSPreventer, that is based on
                                                                                  our previously proposed tool, called JSVerifier, and these
                                                                                  features are highlighted below.
                                                                                  Implementation We assume that Ajax design patterns [12],
  Asynchronous JavaScript and XML (Ajax) [1] make Web
                                                                                      which have been collected from observations in many
applications responsive [2]. Ajax applications improve us-
                                                                                      real-world Ajax applications, describe of the properties
ability because they can handle user events on the client
                                                                                      of the application behaviors expected by developers.
∗Shinichi Honiden and Hironori Washizaki are also affiliated                            Hence, the implemented Ajax design pattern informa-
to National Institute of Informatics, Japan.                                          tion (IADP info) can be leveraged to verify whether
Permission to make digital or hard copies of all or part of this work for             an Ajax application is running as expected.
personal or classroom use is granted without fee provided that copies are not
made or distributed for profit or commercial advantage and that copies bear       Extractor We assume that interactions with Ajax applica-
this notice and the full citation on the first page. Copyrights for components        tions can trigger changes in the application states. In-
of this work owned by others than ACM must be honored. Abstracting with               teractions correspond to how user events, asynchronous
credit is permitted. To copy otherwise, or republish, to post on servers or to        server responses, and timeouts are handled. Because
redistribute to lists, requires prior specific permission and/or a fee. Request       the source code contains these events and event han-
permissions from                                                 dlers, our extraction method [10] statically analyzes
ASE’14 , September 15 - 19 2014, Vasteras, Sweden
Copyright 2014 ACM 978-1-4503-3013-8/14/09 ...$15.00.                                 the relationships among interactions via a call graph                                             and constructs a finite state machine.
Validating Ajax Applications Using a Delay-Based Mutation Technique
Ajax Design Patterns                                † Implemented Ajax Design Pattern Information          This is because our mutation operators change non-
    Implementation                                                                                         functional aspects of Ajax applications, but the ap-
                                                                                                           plications can handle specific server responses after a
                Previous               IADP Info†                Test Data/Oracles     Proposed
                 Work                                                                  Extension           given delay time has elapsed. We expect that test-
               JSVerifier                                                            JSPreventer           ing the mutated source code with the given test data
                                                                                                           and oracles will reveal actual errors. Thus, JSPreven-
                                            Interaction                                                    ter provides developers with executable evidences of
                                            Invariants                                 … …

               Finite State Machines
                                                                                      …… ……
                                                                                     … … ……
                                                                                                           not-easily-executable faults during testing, i.e., subtle
                                                                                     …    …
                                                                                                           network delays are required for revealing actual errors.
                                                                          Mutated Source Code
                     Extractor                                                                      Debugging Outputs from JSVerifier and JSPreventer (i.e.,
                                 Faulty Interaction Sequences                                           an extracted finite state machine, identified faulty in-
                                                                           Actual Errors
                                                                                                        teraction sequences, and revealed actual errors) are
       … …
      …… ……
                                                                                     … …
                                                                                              …         used to debug the original source code until JSVerifier
                                                                                     … …      …
     … … ……                                                         Debugging
     …    …                                                                          …
                                                                                                        does not report any potential faults. JSPreventer can
Faulty Source Code                                                         Debugged Source Code         then be used for validating reliable Ajax applications.

               Figure 1: Overview of JSPreventer                                                       Compared with our previous studies [10, 11], the novelty
                                                                                                    of this study lies in revealing actual errors due to poten-
Formulator and Verifier Using the IADP info as a guide,                                             tial faults in Ajax applications. Since Ajax design patterns
    our verification method [11] instantiates interaction in-                                       aim to improve the usability of Ajax applications, code vi-
    variants, and the applications handle their interactions                                        olations against the design patterns do not always lead to
    in invariant occurrence and order. Whether the ex-                                              actual errors being debugged. Additionally, the faulty inter-
    tracted finite state machine satisfies the interaction                                          action sequences that are identified are counterexamples in
    invariants is automatically verified using the NuSMV                                            the extracted finite state machine. Counterexamples in an
    model checker1 . Finally, JSVerifier obtains faulty in-                                         abstract model can be spurious; therefore, it is important to
    teraction sequences from counterexamples of the verifi-                                         reanalyze them in an actual system [15]. Consequently, we
    cation results, e.g., {onload, init(), onclick, callback(),                                     argue that the novelty of our study is that it helps developers
    ... }, and reports the presence of “potential faults” that                                      validate Ajax applications.
    seem to cause actual errors if executed.                                                           Our contributions are:
   Although we claim that faulty interaction sequences are                                               • A mutation-based testing method that reveals actual
associated with potential faults reported by JSVerifier, the                                               errors due to potential faults in Ajax applications.
following two research questions remain unanswered.
RQ1 Can JSPreventer reveal actual errors caused by po-                                                   • Definitions of synchronous and asynchronous delay-
   tential faults in Ajax applications? This question can                                                  based mutation operators.
   be answered by testing whether actual errors are due
   to potential faults. However, the existing testing tech-                                              • An implementation of our method in JSPreventer.
   niques cannot reveal such errors because the potential
   faults might not be easily executable.                                                                • Case study experiments on real-world Ajax applica-
                                                                                                           tions demonstrating that JSPreventer can reveal ac-
RQ2 Can JSPreventer reveal these errors in a feasible amount                                               tual errors due to potential faults.
   of time? Although testing all possible scenarios in ev-
   ery environment should reveal all errors, such a method                                            This paper is organized as follows. Section 2 uses an exam-
   would be unrealistic.                                                                            ple to illustrate the challenge of Ajax application validation.
                                                                                                    Section 3 describes our mutation-based testing method, while
  This paper presents a mutation-based testing method to
                                                                                                    Section 4 evaluates our method. Section 5 covers related
answer the above questions.
                                                                                                    work, and Section 6 is the conclusion.
Revealer JSPreventer gets the faulty interaction sequences
    as input from JSVerifier and tries to find executable ev-
    idences of the potential faults. However, JSPreventer
                                                                                                    2.    MOTIVATING EXAMPLE
    may not easily execute Ajax applications on the faulty                                             To illustrate the aforementioned challenge, we implemented
    interaction sequences because a specified environment                                           a simple Ajax application called QAsite2 . QAsite is a typical
    does not meet specific conditions to reveal actual er-                                          Q&A website where users can ask and answer each other’s
    rors due to the potential faults. Therefore, we assume                                          questions, such as in Experts Exchange3 and Quora4 . In
    that an unexpected network latency [13] may make                                                implementing QAsite, we referred to the User Action, On-
    the potential faults executable. To emulate an un-                                              Demand JavaScript, and Direct Login patterns described
    expected network latency, we define synchronous and                                             in the Ajax design patterns [12].
    asynchronous delay-based mutation operators (Section                                               Figure 2 briefly depicts the source code, while Figure 3
    3). Although a program mutation technique is basi-                                              shows screenshots of QAsite, which runs as follows.
    cally used for injecting artificial faults [14], we lever-                                      2
                                                                                                      Running examples and analysis results are available from
    age the technique to allow JSPreventer to make po-                                    
    tential faults executable in the specified environment.                                         3
1                                                                                                   4                                                                            
Validating Ajax Applications Using a Delay-Based Mutation Technique
1                                                                 31   function t r y L o g i n ( r e s ) {
 2                    32         if ( isSuccess ( res ) ) {
 3      < s c r i p t t y p e =”t e x t / j a v a s c r i p t ”                33            // C r e a t e l o g o u t a n c h o r
 4           s r c =”j s / p r o t o t y p e . j s ”>          34            // D i s a b l e l o g i n form
 5      < s c r i p t t y p e =”t e x t / j a v a s c r i p t ”>
18               p a r a m e t e r s : ” t a s k=g e t s e e d ” ,             48    . . .
19               onSuccess : handleHttpGetSeed }) ;                            49   QAsite  . . .
20       }                                                                     50   Q u e s t i o n  . . .
21   }                                                                         51   Answers  . . .
22   function h a n d l e H t t p G e t S e e d ( r e s ) {                    52       Good! 
23       /∗∗ S e t r e s p o n s e t o ” s e e d ” v a r i a b l e ∗∗/ }       53    ...
24   ...                                                                       54   
25   function v a l i d a t e L o g i n ( ) {                                  55    ...
26      new Ajax . R e q ues t (LOGIN PREFIX ,                                 56   L o g i n / out  . . .
27           method : ”GET” ,                                                  57       
28           p a r a m e t e r s : ” t a s k=c h e c k l o g i n & . . . ” ,   58       
29           onComplete : t r y L o g i n } ) ;                                59       
30   }                                                                         60   . . . < / body>

                                         Figure 2: Source code of our motivating example: QAsite

                                                              Figure 3: Screenshots of QAsite

                                                         Table 1: Actual errors in the QAsite
       FIS#        Ajax design pattern             Brief explanation of erroneous behavior                                      Actual error
       1           User Action                     QAsite might not prevent multiple calls to validateLogin                     Unnecessary
                                                   when a user unexpectedly double-clicks on the login button.                  server sessions
       2           On-Demand                       QAsite can handle onGood before successful login                             Invalid user
                   JavaScript                      due to a dead link to the mask image file.                                   operations
       3           Direct Login                    QAsite can send requests for login attempts without seed data.               Invalid login
                                                   Then, a user cannot log in with his or her username and password.            attempts
Validating Ajax Applications Using a Delay-Based Mutation Technique
Faulty interaction sequence #3
                                                                                                              Executed interaction sequence #3

                                                                                                                (F-1)&& !hasSeed]
                                                                                                                      Without seed data


                                                                                                                             (E) Immediate resp.
                                                                                       down                      handleHttpGetSeed
                                                    (F-2) QAsite sends account info.   validateLogin                   onmousedown

                                 Figure 4: Finite state machine extracted from QAsite

(i) Page load: Upon a page load, QAsite registers user                 therefore, JSVerifier can find all the interactions in a static
     event handlers for the login form and the “Good!” but-            manner. For example, in Figure 2 (line 8), JSVerifier finds
     ton (lines 7-12). The User Action pattern notes such              that QAsite changes its state when handling the onfocus
     implementations so that Ajax applications can handle              event. To construct a finite state machine based on the in-
     user events anytime. Additionally, the QAsite masks               teractions, JSVerifier leverages a call graph that represents
     the answers (line 54) to prohibit guest users from view-          function caller-callee relationships because the interactions
     ing them. This implementation is similar to the On-               are function callbacks in response to events. Thus, JSVerifier
     Demand JavaScript pattern, in which QAsite should                 outputs an interaction-based finite state machine containing
     handle the click on the “Good!” button after users                application behaviors that are independent of the testing en-
     login.                                                            vironment. Figure 4 shows a finite state machine extracted
                                                                       from QAsite.
(ii) Login: We implemented the login form according to                    Additionally, JSVerifier runs the NuSMV model checker
      the Direct Login pattern. When a user sets the cursor            to verify the correctness of the extracted finite state machine
      on the input forms (lines 8-9), QAsite determines that           (Verifier in Figure 1). However, the model checker cannot
      the user intends to log in, and QAsite asynchronously            determine whether or not the behaviors of Ajax applica-
      retrieves seed data from the server (lines 14-23). When          tions are ones developers expect. Fortunately, Ajax design
      a user clicks on the login button (line 10), QAsite sends        patterns are based on observations in many real-world Ajax
      the username and a hash value with the password and              applications. Therefore, we assume that the design patterns
      seed data for secure communications to validate the              present the properties of the application behaviors expected
      account on the server side (lines 25-30). After receiving        by developers. Hence, we define invariants to be verified
      the validation result (line 29), QAsite creates a logout         from the behavior properties relevant to the interactions.
      widget and disables the login form if the information            For example, the Direct Login pattern indicates that QA-
      is valid (lines 32-35).                                          site should receive the seed data before sending the account
(iii) Answer view: While it is creating the logout widget,             information.
      QAsite removes the mask (lines 32-35). The user can                 JSVerifier also uses the property pattern mappings5 , which
      then view the answers and click the “Good!” button               provide template formulas for finite-state verification, to ex-
      (lines 11 and 39-40). When the user logs out, QAsite             press invariants in verification formulas. As template param-
      disables the logout widget, enables the login form, and          eters, JSVerifier requires developers of functions or events
      masks the answers again (lines 42-46).                           where they implement Ajax design patterns (IADP info)
                                                                       so that it instantiates verification formulas (Formulator in
2.1 Results of Our Previous Studies                                    Figure 1). As an example of the Direct Login pattern in
   Although QAsite seems to run as expected, JSVerifier pro-           QAsite, developers input the handleHttpGetSeed and val-
posed in our previous studies [10, 11] reports the presence            idateLogin functions (lines 22 and 25 in Figure 2) as the
of three potential faults, which are listed in Table 1. Herein,        IADP info. From a counterexample of the verification re-
we explain how JSVerifier works.                                       sults, JSVerifier then identifies faulty interaction sequence
   JSVerifier statically extracts a finite state machine from          #3 regardless of whether or not it is executable in a testing
the source code of Ajax applications (Extractor in Figure              environment (visualized by the bold solid arrow in Figure
1). In our static extraction method, we focus on interactions          4). Thus, JSVerifier reports the presence of potential faults
with Ajax applications, such as mouse clicks and server re-            in Ajax applications that seems to cause actual errors if ex-
sponses, as triggers that can change the application states.           ecuted.
The interactions are implemented at event handlers for user
events or asynchronous communications in the source code;        
Original code             Revealing errors due to potential faults                               1 
                        (Step 1-a)                                                                2 ...
 Faulty Interaction                                                    Executed Interaction       3 < s c r i p t s r c =”h a n d l e C l i c k . j s ” />
 Sequences (FISs)                      Executing on FISs                Sequences (EISs)

    Test data                                                    Yes    Executable            Figure 6: Loading a JavaScript file with improper
                                          FISs = EISs                   faults (EFs)          timing
                        (Step 2)            No
                                   Applying mutation operators         Mutated code           3.1        Executing Faulty Interaction Sequences
                        (Step 1-b)                                                              Because executing Ajax applications results in actual er-
                                       Executing on FISs                    EISs
                                                                                              rors if there are any, JSPreventer first attempts to execute
                                                                       Delay-dependent        the applications in accordance with the identified faulty in-
                                                                 Yes    potential faults
                                          FISs = EISs                      (DdPFs)            teraction sequences (Step 1-a in Figure 5). Although JSPre-
                                            No                                                venter uses the Selenium WebDriver6 in order to make its
                                                                         *-dependent          execution as automated as possible, developers must input
                           Yes            Branch point           No     potential faults
                                            found?                         (*dPFs)            test data that cannot be inferred by any tools, such as a
  Test oracle           (Step 3)                                                              correct username and password, in order to test a successful
                                      Testing mutated code             Actual errors          login scenario. JSPreventer executes applications with test
                                                                                              data and obtains the executed interaction sequences from
                                              End                                             the execution results. If the executed interaction sequences
                                                                                              and faulty interaction sequences are identical, JSPreventer
                      Figure 5: JSPreventer workflow                                          determines that potential faults in the applications are exe-
                                                                                              cutable faults against the implemented Ajax design pat-
2.2 The Challenge                                                                             terns. As previously mentioned, executable faults can be
   The potential faults reported by JSVerifier can be exposed                                 revealed through state-of-the-art methods.
under a specific set of conditions but have little chance of be-                                If the potential faults are not executed, JSPreventer fo-
ing found if the set is unknown. A problem with such faults                                   cuses on the differences between the faulty interaction se-
is that developers have trouble detecting them during testing                                 quences and the executed interaction sequences. Differences
because they cause errors only when complicated conditions                                    may occur because Ajax applications cannot handle certain
are met [16]. Therefore, our challenge in this study is to                                    interactions in faulty interaction sequences. The solid and
reveal actual errors due to faults in developers’ testing envi-                               dashed arrows in Figure 4 respectively represent faulty inter-
ronments before users encounter erroneous behaviors in the                                    action sequence #3 and its executed interaction sequence.
applications.                                                                                 At the meshed circle in the figure, WebDriver emulates the
   As an example of faulty interaction sequence #3, if QA-                                    onfocus user action, but QAsite actually handles the onSuc-
site is validated with browsers in a reliable network (e.g.,                                  cess event because of an immediate server response. How-
with Firefox and Chrome browsers in a local host environ-                                     ever, the ability to manipulate the timing of the applications
ment), it can immediately receive a response to obtain the                                    handling the interactions allows Ajax applications to run as
seed data [(E) in Figure 4] and cannot run along with faulty                                  faulty interaction sequences. Thus, a mutation technique
interaction sequence #3. However, an unexpected network                                       can be used to manipulate the timing.
latency that occurs when the seed data is being obtained en-                                  3.2        Applying Mutation Operators
ables QAsite to send account information without the seed
data [(F-1) and (F-2) in Figure 4], which prevents the user                                     By mutating the source code of the Ajax applications,
from logging in with his or her correct username and pass-                                    JSPreventer enables them to run on faulty interaction se-
word. Despite concerted efforts by developers, identifying                                     quences in a given environment (Step 2 in Figure 5). We
unexpected conditions, such as specific communication de-                                     assume that an unexpected network latency might allow
lays in the unpredictable context of a running application,                                   potential faults to be executable and define the following
is still extremely difficult. Consequently, JSPreventer is de-                                  delay-based mutation operators.
signed to systematically test whether Ajax applications run
as expected on faulty interaction sequences that might not                                    Synchronous delay: A Web browser parses the HTML
be easily executable in the given environments.                                                   code from top to bottom while loading a Web page.
                                                                                                  When the browser finds a script element whose src
                                                                                                  attribute has the location of an external JavaScript
3. OUR APPROACH                                                                                   file, it begins loading the JavaScript file. Because the
   Figure 5 depicts the three-step process that JSPreven-                                         JavaScript code can dynamically manipulate the Web
ter uses for validating Ajax applications. JSPreventer ini-                                       page content, the browser blocks render the remain-
tially executes Ajax applications on the faulty interaction                                       ing HTML code until the JavaScript file is completely
sequences identified by JSVerifier (Step 1-a). If the poten-                                      loaded. Therefore, script elements can be implemented
tial faults associated with these sequences are not easily exe-                                   at the bottom of the HTML code in order to render all
cutable, JSPreventer mutates the source code until the Ajax                                       page elements as fast as possible. This is a well-known
applications do execute the potential faults (Steps 2 and 1-                                      way to increase the perceived performance [17].
b). Then, developers test for the unexpected behavior due to
                                                                                                        Loading a JavaScript file at the improper time may
the potential faults by using the mutated applications (Step
                                                                                                        cause an actual error. Figure 6 shows brief code frag-
3). Additionally, JSPreventer classifies potential faults into
                                                                                                        ments to explain our synchronous delay mutation op-
executable, delay-dependent potential, and *-dependent po-
tential faults, which are defined in Section 3.2.                                       
1   // O r i g i n a l c o d e                                                  Our mutation operators do not change the functionalities of
 2   $ . g e t ( ” m y s c r i p t . php ” , //    URL                           Ajax applications because the applications can still handle
 3       { name : ”Yuta ” } ,                //    query s t r i n g
 4       function ( d a t a ) {              //    callback
                                                                                 a specific interaction after the given delay has elapsed.
 5           a l e r t ( ” Response i s ”          + data ) ;                       To determine where to apply our mutation operators, JSPre-
 6       }) ;                                                                    venter searches for a branch point between the faulty inter-
 7                                                                               action sequences and executed interaction sequences, such as
 8   // Mutated c o d e
 9   new D e l a y e d R e q u e s t ( 3 0 0 0 ) . a p p l y F u n c t i o n (   the meshed circle in Figure 4. The branch point may pre-
10     $ . get ,                         // t a r g e t f u n c                  vent Ajax applications from running on the faulty interac-
11     ” m y s c r i p t . php ” ,       // URL                                  tion sequences. The mutation operators are then applied to
12     { name : ”Yuta ” } , // q u e r y s t r i n g
13     function ( d a t a ) { // c a l l b a c k                                 an interaction in the executed interaction sequences at this
14         a l e r t ( ” Response i s ” + d a t a ) ;                            point, e.g., the onSuccess event handler. Even if the server
15     }) ;                                                                      immediately responds, QAsite does not handle the server
                                                                                 response within the delay period. By iterating Steps 1-b
       Figure 7: Example of our wrapper function
                                                                                 and 2 in Figure 5 until the executed interaction sequences
                                                                                 equal the faulty interaction sequences, JSPreventer artifi-
        erator. In this application, developers implement the                    cially sets up conditions in which potential faults can be ex-
        handleClick function in the handleClick.js JavaScript                    posed. Finally, it classifies the potential faults that become
        file and set the function as the onclick event handler                   executable by applying our mutation operators as delay-
        of the button element. Following common practice,                        dependent potential faults. Thus, JSPreventer provides
        the script element is then implemented at the bottom                     developers with executable evidences of the delay-dependent
        of the HTML code. However, an unexpected network                         potential faults which are not-easily-executable faults be-
        latency may delay loading of the JavaScript file. The                    cause they are not executed without subtle network delays.
        application cannot respond to the onclick event oc-                         However, not all applications can be mutated to produce
        currence because this scenario allows a user to click a                  identical executed interaction sequences and faulty interac-
        button even though the browser has yet to register the                   tion sequences. Faulty interaction sequence #2 in Table
        event handler.                                                           1 means that QAsite can call the onGood function without
        To emulate a JavaScript file loading delay due to an                     receiving the server response for login. Although JSPre-
        unexpected network latency, JSPreventer inserts an ar-                   venter injects the same artificial delay as in the case of
        tificial script element whose src attribute is located in                faulty interaction sequence #3, the “Good!” button can-
        our server-side program. This program then sends an                      not be clicked because the mask works as expected. This
        HTTP response with a given delay time, causing a                         means that JSPreventer cannot make the potential faults
        synchronous delay. This HTTP response consists of                        executable. JSPreventer classifies these potential faults as
        header and body sections in which are set “content-                      *-dependent potential faults, which might cause actual
        type: application/javascript” and an empty string, re-                   errors for reasons other than delayed injection.
        spectively. In this manner, JSPreventer can manip-
        ulate the timing of handling interactions relevant to                    3.3    Testing Mutated Code
        page loading, such as an onload event.                                      Because developers implement Ajax design patterns ac-
                                                                                 cording to their intentions, we assume that they can also
Asynchronous delay: Although the most significant fea-                           implement test oracles to verify whether the patterns are
    ture of Ajax applications is the asynchronous commu-                         correctly implemented in Ajax applications or not. With
    nications between the client and server, an unexpected                       such test oracles, JSPreventer can determine whether the
    network latency can significantly impact this feature.                       mutated code has passed the test (Step 3 in Figure 5). If
    Unlike a synchronous delay, an unexpected network                            the code fails a test, developers can determine the actual
    latency does not inhibit rendering of Web page con-                          errors from the test results.
    tent or handling of user interactions. Consequently,                            JSPreventer outputs actual errors in addition to a finite
    it is extremely difficult to consider all possible states                      state machine and faulty interaction sequences from JSVer-
    where Ajax applications may have to handle a delayed                         ifier. We assume that these outputs will help developers
    asynchronous server response.                                                debug the original source code because they allow the com-
        To emulate a handling delay with an asynchronous                         plex behaviors of Ajax applications to be understood, the
        server response, JSPreventer rewrites the correspond-                    faulty behaviors of a finite state machine to be identified,
        ing JavaScript code fragment by using our wrap func-                     and unexpected behaviors due to potential faults to be ob-
        tion, DelayedRequest, as shown in Figure 7. JSPre-                       served. Thus, developers can use JSPreventer to prevent
        venter finds a target function for an asynchronous server                actual errors due to potential faults in Ajax applications.
        request in the original code and parses its arguments,
        such as URL, query string, and callback function. In-                    3.4    Use Scenario of JSPreventer
        stead of the original code, JSPreventer inserts an in-                      Figure 8 illustrates a JSPreventer use scenario based on
        stantiation statement of the DelayedRequest with a                       the results of our motivating example, assuming that JSPre-
        given delay time, e.g., 3000 msec. The DelayedRequest                    venter is applicable in the context of iterative and incre-
        sends the same asynchronous server request as the orig-                  mental development [18]. In fact, developers often select
        inal one, but does not invoke the callback function even                 a simplified and rapid iteration development life cycle [19].
        if a response is immediately received. After the given                   The scenario in JSPreventer involves five steps: (i) the im-
        delay time has elapsed, the DelayedRequest invokes                       plemented Ajax design pattern information (IADP info) is
        the callback function.                                                   initially inputted to JSVerifier; (ii) JSVerifier suggests faulty
(i) Input IADP Info                                 (ii) Notice potential faults          Faulty Interaction Sequence


       (iii) Input test data and oracles

                                                                                  (v) Confirm correctness and reliability

    (iv) Observe and debug actual errors

                                            Figure 8: JSPreventer use scenario

interaction sequences in the applications against the IADP                optical accessory supplier website was found at the top
info and reports the presence of potential faults. (iii) Devel-           of the search results.
opers provide test data and oracles to JSPreventer; (iv) to
reveal unexpected runs, JSPreventer mutates and tests the         UCDChina and ESA Additionally, we searched for “on-
applications. In this step, if the potential faults become to        blur=checkInput” and “onsubmit=validate”, which are
be executable with subtle network delays, we assume that             representative of the Live Form pattern. The Live
the executable evidences and the revealed actual errors can          Form pattern suggests that Ajax applications should
be of help in validating and debugging the applications; (v)         check form data before making a submission. From
through iterative development, highly reliable Ajax applica-         the search results, we found Chinese and British com-
tions can be released.                                               panies’ portal websites.

4. EVALUATION                                                     4.2      Experimental Setup
  We conducted case studies and evaluated the usefulness of          Our case study experiments were conducted on a 64-bit
JSPreventer by answering the following research questions.        Mac OS X 10.9.2 machine with an Intel Core i5 (2.3GHz)
                                                                  and 16 GB of memory. First, we saved complete Web pages
RQ1 Can JSPreventer reveal actual errors caused by po-            of the subjects by using Mozilla Firefox and obtained HTML,
   tential faults in Ajax applications?                           CSS, and JavaScript source code files. Then, we leveraged
RQ2 Can JSPreventer reveal the errors in a feasible amount        Code Beautifier10 to count lines of these codes, as listed in
   of time?                                                       Table 2 (HT M L, CSS, and JS). The 1K-10K lines of code
                                                                  range represents medium-largish Ajax applications. Note
4.1 Subject Ajax Applications                                     that only client-side source code could be obtained; there-
  Table 2 shows the Ajax applications which we used in            fore, we implemented mock server-side scripts in PHP.
our case studies. To find real-world Ajax applications in            Next, we used JSVerifier to extract the finite state ma-
which developers had implemented Ajax design patterns, we         chine from each subject. Table 2 lists the extraction time
searched by using keywords relevant to Ajax design patterns       (Te ) and the numbers of states and transitions of the finite
with NerdyData7 , which provides a search engine for the          state machine (#states and #trans.).
source code on websites.                                             Table 3 lists the interaction invariants used in our case
2020m We searched for “login controller.js” where James           studies. The interaction invariants to be satisfied were de-
    Dam8 implemented the Direct Login pattern9 . The              termined for each subject (Table 4). In the source code of
                                                                  all the subjects, we found implementations of asynchronous                                                   communications and page load event handlers. Therefore,
  Internet Archive:
9                                                                 10
  Ajax Login System Demo:                      
Table 2: Subject Ajax applications
      Subject        URL                                      HTML CSS JS          #states              #trans.    Te (sec)   Tv (sec)
      2020m                              188           420   2468    15       57         9.434      0.344
      UCDChina                           1978          898   880     32       508        14.166     0.624
      ESA           4751          964   3523    34       602        17.705     0.657

                            Table 3: Interaction invariants derived from Ajax design patterns
      #     Expected application behavior                          Ajax design pattern  Pattern category
      1     Handling user events during asynchronous communications             XMLHttpRequest Call      Fundamental Technology
      2     Registering user event handlers at page load                        User Action              Fundamental Technology
      3     Validating form data before submission                              Live Form                Functionality and Usability
      4     Retrieving seed data before login attempt                           Direct Login             Functionality and Usability
      5     Disabling login form after successful login                         Direct Login             Functionality and Usability

we determined that Invariants #1 (XMLHttpRequest Call)                    verse Lookup12 might allow interceptors to reverse-engineer
and #2 (User Action) should be satisfied in all the subjects.             the hashed value, obtain the user password, and log in 2020m
Additionally, considering the search keywords, we also deter-             illegally. Thus, we could infer a vulnerability to intrusion.
mined that Invariants #4 and #5 (Direct Login) in 2020m                      Additionally, JSPreventer revealed that UCDChina sent
and Invariant #3 (Live Form) in UCDChina and ESA should                   an empty text on its search form, despite developers having
be satisfied. JSVerifier then verified all the invariants for             incorporated a goSearch function to prevent it. A cause of
each subject and reported the verification time (Tv in Table              this erroneous behavior was that UCDChina could display
2) and the verification results (Resultv in Table 4).                     the search form before loading the code fragment containing
  If the verification result was incorrect, JSPreventer exe-              the goSearch function. Since the search form is the source
cuted the classification workflow shown in Figure 5. We set               where users enter the search query, developers in this case
the delay time to be 3 seconds. Additionally, we provided all             had intended to improve the search form. Although they had
the necessary test data and oracles, such as the correct user-            expected only valid search queries would be sent from the
name and password, for successful logins in 2020m. Finally,               search form, the error indicates that UCDChina did not val-
JSPreventer classified the potential faults that caused incor-            idate the search query at all, and therefore, it possibly sent
rect verification results into executable, delay-dependent po-            every search query. If so, there would be an additional pos-
tential, or *-dependent potential faults. The Resultc column              sibility that a server would receive malicious search queries
in Table 4 corresponds to these classification results. JSPre-            (e.g., an SQL code fragment) that might lead to an SQL
venter also reported the execution, mutation, and testing                 injection attack.
times (Tx , Tm , and Tt ).                                                   JSPreventer could find an undefined addthis_close func-
  Additionally, we conducted comparative experiments with                 tion call in ESA by using the synchronous delay operator
Crawljax11 , which is a state-of-the-art tool for finding faults          upon loading an addthis_widget.js file. The undefined
in Ajax applications [6, 8, 20, 9]. The default setting of                function calls can be fatal errors, which might cause ESA to
Crawljax could not find any delay-dependent potential faults              crash.
in the subjects.                                                             Although Crawljax ran until it completed its exploration
                                                                          of the state space, it did not reveal the actual errors found by
4.3 Results and Discussions                                               JSPreventer. These results suggest that JSPreventer reveals
  Revealing actual errors due to delay-dependent                          actual errors due to delay-dependent potential faults.
potential faults (RQ1): JSPreventer could classify po-                       Feasible analysis time (RQ2): For each test subject,
tential faults reported by JSVerifier into executable, delay-             JSPreventer revealed the actual errors within one minute.
dependent potential, and *-dependent potential faults. As                 Most of the Te , Tx , and Tt values were required to initial-
indicated in Table 5, it found actual errors in the subjects.             ize parsers and to launch the test browser. Although Tv in-
We reviewed the source code of the subjects and confirmed                 creased linearly with the size of the extracted finite state ma-
that the actual errors could be exposed.                                  chine, the extracted finite state machines were small enough
  In 2020m, users could not log in even with their correct                to be verified with the NuSMV model checker. Tm was much
username and password because 2020m sent the login re-                    shorter than the other amounts. These results indicate that
quest with the initial value at the declaration statement                 JSPreventer is practical.
of the seed variable. For secure password hashing, 2020m                     False positives in *-dependent potential faults: In
used the MD5 algorithm to a value obtained by adding                      our case studies, three potential faults could not be executed
the seed variable to the user password. Although 2020m                    using JSPreventer and the delay-based mutation operators.
should use the seed variable generated at runtime, it hashed              As for the potential faults at Invariants #2 and #5 in 2020m,
the value with the initial value “0”; hence, 2020m sent the               we observed that Mozilla Firefox prevented 2020m from run-
hashed value using the raw user password. The MD5 Re-                     ning on the identified faulty interaction sequences by making

11                                                                        12                                                    
Table 4: Classification results of our case studies
        Subject         Invariant# (Ajax design pattern)      Resultv           Tx (sec)    Tm (sec)   Tt (sec)   Resultc +
        2020m           1 (XMLHttpRequest Call)               Correct           -           -          -          -
                        2 (User Action)                       False positive    9.643       0.039      11.868     *dPF
                        4 (Direct Login)                      Potential fault   8.147       0.116      8.175      DdPF
                        5 (Direct Login)                      False positive    8.553       -          -          *dPF
        UCDChina        1 (XMLHttpRequest Call)               Correct           -           -          -          -
                        2 (User Action)                       Potential fault   11.677      0.183      11.816     DdPF
                        3 (Live Form)                         Potential fault   9.692       -          -          EF
        ESA             1 (XMLHttpRequest Call)               Correct           -           -          -          -
                        2 (User Action)                       Potential fault   10.520      0.133      10.335     DdPF
                        3 (Live Form)                         False positive    15.307      0.208      14.965     *dPF
         +   EF, DdPF, and *dPF in the column of Resultc represent executable, delay-dependent potential, and *-dependent
             potential faults, respectively.

                               Table 5: Actual errors due to delay-dependent potential faults
                     Subject        Brief explanation                          Inferred vulnerability
                     2020m          Login failed with correct username and password        Intrusion
                     UCDChina       Content search with an empty query                     SQL injection
                     ESA            Undefined function call                                Application crash

a proper rendering block and page transition, respectively.          4.4 Threats to Validity
Since the extracted finite state machine did not contain the            Internal validity threats: We considered two exter-
Web browser behavior, we determined that these potential             nal factors that might affect results in our case study ex-
faults were false positives due to spurious counterexamples.         periments. The results from 2020m, UCDChina, and ESA
Additionally, JSPreventer could not make the potential fault         demonstrate the usefulness of JSPreventer because these
at Invariant #3 in ESA executable. In the source code of             real-world applications were obtained via a public search en-
ESA, we found that a conditional branch was implemented              gine provided by NerdyData. However, their actual server-
to prevent ESA from executing the potential fault. Analyz-           side scripts were not available; therefore, we implemented
ing data-dependent impossible behavior is currently beyond           mock server-side scripts in order to run these real-world ap-
the scope of our research; hence, this result was a false posi-      plications on our machine. Because the mock server-side
tive due to our methods. However, this problem can be miti-          scripts may be a threat to internal validity, we intend to
gated by using execution results to refine the extracted finite      conduct additional case studies using real-world open-source
state machine, i.e., by conducting further dynamic analy-            Ajax applications.
ses together with JSPreventer. A remaining issue is that                Additionally, we provided the necessary information to
it is unknown whether Ajax applications actually execute             run the test subjects (e.g., IADP Info, test data, and or-
*-dependent potential faults. Therefore, we are interested           acles). Although the ability of developers to input correct
in defining additional effective mutation operators to make           information into JSPreventer may affect the internal valid-
*-dependent potential faults executable.                             ity, the results of our case studies show that JSPreventer
   Applying to generic Ajax applications: We designed                can reveal actual errors due to potential faults. It should
JSPreventer for developers who use Ajax design patterns to           be noted that the test scenarios involved typical tests, such
build applications. In actual Ajax application development,          as verifying user login success with the correct username
developers have their own design patterns. If there are not          and password. In the future, we intend to use JSPreventer
given IADP info for the own design patterns, JSPreventer             in actual Ajax application development projects to evalu-
cannot determine code locations where our mutation opera-            ate whether developers can input correct information into
tors should be applied. To test Ajax applications made with          JSPreventer. We are also interested in combining a search-
their own design patterns, developers need to specify the            based testing technique [21] with JSPreventer in order to
code locations of synchronous or asynchronous communica-             generate test data automatically.
tions which they suspect as the cause of errors when specific           External validity threats: Regarding the generality
network delays are present. JSPreventer then mutates the             of our approach, JSPreventer can only reveal actual errors
code fragments at the specified code locations and tries to          due to delay-dependent potential faults. However, potential
reveal the errors.                                                   faults may depend on other reasons, such as unexpected user
   Debugging potential faults: We assume that the ac-                operations and Web browser behaviors. Therefore, we are
tual errors revealed by JSPreventer can help developers de-          going to collect *-dependent potential faults by conducting
bug potential faults. However, this debugging task depends           additional case studies with JSPreventer and define effective
on the skills and experience of the developers. In the future,       mutation operators for them.
we plan to establish a method to support debugging.
Although 2020m, UCDChina, and ESA are practical Ajax              Zheng et al. also conducted a rules-based static analy-
applications, it would be interesting to determine the scala-     sis to detect data races due to asynchronous calls (i.e., the
bility of JSPreventer by obtaining experimental results using     sort of interactions dealt with in this paper) in Ajax applica-
many real-world Ajax applications.                                tions [13]. Although they focused on global variables, which
                                                                  are shared by asynchronous calls, JSPreventer can identify
                                                                  faulty asynchronous calls without data races by leveraging
5. RELATED WORK                                                   the Ajax design patterns as oracles. Additionally, Zheng et
   In this section, we describe the state-of-the-art studies on   al. were not interested in testing unexpected behaviors in
the analysis and testing of Ajax applications.                    the applications if a detected data race occurred. Although
   State-based analysis and testing: Marchetto et al.             they suggested two methods to fix data races, we argue that
introduced the concept of state-based analysis and testing        the actual errors found by JSPreventer can help developers
to Ajax applications [22]. Their tool, ReAjax[23], extracts       debug their applications and confirm the correctness of the
finite state machines from the execution results of Ajax ap-      codes.
plications. The finite state machines consist of Document            Guarnieri et al. introduced a pure static taint analysis
Object Model (DOM)13 instances and the effects of callback         for JavaScript code in order to identify security vulnera-
executions as states and transitions. However, ReAjax re-         bilities such as cross-site scripting and SQL injection [28].
quires developers to manually execute Ajax applications to        Wei et al. pointed out the dynamic feature of JavaScript;
sufficiently trace the execution results.                           JavaScript code can dynamically get additional code from
   Towards automated execution, Mesbah et al. implemented         the server at runtime. They proposed a blended taint anal-
Crawljax [24], which can automatically emulate user ac-           ysis of the JavaScript code that can be collected by executing
tions, i.e., crawling Ajax applications. Crawljax has been        test cases [29]. Although these analyses can output precise
used to detect DOM-related faults [6], to test cross browser      control flows of only the JavaScript code, JSPreventer ana-
compatibility [8], and to determine whether CSS rules are         lyzes stateful behaviors in the HTML, CSS, and JavaScript
unnecessary or not [20]. Additionally, Choudhary et al.           code of Ajax applications.
leveraged Crawljax to improve the accuracy of identifying            Mutation analysis and testing: Mutation analysis is
cross browser issues [9]. Although Crawljax is very useful        a fault-based technique to assess the adequacy of test cases
for finding executable faults, such crawling approaches can-      [14, 30]. It seeds artificial faults into the program under test
not reveal potential faults because they may not be easily        and then measures whether the seeded artificial faults can
executable in the given environments.                             be detected in the test cases. This technique in its original
   Arzti et al. presented Artemis [7] as a way to improve         form might be applicable to Ajax applications [31, 32]. In
code coverage by using the feedback-directed technique [25].      contrast, JSPreventer leverages a deformed mutation analy-
Artemis analyzes historical test execution data and gener-        sis technique, wherein seeded artificial delays do not act as
ates test cases to explore the state space of Ajax applica-       faults to be detected but expose existing potential faults in
tions. The authors noted that Artemis may help Crawljax           Ajax applications. Although JSPreventer cannot be used to
determine what user actions should be emulated. However,          assess the adequacy of test cases, it can reveal actual errors
Crawljax and Artemis cannot be combined in a way that             due to potential faults.
would determine potential faults in Ajax applications.
   Hong et al. have a similar concern of finding concurrency
errors due to the improper timing of Ajax applications han-       6.      CONCLUSION
dling the interactions [26]. Their tool called WAVE is an ex-
                                                                     We proposed a support tool, called JSPreventer, for vali-
tended implementation of the WebKit14 browser framework
                                                                  dating Ajax applications by finding executable evidences of
for managing event invocations on the JavaScript engine.
                                                                  potential faults that may not be easily executed in the given
However, the HTML rendering engine can also affect event
                                                                  development and test environments. We implemented delay-
invocations, such as the rendering block, as we mentioned in
                                                                  based mutation operators to artificially make the poten-
Section 3.2. Since JSPreventer reveals errors due to poten-
                                                                  tial faults executable and reveal actual errors due to delay-
tial faults of Ajax applications on standard Web browsers
                                                                  dependent potential faults. Our experimental results show
that users actually use, it can more precisely suggest erro-
                                                                  that JSPreventer reveals actual errors due to delay-dependent
neous behavior of the applications to developers.
                                                                  potential faults in real-world Ajax applications in a feasible
   Static approach: As for static approaches, Guha et
                                                                  amount of time. By debugging the revealed errors, devel-
al. proposed a static method to prevent Ajax applications
                                                                  opers can develop and validate highly reliable Ajax applica-
from handling invalid server requests [27]. Their frame-
work constructs a request graph through control flow anal-
                                                                     Our future research will proceed in three separate direc-
ysis, which represents how Ajax applications handle asyn-
                                                                  tions. One direction is to develop automated debugging.
chronous server requests with an invariant order. The re-
                                                                  Besides establishing a method that effectively uses solutions
quest graph can be used as an oracle to validate runtime
                                                                  in Ajax design patterns, we plan to combine JSPreventer
server requests. However, if developers cannot correctly im-
                                                                  with mechanisms for determining actual errors in existing
plement Ajax applications, the request graph cannot reject
                                                                  debugger tools, such as Firebug15 . The second direction is
invalid server requests relevant to potential faults. Thus,
                                                                  to implement more mutation operators to reveal actual er-
developers debugging the potential faults using JSPreventer
                                                                  rors due to *-dependent potential faults. The third direction
can construct a more proper request graph from the de-
                                                                  is to expand the range of case studies on large-scale Ajax ap-
bugged applications.
                                                                  plications in the real world.
14                                                                15                                          
7. REFERENCES                                                 [17] Steve Souders. High Performance Web Sites. O’Reilly
 [1] Jesse James Garrett. Ajax: A New Approach to Web              Media, Inc., 2007.
     Applications.                [18] C. Larman and V.R. Basili. Iterative and Incremental
     ajax-new-approach-web-applications.                           Developments: A Brief History. Computer,
 [2] Brent Stearn. XULRunner: A New Approach for                   36(6):47–56, 2003.
     Developing Rich Internet Applications. IEEE Internet     [19] Anthony T. Holdener, III. Ajax: The Definitive Guide.
     Computing, 11(3):67–73, 2007.                                 O’Reilly Media, Inc., 2008.
 [3] Jakob Nielsen and Hoa Loranger. Prioritizing Web         [20] Ali Mesbah and Shabnam Mirshokraie. Automated
     Usability. New Riders Press, Berkeley, CA, 2006.              Analysis of CSS Rules to Support Style Maintenance.
 [4] Internet World Stats. World Internet Usage Statistics         In Proc. Int’l Conf. on Software Engineering (ICSE),
     News and World Population Stats.                              pages 408–418, May 2012.                    [21] Nadia Alshahwan and Mark Harman. Automated Web
 [5] Alessandro Marchetto, Filippo Ricca, and Paolo                Application Testing Using Search Based Software
     Tonella. A case study-based comparison of web testing         Engineering. In Proc. Int’l Conf. on Automated
     techniques applied to AJAX web applications. Int’l            Software Engineering (ASE), pages 3–12, November
     Journal on Software Tools for Technology Transfer             2011.
     (STTT), 10(6):477–492, 2008.                             [22] Alessandro Marchetto, Paolo Tonella, and Filippo
 [6] Ali Mesbah and Arie van Deursen. Invariant-based              Ricca. State-based Testing of Ajax Web Applications.
     Automatic Testing of AJAX User Interfaces. In Proc.           In Proc. Int’l Conf. on Software Testing, Verification
     Int’l Conf. on Software Engineering (ICSE), pages             and Validation (ICST), pages 121–130, April 2008.
     210–220, May 2009.                                       [23] Alessandro Marchetto, Paolo Tonella, and Filippo
 [7] Shay Artzi, Julian Dolby, Simon Holm Jensen, Anders           Ricca. ReAjax: a reverse engineering tool for Ajax
     Moller, and Frank Tip. A Framework for Automated              Web applications. IET Software, 6(1):33–49, 2012.
     Testing of JavaScript Web Applications. In Proc. Int’l   [24] Ali Mesbah, Engin Bozdag, and Arie van Deursen.
     Conf. on Software Engineering (ICSE), pages                   Crawling AJAX by Inferring User Interface State
     571–580, May 2011.                                            Changes. In Proc. Int’l Conf. on Web Engineering
 [8] Ali Mesbah and Mukul R. Prasad. Automated                     (ICWE), pages 122–134, July 2008.
     Cross-Browser Compatibility Testing. In Proc. Int’l      [25] Carlos Pacheco, Shuvendu K. Lahiri, Michael D.
     Conf. on Software Engineering (ICSE), pages                   Ernst, and Thomas Ball. Feedback-Directed Random
     561–570, May 2011.                                            Test Generation. In Proc. Int’l Conf. on Software
 [9] Shauvik Roy Choudhary, Mukul R. Prasad, and                   Engineering (ICSE), pages 75–84, May 2007.
     Alessandro Orso. X-PERT: Accurate Identification of      [26] Shin Hong, Yongbae Park, and Moonzoo Kim.
     Cross-Browser Issues in Web Applications. In Proc.            Detecting Concurrency Errors in Client-side
     Int’l Conf. on Software Engineering (ICSE), pages             JavaScript Web Applications. In Proc. Int’l Conf. on
     702–711, May 2013.                                            Software Software Testing, Verification and Validation
[10] Yuta Maezawa, Hironori Washizaki, and Shinichi                (ICST), pages 61–70, March 2014.
     Honiden. Extracting Interaction-based Stateful           [27] Arjun Guha, Shriram Krishnamurthi, and Trevor Jim.
     Behavior in Rich Internet Applications. In Proc.              Using Static Analysis for Ajax Intrusion Detection. In
     European Conf. on Software Maintenance and                    Proc. Int’l World Wide Web Conf. (WWW), pages
     Reengineering (CSMR), pages 423–428, March 2012.              561–570, April 2009.
[11] Yuta Maezawa, Hironori Washizaki, Yoshinori Tanabe,      [28] Salvatore Guarnieri, Macro Pistoia, Omer Tripp,
     and Shinichi Honiden. Automated Verification of               Julian Dolby, Stephen Teilhet, and Ryan Berg. Saving
     Pattern-based Interaction Invariants in Ajax                  the World Wide Web from Vulnerable JavaScript. In
     Applications. In Proc. Int’l Conf. on Automated               Proc. Int’l Symp. on Software Testing and Analysis
     Software Engineering (ASE), pages 158–168,                    (ISSTA), pages 177–187, July 2011.
     November 2013.                                           [29] Shiyi Wei and Barbara Ryder. Practical Blended Taint
[12] Michael Mahemoff. Ajax Design Patterns. O’Reilly               Analysis for JavaScript. In Proc. Int’l Symp. on
     Media, Inc., 2006.                                            Software Testing and Analysis (ISSTA), pages
[13] Yunhui Zheng, Tao Bao, and Xiangyu Zhang.                     336–346, July 2013.
     Statically Locating Web Application Bugs Caused by       [30] Yue Jia and Mark Harman. An Analysis and Survey of
     Asynchronous Calls. In Proc. Int’l World Wide Web             the Development of Mutation Testing. IEEE Trans.
     Conf. (WWW), pages 805–814, April 2011.                       on Software Engineering (TSE), 37(5):649–678,
[14] R.A. DeMillo, R.J. Lipton, and F.G. Sayward. Hints            September 2011.
     on Test Data Selection: Help for the Practicing          [31] S. Mirshokraie, A. Mesbah, and K. Pattabiraman.
     Programmer. Computer, 11(4):34 –41, April 1978.               Efficient JavaScript Mutation Testing. In Proc. Int’l
[15] Cong Tian and Zhenhua Duan. Detecting Spurious                Conf. on Software Testing, Verification and Validation
     Counterexamples Efficiently in Abstract Model                   (ICST), pages 74–83, March 2013.
     Checking. In Proc. Int’l Conf. on Software               [32] Kazuki Nishiura, Yuta Maezawa, Hironori Washizaki,
     Engineering (ICSE), pages 202–211, May 2013.                  and Shinichi Honiden. Mutation Analysis for
[16] Michael Grottke and Kishor S. Trivedi. Fighting Bugs:         JavaScript Web Applications Testing. In Proc. Int’l
     Remove, Retry, Replicate, and Rejuvenate. Computer,           Conf. on Software Engineering and Knowledge
     40(2):107–109, 2007.                                          Engineering (SEKE), pages 159–165, June 2013.
You can also read
Next slide ... Cancel