AVOIDING TEST DAY LATENCY: Preventing Internet Interruption for Performance-Based Assessment (PBA) By TrueAbility
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
White Paper AVOIDING TEST DAY LATENCY: Preventing Internet Interruption for Performance-Based Assessment (PBA) By TrueAbility®
Prepared by
TrueAbility ®
Performance-based Assessment
for the Digital Workforce™
Performance-based testing is the core of TrueAbility’s
platform. It ensures that individuals are tested on their ability
to correctly perform essential tasks and functions in a real-world
application. Our technology is your vehicle to creating an exam
that leverages performance-based testing in a scalable, secure,
and cost-effective way.
Table of Contents
Overview Page 1
Key FIndings Page 2
The Impact Page 3
The Study Page 6
The Results Page 8
Summary Page 10
The PBA Advantage Page 12
Published February 22nd, 2018
OVERVIEW
Over the last several decades, skill certification as plumbers, electricians, factory workers,
and testing organizations have been reliant on carpenters, and the like. Demonstrating hands-
multiple-choice questions (MCQ) and written on skills of this type of work is often performed
testing techniques to gauge expertise. Most “on-the-job” and under the watchful eye of an
tests have been performed as in-person experienced master of the skill being tested. As
exams conducted in physical test centers. As more of the workforce shifts to computer-based
interest in assessing skills grows, broadening jobs, which require expertise in software and
worldwide user accessibility with computer- technology, performance-based testing of digital
based assessments, while reducing costs and skills requires test-takers to perform a series
increasing exam value has understandably of tasks, not in the presence of an expert, but
become a top priority. As the modern workforce instead while connected to a virtual workspace
continues to use more software and technology, that is hosted online in the cloud.
the introduction of internet-based testing
for computer-based jobs is a welcomed one.
However, requiring test candidates to connect These type of cloud-based
to systems in the cloud requires attention to
environments closely simulate
how exams are initially set up to eliminate
potential access issues. A complex network, real-world job situations and
the internet is critical to the success of any provide the software and tools
online test or training program. Primarily,
network latency must be stable to deliver a
common to the testing subject.
globally consistent, high-quality experience
every time. A test developer can create a
Unlike traditional internet-based testing
proficient exam, but latency from an insufficient
which may require access to a simple online
connection to the exam environment can defeat
multiple-choice exam, performance-based
the test-taking purpose, as well as impact
exams are more robust and thus demand a
scoring reliability and user experience.
more thoughtful approach concerning internet
connectivity. Once achieved, the value of
Maintaining a high success rate for proving such an exam extends well beyond a multiple
knowledge and skill, performance-based choice test. With the rise of PBA and action-
assessment (PBA) has been the foundation based learning, pen and paper -- or even one-
for determining the ability of skilled jobs such dimensional computer testing -- are no longer
sufficient for many types of skill assessment.
1 Avoiding Test Day Latency: Preventing Internet Interruption for PBA Copyright ® 2018 TrueAbility
As a majority of forward-thinking certification and training organizations around the
world move to advance their testing and training labs to PBA, most will seek a solution
to address the most critical hurdle for these exams, the potentially menacing network
latency challenges. Solving this will ensure PBAs execute successfully and reliably.
Performance-
Based Assessment
In 2017, TrueAbility conducted a global study among
(PBA) captures
more educational test developers and testing candidates to better
objectives than any
other assessment understand how network latency impacts PBA, how
method alone.
significant the lag is to the exam experience, and how network
speeds negatively impact the end-user.
KEY FINDINGS
INCLUDE:
1 The more geographical distance between a user and the physical
location of their hosted exam, the greater the amount of
network latency is experienced.
2 Testers who experienced network delays of 201milleseconds
(ms) or more, reported they were not able to perform an
assessment with a high degree of confidence to the results.
3 135ms lag shows to be the upper limit of acceptable latency for
testers.
2 Avoiding Test Day Latency: Preventing Internet Interruption for PBA Copyright ® 2018 TrueAbility
THE IMPACT OF NETWORK
LATENCY ON PERFORMANCE-
BASED ASSESSMENT
The testing industry continues to advance its educational
and assessment options with new methods such as evidence-based
NETWORK LATENCY:
training and evaluation content, remote computer-based testing
An expression of delays
(CBT) and training labs, and online remote proctoring for students
incurred when processing
and test takers in an effort to increase the fidelity, efficacy and global
network data.
reach. However, for these new methods to be successful, there are
numerous technical challenges that must be overcome, the
greatest being performance delays and consistent and reliable
accessibility of these remote environments.
Particularly, for highly-specialized computer-based certifications, a seamless PBA
experience demands computer server performance and network responsiveness.
Accessing high-fidelity tests over the public internet from remote servers is contingent
on several testing environment factors such as internet connectivity or bandwidth.
One of the primary factors affecting the testing process that is less often discussed,
regardless of which method is used, is network latency and its impact on PBA.
Network latency is an expression of delays incurred when processing network data,
measured in milliseconds and referred to as “lag.” Low latency means that delays are
relatively negligible, while high latency connections suffer from longer delays. Network
latency is the round-trip time (RTT) or “the time it takes a given network packet to travel
from source to destination, and back.” Latency as close to zero as possible is ideal.1
Three main contributing factors that
impact network latency are:
1. Physical distance of the transfer of information.
2. The number of hops, or networking devices, the data must cross.
3. And to a lesser extent these days, computer hardware.
1 LifeWire, All About the Effects of Latency on Computer Networks (October 10, 2017) .
3 Avoiding Test Day Latency: Preventing Internet Interruption for PBA Copyright ® 2018 TrueAbility
Latency is increased when information experience. In the context of PBA or action-
travels distance to reach a destination. One based assessments, network latency affects
consideration often overlooked is the quality the time it takes for a test taker’s action
of the network the information travels on from to be recorded, which can be detrimental
between one geographic location to another. to the confidence of the test taker in their
Think of the public internet as a freeway that performance. Actions can be defined as
can easily turn into a sea of congested gridlock. simply as moving the cursor, inputting text
or as complex as entering commands and
The effects of latency for web use have been
processing data. Furthermore, such latency
well publicized. For example, research by
brings into question the reliability of the
enterprise content delivery network provider,
exam—the degree to which an assessment
Akamai, captured data from users engaged in
produces consistent results.
basic web surfing which indicated that even
a 100ms delay in page load speed caused
an average seven percent drop in visitors
moving through the site, while a much larger
two-second delay in web page load time can
increase bounce rate, or the rate visitors leave
Therefore, for certification
the site, by over 100 percent.2
program managers electing to
This research proves the sensitivity users
have around performance, involving a task as
optimize their testing experience
simple as web surfing. Now, introduce similar by running a performance-based
latency into a high-stakes exam that is made to assessment, it is imperative
emulate a real-world work environment—it is
reasonable to infer the level of frustration will
to ensure that the assessment
be exponential. is within an accepted range of
Countless industries and practices depend latency for all users.
heavily on a stable internet connection and
efficient data processing for several reasons.
Perhaps most importantly—more now than
ever—is the speed at which information is
submitted, an element crucial to the user
2 Akamai Online Retail Performance Report: Milliseconds are Critical (April 19, 2017)
4 Avoiding Test Day Latency: Preventing Internet Interruption for PBA Copyright ® 2018 TrueAbilityAmong other factors, network latency can influence the validity of a PBA and therefore
affect the value (real or perceived) of a certification. In his study on the subject, The Effect
of Latency on User Performance in Real-Time Strategy Games, Mark Claypool states that
“knowing how these real-time applications react to latency and network loss is the crucial first
step in designing the next generation network hardware and software that will support their
requirements.” 3
Qualitative data shows that an exam candidate who is regularly, when not taking an exam,
subjected to large amounts of latency may perceive impacted connectivity as standard,
whereas another who is used to frequently non-impacted connectivity, could perceive a minor
amount of lag as an obstacle to completing an exam successfully.
“Until now, there has been little regard for establishing acceptable ranges of latency for exams
across the world for the testing industry,” says David Lister, a certified security expert with
Wiresolve.
TrueAbility has created infrastructure that provides the world’s
most optimal lag-free experience by using multiple tier one cloud
providers and deploying the assessment in the closest region to
the test taker.
3 Claypool http://web.cs.wpi.edu/~claypool/papers/rts/paper.pdf
As networking expert David Lister puts it,
“ TrueAbility’s network architecture has high availability
built in, ensuring that test taker responses will not be
lost as they move across the digital landscape.
“
5 Avoiding Test Day Latency: Preventing Internet Interruption for PBA Copyright ® 2018 TrueAbilityTHE STUDY
In early 2017 TrueAbility set out to examine network latency and its effect and impact on PBA.
This initiative was spurred by a corporate objective to build the first enterprise-grade infrastructure
that delivers performance-based assessment to every corner of the globe, within acceptable latency
ranges.
This network map illustrates the network as end of 2017. This is the current and planned top-tier, public cloud
datacenter connectivity. You can see the GCP POPs and the direct networking cables that connect the world.
To begin, TrueAbility built a team of over 100 global network testers representing over 40
countries across North America, South America, Europe, Africa, Asia, and Australia, and
utilized 14 tier-one datacenters to host sample exams.
The geographically dispersed testers engaged 62 data points in completing a sequence of
networking scenarios. These scenarios included various ways that data interacts with a live
environment, including upload, download, ping, and even jitter testing that clocked the upload and
download speeds. The test considered both objective and subjective factors.
Network latency and actual internet response time was recorded while participants provided
feedback about their perception of the connected experience. In all, 17 in-depth steps were
performed by each tester throughout the study to collect fundamental network data when accessing
a PBA environment.
6 Avoiding Test Day Latency: Preventing Internet Interruption for PBA Copyright ® 2018 TrueAbilityTEST COMPONENT
Comprised of three phases:
PHASE 1: QUANTITATIVE DATA COLLECTION
Testers were required to record testing location information that
included PC Type, Home/Office ISP, and Upload/Download speed
of their connection. Each performed a list of connectivity tests
between their locations, and a point-of-presence (POP) setup in
each datacenter. The connectivity tests recorded the following
data points between the tester and the POP; upload speed to
each datacenter POP; download speed to each datacenter POP;
average packet round-trip time to each datacenter POP; and Jitter.
PHASE 2: QUALITATIVE DATA COLLECTION
Testers logged into various remote environments and performed
a basic typing test. A post-test survey was performed, asking the
tester questions regarding their perception of how much lag there
was during the test, and how much delay existed between the time
keys were pressed versus when the characters appeared on the
screen within the typing test.
PHASE 3: ANALYSIS
The quantitative and qualitative data from both phase 1 and 2
were analyzed to identify trends and patterns, which were then
matched to what a tester considered to be a latent connection
with specific network throughput, round trip, and location data.
7 Avoiding Test Day Latency: Preventing Internet Interruption for PBA Copyright ® 2018 TrueAbilityTHE RESULTS
Latency, fidelity, and validity are critical factors that drive successful PBA
implementation. In order for PBA to exist, internet-based testing environments
must deliver without obstacles or degradation to allow the person taking the
exam to perform in as close to a real-world work environment as possible. Results
from the study reveal geographical distance plays a significant role in delivering
that experience.
When a tester recorded a latent exam experience, the following
Testers who factors were found to be consistent:
experienced
network delays of
201milleseonds 1. The exam environment was not hosted in the closest datacenter to their
(ms) or more, physical location.
reported they
were not able 2. The tester was experiencing an average packet round-trip time over 200ms.
to perform an
assessment with
a high degree of When a tester recorded an optimal exam experience, meaning
confidence to the
results. there was no noticeable latency–or disruptive level of latency–the
following factors were consistent:
1. The test taker’s exam environment was hosted in a datacenter geographically
located near their physical location.
2. The tester’s average packet round trip time was less than 135ms.
Testers located in regions with poor Internet connectivity speeds were found to
have a lower qualitative threshold for what they considered to be latency. For
instance, an average round trip time of over 200ms to these testers in many cases
was not reported as latent.
Testers who experienced network delays of 201ms or more, reported they were
not able to perform an assessment due to challenges operating within the testing
environment and lacked confidence in the system and tools to complete the tasks
on time and accurately. This was also true for over half of all users that reported
a network delay between 135-200ms. Once the testers experienced less than a
135ms delay, they reported environments operated as expected, and assessment
results were accurate.
8 Avoiding Test Day Latency: Preventing Internet Interruption for PBA Copyright ® 2018 TrueAbilityThe research clearly shows that a 135ms lag is the upper limit of acceptable latency for
most testers. It also indicates that the optimal performance RTT was less than 100ms.
In general, users with sub 100ms RTT did not complain about lag.
The benchmarks of the testing are displayed below in Table A
It is required to decrease the Key to this is consistently ensuring acceptable
latency levels of 135ms and below. This is a
physical distance between a
significant finding due to cloud-based PBA
hosted performance-based being a nascent approach to skill certification
exam environment and the test for testing organizations with constituents
around the world. Ultimately, if an internet-
taker as much as possible in based exam fails to meet accessibility
order to remove performance expectations, the value of the entire program
obstacles. is at risk.
9 Avoiding Test Day Latency: Preventing Internet Interruption for PBA Copyright ® 2018 TrueAbilitySUMMARY
Performance-based assessment is a method that presents
incredible opportunities for certification and training
organizations around the world.
The simple fact is the art and science of and the hosted exam environment.
skills testing has traditionally focused To disregard building a truly global
on archaic methods. Automating the provisioning capability will result in an
process is a chance to enhance user unacceptable exam experience leading
experience, boost credibility, significantly to low conversion, unfavorable branding,
reduce costs, improve exam delivery, and potentially catastrophic failure of the
and increase fidelity with more accurate program.
results when delivered without
interruption. Increasing awareness of the value of
performance-based assessment for
For an organization to successfully computer-based exams is driving interest
mitigate the negative effects of high and adoption, and most importantly new
latency in PBA, they must consider both solutions in the marketplace.
the physical location of the end user
ELIMINATING LATENCY WITH
SOPHISTICATED PERFORMANCE-
BASED TECHNOLOGY
Performance-based certification and training based testing to be scalable, secure, and
puts a technical professional in a real-world affordable–and now accessible at a high
environment, so they can “show what they degree.
know by doing.” Demonstrative testing and
TrueAbility having identified “proximity” as
training is a more accurate indicator of skill
the fulcrum in its study, it is therefore critical
level and critical thinking ability. Concerns
for a PBA solution to leverage means to
about implementation generally revolve
provide test takers uninhibited access to the
around costs, and the ability to securely and
performance-based exam environment, as
globally issue exams without interruption.
seen in the “TrueAbility Tiered Regions” in
The good news is that modern technology
Table B below.
has made it possible for performance-
10 Avoiding Test Day Latency: Preventing Internet Interruption for PBA Copyright ® 2018 TrueAbilityTable B: TrueAbility Tiered Regions
TrueAbility solves the proximity issue by leveraging tier-one datacenters around
the globe to host exams. With all major infrastructure providers integrated, TrueAbility’s
platform identifies ideal regions for provisioning the candidate exam based on their physical
location, eliminating unacceptable latency from the experience. As of this published
edition, there are 19 datacenter network regions in use with more coming online annually.
TrueAbility leverages these datacenters locations and high-performance fiber routing that
is illustrated in the “TrueAbility Premium Networking Infrastructure” in Table C below. This
advanced approach to capitalizing on the most advanced cloud technologies serves to
continue to impact delivery and shrink the globe.
By having a cloud-based assessment platform with optimal latency,
skills testing is most closely aligned to a real working experience.
11 Avoiding Test Day Latency: Preventing Internet Interruption for PBA Copyright ® 2018 TrueAbilityTable C: TrueAbility Premium Network Infrastructure
THE PBA ADVANTAGE
With increased service delivery infrastructure, and policies, and procedures designed to ensure
exams can be delivered 24/7 with high performance and availability, PBA gives exam developers
the ability to create and track a solid certification and/or training assessment program where
candidates excel in their testing environment. Organizations who have implemented PBA have
experienced significant benefits:
• Improved accuracy
• Increase exam value
• Seamless exam delivery anywhere, at any time
• Increased user responsiveness and experience
• Reduced costs associated with automation
• Elimination of geographic barriers
• Authenticity and Verification Integrity
• 24/7/365 Exam Support
• Automated Grading
• Detailed Reporting
• Remote Proctoring
• Customized exams
• Decrease cheating and test breach
12 Avoiding Test Day Latency: Preventing Internet Interruption for PBA Copyright ® 2018 TrueAbilityAbout TrueAbility
TrueAbility is an advanced platform for delivering Performance-
based Assessment Solutions for the Digital Workforce™.
Important to the classroom and workplace, TrueAbility provides
solutions built upon scalable performance-based technology,
bridging a gap for both end-users and the entities measuring
their skill by providing the most accurate and comprehensive
environment to learn or demonstrate skillsets.
Performance-based assessment captures more educational
objectives than any other assessment method alone. TrueAbility’s
platform combines this innovation with traditional assessment
methods and a menu of features to create the most optimal
assessment capability available today.
Today, data scientists, actuaries, lawyers, developers, engineers,
technologists, and others prove themselves on the TrueAbility
platform. Founded in 2012, TrueAbility is privately held and
headquartered in San Antonio, Texas.
For more information, please visit :
www.trueability.com
Let TrueAbility help you with your assessment:
Want to see the product what makes us different? Schedule a
live demonstration of TrueAbility PBA.
TrueAbility ®
877 461 8702
info@trueability.com
13 Avoiding Test Day Latency: Preventing Internet Interruption for PBA Copyright ® 2018 TrueAbilityYou can also read
