Use of Throughput To Evaluate a Cursor Control Device (CCD) Performance
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Use of Throughput To Evaluate a Cursor Control Device (CCD) Performance
P. Doyon-Poulin (a) and N. Routhier (b)
(a) École Polytechnique, Montréal QC
(b) Bombardier Aéronautique, Montréal QC
Abstract
The goal of this study was to evaluate the performance of a cursor control device in a mockup
cockpit environment under different physical configurations of eye-to-screen distance and crew
reach. We tested the influence of two factors on the cursor performance: 1) display configuration
and 2) handedness. We found a different in performance for the handedness factor, but it was
less important than what we expected. We found no significant effect for the display
configurations tested. In this study, we used the ISO 9241:9 method to evaluate the cursor
performance. This provided a robust test procedure and baseline results for comparing the cursor
with other input devices.
I. Introduction
Designing interaction devices for a cockpit environment is challenging due to space limitations,
crew variability, and the range of in-flight turbulence and vibrations affecting fine hand
movements. The introduction of the Cursor Control Device (CCD) with integrated graphical
applications in commercial glass cockpits offers multiple advantages in the area of flexibility,
scalability and ease of learning. However, for a device to offer an advantage over previous
technologies, it must demonstrate an equivalent or better level of task performance and support
efficient workflow and crew comfort while reducing workload and error rates. Although some
CCD models are already being used as primary means of interaction in an aerospace
environment, this type of interaction is still novel and, consequently, studies on testing methods
and performance data are scarce.
In this study, we investigated the effect of two factors on the overall performance of a trackball
CCD. First, we tested if the position of the display on which the pilot is interacting has an
influence on the cursor performance. Second, we tested whether handedness influences
Manuscript presented at CASI 2011. p.1performance – that is, to what extent does using the CCD with the dominant or the non-dominant
hand affects performance. The trackball CCD model we tested is depicted on Figure 1.
Figure 1: Trackball CCD. Deleted from manuscript.
Display positioning deserves special consideration in multi-display glass cockpit because pilots
are afforded the option to allocate interactive applications to the display they choose. The CCD
however is not moveable, and there might exist a performance cost in cursor interaction
associated with a particular CCD-display geometry. Handedness is also an important factor to
evaluate for aircraft CCD interactions, since the CCD is in a fixed location on the inboard side of
each seat. Kabbash and colleagues found a performance cost with the non-dominant hand for the
use of the PC mouse and stylus, but not for the trackball (Kabbash, MacKenzie and Buxton,
1993).
To evaluate the effect of those two factors on the cursor performance, we chose to use the test
procedure outlined in ISO 9241:9 with the throughput value as a performance indicator. ISO
9241:9 has been extensively used in the HCI research community to characterize the overall
performance of input devices, thus providing a large amount of baseline results to compare the
CCD tested in this experiment.
The rest of this article is structured as follows. The next section briefly describes the procedure
suggested in ISO 9241:9 to measure throughput. Section III explains the method we used to
evaluate the CCD throughput with three CCD-display geometries and the effect of handedness.
Section IV presents the results we found for the geometry and handedness factors, and section V
discusses the implications our results have for future research on cursor interaction in the aircraft
industry.
II. ISO 9241:9
ISO 9241:9 defines throughput as an overall performance metric. Throughput indicates the rate,
in bits per second (bps), at which a user can select multiple targets with a specific device in a
given time frame. In this sense, it is similar to a device bandwidth: higher throughput means
shorter selection time. However, contrary to other performance metrics that capture only a single
aspect of performance as time or accuracy, throughput includes both the speed and accuracy of
Manuscript presented at CASI 2011. p.2the users in a single metric. This is one of the main reasons why HCI researchers advocate for its
use as a cursor performance metric (Soukoreff and MacKenzie, 2004; MacKenzie, 2003).
The correction for speed-accuracy tradeoff makes the throughput value more robust to an
individual’s tendency to put more emphasis on either speed or accuracy in a test, a notion that is
absent in a time-only measure of performance. MacKenzie and Isokoski (2008) showed that a PC
mouse’s throughput is constant whether participants were asked to emphasize on speed or
accuracy. In both cases, the selection time and error rate were significantly different, but the
throughput value remained the same.
Throughput calculation uses the actual end-point scatter data to adjust results for accuracy. To
give a sense of the adjustment for accuracy, consider a user serially selecting two targets of
width (W) with a center-to-center distance (D). For large targets, as the one depicted on Figure 2,
end-points are gathered near the target’s edge and do not cover the whole target width. In this
case, the effective width (We) is smaller than the actual target width, and the effective distance
(De) is smaller than the actual distance. Inversly, for small targets, movement end-points are
more scattered since users overshoot the target, thus increasing the effective width (for more
details, see Soukoreff and MacKenzie, 2004).
Figure 2 : End-points visualization. For large targets, the effective width is smaller than the
target width.
ISO 9241:9 suggests to use a multidirectional selection task to evaluate throughput. The test
procedure is straightforward: the subject sequentially selects 25 targets laid out around the
circumference of a circle (see Figure 3). The test starts when the subject selects the topmost
Manuscript presented at CASI 2011. p.3target and ends when the subject has gone around the circle and reaches again the top-most
target.
The subject repeats the test over a range of difficulties. That is, once all targets have been
selected, a new trial is presented with a different combination of width and distance. Data
recorded during the test are the movement time (MT) taken to select each target, as well as the
start-points and end-points coordinates.
Figure 3: Selection task suggested by ISO 9241:9. The arrows indicate the selection path.
For each trial of 25 targets, De is the average distance between two successive end-points, while
We is defined with the standard deviation of end-point scatter data (σ) as follows (see also Kong
and Ren, 2006).
We = 4.133σ (1)
Each adjusted width-distance combination defines the effective indice of difficulty (IDe) of a trial.
€ 2 of the logarithm.
Units of IDe are bits, due to the base
" De %
IDe = log 2 $ +1' (2)
# We &
Manuscript presented at CASI 2011. p.4
€Throughput (TP) is then computed as an average over all m trials and n participants. Units of
throughput are bits per second (bps).
1 n # 1 m IDeij &
TP = ∑%% ∑ ( (3)
n i=1 $ m j =1 MTij ('
III. Method
€
Participants
Six participants (all right-handed) volunteered for the experiment. All participants were regular
users of GUI and PC mouse. One participant was familiar with an aircraft CCD.
Apparatus
Tests with the CCD were conducted in a mockup environment reproducing the cockpit physical
ergonomics. Each participant was positioned according to the Eye Reference Point (ERP) (see
Figure 4). Test trials were presented on one of the three 15.1” displays (see Figure 5). Two CCDs
were mounted in the center pedestal in a geometry similar to an aircraft installation. Acceleration
was enabled for the cursor (non-linear gain response) and no additional delay on the cursor
response time was applied.
Figure 4: Ergonomic layout
Manuscript presented at CASI 2011. p.5Procedure
The experiment used a Javascript program implementing Fitts’ task according to ISO 9241:9 as
described in the previous section. Each participant repeated the selection task for 10 width-
distance combinations (see Table 1) and the software randomized the ordering of combinations.
It took roughly 15 minutes for a participant to complete the test in a single configuration.
Table 1: Presented Indices of Difficulty (IDs)
Distance
Width (pixels) ID (bits)
(pixels)
15 664 5.5
20 206 3.5
20 620 5
25 375 4
30 720 4.64
30 255 3.25
40 412 3.5
50 622 3.75
75 525 3
100 300 2
Design
We tested the influence of two factors on the cursor performance: 1) display configuration and 2)
subject handedness. Dependent variables were throughput (bps) and error count (number of
selections outside the target).
For the configuration factor, participants tested three CCD-display geometries using their
dominant hand. Two configurations used the onside cursor and one configuration used the cross-
side cursor. We analyzed the results as a one-way within-subjects design. The total number of
clicks recorded for this test was 6 participants x 3 geometries x 10 trials x 25 targets = 4500
clicks. The three CCD-display geometries tested were (see Figure 5-a):
1. Left CCD, left screen
Manuscript presented at CASI 2011. p.62. Left CCD, head-down screen
3. Right CCD, right screen
For the handedness factor, participants used the onside CCD with their dominant and non-
dominant hand, and changed seat accordingly. Participants then did the same series of selection
tests, seated at a desk, using a PC mouse and notebook with their dominant hand only. This
provided a baseline throughput result for the dominant hand. We analyzed results as a one-way
within-subjects design. The geometries tested were (see Figure 5-b):
1. Dominant hand, left CCD, left screen
2. Non-dominant hand, right CCD, right screen
3. Dominant hand, PC mouse
a b
Figure 5: CCD-Display layout for the (a) display configuration and (b) handedness test
IV. Results
The effect of display configuration on the error count was not significant (F2,10 = .9507, ns), nor
was the effect of handedness (F2,10 = .9209, ns). Overall, less than 5% of all clicks were outside
the target.
Figure 6-a shows the CCD throughput value for each display configuration. The mean
throughput value for the onboard screen was 1.93 bps, while it was 1.87 bps for the head-down
screen and 1.81 bps for the cross-side cursor and outboard screen. The effect of configuration on
throughput is not significant (F2,10 = 1.369, p > 0.05).
Manuscript presented at CASI 2011. p.7Figure 6-b shows the throughput value for the dominant and non-dominant hand for the CCD, as
well as for the PC mouse with the dominant hand for baseline comparison. The mean throughput
value for the CCD with the dominant hand was 1.93 bps, while it was 1.63 bps with the non-
dominant hand and 3.60 bps for the PC mouse with the dominant hand. The effect of handedness
on the CCD throughput value is significant (F1,5 = 9.601, p < 0.05). Not surprisingly, the
difference in throughput between the two devices with the dominant hand is also significant (F1,5
= 254.06, p < 0.00005). Results for the PC mouse were similar to those found in the academic
literature, where the PC mouse throughput is estimated to be anywhere between 3.7 to 4.9 bps
(Soukoreff and MacKenzie, 2004).
a b
Figure 6: Throughput for (a) each display configuration and (b) handedness. Error bars are the
95% confidence interval.
Figure 7 shows a visualization of end-points scatter data for both a large and a small target. For
the purpose of the demonstration, the end-points coordinates of the 25 targets selected in a given
trial are plotted on a single target. Results are from participant #6 with the left cursor, left screen
configuration.
For the large target, end-points are evenly distributed around the target center, although not
covering the whole target area. In this case, the effective width is smaller than the actual width.
For the small target, the participant made four selections outside the target, thus enlarging the
effective width. Movement end-points are gathered near the lower-left area of the target, because
the center of the effective width circle is shifted relative to the target center.
Manuscript presented at CASI 2011. p.8Figure 7: Movement end-points visualization for a large (left) and small (right) target. Results
are from participant #6 with the left cursor, left screen configuration.
V. Discussion
Configurations
Contrary to our expectations, the performance was similar between the three CCD-display
geometries tested in this experiment. We found that the cross-side cursor had a throughput value
6% lower than that of the onside CCD, but we would have expected a higher performance cost
for the use of the cross-side cursor since the participant’s posture is suboptimal. With this setup,
the participant’s arm is fully extended to reach the cross-side CCD and the hand grasps the CCD
at an acute angle. However, it would be reductive to limit our interpretation to the sole
performance metric. During post-experiment interviews, participants reported that the cross-side
cursor manipulation induced neck strain and more pain to their hand because they had to tighten
more muscle groups.
Handedness
Our results showed that using the CCD with the non-dominant hand decreases performance by
16%. While this difference is significant, it is not as severe as we would have expected. During
this study, participants used the CCD with their non-dominant hand in only one out of four
geometries; we would have expected a steeper learning effect for the non-dominant hand
performance. One aspect that was not tested in this experiment is the learning effect resulting
from repeated usage of the CCD. A future experiment should test if the increase in performance
due to learning is the same for both hands.
Manuscript presented at CASI 2011. p.9Benchmark
Even though the CCD throughput is about half that of the PC mouse, its performance is similar
to that of other input devices. In their review article, Soukoreff and MacKenzie (2004)
summarized the results of 9 Fitts’ law studies conforming to ISO 9241:9. They found that the
range of throughput value is [3.7 - 4.9] bps for the PC mouse, [1.6 – 2.55] bps for the isometric
joystick and [0.99 – 2.9] bps for the touchpad. With a throughput of 1.93 bps for the dominant
hand, the CCD prototype evaluated in this study has a performance similar to that of the
joysticks found by Soukoreff and MacKenzie. Its performance is however lower than that of a
tabletop trackball with a throughput of 3.0 bps (MacKenzie, Kauppinen and Silfverberg, 2001).
In addition to benchmarking with other devices, throughput can also be used to quantify the
changes in performance on the same device during the prototyping phase. This is particularly
useful for human factors practitioners who want to improve the overall performance of the
cursor. For instance, in a pilot study, we tested the same CCD with a constant gain (no
acceleration) and found a throughput of 1.33 bps. This means that we have increased the initial
cursor performance by 144% by optimizing software-specific parameters – enabling acceleration
in this case.
Implications for evaluation
In this article, we adapted the method outlined in ISO 9241:9 to evaluate an aircraft CCD. Our
experience with the ISO 9241:9 method supports its use to evaluate a cursor performance for two
reasons. First, because its procedure has been thoroughly validated by the ISO committee and
has since been used extensively by the HCI community. Second, because the use of a standard
performance metric provides a means to compare in a reliable way various interaction devices. In
this sense, the results reported in this study can serve as a baseline comparison for other aircraft
CCDs, whether the device is a trackball or a joystick since the ISO 9241 method is device-
independent.
Limitations and future experiments
The small number of participants who took part in our experiment explains in part the lack of
significance between the three dominant hand configurations. Having more participants would
Manuscript presented at CASI 2011. p.10reduce the error bars and could raise a significant difference between the onside and cross-side
cursor. This should be investigated in a follow-up experiment.
Tests were done in a static cockpit. However, in-flight turbulence imposes a burden to pilots for
target selection due to the jittering of both the screen and the cursor. The effect of turbulence on
the efficiency of cursor interactions could be evaluated in a dynamic, high-fidelity cockpit
simulator.
VI. Conclusion
In this study, we evaluated the performance of a trackball CCD for use in aircraft installation.
We found a difference in performance between the dominant and non-dominant hand, but it was
less important that what we were expecting. Also, we found no significant difference between
three CCD-display geometries configurations when the cursor was used with the dominant hand
only. The CCD reached a throughput value of 1.93 bps in the best configuration. In this
experiment, we used the ISO 9241:9 standard to characterize the overall CCD performance. We
encourage other researchers to use the ISO standard because it provides a robust test procedure
and baseline results from various input devices.
References
International Organisation for Standardisation. (2002). Reference Number: ISO 9241-9:2000(E).
Ergonomic requirements for office work with visual display terminals (VDTs)—Part 9—
Requirements for non-keyboard input devices (ISO 9241-9).
Kabbash, P., MacKenzie, I. S., and Buxton, W. (1993). Human performance using computer
input devices in the preferred and non-preferred hands. Proceedings of the ACM Conference on
Human Factors in Computing Systems – INTERCHI ’93. ACM, New York. pp. 474-481.
Kong, J., and Ren, X. (2006). Calculation of Effective Target Width and Its Effects on Pointing
Tasks, Information and Media Technologies, Vol. 1, No. 2, pp. 1057-1059.
MacKenzie, I. S. (2003). Motor behaviour models for human-computer interaction. In HCI
models, theories, and frameworks: Toward a multidisciplinary science. Edited by J. M. Carroll.
Morgan Kaufmann, San Francisco. pp. 27-54.
Manuscript presented at CASI 2011. p.11MacKenzie, I. S., & Isokoski, P. (2008). Fitts' throughput and the speed-accuracy tradeoff. Proceedings of the ACM Conference on Human Factors in Computing Systems – CHI 2008. ACM, New York. pp. 1633-1636. MacKenzie, I. S., Kauppinen, T., & Silfverberg, M. (2001). Accuracy measures for evaluating computer pointing devices. Proceedings of the ACM Conference on Human Factors in Computing Systems – CHI 2001. ACM, New York. pp. 9-16. Soukoreff, R. W., & MacKenzie, I. S. (2004). Towards a standard for pointing device evaluation: Perspectives on 27 years of Fitts’ law research in HCI. International Journal of Human- Computer Studies, Vol. 61, N. 6, pp. 751-789. Manuscript presented at CASI 2011. p.12
You can also read
