WHAT'S MISSING FROM NATIONAL LANDLINE RDD SURVEYS? THE IMPACT OF THE GROWING CELL-ONLY POPULATION
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Public Opinion Quarterly, Vol. 71, No. 5 2007, pp. 772–792 WHAT’S MISSING FROM NATIONAL LANDLINE RDD SURVEYS? THE IMPACT OF THE GROWING CELL-ONLY POPULATION SCOTT KEETER Downloaded from https://academic.oup.com/poq/article/71/5/772/1805417 by guest on 08 September 2021 COURTNEY KENNEDY APRIL CLARK TREVOR TOMPSON MIKE MOKRZYCKI Abstract The number of cell phone only households has continued to grow – 12.8 percent of all households by the end of 2006, according to the National Health Interview Survey (NHIS). To assess the extent and nature of the potential bias in landline telephone samples created by the absence of cell-only households, the Pew Research Center conducted four independent dual frame studies in 2006, three of which included at least 200 interviews with cell-only respondents. Despite the fact that there are significant differences between cell-only and landline respon- dents on many important variables, across the four surveys with different substantive content we find that including a cell-only sample with a land- line RDD sample produces general population estimates that are nearly SCOTT KEETER is with Pew Research Center, 1615 L St., N.W., Suite 700, Washington, DC 20036, USA. COURTNEY KENNEDY is with Pew Research Center and the University of Michigan, 426 Thompson Street, Room 4050, Ann Arbor, MI 48104, USA. APRIL CLARK is with Pew Research Center, 1615 L St., N.W., Suite 700, Washington, DC 20036, USA. TREVOR TOMPSON is with The Associated Press, 2021 K St., N.W., 6th floor, Washington, DC 20006, USA. MIKE MOKRZYCKI is with The Associated Press, 168 Middle St., West Newbury, MA 01985, USA. The authors are grateful for the assistance of several individuals and organizations. Princeton Survey Research Associates International (PSRAI) and SRBI, Inc. collected the data reported here. The advice and assistance of Jonathan Best, Larry Hugick, Stacy Diangelo, and Julie Gasior at PSRAI, and Chintan Turakhia, Dean Williams, and Mark Schulman at SRBI was invaluable. Anna Fleeman, Charlotte Steeh, Linda Piekarski, Mike Brick, Clyde Tucker, and Stephen Blumberg regularly answered questions for us and provided us with updates from their own research with the cell phone population. The Pew Research Center and The Associated Press provided substantial support for the conduct of this research. We greatly benefited from the help and advice of Andrew Kohut, Michael Dimock, Nilanthi Samaranayake, Robert Suls, Greg Smith, Cary Funk, Juliana Horowitz, Richard Wike, Peyton Craighill, Lee Rainie, and John Horrigan. Paul Lavrakas and two anonymous reviewers provided excellent feedback on an earlier draft. The 2005 Cell Phone Sampling Summit II, organized by Paul Lavrakas and Chuck Shuttles, provided ideas and inspiration for this work. Address correspondence to Scott Keeter; e-mail: skeeter@pewresearch.org doi:10.1093/poq/nfm053 C The Author 2007. Published by Oxford University Press on behalf of the American Association for Public Opinion Research. All rights reserved. For permissions, please e-mail: journals.permissions@oxfordjournals.org
Impact of the U.S. Cell-Only Population 773 identical to those from the landline sample alone. Yet, while the noncover- age problem is currently not damaging estimates for the entire population, we find evidence that it does create biased estimates on certain variables for young adults, 25 percent of whom are cell-only according to the most recent government estimate. Downloaded from https://academic.oup.com/poq/article/71/5/772/1805417 by guest on 08 September 2021 The potential impact of the growing number of cell phone only (CPO) house- holds on poll estimates burst into public consciousness and gained heightened interest in the survey research community during the 2004 U.S. presidential campaign. Were random digit dial (RDD) telephone surveys conducted only on landline phones suffering noncoverage bias – and underestimating Democrat John Kerry’s support – by excluding adults who can only be reached by cell phone? A question about telephone status on the face-to-face 2004 national exit poll of voters by the National Election Pool (a consortium of The Associated Press, ABC, CBS, CNN, FOX, and NBC) found that 7 percent of Election Day voters could be reached only by cell phone. While young people were far more likely to be cell-only, and somewhat more supportive of John Kerry than older voters, they were similar to other voters in their age cohort. This provided initial evidence that pre-election telephone polls, which generally performed well in 2004 in forecasting national and state election outcomes, did not suffer signifi- cant noncoverage bias because of the absence of cell-only respondents, as long as the surveys in the USA weighted data from landline samples appropriately by age (National Council on Public Polls n.d.; Keeter 2006). There were few dual frame studies of landline and cell phone samples at that time, however, and those that were conducted focused largely on the feasibility of surveying cell phone numbers, which poses logistical and legal challenges (Steeh 2004; Brick et al. 2007). Since 2004, the number of CPO households has continued to grow – 12.8 percent of all households by the end of 2006 (and 11.8 percent of all adults), ac- cording to the National Health Interview Survey (NHIS). And recently wireless substitution has happened at an accelerated pace, with the incidence of cell- only households increasing by about one percentage point every six months from 2003 through 2005 but more than two percentage points in each half of 2006 (Blumberg and Luke 2007a). In his 2006 AAPOR presidential address, Cliff Zukin identified the growth of cell phones, particularly among young people, as a “serious coverage problem” and a “particular challenge” to the representativeness of industry-standard RDD samples. With increasing con- cern has come an accelerated pace of study of the issue, as evidenced by the 2003 and 2005 Cell Phone Summits (Lavrakas and Shuttles 2005), numerous papers presented at the Second International Conference on Telephone Survey Methodology (Lepkowski et al. 2007), and at the 2007 Annual Conference of the American Association for Public Opinion Research. As the cell-only population has grown, telephone surveys by Pew and other organizations that rely on landline samples have experienced a sharp decline in
774 Keeter et al. Downloaded from https://academic.oup.com/poq/article/71/5/772/1805417 by guest on 08 September 2021 Figure 1. Weighted and Unweighted Estimates of the Proportion of Adults Aged 18–34 Years based on Pew Research Center Surveys Conducted from 1990 to 2007. the percentage of younger respondents interviewed in their samples. As shown in figure 1, in Pew Research Center surveys over the past five years, the average percentage of respondents aged 18–34 years in unweighted samples declined from 32 percent in 2000 to 21 percent in surveys conducted in 2005, 20 percent in 2006, and 19 percent thus far in 2007 (the population parameter is now 30 percent). This decline is consistent with the fact that the cell-only population is heavily tilted toward young people. What is the consequence of this change in the composition of survey samples? Does the absence of the cell-only adults create a significant bias in national estimates on variables of interest in social and political surveys? The short answer is “no,” not now – or not as yet. This article will describe the basic differences between the cell-only and landline respondents across variables including numerous demographics, social and political attitudes, voting, and other behaviors. It will also describe the consequences of including cell-only respondents in the overall survey samples and the practical issues and cost implications of including cell-only respondents. In 2006, the Pew Research Center conducted four independent dual frame studies, three of which included at least 200 interviews with cell-only
Impact of the U.S. Cell-Only Population 775 respondents. These studies provide evidence for an updated assessment of whether noncoverage of CPO households significantly biases landline survey estimates. And because they covered a wide range of topics, the surveys al- low for an expanded examination of the differences and similarities between cell-only adults and those with landlines.1 The four projects were conducted between March and October 2006. The Downloaded from https://academic.oup.com/poq/article/71/5/772/1805417 by guest on 08 September 2021 following is a brief overview of the surveys with key information summarized in table 1.2 CELL PHONE STUDY The Pew/Associated Press/AOL Cell Phone Study examined the public’s use of and attitudes toward cell phones. Questions included how much and in what ways respondents use their cell phones, how cell phones have affected their lives, likes and dislikes about cell phones, awareness and use of features and services of cell phones beyond voice calling, and a small module of political and values questions. Telephone interviews were conducted from March 8 to March 28, 2006, among a sample of 1,503 adults aged 18 years and older. Of these, 1,286 were cell phone users. Approximately half of the interviews (752) were conducted using a landline number frame, with the remainder (751) conducted from a cell phone number frame including 200 who only had cell phones. Those interviewed from the cell phone frame were offered an incentive of $10 for completing the survey. The 551 respondents interviewed in the cell sample who reported having a landline in their household are excluded from all analyses in this study. 2006 BIENNIAL MEDIA CONSUMPTION The Pew Biennial Media Survey measured the public’s use of and atti- tudes toward the news media. Questions included how respondents receive news, attitudes toward news including enjoyment and appetite for following news stories, credibility of print and broadcast media outlets, and the use of technology for communication, entertainment, and news consumption. The 3,454 telephone interviews were conducted from a standard list-assisted RDD 1. Response rates and question wording for cell phone questions are shown in the appendix. 2. Several design features were constant across the four surveys. In the landline samples, respon- dents were selected using a procedure that asked first to interview the youngest male currently at home; if no male was at home, the interviewer asked for the youngest female. In the cell sam- ples, interviews were conducted with the person answering the call. Cell numbers answered by persons under age 18 were considered ineligible. All four surveys were conducted only in English. A maximum of one voicemail message was left with each cell number and no messages were left on answering machines reached on landline numbers. In all samples, numbers were attempted up to 10 times, and refusal conversion was attempted once for “soft” refusals.
776 Table 1. Sample Sizes in Telephone Service Categories by Study Landline sample respondents Cell sample respondents Landline Dual Cell N Landline Dual Cell N Survey Field period only user only only user only Cell Phone Survey March 8–28, 2006 217 534 1 752 0 552 199 751 GenNext Survey September 6–October 2, 2006 315 909 0 1,251 0 115 130 250 Media Consumption Survey April 27–May 22, 2006 N/A N/A N/A 3,204 Media Consumption Surveya May 15–June 3, 2006 0 0 250 250 Turnout Surveya September 21–October 4, 2006 N/A N/A N/A 1,804 0 0 200 200 a The RDD cell sample design screened for cell-only adults. Keeter et al. Downloaded from https://academic.oup.com/poq/article/71/5/772/1805417 by guest on 08 September 2021
Impact of the U.S. Cell-Only Population 777 landline sample and an RDD sample of cell phones. The 3,204 landline inter- views were conducted from April 27 to May 22, 2006. Supplemental interviews were conducted using a cell phone RDD sample (N = 250) among those who do not have access to a landline at their residence. These CPO interviews were conducted from May 15 to June 3, 2006. Downloaded from https://academic.oup.com/poq/article/71/5/772/1805417 by guest on 08 September 2021 TURNOUT SURVEY The Turnout Poll, conducted in conjunction with The Associated Press at the height of the 2006 congressional election campaign, includes measures about the political candidates and campaigns as well as political engagement. Questions included vote intention and congressional party preference as well as views of and overall attentiveness to the campaigns. The telephone survey of 2,004 adults was conducted from September 21 to October 4, 2006. The sample included 200 CPO interviews. GENNEXT SURVEY The GenNext Survey, conducted in association with MacNeil–Lehrer Produc- tions, explored lifestyles, attitudes, and values of young adults (aged 18–25 years). Questions included many aspects of public opinion including topics such as world views, use of technology, news consumption, and politics and policy issues. Interviews were conducted by telephone from September 6 to October 2, 2006, with a nationally representative sample of 1,501 adults (in- cluding an oversample of young people); 250 of the interviews were conducted with cell frame respondents aged 18–25 years. The numbers for the cell samples were drawn through a systematic equal probability sample from 1,000-banks dedicated to cellular service according to the Telcordia database. Numbers for the landline samples were drawn with equal probability from 100-banks with three or more listed residential numbers. According to the NHIS, approximately 98 percent of U.S. adults live in a household with either a landline or a cell phone (Blumberg and Luke 2007a). Analysis of all four studies produce the same conclusion: Cell-only respon- dents are significantly different from landline respondents in important ways, but they were neither numerous enough nor different enough on the key variables to produce a significant change in overall general population survey estimates when included with the landline samples and weighted according to U.S. Cur- rent Population Survey (CPS) parameters on basic demographic characteristics. However, certain survey topics and sampling frames may be vulnerable to sig- nificant, even dramatic, noncoverage bias if they exclude respondents who can only be reached by cell. This article concludes with evidence regarding the potential for bias in survey estimates for certain variables among young adults.
778 Keeter et al. Comparing and Blending Landline and Cell-Only Samples Across the Cell Phone, Media Consumption, and Turnout surveys, we se- lected for analysis 46 measures of media use, political and social attitudes, and electoral engagement. (The GenNext survey is reported separately in the next section.) For most indicators, the category chosen for analysis was the one most Downloaded from https://academic.oup.com/poq/article/71/5/772/1805417 by guest on 08 September 2021 central to the original purpose of the study; for example, for most measures of media use, we chose the percentage regularly using a given source. Where the choice of a category was not obvious, we selected the substantive category with the largest difference between the landline and cell-only samples. Across these 46 indicators, the mean difference between point estimates for the weighted3 landline sample and the unweighted cell-only sample was 7.8 percentage points. For most comparisons, differences of six percentage points or larger are statistically significant. But as shown in table 2no survey estimate would change by more than two percentage points with the inclusion of respondents from a cell-only sample, and the average change was 0.7 percentage points. Thus, the absence of the cell-only respondents from the landline RDD sampling frame creates only a minimal amount of bias in the weighted general population estimates. This felicitous result is a product of the fact that the cell-only population remains a relatively small part of the total population and the fact that they are not dramatically different on most measures from landline respondents, especially those within the same age cohort. At 11.8 percent of all adults, they are not irrelevant but are, nevertheless, still a small minority. And while being distinct, they are not so different that their presence or absence can shift the total estimates by a non-negligible amount. The differences between the landline and cell-only samples are largest on measures that tend to vary strongly by age. Indicators of political engagement show very large differences, topped by voter registration (78 percent in the landline sample, 49 percent in the cell-only sample). Awareness that the Re- publican Party held the majority in the House of Representatives (at the time of the survey) was much greater among the landline sample (71 percent) than among the cell-only respondents (49 percent). Similarly, cell-only respondents were much less likely to report closely following the election or reading a newspaper yesterday, and more likely to say they are sometimes too busy to vote. 3. The set of weights used for the “landline sample” estimates was created through iterative proportional fitting. The landline sample was poststratified to align with CPS estimates for the cross- classifications of age and education, sex and education, and sex and age, race and Hispanic ethnicity, as well as marginals for region and county-level population density. The set of weights used for the “blended sample” estimates was created by poststratifying to NHIS estimates of telephone status (percent cell-only versus percent with a landline) in addition to the six aforementioned demographic variables.
Table 2. Summary of Survey Estimates by Sampling Frame Landline sample Cell only Blended sample Difference Difference (weighted) (unweighted) (landline plus cell only, (landline minus cell only) (landline minus blended weighted) sample) % % % % % Registered to votec 78 49 76 +29∗∗ +2 Has a cell phoneb 74 100 76 −26∗∗ −2 Ever voted in your precinctc ˆ 90 67 88 +23∗∗ +2 Aware of GOP majorityc 71 49 69 +22∗∗ +2 Sometimes too busy to votec 21 38 22 −17∗∗ −1 Newspaper yesterday: print onlyb 34 19 33 +15∗∗ +1 Closely following electionc ˆ 66 52 65 +14∗∗ +1 Approve of gay marriagea 37 51 37 −14∗∗ 0 Read newspaper yesterdayb 40 27 40 +13∗∗ 0 First time voterc ˆ 5 17 6 −12∗∗ −1 Regularly watch local TV newsb 54 43 53 +11∗∗ +1 Impact of the U.S. Cell-Only Population Always votec ˆ 47 36 47 +11∗∗ 0 Got news on one or more devicesb 7 15 8 −8∗∗ −1 Regularly visit local TV station sitesb 6 14 7 −8∗∗ −1 Conservativec 38 30 37 +8∗∗ +1 Feel guilty when I don’t votec 62 54 61 +8∗∗ +1 Voting doesn’t change thingsc 22 30 23 −8∗∗ −1 Regularly watch network newsb 28 21 27 +7∗∗ +1 Got news on cell phoneb 4 11 5 −7∗∗ −1 Know where to votec ˆ 88 81 88 +7∗∗ 0 House vote: Republican (March)a 37 30 37 +7∗∗ 0 Regularly watch network AM newsb 23 17 22 +6∗∗ +1 779 Downloaded from https://academic.oup.com/poq/article/71/5/772/1805417 by guest on 08 September 2021
Table 2. Continued Landline sample Cell only Blended sample Difference Difference (weighted) (unweighted) (landline plus cell only, (landline minus cell only) (landline minus blended 780 weighted) sample) % % % % % Regularly watch Daily Showb 6 12 7 −6∗∗ −1 Washington issues don’t affect mec 24 30 24 −6 0 Went online from home yesterdayb 42 37 41 +5 +1 Total online yesterdayb 53 48 51 +5 +2 Iraq war was right decisiona 39 44 40 −5 −1 Regularly watch cable TV newsb 34 38 35 −4 −1 Listened to radio news yesterdayb 36 32 36 +4 0 Newspaper yesterday: web onlyb 5 9 6 −4 −1 Regularly visit local newspaper sitesb 8 12 9 −4 −1 Regularly visit blogs about newsb 4 8 5 −4 −1 House vote: Republican (October)c 38 34 38 +4 0 Disapprove of President Busha 54 58 53 −4 +1 Got news online yesterdayb 23 26 23 −3 0 Regularly visit national newspaper sitesb 8 11 8 −3 0 Don’t know enough to votec 60 63 60 −3 0 Party ID: independent/othera 21 18 20 +3 +1 Watched television news yesterdayb 57 59 58 −2 −1 Got news on PDAb 2 4 2 −2 0 Got news from podcastb 2 4 2 −2 0 Regularly visit search engine news sitesb 18 19 18 −1 0 Went online from work yesterdayb 25 24 25 +1 0 Thought quite a lot about electionc 51 52 51 −1 0 Regularly visit TV network news sitesb 14 14 14 0 0 Generally bored by Washingtonc 36 36 36 0 0 Keeter et al. ∗∗ p< .01. SOURCE.—a Cell Phone Study March 8–28, 2006, b Media Consumption Survey April 22–May 22, 2006, c Voter Turnout Survey September 21–October 4, 2006. ˆ based on registered voters. Downloaded from https://academic.oup.com/poq/article/71/5/772/1805417 by guest on 08 September 2021
Impact of the U.S. Cell-Only Population 781 Differences on attitudinal measures tended to be considerably smaller than on measures of political engagement, though cell-only respondents were 14 points more likely to approve of gay marriage (51 percent versus 37 percent). They were also eight points less likely to describe their political ideology as “conservative.” In March, they were seven points less likely to say they would vote Republican in the congressional elections, but in October the difference Downloaded from https://academic.oup.com/poq/article/71/5/772/1805417 by guest on 08 September 2021 between the cell-only and landline respondents was just four points. Other substantive political differences were similarly modest: cell-only respondents were slightly more likely to disapprove of President Bush’s performance but also to say the war in Iraq was the right decision. While newspaper readership is considerably lower among the cell-only, dif- ferences in other media measures are more modest. Internet news sources are as commonly cited by cell-only respondents as by landline respondents. The cell-only are four points more apt to say they regularly watch cable TV news (38 percent versus 34 percent) and six points more likely to say they regularly watch The Daily Show with Jon Stewart. More than twice as many cell-only respondents report ever getting news on some type of portable electronic device such as a cell phone, PDA, or podcast (15 percent versus 7 percent). Although the point is relatively trivial, 100 percent of the cell-only sample has a cell phone, compared with only 74 percent of the landline sample, but the addition of the cell-only respondents to the overall sample changes the total survey estimate of cell phone ownership by only two percentage points. Estimates for Young Adults While the noncoverage problem is currently not damaging estimates for the entire population, it may very well be damaging estimates for subgroups in which using only a cell phone is more common. This concern is particularly relevant for young adults. According to the most recent government estimate, over 25 percent of those under age 30 use only a cell phone (Blumberg and Luke 2007a). In addition to age, Tucker, Brick, and Meekins (2007) report that cell-only status is associated with race/ethnicity, employment, marital status, and home ownership. A high noncoverage rate does not necessarily result in biased subgroup estimates. For example, if young adults accessible through landline samples are similar on the key survey measures to those who are excluded, then a landline- only sample design is sufficient. To assess the similarities between young adults from a landline and their cell-only peers, we compared unweighted results on demographic, lifestyle, and attitudinal questions. This section of the analysis is unweighted because the analytic goal is descriptive – how do these two groups of young adults compare? Here weighting would have the undesirable effect of distorting the raw differences between young adults reached on a landline and those reached on a cell phone. In the final piece of this analysis, we seek
782 Keeter et al. Table 3. Living Situation of Adults Aged 18–25 Years by Telephone Service Living situation Landline sample Cell only Live with parents 50 19 Rent 29 57 Own 14 10 Downloaded from https://academic.oup.com/poq/article/71/5/772/1805417 by guest on 08 September 2021 Live in a dorm 2 12 Other/refused 5 2 100% 100% (329) (130) Figures are unweighted. SOURCE.—GenNext Survey. to answer a different question – to what extent are RDD survey estimates for young adults affected by the inclusion of cell sample respondents? We use weighting to answer the latter question because the goal is gauging the effect on final survey estimates. Data for this analysis come from the GenNext survey, which included an oversample of 18- to 25-year-old respondents in the landline sample and a cell sample of an additional 250 adults in this age group (130 of whom reported not having a landline at home). BIVARIATE RESULTS Young adults who have a landline and young adults who are cell-only differ on several dimensions. One stark difference is their living arrangements, as shown in table 3. Half of the 18- to 25-year-old respondents in the landline sample reported living with their parents. This compares with fewer than one in five (19 percent) of their cell-only peers. The majority (57 percent) of cell- only young adults, by contrast, are renters, compared with 29 percent of their landline peers. These differences, and all those discussed in this section, are significant at least at the .05 level. Telephone status has a loose but perceptible association with certain be- haviors and mores among young adults. Replicating a result from the NHIS (Blumberg and Luke 2007b), we find that cell-only adults in this age group are more likely to report drinking alcohol in the past seven days (57 percent versus 36 percent). They are also more likely to say that it is okay for people to smoke marijuana (52 percent versus 38 percent). No significant differences were observed for self-reported smoking or illegal drug use. With respect to social values, young adults with landlines tend to be more traditional. They are significantly more likely to attend religious services at least once a week relative to their cell-only peers (30 percent versus 20 percent) and to believe that homosexuality should be discouraged by society (37 percent
Impact of the U.S. Cell-Only Population 783 versus 25 percent). Fewer adults aged 18–25 years with landlines say they have ever dyed their hair an untraditional color, compared with those who are cell-only (20 percent versus 29 percent). There is some evidence that young adults with landlines are less technology savvy than their cell-only peers. They are significantly less likely to report having sent or received an e-mail (49 percent versus 65 percent), text message Downloaded from https://academic.oup.com/poq/article/71/5/772/1805417 by guest on 08 September 2021 (42 percent versus 74 percent), or instant message (28 percent versus 38 per- cent) in the previous 24 hours. Similarly, about half (52 percent) of the young adults with landlines have used social networking web sites like Facebook or MySpace, compared with roughly three-fifths (62 percent) of those who are cell-only. MULTIVARIATE RESULTS The bivariate results reported above reveal several significant differences be- tween young adults reached in a landline sample and those who were reachable only on a cell phone. Some of these differences may be the product of com- mon underlying mechanisms. In order to identify the most influential drivers of cell-only status among the measures collected in the GenNext survey, we constructed a logistic regression model based only on respondents aged 18– 25 years. The model was constructed by first testing all main effects shown in table 4 as well as all first-order (two-way) interactions. The two significant interactions from the preliminary model (not shown) were retained in the final model. The model suggests that, within this age group, the most significant in- fluences on cell-only status are having moved away from parents and being unmarried. In addition, being a home owner has a significant though somewhat weaker negative association, while population density is positively related to cell-only status. The effects of fixed person-level characteristics tend to be of a lower magnitude. Blacks are somewhat less likely to be cell-only than other non- whites, and there is a significant main effect for Hispanic ethnicity as well as a significant interaction between Hispanic ethnicity and population density. (His- panics in high-density areas are especially likely to be cell-only.) Gender, family income, and being employed or attending school are not significant predictors of having only a cell phone. On balance, these results indicate that cell-only status has more to do with life-cycle effects than with stable individual characteristics such as gender, race, or ethnicity, although both sets of factors play a role. WEIGHTED SUBGROUP ESTIMATES Thus far we have established that young adults reached in landline samples differ on a number of dimensions from those reachable only in a cell sample. A key question for researchers is whether this difference affects weighted
784 Keeter et al. Table 4. Logistic Regression Model for Predicting Cell-only Telephone Status within the 18–25 Age Group (N = 579) Predictor of cell-only telephone status Estimate SE Odds ratio Live with parents −1.94∗∗∗ 0.27 0.14 Married −1.48∗∗∗ 0.38 0.23 Downloaded from https://academic.oup.com/poq/article/71/5/772/1805417 by guest on 08 September 2021 Own home −0.87∗ 0.35 0.42 Population density (high) 0.81∗∗ 0.30 2.24 Hispanic 0.85∗∗ 0.32 2.34 Black −1.01∗∗ 0.43 0.37 White 0.18 0.30 1.19 All other races (reference cell) Employed or student 0.15 0.35 1.16 Male 0.46 0.25 1.58 Family income (under $20,000) 0.73 0.39 2.08 Interactions Population density (high) ∗ Hispanic 2.48∗∗ 0.88 11.90 Family income (under $20,000) ∗ Male 1.43∗ 0.57 4.17 Likelihood ratio chi-square 97.93 p < 0.001 ∗p < .05. ∗∗ p < .01. ∗∗∗ p < .001. SOURCE.—GenNext Survey. subgroup estimates. To attempt to answer this, we computed two sets of weighted estimates for young adults – estimates based only on the landline sample and blended estimates based on both the landline and cell sample, which in this analysis includes persons reached through the cell sample who had both a cell and a landline phone, as well as cell-only individuals. Weights for the estimates based solely on the landline sample were created by raking the data to the six CPS demographic variables listed above. Weights for the blended estimates were created by a three-step process. First, all respondents aged 26 years or older in the GenNext study were assigned a base weight equal to the estimated total number of telephone numbers in the landline RDD frame servicing residences with only adults aged 26 years and older divided by the estimated number of sampled telephone numbers in the landline RDD frame servicing residences with only adults aged 26 years and older. Similarly, respondents aged 18–25 years who were interviewed in the landline sample were assigned a base weight equal to the estimated total number of telephone numbers in the landline RDD frame servicing residences with an adult aged 18– 25 years divided by the estimated number of sampled telephone numbers in the landline RDD frame servicing residences with an adult aged 18–25 years. Respondents aged 18–25 years who were interviewed in the cell sample were assigned a base weight equal to the estimated total number of telephone numbers
Impact of the U.S. Cell-Only Population 785 in the cell RDD frame servicing residences with an adult aged 18–25 years divided by the estimated number of sampled telephone numbers in the cell RDD frame servicing residences with an adult aged 18–25 years. Second, following procedures described by Brick et al. (2006) we used the simple dual frame estimator developed by Hartley (1962) to combine the data from respondents aged 18–25 years from the landline sample with those from Downloaded from https://academic.oup.com/poq/article/71/5/772/1805417 by guest on 08 September 2021 the cell sample (all of whom aged 18–25 years in the GenNext survey). All respondents aged 18–25 years (regardless of sample) who reported having both a landline and a cell phone were given a weight of 1/2 (0.50), while all other respondents received a weight of 1 at this step. The weight of 1/2 was used as an approximation to the optimal mixing parameter described by Hartley. Minor departures from the optimal mixing parameter generally have little impact on the efficiency of the estimates (Brick et al. 2006). Next, the combined sample (N = 1,501) was raked to a telephone service variable in addition to the demographic variables from the CPS. Control totals for the telephone service variable come from the NHIS, and for the purposes of the GenNext survey, the categories used were “age 18–25 cell-only,” “age 18– 25 cell and landline,” “age 18–25 landline-only,” and “age 26+.” (More details on this weighting procedure are available from the authors.) Given the potential for sampling, nonresponse, and measurement error to compromise the reliability of the steps outlined above, the blended sample estimates are most appropriately viewed as “experimental.” Several statistical difficulties such as estimating the relevant frame parameters and using different selection procedures in the two samples (among other factors) may very well lead to biased estimates. In assessing the blended sample estimates, however, we did not detect any obvious evidence of bias, likely because of the poststrat- ification to key parameters such as age and telephone status among those aged 18–25 years. Third, we assess the potential for bias resulting from noncoverage of cell-only young adults. In particular, we compare the weighted survey estimates for those aged 18–25 years based only on the landline sample in the GenNext survey with the weighted estimates for those aged 18–25 years based on the blended sample. The results are presented in table 5. The weighted estimates for young adults based only on the landline sample differ from the corresponding blended estimate for several measures, although only 3 out of the 35 comparisons reach statistical significance. Approximately two significant results would be expected from sampling variability alone. The direction of many of the differences, while generally not significant, reflect results observed in the unweighted comparisons. The blended estimates, which are based on both the landline and cell samples, suggest that a higher proportion of 18- to 25-year-old respondents use new technologies than is suggested by weighted estimate based on a landline sample only. For exam- ple, the blended sample estimate for the proportion in this age group sending or receiving a text message in the previous 24 hours was 55 percent, which
Table 5. Differences between Estimates for 18- to 25-Year-Old Respondents by Sample Design Landline Cell only Blended sample Difference Difference 786 sample (landline plus cell sample) (landline minus cell only) (landline minus blended sample) Demographics Married 17 8 15 +9∗ +2 Have children 31 19 24 +12∗ +7 Own home 14 10 13 +4 +1 Income under $20,000 24 22 23 +2 +1 Lifestyle Attend religious services weekly or more 31 20 25 +11∗ +6 Currently enrolled in school 37 59 47 −22∗∗ −10∗ Has health insurance 67 72 65 −5 +2 Rely on family for financial assistance 44 42 48 +2 −4 Has a tattoo 35 35 33 0 +2 Smoke (past week) 29 30 27 −1 +2 Drink alcohol (past week) 34 57 41 −23∗∗ −7 Play video games (past week) 34 36 36 −2 −2 Exercise (past week) 80 88 83 −8 −3 Take illegal drugs (past week) 8 11 8 −3 0 Technology usage Use Facebook, MySpace 50 62 57 −12∗ −7 Dated someone met online 12 15 11 −3 +1 In the last 24 hours. . . Sent or received an e-mail 48 65 51 −17∗∗ −3 Sent or received a text message 43 74 55 −31∗∗ −12∗∗ Sent or received an instant message 26 38 33 −12∗ −7 Political attitudes and behaviors Keeter et al. Registered to vote 56 56 47 0 +9∗ Downloaded from https://academic.oup.com/poq/article/71/5/772/1805417 by guest on 08 September 2021
Table 5. (Continued.) Landline Cell only Blended sample Difference Difference sample (landline plus cell sample) (landline minus cell only) (landline minus blended sample) Republican/lean Republican 33 40 37 −7 −4 Conservative 33 28 28 +5 +5 Agree that. . . 0 Government is wasteful and inefficient 43 40 42 +3 +1 Military strength is best way to peace 26 29 26 −3 0 Homosexuality should be accepted 57 68 61 −11∗ −4 I’m satisfied financially 49 53 51 −4 −2 Government should protect morality more 47 40 46 +7 +1 Technology makes people more isolated 69 66 69 +3 0 Say it’s okay for people to. . . Drink a lot of alcohol 25 36 29 −11∗ −4 Gamble 63 71 63 −8 0 Impact of the U.S. Cell-Only Population Have sex outside marriage 54 63 58 −9 −4 Have a baby outside marriage 56 55 51 +1 +5 Smoke marijuana 38 52 40 −14∗ −2 Pirate online music or videos 47 52 43 −5 +4 Use cable/wireless without paying 20 23 19 −3 +1 Sample size 329 130 579 ∗p < .05. ∗∗ p< .01. Missing data excluded from the analysis. Figures in the “Landline sample” and “Blended sample” columns are weighted. Figures in the “cell only” column are unweighted. SOURCE.—GenNext Survey. 787 Downloaded from https://academic.oup.com/poq/article/71/5/772/1805417 by guest on 08 September 2021
788 Keeter et al. compares to 43 percent based on the landline sample. On certain lifestyle mea- sure such as attending church and consuming alcohol, there is also some indi- cation that weighted estimates based only on a landline sample make this age group look somewhat more traditional in their values, but the sample sizes were not large enough to detect significant differences from the blended estimates. With absent gold standard measures, it is difficult if not impossible to deter- Downloaded from https://academic.oup.com/poq/article/71/5/772/1805417 by guest on 08 September 2021 mine with confidence whether the blended or the landline sample estimates are more accurate. The American Community Survey (ACS), a massive national area-probability study conducted by the U.S. Census Bureau, provides bench- mark estimates for young adults on a handful of the items in this analysis. The weighted landline sample estimate mirrors the ACS benchmark for the propor- tion of 18- to 25-year-old respondents who are married (17 percent), while the blended sample yields a slight underestimate (15 percent). On income, both of the GenNext survey estimates overstate the proportion in the age group with an annual income under $20,000 (19 percent). Lastly, in estimating the propor- tion of 18- to 25-year-old respondents enrolled in school, the landline sample understates and the blended sample overstates the ACS figure (41 percent) by equal magnitudes. It should be noted that the ACS is a mixed mode study with somewhat different question wording than the GenNext study. Consequently, the differences noted here may, in part, be attributable to measurement or nonresponse error rather than solely due to coverage error. No such gold standard estimates are available for items pertaining to politi- cal ideology and party affiliation. The results are mixed in the GenNext survey concerning the effect of including a cell sample on subgroup estimates for these items. The blended sample estimate of the proportion of those aged 18– 25 years who are Republican is somewhat higher than the corresponding landline sample estimate. The blended estimate for the proportion who are conservative, however, is somewhat lower than the corresponding landline sample estimate. Neither difference is statistically significant. Furthermore, the landline-only and blended sample estimates were essentially comparable on specific policy questions, such as the role of government in protecting morality and the use of military strength to bring about peace. The absence of a consistent effect for sample design on political items corroborates the Keeter (2006) finding that vote preference is, at least currently, robust to this methodological issue. That is, it appears that political attitudes are generally unrelated to telephone status. We also see from this analysis, however, that other measures such as use of alcohol and new technologies are related to telephone service – leading to a problematic dependence of the estimate on the sample design. Discussion Across four surveys with different substantive content, we find that including a cell-only sample with a landline RDD sample produces general population
Impact of the U.S. Cell-Only Population 789 estimates that are nearly identical to those from the landline sample alone. Thus, there is little evidence here that landline samples produce more bias than dual frame samples, a finding consistent with those reported by Brick et al. (2006). In addition to providing a look at how cell phone users compare with landline respondents socially, politically, and demographically, collectively these studies Downloaded from https://academic.oup.com/poq/article/71/5/772/1805417 by guest on 08 September 2021 also allow us to assess the feasibility of conducting a telephone survey in a cell phone sampling frame. The conclusion is that cell phone surveys are feasible, but are considerably more difficult and expensive to conduct than landline surveys. Interviewing people on cell phones presents unique challenges that require new procedures and have implications for overall costs (Lavrakas and Shuttles 2005). Cell phones are often used by people while they are driving or otherwise distracted, presenting potential safety hazards or, at a minimum, potentially reducing the attention of the respondent to the survey tasks. The four surveys reported here always asked cell sample respondents if they were in a place where it was safe to talk, a question that also provides reluctant respondents with an opportunity to terminate the call. Even if respondents are willing to do the interview at another time, this consideration necessitates the establishment of an appointment for a follow-up call. An important cost consideration is that federal regulations prohibit the use of automated dialing devices when calling cell phones so each number in the cell phone sample must be dialed manually. Somewhat offsetting this is the fact that fewer numbers in the cell phone frame are unassigned. Another important difference occurs in eligibility rates. If the goal is to interview cell-only respondents, rather than everyone in the cell phone frame, significant amounts of screening are necessary. In the two studies that screened for cell-only respondents, just 16 percent were eligible. (In the Pew/AP/AOL Cell Phone Study, 26 percent of those interviewed in the cell frame said they had no landline phone. The reason for the difference is unclear.) In addition, most U.S. cell users incur some type of usage charge or loss of prepaid minutes that discourages cooperation. While cooperation rates in the cell samples were somewhat higher than those in the landline samples, this slight advantage might be a result of the monetary incentive offered to cell-only respondents. The $10 inducement may encourage participation, but it also increases the costs associated with conducting cell phone surveys. As a result of these differences between landline and cell phone samples, data collection costs (apart from overall study design, programming, and analysis costs) are substantially higher for the cell phone samples. Given the logistical and cost implications associated with dual frame de- signs, the utility of including cell-only or cell-frame respondents with landline samples appears marginal, at least at present. Yet the rapid growth in the size of the cell-only population and its continued concentration among younger peo- ple means that its potential impact warrants continued study. And the fact that
790 Keeter et al. landline-based estimates of certain characteristics and behaviors among young people are biased by the absence of the cell-only population means that the choice of whether to employ a dual frame design must be guided by a study’s target population and subject matter. Appendix: Summary of Landline and Cell Phone Sample Downloaded from https://academic.oup.com/poq/article/71/5/772/1805417 by guest on 08 September 2021 Response Rates Survey AAPOR response rate 3 Cell (%) Landline (%) Pew/AP/AOL: Cell Phone Survey 20 30 2006 Biennial Media Consumption Survey 24 29 Pew/MacNeil–Lehrer: GenNext Survey 29 25 Pew/AP: Turnout Survey 25 28 Pew/AP/AOL Cell Phone Survey: ASK LANDLINE SAMPLE ONLY: Do you happen to have a cell phone or not? 1 Yes 2 No 9 Don’t know/Refused CELL SAMPLE AND LANDLINE REACHED ON CELL (IF CELL SAMPLE OR ‘1’ IN Q12 ASK): Is the cell phone your only phone or do you also have a regular telephone at home? 1 Cell is only phone 2 Has regular phone at home 9 Don’t know/refused LANDLINE AND CELL USER [IF (CELL SAMPLE AND Q13 = 2) OR (LANDLINE SAMPLE AND Q2 = 1) ASK]: Thinking about all the phone calls you make, do you make more calls with your cell phone or more calls with your regular home phone? 1 More with cell phone 2 More with home phone 3 About equal (VOL.) 9 Don’t know/refused USE CELL PHONE MORE (IF ANSWERED ‘1’ IN Q19 ASK): Would that be a LOT MORE or just a FEW more with your cell phone? 1 A lot more
Impact of the U.S. Cell-Only Population 791 2 A few more 9 Don’t know/refused (VOL.) USE REGULAR PHONE MORE (IF ANSWERED ‘2’ IN Q19 ASK): Would that be a LOT MORE or just a FEW more with your regular home phone? Downloaded from https://academic.oup.com/poq/article/71/5/772/1805417 by guest on 08 September 2021 1 A lot more 2 A few more 9 Don’t know/refused (VOL.) 2006 Biennial Media Consumption Survey: Is the cell phone your only phone or do you also have a regular telephone at home? 1 Cell is only phone 2 Has regular phone at home 9 Don’t know/refused Pew/MacNeil–Lehrer GenNext Survey: ASK CELL SAMPLE: Is the cell phone your only phone or do you also have a regular telephone at home? 1 Cell is only phone 2 Has regular phone at home 9 Don’t know/refused ASK LANDLINE SAMPLE ONLY: Do you happen to have a cell phone or not? 1 Yes 2 No 9 Don’t know/refused IF R HAS BOTH CELL AND LANDLINE, ASK: Of all the phone calls that you receive, about how many are received on a cell phone? Would you say. . . 1 All or almost all 2 More than half 3 Less than half, or 4 Very few or none 9 Don’t know/refused (VOL.) Pew/AP/AOL Turnout Survey: Is a cell phone your only phone, or do you also have a regular phone where you currently live? 1 Only phone 2 Have regular phone at home 9 Don’t know/refused
792 Keeter et al. References Blumberg, Stephen J., and Julian Luke V. May 14, 2007a. “Wireless Substitution: Early Release of Estimates Based on the National Health Interview Survey, July–December 2006.” Report by the U.S. Centers for Disease Control and Prevention. Blumberg, Stephen J., and Julian Luke V.. 2007b. “The Prevalence of Wireless Substitution and Its Impact on Random-Digit-Dial Health Surveys of Young Adults and Adults in Poverty.” Paper Downloaded from https://academic.oup.com/poq/article/71/5/772/1805417 by guest on 08 September 2021 presented at the Annual Meeting of the American Association for Public Opinion Research, Anaheim, CA. Brick, Michael J., Pat D. Brick, Sarah Dipko, Stanley Presser, Clyde Tucker, and Yangyang Yuan. 2007. “Cell Phone Survey Feasibility in the U.S.: Sampling and Calling Cell Numbers Versus Landline Numbers.” Public Opinion Quarterly 71:23–39. Brick, Michael J., Sarah Dipko, Stanley Presser, Clyde Tucker, and Yangyang Yuan. 2006. “Nonre- sponse Bias in a Dual Frame Sample of Cell and Landline Numbers.” Public Opinion Quarterly 70:780–93. Hartley, Herman O. 1962. “Multiple Frame Surveys.” Proceedings of the Social Statistics Section, American Statistical Association, 203–6. Keeter, Scott. 2006. “The Impact of Cell Phone Noncoverage Bias on Polling in the 2004 Presi- dential Election.” Public Opinion Quarterly 70:88–98. Lavrakas, Paul, and Charles Shuttles. 2005. “Cell Phone Sampling Summit II Statements on Accounting for Cell Phones in Telephone Survey Research in the U.S.” Available at http://www.nielsenmedia.com/cellphonesummit/statements.html, accessed August 26, 2007. Lepkowski, James M., Clyde Tucker, J. Michael Brick, Edith de Leeuw, Lilli Japec, Paul J. Lavrakas, Michael W. Link, and Roberta L. (eds.). 2007. Advances in Telephone Survey Method- ology. New York: Wiley. National Council on Public Polls. N.D. “2004 Poll Analysis.” Available at http://www.ncpp.org/?q=node/26, accessed August 26, 2007. Steeh, Charlotte. 2004. “A New Era for Telephone Surveys.” Paper presented at the Annual Meeting of the American Association for Public Opinion Research, Phoenix, AZ. Tucker, Clyde, Michael J. Brick, and Brian Meekins. 2007. “Household Telephone Service and Usage Patterns in the United States in 2004: Implications for Telephone Samples.” Public Opinion Quarterly 71(1):1–22. Zukin, Cliff. 2006. “The Future is Here! Where Are We Now? And How Do We Get There?” Public Opinion Quarterly 70:426–42.
You can also read