THE VERY SEPARATE WORLDS OF ACADEMIC AND PRACTITIONER PERIODICALS IN HUMAN RESOURCE MANAGEMENT: IMPLICATIONS FOR EVIDENCE-BASED MANAGEMENT
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
娀 Academy of Management Journal
2007, Vol. 50, No. 5, 987–1008.
THE VERY SEPARATE WORLDS OF ACADEMIC AND
PRACTITIONER PERIODICALS IN HUMAN
RESOURCE MANAGEMENT:
IMPLICATIONS FOR EVIDENCE-BASED MANAGEMENT
SARA L. RYNES
TAMARA L. GILUK
KENNETH G. BROWN
University of Iowa
It is hardly news that many organizations do not tific evidence. Although this point may seem obvi-
implement practices that research has shown to be ous, it is hardly trivial. For example, unlike
positively associated with employee productivity medicine, education, or law, management is not
and firm financial performance (e.g., Hambrick, truly a profession (Leicht & Fennell, 2001; Trank &
1994; Johns, 1993; Pfeffer & Sutton, 2000). Indeed, Rynes, 2003). As such, there is no requirement that
the failure to implement research-supported prac- managers be exposed to scientific knowledge about
tices has been observed in nearly every field where management, that they pass examinations in order
there is a separation between those who conduct to become licensed to practice, or that they pursue
research and those who are in a position to imple- continuing education in order to be allowed to
ment research findings (Lewis, 2003; Rogers, 1995; maintain their practice. Furthermore, since the first
Straus, Richardson, Glasziou, & Haynes, 2005). choice of most managers seeking information is to
The gap between science and practice is so per- consult other managers (e.g., Brown & Duguid,
sistent and pervasive that some have despaired of 2002; Wenger, McDermott, & Snyder, 2002) and
its ever being narrowed. Nevertheless, over the past since extremely few managers read academic pub-
decade or so, attempts to deal with the problem lications (Rynes, Colbert, & Brown, 2002), the ques-
have evolved in the form of movements toward tion of how to inform managers about scientific
“evidence-based” practice in such fields as medi- evidence is anything but trivial.
cine, education, marketing, rehabilitation, and psy- One way in which aspiring managers can learn
chology (APA Task Force, 2006; Ford, 2005; Law, about management-related evidence is through for-
2002; Southworth & Conner, 1999; Straus et al., mal education. However, even the acquisition of a
2005). formal master’s or bachelor’s degree in business is
In the field of management, the nascent move- no guarantee that a student has learned evidence-
ment toward evidence-based practice is known as based principles. This is because many textbooks
“evidence-based management,” or EBM. According do not cover research findings, and many individ-
to Rousseau, “Evidence-based management means uals teaching in business schools do not have
translating principles based on best evidence into Ph.D.’s and are unlikely to know about scientific
organizational practices. Through evidence-based evidence in their field of instruction (Trank &
management, practicing managers develop into ex- Rynes, 2003). Furthermore, there are millions of
perts who make organizational decisions informed managers who do not hold formal degrees in man-
by social science and organizational research–part agement. How might these managers receive infor-
of the zeitgeist moving professional decisions away
mation that is consistent with the best available
from personal preference and unsystematic experi-
scientific evidence about how various management
ence toward those based on the best available sci-
practices influence business outcomes?
entific evidence” (2006: 256).
One possible way is through periodicals aimed at
For evidence-based management (EBM) to take
practitioners, either in specialty areas or in general
root, it is necessary—though far from sufficient—
management. For example, in the area of human
that managers be exposed to, and embrace, scien-
resource (HR) management, Rynes, Colbert, and
Brown (2002) found that by far the most widely
The authors would like to thank Bennett Postlethwaite read periodical is HR Magazine, which is pub-
for insightful comments on earlier versions of this article lished by HR’s major professional association, the
and Todd Darnold and Jaclyn Tholl for their assistance in Society for Human Resource Management (SHRM)
coding the data. and has a circulation of more than 200,000. An-
987
Copyright of the Academy of Management, all rights reserved. Contents may not be copied, emailed, posted to a listserv, or otherwise transmitted without the copyright holder’s express
written permission. Users may print, download or email articles for individual use only.988 Academy of Management Journal October
other HR periodical that is relatively widely read, cent of practitioners actively disagreeing with or
and that aims specifically to create a bridge be- not knowing about) the following research
tween scientists and practitioners of HR, is Human findings:1
Resource Management. Alternatively, in the case of • Intelligence predicts job performance better than
general management, the most highly regarded pe- conscientiousness (Schmidt & Hunter, 1998).
riodical is the Harvard Business Review, which has • Screening for intelligence results in higher job
a circulation of 240,000 and is published in 12 performance than screening for values or values
languages. HBR is another publication that at- fit (Meglino & Ravlin, 1998; Schmidt & Hunter,
tempts to bridge the worlds of science and practice 1998).
and that has at least some readership among HR • Being very intelligent is not a disadvantage for
managers, directors, and vice presidents (Rynes et performing well on a “low-skilled” job (Hunter,
al., 2002). 1986; Schmidt & Hunter, 1998).
In this editorial, we examine the extent to which • Personality inventories vary considerably in
three important HR-related research findings are terms of how well they predict applicants’ job
being “translated” and “transferred” to practitio- performance (Barrick & Mount, 1991; Gardner &
ners via these three widely read periodicals. We Martinko, 1996).
then discuss some implications of our findings for • Integrity tests successfully predict whether
the prospects of EBM in HRM and invite commen- someone will steal, be absent, or otherwise take
taries from other individuals who are in a good advantage of employers, even though individu-
position to reflect on our findings. First, however, als can “fake good” on them (Ones, Viswesvaran,
we explain how we chose our topics of study. & Schmidt, 1993; Ones, Viswesvaran, & Reiss,
1996).
WHAT SCIENTIFIC EVIDENCE SHOULD • Integrity tests do not have adverse impact on
WE STUDY? racial minorities (Ones & Viswesvaran, 1998).
• Goal setting is more effective for improving per-
Because the task of moving toward EBM is so formance than is employee participation in de-
daunting, priorities must be set as to what specific cision making (Locke, Feren, McCaleb, Shaw, &
types of scientific evidence are most important to Denny, 1980; Locke & Latham, 1990; Wagner,
translate and transfer. At the risk of making a some- 1994).
what obvious point, Rousseau and McCarthy (2007) • The tendency to make errors in performance ap-
argued that scholars should begin EBM by focusing praisal is very difficult to eradicate through
on issues about which there is a clear scientific training (London, Mone, & Scott, 2004).
consensus on findings. In addition, an issue should • People’s actual behavior suggests that pay is
be important rather than trivial (Priem & Rosen- much more important to them than they imply in
stein, 2000). To use a medical analogy, we should surveys (Rynes, Gerhart, & Parks, 2005; Rynes,
focus on “number one killer” issues before moving Schwab, & Heneman, 1983).
on to less consequential concerns. Third, we
should focus most of our attention on topics for As these findings show, the two largest areas in
which the scientific findings are not obvious to which a gap looms between research results and
practitioners—that is, on problems that managers, practitioner knowledge or beliefs are (1) the impor-
left to their own devices, will likely “solve” by tance of intelligence in predicting job performance
doing something other than what sound research and (2) the usefulness of personality and integrity
evidence would support (Gordon, Kleiman, & tests for predicting job performance and counter-
Hanie, 1978; Priem & Rosenstein, 2000). productive work behaviors. However, Rynes and
colleagues (2002) did not determine the extent to
which the HR research community regarded each
Studying Practitioners’ Views
of their 35 items as “important.”
In the HR area, previous research has already
identified a number of clear scientific findings that
Web Survey of HR Researchers
are not obvious to practitioners. Specifically, Rynes
and her colleagues (Rynes et al., 2002) surveyed Therefore, to provide this third necessary piece
nearly 1,000 HR vice presidents, directors, and of information for prioritizing research findings for
managers to identify which of 35 well-documented
research findings HR practitioners widely disbe-
lieve. Their results showed widespread disagree- 1
See Rynes et al. (2002) for additional documentation
ment or lack of knowledge (i.e., more than 50 per- regarding these research findings.2007 Rynes, Giluk, and Brown 989
EBM, we conducted a Web-based survey of HR TABLE 1
research experts. Specifically, we surveyed the ed- Editorial Board Members’ Assessments of the
itorial board members of four journals: Personnel Most Fundamental Findings from
Psychology (PP), the Journal of Applied Psychology Human Resources Researcha
(JAP), the Academy of Management Journal (AMJ),
Number of
and Human Resource Management (HRM).2 Each
Finding Responses
board member was asked the following question,
which was answered in an open-ended format: “In
your opinion, what are the five most fundamental General mental ability is the strongest, or one of 22
the strongest, predictors of performance
findings from human resources research that all
practicing managers should know? Your answer Setting goals and providing feedback is a highly 22
need not cite specific studies—we are interested in effective motivational practice
fundamental, generalizable principles.” In all, 208
HR practices are important to organizational 21
board members were contacted. Usable responses outcomes
were received from 85 board members, for a 41
percent response rate. Of these, 174 served on one Structured interviews are more valid than 16
unstructured ones
board, 30 on two boards, and four on three boards.
To analyze the results, the first author evaluated Valid selection practices are very important to 15
the items, sorted them into theme-based categories, performance outcomes
and attached tentative names to the categories.3
Personality is related to performance 11
The second author was then provided with the
category names and asked to independently sort a
Findings with ten or more responses on a Web-based survey
the items. The two raters agreed on 71 percent of of board members from a set of academic journals.
the category choices, and the third author resolved
the differences.
Table 1 presents our results for the six topics Combining the findings from Rynes et al. (2002)
receiving at least ten mentions. This table shows and the board member survey reveals that three
that seven of the nine items identified by Rynes, content areas stand out as both containing clear and
Colbert, and Brown (2002) as exhibiting large gaps important research findings and suffering a gap
between scientific findings and practitioner beliefs between HR researchers’ and HR practitioners’
are also regarded as very important findings by evaluations of these findings: the importance of
researchers: the three items pertaining to intelli- intelligence or GMA for performance; the impor-
gence (also known as “general mental ability,” or tance of goal setting and feedback for performance,
“GMA”), the three items relating to personality, and the validity of personality (of which integrity
and the item concerning the effectiveness of goal tests are one representation) for predicting perfor-
setting for improving performance.4 mance. Thus, these became the three content areas
examined for coverage in practitioner and bridge
journals during the main phase of our research.
2
In the case of AMJ, we surveyed only those board
members for whom HR was a primary research area. For Research Questions
HRM, we surveyed only those board members who were
academics. Three major questions governed our examination
3
Items could also be sorted into more than one cate- of coverage of these three topics—intelligence, per-
gory, if appropriate. For example, the item, “Cognitive sonality, and goal setting—in practitioner and
ability and personality tests are valid predictors of per- bridge periodicals:
formance,” was sorted into both the “general mental abil-
1. How much coverage did each of these three top-
ity” and “personality” categories.
4
ics receive in major practitioner and bridge pe-
Illustrative responses for the GMA category in-
riodicals between 2000 and 2005?
cluded, “Cognitive ability is the single most important
2. To what extent is the content of coverage in
predictor of human performance,” “Ability tests have
high validity,” and “General mental ability is a valid practitioner and bridge journals consistent or
predictor of all job performance.” Responses for the goal
setting category included, “Goals really matter,” “Setting
specific, difficult attainable goals increases perfor- for personality included, “Conscientiousness predicts
mance,” “Specific, difficult goals with feedback are performance in most jobs,” “[We should] hire people on
highly effective motivators,” and “The power of goal- the basis of ability and personality,” and “Effect of per-
setting and similar motivational techniques.” Responses sonality on performance.”990 Academy of Management Journal October
inconsistent with peer-reviewed research coverage of GMA, personality, and goal setting in
findings? the three periodicals, with articles being the unit of
3. What sources of evidence are presented in each analysis. However, we excluded all articles that
periodical? were not at least a full page long, as articles of less
than a page are quite common in HR Magazine and
generally do not contain research-relevant informa-
METHODS
tion (e.g., book and product reviews, current event
Sample updates, awards, or profiles of companies or HR
practitioners). In addition, we eliminated the
To investigate the extent to which the three im-
“Forethought” sections of HBR (because these
portant HR research findings had received coverage
pieces are, again, less than a page long), as well as
in practitioner and bridge journals since Rynes et
HBR’s fictional case studies. Application of these
al. (2002) collected their data (in 1999), we con-
criteria resulted in the coding of 1,490 articles: 785
structed a database of articles from HR Magazine,
for HR Magazine, 168 for HRM, and 537 for HBR.
Human Resource Management (HRM), and the Har-
vard Business Review (HBR) for the six-year period
2000 – 05. Each of these periodicals represents a Database
somewhat different slice of the practitioner do-
Information about each of the 1,490 articles was
main. HR Magazine is a specialist periodical, focus-
gathered from Business Source Premier and en-
ing on HR managers. Patterns of both readership
tered into a spreadsheet. For each article, we re-
and authorship suggest that HR Magazine has a
corded the abstract and the first five keywords
very strong practitioner focus. For example, 84.2
listed by Business Source Premier, as well as basic
percent of the practitioners surveyed by Rynes and
bibliographic information (e.g., authors, journal,
her coauthors (Rynes et al., 2002) “usually” or “al-
volume, and page numbers). In addition, we coded
ways” read HR Magazine, and very few of its arti-
whether the authors were all academics, all practi-
cles have academic authors or coauthors (only 6.6
tioners, or a mix of academics and practitioners.
percent between 2000 and 2005). HRM is also a
To facilitate article searches on particular topics,
specialist periodical, read by fewer practitioners
we began by creating a master list of the keywords
than HR Magazine, but more likely to be read by
that occurred in HR Magazine, HRM, and two top-
those with higher education and position levels
tier academic journals specializing in HR content
(Rynes et al., 2002). Between 2000 and 2005, aca-
(the Journal of Applied Psychology and Personnel
demics authored the majority of HRM articles
Psychology).6 This process resulted in 289 key-
(64%); practitioners authored 20 percent, and
words. To reduce this large number of keywords,
mixes of academics and practitioners wrote 16 per-
all three authors jointly used the card sort method
cent. Finally, HBR is the most widely read and most
to create a smaller set of broader categories. For
highly respected general management bridge jour-
example, the general category “selection” included
nal for managers. We regard it as more of a “bridg-
the following keywords: “ability—testing,” “appli-
ing” than “practitioner” journal because it is read
cations for positions,” “assessment centers,” “cog-
by both academics and managers, and because its
nitive abilities test,” “employee screening,” “em-
articles are almost evenly authored by academics
ployee selection,” “employment interviewing,”
and practitioners (from 2000 through 2005, aca-
“employment tests,” “examinations,” “interview-
demics authored 41 percent; practitioners, 45 per-
ing,” “interviews,” and “personality tests.” We
cent; and combinations, 14 percent).5
placed all keywords that were difficult to classify
The intent of the content analysis was to seek
in a “miscellaneous” category. These steps resulted
in 57 initial categories. An advanced graduate stu-
dent in human resources then performed the same
5
We realize there are other practitioner and bridge
periodicals that contain HR-related content. However,
6
we believe that the three selected periodicals represent We used two academic journals, a bridge journal,
the clearest exemplars of the three genres (specialist and a practitioner journal in HR in generating keywords
practitioner, specialist bridge, and generalist bridge), at in order to make sure that both practitioner and academic
least in North America. For example, Deadrick and Gib- concepts of the field of HR management were incor-
son (2007) also chose HR Magazine and HRM as their porated. We did not incorporate HBR at the keyword
“professional-oriented” comparison points to two HR ac- generation stage because it is a general management
ademic journals, the Journal of Applied Psychology and journal with many keywords being clearly outside the
Personnel Psychology, in their analysis of the HR re- range of HR management (e.g., marketing, operations
search-practice gap. management).2007 Rynes, Giluk, and Brown 991
card sort using the author-generated categories, TABLE 2
placing 76 percent of the items in the author-gen- Percent Coverage of General Mental Ability,
erated categories. Discussion between the third au- Personality, and Goal Setting, by Periodicala
thor and the graduate student was used to create
HR Human Resource Harvard Business
consensus on the remaining keywords.
Topic Magazine Management Review
To search for articles related to the use of either
intelligence/GMA or personality in selection, we Ability 0.0% 1.2% 0.4%
initially conducted a broad search by focusing on Personality 0.4 1.2 0.6
articles including any of the keywords that re- Goal setting 0.6 0.6 0.6
flected either “selection” or “recruiting”7 (since the Total number 785 168 537
two functions often occur simultaneously and are of articles
difficult to separate in practice), or any mention of
a
“intelligence,” or any personality trait. This search Figures represent the percentages of all full-length articles
appearing in each periodical between 2000 and 2005 generated
yielded 98 articles from HR Magazine, 21 from
by both keyword and manual searches of article content by two
HRM, and 23 from HBR. Interrater reliability was of the three authors (see the text for details). To be included in
not an issue, since the keywords were taken di- the table, articles on “personality” had to discuss personality in
rectly from the spreadsheet. However, because the context of selection (as opposed to postselection manage-
some of the keywords were very broad (e.g., “psy- ment of different personality types).
chological tests,” “college students”), not all of the
keyword-identified articles truly focused on selec- 168, 1.2%) that discussed the ability-performance
tion. To deal with this reality, the first and second link, and HBR also had two (of 537, 0.4%).8
authors independently reviewed all 142 articles Results were not much different for personality
and highlighted those they thought were inappro- or goal setting. The role of personality in selection
priately categorized. The few cases of disagreement was the topic of three articles (0.4%) in HR Maga-
(less than 10 percent in each of the three categories) zine, two in HRM (1.2%), and three (0.6%) in HBR.
were resolved via joint discussion. The omission of Similarly, there were five articles (0.6%) on goal
non-selection-related articles resulted in a subset of setting in HR Magazine, one (0.6%) in HRM, and
116 articles: 91 from HR Magazine, 20 from HRM, three in HBR (0.6%).
and 5 from HBR. Thus, there is a clear gap in the extent of coverage
Similar steps were followed for goal setting—that of GMA, personality, and goal setting between ac-
is, initial keyword searches (keywords were “feed- ademic journals on the one hand9 and practitioner
back,” “goals,” and “goal setting in personnel man- and bridge periodicals on the other. The nearly
agement”), followed by examination of abstracts nonexistent coverage of intelligence, personality,
and article content to eliminate irrelevant articles and goal setting by practitioner and bridge journals
(e.g., ones on 360-degree feedback that did not con- is consistent with (and may be linked to) Rynes et
tain any discussion of goals). These steps produced al.’s (2002) finding that the largest gaps between
12 goal setting articles: 5 from HR Magazine, 1 from research findings and practitioner beliefs occur in
HRM, and 6 from HBR. these areas.
RESULTS Research Question 2: Research Consistency
of Coverage
Research Question 1: Extent of Coverage
Beyond this difference in quantity of coverage, it
Our first search was for articles related to the role
is also interesting to examine the extent to which
of GMA in job performance. Despite the high de-
gree of importance placed by research academics
on the findings related to the intelligence-perfor- 8
In general, we would not expect HBR to provide as
mance link, our search revealed almost no coverage much relative coverage of HR issues as the other two
of this topic in the three practitioner and bridge periodicals, given that it is a general management (rather
periodicals. Specifically, HR Magazine had no ar- than an HR-focused) periodical.
9
ticles (of 785 total, 0%) regarding GMA over that For example, analogous figures in Journal of Applied
time period (see Table 2). HRM had two articles (of Psychology, a top-tier academic journal in this area, were
3.2 percent for GMA, 5.9 percent for personality, and 2.5
percent for goal setting. Figures for another top-tier aca-
7
Keywords for “recruiting” were “college stu- demic journal, Personnel Psychology, were 6.9 percent
dents,” “employees—recruiting of,” and “help-wanted for GMA, 6.3 percent for personality, and 4.2 percent for
advertising.” goal setting.992 Academy of Management Journal October
coverage of these topics in practitioner and bridge business settings. But in rejecting IQ testing alto-
journals is consistent with research evidence. We gether, hiring managers have turned their backs on
discuss each of the three topics in turn. the single most effective assessment of cognitive
Intelligence/GMA. No articles on GMA appeared abilities, simply because there isn’t a version that
applies to the corporate world. They have dismissed
in HR Magazine over the relevant time period.
the one method that could help them identify busi-
However, two articles in HRM did deal with the ness stars. (2005: 100)
role of GMA in selection. In the first, O’Leary, Lind-
holm, Whitford, and Freeman (2002) explained the He recommended, as a remedy, situational inter-
recruitment and selection practices of the U.S. fed- views that focus on “cognitive subjects associated
eral government. These practices include the use of with executive work: accomplishing tasks, working
a variety of cognitive and noncognitive tests de- with and through others, and judging oneself. The
signed to match individuals’ abilities, personality, questions shouldn’t require specific industry ex-
and social skills with the requirements of four dif- pertise or experience. Any knowledge they call for
ferent occupational groups (administrative sup- must be rudimentary and common to all execu-
port, professional, managerial, and trades/labor). tives” (Menkes, 2005: 102). This recommendation
The authors cited a considerable amount of aca- is consistent with a considerable amount of empir-
demic research on both the validity and utility of ical evidence (e.g., Latham & Saari, 1984; Schmidt
alternative selection devices and provided descrip- & Hunter, 1998, 2000) and provides a counterpoint
tions of how research findings guide OPM’s inter- to the commonly held (but incorrect) assumption
nal selection and placement research. In short, the that intelligence can only be assessed with “intel-
article frequently references the academic literature ligence tests.”
on GMA and is highly consistent with it. In the second article related to intelligence,
The other relevant HRM article was part of a “Deep Smarts,” Leonard and Swap wrote:
special issue (Burke, Drasgow, & Edwards, 2004)
designed to illustrate how psychology-based re- When a person sizes up a complex situation and
search can be usefully applied in HR management. comes to a rapid decision that proves to be not just
Articles for this issue were authored by academic- good but brilliant, you think, “That was smart.”
practitioner teams in nine areas of HR practice, After you’ve watched him do this a few times, you
realize you’re in the presence of something special.
including recruitment and selection. Because of the
It’s not raw brainpower, though that helps. It’s not
special issue’s overriding focus on the applicability
emotional intelligence, either, though that, too, is
of psychological research, the article on recruit- often involved. It’s deep smarts, the stuff that pro-
ment and selection (Ryan & Tippins, 2004) is also duces that mysterious quality, good judgment.
highly consistent with research evidence. For ex- (2004: 88)
ample, Ryan and Tippins (2004) drew on previous
research to compare various selection tools (includ- This article maps less well onto peer-reviewed re-
ing GMA tests, integrity tests, and measures of con- search findings than does the Menkes (2005) arti-
scientiousness) on validity, costs, and sizes of av- cle. For example, what Leonard and Swap call
erage group differences in scores (e.g., male versus “deep smarts” is what academic researchers call
female, and white versus black, Hispanic, and “expert judgment”—a process whereby individuals
Asian). In addition, they discussed various selec- subconsciously match complex environmental
tion tools and strategies in terms of both their use- stimuli with some deeply held category, pattern, or
fulness for reducing adverse impact and their likely feature acquired over many years of experience
impact on applicants’ perceptions (an important (Dane & Pratt, 2007; Simon, 1996). As such, the
aspect in recruitment). Thus, the HRM articles on term “deep smarts” contributes to what researchers
GMA, though not numerous, are highly consistent call “construct proliferation,” or creating new la-
with research findings. bels for phenomena that have already been well-
HBR also published two articles on intelligence researched under another name.
during this period. In “Hiring for Smarts,” Menkes In addition, by introducing the word “smarts” to
presented a largely research-consistent argument indicate a combination of intelligence and experi-
for assessing intelligence when hiring managers: ence in a particular type of job or activity, all three
constructs (smarts, intelligence, and experience)
So much has been written about leadership person-
ality and style that hiring managers are in danger of become muddied. And although it is true that in-
neglecting the most critical factor in executives’ suc- telligence (what the authors call “raw smarts”) is
cess: intelligence. . . . Historically, the only reliable insufficient for producing deep expert knowledge,
measure of such brainpower has been the standard it will still be the best predictor of it at any given
IQ test which, for good reasons, is rarely used in level of experience (i.e., holding experience con-2007 Rynes, Giluk, and Brown 993
stant). This is because of what intelligence is and behaviors: Andrews (2005), Bates (2002), and Krell
the way it works: (2005). Andrews (2005) began with a discussion of
personal and business ethics and then asked
Intelligence is the ability to grasp and reason cor-
whether personality tests can help detect those
rectly with abstractions (concepts) and solve prob-
lems. However, perhaps a more useful definition is
likely to engage in unethical or other counterpro-
that intelligence is the ability to learn. Higher intel- ductive behaviors. For the most part, she took the
ligence leads to more rapid learning, and the more research-consistent position that they can, citing a
complex the material to be learned, the more this is variety of research psychiatrists and psychologists
true. . . . Why does GMA predict job performance? to support the case.
The primary reason is that people who are more On the other hand, some claims made in the
intelligent learn more job knowledge and learn it article go far beyond scientifically substantiated
faster. . . . Even when workers have equal job knowl- evidence. For example, at one point, Andrews cited
edge the more intelligent workers have higher job a senior vice president of HR as saying, “You can
performance. This is because there are problems pick up a multitude of clues about a person’s char-
that come up on the job that are not covered by
acter by simply having a restaurant meal together.
previous job knowledge, and GMA is used directly
on the job to solve these problems. (Schmidt &
You’ll see how they interact with the waiter or the
Hunter, 2000: 3–5) people sitting at adjacent tables. I sometimes say,
‘Gee, how much of a tip do you think we should
More generally, keyword searches in HBR turned leave?’ Then, based on whatever percentage they
up a number of additional articles that further suggest, I ask why. I want to see how they make
“muddy the construct waters” with respect to in- those decisions. A lot of it bears on how they view
telligence and its relationship to job performance. the world in a more general sense” (2005: 56). This
For example, in the period 2000 – 05, HBR con- type of screening behavior is not supported by re-
tained more articles that covered “emotional intel- search findings. Rather, it is an example of using
ligence” and “social intelligence” (e.g., Coutu, non-job-related criteria that are likely to reflect a
2004; Goffee & Jones, 2005; Goleman, 2000, 2004) hiring manager’s personal predilections more than
than articles that covered “intelligence” or “cogni- a candidate’s ability to do a job. As such, this quote
tive ability,” despite the fact that emotional intelli- represents a selection tactic that is low in validity
gence and social intelligence have far weaker re- and utility but high in exposure to potential legal
search bases in top-tier peer-reviewed psychology liability.
journals and that some definitions of “emotional A second HR Magazine article by Bates (2002) is
intelligence” are so broad as to include nearly all also a mix of research-consistent and questionable
important human traits, including a hefty chunk of claims. For example, in keeping with research evi-
GMA (Murphy, 2006). dence, he wrote that “consensus is building in the
In sum, of the two periodicals that addressed the research community that five factors shape our
usefulness of intelligence in selection, only HRM overall personality” (Bates, 2002: 30). However, the
provided research-consistent information. How- five traits he cited (“need for stability, whether we
ever, HBR provided mixed coverage, with the arti- are solitary or social, whether we strive more for
cle by Menkes (2005) providing research-consistent innovation or efficiency, the degree to which we
information, but articles by Coutu (2004), Goffee stick to our positions or accept others’ ideas, and
and Jones (2005), Goleman (2000, 2004) and Leo- whether we are more linear or flexible in our ap-
nard and Swap (2004) providing either research- proach to goals” [Bates, 2002: 30]) are not entirely
inconsistent or, at best, only partially research- consistent with the Big Five that have generally
consistent information. been used in selection research: emotional stabil-
Personality. The two HRM articles that covered ity, extraversion, openness to experience, agree-
GMA in a research-consistent fashion (O’Leary et ableness, and conscientiousness (Barrick & Mount,
al., 2002; Ryan & Tippins, 2004) also reviewed the 1991; Digman, 1986; McCrae & Costa, 1987). This
research evidence on the validity of various aspects confusion was exacerbated later in the article,
of personality in selection. As such, although one where the five dimensions of a proprietary person-
would not describe HRM’s coverage of these issues ality inventory offered by a consulting firm were
as “extensive,” it is consistent with the best avail- listed as “need for stability, extraversion, original-
able scientific evidence on personality, as was ity, accommodation, and consolidation” (Bates,
HRM’s coverage of GMA. 2002: 31).
Although HR Magazine did not cover GMA at all In addition, Bates stated that “there are no
in the relevant period, it did publish three articles ‘wrong’ answers to personality tests— only results
on personality assessment as a predictor of various that suggest an individual is better-suited to one994 Academy of Management Journal October type of work than another” (2002: 30). Although it criteria [sic] for selection, development, and reten- is true that certain personality traits (such as extra- tion” (2005: 49 –50). From a scientific perspective, version) are more predictive of performance in this suggested use of personality assessments some jobs than others, one of the Big Five traits amounts to “capturing” the current decision model (conscientiousness) has been found to be a positive of a decision maker. Unfortunately, however, it predictor of performance in all job types (Barrick & does not demonstrate that an applicant so assessed Mount, 1991). In addition, scores on three of the can do the job or that the decision maker’s current Big Five factors (conscientiousness, emotional sta- model is a valid one. In fact, related research (on bility, and agreeableness—the factors that domi- employment interviews) suggests that interviews nate most personality-based integrity tests) have are considerably more valid if managers are not been found to be good predictors of counterproduc- allowed to develop “preconceptions” (e.g., by tive behaviors such as fighting, stealing, and absen- viewing résumés) prior to conducting interviews teeism over all job categories (Ones et al., 1993). (e.g., McDaniel, Whetzel, Schmidt, & Maurer, Moreover, when employers use personality-based 1994). As such, this suggested use of personality integrity tests, they are certainly counting some assessments is more likely to entrench idiosyn- answers as “better” than others, regardless of the cratic judgments of hiring managers, providing an job in question. aura of scientific respectability to what are merely The final personality-related article in HR Maga- individual assumptions, predilections, or biases zine (Krell, 2005) describes how personality tests that selection researchers have been arguing against are being used for a variety of purposes other than for years. external hiring. These include individual develop- Overall, then, the few articles appearing in HR ment, team communications, conflict resolution, Magazine on the topic of personality assessment coaching, and placement. Overall, the article con- represent a mixture of research-consistent and re- tains a mixture of research-consistent and -incon- search-inconsistent information. Although they ac- sistent statements, along with some claims that are curately portray the ascendancy of five (give or take difficult to link to any clear research literature. one) dimensions of personality in the research On the research-consistent side, Krell (2005), like realm and the fact that some of the dimensions are Bates (2002), correctly indicated that most experts differentially associated with performance on dif- believe there are between four and six basic per- ferent types of jobs, they do not convey the fact that sonality dimensions and that acceptance of person- conscientiousness is a predictor of performance in ality assessment is growing. He also described one all jobs, or that a combination of conscientiousness, company’s use of concurrent empirical validation emotional stability, and agreeableness is a good of personality measures (a desirable practice if suf- general predictor of counterproductive behaviors ficient sample sizes are available) and referred across occupations. In addition, they make a num- readers to a variety of online support tools for using ber of claims that are inconsistent with existing personality assessments (including SHRM white research findings. The general sense that we were papers on test validation and using personality as- left with after reading these articles was an impres- sessments in selection, and a legal report about the sion that they overpromised as to what personality use of integrity testing). assessments can do, underexplained the differ- On the other hand, the article also discussed a ences between types of personality assessments, number of practices that do not have clear research and overreached in terms of their legitimate foundations and may be problematic. For example, applications. one quoted executive enthused: “The science be- Turning to HBR, we found 12 articles that con- hind cultural fit is extremely important and goes tain personality-related keywords and 4 that con- right to the bottom line” (Krell, 2005: 51). In fact, tain “selection” keywords. However, upon exam- the “science” of cultural fit suggests that although ining the articles, we found that very few of there are clear relationships between cultural fit them make any direct references to the use of and employee satisfaction and retention, results personality variables in selection. Rather, most with respect to job or unit performance are much focus on the management of individuals with par- more open to question (e.g., Janis, 1983; Meglino & ticular (usually “problematic”) personality char- Ravlin, 1998). acteristics (Waldroop & Butler, 2000), or discuss In another place, Krell quoted a consultant who how dysfunctional personality characteristics of argued: “Using personality assessments to confirm CEOs can be better managed or self-managed (e.g., HR professionals’ instincts is a benefit of these Goffee & Jones, 2000; Khurana, 2002; Maccoby, tools. . . . You know you like them. . . . Now you 2000; Tedlow, 2001). Another set of articles focuses can determine exactly why that is and use that on how leaders’ relationships with close confidants
2007 Rynes, Giluk, and Brown 995
(Sulkowicz, 2004), coaches (Berglas, 2002), or fol- matic leadership tends to be defined much more
lowers (Offermann, 2004) can degenerate into psy- broadly and is often equated with transformational
chologically destructive patterns that compromise leadership, especially transformational leader-
a leader’s effectiveness. Additionally, the four arti- ship’s visioning and role modeling dimensions. It
cles about selection in general (Bennis & O’Toole, includes not only having a dynamic, charismatic
2000; Butler & Waldroop, 2004; Sorcher & Brant, style, but also communicating a compelling vision
2002; Wetlaufer, 2000) focus mostly on hiring pro- and serving as a role model of the values of an
cedures (e.g., agreeing on the job description, cre- organization.
ating interview questions, resolving political con- In the academic literature, personalized and so-
flicts) and candidate skills or behaviors rather than cialized charismatic leadership are often distin-
personality traits. guished. Personalized charismatic leaders tend to
However, two of these articles deal at least partly be described as self-centered and sometimes even
with the evaluation of personality traits in a CEO manipulative. They are interested in pursuing their
selection context. In “Don’t Hire the Wrong CEO,” own goals, rather than the goals of a collective:
Bennis and O’Toole (2000: 174 –175) warned
In the personalized relationship, followers are con-
against “candidates who act like CEOs. . . . Boards
fused and disoriented before joining the relation-
often are seduced by articulate, glamorous— dare
ship, and the relationship provides them with a
we say it— charismatic dreamers who send multi- clearer sense of self and greater self confidence. This
ple frissons down their collective spines. . . . In type of relationship is based mainly on followers’
fact, (however), many of the greatest corporate lead- personal identification with the leader, rather than
ers come up short on the charisma scale, because on their identification with or acceptance of the
charisma typically goes hand-in-hand with inflated leader’s message. (Howell & Shamir, 2005: 100)
ego.” Similarly, in “The Curse of the Superstar
CEO,” Khurana also warned about the dangers of In contrast, socialized charismatic leaders work for
charismatic leaders: “When companies look for the good of the collective:
new leaders, the one quality they seek above all In the socialized relationship, followers have a clear
others is charisma. The result, more often than not, sense of self and a clear set of values, and the char-
is disappointment— or even disaster” (2002: 60). In ismatic relationship provides them with a means for
other words, the two HBR articles that deal with expressing their important values within the frame-
personality in leader selection are essentially warn- work of a collective action. Followers in this type of
ings against charismatic leaders. relationship derive their sense of direction and self-
How do these warnings square with academic expression not from personal identification with the
research on charismatic leadership? In one sense, it leader but from the leader’s message. In this rela-
is difficult to make comparisons, because the term tionship followers place constraints on the leader’s
influence, play an active role in determining the
“charismatic leader” seems to be used differently in
values expressed by the leader, are less dependent
the academic and practitioner literatures.10 In the on the leader, and are less open to manipulation by
practitioner literature, “charisma” is a synonym for the leader. (Howell & Shamir, 2005: 100)
charm or mysticism. Indeed, Khurana traced the
word “charisma” to the various “charisms, or gifts This distinction helps to explain why the authors
of the Holy Spirit, that Christians may possess” in HBR see charismatic leadership as generally neg-
(2002: 60). This use of the word suggests that cha- ative, but academic researchers see it as ambiguous,
risma is “style” rather than “substance.” Relatedly, though generally more positive. Overall, there is
Howell and Shamir wrote, “Theories of charismatic “accumulating evidence that demonstrates both the
leadership have been accused of promoting a ‘he- positive and negative outcomes of charismatic
roic leadership’ stereotype (Beyer, 1999; Yukl, leadership” (Howell & Shamir, 2005: 97). However,
1998), which depicts leaders as heroic figures that more of the academic evidence falls on the positive
are single-handedly capable of determining the fate side (Judge & Piccolo, 2004)—a result that is prob-
and fortunes of groups and organizations. In this ably due in part to the fact that academics tend to
heroic conception, the leader is omnipotent, and measure charismatic leadership in a way that is
followers are submissive to the leader’s will and consistent with socialized charismatic leadership.
demands” (2005: 96). In contrast, Khurana (2002) and Bennis and
In contrast, in the academic literature, charis- O’Toole (2000) seem to be describing personalized
charismatic leadership, or the “dark side” of char-
ismatic leadership.
10
We thank Amy Colbert for help in interpreting the Viewed from the vantage point of EBM, it is
academic and practitioner literatures on leadership. significant that the HBR articles on personality—996 Academy of Management Journal October
including those that deal with management and In an effort to be hands-off and not become a much-
self-management, as well as selection—make no maligned “micro-manager,” supervisors have gone
mention whatsoever of the huge scientific discov- to the opposite extreme and completely abdicated
their primary role as managers. . . . Under-manage-
ery of the robust Big Five personality factors. This
ment is the overwhelming common denominator in
omission is particularly striking in that the discov-
most cases of suboptimal workplace performance at
ery of the Big Five goes back more than 20 years all levels. The under-managed worker struggles be-
(e.g., Digman, 1986; McCrae & Costa, 1987). As cause his supervisor is not sufficiently engaged to
such, none of the excitement that academics have provide the direction and support he needs and,
felt about being able to consolidate research evi- therefore, is unable to help with resources and prob-
dence on these “meta” factors of personality (as lem-solving. The manager cannot judge what expec-
opposed to having to deal with dozens or even tations are reasonable, and he cannot set goals and
hundreds of narrower personality traits) has deadlines that are ambitious but still meaningful.
reached HBR’s audience. Similarly, the academic (2004: 119)
distinction between “personal” and “socialized”
charismatic leadership seems to have passed HBR’s In short, Tulgan provided a good explanation of the
notice, despite the fact that it might help to clear up strong research finding that goal setting with feed-
back is a far more effective motivator of perfor-
some of the conflicting views about the merits and
mance, on average, than is empowerment (Latham,
shortfalls of charismatic leaders.
2006; Locke et al., 1980; Rynes et al., 2002).
Overall, in all three journals, the amount of at-
In the final article in HR Magazine that we exam-
tention paid to personality is not very great, and
ined, Carrison (2003) focuses on a particular form
certainly smaller than the amount of attention de-
of goal setting: setting deadlines. He describes com-
voted to it in academic journals. However, its cov-
monalities in management practices over three
erage in HRM tends to be research-consistent, while
large construction projects that all managed to meet
coverage in HR Magazine represents a mixture of
ambitious scheduling goals. These commonalities
research-consistent (e.g., the Big Five and the po-
included giving the goals a great deal of publicity,
tential for using personality in selection is clearly
stressing the schedule at all points in the process,
there) and non-research-consistent reporting (e.g.,
holding emergency meetings at the first signs of
occasional recommending of nonbehavioral inter-
slippage, holding all managers accountable to each
view questions or questions that have nothing to do other, getting managers’ input on and commitment
with the predictive dimensions of the Big Five). to the schedule, and celebrating on-time milestones
Finally, treatment of personality in HBR seems to along the way. All these principles are consistent
be completely divorced from academic research on with the results of goal setting research (Latham,
personality, with no mention of the Big Five, con- 2006). In short, when HR Magazine did report on
tinued discussion of narrow rather than broad per- goal setting as the central topic of interest, it tended
sonality traits, and no research-based summary of to do so in a research-consistent fashion.
generalizable personality-performance relationships. HBR published six articles that deal at least
Goal setting. Turning next to goal setting, we partly with goals or goal setting. Once again, how-
found that less than 1 percent of the articles in HR ever, some of the articles are tangential to the issues
Magazine focus on the usefulness of goal setting for covered by the well-documented body of goal set-
improving performance. Of the five articles in HR ting research. For example, one article deals with
Magazine that mentions goal setting, three of them assessments of individual motivations and compet-
mentions it rather incidentally (i.e., as part of a ing commitments (Kegan & Lahey, 2001), and an-
variable pay system in Frase-Blunt [2001] and other discusses ways to reframe goals to tap into
Garvey [2000], or as an available feature in an on- individual differences in motivation (Nicholson,
line performance management system in Robb 2003).
[2004]). However, three articles discuss principles of
However, in “The Under-management Epi- goal setting that map onto academic research.
demic,” Tulgan (2004) hit the basic findings from These articles focus mostly on the principles of
goal setting research right on the head. Specifically, frequent feedback with respect to progress toward
he discussed the value of specific, challenging, and goals, as well as the importance of goal acceptance.
meaningful goals; accurate monitoring and docu- For example, in “Management by Whose Objec-
mentation of progress toward goals, and specific tives?,” Levinson argued that one of the reasons for
feedback on performance with guidance for im- the failure of “Management by Objectives” is that
provement (Latham, 2006). Tulgan went on to say: “unit managers are forced to commit to goals they2007 Rynes, Giluk, and Brown 997
don’t believe are realistic” (2003: 107).11 Relatedly, six in HBR: three tangential and three research-
Parcells (2000) focused on the importance of setting consistent, although their coverage was partial.
goals that permit “small wins,” rather than an “ul-
timate” goal that seems unattainable. In the third
Research Question 3: Sources of Evidence
article, “Turning Great Strategy into Great Perfor-
mance,” Mankins and Steele (2005) emphasized The preceding analyses suggest little correspon-
the importance of communicating strategic goals in dence between what is being published in aca-
simple, concrete language and of clearly identify- demic versus practitioner and bridge journals with
ing priorities. All these recommendations are con- respect to the three most important findings of HR
sistent with goal setting research, although no ref- research (as perceived by researchers). Areas con-
erence is made to this research, and pieces of the sidered to be very important by researchers receive
relevant goal setting findings are not emphasized little coverage in practitioner and bridge journals
(particularly, the importance of setting difficult but and, when they do receive coverage, it is as likely to
attainable goals). be research-inconsistent as research-consistent, ex-
Finally, HRM published one highly research-con- cept in HRM.
sistent article on goal setting (London et al., 2004). This situation makes the question of who, or
This article was part of the same special issue on what, is cited as evidence in practitioner and bridge
applications of psychological research to HR man- journals an interesting one. Thus, we examined all
agement that was mentioned in previous sections. 152 articles that dealt with selection/recruitment
In summary, coverage of goal setting in the se- (n ⫽ 141) or goal setting (n ⫽ 11) to examine what
lected practitioner and bridge publications was sources of evidence each periodical used. These
quite scarce, particularly when the large effect sizes analyses provide some indication as to what
found in goal setting research are taken into ac- sources of information are viewed as most legiti-
count. Moreover, approximately half of the articles mate or credible at each periodical.
that did mention goal setting did so only peripher- HRM. We tallied the evidentiary bases of the
ally. Of the very small number of all articles that three journals in different ways, because the con-
dealt more than incidentally with goal setting, tent and format of each periodical differ. Of the
however, the coverage was largely research-consis- three, HRM most closely resembles top-tier aca-
tent (particularly with respect to the importance of demic HR journals such as the Journal of Applied
goal acceptance). Psychology and Personnel Psychology. For exam-
Summary. Our analysis of Research Question 2 ple, like articles in academic journals, HRM articles
suggests that with respect to the importance of in- tend to cite a fair number of peer-reviewed research
telligence or GMA to job performance, there has articles as sources of evidence (36.7 citations on
been only sporadic (but accurate) transfer of re- average, with a standard deviation of 20.2). In ad-
search findings to HRM, limited but mostly re- dition, journals receiving the most citations in
search-inconsistent transfer to HBR, and no transfer HRM are research- rather than practice-oriented.
to HR Magazine. With respect to personality, the Specifically, the top five journals cited in HRM
results for HRM mirror those with respect to intel- over this period were all peer-reviewed ones: the
ligence—very limited, but research-consistent, cov- Journal of Applied Psychology (9.8% of all cita-
erage. In the case of HR Magazine, coverage is also tions), Personnel Psychology (6.4%), the Academy
at a very low level (⬍ 1%), and claims are a mix of Management Journal (5.6%), HRM (4.4%), and
of research-consistent and research-inconsistent. the Academy of Management Review (2.5%). In
However, on the positive side, HR Magazine is at contrast, HBR and HR Magazine (neither of which
least transmitting information about there being is peer-reviewed) each accounted for only 1.1 per-
five (or so) basic personality characteristics, which cent of HRM’s total citations.
cannot be said of HBR. In fact, HBR mentioned Another similarity to top-tier journals is that
neither the discovery of the Big Five personality most HRM articles are either original research or
traits, nor the academic literature on charismatic or literature reviews. For example, of the 21 recruit-
transformational leadership. With respect to goal ment, selection, and goal setting articles found be-
setting, we found one relevant and research-consis- tween 2000 and 2006, 7 reported the results of
tent article in HRM; five articles in HR Magazine, of survey research (either questionnaire- or interview-
which three provided only peripheral coverage and based), 6 were based on either single- or multiple-
two provided research-consistent information; and organization case studies, 5 presented literature re-
views, 2 reported the results of experiments, and 3
presented typologies or “best practices” based on
11
HBR originally published this article in 1970. either cases or qualitative analyses.You can also read
