Bibliometric analysis of engineering education research publications - A reference discipline approach
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Bibliometric analysis of engineering education research
publications - A reference discipline approach
P Neto1
Lecturer
Escola Superior Tecnologia Barreiro/ Setubal Polytechnic Institute
Barreiro, Portugal
pedro.neto@estbarreiro.ips.pt
B Williams
Lecturer
Escola Superior Tecnologia Barreiro/ Setubal Polytechnic Institute
Barreiro, Portugal
bill.williams@estbarreiro.ips.pt
Conference Topic: Engineering Education Research
Keywords: Discipline, Engineering Education Research, Bibliometric Analysis
INTRODUCTION
In recent years, bibliometric analysis of publications has been receiving growing attention in
engineering education research as an approach that can bring a number of benefits. In this
paper, we use reference discipline and author affiliation analysis of journal articles to provide
data on the development of Engineering Education Research (EER) as a scholarly research
field [1] and the diffusion of engineering education innovations [2]. In this study we analyze a
total of 139 papers published in 6 engineering education journals in 2011 with regard to
reference discipline and the geographical location of the authors published and we discuss the
results. Analysis of reference disciplines has been used by scholars in the new disciplines of
Information Systems and Enterprise Engineering research for the past two decades and
involves a study of the disciplines referenced and cited in research papers to track the
developing maturity of these fields of research. We propose that a similar approach can be
useful in EER as an indicator of interdisciplinarity and a form of measuring the developing
maturity of EER as a field and cross-fertilization between EER and other research fields. In
addition, when this is combined with data on the affiliation of authors we can also obtain data
on cross-fertilization within the field of EER beyond national boundaries.
1 REFERENCE DISCIPLINE
In this paper we adopt a concept which we believe can serve as a useful boundary object
linking the EER community with researchers in the fields of information science and
enterprise engineering. The term reference discipline [3], has been an important component of
1
Corresponding Author
P Neto, pedro.neto@estbarreiro.ips.ptthe vocabulary and conduct of Information Science (IS) scholars over the past two decades.
Early empirical studies such as [4] placed the IS discipline at the end of an intellectual food
chain based on the results of a citation analysis. More recently, some scholars (e.g., [5], [6])
suggest that perhaps this research field has matured to an extent where it might be serving in
the role of reference discipline, while others (e.g., [7], [8]; [9]), rejecting the possibility of a
two-way interaction, maintain that IS has yet to reach a stage where it can serve in that role.
Likewise scholars of the new discipline of Enterprise Engineering have also made extensive
use of this concept while attempting to track the development of their research area [10].
Fig. 1. Evolution of an Emergent Discipline [15]
The term has come to refer to those disciplines X that provide foundational, methodological,
or other inputs to another discipline/s Y such that the state of knowledge in Y is advanced
through inputs provided by X. In other words if Y cites X in order to develop and advance the
state of its knowledge, X becomes a reference discipline for Y. Despite agreement at this
basic level, the term reference discipline remains an imprecise concept [11]. For instance,
some IS scholars (see [8], or [9]) use a hard definition whereby a discipline is considered as a
reference discipline when providing extensive input to other disciplines; therefore there is
presence of citations in any randomly selected piece of work in the other discipline. This
implies an expectation that any work in a discipline should (ideally) cite its reference
discipline. A softer, less restrictive view of a reference discipline (see [6], [4]; [12]) describes
it as simply one that contributes to another discipline. For our purposes which involve
tracking interdisciplinarity we opted to use the latter soft definition. Disciplines build
competent knowledge claims by relying on both intra-disciplinary as well as inter-disciplinary
work. In general, as disciplines mature and start to form a distinct identity of their own, they
not only depend less on other disciplines for supporting their work, but also tend to
increasingly inform work in other fields. Miner in his critique of the organization sciences,
[13], argued that the progress towards maturity of an emergent discipline can be seen by how
much it depends on its own sources and how much it contributes to other disciplines.
Similarly, economist Joseph Schumpeter has stressed the notion that any field of inquiry that
justifiably earns the distinction of being called a discipline should provide the world with
theories and paradigms and draw more upon work within the discipline [14]. Fig. 1, from the
work of Grover on the development of IS as a discipline [15], illustrates how an emergent
discipline evolves from being initially very much dependent upon concepts, models and
frameworks developed in existing disciplines, but over time accumulates those of its own and
may eventually come to serve as a reference discipline for others.
It is probably early days to consider EER as a reference discipline although Johri and Olds in
a recent review paper propose that we can already identify “many areas of mutual benefit for
engineering education and the learning sciences and many potential areas of collaborativeresearch that can contribute not only to engineering learning but to the learning sciences”
[16].
2 OBJECTIVES
EER is gradually gathering recognition as an emergent field [17], [18]. By means of a
bibliometric analysis of publications, this work aims to provide data which can help scholars
track cross-fertilization between discipline areas [2], across international borders [1] and
which may also provide indicators of the maturity of EER as a field of research [19],[20].
3 METHOD
A total number of 139 articles published in 6 engineering education journals in 2011 were
analysed with regard to reference discipline and the country affiliation of the authors. The
journals and the corresponding number of articles can be seen in Table 1. The journals were
selected to obtain a mix between US based (JEE and AEE) and non-US general engineering
education journals AJEE and IJEE), and also to include two disciplinary engineering
education journals, CEE and IEEE Trans Educ. Knowing that similar work was already
underway by other researchers for the European Journal of Education Research, we opted not
to include it in our sample.
For the research discipline analysis, each of the journal articles was classified independently
by both authors and the results later discussed until the classifications were consensual. To
satisfy our classifications system, a discipline needed to be specifically shown by the authors
to underpin their research. Such references were normally found in the background or
methodology sections of the papers and were frequently mentioned as keywords. Any
disciplines which only appeared in the bibliographic references section were not considered
reference disciplines.
Table 1. Publications and Number of articles analysed for 2011
Publications Number of articles
Advances in Engineering Education (AEE) 22
Australasian Journal of Economics Education (AJEE) 21
Chemical Engineering Education (CEE) 30
IEEE Transactions on Education (IEEE Trans Educ) 21
International Journal of Engineering Education (IJEE) 20
Journal of Engineering Education (JEE) 25
4 RESULTS
4.1 Reference Discipline
The reference disciplines identified in the journals are shown in Table 2. It is possible to
identify 47 reference disciplines in the 139 Engineering Education articles analysed. We note
that the two disciplinary journals had very few examples of reference disciplines, three of the
general EER journals had broadly similar values corresponding to around 25% of papers
analysed including reference disciplines while the JEE stands out as having almost one per
paper on average (Fig. 2). Table 3 shows the number of articles citing a specific Reference
Discipline.Table 2. Reference Disciplines found in individual articles
Journal Reference Disciplines/article
Journalism
Teacher Training
Psychology
AEE
Organizational Change (sustainability)
Community Studies
Medicine
Learning Science (metacognitive skills)
Sociology (migration patterns & under-employment of migrant engineers)
Education Policy (internationalization)
AJEE
Learning Science (learning cultures)
Communication Studies (discourse analysis)
Learning Theory (competence theory - DeSeCo)
CEE Phenomenology
Organizational Studies (Porter's value chain)
Scientometrics
IJEE Learning Theory (Bloom's Taxonomy, learning models)
Learning Theory (context awareness)
Game Theory (modelling)
Learning Theory - Dewey
Gender Studies; Psychology (self-efficacy)
Learning Theory (constructivism)
Communication Studies (discourse analysis)
Medicine (survival analysis)
Sociology (identity theory); Organization Studies (self-managed work teams)
Psychology (information fluency); Psychology (metacognitive reading)
Organization Studies (complex adaptive systems)
Psychology (mathematics anxiety)
Learning Theory (collaboration; constructivism); Psychology (self-efficacy)
JEE Psychology (cognitive psychology)
Psychology (cognition development)
Psychology (self-efficacy)
Sociology (identity theory)
Learning Theory (social cognitive theory); Communication Studies (discourse analysis);
Psychology (self-efficacy)
Philosophy (ethics)
Learning Theory (competency formation)
Psychology and Sociology (social integration)
Psychology (technological creativity studies)
Psychology (psychometrics)
JEE
IJEE
IEEE
Journals
CEE
AJEE
AEE
0.00 0.25 0.50 0.75 1.00
N. of Ref. Disc./ N. of articles
Fig. 2. Number of Reference Disciplines per number of articlesTable 3. Number of articles citing a specific Reference Discipline
Reference N. of articles Reference N. of articles Reference N. of articles
Disciplines citing Disciplines citing Disciplines citing
Communication
3 Learning Science 2 Philosophy 1
Studies
Community Studies 1 Learning Theory 8 Psychology 15
Education Policy 1 Medicine 2 Scientometrics 1
Game Theory
1 Organization Studies 3 Sociology 4
Modelling
Organizational
Gender Studies 1 1 Teacher Training 1
Change
Journalism 1 Phenomenology 1
4.2 Affiliation analysis
The geographical location of the authors published is summarized in Table 4. The number of
countries found in the analysed journals is 27. From a total of 231 authors, 55% are from US,
16% from Australia, 11% from Spain and 18% from other countries. Fig. 3 to 8 show the
distribution of the author’s contribution in terms of the geographical location. Comparing the
results on those figures can lead to the conclusion that for the year 2011 AEE, AJEE and JEE
were mainly local in terms of geographical location of the authors whereas CEE, IEEE Tran
Educ and IJEE tend to have a broader range of participation.
Australia
Australia
New
Turkey
Zealand
USA
USA
Fig. 3. Author affiliations in AEE Fig. 4. Author affiliations in AJEE
Australia
Austria Canada
Canada China
Cyprus
India Finland
Ireland Greece
Portugal India
Iran
Puerto Rico Japan
South Africa Malaysia
Spain Slovenia
Spain
USA UK
USA
Fig. 5. Author affiliations in CEE Fig. 6. Author affiliations in IEEE Trans EducAustralia
Canada
Australia
China
Colombia Finland
Mexico Korea
Serbia Singapore
Spain
USA
Taiwan
USA
Fig. 7. Author affiliations in IJEE Fig. 8. Author affiliations in JEE
Table 4. Geographical affiliation of published authors
Number of articles in each journal
Affiliation IEEE Trans Total
AEE AJEE CEE IJEE JEE
Educ
Australia 8 52 5 3 1 69
Austria 1 1
Canada 1 1 4 6
China 5 4 9
Colombia 2 2
Cyprus 1 1
Finland 3 1 4
Greece 6 6
India 4 4 8
Iran 3 3
Ireland 1 1
Japan 1 1
Korea 1 1
Malaysia 3 3
Mexico 3 3
New Zealand 1 1
Portugal 4 4
Puerto Rico 1 1
Serbia 5 5
Singapore 1 1
Slovenia 5 5
South Africa 2 2
Spain 2 21 24 47
Taiwan 2 2
Turkey 1 1
UK 4 4
USA 86 3 72 17 14 39 2315 DISCUSSION
In the case of IEEE Transactions on Education and the Journal of Chemical Engineering
Education, our results shows little evidence of reference disciplines which suggests that the
research of the authors of the papers published in these discipline-based journals tends to be
informed by concepts within rather narrow disciplinary confines. By contrast, the results for
the general engineering education journals, particularly JEE, show that the authors appear to
be reading more broadly and their work is informed by both engineering education research
and other reference disciplines. This could be interpreted as a sign of interdisciplinarity and
developing maturity of EER as a field of research.
On the other hand, from an internationalization perspective, the limited number of countries
involved in publishing research in the journals studied, seems to point to a lack of cross-
dissemination of EER findings and suggests there may be barriers limiting the diffusion of
proven educational innovation. Although the work presented, 139 articles from 6 EER
journals in 2011, represents a relatively small sample of the overall volume of EER
publications, we believe the study demonstrates the value of a bibliometric analysis approach
to EER. It allows to study the developing maturity of the field, identify possible barriers to
cross-fertilization and to help authors choose suitable publication channels.
REFERENCES
[1] Jesiek, BK, Borrego M, Beddoes K, Hurtado M, Rajendran P and Sangam, D (2011),
Mapping Global Trends in Engineering Education Research, 2005–2008, Int. J. Eng.
Educ., Vol. 27, No. 1., pp. 77-90.
[2] Borrego, M., Froyd, J. E., & Hall, T. S., (2010). Diffusion of Engineering Education
Innovations: A Survey of Awareness and Adoption Rates in U.S. Engineering
Departments. Journal of Engineering Education, 99 (3), 185-207.
[3] Keen, PGW (1980), MIS Research: Reference Disciplines and a Cumulative
Tradition, Proc. Of the First International Conference on Information Systems,
Philadelphia, PA, 1980, pp. 9-18.
[4] Culnan, MJ and Swanson EB (1986), Research in Management Information Systems,
1980-1984: Points of Work and Reference, MIS Quarterly, Vol. 10, No. 3 pp. 289-
302.
[5] Baskerville, R and Myers, M (2002), Information Systems as a Reference Discipline,
MIS Quarterly, Vol. 26, No. 1 pp. 1-14.
[6] Grover, V, Gokhale, R, Lim, J. and Ayyagari, R (2006), About Reference Disciplines
and Reference Differences: A Critique of Wade et al., Journal of the Association for
Information Systems, Vol. 7, No. 5, pp. 336-350.
[7] Benbasat, I and Weber, R (1996), Research commentary: Rethinking diversity in
information systems research, Information Systems Research, Vol. 7, No. 4, pp 389-
399.
[8] Orlikowski, W and Barley, S (2001), Technology and institutions: What can research
on information technology and research on organizations learn from each other? MIS
Quarterly, Vol. 25, No. 2, 145-165.[9] Wade, M, Biehl, M and Kim, H (2006), Information Systems Is NOT A Reference
Discipline (And What We Can Do About It), Journal of the Association for
Information Systems, Vol. 7 No. 5, pp. 247-269.
[10] Liles, DH, Johnson, ME et al. (1996), The Enterprise Engineering Discipline, Proc.
of the Fifth Annual Industrial Engineering Research Conference, Minneapolis, MN.
[11] Westin, SM, Roy, M and Kim, CK (1994), Cross-fertilizations of knowledge: the
case of MIS and its reference disciplines, Information Resources Management
Journal, Vol. 7, No. 2, pp. 24-34.
[12] Cheon, M. J., C. Lee, and V. Grover (1992) "Research in MIS - Points of Work and
Reference: A Replication and Extension of the Culnan and Swanson Study,"
ACM SIGMIS Database (23) 2, pp. 21-29.
[13] Miner, J. B. (1984) "The Validity and Usefulness of Theories in an Emerging
Organizational Science," The Academy of Management Review (9) 2, pp. 296-
306iner (1984)
[14] Schumpeter, JA and Opie, R (1934), The theory of economic development; an
inquiry into profits, capital, credit, interest, and the business cycle, Cambridge, Mass,
Harvard University Press.
[15] Grover, V, Gokhale, R, Lim, J, Coffey, J, and Ayyagari, R (2006), A Citation
Analysis of the Evolution and State of Information Systems within a Constellation of
Reference Disciplines, Journal of the Association for Information Systems, Vol. 7,
No. 5, pp. 270-325.
[16] Johri, A and Olds, B (2011), Situated Engineering Learning: Bridging Engineering
Education Research and the Learning Sciences, Journal of Engineering Education,
Vol. 100, No. 1, pp. 151-185.
[17] NEERC (National Engineering Education Research Colloquies) Special Report
(2006), The Research Agenda for the New Discipline of Engineering Education, J.
Eng. Educ., Vol. 94, No. 4, pp. 257-261.
[18] Streveler, R and Smith K (2010), Guest Editorial: From the Margins to the
Mainstream: The Emerging Landscape of Engineering Education Research, J. Eng.
Educ., Vol. 99, No. 4, pp. 285-287.
[19] Borrego, M (2007), Development of Engineering Education as a Rigorous Discipline:
A Study of the Publication Patterns of Four Coalitions, J. Engr. Educ., Vol. 96 No. 1,
pp. 5-18.
[20] Fensham, P (2004), The Evolution of Science Education as a Field of Research:
Defining an Identity, New York, NY: Springer Publishing Company.You can also read
