Human Factors and Organizational Issues Section Synopsis IMIA Yearbook 2021
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Published online: 2021-09-03
100
© 2021 IMIA and Georg Thieme Verlag KG
Human Factors and Organizational Issues
Section Synopsis IMIA Yearbook 2021
Yalini Senathirajah1, Michelle Hribar2, Section Editors of the IMIA Yearbook Section on
Human Factors and Organizational Issues
1
U. Pittsburgh School of Medicine, Dept. of Biomedical Informatics, Pittsburgh, PA, USA
2
Oregon Health & Science University, Portland, OR, USA
Summary 1 Introduction in each co-editor creating a set of 15 can-
didates each, which were then discussed.
Objective: To select the best papers that made original and high
impact contributions in the area of human factors and organiza-
Papers published in 2020 on human factors From this search we selected 13 best papers;
tional issues in biomedical informatics in 2020.
and organizational issues (HFOI) focused on these were sent for additional peer review
Methods: A rigorous extraction process based on queries from
user healthcare technology and data needs, by a set of external reviewers and two chief
Web of Science® and PubMed/Medline was conducted to iden-
for both patients and physicians. In addition, editors, with each paper being reviewed by
tify the scientific contributions published in 2020 that address
several papers focused on the downstream at least four people. We categorized the best
human factors and organizational issues in biomedical informat-
impact of Electronic Health Record (EHR) papers according to four themes: 1) data
ics. The screening of papers on titles and abstracts independently
use and usability on provider burnout. Not presentation for accuracy and understanding;
by the two section editors led to a total of 1,562 papers. These
surprisingly, there were also many papers 2) EHR usability and clinician burnout; 3)
papers were discussed for a selection of 12 finalist papers, which
about designing and implementing new clinical system design; and 4) timely topics.
were then reviewed by the two section editors, two chief editors,
systems for Coronavirus Disease 2019 We then selected the best four papers using
and by three external reviewers from internationally renowned
(COVID-19), as well as on design consider- these reviews, while also ensuring the papers
research teams.
ations for achieving health equity, both very represented a variety of themes and had
Results: The query process resulted in 12 papers that reveal
timely topics in 2020. Finally, there were strong methods and results.
interesting and rigorous methods and important studies in
other papers that focused on design factors
human factors that move the field forward, particularly in clinical
for healthcare systems. Together these papers
informatics and emerging technologies such as brain-computer
provide a small cross section of the best HFOI
interfaces. This year three papers were clearly outstanding and
research in biomedical informatics in 2020. 3 Results
help advance in the field. They provide examples of applying
Table 1 lists the top four papers, in order of
existing frameworks together in novel and highly illuminating
the first author’s surname.
ways, showing the value of theory development in human factors.
Emerging themes included several which discussed physician 2 Methods Adler-Milstein et al. [1] combined com-
mon objective measures of EHR use with
burnout, mobile health, and health equity. Those concerning the
We collected papers from 2020 by a standard Maslach Burnout inventory survey responses
Corona Virus Disease 2019 (Covid-19) were included as part of
search in PubMed and Web of Science®. for 87 clinicians in primary care clinics of
that section.
Searches were performed in January 2020 a large, urban, academic medical center to
Conclusion: The selected papers make important contributions to
to identify peer-reviewed journal articles determine how EHR use might be affecting
human factors and organizational issues, expanding and deep-
published in the English language in 2020, burnout. They found that 34% of clinicians
ening our knowledge of how to apply theory and applications of
and related to HFOI research in biomedical had high cynicism and 51% had high emo-
new technologies in health.
informatics. In addition to the search of tional exhaustion. Clinicians who had the
electronic databases, manual searches of key highest measures of EHR time after hours
Keywords themes were performed in major biomedi- had much higher odds of high exhaustion;
Human factors, human-computer interaction, organizational cal journals (e.g., Journal of the American this trend also held for clinicians with the
issues, electronic health records, mobile health Medical Informatics Association, Methods most message volume. No trends were ob-
of Information in Medicine, Journal of Med- served for cynicism.
Yearb Med Inform 2021:100-4 ical Internet Research, …etc.). Independent In their paper, Brewer et al. [2] present
http://dx.doi.org/10.1055/s-0041-1726524 searches by the two co-editors resulted in contextually tailored case examples of stud-
an initial set of 1,562 papers. Independent ies and projects which aim to address health
reading of the titles and abstracts resulted inequities by involving the community, i.e.,
IMIA Yearbook of Medical Informatics 2021101
Human Factors and Organizational Issues Section Synopsis IMIA Yearbook 2021
Table 1 Best paper selection of articles for the IMIA Yearbook of Medical Informatics 2021 in the section 'Human Factors and Organizational paper focused on presenting results from
Issues'. The articles are listed in alphabetical order of the first author’s surname. AI models for diagnosing skin lesions. In
addition, Stonbraker et al. [5] designed a
Section
thorough study of data presentations of
Human Factors and Organizational Issues longitudinal patient reported outcomes to
determine which ones promoted the best
Adler-Milstein J, Zhao W, Willard-Grace R, Knox M, Grumbach K. Electronic health records and burnout: Time spent on the comprehension. They found that patients
electronic health record after hours and message volume associated with exhaustion but not with cynicism among primary care
interpreted the visualizations very literally
clinicians. J Am Med Inform Assoc 2020;27(4):531-8.
Brewer LC, Fortuna KL, Jones C, Walker R, Hayes SN, Patten CA, Cooper LA. Back to the future: Achieving health equity and that while they preferred bar charts with
through health informatics and digital health. JMIR Mhealth Uhealth 2020;8(1):e14512. emojis, they could confuse the meaning of
Reading Turchioe M, Grossman LV, Myers AC, Baik D, Goyal P, Masterson Creber RM. Visual analogies, not graphs, increase the emojis if they were not precisely tailored
patients’ comprehension of changes in their health status. J Am Med Inform Assoc 2020;27(5):677-89. to the symptom or outcome they were rep-
Tschandl P, Rinner C, Apalla Z, Argenziano G, Codella N, Halpern A, Janda M, Lallas A, Longo C, Josep Malvehy J, Paoli resenting.
J, Puig S, Rosendahl C, Soyer HP, Zalaudek I, Kittler H. Human–computer collaboration for skin cancer recognition. Nat
Med 2020;26(8):1229–34.
3.2 EHR Usability and Clinician
Burnout
co-design approaches. It strategically merged example of a continued emerging emphasis The Adler-Milstein paper et al. paper [1]
health services research with communi- on visualization for patients, including those described above fell into this category. An-
ty-based participatory research (CBPR) for with lower literacy, and cautioned against other excellent paper by Melnick et al. [6]
innovation and development. The primary over-reliance on graphs. performed a secondary analysis of a survey
contribution of this paper is a set of resilience Finally, Tschandl et al. [4] tested a variety of US physicians from all specialties to de-
strategies that could be supported by innova- of representations of artificial intelligence termine the association between physicians’
tions in health information technology in fu- (AI) for rating skin lesions for providing perceived EHR usability and workload and
ture research. The paper addresses the serious decision support to physicians and measured their professional burnout. They found a
problem of health equity and calls attention to the resulting accuracy and susceptibility to strong relationship between EHR usability
the possibility that the new massive activity randomly introduced errors. They found that and workload, with more favorable usability
in health informatics including digital health when they provided the probabilities for the associated with a lower task load.
may have an unintended consequences: to full range of diagnoses, the decision support Another innovative paper by Weir et al.
worsen health inequities and create new forms improved the physician’s diagnostic accuracy [7] was a position paper that explored how
of inequity. Thus, designers and informati- by 13% but that the decision support for de- burnout is affected by three aspects of cogni-
cians must guard against these consequences termination of malignancy did not improve tion and motivation: 1) goal-based decision
in their ongoing work. accuracy. They also found that physicians making; 2) drive for sense-making and
Patient-reported outcomes (PROs) in- were susceptible to accepting randomly meaning; and 3) need for agency, autonomy,
creasingly use self-monitor health status incorrect predictions once they had used the and control. Through a thorough discussion
and visualization over time would help AI based decision support tools. of prior research, the paper hoped to spur
patients interpret their data. The study of Although not selected as best papers, the research that applies concepts of cognitive
Reading Turchioe et al. [3] included 40 remaining candidate best papers were just science to understanding and mitigating
hospitalized patients and compared four as excellent and influential. We summarize physician burnout.
visualization conditions: (1) text-only; them below, organized by the four major
(2) text plus visual analogy; (3) text plus themes for the papers: 1) data presentation
number line; and (4) text plus line graph for accuracy and understanding; 2) EHR
assessed objective comprehension using usability and clinician burnout; 3) clinical 3.3 Clinical Systems Design
the International Organization for Stan- system design; and 4) timely topics. This theme had the broadest topics of papers,
dardization protocol. Secondary outcomes from user and patient designed systems, to the
included response times, preferences, risk negative impact of a medication ordering sys-
perceptions, and behavioral intentions. As tem, to the state of application programming
a result, 63% correctly comprehended the 3.1 Data Presentation for interfaces in healthcare. A paper by Farao
text-only condition and 60% comprehended Accuracy and Understanding et al. [8] described a user-centered design
the line graph condition, compared with Two of the four selected best papers fell into framework for mobile health applications
83% for the visual analogy and 70% for the this category; the Reading-Turchioe et al and demonstrated its use for an app that
number line (P< 0.05) conditions. This was paper [3] focused on presenting healthcare read the tuberculin skin test. Combining the
the top voted paper and provided the best data to patients while the Tschandl et al. [4] information systems research framework with
IMIA Yearbook of Medical Informatics 2021102
Senatirajah et al.
design thinking, the new framework allowed ed contact tracing application. They found 4. Tschandl P, Rinner C, Apalla Z, Argenziano G,
for iterative refinement of designs through widespread support for such an application Codella N, Halpern A, et al. Human–computer
collaboration for skin cancer recognition. Nat Med
interactions with users. A paper by Ng et al. and concluded it could be a viable tool for 2020 Aug;26(8):1229–34.
[9] described patient designed enhancements limiting spread of the virus. 5. Stonbraker S, Porras T, Schnall R. Patient prefer-
to a continuous glucose monitoring system ences for visualization of longitudinal patient-re-
for Type I diabetes. These “do-it-yourself ” ported outcomes data. J Am Med Inform Assoc
innovations formed the Nightscout Project, 2020 Feb 1;27(2):212–24.
6. Melnick ER, Harry E, Sinsky CA, Dyrbye LN,
a continually expanding set of open-source
technologies driven by a community of Type
4 Conclusion Wang H, Trockel MT, et al. Perceived Electronic
Health Record Usability as a Predictor of Task
I diabetic patients, their parents, and their The HFOI papers this year reflected import- Load and Burnout Among US Physicians: Me-
caregivers. Next, a paper by MacKenna et ant progress and diversification of themes. diation Analysis. J Med Internet Res 2020 Dec
al. [10] described how a poorly designed The pandemic and issues it has raised formed 22;22(12):e23382.
an important set of advances, but also 7. Weir CR, Taber P, Taft T, Reese TJ, Jones B, Del
medication ordering system resulted in 9.5 Fiol G. Feeling and thinking: can theories of hu-
million excess cost in one year for ordering expanded existing topics such as clinician man motivation explain how EHR design impacts
more expensive generic medication than burnout and the importance of usability in clinician burnout? J Am Med Inform Assoc 2021
was necessary. Finally, a paper by Dullabh patient-facing interventions. The emergence Apr 23;28(5):1042–6.
et al. [11] used a socio-technical model to of equity and usability as a serious concern is 8. Farao J, Malila B, Conrad N, Mutsvangwa T,
overdue, and all these advances are welcome Rangaka MX, Douglas TS. A user-centred design
structure an analysis of the current state framework for mHealth. PLOS One 2020 Aug
of application programming interfaces in expansions in the field. 19;15(8):e0237910.
healthcare for providing necessary interop- 9. Ng M, Borst E, Garrity A, Hirschfeld E, Lee J.
erability and data sharing. Not surprisingly, Acknowledgements Evolution of Do-It-Yourself Remote Monitoring
they found a large amount of potential use We would like to acknowledge the support Technology for Type 1 Diabetes. J Diabetes Sci
of Adrien Ugon, Martina Hutter, Kate Fultz Technol 2020 Sep;14(5):854–9.
cases and existing applications, most of which 10. MacKenna B, Curtis HJ, Walker AJ, Bacon S,
focused on filling a physician’s need, read ca- Hollis, Lina Soualmia, Brigitte Séroussi, and Croker R, Goldacre B. Suboptimal prescribing
pabilities were widely provided, but not write, the whole Yearbook Editorial Committee as behaviour associated with clinical software design
FHIR was the overwhelming standard that is well as the reviewers for their contribution features: a retrospective cohort study in English
NHS primary care. Br J Gen Pract J R Coll Gen
used, and it enabled communication of the to the selection process of the Human Factor Pract 2020 Sep;70(698):e636–43.
common clinical dataset, and validating data and Organizational Issues section for the 11. Dullabh P, Hovey L, Heaney-Huls K, Rajendran
from external sources complicated providers’ IMIA Yearbook 2021. N, Wright A, Sittig DF. Application Programming
workflows. Interfaces in Health Care: Findings from a Cur-
rent-State Sociotechnical Assessment. Appl Clin
Inform 2020 Jan;11(1):59–69.
References 12. Reeves JJ, Hollandsworth HM, Torriani FJ, Taplitz
R, Abeles S, Tai-Seale M, et al. Rapid response to
3.4 Timely Topics 1. Adler-Milstein J, Zhao W, Willard-Grace R, Knox COVID-19: health informatics support for out-
M, Grumbach K. Electronic health records and break management in an academic health system.
This final theme included timely topics burnout: Time spent on the electronic health record J Am Med Inform Assoc 2020 Jun 1;27(6):853–9.
from 2020: COVID-19 and health equity. after hours and message volume associated with 13. Altmann S, Milsom L, Zillessen H, Blasone
One of the best papers, Brewer et al. [2] exhaustion but not with cynicism among primary R, Gerdon F, Bach R, et al. Acceptability of
demonstrated how user-centered design care clinicians. J Am Med Inform Assoc 2020 Apr App-Based Contact Tracing for COVID-19:
can have a big impact on addressing health 1;27(4):531–8. Cross-Country Survey Study. JMIR MHealth
2. Brewer LC, Fortuna KL, Jones C, Walker R, Hayes UHealth [Internet]. 2020 Aug 28 [cited 2021 May
equity. Two other papers were on COVID-19. SN, Patten CA, et al. Back to the Future: Achieving 24];8(8). Available from: https://www.ncbi.nlm.
Reeves et al. [12] described the design and Health Equity Through Health Informatics and nih.gov/pmc/articles/PMC7458659/
implementation of EHR based tools for Digital Health. JMIR MHealth UHealth 2020 Jan
managing multiple aspects of COVID-19 14;8(1):e14512.
and is included in the special section of the 3. Reading Turchioe M, Grossman LV, Myers AC, Correspondence to:
Baik D, Goyal P, Masterson Creber RM. Visual Yalini Senathirajah
yearbook. Another paper by Altmann et al. analogies, not graphs, increase patients’ compre- University of Pittsburgh School of Medicine
[13] employed a survey across five countries hension of changes in their health status. J Am Med Pittsburgh, USA
to determine the acceptability of an automat- Inform Assoc 2020 May 1;27(5):677–89. E-mail: yalini@pitt.edu
IMIA Yearbook of Medical Informatics 2021103
Human Factors and Organizational Issues Section Synopsis IMIA Yearbook 2021
Appendix: Summary of Best No objective measures were associated with
high cynicism. These results are important
first project, FAITH! (Fostering African
American Improvement in Total Health),
Papers Selected for the 2021 since they fill an important gap of correlating described an intervention based on mHealth
Edition of the IMIA Year- self-reported EHR use with widely reported
objective EHR metrics of use, as well as cor-
and face to face church-based health educa-
tion and social support, especially directed
book, Section Human Factors relating objective measures of EHR use with at cardiovascular disease which has double
components of burnout. Since prior studies the mortality rate in African Americans com-
and Organizational Issues have correlated self-reported EHR use time pared to whites, and a higher incidence. They
with burnout, there was a need to validate used iterative formative design processes
Adler-Milstein J, Zhao W, Willard-Grace R, this correlation with objective measures of with a team of clinicians, technologists,
Knox M, Grumbach K use time, particularly as interventions are de- behavioral and social scientists, and made
signed to address EHR burden. Interestingly, use of patient preferences, such as using
Electronic health records and burnout: after hours EHR use was associated with spiritual verses and messaging. This led to
Time spent on the electronic health record exhaustion, but not cynicism, suggesting high apps ratings and acceptability, usability,
after hours and message volume associated that providers were feeling overwhelmed and satisfaction. Blood pressure, diet, and
with exhaustion but not with cynicism by work but did not develop cynicism as physical activity all significantly improved,
among primary care clinicians a response. The authors were hopeful that and the app had a 98% retention rate, re-
J Am Med Inform Assoc 2020;27(4):531-8 interventions designed to reduce the EHR markable for a health app. Tailored visual
burden (i.e., after hours work and message study results were also fed back to the com-
Primary care physicians have among volume) could help reduce these providers’ munity. The second project, Peertech CBPR
the highest rate of burnout of all clinical exhaustion. It is possible, however, that partnership, addressed premature mortality
specialties; further, these physicians have providers who have developed cynicism are in people with serious mental illness (SMI)
self-reported after hours work as a key those who have potentially inappropriately sch as bipolar disorder, major depressive
factor for feelings of burnout. This study cut short their EHR use as a coping mecha- disorder, and schizophrenia. The partner-
compared self-reported and objective mea- nism. Further studies are needed to further ing academic-community team identified
sures of electronic health record (EHR) work illuminate the relationship between cynicism this as a major health disparity. This led to
and proficiency and then correlated them and EHR burden in order to design effective co-creation of a smartphone app: Peer- and
with burnout measures for primary care interventions. Technology-Supported Self-Management
providers to better understand and improve Training (PeerTECH), which carried out
factors driving burnout. To do this, the study Brewer LC, Fortuna KL, Jones C, Walker R, simultaneous management of mental and
used self-reported measures of EHR use Hayes SN, Patten CA, Cooper LA chronic health conditions in patients aged
and proficiency, EHR supplied metrics for over 60. The approach consisted of equal
after-hours EHR use, messaging volume, Back to the future: Achieving health equity partnership between patients, certified peer
and proficiency, and Maslach Burnout In- through health informatics and digital specialists (CPS), leaders, and scientists
ventory exhaustion and cynicism subscale health from idea conception, defining of research
responses for 87 primary care providers in an JMIR Mhealth Uhealth 2020;8(1):e14512 questions, intervention development and
urban academic medical center. The results usability testing extending to dissemination.
showed that the subjective, self-reported This paper called attention to the serious Results included statistically significant
perceived EHR use time was correlated with issues of health equity and its systemic ef- psychiatric self-management (on the Illness
objective measures of EHR use after-hours fects and unintended consequences. These Management and Recovery Scale (IMRS)
and on unscheduled days, message volume, issues were brought to the fore during the (p104
Senatirajah et al.
end users and populations; 83% for the visual analogy and 70% for the age retrieval to present similar images with
4. Integrate community engagement through number line (P< 0.05) conditions. The results known diagnoses and the fourth presented
user-centered design or participatory supported using visual analogies rather than previously collected probabilities for each
design; text to display longitudinal PROs but the of the seven diagnoses as determined by
5. Gain an understanding of community authors cautioned against relying on graphs 511 human raters. A total of 302 human
partner technology infrastructure for ca- (known high prevalence of inadequate graph raters from 41 countries diagnosed batches
pacity building to support and strengthen literacy). Discrepancies between comprehen- of images both without decision support and
community-based health informatics sion and preferences suggested factors other then with one type of decision support. The
interventions; and than comprehension influenced preferences. results showed that the first representation
6. Plan the appropriate amount of time Future researchers should assess comprehen- improved the accuracy of human raters from
and resources to devote to community sion rather than preferences to guide pre- 63.6% to 77.0% (increase of 13.3%, 95% CI
engagement processes for intervention sentation decisions. This paper was the best 11.5% to 15.2%; P = 4.9 × 10−35, two-sided
development and sustainability. example of a continued emerging emphasis paired t-test, t = 14.5, d.f. = 301; n = 302 rat-
on visualization for patients and clinicians ers). No improvement was observed for the
The importance of this paper is in addressing to improve comprehension and workflow. second or third representation, with some
a current timely issue, with an unprecedented improvement for the fourth. They observed
degree of success in retention and disease Tschandl P, Rinner C, Apalla Z, Argenziano G, an inverse relationship between the net
control effectiveness, serving as an example Codella N, Halpern A, Janda M, Lallas A, gain accuracy increase due to AI and rater
of how to conduct such design successfully. Longo C, Josep Malvehy J, Paoli J, Puig S, experience; more inexperienced raters more
Rosendahl C, Soyer HP, Zalaudek I, Kittler H frequently changed their initial ratings after
Reading Turchioe M, Grossman LV, Myers viewing the AI results than experts did.
AC, Baik D, Goyal P, Masterson Creber RM Human–computer collaboration for skin After establishing a positive impact of AI
cancer recognition on diagnostic accuracy, the study tested the
Visual analogies, not graphs, increase impact of errors in AI on diagnostic accu-
patients’ comprehension of changes in their Nat Med 2020;26(8):1229–34
racy. These errors could be due to models
health status Image-based artificial intelligence (AI) has classifying images outside the domain of
J Am Med Inform Assoc 2020;27(5):677-89 the potential to improve visual diagnostic images used to train the model, something
accuracy in healthcare. Recent studies in which is a big concern for AI researchers.
The aim of this paper was to understand dermatology have shown that the accura- They intentionally showed raters incorrect
which visualizations are most effective in cy of AI for identifying skin lesions was multiclass probabilities, which decreased
aiding patients to self-monitor their health equivalent to or better than human experts the providers’ accuracy back to where it
status, via the use of patient-reported out- in controlled experimental studies; further, was without AI. The paper was very thor-
come (PRO) measures in hospitalized pa- human-AI cooperation can improve accuracy ough and presented several other analyses
tients, with a focus on objective comprehen- even more, but studies have not determined such as identifying two different diagnostic
sion. 40 hospitalized patients were included the best ways to incorporate AI into clinical categories that benefitted most of clinical
to compare fourvisualization conditions: (1) workflows for improving diagnostic accu- decision support based on AI, measuring the
text-only, (2) text plus visual analogy, (3) text racy in real healthcare settings. This study tendency of raters to change their minds,
plus number line, and (4) text plus line graph explored the impact of different representa- and determining how background features
assessed objective comprehension using tions of AI based clinical decision support affected both the AI and the human rater’s
the International Organization for Stan- for identifying skin lesions on the accuracy classification. Based on these multiple
dardization protocol. Secondary outcomes of clinical diagnoses. The first representation analyses, this paper justifiably advocates for
included response times, preferences, risk presented the probability that the lesion was studying human computer collaboration in
perceptions, and behavioral intentions. Six- in one of seven diagnostic categories: four real world clinic settings when evaluating the
ty-three percent correctly comprehended the malignant and three benign. The second performance of AI for any and all healthcare
text-only condition and 60% comprehended collapsed the seven categories into two: applications, not just those in dermatology.
the line graph condition, compared with malignant and benign. The third used im-
IMIA Yearbook of Medical Informatics 2021You can also read
