Human Factors and Organizational Issues Section Synopsis IMIA Yearbook 2021

Page created by Janice Reid
Published online: 2021-09-03

    © 2021                       IMIA and Georg Thieme Verlag KG

     Human Factors and Organizational Issues
     Section Synopsis IMIA Yearbook 2021
     Yalini Senathirajah1, Michelle Hribar2, Section Editors of the IMIA Yearbook Section on
     Human Factors and Organizational Issues
       U. Pittsburgh School of Medicine, Dept. of Biomedical Informatics, Pittsburgh, PA, USA
       Oregon Health & Science University, Portland, OR, USA

     Summary                                                               1 Introduction                                   in each co-editor creating a set of 15 can-
                                                                                                                            didates each, which were then discussed.
     Objective: To select the best papers that made original and high
     impact contributions in the area of human factors and organiza-
                                                                           Papers published in 2020 on human factors        From this search we selected 13 best papers;
     tional issues in biomedical informatics in 2020.
                                                                           and organizational issues (HFOI) focused on      these were sent for additional peer review
     Methods: A rigorous extraction process based on queries from
                                                                           user healthcare technology and data needs,       by a set of external reviewers and two chief
     Web of Science® and PubMed/Medline was conducted to iden-
                                                                           for both patients and physicians. In addition,   editors, with each paper being reviewed by
     tify the scientific contributions published in 2020 that address
                                                                           several papers focused on the downstream         at least four people. We categorized the best
     human factors and organizational issues in biomedical informat-
                                                                           impact of Electronic Health Record (EHR)         papers according to four themes: 1) data
     ics. The screening of papers on titles and abstracts independently
                                                                           use and usability on provider burnout. Not       presentation for accuracy and understanding;
     by the two section editors led to a total of 1,562 papers. These
                                                                           surprisingly, there were also many papers        2) EHR usability and clinician burnout; 3)
     papers were discussed for a selection of 12 finalist papers, which
                                                                           about designing and implementing new             clinical system design; and 4) timely topics.
     were then reviewed by the two section editors, two chief editors,
                                                                           systems for Coronavirus Disease 2019             We then selected the best four papers using
     and by three external reviewers from internationally renowned
                                                                           (COVID-19), as well as on design consider-       these reviews, while also ensuring the papers
     research teams.
                                                                           ations for achieving health equity, both very    represented a variety of themes and had
     Results: The query process resulted in 12 papers that reveal
                                                                           timely topics in 2020. Finally, there were       strong methods and results.
     interesting and rigorous methods and important studies in
                                                                           other papers that focused on design factors
     human factors that move the field forward, particularly in clinical
                                                                           for healthcare systems. Together these papers
     informatics and emerging technologies such as brain-computer
                                                                           provide a small cross section of the best HFOI
     interfaces. This year three papers were clearly outstanding and
                                                                           research in biomedical informatics in 2020.      3 Results
     help advance in the field. They provide examples of applying
                                                                                                                            Table 1 lists the top four papers, in order of
     existing frameworks together in novel and highly illuminating
                                                                                                                            the first author’s surname.
     ways, showing the value of theory development in human factors.
     Emerging themes included several which discussed physician            2 Methods                                            Adler-Milstein et al. [1] combined com-
                                                                                                                            mon objective measures of EHR use with
     burnout, mobile health, and health equity. Those concerning the
                                                                           We collected papers from 2020 by a standard      Maslach Burnout inventory survey responses
     Corona Virus Disease 2019 (Covid-19) were included as part of
                                                                           search in PubMed and Web of Science®.            for 87 clinicians in primary care clinics of
     that section.
                                                                           Searches were performed in January 2020          a large, urban, academic medical center to
     Conclusion: The selected papers make important contributions to
                                                                           to identify peer-reviewed journal articles       determine how EHR use might be affecting
     human factors and organizational issues, expanding and deep-
                                                                           published in the English language in 2020,       burnout. They found that 34% of clinicians
     ening our knowledge of how to apply theory and applications of
                                                                           and related to HFOI research in biomedical       had high cynicism and 51% had high emo-
     new technologies in health.
                                                                           informatics. In addition to the search of        tional exhaustion. Clinicians who had the
                                                                           electronic databases, manual searches of key     highest measures of EHR time after hours
     Keywords                                                              themes were performed in major biomedi-          had much higher odds of high exhaustion;
     Human factors, human-computer interaction, organizational             cal journals (e.g., Journal of the American      this trend also held for clinicians with the
     issues, electronic health records, mobile health                      Medical Informatics Association, Methods         most message volume. No trends were ob-
                                                                           of Information in Medicine, Journal of Med-      served for cynicism.
     Yearb Med Inform 2021:100-4                                           ical Internet Research, …etc.). Independent          In their paper, Brewer et al. [2] present                              searches by the two co-editors resulted in       contextually tailored case examples of stud-
                                                                           an initial set of 1,562 papers. Independent      ies and projects which aim to address health
                                                                           reading of the titles and abstracts resulted     inequities by involving the community, i.e.,

    IMIA Yearbook of Medical Informatics 2021
                                                                                                                        Human Factors and Organizational Issues Section Synopsis IMIA Yearbook 2021

Table 1 Best paper selection of articles for the IMIA Yearbook of Medical Informatics 2021 in the section 'Human Factors and Organizational   paper focused on presenting results from
Issues'. The articles are listed in alphabetical order of the first author’s surname.                                                         AI models for diagnosing skin lesions. In
                                                                                                                                              addition, Stonbraker et al. [5] designed a
                                                                                                                                              thorough study of data presentations of
  Human Factors and Organizational Issues                                                                                                     longitudinal patient reported outcomes to
                                                                                                                                              determine which ones promoted the best
   Adler-Milstein J, Zhao W, Willard-Grace R, Knox M, Grumbach K. Electronic health records and burnout: Time spent on the                   comprehension. They found that patients
    electronic health record after hours and message volume associated with exhaustion but not with cynicism among primary care
                                                                                                                                              interpreted the visualizations very literally
    clinicians. J Am Med Inform Assoc 2020;27(4):531-8.
   Brewer LC, Fortuna KL, Jones C, Walker R, Hayes SN, Patten CA, Cooper LA. Back to the future: Achieving health equity                     and that while they preferred bar charts with
    through health informatics and digital health. JMIR Mhealth Uhealth 2020;8(1):e14512.                                                     emojis, they could confuse the meaning of
   Reading Turchioe M, Grossman LV, Myers AC, Baik D, Goyal P, Masterson Creber RM. Visual analogies, not graphs, increase                   the emojis if they were not precisely tailored
    patients’ comprehension of changes in their health status. J Am Med Inform Assoc 2020;27(5):677-89.                                       to the symptom or outcome they were rep-
   Tschandl P, Rinner C, Apalla Z, Argenziano G, Codella N, Halpern A, Janda M, Lallas A, Longo C, Josep Malvehy J, Paoli                    resenting.
    J, Puig S, Rosendahl C, Soyer HP, Zalaudek I, Kittler H. Human–computer collaboration for skin cancer recognition. Nat
    Med 2020;26(8):1229–34.

                                                                                                                                              3.2 EHR Usability and Clinician
co-design approaches. It strategically merged                           example of a continued emerging emphasis                              The Adler-Milstein paper et al. paper [1]
health services research with communi-                                  on visualization for patients, including those                        described above fell into this category. An-
ty-based participatory research (CBPR) for                              with lower literacy, and cautioned against                            other excellent paper by Melnick et al. [6]
innovation and development. The primary                                 over-reliance on graphs.                                              performed a secondary analysis of a survey
contribution of this paper is a set of resilience                          Finally, Tschandl et al. [4] tested a variety                      of US physicians from all specialties to de-
strategies that could be supported by innova-                           of representations of artificial intelligence                         termine the association between physicians’
tions in health information technology in fu-                           (AI) for rating skin lesions for providing                            perceived EHR usability and workload and
ture research. The paper addresses the serious                          decision support to physicians and measured                           their professional burnout. They found a
problem of health equity and calls attention to                         the resulting accuracy and susceptibility to                          strong relationship between EHR usability
the possibility that the new massive activity                           randomly introduced errors. They found that                           and workload, with more favorable usability
in health informatics including digital health                          when they provided the probabilities for the                          associated with a lower task load.
may have an unintended consequences: to                                 full range of diagnoses, the decision support                            Another innovative paper by Weir et al.
worsen health inequities and create new forms                           improved the physician’s diagnostic accuracy                          [7] was a position paper that explored how
of inequity. Thus, designers and informati-                             by 13% but that the decision support for de-                          burnout is affected by three aspects of cogni-
cians must guard against these consequences                             termination of malignancy did not improve                             tion and motivation: 1) goal-based decision
in their ongoing work.                                                  accuracy. They also found that physicians                             making; 2) drive for sense-making and
    Patient-reported outcomes (PROs) in-                                were susceptible to accepting randomly                                meaning; and 3) need for agency, autonomy,
creasingly use self-monitor health status                               incorrect predictions once they had used the                          and control. Through a thorough discussion
and visualization over time would help                                  AI based decision support tools.                                      of prior research, the paper hoped to spur
patients interpret their data. The study of                                Although not selected as best papers, the                          research that applies concepts of cognitive
Reading Turchioe et al. [3] included 40                                 remaining candidate best papers were just                             science to understanding and mitigating
hospitalized patients and compared four                                 as excellent and influential. We summarize                            physician burnout.
visualization conditions: (1) text-only;                                them below, organized by the four major
(2) text plus visual analogy; (3) text plus                             themes for the papers: 1) data presentation
number line; and (4) text plus line graph                               for accuracy and understanding; 2) EHR
assessed objective comprehension using                                  usability and clinician burnout; 3) clinical                          3.3 Clinical Systems Design
the International Organization for Stan-                                system design; and 4) timely topics.                                  This theme had the broadest topics of papers,
dardization protocol. Secondary outcomes                                                                                                      from user and patient designed systems, to the
included response times, preferences, risk                                                                                                    negative impact of a medication ordering sys-
perceptions, and behavioral intentions. As                                                                                                    tem, to the state of application programming
a result, 63% correctly comprehended the                                3.1 Data Presentation for                                             interfaces in healthcare. A paper by Farao
text-only condition and 60% comprehended                                Accuracy and Understanding                                            et al. [8] described a user-centered design
the line graph condition, compared with                                 Two of the four selected best papers fell into                        framework for mobile health applications
83% for the visual analogy and 70% for the                              this category; the Reading-Turchioe et al                             and demonstrated its use for an app that
number line (P< 0.05) conditions. This was                              paper [3] focused on presenting healthcare                            read the tuberculin skin test. Combining the
the top voted paper and provided the best                               data to patients while the Tschandl et al. [4]                        information systems research framework with

                                                                                                                                                          IMIA Yearbook of Medical Informatics 2021
Senatirajah et al.

design thinking, the new framework allowed        ed contact tracing application. They found               4. Tschandl P, Rinner C, Apalla Z, Argenziano G,
for iterative refinement of designs through       widespread support for such an application                   Codella N, Halpern A, et al. Human–computer
                                                                                                               collaboration for skin cancer recognition. Nat Med
interactions with users. A paper by Ng et al.     and concluded it could be a viable tool for                  2020 Aug;26(8):1229–34.
[9] described patient designed enhancements       limiting spread of the virus.                            5. Stonbraker S, Porras T, Schnall R. Patient prefer-
to a continuous glucose monitoring system                                                                      ences for visualization of longitudinal patient-re-
for Type I diabetes. These “do-it-yourself ”                                                                   ported outcomes data. J Am Med Inform Assoc
innovations formed the Nightscout Project,                                                                     2020 Feb 1;27(2):212–24.
                                                                                                           6. Melnick ER, Harry E, Sinsky CA, Dyrbye LN,
a continually expanding set of open-source
technologies driven by a community of Type
                                                  4 Conclusion                                                 Wang H, Trockel MT, et al. Perceived Electronic
                                                                                                               Health Record Usability as a Predictor of Task
I diabetic patients, their parents, and their     The HFOI papers this year reflected import-                  Load and Burnout Among US Physicians: Me-
caregivers. Next, a paper by MacKenna et          ant progress and diversification of themes.                  diation Analysis. J Med Internet Res 2020 Dec
al. [10] described how a poorly designed          The pandemic and issues it has raised formed                 22;22(12):e23382.
                                                  an important set of advances, but also                   7. Weir CR, Taber P, Taft T, Reese TJ, Jones B, Del
medication ordering system resulted in 9.5                                                                     Fiol G. Feeling and thinking: can theories of hu-
million excess cost in one year for ordering      expanded existing topics such as clinician                   man motivation explain how EHR design impacts
more expensive generic medication than            burnout and the importance of usability in                   clinician burnout? J Am Med Inform Assoc 2021
was necessary. Finally, a paper by Dullabh        patient-facing interventions. The emergence                  Apr 23;28(5):1042–6.
et al. [11] used a socio-technical model to       of equity and usability as a serious concern is          8. Farao J, Malila B, Conrad N, Mutsvangwa T,
                                                  overdue, and all these advances are welcome                  Rangaka MX, Douglas TS. A user-centred design
structure an analysis of the current state                                                                     framework for mHealth. PLOS One 2020 Aug
of application programming interfaces in          expansions in the field.                                     19;15(8):e0237910.
healthcare for providing necessary interop-                                                                9. Ng M, Borst E, Garrity A, Hirschfeld E, Lee J.
erability and data sharing. Not surprisingly,     Acknowledgements                                             Evolution of Do-It-Yourself Remote Monitoring
they found a large amount of potential use        We would like to acknowledge the support                     Technology for Type 1 Diabetes. J Diabetes Sci
                                                  of Adrien Ugon, Martina Hutter, Kate Fultz                   Technol 2020 Sep;14(5):854–9.
cases and existing applications, most of which                                                             10. MacKenna B, Curtis HJ, Walker AJ, Bacon S,
focused on filling a physician’s need, read ca-   Hollis, Lina Soualmia, Brigitte Séroussi, and                Croker R, Goldacre B. Suboptimal prescribing
pabilities were widely provided, but not write,   the whole Yearbook Editorial Committee as                    behaviour associated with clinical software design
FHIR was the overwhelming standard that is        well as the reviewers for their contribution                 features: a retrospective cohort study in English
                                                                                                               NHS primary care. Br J Gen Pract J R Coll Gen
used, and it enabled communication of the         to the selection process of the Human Factor                 Pract 2020 Sep;70(698):e636–43.
common clinical dataset, and validating data      and Organizational Issues section for the                11. Dullabh P, Hovey L, Heaney-Huls K, Rajendran
from external sources complicated providers’      IMIA Yearbook 2021.                                          N, Wright A, Sittig DF. Application Programming
workflows.                                                                                                     Interfaces in Health Care: Findings from a Cur-
                                                                                                               rent-State Sociotechnical Assessment. Appl Clin
                                                                                                               Inform 2020 Jan;11(1):59–69.
                                                  References                                               12. Reeves JJ, Hollandsworth HM, Torriani FJ, Taplitz
                                                                                                               R, Abeles S, Tai-Seale M, et al. Rapid response to
3.4 Timely Topics                                 1. Adler-Milstein J, Zhao W, Willard-Grace R, Knox           COVID-19: health informatics support for out-
                                                     M, Grumbach K. Electronic health records and              break management in an academic health system.
This final theme included timely topics              burnout: Time spent on the electronic health record       J Am Med Inform Assoc 2020 Jun 1;27(6):853–9.
from 2020: COVID-19 and health equity.               after hours and message volume associated with        13. Altmann S, Milsom L, Zillessen H, Blasone
One of the best papers, Brewer et al. [2]            exhaustion but not with cynicism among primary            R, Gerdon F, Bach R, et al. Acceptability of
demonstrated how user-centered design                care clinicians. J Am Med Inform Assoc 2020 Apr           App-Based Contact Tracing for COVID-19:
can have a big impact on addressing health           1;27(4):531–8.                                            Cross-Country Survey Study. JMIR MHealth
                                                  2. Brewer LC, Fortuna KL, Jones C, Walker R, Hayes           UHealth [Internet]. 2020 Aug 28 [cited 2021 May
equity. Two other papers were on COVID-19.           SN, Patten CA, et al. Back to the Future: Achieving       24];8(8). Available from: https://www.ncbi.nlm.
Reeves et al. [12] described the design and          Health Equity Through Health Informatics and    
implementation of EHR based tools for                Digital Health. JMIR MHealth UHealth 2020 Jan
managing multiple aspects of COVID-19                14;8(1):e14512.
and is included in the special section of the     3. Reading Turchioe M, Grossman LV, Myers AC,            Correspondence to:
                                                     Baik D, Goyal P, Masterson Creber RM. Visual          Yalini Senathirajah
yearbook. Another paper by Altmann et al.            analogies, not graphs, increase patients’ compre-     University of Pittsburgh School of Medicine
[13] employed a survey across five countries         hension of changes in their health status. J Am Med   Pittsburgh, USA
to determine the acceptability of an automat-        Inform Assoc 2020 May 1;27(5):677–89.                 E-mail:

IMIA Yearbook of Medical Informatics 2021
                                                                                  Human Factors and Organizational Issues Section Synopsis IMIA Yearbook 2021

Appendix: Summary of Best                        No objective measures were associated with
                                                 high cynicism. These results are important
                                                                                                      first project, FAITH! (Fostering African
                                                                                                      American Improvement in Total Health),
Papers Selected for the 2021                     since they fill an important gap of correlating      described an intervention based on mHealth
Edition of the IMIA Year-                        self-reported EHR use with widely reported
                                                 objective EHR metrics of use, as well as cor-
                                                                                                      and face to face church-based health educa-
                                                                                                      tion and social support, especially directed
book, Section Human Factors                      relating objective measures of EHR use with          at cardiovascular disease which has double
                                                 components of burnout. Since prior studies           the mortality rate in African Americans com-
and Organizational Issues                        have correlated self-reported EHR use time           pared to whites, and a higher incidence. They
                                                 with burnout, there was a need to validate           used iterative formative design processes
Adler-Milstein J, Zhao W, Willard-Grace R,       this correlation with objective measures of          with a team of clinicians, technologists,
Knox M, Grumbach K                               use time, particularly as interventions are de-      behavioral and social scientists, and made
                                                 signed to address EHR burden. Interestingly,         use of patient preferences, such as using
Electronic health records and burnout:           after hours EHR use was associated with              spiritual verses and messaging. This led to
Time spent on the electronic health record       exhaustion, but not cynicism, suggesting             high apps ratings and acceptability, usability,
after hours and message volume associated        that providers were feeling overwhelmed              and satisfaction. Blood pressure, diet, and
with exhaustion but not with cynicism            by work but did not develop cynicism as              physical activity all significantly improved,
among primary care clinicians                    a response. The authors were hopeful that            and the app had a 98% retention rate, re-
J Am Med Inform Assoc 2020;27(4):531-8           interventions designed to reduce the EHR             markable for a health app. Tailored visual
                                                 burden (i.e., after hours work and message           study results were also fed back to the com-
Primary care physicians have among               volume) could help reduce these providers’           munity. The second project, Peertech CBPR
the highest rate of burnout of all clinical      exhaustion. It is possible, however, that            partnership, addressed premature mortality
specialties; further, these physicians have      providers who have developed cynicism are            in people with serious mental illness (SMI)
self-reported after hours work as a key          those who have potentially inappropriately           sch as bipolar disorder, major depressive
factor for feelings of burnout. This study       cut short their EHR use as a coping mecha-           disorder, and schizophrenia. The partner-
compared self-reported and objective mea-        nism. Further studies are needed to further          ing academic-community team identified
sures of electronic health record (EHR) work     illuminate the relationship between cynicism         this as a major health disparity. This led to
and proficiency and then correlated them         and EHR burden in order to design effective          co-creation of a smartphone app: Peer- and
with burnout measures for primary care           interventions.                                       Technology-Supported Self-Management
providers to better understand and improve                                                            Training (PeerTECH), which carried out
factors driving burnout. To do this, the study   Brewer LC, Fortuna KL, Jones C, Walker R,            simultaneous management of mental and
used self-reported measures of EHR use           Hayes SN, Patten CA, Cooper LA                       chronic health conditions in patients aged
and proficiency, EHR supplied metrics for                                                             over 60. The approach consisted of equal
after-hours EHR use, messaging volume,           Back to the future: Achieving health equity          partnership between patients, certified peer
and proficiency, and Maslach Burnout In-         through health informatics and digital               specialists (CPS), leaders, and scientists
ventory exhaustion and cynicism subscale         health                                               from idea conception, defining of research
responses for 87 primary care providers in an    JMIR Mhealth Uhealth 2020;8(1):e14512                questions, intervention development and
urban academic medical center. The results                                                            usability testing extending to dissemination.
showed that the subjective, self-reported        This paper called attention to the serious           Results included statistically significant
perceived EHR use time was correlated with       issues of health equity and its systemic ef-         psychiatric self-management (on the Illness
objective measures of EHR use after-hours        fects and unintended consequences. These             Management and Recovery Scale (IMRS)
and on unscheduled days, message volume,         issues were brought to the fore during the           (p
Senatirajah et al.

   end users and populations;                       83% for the visual analogy and 70% for the        age retrieval to present similar images with
4. Integrate community engagement through           number line (P< 0.05) conditions. The results     known diagnoses and the fourth presented
   user-centered design or participatory            supported using visual analogies rather than      previously collected probabilities for each
   design;                                          text to display longitudinal PROs but the         of the seven diagnoses as determined by
5. Gain an understanding of community               authors cautioned against relying on graphs       511 human raters. A total of 302 human
   partner technology infrastructure for ca-        (known high prevalence of inadequate graph        raters from 41 countries diagnosed batches
   pacity building to support and strengthen        literacy). Discrepancies between comprehen-       of images both without decision support and
   community-based health informatics               sion and preferences suggested factors other      then with one type of decision support. The
   interventions; and                               than comprehension influenced preferences.        results showed that the first representation
6. Plan the appropriate amount of time              Future researchers should assess comprehen-       improved the accuracy of human raters from
   and resources to devote to community             sion rather than preferences to guide pre-        63.6% to 77.0% (increase of 13.3%, 95% CI
   engagement processes for intervention            sentation decisions. This paper was the best      11.5% to 15.2%; P = 4.9 × 10−35, two-sided
   development and sustainability.                  example of a continued emerging emphasis          paired t-test, t = 14.5, d.f. = 301; n = 302 rat-
                                                    on visualization for patients and clinicians      ers). No improvement was observed for the
The importance of this paper is in addressing       to improve comprehension and workflow.            second or third representation, with some
a current timely issue, with an unprecedented                                                         improvement for the fourth. They observed
degree of success in retention and disease          Tschandl P, Rinner C, Apalla Z, Argenziano G,     an inverse relationship between the net
control effectiveness, serving as an example        Codella N, Halpern A, Janda M, Lallas A,          gain accuracy increase due to AI and rater
of how to conduct such design successfully.         Longo C, Josep Malvehy J, Paoli J, Puig S,        experience; more inexperienced raters more
                                                    Rosendahl C, Soyer HP, Zalaudek I, Kittler H      frequently changed their initial ratings after
Reading Turchioe M, Grossman LV, Myers                                                                viewing the AI results than experts did.
AC, Baik D, Goyal P, Masterson Creber RM            Human–computer collaboration for skin             After establishing a positive impact of AI
                                                    cancer recognition                                on diagnostic accuracy, the study tested the
Visual analogies, not graphs, increase                                                                impact of errors in AI on diagnostic accu-
patients’ comprehension of changes in their         Nat Med 2020;26(8):1229–34
                                                                                                      racy. These errors could be due to models
health status                                       Image-based artificial intelligence (AI) has      classifying images outside the domain of
J Am Med Inform Assoc 2020;27(5):677-89             the potential to improve visual diagnostic        images used to train the model, something
                                                    accuracy in healthcare. Recent studies in         which is a big concern for AI researchers.
The aim of this paper was to understand             dermatology have shown that the accura-           They intentionally showed raters incorrect
which visualizations are most effective in          cy of AI for identifying skin lesions was         multiclass probabilities, which decreased
aiding patients to self-monitor their health        equivalent to or better than human experts        the providers’ accuracy back to where it
status, via the use of patient-reported out-        in controlled experimental studies; further,      was without AI. The paper was very thor-
come (PRO) measures in hospitalized pa-             human-AI cooperation can improve accuracy         ough and presented several other analyses
tients, with a focus on objective comprehen-        even more, but studies have not determined        such as identifying two different diagnostic
sion. 40 hospitalized patients were included        the best ways to incorporate AI into clinical     categories that benefitted most of clinical
to compare fourvisualization conditions: (1)        workflows for improving diagnostic accu-          decision support based on AI, measuring the
text-only, (2) text plus visual analogy, (3) text   racy in real healthcare settings. This study      tendency of raters to change their minds,
plus number line, and (4) text plus line graph      explored the impact of different representa-      and determining how background features
assessed objective comprehension using              tions of AI based clinical decision support       affected both the AI and the human rater’s
the International Organization for Stan-            for identifying skin lesions on the accuracy      classification. Based on these multiple
dardization protocol. Secondary outcomes            of clinical diagnoses. The first representation   analyses, this paper justifiably advocates for
included response times, preferences, risk          presented the probability that the lesion was     studying human computer collaboration in
perceptions, and behavioral intentions. Six-        in one of seven diagnostic categories: four       real world clinic settings when evaluating the
ty-three percent correctly comprehended the         malignant and three benign. The second            performance of AI for any and all healthcare
text-only condition and 60% comprehended            collapsed the seven categories into two:          applications, not just those in dermatology.
the line graph condition, compared with             malignant and benign. The third used im-

IMIA Yearbook of Medical Informatics 2021
You can also read