Lars G. Hemkens EVALUATION DIGITALER THERAPEUTIKA - NEUE STUDIENDESIGNS UND REAL WORLD EVIDENCE - Health Innovation Hub

EVALUATION DIGITALER THERAPEUTIKA
NEUE STUDIENDESIGNS UND REAL WORLD EVIDENCE

              Lars G. Hemkens
            Heidelberg, 27. Februar 2020

                                           web version

…

    University of Basel, Department of Clinical Research

“Women have
fewer teeth than
men“
            Aristoteles; HA II 3, 501b19

“Pneumonia   is one of
the diseases in which
 a timely venesection
        may save life“
        Sir William Osler; Principles and Practice of Medicine;1892

   Bloodletting for pneumonia
   Mercury for syphilis
   Lobotomy for psychiatric diseases
   Heroin for cough
   Bedrest in myocardial infarction
   Low-fiber diet in diverticulosis
   Babys sleep on stomach
   Surgery for peptic ulcers
   Intensive glucose lowering in type 2 diabetes
   Hormone replacement therapy in women
   Vitamine E in cardiac diseases
   Antiarrhythmic in myocardial infarction
   Stents in stable CVD

„Approximately   90% of new drugs
  entered into clinical development on
   promising preclinical findings
 fail to yield sufficient efficacy and
  safety to receive …(FDA) license“

                    Benjamin et al. PLoS Biol 2017 (citing Hay et al. Nature
                    Biotech 2014; 32: 40-51).
                    )

BIG DATA
ROUTINELY COLLECTED DATA

Insurance data / claims data
 Electronic Health Records
          Registries
           Microsoft
             Bing
            Google
            Twitter
           Amazon
          Facebook
          Whatsapp
            Netflix
           LinkedIn
          Wearables
             Fitbit
            Firefox
   IoT (Internet of Things)

Real World Data
“Real-world data are the data relating to patient health status
and/or the delivery of health care routinely collected from a
variety of sources. RWD can come from a number of sources,
for example:
 Electronic health records (EHRs)
 Claims and billing activities
 Product and disease registries
 Patient-generated data including in home-use settings
 Data gathered from other sources that can inform on health
  status, such as mobile devices”

                        www.fda.gov/science-research/science-and-research-special-topics/real-world-evidence
                        Accessed 23 Sep 2019

"Data collection under real-world circumstances
maximizes representativeness and generalizability,
minimizes costs and effort, and allows the capture of
information in large populations and many clinical
events in large data sets that are continuously
updated and cover long periods."

                                     Hemkens et al. CMAJ 2016

Safeguards of clinical trial research
          often lacking
Protocols
Registration
Research networks
Prospective Analysis
...

CONFOUNDING

From http://tylervigen.com/view_correlation?id=28807
Thanks to Tyler Vigen for sharing (accessed 22 February 2020)

From http://tylervigen.com/view_correlation?id=28672
Thanks to Tyler Vigen for sharing (accessed 22 February 2020)

From http://tylervigen.com/view_correlation?id=28669
Thanks to Tyler Vigen for sharing (accessed 22 February 2020)

From http://tylervigen.com/view_correlation?id=28829
Thanks to Tyler Vigen for sharing (accessed 22 February 2020)

Nationwide Big Data

       From www.tylervigen.com/spurious-correlations
       Thanks to Tyler Vigen for sharing (accessed 8 March 2017)

Nationwide Big Data

       From www.tylervigen.com/spurious-correlations
       Thanks to Tyler Vigen for sharing (accessed 8 March 2017)

Some Confounder Relationships
BMI and physical activity in early childhood with atopy
                          Paternal factors?
                          Ethnicity?
                          Resp. infections?
                          Socioeconomics?
                          Patient preferences and values?

                          Byberg et al. Clinical and Translational Allergy 2016;6:33; Figure S1
                          URL: https://ctajournal.biomedcentral.com/articles/10.1186/s13601-016-0124-9
                          CC-BY 4.0: http://creativecommons.org/licenses/by/4.0/ No changes were made

Some Confounder Relationships
 Alcohol and Cardiovascular disease

     Wallach et al. (PrePrint/under review)
     www.researchsquare.com/article/8e138689-12d5-43fa-87c9-9766d9e11a53/v1

97 studies:
           Pick
           your
          effect
Chu et al. Int J Epidemiol. 2020

"Five of 6 highly-cited
nonrandomized studies had been
contradicted or had found stronger
effects vs 9 of 39 randomized
controlled trials (P = .008)."

                        Ioannidis JAMA 2005

16 clinical questions / 16 RCD studies
36 subsequent RCTs
17 275 patients
835 deaths
Propensity Scores
Mortality

               Hemkens et al. BMJ 2016;352:i493
               URL: http://www.bmj.com/content/bmj/352/bmj.i493.full.pdf
               CC-BY 4.0: http://creativecommons.org/licenses/by/4.0/ No changes were made

RCD studies find 31% larger benefits than later RCTs
                    (ROR 1.31)

          In various re-analyses ROR from 1.25 to 1.58

                           Hemkens et al. BMJ 2016;352:i493
                           URL: http://www.bmj.com/content/bmj/352/bmj.i493.full.pdf
                           CC-BY 4.0: http://creativecommons.org/licenses/by/4.0/ No changes were made

“Big Data presents many
exciting new opportunities, but
there is no underlying
revolution in the logic of how
claims about the effects of
treatment should be justified.”
S Goodman, S Schneeweiss, M Baiocchi JAMA 2017

                                         JAMA 2017

…

Non-Randomized         Randomized

                 vs.
RCD                         Active

Learning from Winners

41 SHADES OF BLUE

„..a team at Google
  couldn’t decide between two blues,
            so they’re testing
      41 shades between each blue
  to see which one performs better...
I can’t operate in an environment like that.“

                Douglas Bowman, Google’s previous visual design leader left Google
                https://stopdesign.com/archive/2009/03/20/goodbye-google.html

Today,
Microsoft and several other leading companies
    including Amazon, Booking.com,
   Facebook, and Google each conduct

    more than 10000 online
    controlled experiments
               annually…

                      Kohavi R, Thomke HS. Havard Buisness Review 2017

„The idea increased
 Bing’s revenue by          12%
(over   $120M at the time)“
                         R. Kohavi
                         http://bit.ly/2017ABTestingTutorial

“Only one third
of the ideas tested at Microsoft
improved the metric(s) they
    were designed to improve”

             Kohavi, R. et al. 2013 Online Controlled Experiments at Large Scale.
             Retrieved from http://bit.ly/ExPScale

Kohavi et al. Trials 2020;21:150
URL: https://trialsjournal.biomedcentral.com/track/pdf/10.1186/s13063-020-4084-y
CC-BY 4.0: http://creativecommons.org/licenses/by/4.0/ No changes were made

Kohavi et al. Trials 2020;21:150
URL: https://trialsjournal.biomedcentral.com/track/pdf/10.1186/s13063-020-4084-y
CC-BY 4.0: http://creativecommons.org/licenses/by/4.0/ No changes were made

Learning Health Care System

RCD for RCT

RCD for RCT
Identification/recruitment
 Screen EHR-database for eligible
   patients
 Inform Study design
    Power Calculation/ Feasibility

 Send recruitment message

RCD for RCT
Extended follow-up
 Long-term outcomes
 Expanding previous trials
   (Registry Linkage)

RCD for RCT
Outcome Assessment
 Length of hospital stay
 Adverse events
 Complications
 ICU stay
 Re-admission
 Ressource use / Costs

 Many of the most patient-
relevant outcomes!

RCD for RCT
Intervention
 EHR-based diagnostic support
   tools (e.g. AI, algorithms)
 Alert systems
 Prescription feedback

 Learning healthcare system

RCD for RCT – Kosten

                                                   Mc Cord et al. CMAJ Open 2019
     https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6375253/pdf/cmajo.20180096.pdf

Schnelle Antworten
Glaubwürdige Evidenz
   Geringe Kosten

Randomisierte Studien

   Späte Antworten
 Glaubwürdige Evidenz
     Hohe Kosten

       ???

The Magic of
Randomization

WHY NOT ???
                                          MODELLIERUNG                   RANDOMISIERUNG

Datenmenge                                   deutlich mehr

Datenqualität                              höhere Anforderung

Statistische Modelle                           komplexer

Annahmen / Verständnis Wirkprinzip             komplexer

Risiko für Fehler/Bias                       deutlich höher

Einverständnis                                                                erforderlich

Kosten                                         eher höher

                                         Vorteil Modellierung         Vorteil Randomisierung 

IMMER: Registrierung, Studienprotokoll, Analyseplan, Ethikvotum, Follow-up, Publikationen…

“Obstacles to randomized trials
should be removed to protect
patients…
Promotion of nonrandomized analyses
… is a false solution to the problems
caused by the bureaucratic burdens
imposed on … trials...”
R Collins, L Bowman, M Landray, R Peto, NEJM 2020

                    “The Magic of Randomization versus the Myth of Real-World Evidence”
                    NEJM 382;7 February 13, 2020

“Sometimes when you innovate,
you make mistakes.

 It is best to admit them quickly,
 and get on with improving your
                other innovations“
                              Steve Jobs

PD Dr. med. Lars G. Hemkens MPH                                                                     Lars.Hemkens@usb.ch
Department of Clinical Research
University Hospital Basel Switzerland                                                               @LGHemkens
QUEST (Quality | Ethics | Open Science | Translation) Visiting Fellow
Berlin Institute of Health (Charité and Max Delbrück Center) | Berlin | Germany

      Special Thanks

      JPA Ioannidis (U Stanford)                                                  U Dirnagl (Charite / BIH / QUEST)
      D Contopoulus-Ioannidis (U Stanford)                                        B Kasenda (iomedico/U Basel)
      R Salman (U Edinburgh)                                                      TV Pereira (U Toronto)
      H Ewald (U Basel)                                                           F Naudet (METRICS / INSERM)
      KA Mc Cord (U Basel)                                                        O Fröbert (U Orebro)
      M Briel (U Basel)                                                           B Speich (U Oxford)
      C Pauli-Magnus (U Basel)                                                    R Kohavi (Microsoft/Airbnb)
      HM Verkoijen (U Utrecht)

Weiterführende Informationen
   Fröbert et al. (TASTE trial). Thrombus aspiration during ST-segment elevation myocardial infarction. N Engl J Med. 2013
    www.ncbi.nlm.nih.gov/pubmed/23991656
     Nationwide registry based RCT (Scandinavia); 0% Loss-to-follow-up; >7000 pts; costs about 350‘000 EUR

   Hemkens et al. Personalized Prescription Feedback Using Routinely Collected Data to Reduce Antibiotic Use in Primary Care:
    A Randomized Clinical Trial. JAMA Intern Med. 2017 www.ncbi.nlm.nih.gov/pubmed/28027333
     Nationwide admin/insurance data RCT (Switzerland); 2900 GPs, >10‘000‘000 consultations; 2 yrs; costs about 300‘000 USD

   The Magic of Randomization versus the Myth of Real-World Evidence. Collins R, Bowman L, Landray M, Peto R.
    N Engl J Med. 2020 www.ncbi.nlm.nih.gov/pubmed/32053307
     „Nonrandomized observational analyses have been promoted as alternatives to randomized clinical trials. However,
    randomization ensures balance between groups, whereas nonrandomized studies are often biased by between-group
    differences. Efforts to reduce the cost and complexity of clinical trials are preferable to relying on observational studies.“

   Using electronic health records for clinical trials: Where do we stand and where can we go?
    Mc Cord KA, Hemkens LG. CMAJ. 2019 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4868623/pdf/188e158.pdf

   How Routinely Collected Data for Randomized Trials Provide Long-term Randomized Real-World Evidence.
    Hemkens LG. JAMA Netw Open. 2018 Dec 7;1(8):e186014
    https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2719124

   Nonrandomized studies using causal-modeling may give different answers than RCTs: a meta-epidemiological study.
    Ewald et al. J Clin Epidemiol 2020 https://www.ncbi.nlm.nih.gov/pubmed/31704350

   Creating a Learning Health System through Rapid-Cycle, Randomized Testing Horwitz LI, Kuznetsova M, Jones SA.
    N Engl J Med 2019; www.ncbi.nlm.nih.gov/pubmed/31532967  „Randomized tests of quality-improvement interventions were
    embedded within an existing health care system. Quality and efficiency were improved by abandoning ineffective interventions
    and continuing successful quality-improvement strategies.“