QUEST-KRITERIEN EINFÜHRUNGSVERANSTALTUNG CSP 2018 - Berliner Institut für Gesundheitsforschung 13. September 2018 - Berliner Institut für ...
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
EINFÜHRUNGSVERANSTALTUNG CSP 2018 QUEST-KRITERIEN Berliner Institut für Gesundheitsforschung 13. September 2018
Die folgenden Folien sind als Anregung zu verstehen, sie ersetzen nicht die AKTIVE Auseinandersetzung mit den QUEST-Kriterien im Hinblick auf Ihre spezifische Forschungsfrage. Bei Fragen wenden Sie sich gerne an Dr. Miriam Kip (miriam.kip@bihealth.de oder quest@bihealth.de). QUEST Tool box: (https://www.bihealth.org/en/quest-center/mission-approaches/englische- uebersetzung/the-quest-toolbox/)
MERIT-Project
Development of attributes of a robust and innovative
research/ Merkmale einer robusten und innovativen
Miriam Kip
Forschung (MERIT)
QUEST-criteria:
Axel Pries open questions:
• priority setting
• strategies of scientific rigor
• transparency and dissemination or results
• participation
• intramural funding schemes, e.g. CSP, Validation fund
• introduction to the doctoral/dissertation program at Charité
PRIORITY SETTINGS
Warum?
• Vorhandene Evidenz darlegen
• Wissenslücken identifizieren
• Bisherige Studienqualität kritisch bewerten
• Identifikation von Faktoren, die die Effektivität einer Maßnahme
beeinflussen
• Ableiten wichtiger Informationen hinsichtlich Design neuer
Studien
• “evidence-based trial design”
- Reduce waste in future research
- Reduce risk for humans and animals
- Reduce risk of unnecessary enrollment of humans
Wie? (a very short introduction)
• Clinical Interventions - PICOS
• Population Meshterms
Volltextsuche
• Intervention general expressions
• Control/Comparators
Boolsche Operanden
• Outcome
• Study design Filter
• preclinical
• (a) treatment/intervention
• (b) disease or condition of interest
• (c) animal species/cell population studied
• (d) outcome measures
Wo? • PubMed, Embase • Clinicaltrials.gov, Metaregister der WHO etc.. • Eigene Vorstudien (dabei Daten nachvollziehbar darstellen)
Ressourcen (Auswahl) • BMJ 2011;343:d5928 doi: 10.1136/bmj.d5928 • Hooijmans et al. BMC Medical Research Methodology 2014, 14:43 http://www.biomedcentral.com/1471-2288/14/43 • http://syrf.org.uk/library/ • https://www.york.ac.uk/media/crd/Systematic_Reviews.pdf • https://www.cochrane.de/de/ressourcen • https://www.radboudumc.nl/en/research/radboud-technology- centers/animal-research-facility/systematic-review-center-for-laboratory- animal-experimentation • https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3265183/pdf/LA-11-087.pdf • https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3104815/pdf/LA-09-117.pdf
STRATEGIES OF SCIENTIFIC RIGOR (I)
Highly effective treatments in animal stroke models PLoS Biol. 2010;8(3):e1000344.
Phase III studies show no effect Minnerup et al. Exp.Transl. Stroke Med. 2014;6:2
Possible
Solutions
Reduce Bias!
Use blinding, randomization,in/exclusion criteria.
Report results according to guidelines (e.g. ARRIVE, CONSORT, PRISMA, etc.)
Pre-register
Use statistics sensibly!
Go beyond the bar plot! Show individual data points and distributions.
Think biological significance, think effect size.
Consult a statistician.Bias influences effect size
Stroke models (NXY-095) Alzheimer's disease models
models
Improvement in behavioural outcome
1.2
(Standardised Effect Size)
1.0
Reduction in infarct size
Reduction in infarct size 0.8
0.6
0.4
0.2
0.0
Yes No
Blinded conduct of Blinded assessment Blinded assessment of
experiment of outcome outcome
> 30 studies > 500 animalsTools and Resources Statistical Consulting (CRU and Biostats) QUEST toolbox (includes tools for Figure creation) Experimental Design Assistant for animal experiments Courses at the QUEST Center and Promotionskolleg
STRATEGIES OF SCIENTIFIC RIGOR (II)
Clinical Scientist
Prof. Dr. Geraldine Rauch
Dr. Jochen Kruppa
Institute of Biometry and
Clinical Epidemiology
geraldine.rauch@charite.de
UNIVERSITÄTSMEDIZIN BERLINAuswahl von Endpunkten Endpunkte: Diejenigen Merkmale in klinischen Studien, anhand derer der Erfolg der Studie gemessen werden soll. Man unterscheidet dabei zwischen primären und sekundären Endpunkten. Primäre Endpunkt: Erfasst dabei das Hauptziel der Studie. Dieses wird am Ende der Studie mit einem statistischen Test überprüft. Sekundären Endpunkte: Erfassen dabei weitere Aspekte der Studie. Die Auswertung erfolgt rein deskriptiv. Bei der Auswahl von Endpunkten sollte Folgendes beachtet werden: Angemessen für die medizinische Fragestellung Möglichst objektiv erfassbar Möglichst hohes Skalenniveau (vgl. spätere Folien)
Begriffe Zielgröße / Endpunkt: das Merkmal, das als Ergebnis einer Untersuchung interessiert, z.B. eine unter dem Einfluss der Therapie sich verändernder Laborwert oder ein Krankheitssymptom Einflussgrößen: alle Merkmale die im funktionellen Zusammenhang zur Zielgröße stehen, z.B. bestimmte Behandlungsformen, Therapiemaßnahmen Störgrößen/Confounder: Einflussgrößen, deren Untersuchung nicht Ziel der Studie ist, z.B. die unerwünschte Abhängigkeit vom Alter oder Geschlecht. Störgrößen sollten entweder eliminiert oder in der Analyse der Zielgrößen berücksichtigt werden. Störgrößen sind jedoch nicht immer alle erfassbar.
Harte und weiche Endpunkte
• Harte Endpunkte
– lassen sich direkt erheben
– Beispiele:
• Überlebenszeit
• Tumoransprechen
• Weiche Endpunkte
– lassen sich nur indirekt erheben
– Beispiele:
• Lebensqualität
• SchmerzempfindenStudiendesign
nicht kontrolliert/ kontrolliert/
einarmig mehrarmig
Nicht für jede Fragestellung
ist jedes Design möglich!
nicht randomisiert randomisiert
einfach
offen verblindet
doppelt
Goldstandard für Studien
monozentrisch multizentrisch zum WirksamkeitsnachweisStudientypen
Es gibt viele verschiedene Studientypen
Klassifizierung nach unterschiedlichen Kriterien möglich:
• Fragestellung (z.B. Therapiestudie, Diagnosestudie,
Prognosestudie),
• Blickrichtung (prospektiv, retrospektiv),
• “Aktivität” des Forschers
– Beobachtungsstudien (Querschnittsstudien, Kohortenstudien,
Fall-Kontroll-Studien)
– Interventionsstudien (nichtrandomisierte Studien,
randomisierte Studie)
• Bei Arzneimittelstudien: „Phase“ der Erprobung (Phase I, II, III, IV)
• usw.
Hier kein umfassender Überblick möglich!Quelle: Dtsch Arztebl 2009; 106(15)
Literaturempfehlung
• Schumacher M, Schulgen G (2008, 3. Auflage): Methodik klinischer
Studien, Springer.
• Sachs L (1993, 7. Auflage): Statistische Methoden: Planung und
Auswertung, Springer.
• Weiß C (2010, 5. Auflage): Basiswissen Medizinische Statistik, Springer.
• Gonick L (1993): The Cartoon Guide to Statsitics, HarperCollins PublisherSTRATEGIES OF SCIENTIFIC RIGOR (III)
Reveal, Don’t Conceal: Transforming Data
Visualization to Improve Transparency
Tracey L. Weissgerber, PhD
Twitter: @T_WeissgerberData presentation is the foundation of our
collective scientific knowledge…
Figures are especially important.
They often show data for key findings.Many different data distributions can lead to the
same bar graph…
Symmetric Outlier Bimodal Unequal n
30
20
10
0
Test p value
T-test: equal var. 0.035 0.074 0.033 0.051
T-test: Unequal var. 0.035 0.076 0.033 0.035
Wilcoxon 0.056 0.10 0.173 0.067
Weissgerber et al., PLOS Biology 2015Why you shouldn’t use a bar graph even if
your data are normally distributed
Bar graph Bar graph Univariate
(mean ± SE) with points scatterplot
Zone of
Range of Invisibility
Observed
Heart rate (beats/min)
Values
Zone of
Irrelevance
0
Sedentary Exercise Sedentary Exercise
Trained Trained
Bar graphs
1. Don’t allow you to critically evaluate continuous data
2. Arbitrarily assign importance to bar height, instead of showing
how the difference between means compares to the variability
Weissgerber et al., JBC 2017Graphics for:
- Cross sectional studies
- Experimental studies with independent groups
Dotplot Boxplot with Boxplot Violin plot Bar graph
points (with or
without
points)
Outcome Continuous Continuous Continuous Continuous Counts &
variable proportions
Sample size Small Medium Large Medium to Any
Large
Data Any Any Do not use for Any N/A
distribution bimodal data
Free violin plot tool: https://interactive-graphics.shinyapps.io/violin/Free Tools for Interactive Graphics
20 Group 1
Group 2
15 Group 3
10
5
0
Condition 1 Condition 2 Condition 3
Dot, box or violin plot: http://statistika.mfub.bg.ac.rs/interactive-
dotplot/
Interactive line graph:
http://statistika.mfub.bg.ac.rs/interactive-graph/
Additional resources: Twitter @T_WeissgerberWhy we need to report more than “Data were
analyzed by t-tests and ANOVA”
• Meta-research studies show that statistical
errors are common. These include:
– Failing to specify what test was used
– Using suboptimal or inappropriate tests
– Incorrect p-values
• T-tests and ANOVA are the most common
analysis techniques in many basic
biomedical science fieldsWhy we need to report more than “Data were
analyzed by t-tests and ANOVA”
Systematic review: Many physiology papers are
missing information needed to determine what type
of ANOVA was performed
Essential Details Papers with Missing
Information
Number of factors 17%
Names of factors 54%
Post-hoc tests 27%
Between vs. within-subjects factors for 63%
repeated measures ANOVAPapers rarely contain information needed to verify
the test result
Essential Details Papers with Missing
Information
T-tests ANOVA
(n = 163) (n = 225)
Test statistic 96% 95%
Degrees of freedom * 7% 97%
Exact p-value 69% 78%
* Exact sample size is also acceptable for t-testsThis information is essential to identify bias &
correct errors
1. Confirm that the correct test was used
2. Confirm test results
– Errors in reported p-value are common; may alter
conclusions in 1/8 papers1
3. Assess bias: Were observations excluded
without explanation?
– Among papers with animal models of cancer & stroke2
• 7-8% excluded animals without explanation
• 2/3 didn’t have enough information to assess
1Nuitjen et al., Behav Res Methods 2015
2Holman et al., PLOS Biol 2016Solutions
• Report exactly what test you used
Test Reporting
T-tests Unpaired vs. paired, equal vs. unequal variance
ANOVA Number & names of factors, between vs.
within subjects factors, post-hoc tests, any
interaction terms included in the ANOVA
More complex tests Detail needed to reproduce analysisWhy does this matter? An example…
These two ANOVA Repeated Measures
tests… without repeated measures ANOVA
…see the data Compares 60 Compares 60
differently 3 independent groups 40
10 pairs 40
(n = 30, 10/group) of related
20
20
observations
0 (n = 10) 0
T1 T2 T3
0 T1 1 T2 2 T3 3
…test different Null hypothesis: Null hypothesis:
hypotheses Mean T1 = Mean T2 = Mean T3 Mean T1 = Mean T2 = Mean T3
when population means are related
…use More unexplained variability Less unexplained variability – we can
information account for the effect of “subject”.
differently
Sums of squares Sums of squares
Conditions (between groups) 240 Conditions (between groups) 240
Residual (within groups) 2604 Residual (within groups) 2604
Subjects 2195
? Error 409
Total 2844 Total 2844
…give different overall p = 0.304 overall p = 0.016
resultsSolutions
• Report exactly what test you used
• Improve clarity by describing simple tests in
figure & table legends
• Report test statistic, degrees of freedom,
exact p-value
• Use Statcheck: http://statcheck.io
• Deposit code: Make your analyses
reproducible
Clear reporting allows you to identify &
correct errors prior to publication!STAKEHOLDER ENGAGEMENT
Patient Engagement
in biomedical research:
From research subjects to partners in research
Miravittles M et al. (2013), Respiratory Medicine 107, 1977-1985
Engaging patients as
partners and not just
subjects in research
can improve research!
They hold important
experience-based
expertise from living
with diseases.Patient Engagement
in biomedical research:
Key areas for patient engagement
Geissler J et al. (2017), Therapeutic Innovation & Regulatory Science 51(5), 612-619.
Key areas
Research phases
Key areasPatient Engagement
in biomedical research:
Important resources
Possible gatekeeper to
Best Practice Examples
identify patients…
…although so far no
institution appointed.Patient Engagement in biomedical research: Who else should be involved? Other stakeholders whose involvement might benefit research: • Political actors involved in decision-making about medical products, e.g. G-BA, IQWiG, BfArM, PEI? • Industry actors from the pharmaceutical and biotech world? • Clinicians using the research outcomes? • Training institutions teaching the findings? • …
Patient Engagement in biomedical research: Further open questions • Who should represent (and why) particular stakeholder groups? • Do representatives need particular training? • How should engagement activities be structured to ensure a level playing field? • What conflicts of interest exist and how can they be managed? • Is there a need for a coordinating institution for patient engagement? • …
You can also read
