"Machine-Learning Predictive models for the follow-up management in spine surgery" - DataREG
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
“Machine-Learning Predictive models for the
follow-up management in spine surgery”
Dott. Ing. Linda Dui Dott. Ing. Federico Cabitza, PhD
Data Scientist, DataReg IRCCS Istituto Ortopedico Galeazzi
Università degli Studi di Milano-Bicocca
Pedro Berjano, MD
Spine 4 Unit
IRCCS Istituto Ortopedico Galeazzi
➢ Datareg @ Istituto Ortopedico Galeazzi (up-to-date 15 Nov 2017) ➢ 22 teams ➢ 3 specialties: spine surgery, hip&knee prothesics, ankle-foot prothesics. ➢ Medical users: 95 (spine) + 112 (others) ➢ Patient 0: 11 November 2015 ➢ Enrolled patients to date: 2,173 (spine) + 1,432 (others) ➢ Enrolled patients yearly: ~3500 (long term) ➢ PROMS only Questionnaires : 16,096 (spine) + 13,636 (others) ➢ PROMs items: 347,568 (spine) 53,051 (others) ➢ PROMS only Questionnaires yearly (long term): 12-15 k
Questionnaires compilation rate (15/11/2017)
100%
1438
90%
80%
5272
70%
5377 4231
60% 1877
50%
9562
40%
30%
4064
20%
2715 1941
10% 529
0%
Pre-op 3 moths 6 moths 12 moths 24 moths
filled not filled
Questionnaires compilation rate (tomorrow)
100%
1438
90%
80%
5272
70%
5377 4231
60% 1877
50%
9562
40%
30%
4064
20%
2715 1941
10% 529
0%
Pre-op 3 moths 6 moths 12 moths 24 moths
filled not filled
RESPONSE PREDICTIONS
SPINE :: FOLLOW-UP E-MAILS REPLY
RESPONSE PREDICTIONS
SPINE :: FOLLOW-UP E-MAILS REPLY
Prediction of the response to e-mail alerts for
PROMs compilation in follow-up
Selected features: 6. Month
7. Month day
1. Gender
8. Year day
2. Age
9. Week day
3. Step
4. Protocol
5. Year
Target variable:
➢ Form online filled in or not
RESPONSE PREDICTIONS
SPINE :: FOLLOW-UP E-MAILS REPLY
Prediction of the response to e-mail alerts for
PROMs compilation in follow-up
All categorical (nominal) variables have Selected features: 6. Month
7. Month day
been binarized. 1. Gender
8. Year day
2. Age
9. Week day
3. Step
4. Protocol
5. Year
Target variable:
➢ Form online filled in or not
RESPONSE PREDICTIONS
We show the PCA representation of the dataset.
Replied (1)
Didn’t reply (0)
RESPONSE PREDICTIONS
The most accurate model for this problem is Random Forest (trained with 10-
fold cross validation)
SPINE :: FOLLOW-UP E-MAILS REPLY
Prediction of the response to e-mail alerts for
PROMs compilation in follow-up
Selected features: 6. Month
7. Month day
1. Gender
8. Year day
2. Age
9. Week day
3. Step
4. Protocol
5. Year
Target variable:
➢ Form online filled in or notRESPONSE PREDICTIONS
The most accurate model for this problem is Random Forest (trained with 10-
fold cross validation)
SPINE :: FOLLOW-UP E-MAILS REPLY
Prediction of the response to e-mail alerts for
PROMs compilation in follow-up
Selected features: 6. Month
7. Month day
1. Gender
8. Year day
2. Age
9. Week day
3. Step
4. Protocol
Training Test 5. Year
Target variable:
Reply 4307 (54%) 1077 ➢ Form online filled in or not
No reply 3612 (46%) 903
Total 7919 1980RESPONSE PREDICTIONS
The most accurate model for this problem is Random Forest (trained with 10-
fold cross validation)
.99 SPINE :: FOLLOW-UP E-MAILS REPLY
Prediction of the response to e-mail alerts for
PROMs compilation in follow-up
Selected features: 6. Month
7. Month day
1. Gender
8. Year day
2. Age
9. Week day
3. Step
4. Protocol
5. Year
Target variable:
➢ Form online filled in or not
Accuracy: 93.1%
TP rate: 93.1%
FP rate: 8.1%
Precision: 93.6%
Recall: 93.1%
F-score: 93.0% (on test set)Surgery at the IRCCS IOG We stratify our patients in those who, 3 months after the operation, claim to have got better and those who assert to have got worse. How to make the concept of improvement measurable, ie how to appraise what the patient values more?
1st example: Herniated Disk Surgery
The Oswestry Disability Index
The Oswestry Disability Index
expresses annoyance (in terms
of disability and quality of life)
of patients suffering from back
pain. The lower the index the
better. The treatment efficacy,
and hence the improvement, is
reflected by a reduction of this
index over time.1st example: Herniated Disk Surgery
The Oswestry Disability Index
The Oswestry Disability Index
expresses annoyance (in terms
of disability and quality of life)
of patients suffering from back
pain. The lower the index the
better. The treatment efficacy,
and hence the improvement, is
reflected by a reduction of this
index over time.
The surgeon proposed a MCSD of
10%.
On the ODI this means 10 points.1st example: Herniated Disk Surgery
The Oswestry Disability Index
The Oswestry Disability Index
expresses annoyance (in terms
of disability and quality of life)
of patients suffering from back
pain. The lower the index the
better. The treatment efficacy,
and hence the improvement, is
reflected by a reduction of this
index over time.
The surgeon proposed a MCSD of
10%.
On the ODI this means 10 points.
Such a MCSD correponds to an
effect size of ~0.47 (Cohen d), hence
we need ~143 patients to detect it
(beta=.8, alpha=.05)1st example: Herniated Disk Surgery
The Oswestry Disability Index
The Oswestry Disability Index
expresses annoyance (in terms
of disability and quality of life)
COMI*: Overall, of patients suffering from back
how much did the
pain. The lower the index the
operation in your
hospital help your
better. The treatment efficacy,
back problem? and hence the improvement, is
reflected by a reduction of this
index over time.
✓ Did not help
✓ Made things worse
✓ Helped a lot
✓ Helped
*: CORE OUTCOME MEASURES INDEX1st example: Herniated Disk Surgery
The Oswestry Disability Index
The Oswestry Disability Index
expresses annoyance (in terms
of disability and quality of life)
of patients suffering from back
pain. The lower the index the
better. The treatment efficacy,
and hence the improvement,
reflects in a reduction of this
index over time.
The average ODI reduction (delta score) for
those claiming to have got better (N=156) is
The average ODI the
-14.8, while reduction
average (delta score) for for
ODI reduction
thosethose
claiming to have
claiming got better
to have (N=109)
got worse is was
(N=8)
-12.54, while the average ODI reduction for
+8.1.
thoseThe
claiming to have
difference btwgot worse
these mean(N=22)
values is
was -1.59.
statistically significant (p = 0.015).
The difference btw these mean values is
statistically significant (p = 0.046).1st example: Herniated Disk Surgery
The Oswestry Disability Index
The Oswestry Disability Index
expresses annoyance (in terms
of disability and quality of life)
of patients suffering from back
pain. The lower the index the
better. The treatment efficacy,
and hence the improvement,
reflects in a reduction of this
index over time.
We found a Minimal clinically Important
Difference (MID) threshold, which is anchor
based and distribution based. This basically
confirms the clinical knowledge of the
surgeon.
This threshold can be considered the upper
bound of the CI of the mean improvement:
for this pathology: 11.5 on the ODI.1st example: Herniated Disk Surgery
The Oswestry Disability Index
The Oswestry Disability Index
expresses annoyance (in terms
of disability and quality of life)
of patients suffering from back
pain. The lower the index the
better. The treatment efficacy,
and hence the improvement,
reflects in a reduction of this
index over time.
We found a Minimal clinically Important
Difference (MID) threshold, which is anchor
based and distribution based. This basically
confirms the clinical knowledge of the
surgeon.
This threshold can be considered the upper
bound of the CI of the mean improvement:
for this pathology: 11.5 on the ODI.
Effect Size: 0.55
(Cohen’s d)IMPROVEMENT PREDICTIONS
SPINE :: ODI SCORE :: 3 MONTHSIMPROVEMENT PREDICTIONS
SPINE:: ODI SCORE :: 3 MONTHS
Prediction of the ODI score for spine surgery
(1st int.) after 3 months.
Selected features: 11. Impianti
12. Accesso anteriore/posteriore
1. Genere
13. Rischio anestesiologico
2. Età
14. Profilassi
3. BMI
15. Tecnica chirurgica
4. Livello patologia principale
16. Perdita ematica
5. Patologia aggiuntiva
17. Trasfusione ematica
6. Num interventi precedenti alla
18. Tecniche per favorire la fusione
colonna
19. Stabilizzazione rigida/con
7. Precedenti trattamenti
conservazione della mobilità
8. Fumo
20. Tecniche percutanee
9. Flags
10. Scopo intervento
Target variable:
➢ Delta ODI score 0-3 monthsIMPROVEMENT PREDICTIONS
All categorical (nominal) variables have
been binarized. All numerical (scalar)
SPINE:: ODI SCORE :: 3 MONTHS
variables (the ODI pre-op score) have been
Prediction of the ODI score for spine surgery
standardized and normalized. Variables
(1st int.) after 3 months.
with equal answers in 99% cases have
been removed. Feature selection Selected features: conservativo
3. Scopo dell’interrvento:
performed with Wrapper method. 1. Patologia aggiuntiva: malattia
risoluzione del dolore periferico
degenerativa
4. Stabilizzazione rigida: posteriore
2. Precedenti trattamenti per le
5. Pre-op ODI score
patologie principali: >12 mesi
Target variable:
➢ Delta ODI score 0-3 monthsIMPROVEMENT PREDICTIONS
We show the PCA representation of the dataset.
Better (>11)
Worse (≤11)IMPROVEMENT PREDICTIONS
The most accurate model for this problem is Random Forest (trained with 3-fold
cross validation)
SPINE:: ODI SCORE :: 3 MONTHS
Prediction of the ODI score for spine surgery
(1st int.) after 3 months.
Selected features: conservativo
3. Scopo dell’interrvento:
1. Patologia aggiuntiva: malattia
risoluzione del dolore periferico
degenerativa
4. Stabilizzazione rigida: posteriore
2. Precedenti trattamenti per le
5. Pre-op ODI score
patologie principali: >12 mesi
Target variable:
➢ Delta ODI score 0-3 monthsIMPROVEMENT PREDICTIONS
The most accurate model for this problem is Random Forest (trained with 3-fold
cross validation)
SPINE:: ODI SCORE :: 3 MONTHS
Prediction of the ODI score for spine surgery
(1st int.) after 3 months.
Selected features: conservativo
3. Scopo dell’interrvento:
1. Patologia aggiuntiva: malattia
risoluzione del dolore periferico
degenerativa
4. Stabilizzazione rigida: posteriore
2. Precedenti trattamenti per le
5. Pre-op ODI score
patologie principali: >12 mesi
Training Test
Target variable:
Better 26 (58%) 20 ➢ Delta ODI score 0-3 months
Worse 19 (42%) 6
Total 45 26IMPROVEMENT PREDICTIONS
The most accurate model for this problem is Random Forest (trained with 3-fold
cross validation)
.93 SPINE:: ODI SCORE :: 3 MONTHS
Prediction of the ODI score for spine surgery
(1st int.) after 3 months.
Selected features: conservativo
3. Scopo dell’interrvento:
1. Patologia aggiuntiva: malattia
risoluzione del dolore periferico
degenerativa
4. Stabilizzazione rigida: posteriore
2. Precedenti trattamenti per le
5. Pre-op ODI score
patologie principali: >12 mesi
Target variable:
➢ Delta ODI score 0-3 months
Accuracy: 73.1%
TP rate: 73.1%
FP rate: 19.7%
Precision: 82.3%
Recall: 73.1%
F-score: 75.1% (on test set)Today
Tomorrow? 30
These slides are not intended nor produced to be made publicly available. However, I share them to a limited readership (those who possess the Web address where the slides are stored) in an effort of disseminating scientific ideas and reflections, with no intended aim of economic gain, and to get feedback and comments on those ideas, which I expresss under a CC-BY 4.0 license. I took the greatest possible care to identify all image copyright holders correctly. However, if I have omitted to do so in some individual instances, I would be most grateful if these copyright holders would inform me forthwith. It is my policy to immediately remove, upon notification and identification, any specific image displayed on this website for which the copyright holder deems the fair use (see below) cannot be associated. Upon request, I will remove immediately the content and update the content accordingly. FAIR USE NOTICE These slides may contain copyrighted (©) material the use of which has not always been specifically authorized by the copyright owner. Such material is made available to advance understanding of ecological, political, historical, human rights, economic, artistic expression, democracy, scientific, moral, ethical, and social justice issues, etc. It is believed that this constitutes a ’fair use’ of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, this material is distributed without profit to those who have expressed a prior general interest in receiving similar information for research and educational purposes. ** COPYRIGHT NOTICE** In accordance with Title 17 U.S.C. Se
You can also read
