Eteam Project: Between-user reliability exercise - J Lamb, K Galea, B Miller, L MacCalman, M van Tongeren, G Hazelwood and S Rashid
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
eteam Project:
Between-user reliability
exercise
J Lamb, K Galea, B Miller, L MacCalman,
M van Tongeren, G Hazelwood and S Rashid
INSTITUTE OF OCCUPATIONAL MEDICINE www.iom-world.org
Overview Background and aims Format Coverage Results Main sources of variation in tools Conclusions Recommendations
Aim: Examine how consistent tool users are in
making choices in comparison with other users
User 2 + BURE: Large
exposure scale remote-
situation completion
exercise (~
User 1 + 1 User 3 + 150
participants)
exposure exposure Evaluation
situation situation of Between-
1 1 User
Reliability
Same In-person
tool workshop for
more detailed
estimate feedback (~20
? participants)
Confidence in a tool’s predictions requires
confidence in its reliability
BURE Format
Collect tool estimates from multiple users for a selection
of common exposure situations
6 tools: participants asked to generate inhalation &
dermal estimates for each tool- situation combination
Simple guides on tool installation and use
Standard worksheets used to collect results
Background questionnaire
Final feedback questionnaire
Exposure situations
• 20 varied workplace • Information provided on
situations: inhalation +/-
Vapour pressure
dermal exposure potential
Molecular weight
• Standard 1 page A4 format
CAS number
• Textual description of
typical workplace exposure • Variable information on
settings other exposure
determinants e.g. RMMs,
• Professional & industrial task duration, environment
settings
• Powders, liquids and fumes
Situation 4:Situation
Use of Xylene
4: Use ofinXylene
Formulations
in Formulations
- Mixing
- Mixing
of chemicals
of chemicals
in an
in an
Open
Open
Vessel
Vessel
Please assess
Please
inhalation
assess inhalation
and dermaland exposure
dermal exposure
to xylene
to xylene
in theinsituation
the situation
described
described
below.
below.
When entering
When data
entering
into data
the into
toolstheduring
tools during
the exercise,
the exercise,
please
please
useusethethe
CAS
CASnumber,
number,
molecular weight
molecular
andweight
vapour
andpressure
vapour pressure
value (which
value (which
is forispure
for pure
xylene
xylene
(mixed
(mixed
isomers))
isomers))
given in thegiven
tableinbelow.
the table below.
1. General
1. Description
General Description
of Exposure
of Exposure
Situation
Situation
This situation
Thisinvolves
situationindustrial
involves industrial
mixing ofmixing
liquidof chemicals,
liquid chemicals,
including
including
xylene.
xylene.The
Theoperator
operator
stands on astands
platformon aabove
platform
theabove
vessel
thetovessel
mix the
to mixraw
thematerials
raw materials
for the
for the
process,
process,
which
whichtakes
takes
place in Work
place
Area
in Work
D. Area D.
The mixed product
The mixed (Product
product D)
(Product
contains
D) contains
60% xylene
60% xylene
(mixed(mixed
isomers).
isomers).
Product
Product
DD is is
mixed
mixedinin
50 litre batches.
50 litre batches.
The processThe
takes
process
place
takes
at room
place temperature (20 oC).
at room temperature (20 oC).
There are fixed
Therecapture
are fixedhoods
capture
above
hoodsthe
above
mixing
the mixing
processprocess
and adequate
and adequate
general
general
ventilation.
ventilation.
The activityThe
takes
activity
place takes
for 5
place
hours
for per
5 hours
8 hour
per 8shift.
hour shift.
There is noThere
personal
is no protective
personal protective
equipmentequipment
and noand
respiratory
no respiratory
protective
protective
equipment
equipmentworn
worn
during the activity.
during the activity.
2. Product/
2. Substance
Product/ Substance
Information
Information
Molecular
Molecular Vapour
Vapour Concentration
Concentration
SubstanceSubstance CAS CAS
Product Product
Supplier Supplier Weight/
Weight/pressure
pressure
at at ofofXylene
Xylene in
Name Name Number
Number -1 -1
gmolgmol 20o20
C/oC/
PaPa Product
ProductDD (%)
(%)
Xylene Xylene
Product D Product
Supplier
D DSupplier (mixed
D (mixed
1330-20-7
1330-20-7 106 106 1200
1200 60
60
isomers)isomers)
Results: BURE participant population
Sector Experience of tools
• majority consultancy/ • Most experience of ECETOC
industry (57%) TRAv2/v3, then
Stoffenmanager
Location
• mainly EU (84%) Exposure assessment
experience
• even split across all
Main reason for carrying out
categories (~20% each
exposure assessments
category)
• REACH exposure
assessment (40%)
English language ability
• majority self-assessed as
native/excellent/good
Final dataset
Number of estimates used in analyses
800
700
600
500
Inhalation
estimates
400
300 Dermal
estimates
200
100
0
Assessor-related variation/ total variation-
all situations
Ratio
Tool N Varassessor Varresidual VarTotal (97.5%ile:
2.5%ile)
Inhalation exposure
ECETOC TRAv3 (mg/m3) 350 0.09 2.53 2.63 577
ECETOC TRAv2 (mg/m3) 405 0.28 1.91 2.19 331
MEASE (mg/m3) 398 0.35 6.07 6.43 20746
EMKG-EXPO-TOOL
397 0.28 3.72 4.00 2540
(mg/m3)
STOFFENMANAGER
309 0.60 1.59 2.20 335
(mg/m3)
Dermal exposure
ECETOC TRAv3
350 0.47 1.59 2.06 278
(mg/kg/day)
ECETOC TRAv2
405 0.18 1.12 1.31 90
(mg/kg/day)
MEASE (mg) 398 0.78 3.69 4.47 3975
RISKOFDERM (hands) (mg) 742 0.55 6.11 6.66 24744
Assessor-related variation/ total variation-
applicable situations only
Ratio
Tool N Varassessor Varresidual VarTotal (97.5%ile:
2.5%ile)
Inhalation exposure
ECETOC TRAv3 (mg/m3) 326Variation related to participants’
characteristics
No obvious or consistent Regulators are not obviously
trends observed conservative, industry not
obviously optimistic
More experience in English language ability
assessing exposure does may have some small effect
not seem to reduce the for MEASE, however not
amount of variation consistent
People who do more REACh
assessments are no more
consistent than othersParticipants generally had less
experience of:
Professional/ end use vs
industrial situations
Solids vs liquids
Inhalation vs dermalHow uncertain were participants
when choosing inputs?
Level (%) of uncertainty experienced in choosing
input parameters- inhalation
100
90
80
70
60
50
40
30
20
10
0
Substance Operational Task/ activity Risk
characteristics conditions management
measures
no uncertainty minor uncertainty major uncertainty missingHow easy did participants find translation of
situations into tool inputs?
Ease of Translation into Tool Parameters (%)
Inhalation
Tool Neither easy
Very easy/ Difficult/
nor Missing
easy very difficult
difficult
ECETOC TRAv2 49 16 6 28
ECETOC TRAv3 36 23 10 30
EMKG-EXPO-TOOL 45 18 8 29
MEASE 42 21 9 28
STOFFENMANAGER 27 29 14 30Less Differences
variability for in variability
industrial between
settings than physical
professional? forms?
More More
contextual variability
information/ when
less outside of
variation? Detailed tool scope?
look at
situationsSituation 7:
Changing of filters in paint spray booth
Inhalation estimates Dermal estimatesSituation 11:
Small scale weighing of amoxicillin powder
102
101
In h a la tio n E s tim a te (m g /m 3 )
100
10-1
10-2
10-3
10-4
TRAv2 TRAv3 MEASE EMKG SM90
Inhalation estimates Dermal estimatesSituation 15: Packing of nickel powder Inhalation estimates Dermal estimates
Common Sources of variation
Choice of PROC code/ Risk management
handling description measures
• Assessing main process or • Variety within situation
subtask?
Choice of industrial vs
professional
Dustiness
• Participants and delegates
• Intrinsic dustiness or linked to seemed to struggle with this
energy in process
• No consistent determining
• Difficult to assess non-visually factor
Duration of activity
• “borderline” timesSources of variation:
ECETOC TRAv3 & ECETOC TRAv2
Choice of PROC code Choice of industrial
vs professional
• Assessing main process or
subtask?
Duration of activity
• Handling/ transfer activities:
8a/8b/9 (TRAv3) and PROC Dustiness
1/4/8a for liquid transfer (TRAv2)
• Provision of guidance/ examples
mentioned at workshop
• Inconsistency of PROC choice
between two tool versions for
same situationSources of variation: EMKG-EXPO-Tool
General Scale of use
• Fewer inputs but still large • Limited information given so
amount of variation judgement required- higher
levels of variation
• No in-tool task parameter so
more uncertainty when trying Dustiness
to translate activity
• More uncertainty re liquid Control approach
characteristics than other
tools: absence of concentration
factor
Volatility
• some discrepancies in
allocation- reading ability!Sources of variation: MEASE
Physical form Contact level
• Majority of situations outside • Difficulty in separating out
scope of tool frequency of contact from
intrinsic dustiness
• Aqueous solution instead of
liquid- in BURE more variation
if chosen erroneously Pattern of exposure
• Dustiness control
• Awareness/ use of tool
guidance?
PROC code
• Large variation in choice Scale of operation
• Awareness of tool
metal/inorganic-specific Duration of activity
options / glossarySource of variation: Stoffenmanager
Activity description Control measures
• Choice between similar
descriptions: e.g.
agreement that small
quantities but which Dustiness
handling type? • Similar reasons for variation
to other tools
• More choices but balanced
Room size by use of descriptions?
• Little information given
therefore judgement
needed - some variation
notedSources of variation- RISKOFDERM
DEO- task Contact
• Less variation than other • Observed variation in level
tools- more delineation of contact
between categories • Difficulty in separation of
likelihood of contact from
Application rate frequency?
Cumulative duration Dustiness
• Information given but
interpreted very differently
Estimates much higher
than for other dermal
toolsOther sources of variation
Erroneous choices Lack of awareness of
• physical form of molten metals tool guidance
• dermal exposure situations • Tendency to use basic use
instructions provided rather
than actual tool information
Differences in Typographical/
interpretation/ mis-reading transcription errors
of information
• Inclusion/ exclusion of described
risk management measuresLimitations of BURE
Recruitment may not have reached typical tool
users
Self-selection regarding English language
Different to iteration process used under REACh
• Workplace specific situations used vs sector generic scenarios
• Assessment outputs are the estimate and the tool parameter
choicesConclusions
Most variation between users is not obviously attributable to
their personal characteristics
Ease of translation and level of uncertainty are not predictors
of level of variation
Perceived level of uncertainty greater for dermal assessments
and for solids- general levels of experience of these tasks?
Participants, on occasion, conflate determinants when
allocating inputs which may affect variation and validity of the
estimate
Assessment of overall process type rather than described
exposure-prone taskConclusions (2)
Professional situations gave rise to more variation in
estimates- lower familiarity with these activities?
Allocation of level of dustiness seems to be challenging and
variable
For all tools, the choice of task/ activity for a given situation
showed great variation between people who were assessing
the same, reasonably well-described exposure settings
Similar findings in reliability studies for other assessment tools
Overall, the exercise suggests that between user variation in
interpretation of exposure determinants could be an important
issue for the standardisation of REACh processesRecommendations……
Development of
Tool guidance regular
and help evaluation
functions need programme to
to be read by help users refine
users their assessment
performance
Quality control
Group
assessments →
assessing More consistent
separately then assessments?
reaching
consensus may
reduce errors?Acknowledgements • eteam Project Advisory Board • Marlyn Davis, IOM • BURE and workshop participants
You can also read
