Validating AMA frameworks - A Regulator's Experience in Japan
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Validating AMA frameworks
- A Regulator’s Experience in Japan
2nd International Conference on Operational Risk
Sao Paulo, Brazil, June 5, 2009
Tsuyoshi Nagafuji
Financial Services Agency, Japan
This presentation does not necessarily express
established views or policies of the FSA.Overview : Objective
Sharing my experience in validating and approving Japanese
banks’ AMA applications.
¾ Presenting what we have done or what we are actually doing in
Japan, rather than what we hope to do.
¾ Focusing on the factors that remain until the final stage for
application, which banks find difficult and time consuming to
address.
9 Model – sensitivity analysis / stress testing: Do you know all
the possibilities for strange behavior?
9 Scenarios – rules and documentation: Have you done your
best to exclude subjectivity?
9 Use test: Are you actually using the framework? Is it really
working?
2Overview : Method
I am using a realistic “model of AMA models" as an example, in
consideration of anonymity.
¾ I am using “The model of AMA models” that I presented at the
“Operational Risk Scenario Analysis Workshop” held at Bank of
Japan, the central bank, in 2006*.
* The model presented here is the same as the one I presented in 2006, but the description
is simplified. Please see “Quantification of Operational Risk Using Scenario Data (Nagafuji,
2006)” for the details.
¾ The model is extremely simplified but still retains some
aspects of typical AMA models used by Japanese banks.
9 The model is based on real internal data and real scenario
data from major Japanese banks.
9 The model has a similar structure to typical Japanese
models.
3Overview : Outline
Presentation Overview (5 minutes)
1. Context (5 minutes)
2. Sample Model (10 minutes)
3. Validation of the Sample Model (15 minutes)
Concluding Remarks (5 minutes)
Q & A (5 minutes)
(Total: 45 minutes)
4Overview
1. Context
2. Sample Model
3. Validation
Concluding Remarks
Appendix
5Context: Japanese Banking Industry
Consists of three "Mega” banks and many smaller banks.
Foreign banks play a very small role.
World banks by Tier I capital ($billion) (Source: The Banker, July 2008)
MUFG
120 ( $82 bil)
100
Mizuho
6 ($49 bil)
80
SMFG
60
15 ($44
18
40
28
38
20 48 50 66
… … … … … …81 …89…94
0
u
G
co
R via
S
SCC
C lays
JP S
co
IN it
BOM
A
o
H
R iti
C
C
BB
IC g
H BC
rc as
Ita
d
ab
B
tA
BO
B
PaBO
C
SB
an
re
es
Ba rib
ho
C
(Now merged)
di
C
b
H
ad
ac
re
ni
ni
Br
W
U
U
C
Top 20 Banks Top 21 – 100
98 banks in the Top 1,000 list. 14 banks in the Top 1,000 list.
6Context: Op Risk in Japanese Banks
Operational risk losses are extremely small.
Total annual loss Loss frequencies
amount (# of losses greater or equal to
(Average Dollar Amount by year, $20,000, per year, per total assets of
percentages of total assets) $1 billion)
US banks US banks
Japanese Japanese About 1/20
banks About 1/40 banks
0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0 0.5 1 1.5 2
* Both figures are medians of the banks that participated in the exercise
(Source) 2004 U.S. LDCE, 2007 Japan LDCE (See “Appendix: References about
Japanese AMA implementation" for detail).
7Context: Application Timetable
■ Typical validation process
1. Preparation
1. Preparation
Banks are encouraged to develop their framework
to a practical level and use it for their internal
Ready? purposes before going into the parallel run.
2. Parallel Run (At least one year)
2. Parallel Run Two capital calculations are verified through visits
and regular discussions.
OK? 3. Approval
Banks that do not meet the requirement stay at
stage 2 or go back to stage 1.
3. Approval
■ Currently one banking group has been approved for the AMA.
8Overview
1. Context
2. Sample Model
3. Validation
Concluding Remarks
Appendix
9Sample Model: Overview
Several major banks are treated as if they were a single big bank.
9 Their internal loss data and scenario data are put into a very
simple loss distribution model (LDA).
Bank A
Internal Loss
Data
Bank B
LDA Model Results
Bank C
Scenario Data
Bank X
10Sample Model: Quantification Model
■ Monte Carlo simulation (100,000 simulations)
¾ Frequency: Poisson distribution
λ = Frequency of Scenarios
+ Frequency of Real Loss Data
¾ Severity: Empirical distribution
■ Single unit of measure (= top of the house calculations)
11Sample Model: Internal Loss Data
Banks’ internal loss data are used as if they were from one big
bank.
Bank A
Internal Loss
Data
Bank B
LDA Model Results
Bank C
Scenario Data
Bank X
Each data point is assumed to have a frequency of once in
10 years, because the observation period is 10 years
12Sample Model: Scenarios (1/4)
■ Each bank’s scenarios for their quantification are used.
1) Independent scenarios:
Scenarios that hit each bank independently
Æ Each bank’s scenarios are put into the model as they are.
2) Common scenarios:
Scenarios that could hit all banks at the same time: earthquakes and inter-bank
settlement system failures.
Æ Each bank’s scenarios are aggregated to a single scenario and then put into the
model.
13Sample Model: Scenarios (2/4)
■ Common Scenarios 1: Earthquakes
Losses by historical earthquakes are estimated for each bank and then aggregated.
Description (year , Frequency Severity Details
magnitude of earthquake) (once in X (largest
years) =100)
Earthquake in Tokyo 1,200 100 Earthquake greater than any of those below is assumed.
Keian (1649, 7.1) 49 (Frequency) 8 large-scale earthquakes between 1600 and
Genroku (1703, 8.2) 85 1925 in Tokyo, Nagoya and Osaka are listed, assuming each
Ansei Edo (1855, 6.9) 55 will occur once every 400 years.
Meiji Tokyo (1894, 7.0) 400 each 47 (Severity)
Great Kanto (1923, 7.9) 82 The damage to the building, furniture and the
Hoei (1707, 8.4) 57 opportunity cost due to interruption of business are
Ansei (1854, 8.4) 50 calculated based on the earthquake intensity and quake
Nobi (1881, 8.0) 55 resistance of the buildings.
Extra work cost, damage to the machines and
equipment and the opportunity cost due to business
interruption are calculated. Damage to the computer center
and paralysis of the head office functions are assumed.
Declines in the value of the loans (including
impairment of the value of collateral) are not factored in.
Tokyo (1926) - Aichi Average (Frequency) 61 earthquakes occurred between 1926 and 97
(1997) (61 earthquakes) 77 each 0.4 (of intensity 5- or higher) are listed, assuming each will occur
once every 77 years.
(Severity) as shown above.
14Sample Model: Scenarios (3/4)
■ Common Scenario 2: Failure in the settlement system
A scenario was created where “a failure occurs in the computer systems commonly
used by the banks once every twenty years, causing total damage of JPY 20 billion.”
The following scenarios from a bank were referred to in creating this scenario.
Frequency Severity Details
Once in JPY several billions A failure in the accounting system or in the domestic
several ($US tens of millions) network, which would take 12 hours for full recovery.
decades
Once in JPY several 1) A failure occurs in the communication infrastructure, or, 2)
several hundred millions there is a flaw in the emergency handling procedures,
decades ($US several millions) causing interruption of the settlement operation for half a
day. The compensation for damage paid to securities
exchanges as clearing agents in charge of settlement of the
government bonds is included.
Once in JPY several billions Foreign exchange / settlement operations are not performed
several ($US tens of millions) for a full day due to a system failure
decades
Once every JPY several A failure occurs in the Zengin System just after 9:00 am. The
several years hundred millions system recovers at around noon. However, the settlement
($US several millions) operation is erratic during that day.
15Sample Model: Scenarios (4/4)
■ Independent Scenarios
Actual scenarios collected from banks are used as they were.
BIS event Major scenarios (scenarios for larger amounts of losses)
types # of scenarios Examples
Internal Fraud 30 Fraud in the market trading functions, withdrawal of customer funds
External Fraud 3 Swindles, compromised online banking
Employment 5 Discrimination
Clients, 30 Lender’s liability, inappropriate advice to customers, failure to
Products explain the risks, etc
Physical 11 Terrorist attacks
assets
Systems 12 Failure in the accounts transfer system, including interruption of the
accounting system
Process 38 Failure in bond settlement (overseas), improper identity verification,
error in cash transfer, etc
Total 129
Made-up scenarios for banks that did not have scenarios are also used.
→ Some scenarios from scaled by the total assets of each bank.
16Sample Model: Results
Risks at the confidence levels of 99% and 99.9% are quantified.
Æ The risk at a confidence level of 99.9% is about one third of the Basic Indicator
Approach (BIA) amount.
99% 99.9% EL (BIA)
23% 34% 7% (100%)
Big scenarios, especially earthquakes, contribute much to the
results.
17Sample Model: Recap
The model retains some aspects of AMA models used by
Japanese banks.
Sample Model
Internal loss data and
External loss data scenarios are directly
and BEICFs Internal Losses input into the model.
inform scenarios
Model Risk Amount
External
Loss Data
9 Scenarios
are estimated 9 Number of cells is
Scenarios as individual relatively small.
BEICFs* data points. 9 “Empirical
9 Number of distribution” is used
scenarios are by some banks.
large.
*Business Environment and Internal Control Factors
18Overview
1. Context
2. Sample Model
3. Validation
Concluding Remarks
Appendix
19Validation of the Sample Model: Overview
Three major points that banks have trouble with in completing
their AMA application.
(1) Model: Sensitivity analysis / stress testing
(2) Scenarios: Rules and documentations Sensitivity analysis/
(3) Use test: Are you actually using the framework? stress testing: Do you
know all the possibilities
for strange behavior?
Internal Losses (1)
Model Risk Amount
External Loss (2)
Data
Rules and documentations:
Scenarios
Scenarios Have you done your best to
exclude subjectivity?
BEICFs
Are you actually using
the framework? Is it
(3) Use test really working?
20Validation: Model (1/3)
Statistical integrity of the model is essential, but has not been a
determining factor at the last stage.
9 LDA model
Independence between frequency and severity should be accounted for.
9 Choice of distributions and estimation methods
Those may not be great discussion points as the sample model uses empirical
distribution.
9 Granularity
Independence among the data points should be accounted for.
Æ As long as assumptions in the model are clarified and accounted
for, this factor is not a decisive one.
21Validation: Model (2/3)
Instead, sensitivity analysis / stress testing has been a great
challenge at the last stage.
9 No model is free from “strange” (counter-intuitive) behavior.
9 Comprehending all the possibilities of “strange” behavior is
time consuming, especially when the model is complex.
Massive losses have a large impact.
A small change in frequency for a massive loss may have a large impact.
(Frequency: Poisson, Severity: Empirical)
Data set 1) and data set 2) give completely different risk, although the only
difference is the frequency of a single big loss!
Losses EL 99.9％
1) One JPY100 billion Loss (Once in 999 years) 0.1 100
3 / 1,000,000
+ 100 JPY 10,000 Losses (Each once in 10 years) billion billion
2) One JPY100 billion Loss (Once in 1000 years) 0.1 0.0003
+ 100 JPY 10,000 Losses (Each once in 10 years) billion billion
22Validation: Model (3/3)
Comprehending unusual (counter-intuitive) behavior of a model
is essential for regulatory and internal purposes.
We request banks:
9 To comprehend possible “strange” behaviors of their model.
9 To address those “strange” behaviors.
Accept them (Management should fully understand the
consequences).
Revise and reconstruct their models.
Æ May take a lot of time.
23Validation: Scenarios (1/3)
Estimation of frequency and severity is essential, but not a
determining factor at the last stage.
Æ Scenarios for the sample model must include “once in 1000
years” events, as the model uses empirical distribution for
severity.
Æ We verify this through
9 Checking the logic.
9 Checking facts that scenarios are based on.
9 Benchmarking scenarios between banks.
Earthquakes:
Is the use of past earthquakes appropriate?
What is included as losses from earthquakes?
Is the latest seismological knowledge utilized?
System failures:
Are the statistics on computer failures utilized?
What is the accuracy of the statistics?
Are the statistics used appropriately?
Æ As it is impossible and inappropriate to press one specific
view, this factor has not been a decisive one at the last stage.
24Validation: Scenarios (2/3)
Instead, setting rules in making scenarios and making thorough
documentation has played a determining role at the last stage.
(A hypothetical example)
External events that The bank sets up a
could happen to the rule in judging
bank are made into whether a particular
External scenarios. event can happen
Loss Data to it.
Scenarios
BEICFs
The bank sets the frequency The bank sets up a
of scenarios based on the rule in judging
RCSA scores (ex. How whether a particular
effective are controls?). event can happen
to it.
25Validation: Scenarios (3/3)
We request banks to:
9 Do their best to set rules that ensure the same estimate
regardless of who the estimator is.
9 Fully document and account for subjective judgments that
remain.
This looks easy to accomplish, but turns out to be very
challenging, because:
9 Rules can be set only after experience is accumulated.
9 This is often neglected until the last stage.
26Validation: Use test (1/3)
“Use test” is a strong tool to improve an AMA framework. Thus, it
is very challenging and often becomes a determining factor at the
last stage.
Our “use test” in Japan is not special.
¾ Banks should show that they use their AMA framework in their
day-to-day risk management (= The framework is not
exclusively for regulatory purposes).
¾ Use test is based on the idea that supervisors can be more
confident with an AMA framework that is “really used”.
27Validation: Use test (2/3)
Banks decide how to demonstrate their compliance with the use
test (= Banks decide how to use their AMA framework).
Many Japanese banks choose to base their “risk management
cycle” on their AMA model.
Evaluate risk Implement
reduction risk reduction
measures based measures
on the model. １.Plan 2. Do based on the
model.
Model/
Risk reduction measures
・ Introducing double Risk
checking
・ Computerize operations
・ Restricting operations 3. See
・…
Verify the
results using
the model.
28Validation: Use test (3/3)
“Use test” imposes improvement on their overall AMA framework.
¾ It imposes improvement on the model.
9 The model needs to be practically free from “counter-intuitive” behavior
9 The model needs to be sensitive enough.
¾ Understanding by management and business units is essential.
Thus, it often becomes a determining factor at the final stage.
¾ When the AMA framework does not meet the use test
requirement, it needs modification.
¾ When the modification is drastic, banks are required to take the
“use test” again, which needs at least half a year to complete.
29Overview
1. Context
2. Sample Model
3. Validation
Concluding Remarks
Appendix
30Concluding Remarks: In Theory…
■ In theory, capturing 99.9% risk is the single most important
requirement of AMA models.
9 Do distribution assumptions capture 99.9% risk?
9 Are scenarios representing once-in-1000-year events?
■ However, this requirement does not turn out to be the remaining
factor at the last stage.
9 Choice of distribution or estimation of scenarios boils down to
subjective judgments, which are argumentative, but not
decisive.
9 Banks that cannot address these issues cannot enter the
parallel run.
31Concluding Remarks: In Practice…
■ The following practical factors have often played a decisive role
at the last stage.
1. Comprehending unusual (counter-intuitive) behavior of the
model and preparing for it.
2. Setting rules and perfecting documentation to minimize the
subjectivity of scenarios.
3. Meeting the “Use test” requirement
■ Those factors ensure workable framework both for internal and
regulatory purposes.
32Concluding Remarks: Challenges for Banks
■ We do not press banks to have an ideal or very sophisticated
framework. Rather, we ask them to have a practical, reliable
framework to meet requirements for regulatory purposes.
■ After all, it is up to banks to build an AMA framework that truly
enhances their risk management.
33ご清聴ありがとうございました
Questions?
For further questions, feel free to contact:
34Appendix: References about Japanese AMA implementation
Sample model used in this presentation
9 Quantification of Operational Risk Using Scenario Data (July, 2006)
http://www.boj.or.jp/en/type/release/zuiji_new/data/fsc0608be3.pdf
Losses of Japanese banks (Page 7 of this presentation)
9 Results of the 2007 Operational Risk Data Collection Exercise (August, 2007)
http://www.fsa.go.jp/en/news/2007/20070810-2.pdf
9 A research paper comparing the loss data between the U.S. and Japan (April, 2008)
http://www.boj.or.jp/en/type/release/adhoc/data/risk0804a.pdf
(For the results of U.S. LDCE, see http://www.bos.frb.org/bankinfo/qau/papers/pd051205.pdf)
Others
9 Use of External Data for Operational Risk Management Workshop (April, 2008)
http://www.boj.or.jp/en/type/release/adhoc/fsc0804a.htm
9 The Effect of the Choice of the Loss Severity Distribution and the Parameter Estimation Method
on Operational Risk Measurement (December, 2007)
http://www.boj.or.jp/en/type/ronbun/ron/research07/ron0712c.htm
9 Discussions on Further Advancing Operational Risk Management (Part1: June 2006, Part2:
August 2006)
Part1: http://www.boj.or.jp/en/type/release/zuiji_new/fsc0608c.pdf
Part2: http://www.boj.or.jp/en/type/release/zuiji_new/fsc0612a.pdf
9 Operational Risk Scenario Analysis Workshop (July 2006)
http://www.boj.or.jp/en/type/release/zuiji_new/fsc0608a_add.htm
35You can also read
