Multi-Manager Mindset Edition 1, 2018 - STANLIB Multi-Manager
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Multi-Manager Mindset
Edition 1, 2018@stanlib
stanlib
stanlibmultimanager.comContents
02
Foreword
03
Understanding performance evaluation
08
Manager evaluation post selection
11
Key factors to consider when
evaluating asset managers
14
The perils of a CPI benchmarkForeword
In the latest edition of Mindset, we focus on performance
measurement and evaluation – asking the how, why and so
what questions that are so important when we build and
manage portfolios.
Performance measurement and evaluation is complex. Amira Abbas writes about the perils of using CPI as a
It seems where there is a pro to one method and benchmark. It may make sense to an investor looking
measure, there is also a con. We need to be aware for a real return, but as Amira explains, using CPI as
of these and use a number of measures to ensure we a benchmark does not tell us much about manager
valuable insight and build portfolios that meet our performance.
clients’ needs.
Enjoy this Mindset and look out for our next 2018
Joao Frasco, our chief investment officer, leads with edition.
an article looking at how to understand performance
Send us your feedback and comments and remember
evaluation. Joao shows why we must use the
to visit our website stanlibmultimanager.com for
dimensions of time, tracking error, probabilities and
regular updates from the team.
expectations when we evaluate performance. He
highlights that it is not just about measurement and
attribution, we also need to appraise performance so De Wet van der Spuy
we know what action to take. Managing Director, STANLIB Multi-Manager
Lubabalo Khenyane dispels the myth that you can
never predict which managers will outperform in the
future in his article. He examines how performance
is evaluated after managers are selected, and how we
need to be aware of our own biases when we monitor
and review performance.
It is not all about the numbers as Sonal Bhagwan
explains in her article on key factors to consider when
evaluating asset managers. She looks at how the six
Ps of philosophy, process, people, price, performance
and the physical environment are analysed along with
the numbers to see if performance is likely to be
repeated.
PAGE 2 STANLIB MULTI-MANAGER | MULTI-MANAGER MINDSETUnderstanding performance
evaluation
By Joao Frasco, Chief Investment Officer, STANLIB Multi-Manager
Performance evaluation is one of those topics that is
conceptually easy to understand, but your understanding
begins falling apart once you get into the detail.
Fortunately, there are great frameworks for thinking Framework
about this “problem”, and great tools for helping with
the exercise. In this article, I’ll tackle the problem from There are many different frameworks for tackling
the perspective of uncertainty, which will be useful performance evaluation, but we will focus on an easy
for everyone in the value chain, all the way from to understand and well recognised framework, which
investors to the asset managers who ultimately make is taught by the CFA Institute in their Certificate in
the security selection and asset allocation decisions. Investment Performance Measurement. Essentially,
performance evaluation consists of three main
The topic is complex so I will begin by providing an components, namely: performance measurement,
introduction on some of the fundamental concepts performance attribution and performance appraisal.
needed to understand how this complexity can be Let’s look at these in a little more detail.
tackled.
Performance measurement is the starting point, and
Why evaluate performance? is concerned with measuring the performance realised.
This may appear to be a relatively simple exercise, but
The purpose of performance evaluation is to it comes with lots of complexity, so let’s unpack this a
understand how something measures up against our little further by asking a couple of related questions,
expectations or goals and objectives. An investor or such as:
adviser or multi-manager may want to understand how
an appointed asset manager has performed relative • Are we measuring returns or risk or something
to its benchmark. Alternatively, an investor may want else, such as costs?
to understand how an adviser has performed relative • Are we measuring a client account, or a fund, or a
to other advisers. composite of funds or accounts?
There are many reasons for doing performance • Are we measuring returns gross or net of fees and
evaluation, but if we focus on the objective of costs?
understanding performance, we realise that the
purpose is ultimately to get actionable information. • Over what period of time are we measuring, or
That information could result in the hiring or firing are we interested in multiple periods?
of a manager or an adviser. To get to that decision, • What are we measuring performance against – a
however, we need to understand investments benchmark, an objective or peers?
intimately, so that we recognise the limitations of the
exercise, and hence the limitations on decisions we • What return measure are we using, for example,
take. This requires an understanding of the uncertainty time-weighted or money-weighted, and what
inherent in performance evaluation. formula is required?
PAGE 3 STANLIB MULTI-MANAGER | MULTI-MANAGER MINDSETThese are just some of the important questions we should be taken. If performance was bad (or good), and
need to understand before embarking on measuring the attribution points to sources of the returns which
performance. would not have been expected, should a manager be
fired (or hired)? There are many possible implications
Performance attribution is the next step, and looks
for the results of the appraisal, and understanding the
at how the performance observed was derived. Again,
analysis is critical if you are to make great decisions
there are many different ways to “slice and dice” this
from the results.
analysis, but that doesn’t mean that they are all equally
valid. It is important to understand the manager’s (or
adviser’s) investment philosophy and process so that Past performance used as the only
the right analysis can be performed. Not doing so dimension to perform appraisal
may lead to drawing conclusions from faulty analysis.
If your analysis was simply to consider whether a
A quick example will help to explain this. Let’s assume
manager has outperformed an appropriate benchmark
that an asset manager has been appointed to manage
or not, you should expect half of all managers with
a sovereign bond mandate. It would be a grave mistake
“no skill” to outperform over any time period (no
to measure that asset manager’s performance relative
matter how long). So looking at past performance is
to a credit bond benchmark or credit bond peers.
as good as flipping a coin for decision making - that
Performance appraisal is the final step in the is, it is worthless. Table 1 below illustrates this point.
process but the most important. Unfortunately, most
That is why using past performance as the only input
performance evaluation exercises stop before this
in deciding whether an asset manager is skilful or not
step is adequately completed and therefore nothing
is a waste of time, especially if the analysis is done
results from the previous two steps. This step is
using flawed methodologies (which it very often is).
concerned with the decision making element of
performance evaluation, by asking “so what?” What While I could fill a textbook with all the information
can be deduced or inferred from the performance required to unpack this topic completely, I will try to
measurement and attribution and what action, if any, cover the high level summary here instead.
Table 1: Probability of outperforming benchmark (when the manager has no skill)
Alpha Tracking error
0% 1% 2% 3% 4% 5% 6% 7%
Period Probability of single manager outperforming benchmark
1 month 50% 50% 50% 50% 50% 50% 50%
3 months 50% 50% 50% 50% 50% 50% 50%
6 months 50% 50% 50% 50% 50% 50% 50%
1 year 50% 50% 50% 50% 50% 50% 50%
3 years 50% 50% 50% 50% 50% 50% 50%
5 years 50% 50% 50% 50% 50% 50% 50%
10 years 50% 50% 50% 50% 50% 50% 50%
20 years 50% 50% 50% 50% 50% 50% 50%
PAGE 4 STANLIB MULTI-MANAGER | MULTI-MANAGER MINDSETMore dimensions required to appraise coin. If you had an unbiased coin (equally likely to
manager skill – time period and tracking land on heads or tails), and you flipped it many
times, the chance of you getting a value far above
error or below the 50% mark (for heads or tails), would
Now, if the analysis were to instead focus on the drop with the number of flips. For example, the
managers achieving a minimum level of alpha (say 1%, chance of you flipping 60% heads (or more) in
gross), then the probability of managers with no skill ten flips of the coin i.e. 6 heads or more, would
achieving this will fall as the time period of the analysis be around 13% (not very likely, but certainly not
increases. The table below will demonstrate these rare). If however, you flipped the coin 50 times,
probabilities along two dimensions under idealised the chance of you flipping 60% heads (or more)
assumptions. The first dimension is the time period would drop to 6% (less than half the previous
used for the analysis, and this is observed by looking probability). The probability would decrease
at the rows along the leftmost column. The second further the more you kept flipping
dimension is tracking error (or active risk), observed • Secondly, the probabilities increase with tracking
by looking at the columns along the top row. error for any given period. Let’s consider another
example to help us understand this. I like to use
There is a lot of useful information in this table, so
insurance as another great example of uncertainty.
let’s examine some of it:
Imagine you have two different insurers, offering
• Firstly, the probabilities of outperforming the two different kinds of cover. The one offers cover
benchmark by 1% drop as the period increases, on regular cars which cost of average R100 000.
for any given level of tracking error. This is The other offers cover on high performance cars
analogous to how casinos operate. With the which cost of average R1 million. The probability
odds slightly in their favour, the probability of of a car crash is exactly the same in both cases (not
them making money increases as the number true in reality), and the insurer collects enough
of independent bets increases. Let’s explain this in premiums to cover the cost of the risk (also
using another classic example, namely flipping a not true in practice, as insurers need to cover a
Table 2: Probability of outperforming benchmark by 1% (when the manager has no skill)
Alpha Tracking error
1% 1% 2% 3% 4% 5% 6% 7%
Period Probability of single manager outperforming benchmark by 1% or more
1 month 39% 44% 46% 47% 48% 48% 48%
3 months 31% 40% 43% 45% 46% 47% 47%
6 months 24% 36% 41% 43% 44% 45% 46%
1 year 16% 31% 37% 40% 42% 43% 44%
3 years 4% 19% 28% 33% 36% 39% 40%
5 years 1% 13% 23% 29% 33% 35% 37%
10 years 0% 6% 15% 21% 26% 30% 33%
20 years 0% 1% 7% 13% 19% 23% 26%
PAGE 5 STANLIB MULTI-MANAGER | MULTI-MANAGER MINDSETplethora of additional costs). Let’s now assume left half of the table (triangle), as per Table 3 below.
that both insurers have exactly the same amount
So how could you legitimately use past performance
in rand of cars under insurance (say R100 million,
in a performance evaluation exercise, and what
which implies 1 000 cars for the first insurer, and
does the analysis in Table 3 tell us about the pitfalls
100 cars for the second insurer). If both decided
of doing so?
to hold just R1 million additional capital to ensure
that it could meet all claims, would they have the If we were to change the hurdle to 2%, and consider a
same probability of failure? The answer is no, manager with a tracking error of 4%, we can calculate
the second insurer would only be able to suffer that the probability of outperformance drops to
one additional loss more than expected before 19% for a period of three years, from 40% for three
running out of capital, whereas the first insurer months. The implication, is that a manager with no
could suffer an additional 10 losses (a much less skill is half as likely to outperform that hurdle if you
likely event) consider the performance over three years instead of
This may however be counter-intuitive for some who three months, which in turn implies that you are half
have read that managers will hug the benchmark as likely to erroneously assume that the manager has
so that they are not caught out for having no skill skill (although there is still a one in five chance of you
(through underperformance). While this is correct as being wrong).
the probability of underperforming by any amount What if you were to increase the tracking error to
greater than 0% (ignore the special case of 0%) will 6% (50% increase)? What time frame would now be
similarly increase with the holding period, there is appropriate to get back to the same probability? Again,
still a reason for managers to take more risk, which is we can calculate that the time period would now need
that the chance of outperforming also increases, and to be increased to seven years (133% increase).
represents a free option on clients’ assets.
So you should begin appreciating that time and
Increasing the hurdle (alpha) from 1% to 2%, will drop tracking error are two important dimensions in
all of the probabilities, but more so for the bottom performance evaluation.
Table 3: Probability of outperforming benchmark by 2% (when the manager has no skill)
Alpha Tracking error
2% 1% 2% 3% 4% 5% 6% 7%
Period Probability of single manager outperforming benchmark by 2% or more
1 month 28% 39% 42% 44% 45% 46% 47%
3 months 16% 31% 37% 40% 42% 43% 44%
6 months 8% 24% 32% 36% 39% 41% 42%
1 year 2% 16% 25% 31% 34% 37% 39%
3 years 0% 4% 12% 19% 24% 28% 31%
5 years 0% 1% 7% 13% 19% 23% 26%
10 years 0% 0% 2% 6% 10% 15% 18%
20 years 0% 0% 0% 1% 4% 7% 10%
PAGE 6 STANLIB MULTI-MANAGER | MULTI-MANAGER MINDSETThis is equally important when evaluating manager
There are performance in the context of the investment
decisions they make. For example, if a manager had
many important to invest in a particular company on the basis of a
considerations when doing low probability event that would make the company
performance evaluation, very profitable (say, a blockbuster drug), and the event
doesn’t occur and hence the investment turns out to
required to ensure that the have been a poor investment, would this represent a
analysis is meaningful and bad decision? You actually shouldn’t be making this
assessment based on a single investment, because
the conclusions are robust. the probability of the event is critical.
If the event was expected to have a probability of
Probabilities and expectations – another 1% (this could still make sense from an investment
dimension thesis point of view if the expected return was
sufficiently high), you would need to assess many of
Just like statistics are not intuitive, probabilities are these low probability events together. In this case,
sometimes even worse. How often have you heard 100 such events would not be enough to have much
(or said) that weather forecasters have no idea what confidence in your assessment because 1% of 100 is
they are doing, because they said there was only a only 1, making a zero events outcome quite likely.
30% chance of rain, and it rained (or similarly, not
equivalently, there was a 70% chance of rain, and it So prior probabilities and expectations are another
didn’t rain). People often assume that low probability important dimension in performance evaluation.
events don’t occur, but are very often happy to gamble There are many other important considerations
on low probability events (like the lottery). when doing performance evaluation, required
to ensure that the analysis is meaningful and the
To properly asses the skill of a weather forecaster, you conclusions are robust. Unfortunately, we have only
would need to compare their predictions to reality scratched the surface on this very important topic.
over many observations (not just a few, and certainly
not just one). For example, if you observed that there Conclusion
were 100 times that the forecaster said that the chance
of rain was only 30%, you should expect it to rain 30 You may come away from reading this with a sense
times (plus or minus some reasonable error, which that I have not provided you with solutions, but
you can calculate if you want to make some further rather only highlighted some of the pitfalls. This
assumptions about how confident you want to be in is intentional and important because I often see
the result). Now expecting it to rain 30 times out of a people seeking refuge in numbers (calculated very
hundred is very different from not expecting it to rain. precisely), believing that they hold the answers.
This article was meant to give you a sense of the
How does this translate into performance evaluation?
uncertainty that remains, even when doing the
Well, if you now consider the 19% probability referred
analysis robustly, and why decision making in the
to above (Table 3), it means that 19 out of 100 times
context of uncertainty is important.
you would still be wrong, even though you were
careful to extend the performance evaluation period We should never throw the baby out with the bath
from three months to three years for a manager with water. Having an understanding of the uncertainty
a tracking error of 4%. So you would be assuming that will allow us to better appreciate what confidence
the manager was skilful because she had outperformed we should have in the decisions we take, and what
the benchmark by 1% over three years, and this was outcomes we should expect when observed over
unlikely to occur by chance. multiple observations.
PAGE 7 STANLIB MULTI-MANAGER | MULTI-MANAGER MINDSETManager evaluation post
selection
By Lubabalo Khenyane, Portfolio Manager, STANLIB Multi-Manager
Investors should always remember their client fiduciary
duties and own biases when deciding which asset managers
to invest with.
Allocators of capital should strive to understand Understanding the landscape
performance evaluation before and during their
assessment of the value add from passive and active asset First we need to highlight that prior to evaluating
managers. It is not a simple task. The CFA Institute in its managers, we need to understand the market drivers
Certificate in Investment Performance Measurement of the various asset classes. For example, to construct
(CIPM) Program identifies some of the key aspects an equity fund in the early 2000s, you could classify
you should consider when evaluating performance as the equity market in South Africa into two investment
Performance Measurement, Performance Attribution styles – value and growth. Similarly, you could group
and Performance Appraisal. SA equity asset managers into those two buckets.
Value managers focused on buying cheap companies
This helps when assessing historical performance based on metrics such as low price to earnings ratios,
but our responsibility as capital allocators stretches while growth managers focused more on high return
beyond this. Our job is to identify skilful asset on earnings and other metrics.
managers that might or might not have performed
well historically but are likely to outperform their Since then, the market environment, and the asset
benchmarks and peers in future. Some people management industry has evolved, and today there
would argue this is trying to predict the future, an are more nuances to consider than those two simple
impossible task! dimensions. Why is this important? Today we need to
understand that most asset managers move between
That’s not entirely true. We use our comprehensive various investment styles depending on where they
manager research process to analyse, among other see opportunities. This makes both quantitative
things, a manager’s philosophy, process, people, and qualitative assessment of an asset manager
and portfolio construction and risk management important. Fortunately, return and holdings data to
process to decide on the likelihood of positive perform quantitative analysis is widely disseminated
future performance. Sonal Bhagwan takes a manager in the market. Historic returns data allows you to do
research perspective in her article and talks about performance measurement and risk analysis.
how we qualitatively assess asset managers.
Our manager research work culminates in a list of a Focus on the manager first
select few asset managers that we may invest with,
although not all of them would make it into our Using returns data we can calculate a manager’s
funds. In this article, we explore what we consider historic active returns against appropriate benchmarks,
when selecting managers, and how we assess their against peers and outperformance targets. During this
success or failure after the “hiring” decision. The analysis, we pay special attention to what was happening
article touches on how our individual biases can in the market when a manager out or underperformed
potentially affect our decisions. and whether there is a trend in a manager’s active
PAGE 8 STANLIB MULTI-MANAGER | MULTI-MANAGER MINDSETreturn in certain market environments. We assess a most asset allocators are guilty of not doing so (surely
manager’s skill by using measures such as information great performance should never be questioned, or so
ratios which quantifies a manager’s active return per they think).
unit of active risk taken. Moreover, we look at their
In this review, all performance measures and risk
active return volatility through measures such as
metrics calculated before we hired the manager are
tracking error.
again assessed. At this stage, we have the advantage
We do this analysis for several managers and it gives of having sight of the manager’s daily holdings to
us a sense of which managers have historically fared perform an attribution analysis. The key to attribution
better. This information helps but is not enough to say is to always analyse decisions that an asset manager
Manager A should be hired instead of Manager B. actually made, not the circumstantial ones.
So we continue with our analysis and run other For example in a multi-asset class fund, you cannot
measures such as cluster analysis and correlation give credit or criticise a portfolio manager for security
metrics to understand which managers have similar selection if all they make is the asset allocation decision
active return profiles. We need this information to and gain exposure to asset classes by investing in
build diversified portfolios. building blocks managed by other portfolio managers.
We then incorporate holdings data into the analysis. Attribution helps us to identify and analyse the
Using holdings data we can further our risk analysis drivers of performance, a piece of information that is
by evaluating metrics such as contribution to tracking important when engaging with the manager regarding
error, active positions and beta, a measure of sensitivity performance.
to the market. We also analyse the manager’s holdings
signature to identify which parts of the market they Shine the light on ourselves
prefer. The challenging part in all of this is when we have to
We analyse the quantitative information and shift our focus from the asset manager to ourselves
incorporate our understanding of the manager gained and really start asking difficult questions. This might
from the qualitative assessment process to try to include questions about whether we really understood
understand the repetitive nature of the manager’s the manager before we hired them. Behavioural
performance. When incorporating qualitative finance teaches us that as humans we have several
information, it is important to note defining moments biases, and as allocators of capital this can shape which
in a manager’s life. These are moments where asset managers we choose for our funds.
structural changes have happened such as a senior
The resultant effect of this is investing only with
portfolio manager leaving the company, significant
managers that resonate with us and not those that
team and culture changes or a change in philosophy.
our analysis suggests we should hire. It is important
We then decide on which managers we want to invest to be aware of our individual biases and create a
in and the weightings of each. Once included in one team culture that encourages colleagues to question
of our funds, we continue to monitor and review the each other’s logic. We need to consistently revisit
manager’s role in the fund, in the context of why they the investment case that was compiled at the time of
were included (the portfolio construction framework). hiring the manager and ask ourselves if, given what
This is a challenging task, especially when a manager we know about the manager today, we would still
underperforms in an environment you thought would hire them.
suit them. We also ask questions when a manager
If the answer is yes, we keep our investment as
outperforms in an unfavourable environment, but
even good managers underperform. However, if the
PAGE 9 STANLIB MULTI-MANAGER | MULTI-MANAGER MINDSETanswer is no, we then need to engage with the asset
The fiduciary duty carried manager on how performance can be improved, and
by allocators of capital is of where necessary part ways. Team work once again
paramount importance to clients’ becomes a necessity as colleagues can sometimes
pick up something you might have missed or give a
wealth. It is imperative to always different perspective.
exhibit care and diligence Conclusion
when performing it.
The fiduciary duty carried by allocators of capital
Performance evaluation plays is of paramount importance to clients’ wealth. It
an important role in quantifying is imperative to always exhibit care and diligence
when performing it. Performance evaluation plays
how well we have delivered on an important role in quantifying how well we have
our client’s objectives. delivered on our client’s objectives. Prior to hiring
asset managers, we should conduct a comprehensive
analysis focusing on quantitative and qualitative
aspects.
Historical returns and holdings data is widely available
and we need to use it to measure a manager’s
performance and the risks they have taken to produce
that performance. We should use attribution analysis
to identify the drivers of their performance and
engage with them on those drivers to ascertain
the likelihood of their persistence going forward.
Equally important is to ask difficult questions when
performance is different to our expectations and not
shy away from taking decisive actions. Investors need
to appreciate that performance evaluation does not
stop after a manager has been hired but continues
until they are fired.
We also need to be aware of our own biases and
not hold back on rectifying our mistakes when new
information suggests we were wrong in our initial
analysis. Finally, our clients entrust us with their hard
earned money because they believe we have their
best interests in everything we do. We need to prove
that we are worthy of that trust every single day.
PAGE 10 STANLIB MULTI-MANAGER | MULTI-MANAGER MINDSETKey factors to consider when
evaluating asset managers
By Sonal Bhagwan, Manager Research Analyst, STANLIB Multi-Manager
Performance evaluation is carried out to find the “best”
manager to meet a specific portfolio need, or to check
if an existing manager still meets the client’s original
portfolio need.
Performance evaluation is the process of: momentum, quality), especially in the case of an
equity manager. In active and passive investment
• Understanding how a manager’s performance
management, a manager makes assumptions about
was achieved
market efficiency.
• Testing if their data aligns with their investment
philosophy and process How a manager executes to deliver on their
philosophy is known as the investment process.
• Determining if their investment process can
Portfolio construction forms a large part of this
achieve reasonably consistent and satisfactory
process. Portfolio construction should provide
returns
detail on how the manager allocates capital to
The process entails a qualitative and quantitative assets, the interplay of decision making between
analysis using several sources of information. A the team members and the resulting allocation
combination of completed questionnaires provided decisions, liquidity management, and risk measures
by managers and desktop research forms an used in the investment process.
understanding of the manager. This is followed by a
due diligence to confirm the manager’s claims. The Testing what managers say
due diligence is used to clarify and investigate any
inconsistencies. An essential step in performance evaluation is
testing whether the manager does what they state
Six Ps are used as a broad guideline in the evaluation in their philosophy and process, and if this creates a
process: philosophy, process, people, price, sustainable competitive advantage.
performance and the physical environment.
A change in a manager’s philosophy over time
What are managers saying? causes uncertainty, but you need to investigate
further to determine if this is a cause for concern.
A manager provides a set of principles that guides If fundamental long-term shifts in a market have led
their investment decision making, which is referred to the manager’s philosophy evolving, this could be
to as a manager’s philosophy, or alpha thesis. This interpreted as good because the manager is able
philosophy helps you understand how and why to adapt to the market. However, if the philosophy
a manager generates alpha. It is the foundation changed on the whim of short-term volatile factors
of an investment process. Having a well-defined such as performance and flows, this creates a
investment philosophy and process enables effective question about the repeatability of the investment
testing of these factors. process.
Managers make assumptions about what guides Why is repeatability important? If a manager has
markets which impacts their investment philosophy. successfully delivered alpha in the past, a repeatable
These assumptions may depict the style of a manager process carries some weight in having the potential
(risk factor exposures including value, growth, to deliver positive alpha in the future.
PAGE 11 STANLIB MULTI-MANAGER | MULTI-MANAGER MINDSETPerformance can be evaluated from qualitative and performance appraisal, which is an overall skill
quantitative perspectives. assessment. Performance appraisal measures can
differ by the measure of risk used. This is important to
Quantitative analysis ensure that risk-taking is not confused with skill. The
The first point of reference is a manager’s returns diagram at the bottom left provides a brief description
over a specified period. Even though net returns are of a few risk-adjusted return metrics that could be
appropriate when considering investment options, used in performance appraisal.
gross returns should be utilised when evaluating skill,
as fees have no bearing on skill. Additionally, you may You need to keep in mind that past performance
be able to negotiate fees. does not necessarily indicate skill or guarantee future
performance because there will be times when
Quantitative analysis does not only include return skilled managers underperform, and managers with
analysis. Looking at holdings and transactions are no skill outperform. However, one of the objectives
equally important, and in many cases, much more of performance evaluation is to ascertain whether a
important. manager’s process can be expected to deliver alpha
Alpha is another relevant measure. Alpha is defined as going forward.
the portfolio return in excess of the relevant benchmark At the same time, taking cues from underperformance
return. Methods used to identify sources of portfolio may not be a correct or useful manner of assessing
alpha are referred to as return attribution techniques, performance. Even though this may be a delayed
which attempt to understand the consequences of signal of lack of skill, or lack of adherence to
active investment decisions and provide evidence investment process, or an operational problem, it
towards assessing claimed competencies. could just simply be “noise” or a random outcome.
In an attempt to deliver returns, a manager takes
Limitations of quantitative analysis
on risk, which can be defined as “exposure to
In the case of developing economies, financial
uncertainty”1. As with return attribution, risk
markets are typically illiquid, transaction volume is
attribution examines the risk implications of
low, and transaction costs high. These characteristics
investment decisions and can be used to analyse
arise as a result of many factors associated with
consistency with a manager’s investment philosophy.
developing economies, such as low levels of
Related to return attribution is investment perfor- regulation, poor accounting standards, weak
mance appraisal. Return attribution complements investor protection, political risk, and inflation risk.
This leads to less informational efficiency and high
asset price volatility.
Sharpe ratio
Additional return for bearing risk Under these circumstances any conclusion
above the risk-free market rate reached through quantitative assessments may be
questionable because data may not be up to date or
fully representative of the market. This conundrum
Treynor ratio is very relevant for many African financial markets in
Excess return per unit of systematic an infancy stage. Any quantitative assessment needs
risk an understanding of these limitations.
Qualitative analysis
Information ratio Some of the critical yet softer aspects of performance
Active return per unit of active risk or evaluation fall under people and the physical
tracking error environment which includes organisational culture.
1
CIPM (2017), Risk Measurement and Risk Attribution, 2018 CIPM Level 1 Notes
PAGE 12 STANLIB MULTI-MANAGER | MULTI-MANAGER MINDSETWhen a manager’s investment process is highly an ethical guideline for calculating and presenting
dependent on the skills or decision making of an investment performance history for investment
individual or a few key individuals, this would indicate management firms across the world.
“key-man risk”. The repeatability of a process is at
GIPS gives recommendations regarding input data,
risk should the key individual leave the firm.
calculation methodology and composite construction
Another people consideration is having investment with regards to performance figures. Composite
people with the right credentials and experience, refers to the combination of portfolios (one or more),
vital to creating and executing an investment process. which have a similar strategy, objective or mandate. In
Without this, even a good, consistent and repeatable representing historical performance, there are multiple
process could fail to deliver alpha. Why? An individual factors that GIPS recommends for consideration to
who does not have the right credentials or has not ensure there are no potential misrepresentations.
experienced a market crash, or liquidity crunch, or
When carrying out any benchmark relevant assessment,
other somewhat “rare” market phenomena, may not
you need to ensure an appropriate benchmark is
be able to appreciate the impact of these events on
specified. Benchmark specification errors result in
the investments they manage.
incorrect performance measurement, which negatively
Remuneration is important to attract and retain staff. impacts the attribution and appraisal analyses.
Remuneration related to product performance leads Changing a benchmark to mask underperformance is
to aligned interest between the firm and individual. inappropriate and unethical.
On the other hand, remuneration is not as vital as The impact of fees (price in the 6 Ps) needs to be
flexibility in working hours and funding further studies evaluated because the net return is the relevant return
for some employees. Each individual has different for the investor. Investors may not want to invest with
motivational factors. a skilful manager if the manager keeps most of the
Why is it important to retain staff? If a firm is unable excess returns in fees, merely using clients’ money to
to retain staff and there is high turnover, institutional generate fees. Management fees are not important in
knowledge and experience may be lost, which could evaluating skill, but they are critical in deciding whether
impact the manager’s performance. or not to invest in a skilful manager.
Related to the above is organisational culture. Conclusion
Investment individuals are motivated and have a greater
chance of being retained when an organisation’s Performance evaluation from a manager research
culture encourages positive traits. At the same time, perspective comes down to assessing and
if there is instability due to staff movements, this understanding a manager’s philosophy and process
does not necessarily imply concern. Some individuals while determining the impact of their process on their
get the opportunity to progress in their career and ability to deliver alpha over time. It involves a detailed
staff turnover can be a sign that the current firm is a qualitative and quantitative assessment of the manager
good grooming ground for future leaders. Thus, staff across many dimensions.
mobility is something that needs to be investigated Looking only at past returns ignores a plethora of
before reaching any conclusions. important aspects of the manager’s investment
philosophy and process. Not only should quantitative
Behind the figures analysis be as broad as possible along many other
dimensions (such as risk, holdings and transactions,
When quantitatively assessing a manager you must
mandates across the manager), but qualitative analysis
take Global Investment Performance Standards (GIPS),
should supplement the evaluation and consider the
benchmarks and fees into account.
future to determine whether the proposition is likely
GIPS are a voluntary set of standards that provide to produce the required results.
PAGE 13 STANLIB MULTI-MANAGER | MULTI-MANAGER MINDSETThe perils
of a CPI benchmark
By Amira Abbas, Research Scientist, STANLIB Multi-Manager
A discussion of the pitfalls of using one benchmark in
particular, CPI (the consumer price index), with regard to
performance evaluation.
Evaluating the performance of an asset manager is decreased because it’s eroded by inflation.
often subjective. Broadly speaking, there are three
elements to consider: What makes a good benchmark?
• Measurement – choosing an appropriate benchmark
A benchmark should serve as a point of reference to
• Attribution – determining which asset classes or which other things may be meaningfully compared.
instruments contribute to relative performance
• Appraisal – understanding the meaning of the A good benchmark is one that is fully consistent with
performance a manager’s investment process, style and philosophy.
This consistency makes it easier to evaluate the skill
All come with their own complexities, regardless of of a manager, and their ability to exploit perceived
how a manager structures their fund. opportunities by taking “off-benchmark” bets that
In this article I discuss the pitfalls of using one translate into alpha.
benchmark in particular, CPI (the consumer price
An appropriate benchmark serves as a sanity check
index), with regard to performance evaluation. Instead
of addressing how performance evaluation should to ensure a manager follows the style and philosophy
be done in this context, I will instead focus on the they claim to follow.
important dimensions to consider. Benchmarks should communicate information about
the manager’s investable universe and provide an
Why is CPI used as a benchmark? indication of acceptable levels of risk versus return. To
do this they need to conform to the characteristics of
Despite its limitations, CPI is widely used as a
good benchmarks:
benchmark, and with good reason. In a goal-
based world, an absolute return (nominal or real) is • Investable – it should be possible to replicate and
necessary if you are saving towards a goal, as you hold the benchmark, i.e. the weights and securities
need a mechanism to discount your investments and in the benchmark should be identifiable and
liabilities. CPI plus an additional percentage represents available for investment
a real return over inflation and makes sense to all sorts
of investors. • Appropriate – the benchmark should be consistent
with investment style and reflective of the
Contrast this with a target of 2% alpha over an index manager’s investment opinions
other than CPI, say an equities index. This gives no
insight into the relative value of money. The fund • Accountable – the benchmark chosen signifies
could have achieved its objective of say 5% while the the manager accepts ownership of the constituents
index returned 3%. If inflation over the period was 6%, and is held accountable for significant deviations1
the value of money invested and its purchasing power
1
CIPM Principles Reading, CFA Institute (2017)
PAGE 14 STANLIB MULTI-MANAGER | MULTI-MANAGER MINDSETWhy is CPI a bad benchmark? The first is the dispersion of the returns (which can
be measured around its average value in the case of
CPI is not investable nominal returns, or around a benchmark in the case
Asset managers cannot directly invest in CPI and of active returns). The second, is how the certainty of
achieve its return – unlike an equity index, for example, the average increases with the sample size, that is the
which may be closely replicated by purchasing the average value of the returns becomes more certain
index’s underlying stocks. Even inflation-linked bonds as we extend the period over which we measure
do not represent investing in CPI. returns (this is often misstated as time diversification).
CPI does not provide any information about a Yes, the above includes some implicit assumptions
manager’s philosophy or investment style about the processes generating the returns, but let
It may appear as if a manager is risky by taking large us ignore that complexity for the sake of not getting
off-benchmark bets (assumed from a large tracking overly complex.
error relative to CPI), but this is merely because most While the above uncertainty exists even for great
asset class returns are volatile relative to CPI over benchmarks, things get more complex when we
time. This makes it difficult to evaluate a manager’s consider CPI objectives as benchmarks, since asset
performance. class returns are generally volatile (uncertain) by
comparison.
How do we evaluate a fund with a CPI
So how do we choose an appropriate time period
benchmark?
for evaluation, and how certain can we be that we
From the get go, let us be clear that we are going to will achieve the CPI objective over that time period?
separate manager performance evaluation from fund These questions are difficult to answer and involve a
performance. We will come back to how to evaluate lot of subjectivity, even when conducting quantitative
manager performance when the benchmark given is analyses.
CPI. Investors will often question why a fund does not
beat its benchmark every year. They fail to recognise A simple example
that returns (both nominal and active, that is relative
to the benchmark) are inherently uncertain, which is Consider a SA equity-only fund, with a benchmark
the very definition of risk in investments. There are of CPI+7%. We will look at excess returns and
however two important dimensions that aid with the examine the complexities that arise in assessing the
understanding of this risk. performance of the fund.
Probabilities versus active returns and holding periods
Holding period
Active returns (less than or equal to) 1 year 3 years 5 years 7 years
-30% 2% 99% >99% >99% >99%
PAGE 15 STANLIB MULTI-MANAGER | MULTI-MANAGER MINDSETThe table above uses historical South African equity to see what time period corresponds to a 60% chance
asset class returns relative to CPI+7%. If we assume that of achieving an alpha of 0% or more (i.e. a 40% chance
future returns can be parameterised based on these of achieving 0% alpha or less). Looking at the row
historical returns we can estimate the probabilities of containing 0% alpha, this corresponds closely to a seven
certain outcomes over different time horizons. year holding period. Therefore, considering a fund’s
performance over a seven-year period, would only get
Consider the first row of the table where the
us to being 60% sure that the fund would achieve this
probability of underperforming CPI+7% by 20% or
benchmark. Unfortunately, that also means that there is
more, is about 7% if we look at a one year holding
still a 40% chance that the fund will underperform this
period, but less than 1% for three, five, and seven year
benchmark (a very likely event).
holding periods.
So what can we conclude ex-post if the fund
There are important take-aways from this table.
underperforms or outperforms the benchmark?
Notice how active returns (annualised) fall into Well, very little, especially when the probabilities
a narrower band as the holding periods increase are so high in both cases. Ideally, you want an event
(commonly referred to as the funnel of doubt). The to have a very low probability to conclude that it is
probabilities are contained between active returns of unlikely to have occurred by chance. Unfortunately,
approximately -10% to 20% for 7 year holding periods unlikely does not imply that it can’t occur, and you
as opposed to -30% to 40% for 1 year holding periods. could still get to the wrong conclusion.
This confirms our previous contention that the sample
Also, because these probabilities are very close to
average becomes more certain as the sample size
50%, extending the time period even longer will
increases, that is the return becomes more certain
not help much either. These probabilities are close
as the holding period increases. We often hear “if
to 50% because the historical returns were close to
you’re investing in equities, you need to be invested
CPI+7%. We will therefore need to flex a different
for a longer time horizon due to higher volatility of
parameter if we wanted to increase the certainty of
returns”, so this makes intuitive sense.
achieving the objective.
Given that we can expect the results to become
more certain as we increase the time frame over 2. The time horizon (holding period of the analysis)
which we do the analysis, how do we decide on Suppose we do not know how certain we want to
how long is enough? Surely waiting forever is not be, but we know over what time frame we would
an option. In fact, we probably want to wait as little like to do the assessment. Perhaps the manager
time as possible, for a number of different reasons. has provided guidance that they aim to achieve the
We should therefore understand that we will need to objective over three year rolling periods. In this case
compromise between waiting too long to be certain we can use our model to see what probability is
as the information will become worthless, and waiting associated with the three year period. Again, looking
too little and being very uncertain. There are three at the row containing 0% alpha, and the 3 year
variables that we can flex to help us address this. column, we see that underperforming the benchmark
translates into a 44% probability (a 56% probability of
1. The certainty of achieving the objective outperformance).
How certain do we want to be (ex-ante) of achieving This is again close to 50% for exactly the same
the objective? Remember that in financial markets (as reasons highlighted above. You will notice that these
in life) nothing is certain (not even death and taxes, as two variables are closely related, that is increasing
some countries have zero taxes, and pond scum are the certainty requires increasing the holding period
immortal). and vice versa. At this point, things may be looking
Say we would like to be 60% sure that we outperform a little bleak, but we have one more variable that we
the CPI objective. Using Table 1, we can work backwards can flex, and this one will come to the rescue.
PAGE 16 STANLIB MULTI-MANAGER | MULTI-MANAGER MINDSET3. The CPI objective (or benchmark)
If I ask you to be 99% confident in your answer,
The third variable that we can vary is the CPI objective you would aim for a much wider range, say between 5
itself, or equivalently, the alpha sought. In the above 000km and 20 000km. Incidentally, the circumference
example, we chose an objective that was close to the of the moon is approximately 10 921km.
historical average return of the asset class. We should
therefore expect the probabilities of outperforming
or underperforming this to be close to 50%, by Back to manager performance evaluation
definition of our model of future returns.
I promised to come back to this and provide some
If we instead consider a benchmark of CPI+5%, the guidance about how you would do this in the context
probability of underperformance drops. Now the of CPI+ benchmarks. In a nutshell, you throw out the
dimension of time makes a much bigger difference. CPI+ benchmark, and substitute it with something
If we again consider Table 1 above, at 0% alpha, the more appropriate. In some cases this will be fairly
probabilities of underperformance are 46%, 44%, straightforward, for instance use an appropriate
42% and 41% for one, three, five, and seven year equity index for an equity mandate. In other cases
holding periods respectively. With a revised objective it may be a little more complex, such as what do
of CPI+5%, these probabilities would drop to 42%, you use for a balanced or high equity multi-asset
36%, 32% and 29% for the same respective periods class fund (a composite of various indices may be
(not shown in Table 1). The benefits of a longer time appropriate in these cases). There are many tools and
period become more pronounced as expected. techniques to help with this exercise, and tracking
There is an important compromise that is happening error (or equivalently, the r-squared from a linear
here, that we shouldn’t lose sight of. By lowering the regression) is a good starting point.
target we are measuring against, we are improving the
chance of achieving (or more accurately exceeding) Conclusion
it. We are not changing the expected outcome, and Performance evaluation can be complex at the best
we should be careful to not confuse these two issues. of times, and downright impossible at the worst of
This is in some ways analogous to making predictions times. Performance evaluation versus traditional
or guessing the value of quantities you don’t know. benchmarks has its complexities, but may at least
One way of improving your success rates, is to make provide insight into the skill of an asset manager.
your prediction or guess less precise - perhaps a CPI objectives on the other hand, make economic
wider range. and intuitive sense, but introduce a range of unique
An analogy will help clarify this point. complexities, making performance evaluation
impossible. It is important to understand the
uncertainty that arises when faced with these
I ask you to estimate the circumference of the moon benchmarks, and the variables that can be flexed to
in kilometres, and to be 50% confident in your reduce this uncertainty. The confidence (certainty) in
answer. If I asked you to estimate many different the results, and holding period for the analysis are
things at this level of confidence, you should expect to two such variables. A third is the objective itself (or
get approximately half of them right, and half of them the level of relative performance sought).
wrong. You may have no idea what the circumference
Unfortunately, this does little to help in evaluating
of the moon is so you probably want a fairly large
the performance of the underlying manager, but
range for your estimate for example between
there are a range of tools and techniques that can
8 000km and 15 000km.
assist in this exercise.
PAGE 17 STANLIB MULTI-MANAGER | MULTI-MANAGER MINDSETE
T
T
W
+27 (0)11 448 6000
contact@stanlib.com
0860 123 003 (SA only)
stanlibmultimanager.com
PO Box 203 Melrose Arch 2076
GPS coordinates S 26.1347°, E 28.0686°
17 Melrose Boulevard Melrose Arch 2196
STANLIB Multi-Manager LimitedRegistration No: 1999/012566/06. A Financial Services Provider licensed under the Financial Advisory and Intermediary Services Act, 37 of 2002. FSP license No. 26/10/763You can also read
