USE OF DELPHI METHOD IN ABA SCLAID PUBLIC DEFENSE WORKLOAD STUDIES: A REPORT ON LESSONS LEARNED
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
USE OF DELPHI METHOD IN ABA SCLAID PUBLIC DEFENSE WORKLOAD STUDIES: A REPORT ON LESSONS LEARNED
Copyright © 2021 American Bar Association The views expressed herein represent the opinions of the authors. They have not been approved by the House of Delegates or the Board of Governors of the American Bar Association and, accordingly, should not be construed as representing the position of the Association or any of its entities.
USE OF DELPHI METHOD IN ABA SCLAID PUBLIC DEFENSE WORKLOAD STUDIES: A REPORT ON LESSONS LEARNED By Stephen F. Hanlon, Malia N Brink, and Norman Lefstein
Table of Contents
Acknowledgments................................................................................................................................ 1
Introduction............................................................................................................................................ 3
I. The History of Efforts to Set Limits on Public Defender Workloads................................ 7
A. The 1973 National Advisory Commission Standards.................................................................. 7
B. Weighted Caseload Studies........................................................................................................ 8
II. Background to ABA SCLAID’s Development of a Workload Study Methodology....... 11
III. Overview of ABA SCLAID Defender Workload Studies..................................................... 14
A. The System Analysis: The “World of Is”.................................................................................... 14
B. The Delphi Process: The “World of Should”............................................................................. 14
1. The Research Team............................................................................................................ 15
2. Local Support Necessary for a Successful Workload Study............................................... 15
3. Working as a Team.............................................................................................................. 16
4. Participants.......................................................................................................................... 16
5. Case Type/Case Task Selection.......................................................................................... 17
6. The Structure of the Delphi Process.................................................................................... 18
7. The Report........................................................................................................................... 19
IV. Lessons Learned and Open Questions..................................................................................21
A. Decisions in System Analysis................................................................................................... 21
1. Timekeeping........................................................................................................................ 22
2. The FTE and Caseload Methodology.................................................................................. 24
B. Decisions in the Delphi Process............................................................................................... 25
1. How Many Delphi Panels?................................................................................................... 26
2. How Many Case Types/Case Tasks?.................................................................................. 27
3. Trial v. Plea Analysis............................................................................................................ 28
4. Survey Interface Options..................................................................................................... 28
5. Structured Feedback........................................................................................................... 31
6. Attrition................................................................................................................................. 32
C. Timeline for a Workload Study.................................................................................................. 32
D. Costs of a Workload Study........................................................................................................ 32
V. Conclusion.....................................................................................................................................34
U s e o f D e l p h i M e t h o d i n A B A S C L A I D P u b l i c D e f e n s e Wo r k l o a d S t u d i e sAcknowledgments
This report1 provides an overview of efforts to quantify, with reliable data and analytics, maximum
caseloads for public defenders.2 In particular, the report details the Delphi method as utilized
by several major accounting and consulting firms, working with the American Bar Association
Standing Committee on Legal Aid and Indigent Defense (ABA SCLAID), to establish jurisdiction
specific workload standards for public defense providers.
This report was authored by three individuals who each worked on the ABA SCLAID workload
studies: Stephen F. Hanlon, who has served as the Project Director on all of the ABA SCLAID
workload studies, Malia N. Brink, who edited the Rhode Island and Colorado reports and then
served as Deputy Project Director on the Indiana Study, and Norman Lefstein, who consulted on all
of the ABA SCALID workload studies before his death in August 2019.
Stephen F. Hanlon founded the Community Services Team (CST) at Holland & Knight in 1989 and
served as the Partner in Charge of the CST for the next 23 years. Since his retirement from Holland
& Knight at the end of 2012, Mr. Hanlon has confined his practice to assisting and representing
public defenders with excessive caseloads. He served as general counsel to the National
Association for Public Defense and is a Professor of Practice at Saint Louis University School
of Law. Mr. Hanlon was lead counsel for the Missouri Public Defender in State ex rel. Mo. Public
Defender Commission, 370 S.W.3d 592 (Mo. 2012)(en banc), which was the first state supreme
court case to uphold the right of a public defense organization to refuse additional cases when
confronted with excessive caseloads.
Malia N. Brink has spent over 15 years working on criminal justice reform issues with a focus on
public defense reform. She currently serves as the Counsel for Indigent Defense to ABA SCLAID.
Before joining the ABA, Ms. Brink served as the Public Defense Project Director at the Justice
Programs Office of American University and the Director of Institutional Development and Policy
Counsel at the National Association of Criminal Defense Lawyers. Ms. Brink also serves as a
Lecturer in Law at the University of Pennsylvania Law School.
Norman Lefstein was a renowned legal scholar and academic, whose 45 years of scholarship
focused on indigent defense, criminal justice, and professional responsibility. In 2011, the ABA
published his book, Securing Reasonable Caseloads: Ethics and Law in Indigent Defense. In
addition, he played a major role as co-reporter in writing Justice Denied: America’s Continuing
Neglect of Our Constitutional Right to Counsel, published by the Constitution Project in 2009.
During the 1990s, Dean Lefstein was chief counsel for the Subcommittee on Federal Death
Penalty Cases, and in this capacity, he directed the preparation of Federal Death Penalty Cases:
Recommendations Concerning the Cost and Quality of Defense Representation, which was
approved by the Judicial Conference of the United States. Professor Lefstein served as Dean of the
IU McKinney School of Law from 1988-2002. Prior to becoming the law school’s leader, Professor
Lefstein was a faculty member for 12 years at the University of North Carolina School of Law in
1
This report covers the ABA SCLAID workload studies through early 2020. Although the Indiana Project was underway
during the drafting of this report, it had not yet been completed. The Indiana Project report was then released in July 2020.
See ABA Standing Committee on Legal Aid and Indigent Defendants, The Indiana Project: An Analysis of the Indiana
Public Defense System and Workload Standards (ABA 2020), available at https://www.americanbar.org/content/dam/
aba/administrative/legal_aid_indigent_defendants/the-indiana-project-july-2020.pdf.
2
Throughout this report we use the term public defenders to include all attorneys who provide public defense services,
whether as full-time employees of a governmental or non-profit office, as contractors or pursuant to court-appointment.
U s e o f D e l p h i M e t h o d i n A B A S C L A I D P u b l i c D e f e n s e Wo r k l o a d S t u d i e s | 1Chapel Hill. Before moving into academia, Professor Lefstein served as director of the Public
Defender Service for the District of Columbia, an Assistant United States Attorney in Washington,
D.C., and as a staff member of the Office of the Deputy Attorney General of the U.S. Department
of Justice. Early in his career, he directed a large-scale Ford Foundation research project in which
legal representation was furnished to juveniles in three metropolitan cities.
The authors must acknowledge the tremendous work of the accounting and consulting firms with
which ABA SCLAID partnered in conducting the public defense workload studies: RubinBrown,
Posterwaithe and Netterville, and Blum Shapiro.
Particular thanks are owed to past Chairman of RubinBrown, James Castellano, and the entire
firm. In partnering on the first ABA SCLAID workload study in Missouri, RubinBrown conducted a
comprehensive review of not only the use of the Delphi method, but also other potential methods,
and developed a national blueprint for conducting these workload studies
The authors also wish to acknowledge the members and staff of ABA SCLAID. Without their
advocacy and support, the workload studies would not have been possible. Finally, the authors
wish to thank the National Association of Criminal Defense Lawyers (NACDL) for partnering with
us on the Rhode Island Project, as well as this report. Special thanks to Bonnie Hoffman, NACDL’s
Director of Public Defense Reform and Training, for editing this report.
U s e o f D e l p h i M e t h o d i n A B A S C L A I D P u b l i c D e f e n s e Wo r k l o a d S t u d i e s | 2Introduction
In Gideon v. Wainwright,3 the United States Supreme Court recognized that the Sixth Amendment
right to counsel extends to state criminal proceedings and that every person who stands accused
of a crime requires the guiding hand of counsel to assist in their defense. In so doing, the Court
declared defense counsel for the accused essential to ensuring fairness and equality in our
criminal justice system.
[I]n our adversary system of criminal justice, any person haled into court, who is too
poor to hire a lawyer, cannot be assured a fair trial unless counsel is provided for
him. . . That government hires lawyers to prosecute and defendants who have the
money hire lawyers to defend are the strongest indications of the widespread belief
that lawyers in criminal courts are necessities, not luxuries. The right of one charged
with crime to counsel may not be deemed fundamental and essential to fair trials
in some countries, but it is in ours. From the very beginning, our state and national
constitutions and laws have laid great emphasis on procedural and substantive
safeguards designed to assure fair trials before impartial tribunals in which every
defendant stands equal before the law. This noble ideal cannot be realized if the poor
man charged with crime has to face his accusers without a lawyer to assist him.4
The purpose of the Sixth Amendment cannot be achieved simply by requiring that a person with
a law license stand by the side of the accused individual in court. Defense attorneys must provide
effective and zealous assistance of counsel pursuant to prevailing professional norms, including
the rules of professional conduct.5
In the 50 years since the Gideon decision, America’s public defense providers have struggled to
meet these standards. Excessive caseloads are routinely identified as a root cause of the inability
In the 50 years since the Gideon decision, America’s public defense providers have struggled to
meet these standards. Excessive caseloads are routinely identified as a root cause of the inability
to meet standards, as detailed in the numerous national reports analyzing this crisis.7 In short,
excessive caseloads stand as a core, overarching issue in public defense.8
3
372 U.S. 335 (1963).
4
Id. at 344.
5
Strickland v. Washington, 466 U.S. 668 (1983).
6
Throughout this report the term “caseload” refers to the total number and different kinds of cases assigned to a public
defense attorney or organization during a certain period of time. “Workload” is a broader concept, including all of an
attorney’s responsibilities: the cases on which an attorney works during the course of a year, as well as the many other
responsibilities not pertaining specifically to cases for which the attorney is responsible. This would include administrative
responsibilities, training time, supervision time, etc. As discussed in more detail later in this report, the Delphi survey
process addresses only the time requirements of attorneys for legal representation tasks performed on their various types
of cases under study.
7
See, e.g., Gideon Undone: The Crisis in Indigent Defense Funding – Transcript of a Hearing on the Crisis in Indigent
Defense Funding held during the Annual Conference of the National Legal Aid and Defender Association, November 1982
(ABA 1983), available at https://www.americanbar.org/content/dam/aba/administrative/legal_aid_indigent_defendants/
downloads/indigentdefense/gideonundone.authcheckdam.pdf; National Right to Counsel Committee, Justice Denied:
America’s Continuing Neglect of Our Constitutional Right to Counsel (The Constitution Project 2009), available at https://
www.opensocietyfoundations.org/reports/justice-denied-americas-continuing-neglect-our-constitutional-right-counsel;
Joel M. Schumm, National Indigent Defense Reform: The Solution is Multifaceted (ABA and NACDL 2012), available
at https://www.nacdl.org/Document/NationalIndigentDefenseReformTheSolutionisMult-(1) ; Andrea M. Marsh, State of
Crisis: Chronic Neglect and Underfunding for Louisiana’s Public Defense System (NACDL 2017), available at https://www.
nacdl.org/Document/StateofCrisisChronicNeglectUnderfundingLouisianaPD.
8
For an extensive summary of the depth and breadth of caseload issues, see Norman Lefstein, Securing Reasonable
Caseloads: Ethics and Law in Public Defense 12-19 (ABA 2011) available at: https://www.americanbar.org/content/dam/
aba/publications/books/ls_sclaid_def_securing_reasonable_caseloads.authcheckdam.pdf.
U s e o f D e l p h i M e t h o d i n A B A S C L A I D P u b l i c D e f e n s e Wo r k l o a d S t u d i e s | 3In 2003, following the 40th anniversary of Gideon, the American Bar Association’s Standing
Committee on Legal Aid and Indigent Defendants (ABA SCLAID) published a detailed report on
the state of public defense9 nationally.10 The report was based on testimony submitted at a series of
public hearings held around the country. It concluded that “too often . . . crushing workloads make
it impossible for [public defenders] to devote sufficient time to their cases, leading to widespread
breaches of professional obligations.”11 The report cited examples of excessive caseloads–often
with lawyers handling more than 1,000 cases per year–in Maryland, New York, Pennsylvania,
Rhode Island, and Nebraska.12
Five years later, in 2009, the National Association of Criminal Defense Lawyers (NACDL) published
Minor Crimes, Massive Waste: The Terrible Toll of America’s Broken Misdemeanor Courts, a report
looking specifically at misdemeanor courts.13 Again it documented that defenders from across the
country had massive caseloads, some handling in excess of 2,000 cases per year.14 More recently,
a 2016 series of articles noted Kentucky defenders averaged 448 cases in 2015 and Missouri
defenders often have upwards of 150 clients at one time.15 Data from the Texas Indigent Defense
Commission indicates more than 300 private attorneys each accepted over 250 assigned cases
in 2015, with a number of those attorneys accepting 500 cases or more.16 Notably, this is often in
addition to the private, retained cases these attorneys also accepted during the same year.
The simple reality is that, regardless of talent and experience, a lawyer with too many clients
cannot comply with their professional and ethical duties to each and every individual client
who is entrusted to them.17 For example, the Model Rules of Professional Conduct (Model
Rules) require all attorneys–including public defenders18 –to provide competent and diligent
representation.19 Competence requires not only legal knowledge and skill, but the “thoroughness
and preparation reasonably necessary for the representation.”20 An essential element of
competence is “adequate preparation.”21
9
In this report the term “public defense” includes institutional defender offices, such as the public defender, and private
attorneys who are providing representation to individual criminal defendants unable to afford counsel.
10
ABA Standing Committee on Legal Aid and Indigent Defendants, Gideon’s Broken Promise: America’s Continuing Quest
for Equal Justice (2004), available at https://www.americanbar.org/content/dam/aba/administrative/legal_aid_indigent_
defendants/ls_sclaid_def_bp_right_to_counsel_in_criminal_proceedings.authcheckdam.pdf
11
Id. at 16.
12
Id.
13
Robert Boruchowitz, Malia Brink, and Maureen Dimino, Minor Crimes, Massive Waste: The Terrible Toll
of America’s Broken Misdemeanor Courts (NACDL 2009), available at https://www.nacdl.org/Document/
MinorCrimesMassiveWasteTollofMisdemeanorCourts.
14
Id. at 20-21.
15
Oliver Laughland, “The Human Toll of America’s Public Defender Crisis,” The Guardian, Sept. 7, 2017, available at https://
www.theguardian.com/us-news/2016/sep/07/public-defender-us-criminal-justice-system.
16
Texas Indigent Defense Commission, Indigent Defense Data for Texas (2015), available at http://tidc.tamu.edu/public.net/
Reports/AttorneyCaseLoad.aspx.
17
See American Bar Association, Eight Guidelines of Public Defense Related to Excessive Workloads (2009), at Guideline
1 (“[I]f workloads are excessive, neither competent nor quality representation is possible.”), available at https://www.
americanbar.org/content/dam/aba/administrative/legal_aid_indigent_defendants/ls_sclaid_def_eight_guidelines_of_
public_defense.pdf.
18
See ABA Standing Committee on Ethics and Professional Responsibility, Formal Opinion 06-441: Ethical Obligations
of Lawyers Who Represent Indigent Criminal Defendants When Excessive Caseloads Interfere With Competent and
Diligent Representation (May 13, 2006), (“The obligations of competence, diligence, and communication . . . apply
equally to every lawyer.”), available at https://www.americanbar.org/content/dam/aba/administrative/legal_aid_indigent_
defendants/ls_sclaid_def_ethics_opinion_defender_caseloads_06_441.authcheckdam.pdf.
19
ABA Model Rules of Professional Conduct (2016), Rule 1.1. (“A lawyer shall provide competent representation to a client.
Competent representation requires the legal knowledge, skill, thoroughness and preparation reasonably necessary for the
representation.”) and Rule 1.3 (“A lawyer shall act with reasonable diligence and promptness in representing a client.”),
available at https://www.americanbar.org/groups/professional_responsibility/publications/model_rules_of_professional_
conduct/model_rules_of_professional_conduct_table_of_contents/.
20
Id. at Rule 1.1.
21
Id. at Comment 5 (emphasis added).
U s e o f D e l p h i M e t h o d i n A B A S C L A I D P u b l i c D e f e n s e Wo r k l o a d S t u d i e s | 4The ABA Defense Function Standards establish standards of practice for criminal defense
lawyers.22 They include the need to investigate case facts,23 research the law,24 communicate
with clients,25 negotiate with prosecutors,26 file appropriate motions,27 and prepare for court.28
Importantly, defense attorneys must perform these tasks regardless of whether the case proceeds
to trial or is resolved by a guilty plea.29 Defenders cannot do all these things for all clients when
they have too many cases. By necessity, if a lawyer has too many cases, competent work for one
client will result in unreasonable delays or a lack of work for other clients.30
Because excessive workloads create a grave risk of harm to clients, the Model Rules of
Professional Conduct require lawyers to limit their workloads. Moreover, both ABA practice
standards and Model Rules require that public defenders take steps to prevent or correct excessive
workloads.31 “Continued representation in the face of excessive workloads imposes a mandatory
duty to take corrective action in order to avoid furnishing legal services in violation of professional
conduct rules.”32 Attorneys who face excessive workloads must withdraw from cases or refuse
additional cases.33 The ABA Eight Guidelines of Public Defense Related to Excessive Workloads
require that providers take corrective action in advance, “to avoid furnishing legal services in
violation of professional conduct rules.”34
When an excessive caseload forces a lawyer to choose among the interests of clients, depriving
some if not all of them of competent and diligent defense services, the situation also constitutes
a conflict of interest.35 While the lawyer may be able to provide reasonably effective assistance
of counsel and meet ethical obligations for some clients, doing so requires the lawyer to sacrifice
duties owed and the provision of effective assistance to other clients. In this situation, “there is
a significant risk that the representation of one or more clients will be materially limited by the
lawyer’s responsibilities to another client[.]”36
22
American Bar Association, Criminal Justice Standards for the Defense Function (Defense Function Standards) (4th ed.
2015) at Standard 4-1.1 (a) (“These Standards are intended to address the performance of criminal defense counsel
in all stages of their professional work.”), available at https://www.americanbar.org/groups/criminal_justice/standards/
DefenseFunctionFourthEdition/#4-1.1. Note that the United States Supreme Court found that these standards are
“valuable measures of the prevailing professional norms of effective representation.” Padilla v. Kentucky, 559 U.S. 356,
367 (2010).
23
Defense Function Standards at Standard 4-4.1.
24
Id. at Standard 4-4.6
25
Id. at Standards 4-3.1, 4-3.3, 4-3.9, 4-5.1, and 4-5.4
26
Id. at Standards 4-6.1, 4-6.2, and 4-6.3.
27
Id. at Standards 4-3.2, 4-7.11, and 4-8.1.
28
Id. at Standard 4-4.6.
29
Id. at Standards 4-4.1 and 4-6.1(b).
30
See ABA Eight Guidelines, supra n. 17, Comment to Guideline 1.
31
See ABA Model Rules, supra n. 19, at Rule 1.3, Comment 2 (“A lawyer’s workload must be controlled so that each matter
can be handled competently.”); see also Defense Function Standards, supra n. 22, at Standard 4-1.8(a) (Lawyers “should
not carry a workload that, by reason of its excessive size or complexity, interferes with providing quality representation,
endangers a client’s interest in independent, thorough or speedy representation, or has significant potential to lead to
the breach of professional obligations.”); American Bar Association, Ten Principles of a Public Defense Delivery System
(2002), Principle 5 (“Defense counsel’s workload is controlled to permit the rendering of quality representation.”),
available at https://www.americanbar.org/content/dam/aba/administrative/legal_aid_indigent_defendants/ls_sclaid_def_
tenprinciplesbooklet.authcheckdam.pdf.
32
ABA Eight Guidelines, supra n. 17, at the Comment to Guideline 6.
33
ABA Model Rules, supra n. 19, Rule 1.16(a)(1) (“A lawyer shall not represent a client or, where representation has
commenced, shall withdraw from the representation of a client if the representation will result in a violation of the rules of
professional conduct or other law.”); and ABA Formal Opinion 06-441, supra n. 18.
34
ABA Eight Guidelines, supra n. 17, at the Comment to Guideline 6.
35
Id. at the Comment to Guideline 1, citing In re Order on Prosecution of Criminal Appeals by the Tenth Judicial Circuit
Public Defender, 561 So. 2d 1130, 1135 (Fla. 1990).
36
ABA Model Rules, supra n. 19, Rule 1.7(a)(2).
U s e o f D e l p h i M e t h o d i n A B A S C L A I D P u b l i c D e f e n s e Wo r k l o a d S t u d i e s | 5Defenders with excessive caseloads inevitably engage in triage and clients suffer real harm as
a result. For example, a prosecutor in Miami extended a time-sensitive plea offer of 364 days in
jail and seven years of probation to a public defender for a client charged with a significant felony.
Because the defender had 11 cases set for trial on the same day that the prosecutor required
a response, the defender failed to convey the plea offer to the client. When she contacted the
prosecutor after the deadline to accept the offer the prosecutor said the offer was no longer
available. The client later pleaded guilty with a significantly less favorable agreement, receiving five
years in prison plus probation.37 For this client, the cost of an overburdened public defender was an
additional four years of incarceration.
Excessive caseloads also prevent defenders from reviewing and assessing case weaknesses in a
timely manner. Consider the case of Donald Gamble, who was charged with two counts of robbery
and initially represented by a defender with too many cases. Mr. Gamble was detained pretrial and
no progress was made on his case for a year, after which his public defender resigned her position.
When a Loyola law professor subsequently accepted the case, she reviewed the file and quickly
identified conflicts between security footage and other evidence in the case. She presented the
conflicting evidence to the judge and, a few days later, the prosecutor dropped all charges.38 For
Donald Gamble, the cost of an overworked public defender was a year in jail despite clear evidence
of innocence.
In 2017 current and former Orleans Parish public defenders articulated their grave concerns
regarding the inevitable errors caused by excessive caseloads on an episode of the television news
show 60 Minutes. Nine defenders shared that they all believed innocent clients had gone to jail
because they lacked the time to properly represent their clients.39 An overburdened lawyer simply
cannot meet the effectiveness standards of the Sixth Amendment right to counsel.
But how many cases are too many? For almost 45 years, the public defense community has
struggled to develop reliable caseload/workload limits. Most recently, ABA SCLAID and others
have utilized the Delphi methodology in seeking to provide meaningful limits for public defenders
in particular jurisdictions. As of this writing, ABA SCLAID has contributed to studies completed in
Colorado, Louisiana, Missouri, and Rhode Island, and consulted on a study completed in Texas.40
A workload study was attempted in Tennessee but failed for reasons that will be discussed below.
An additional ABA SCLAID study was subsequently completed during the drafting of this article
in Indiana.
This report will detail the methodology used in the ABA SCLAID workload studies and share lessons
learned. The purpose of this report is to assist public defense organizations41 in determining whether
they have the necessary infrastructure and resources to undertake similar studies, and to assist
other research entities that may seek to conduct such studies. Part I of this report reviews the history
of efforts to develop reliable workload limits for public defenders. Part II provides an overview of the
Delphi method used by ABA SCLAID and its research partners. Part III delves deeper into the ABA
SCLAID use of the Delphi method, looking at decisions made during implementation.
37
Lefstein, supra n. 8 at 61-2.
38
CBS, 60 Minutes, Inside NOLA Public Defenders’ Decision to Refuse Felony Cases, April 16, 2017, available at http://
www.cbsnews.com/news/inside-new-orleans-public-defenders-decision-to-refuse-felony-cases/.
39
Id.
40
ABA SCLAID was the organization responsible for the law and standards on the Missouri, Louisiana, Rhode Island,
and Colorado studies, as well as the failed study in Tennessee. Additionally, ABA SCLAID study personnel consulted
on the study in Texas. Highly respected national accounting and consulting firms partnered with ABA SCLAID and
were responsible for the data analysis portions of these studies. An experienced academic research organization was
responsible for the data analysis portion of the Texas study.
41
Workload studies are generally commissioned by an organization with oversight responsibility for public defense in a
jurisdiction. This can be a Commission, as in Indiana, or a public defender office or agency, as in Colorado or Rhode
Island. In this report we call these entities, collectively, public defense organizations.
U s e o f D e l p h i M e t h o d i n A B A S C L A I D P u b l i c D e f e n s e Wo r k l o a d S t u d i e s | 6I. The History of Efforts to Set Limits on Public Defender Workloads
A. The 1973 National Advisory Commission Standards
In 1973, 10 years after Gideon, the National Advisory Commission on Criminal Justice Standards
and Goals (NAC) endeavored to set the first public defender caseload limits (NAC Standards). The
NAC adopted the recommendation of a National Legal Aid and Defender Association (NLADA)
committee which stated that individual defenders’ annual caseloads should not exceed 150
felonies, 400 misdemeanors (excluding traffic cases), 200 juvenile cases, 200 mental health cases,
or 25 appeals, or a proportional combination thereof.42 The NAC Standards were considered the
national caseload standards for many years.
In 2002, the ABA promulgated its Ten Principles of a Public Defense Delivery System.43 Principle
5 of that document directs that a lawyer’s workload be “controlled to permit the rendering of
quality representation.”44 The Commentary to the Principle states “[n]ational caseload
standards should in no event be exceeded” and cites to the NAC Standards as an example
of such national standards.45
In 2007, NLADA’s American Council of Chief Defenders (ACCD) adopted a resolution
recommending “public defender and assigned counsel caseloads not exceed the NAC
recommended levels.”46 An accompanying memorandum observed that while the NAC Standards
“have proved resilient,” the NAC levels must be altered to account for myriad local practices,
including insufficient support staff levels, complex or severe sentencing schemes, particularly
complex cases, and the need to advise clients regarding collateral consequences.47
The exceptions and need for alterations noted in the ACCD’s resolution and accompanying
memorandum highlight the limited usefulness of the NAC Standards. Because the Standards group
cases together in very broad classifications (e.g., felony, misdemeanor, juvenile), they do not allow
for consideration of the variation among case categories. For example, the “felony” classification
encompasses everything from theft and drug possession to rape and murder. Additionally, other
common case types, such as probation violations, are not listed at all. For this reason, the ACCD
recommended “that each jurisdiction develop caseload standards for practice areas that have
expanded or emerged since 1973 and for ones that develop because of new legislation.”48 One
suggestion from the ACCD was to develop adjustments or case weights to address how different
factors impact or alter the appropriate caseload standard in a particular jurisdiction.49
The Missouri State Auditor raised similar concerns and rejected the use of the NAC Standards in
an October 2012 report. The Auditor was asked to review the Missouri Public Defender’s caseload
crisis protocol, under which the public defender’s office sought to refuse cases when the caseloads
42
National Advisory Commission on Criminal Justice Standards and Goals (1973) at Standard 13.12-Workload of Public
Defenders, available at http://www.nlada.org/defender-standards/national-advisory-commission.
43
Ten Principles, supra n. 31.
44
Id. at Principle 5.
45
The NAC Standards are, as a result of this citation and a citation in ABA Formal Opinion 06-441 (supra n. 18), often
erroneously referred to as the ABA Standards.
46
American Council of Chief Defenders Statement on Caseloads and Workloads (August 24, 2007), available at https://
www.americanbar.org/content/dam/aba/administrative/legal_aid_indigent_defendants/ls_sclaid_def_train_caseloads_
standards_ethics_opinions_combined.authcheckdam.pdf.
47
Id. at 12.
48
Id.
49
Id. at 2 (“One system that can be utilized to arrive at an appropriate maximum limit for complex cases is a case credit
system that allows multiple credits for specific types of cases and recognizes that lawyers can handle fewer of those
cases per year.”).
U s e o f D e l p h i M e t h o d i n A B A S C L A I D P u b l i c D e f e n s e Wo r k l o a d S t u d i e s | 7were too high. In the protocol, the refusal limit was based substantially on the NAC Standards,
with adjustments made for things like complexity through a case-weighting scheme similar to what
was recommended in the ACCD opinion.50 In rejecting the protocol, the Auditor noted that there
was “very little information regarding the methodology and factors considered in the development
of the [NAC S]tandards.”51 He further noted, as the ACCD had, that the NAC Standards “did not
distinguish between types of felony offenses and were not established for certain types of cases.”52
The Auditor concluded that “[w]ithout adequate information to support how the national caseload
standards were derived or maintaining documentation to support assumptions and decisions
regarding case weights, the [Missouri State Public Defender] is unable to demonstrate it has
accurately converted the standards to case weights.”53
B. Weighted Caseload Studies
Researchers have also explored weighted caseload studies as a way to set appropriate caseload
limits. Like the ACCD recommendation, these studies establish case weights and adjustments from
a norm, but instead of the norm being the NAC Standards, the norm was often the current caseload
in the jurisdiction. Many public defense weighted caseload studies were conducted between
1995 and 2010 by the Spangenberg Group.54 The National Center for State Courts (NCSC) 55 also
conducted three statewide weighted caseload studies: Maryland (2005),56 New Mexico (2007),57
and Virginia (2010).58 RAND recently completed a statewide workload study in Michigan (2019).59
The federal defender system has also used a variation of case weighting to address caseloads.60
A review of the NCSC New Mexico workload study demonstrates how these studies typically
work. First, groups of public defenders and private contract lawyers met to determine relevant
workload factors and tasks associated with effective representation in each type of case. For New
Mexico, 11 different types of cases were chosen for the study.61 Second, the NCSC trained the
participating lawyers on how to properly track and record time using a web-based program, and
had attorneys record the time spent on both case-related and non-case related activities for a six-
week period.62 The NCSC opined that this short period of time, given the high level of participation,
was sufficient to produce “a valid and reliable snapshot from which to develop case weights.”63 The
50
See Thomas A. Schwiech, Missouri State Auditor, Missouri State Public Defender (Oct. 2012), available at https://app.
auditor.mo.gov/Repository/Press/2012-129.pdf.
51
Id. at 14.
52
Id.
53
Id.
54
The Spangenberg Group was a nationally recognized research and consulting firm specializing in improving justice
programs. The Spangenberg Group conducted weighted caseload studies in Colorado (1996) and Tennessee (1999).
For more information, see The Spangenberg Group, Keeping Public Defender Workloads Manageable, Bureau of Justice
Assistance, Jan 2001, available at https://www.ncjrs.gov/pdffiles1/bja/185632.pdf.
55
The National Center for State Courts has conducted workload and resource assessments over the years for various
aspects of the criminal justice system, including judges, probation officers, and defenders. A complete listing of their work
is available at https://www.ncsc.org/services-and-experts/areas-of-expertise/workload-assessment.
56
Brian J. Ostrom, Maryland Attorney and Staff Workload Assessment, 2005, available at https://cdm16501.contentdm.oclc.
org/digital/collection/ctadmin/id/414.
57
Daniel J. Hill, A Workload Assessment Study for the New Mexico Trial Court Judiciary, New Mexico District Attorneys’
Office and the New Mexico Public Defender (National Center for State Courts 2007), available at https://cdm16501.
contentdm.oclc.org/digital/collection/ctadmin/id/1084.
58
Matthew Kleiman, Ph.D. and Cynthia Lee, J.D., Virginia Indigent Defense Commission Attorney and Support Staff
Workload Assessment, available at https://cdm16501.contentdm.oclc.org/digital/collection/accessfair/id/189.
59
See, Nicholas M. Pace, et. al., Caseload Standards for Indigent Defenders in Michigan (RAND Corporation 2019),
available at https://www.rand.org/pubs/research_reports/RR2988.html.
60
See Nicholas M. Pace, et al., Case Weights for Federal Defenders (RAND Corporation 2011), available at https://www.
rand.org/content/dam/rand/pubs/technical_reports/2011/RAND_TR1007.pdf.
61
New Mexico Workload Assessment Study, supra n. 57 at 74.
62
Id. at 78.
63
Id.
U s e o f D e l p h i M e t h o d i n A B A S C L A I D P u b l i c D e f e n s e Wo r k l o a d S t u d i e s | 8NCSC concluded from that data that attorneys were spending, for example, almost seven hours on
each non-violent felony case64 and called this a “preliminary case weight.” The “case weight” data
collected and documented what the attorneys were currently doing on cases operating under the
caseload and time constraints they were currently experiencing. This represented current practice
or the measure of the world of “what is.”
To move from that world of “what is”–which does not necessarily capture the time necessary to
perform essential tasks effectively–to the world of “what should be,” the NCSC used a “quality
adjustment process,” which consisted of two parts. First, a web-based “sufficiency of time survey”
was sent to the attorneys asking whether they had sufficient time to perform key tasks of effective
representation.65 Then, after data was gathered, veteran public defenders from offices across the
state were convened to consider the results from the time study and various areas of concern
identified by the sufficiency of time survey. In New Mexico, these attorney groups reviewed 90
distinct events related to attorney performance. Of these 90 decision points, quality adjustments
were made to 21.66 In each instance in which a quality adjustment was made, the group was asked
to provide a rationale and justify any increase in attorney time.”67
“The combination of the case-related time study data (representing current practices, or ‘what is’)
and the quality adjustment data (representing preferred practices, or ‘what should be’) creates a
[final] ’case weight’ for each case-type category.”68 The NCSC staff then determined the number
of days per year (233 days) and hours per day (6.25 hours) attorneys had to perform case-related
activities. Applying these case weights to the time allotted, the NCSC then produced annualized
caseload limits. At the conclusion of this process, the quality adjusted caseloads in New Mexico
came out very close to the NAC Standards, with slightly fewer felonies allowed: 144 felonies (or 138
felonies, including murder), rather than the 150 in the NAC Standards, and more misdemeanors
(414 in comparison to the NAC’s standard of 400) and juvenile cases (251 in comparison to the
NAC standard of 200).69
Applying the case weights to the projected caseload of the public defender and dividing them
by the hours that an attorney has for case work, the NCSC was also able to produce a full-time
employee equivalence staffing number.70 Based on those calculations, the NCSC study determined
that the New Mexico Public Defender program needed 41 additional attorneys, an increase from
169 to 210.71
In summary, weighted caseload studies start with the actual time spent, as calculated during a time
study. This time study is the basis for all other calculations. An attorney survey determines whether
they believe this is sufficient time to complete the tasks. Thereafter a smaller, more experienced
group of public defenders reviews the time allotment and based on the survey results and personal
experience, determines whether additional time should be added to “perform essential tasks and
functions effectively.”72
64
Id. at 78-79.
65
Id. at 83-84.
66
Id. at 85. It is noteworthy that the time-sufficiency survey followed by the meeting of veteran public defenders used in this
type of workload study is also viewed as an application or modification of a Delphi method.
67
Id. at 86.
68
Id. at 10.
69
Id. at 87. Note that in this study the NAC Standards are referred to as the ABA Standards.
70
Id. at 10-11.
71
Id. at 5. Interestingly, the study concluded that the District Attorneys should also receive an increase of 41 FTE attorneys,
from 324 to 365.
72
Id. at 79.
U s e o f D e l p h i M e t h o d i n A B A S C L A I D P u b l i c D e f e n s e Wo r k l o a d S t u d i e s | 9While weighted caseload studies provide critical insights into how to conduct a more structured
inquiry into public defender caseloads, ABA SCLAID and its partners found certain aspects of the
weighted caseload studies problematic. First, the length of time studied often is insufficient. The
NCSC New Mexico study, for example, used six weeks of time data, which is not long enough to
cover even low-level cases from opening to close. Extrapolations made from such a limited sample
are prone to error. Second, beginning consideration with current time expenditures as the basis
for any of the calculations reinforces existing systemic deficiencies.73 Third, the time adjustment
process was not tied to particular standards; instead, attorneys surveyed were simply asked to
use their experience to determine whether they generally had enough time for a particular task.
Similarly, the small group of experienced public defenders asked to evaluate the need for time
adjustments operated without reference to standards. In this process, “individual lawyers might be
reluctant to admit that they should have spent more time on their cases, regardless of whether the
data was submitted anonymously. There is also some risk that defenders might not appreciate that
they should have spent more time on their cases, simply because they have not done so in the past
and believe what they have been doing is perfectly fine.”74
73
For this reason, the RubinBrown and all other accounting and consulting firms that have worked on ABA workload studies
have rejected the use of timekeeping data as an anchor, even when the time data is reliable. Instead, the ABA studies
use law and standards as the anchor. This risk was also well-described in the recent RAND workload study of Michigan:
“Providing such information to decisionmakers undoubtedly runs the risk of an anchoring bias, where judgments (such as
estimates of the average amount of attorney time needed for effective representation) are excessively influenced by initial
values presented to those decisionmakers (such as findings of a time study showing the average amount of attorney time
currently spent on cases).” Pace, supra n. 59, at 23. The RAND researchers chose to show the time data to its panels
despite acknowledging that there was no way to fully avoid the anchoring bias, although RAND also found that the ABA’s
decision not to show this data to panelists was reasonable. See id. at 18 (noting that the ABA studies are among the “well-
tested models” for how to conduct public defense workload studies).
74
Lefstein, supra n. 8 at 150.
U s e o f D e l p h i M e t h o d i n A B A S C L A I D P u b l i c D e f e n s e Wo r k l o a d S t u d i e s | 10II. Background to ABA SCLAID’s Development of a Workload
Study Methodology
A series of events in 2011 and 2012 drove ABA SCLAID to consider an alternative method
for examining defender workloads. In 2011, the ABA published Norman Lefstein’s Securing
Reasonable Caseloads: Ethics and Law in Public Defense, which detailed the various efforts
undertaken thus far and the difficulties with each of those efforts. In that same year, proceedings
also began in the Missouri case State v. Waters.75
In Waters, relying upon the Missouri Public Defender’s protocol for providing a certification of
unavailability once a district defender’s office exceeded the established caseload maximums, an
office moved to be relieved of further representation of clients due to excessive caseloads. The
office also cited to the obligations the attorneys bore under the applicable Rules of Professional
Conduct and the obligation to provide effective assistance of counsel under the Sixth Amendment.
Although the trial judge conceded that the public defenders’ caseloads were excessive, the court
nevertheless ordered the office to continue representing new eligible defendants.76
On appeal, the Missouri Supreme Court found that the trial court exceeded its authority by
appointing the public defender to represent a defendant in contravention of the Rules of
Professional Conduct, the Sixth Amendment, and the Missouri Public Defender’s caseload
maximum rule, and ordered the trial court to vacate its order.77 Thus, Waters stands for the
proposition that when a public defender office can demonstrate that it has so many cases that
its lawyers cannot provide competent and effective representation to all their clients, lawyers
may–indeed, must–refuse additional appointments and judges may not appoint them to represent
additional defendants.
Although Waters was decided on July 31, 2012, in October 2012, as noted above, the Missouri
State Auditor found the Missouri Public Defender’s caseload crisis protocol in invalid because it was
based substantially on the NAC Standards, which lacked sufficient support. As a result, the promise
of Waters and its mechanism for declining cases was dependent on finding a more reliable way to
measure whether a public defense provider has too many cases.
In the immediate aftermath of the Waters decision and Missouri State Auditor’s report, Stephen
F. Hanlon, who was lead counsel for the Missouri State Public Defender (MSPD) in Waters,
was retained by MSPD to find a reliable alternative to using the NAC Standards to enforce the
Waters decision. Mr. Hanlon investigated methodological options, including the recommendation
made by Professor Norman Lefstein in Securing Reasonable Caseloads to consider utilizing the
Delphi method. Mr. Hanlon then conducted an extensive literature review of the Delphi method,
preliminarily concluding it could have merit to apply to the problem of reliably determining
appropriate defender caseloads. He then began to develop a potential plan for a Delphi-based
public defender workload study.
Realizing that additional expert assistance would be required to design and conduct a workload
study, Mr. Hanlon began a search for an accounting firm that could complete the requisite research,
design and conduct a workload study in Missouri. He identified RubinBrown, one of the nation’s
75
State v. Waters, 370 S.W.3d 592 (Mo. 2012) (en banc).
76
See Stephen F. Hanlon, The Appropriate Legal Standard Required to Prevail in a Systemic Challenge to an Indigent
Defense System, 61 St. Louis L.J. 625, 636 (2017). Mr. Hanlon was lead counsel for the Missouri Public Defender in
Waters.
77
Waters, 370 S.W. 3d at 607.
U s e o f D e l p h i M e t h o d i n A B A S C L A I D P u b l i c D e f e n s e Wo r k l o a d S t u d i e s | 11leading accounting and consulting firms. He submitted the results of his literature review and his
initial design work to RubinBrown and proposed to ABA SCLAID that it retain the accounting firm
RubinBrown to:
• Conduct a thorough literature review of previous public defender workload studies and
the Delphi method;
• Determine whether the Delphi method was a reliable research method capable of
generating a reliable consensus of expert opinion for a workload study for the Missouri
Public Defender or determine an appropriate research method for setting appropriate
workload for a public defender office; and
• Conduct a reliable workload study of the Missouri Public Defender system that would
have, as its basis, ABA practice standards and the Rules of Professional Conduct.
After an exhaustive literature review, RubinBrown concluded that the Delphi method was a reliable
research tool to determine the appropriate workload for a public defender office because it was
capable of generating a reliable consensus of expert opinion. As Professor Lefstein had observed:
The Delphi method is based on a structured process for collecting and distilling
knowledge from a group of experts by means of a series of questionnaires
interspersed with controlled opinion feedback. Delphi is used to support judgmental
or heuristic decision-making, or, more colloquially, creative or informed decision-
making. The technique is recommended when a problem does not lend itself to
precise measurement and can benefit from collective judgments, which is precisely
the situation when a defense program considers how much additional time its lawyers
need to spend on a whole range of activities involving different kinds of cases.78
RubinBrown noted that the Delphi method, developed by researchers at the Rand Corporation
over 60 years ago, “has been employed across a diverse array of industries, such as health care,
education, information systems, transportation, and engineering.”79 “The purpose of its use beyond
forecasting has ranged from ‘program planning, needs assessment, policy determination, and
resource utilization.’”80 The ABA SCLAID public defender workload studies are, in many ways,
program planning and needs assessment studies. Respecting the accuracy of opinions reached by
Delphi panelists, RubinBrown observed that “researchers have found that the majority of studies
provide compelling evidence in support of the Delphi method” as compared to “unstructured
interacting groups.”81
Thereafter, Mr. Hanlon, ABA SCLAID and RubinBrown outlined the following essential features of a
public defense workload study:
• The professionals conducting the workload study must be the facilitators of the study,
not the arbiters of what is “appropriate”. The governing principle with respect to the
professional judgment of the Delphi panel must be “[l]et the chips fall where they may.”
78
Lefstein, supra n. 8 at 147.
79
ABA Standing Committee on Legal Aid and Indigent Defendants, The Missouri Project: A Study of the Missouri Public
Defender System and Attorney Workload Standards (ABA 2014), at 9, available at https://www.americanbar.org/content/
dam/aba/administrative/legal_aid_indigent_defendants/ls_sclaid_pub_def_mo_workstudies_rept.pdf.
80
Id. at 10 (citing Chia-Chien Hsu & Brian A. Sandford, The Delphi Technique: Making Sense of Consensus, 12 Prac.
Assessment, Res.& Eval., 1 (2007), https://scholarworks.umass.edu/cgi/viewcontent.cgi?article=1177&context=pare).
81
Id. at 10.
U s e o f D e l p h i M e t h o d i n A B A S C L A I D P u b l i c D e f e n s e Wo r k l o a d S t u d i e s | 12• The professional judgments must come from both public defenders82 and private practice
criminal defense lawyers.
• A successful workload study requires two areas of expertise: (1) surveying and data
analysis, and (2) law and standards.
• Legal, practice and ethical standards, not the results of any timekeeping data, are the
appropriate anchors83 for the professional judgments in the study.
• In this standards-based inquiry, the standards that drive the study are the ABA
Criminal Justice Standards,84 the applicable Rules of Professional Conduct,85
and the United States Supreme Court’s holding in Strickland v. Washington that
an indigent criminal defendant is entitled to “reasonably effective assistance of
counsel under prevailing professional norms.”86
• In particular, the instructions to the adult criminal Delphi panel, which serve much
the same function as jury instructions, would emphasize ABA Defense Function
Standard 4-6.1(b): “In every criminal matter, defense counsel should consider the
individual circumstances of the case and of the client, and should not recommend
to a client acceptance of a disposition offer unless and until appropriate
investigation and study of the matter has been completed. Such study should
include discussion with the client and an analysis of relevant law, the prosecution’s
evidence, and potential dispositions and relevant collateral consequences.
Defense counsel should advise against a guilty plea at the first appearance,
unless, after discussion with the client, a speedy disposition is clearly in the client’s
best interest.”87
• Timekeeping data should not be used as an anchor for the professional judgment
of the Delphi panel to avoid institutionalizing current practices, which may be
deficient.
Following these basic principles, ABA SCLAID began its work in Missouri in early 2013.
82
As addressed in note 2, supra, this report uses the term public defenders to include all types of public defense providers.
The selection of participants in each jurisdiction reflects how public defense is provided in the jurisdiction being studied.
Where public defense is provided, in part, by contractors or court-appointed counsel, efforts are made to ensure that
these types of public defenders are included in the Delphi process.
83
As a term of art in the science of research methodology, an anchor is used to constrain the consensus professional
judgment of the Delphi panel.
84
The United States Supreme Court has found that these standards are “valuable measures of the prevailing professional
norms of effective representation.” Padilla, 559 U.S. at 367. These Standards are the result of a lengthy process that
began in 1964, and, importantly, they “are the result of the considered judgment of prosecutors, defense lawyers, judges,
and academics who have been deeply involved in the process.” Martin Marcus, The Making of the ABA Criminal Justice
Standards: Forty Years of Excellence, 23 Crim. J. 10 (2009), available at https://www.americanbar.org/content/dam/aba/
publications/criminal_justice_magazine/makingofstandards_marcus.pdf.
85
The ABA Model Rules of Professional Conduct, supra n. 19, have been adopted by most jurisdictions.
86
Strickland, 466 U.S. at 686, 688.
87
Defense Function Standards, supra n. 22 at Standard 4-6.1(b). In 2012, the United States Supreme Court, in Missouri v
Frye, citing to the Department of Justice, Bureau of Statistics, noted that “ninety-four percent of state convictions are the
result of guilty pleas.” Missouri v. Frye, 132 S.Ct. 1399, 1407 (2012).
U s e o f D e l p h i M e t h o d i n A B A S C L A I D P u b l i c D e f e n s e Wo r k l o a d S t u d i e s | 13III. Overview of ABA SCLAID Defender Workload Studies
ABA SCLAID’s application of the Delphi method to public defender workload studies requires two
steps: (1) a system analysis (the “world of is”) and (2) the Delphi process (the “world of should”).
The system analysis data is ultimately compared to the workload standards as determined through
the Delphi method to identify potential gaps, if any, in the current system.
A. The System Analysis: The “World of Is”
The system analysis is an examination of the current and historical workload of the public defense
system under study. The system analysis should include staffing numbers and caseloads for public
defense attorneys going back, if possible, at least three years.88 If the data cannot be gathered directly
from the public defense organization, it may be available from the courts or other relevant agencies.
In states where the public defense organization has comprehensive oversight authority over all
public defense providers and where all providers use case management systems that collect core
data on caseloads, most of the data is relatively easy to collect. In states with less centralized
systems, where public defense providers have little statewide oversight, use different case
management systems with different data entry criteria, or use a large number of contractors or part-
time public defense providers and do not gather information on their non-public defense caseloads,
the data can be significantly harder to collect.
Additionally, to be most effective, this system analysis should include timekeeping data to show how
current public defense attorneys are expending their time.89 Timekeeping data tracks time spent
on specific tasks and the particular type of case for which the task is being done. Timekeeping
data allows for a more robust understanding of the time public defense providers in the jurisdiction
currently spend on case specific work, and the time currently spend on other job requirements,
such as administrative tasks, travel time, training, and supervision, which are not captured in the
Delphi process. Timekeeping also allows for more granular comparisons between historical time
expenditures and the time recommendations that result from the Delphi process.
Issues that arise during the System Analysis and how they were addressed during ABA SCLAID
workload studies are discussed in detail in Section IV(A), below.
B. The Delphi Process: The “World of Should”
In ABA SCLAID workload studies, the Delphi process is utilized to determine the amount of
time attorneys should spend on given cases. To measure this, the Delphi process leverages the
expertise of participants–here criminal defense practitioners in the relevant jurisdiction–to arrive at
a consensus on the two key decisions: (1) the amount of time attorneys should expect to spend on
average on a given Case Task for a typical case of the particular Case Type to provide competent
representation and deliver reasonably effective assistance of counsel under prevailing professional
norms (Time), and (2) in what percentage cases of this Case Type should the particular Case Task
be performed (Frequency).90
88
Three years appears sufficiently long to show trends. See ABA Standing Committee on Legal Aid and Indigent
Defendants, The Louisiana Project: A Study of the Louisiana Public Defender System and Attorney Workload Standards,
at 7-10 (2017), available at https://www.americanbar.org/content/dam/aba/administrative/legal_aid_indigent_defendants/
ls_sclaid_louisiana_project_report.pdf.
89
Timekeeping with sufficient accuracy and consistency to allow for reliable comparisons has proven difficult in several
jurisdictions. For a more robust discussion of whether to conduct timekeeping and how to complete a system analysis
without timekeeping, see infra Section IV(A).
90
See infra Section III(B)(5) for the definitions of Case Tasks and Case Types.
U s e o f D e l p h i M e t h o d i n A B A S C L A I D P u b l i c D e f e n s e Wo r k l o a d S t u d i e s | 14You can also read
