TRENDS IN HPC AND DATA CENTER POWER, PACKAGING, AND COOLING - MICHAEL K PATTERSON, PHD, PE, DCEP POWER, PACKAGING, AND COOLING INTEL, TECHNICAL ...

Trends in HPC and Data Center Power,
Packaging, and Cooling
Michael K Patterson, PhD, PE, DCEP
Power, Packaging, and Cooling
Intel, Technical Computing Systems Architecture and Pathfinding

Legal Disclaimer
Today’s presentations contain forward-looking statements. All statements made that are not historical facts are subject to a number of risks and uncertainties, and actual results
may differ materially.

NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN
INTEL’S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED
WARRANTY, RELATING TO SALE AND/OR USE OF INTEL® PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE,
MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. INTEL PRODUCTS ARE NOT INTENDED FOR USE IN
MEDICAL, LIFE SAVING, OR LIFE SUSTAINING APPLICATIONS.

Intel does not control or audit the design or implementation of third party benchmarks or Web sites referenced in this document. Intel encourages all of its customers to visit the
referenced Web sites or others where similar performance benchmarks are reported and confirm whether the referenced benchmarks are accurate and reflect performance of
systems available for purchase.

Intel processor numbers are not a measure of performance. Processor numbers differentiate features within each processor family, not across different processor families. See
www.intel.com/products/processor_number for details.

Intel, processors, chipsets, and desktop boards may contain design defects or errors known as errata, which may cause the product to deviate from published specifications.
Current characterized errata are available on request.

Intel, Intel Xeon, Intel Core microarchitecture, and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other
countries.
*Other names and brands may be claimed as the property of others
Copyright © 2011, Intel Corporation. All rights reserved.

Overview
Data Center Choices
IT & HPC Drivers
Metrics
Power
Cooling
Density
Resources

Data Center Choices
Air cooling vs liquid cooling?
Hot-aisle vs Cold-aisle?
Raised floor vs concrete?
Bricks and mortar vs containers?
New building vs existing?
UPS as part of HPC?
Rack density?
Feed from above or below?
1st cost or TCO?
Reliability level – Tier 1 To Tier IV?

HPC data centers; many hurdles
• Power and performance challenges exist to get to Exascale
• Preparing for Exascale? Aren't we a little early?
• The key facts…
 • Data Center life cycle – 10-15 years
 • HPC cluster life cycle – 3-5 years
• Leads to interesting results…

Exascale by 2020
 Business-as-usual ~47 MW
 (2X perf/watt every 16 months)
(yes the trends have slowed, but still making progress)

NCAR - Home to an Exaflop SuperComputer

NCAR Yellowstone - New supercomputing center in Wyoming

Exascale at #1 by 2020
NCAR will be 10 years old in 2022
 Exascale at #500 by 2027

 NCAR at 10 years old
 2021

 2023

 2025

 2027
 Data Centers should be built to last 10-15 years

Integration
 Enabled by leading edge process technologies

 Integrated Today Possible Tomorrow**

System level benefits in cost, power, density, scalability &
 performance
 **Future options are forecasts and subject to change without notice.

Intel’s Scalable System Framework
A Configurable Design Path Customizable for a Wide Range of HPC & Big Data Workloads

 Small Clusters Through Supercomputers

 Compute Memory/Storage
 Compute and Data-Centric Computing
 Fabric Software Standards-Based Programmability

 Intel Silicon
 On-Premise and Cloud-Based
 Photonics

 Intel® Xeon® Processors Intel® True Scale Fabric Intel® SSDs Intel® Software Tools
 Intel® Xeon Phi™ Intel® Omni-Path Intel® Lustre-based Solutions HPC Scalable Software Stack
 Coprocessors Architecture
 Intel® Silicon Photonics Intel® Cluster Ready Program
 Intel® Xeon Phi™ Processors Intel® Ethernet Technology
 11

Intel SSF enables Higher Performance & Density

A formula for more performance….

advancements in CPU architecture
  advancements in process technology
  integrated in-package memory
  integrated fabrics with higher speeds
  switch and CPU packaging under one roof
  all tied together with silicon photonics

 = much higher performance & density

 12

THE FUTURE

The Most Advanced Supercomputer Ever Built
An Intel-led collaboration with ANL and Cray to accelerate discovery & innovation

 >180 PFLOPS 18X higher
 (option to increase up to 450 PF)
 performance*
 >50,000 nodes Prime Contractor
 13MW >6X more energy
 2018 delivery efficient*
 Subcontractor
Source: Argonne National Laboratory and Intel. *Versus ANL’s current biggest system named MIRA (10PFs and 4.8MW)
Other names and brands may be claimed as the property of others. 14

Aurora | Built on a Powerful Foundation
Breakthrough technologies that deliver massive benefits
 Compute Interconnect File System

3rd Generation 2nd Generation Intel®
Intel® Xeon Phi™ Intel® Omni-Path *

 >17X performance† >20X faster† >3X faster†
 FLOPS per node >500 TB/s bi-section bandwidth >1 TB/s file system throughput

 >12X memory bandwidth† >2.5 PB/s aggregate node link
 bandwidth
 >5X capacity†
 >30PB/s aggregate >150TB file system capacity
 in-package memory bandwidth

 Integrated Intel® Omni-Path Fabric
 Processor code name: Knights Hill

 † Comparisons are versus Mira—Argonne National Laboratory’s current largest HPC
Source: Argonne National Laboratory and Intel
*Other names and brands may be claimed as the property of others. system, Mira. See Aurora Fact Sheet for details 15

Aurora

Processor Xeon Phi™ Knights Hill

Nodes >50,000

Performance 180 PF Peak

Power 13 MW

Space ~3000 sq ft (~280 m2)

Cooling Direct Liquid Cooling

Efficiency >13 GF/w

 All the details: Aurora Fact Sheet at intel.com
 http://www.intel.com/content/www/us/en/high-performance-
 computing/aurora-fact-sheet.html?wapkw=aurora

How did we do this?
 In package memory
  Closer to CPU, stacked memory
 Fabric Integration
  Package connectivity
 Advanced Switches with higher radix and higher speeds
  Closer integration of compute and switch
 Silicon Photonics
  Low cost, outstanding performance but thermal
 challenges do exist
 All this drives changes

 17

So what have we learned over the last three years?
Todays focus is on Power, Packaging, and Cooling (PPC)
 Metrics
  How do we measure and compare?
 Power
  400Vac, 3ph, >100 kW / cabinet for the very high end
 Packaging
  High density computing – significant computing in a small package
  Weight becomes a key design parameter
 Cooling
  Liquid cooling; good for some. Cooler is better, to a point
  Aurora ~100% liquid cooled
  Air cooling still very core to HPC

The Data center

 Data Center
 Chiller CRAC
 Plant Unit

 IT
 Utility UPS PDU
 Equipment

 Site

PUE
 
 =
 
 • Introduced in 2006 by Malone and Belady
 • Developed and agreed to by EU Code of Conduct, DOE,
 EPA, Green Grid, ASHRAE, etc…
 • Has led Energy Efficiency drive in Data Centers
 • PUE Average in 2007 ~ 2.5
 • Best in Class 2016:
 NREL= 1.06, LRZ= 1.15, NCAR~1.2,
 ORNL= 1.25, TU Dresden < 1.3

PUE – simple and effective

PUEs: Reported and Calculated
 PUE
 Global bank’s best data center (of more than 100) 2.25
 EPA Energy Star Average 1.91
 Intel average >1.80
 Intel Jones Farm, Hillsboro 1.41
 ORNL 1.25
 T-Systems & Intel DC2020 Test Lab, Munich 1.24
 Google 1.16
 Leibniz Supercomputing Centre (LRZ) 1.15
 Containers 1.1-1.6
 National Center for Atmospheric Research (NCAR) 1.10
 Yahoo, Lockport 1.08
 Facebook, Prineville 1.07
 National Renewable Energy Laboratory (NREL) 1.06

PUEs: Reported and Calculated Liquid cooling is
 required for density,
 PUE
 Global bank’s best data center (of more than 100)
 but not necessarily for
 2.25
 efficiency
 EPA Energy Star Average 1.91
 Intel average >1.80
 Intel Jones Farm, Hillsboro 1.41 A-FC
 ORNL 1.25 LC
 T-Systems & Intel DC2020 Test Lab, Munich 1.24 A-FC
 Google 1.16 A-FC
 Leibniz Supercomputing Centre (LRZ) 1.15 LC
 Containers 1.1-1.6
 National Center for Atmospheric Research (NCAR) 1.10 LC
 Yahoo, Lockport 1.08 A-FC
 Facebook, Prineville 1.07 A-FC
 National Renewable Energy Laboratory (NREL) 1.06 LC

but PUE isn't perfect, consider…..
 + + ( + )
 =
 ( + )

 data center

 UPS & PDU
 pwr

 fan IT

 fan

Three variations…
 a) + + ( + )
both =
 ( + )
fans

 b) + ( + )
 =
 IT ( + )
fans
only

 c) + + 
bldg =
 
fan
only PUEb < PUEa < PUEc but is (b) best?
 We don’t know….

Can we define a “server-PUE”? Maybe ITUE?
 + + + + 
 = = =
 
 Data Center Server
 Power dist UPS, line losses, PDUs PSU, VRs, board losses
 losses
 Cooling losses Chiller, CRAC, Pumps, Fans Fans, Pumps
 Misc losses Security, Lighting, Building Indicators, Platform
 Control Control
 IT Servers, Storage, Network Processor, Memory, Disk
 + + + + 
 = =
 
 ℎ 
 ITUE = ℎ 

ITUE
 Cooling (j)

 (f)
 CPU/
 Wall Mem/
 Drive

 (g)
 PSU (h) VRs (i)

 ℎ 
 = =
 ℎ 

The next step…
PUE and ITUE are both:
 dimensionless ratios
 Represent the burden or “tax” of infrastructure
 “1” is ideal, values larger than 1 are worse
 Values less than 1 are not allowed

 So why not:

 = 

TUE

 + 
 = = = =
 
 + 
 = × =
 

Does it work?
 a)
 both + + + 
 fans =
 
 b)
 IT + + 
 =
 fans 
 only

 c) + + 
 bldg =
 
 fan
 only The lowest TUE yields the lowest energy
 use. Yes, it works!

ITUE / TUE
 • Paper available
 • email me or from ISC 13
 • Best Paper Award at ISC

 • Use the metric!
 • Ask for projected ITUE in future
 procurements
 • Good cluster to cluster efficiency
 comparison
 • Begin to develop monitoring strategy

 • Be aware of limits
 • Does not include workload / output
 • Difficult to use on older machines
 • Don’t ask for everything; likely to expensive

 31

Power
Trends in the very high end….
 Power now 480 Vac 3ph (400 Vac in Europe)
 >100 kW / cabinet
 In-cabinet 380 Vdc for optimized delivery
 Power management and power monitoring allows optimized performance and
 efficiency

 More typical HPC
  400 Vac 3ph, 230Vac 1ph
  48 Vdc in the rack can reduce weight, cost, and size

 HVDC (380 Vdc) is an option; primary reasons why are 1st cost and renewables

Power Delivery Challenges in the horizon
Variable Power Cap
  Several reasons
 − Peak Shedding
 − Reduction in renewable energy

Power rate of change
  Ex: Hourly or Fifteen minute average in platform power should not exceed by
 X MW.
Controlled Power Ramp up/down – economic or technical issues
  Challenge to do this at a reasonable cost and with energy efficient
 mechanisms

Power
Europe primary power
• 400 Vac 3ph
• High density racks could use 3ph, 1ph if not high power; better options for PSUs (PUE/ITUE)
• Likely that most storage and system racks would do well on 230 Vac 1ph
• Consider rating schemes for equipment (PSU): Platinum, Gold, Silver, etc… (ITUE)
• Board power: same direction, higher cost components very often have a good ROI (ITUE)
UPS (Uninterruptable power supply)

• Generally HPC would rather spend money on compute than UPS, generally European power
 quality is good enough without
• Please don’t use UPS for power quality reasons.  Also they waste 2-5%. (PUE)
• Do use UPS and redundant feeds for fabric and storage and service nodes
Power Management
• Tools available, must be built for the site needs

Packaging
Rack and cluster weight and density
 Strong correlation between weight and power
  Some studies have shown kg/kW is ~constant across rack size
  Goal is to reduce this ratio
 Packaging
  High density computing – network topology optimization and high node count per rack (lots
 of power) make for dense cabinets
 Rack weight density
  Design limit: Floor tiles at 500 lbs/sf ~ 2500 kg/m2 for high end.  …more than many DCs
 White space vs utility space
  Compute density increasing, infrastructure support equipment is not
 What's the trend for machine room area?

I must need a huge data center for PetaScale and
ExaScale computing – Right?

Video Credit to Helmut Satzger, LRZ, Munich, thanks for sharing! 37

38

Do I need a huge data center?
 • Facility area for Compute Cluster does not have to
 be huge. Significant compute density in small
 packages
 • At Aurora density, the 3.2 LRZ PF fits in 5 m2

 • Don’t forget:
 • If Storage is going to be large then you will
 need additional floor space.
 • If you are going to be using Xeon instead of
 Xeon Phi then you may need additional floor
 space
 • Utility & infrastructure space continues to grow

 Rack density (kW/rack & kg/m2) have a wide range of
 choices, but the local data center may restrict these
 39

Cooling

Why liquid? Why Air?
 Power per node continues to rise
 Rack density limits airflow path
 But air-cooling can cost less
 Increased thermal performance of liquid (vs air) allows more
 free-cooling
  Thermal resistance from chip to liquid in a cold plate is smaller
 than chip to air over a heat sink

Cooling
Air Cooling
• ASHRAE A1 thru A4
• Containment a “must” for good PUE and low TCO
• Hot Aisle and Cold Aisle an operational choice, not an efficiency choice
• Free –air cooling should always be checked for applicability
 • Corrosion a real issue depending on air-quality
• Air-Cooling limits rack density, but good performance density can still be had

• If you do air-cooling in a new data center; the VERY FIRST MOST IMPORTANT
 consideration is to design the building around the airflow path. Get that done,
 then bring in the building architects. 

Class

A2

Why containment?

 Recirculation
 Excess Air
 Poor IT temperature
 IT Rack

 IT Rack

 CRAH
 With out

 No mixing
 Less airflow
 Higher DT
 With

Airflow management is the number one tool in
your toolbox for data center improvement – it
can solve more problems than any other tool!

 CERN cold aisle containment

Cooling
Liquid Cooling
• ASHRAE W1 thru W4
• Many different varieties, they have VERY different performance, cost, and
 efficiency results
• Water quality is an important issue
 • Consult ASHRAE guide; monitor!
• Immersion cooling is not on our roadmap, we still keep current but issues
 exist
 • Oil immersion
 • Two-phase immersion
• Liquid cooling can offer better performance, better reliability
 • High density systems (Aurora, etc, are fully liquid cooled)

All ASHRAE work has been
incorporated into the 2nd
Edition.

Tip: 2nd Edition now available for
purchase in the ASHRAE bookstore.

More important tip:
Chap 5 covers Facility water (FWS)
Chap 6 covers IT loop water (TCS)
These are very different! Specify the
right water.

 47

2011 ASHRAE Liquid-Cooled Thermal Guidelines
 Typical Infrastructure Design
 Facility Supply IT Equipment
 Classes Supplemental
 Main Cooling Water Temp (C) Availability
 Cooling
 Equipment
 Equipment
 W1 Water-side 2 – 17
 Chiller/Cooling
 Economizer Now available
 W2 Tower 2 – 27
 Chiller
 W3 Cooling Tower Chiller 2 – 32
 Becoming
 Water-side available,
 Economizer (with dependent on
 W4 Nothing 2 – 45
 drycooler or cooling future demand
 tower)
 Building Heating Cooling Tower
 W5 > 45 Not for HPC
 System

 Required Cooling Infrastructure: Balance of
 Silicon/Datacenter

System Definitions – all different, all about how
close the liquid gets to the components
 Liquid Enhanced
 Air Cooling Air Cooling Liquid Cooling

 Facility Facility Facility

 Rack Rack Rack

 Radiator
CPU Other Other CPU Other

 Liquid Rack Cooling CPU Hybrid Liquid Cooling
 Chassis
 Facility Facility

 Key:
 Liquid Rack
 Rack
 Air
 Server
CPU Other Chassis CPU Other

Liquid Cooling Technologies
Local Immersion
 Coldplate Node-Level
Pump-
 with Remote Coldplate
Coldplate
 Pump

A proposal….
• As a starting point, use the coolest water you can make without a chiller
• Always be above the dewpoint (to prevent condensation in the machine)
• Cooler temperatures promote:
  Lower leakage
  More turbo frequencies
  Higher stability
  More time to recover in an upset condition
  Better reliability
  Reduced flow rates

Note - May consume more water, not applicable if after heat recovery

W2

ASHRAE
TC 9.9 Committee
http://tc99.ashraetcs.org/
Books
https://www.ashrae.org/resources--
publications/bookstore/datacom-series

EE HPC WG
http://eehpcwg.llnl.gov/
Hot for Warm Water Cooling
http://eetd.lbl.gov/sites/all/files/publications/lbnl-5128e.pdf

The Green Grid
http://www.thegreengrid.org/
http://www.thegreengrid.org/en/Global/Content/Tools/NAme
ricanFreeCoolingTool

EU Code of Conduct for Data Centres
http://iet.jrc.ec.europa.eu/energyefficiency/ict-codes-
conduct/data-centres-energy-efficiency

Summary
• Data Center Design is straightforward, but can be daunting if not fully
 understood, unfortunately still very site-dependent
• Resources are available!
• Modular build out is best; plan for the end state, provision just for today
• PPC for HPC
 • Power delivery at higher voltages with less redundancy than Enterprise
 • Density has value, but Packaging can challenge most data centers
 • Air and Liquid Cooling have their place, choose the right one for performance and
 value

Thanks for your attention

 Questions?
 michael.k.patterson@intel.com