Highlights of HEPIX virtual workshop - Oct. 15, 2020 - Indico

Highlights of HEPIX virtual
        workshop
        Oct. 15, 2020

HEPIX Statistics
•   321 registered participants
     – ~90 connected in morning sessions and ~70 connected in evening sessions
     – 27 countries represented
•   45 presentations over 4 days
     – 9 out of 11 North American presentations came by BNL and 2 from Univ. of Michigan (GLT2)
     – Sadly, no other DOE lab or U.S. institute made a presentation
•   Indico link to presentations here:
     – https://indico.cern.ch/event/898285/timetable/#all.detailed
•   Sessions split between 9am-12noon and 5-8pm (CEST) Monday-Thursday to
    accommodate participants over multiple time zones
•   Conducted virtually via Zoom
     – Link (passcode-protected) only available to registered participants to avoid Zoom bombing
       (blocked one person)
     – All sessions recorded
     – Modified organizational effort—3 people per session (chair, timekeeper/comment watcher
       and video recorder)
•   Normally, we would have ~65 presentations and ~100 participants at in-person
    meetings spread over 4.5 days

Site Reports track
• 8 presentations (3 from Asia, 4 from Europe and BNL)
• CERN released Indico 2.3 (many external contributions,
  BNL/FNAL on upcoming v3.x)
• IHEP migrated from Puppet 5 to 6
• BNL presented updates on new data center, support
  for NSLS-II, Drupal CMS, BNLBox, ELK, Digital
  Repositories and InCommon IDP (among other topics)
• Diamond (UK Light Source) has a regular presence at
  HEPIX
   – Meeting with other Photon Science facilities (including
     BNL) on Friday @ 11 am – ask me for Zoom details if
     interested

KEKCC

• KEK completed transition to newer system (100% replacement
  every ~4 years)
    – concern that demand is outstripping resources
    – looking at cloud resources

ASGC

• ASGC uses DiCOS (Distributed Cloud Operating System) as scalable
  and flexible platform to support its multi-disciplinary activities

PIC

• PIC reported a plan to migrate tape storage from
  Oracle SL8500 to IBM TS4500

DESY

• DESY reported increasing demand for (disk and tape)
  storage for its Photon Sciences program

End-User IT Services
• 7 presentations (3 by BNL)
• Carlos presented on Digital Repositories at
  SDCC, along with plans to migrate to
  InvenioRDM in 2021
• Chris/Louis discussed CMS migration to Drupal
  and in-progress work with various websites
  (webdocs, sPHENIX, USATLAS, etc)

Virtual Meetings

• Survey on experiences with virtual meetings

ccPortal

• Python-based system to replace paper-based application forms
• Production roll-out in Sep. 2020, fully implemented in November

BNLBox

Network & Security track
• 8 presentations (one by BNL)
• Informative security update from CERN
   – Increase in Phishing, Ransonware, Zoom bombing and HPC systems
   – By-product of work from home due to COVID-19 guidelines
• Increasing adoption of PerfSONAR in Europe/North America for
  network monitoring—ELK analytics demo at upcoming topical
  meeting
• RNT WG presented on structure & activities in potential DC network
  designs for the HL era—potentially become part of current
  TechWatch WG
• Shigeki’s presentation described recent AAI and FIM activities
   – Support for AD and SDCC for authentication
   – Joined InCommon as IDP in May 2020
   – Change in password policy

News from the IPv6 WG

Recent Security Incident

Simulated phishing campaign at CERN

Computing & Batch Services track
• 2 presentations
• Continued progress towards replacement of HS06
  with HEP-Score
  –   Usage of containers to run benchmark
  –   Includes tests at HPC centers (ie, SDSC)
  –   Uses HEP-specific (mainly LHC) codes
  –   GPU benchmarking included in software suite
• Defining policy for adoption of HEP-Score as
  benchmark (pledges, accounting, procurements,
  etc)
• HEP-Score v1.0 to be released in coming weeks

HTCondor and ML algorithms

•   Create ML algorithm to estimate wait-time for Condor jobs
•   Algorithm based on job and machine ads
•   Integrate this to newly-deployed user portal @ IN2P3

HEPS CC

• New LS being built in China (ETA is 2024)
• Tasked with designing online and offline systems for entire
  experimental lifecycle

Storage & Filesystems
• 9 presentations (one by BNL)
• CVMFS updates
   – Support for containers (Singularity, Docker, Podman, etc)
   – Migration to S3 for Stratum 0 and dedicated caches (ie, for ATLAS)
• BNL presented on thought process in choosing IBM to replace
  Oracle silo technology but still keep LTO media
   – IBM cheaper than SpectraLogic
   – IBM native tape media faster but more expensive than LTO
• Evaluation of Lustre/ZFS for LCLS-II @ SLAC
   – To address I/O rates 3x higher than LCLS
   – Also discussed deployment of HA-Lustre with Kubernetes
• dCache presented on integrating a more secure approach to shared
  storage in the cloud—leverage dCache REST interface

XCache @ IN2P3

• Test of Xcache for data access from remote sites
• IN2P3 involved in LSST and HL-LHC activities

FTS update

• FTS v3.10.0 available in Nov. 2020

Basic IT Services
• 4 presentations (one by BNL)
• An update on CERN’s MALT project to migrate to
  open-source software and minimize use (and
  cost) of commercial solutions
  – Affects AAI, mail, calendaring, collaborative tools, etc
  – Estimated completion by Oct. 2021
• Matt reported on the expanding use of ELK to
  monitor services such as BNLBox and Globus—
  plan to expand in the near future to other
  services

MALT Project

IT Facilities & Business Continuity
• 2 presentations
• CERN has had several internal discussions
  – Categorize services (emergency, essential, basic, etc),
    inter-dependencies and staff roles
  – Potential impact off-site (ie, CVMFS, Indico, etc)
  – Open invitation for discussions with other institutions
• CERN using ELK for anomaly detection
  – Use ML algorithms
  – Huge footprint (36 clusters and ~430 nodes)

Clouds, Grids & Virtualization
• 6 presentations (2 by BNL)
• CERN’s OpenStack cloud installation
   – Re-tasked some Wigner systems for LHCb
   – Now includes GPU-based system
   – Hardware refresh underway
• BNL presentation by Chris Hollowell on Podman
   – Default for RHEL/CentOS 8, alternative to Docker
   – Potentially useful for NSLS-II
• BNL presentation on virtual online computing
   – Remote data center model
   – Flexible (re-task between online/offline on demand)

Kubernetes @ CERN

• One of the CERN use cases

Security Risks with Virtual Meetings
• COVID-19 related impact
   – Malicious attacks tailored to videoconferencing tools
   – Ransomware attacks on the rise
   – Online events are perfect watering holes
• Reduced protection from campus security
   – Use VPN is possible
• Security test (a HEPIX tradition)
   – registered fake people (success!)
   – Careful with links on chat box

Summary
• Peter van der Reest (DESY) replaces Helge
  Meinhard (CERN) as European HEPIX co-chair
• Next meeting scheduled for Taipei, in lieu of
  the Spring 2020 meeting that was cancelled
  due to the COVID-19 pandemic
  – Seems unlikely to be a in-person meeting, so we
    are preparing for another virtual meeting
• Future meeting hosts include Strasbourg,
  Karlsruhe, Lugano, Univ. of Oklahoma, etc