News from FS-EC - DESY Confluence

Page created by Peter Chang
 
CONTINUE READING
News from FS-EC - DESY Confluence
News from FS-EC

Petra Days 2018

     Windows/Linux Upgrades

     Data collection and processing
         • Fast Scans: PiLC
         • 2D Detector Integration
         • Visualization: LaVue
         • Data Processing: ADAPT
         • Central Data Mgt. and Computing
         • Scientific Computing Support

     Sardana: New Tool
News from FS-EC - DESY Confluence
Windows/Linux Upgrades                                       S. Rowoldt, H. Blume

Windows 10 upgrade
• Windows-7 support ends on 14.1.2020
• Status: 50/900 under W-10
• So far, only new PC have been installed with Windows-10
• General migration process will start soon
• Hardware replacements will be necessary (150-200 PCs)

Linux: Debian-9
• 240 PCs
• Rebuild Tango servers
• VME tests have been successful
• Sardana still needs some attention

                     Thorsten Kracht | Experiment Control | July 19, 2017 | Seite 2
News from FS-EC - DESY Confluence
PiLC: Continuous Scans                                                          T. Spitzbart, T. Nunez

                                                               PiLC

                                                                                   Tango

                                 Counts

                                                                               Trigger
                                    Encoder

Trigger modes                                                    New feature: array-defined trigger pattern
• On time, on position                                           • P21: U. Lienert, Th. Baecker
• Start on time then trigger on position                         • Every trigger position is individually defined
• Start on position then trigger on time                         • Controls Perkin Elmer and shutter
• Trigger on position inside a sensitive region                  • Take care of detector ‘after-glow’
• Encoder and counters can be stored                             • Interface: list with trigger-channel, position and trigger-level.

                                  Thorsten Kracht | Petra Days | Feb. 9, 2017 | Seite 3
News from FS-EC - DESY Confluence
PiLC: Applications                                                     T. Spitzbart

Nummer   Beamline   Projekt Titel                                             Ort
     1   P64        4x VFC und 1xPT100 Messung                                PETRA III EX
     2   P65        4X VFC                                                    PETRA III EX
     3   P06        Continuous Scan - SlaveFreq                               PETRA III
     4   P09        XCMD                                                      PETRA III
     5   P06        10x ADC Messung                                           PETRA III
     6   P21        Status: Unbekannt da die Firmware selbst erstellt wurde   PETRA III
     7   P07        Status: Unbekannt da die Firmware selbst erstellt wurde   PETRA III
     8   P03        Trigger generator für 3 Steuerdüsen                       PETRA III
     9   P06        Continuous Scan                                           PETRA III
    10   P11        Continuous Scan                                           PETRA III
    11   P02        Continuous Scan                                           PETRA III
    12   P06        Continuous Scan                                           PETRA III
    13   P10        Shuttersteuerung                                          PETRA III
    14   FLASH      Laser Triggergenerator                                    FLASH
    15   P06        4xPT1000 Messung                                          PETRA III
    16   FS-CXS     LED und Kamera Triggergenerator                           Geb. 25B Keller
    17   P23        Continuous Scan                                           PETRA III EX
    18   P03        Trigger generator für 3 Steuerdüsen                       Schweden
    19   FS-NL      Pumpprobe                                                 Geb 3
    20   P07        Status: Unbekannt da die Firmware selbst erstellt wurde   PETRA III
    21   P03        Continuous Scan - SlaveADC                                PETRA III
    22   P21        array-defined trigger pattern                             PETRA III EX
    23   P06        Continuous Scan - SlaveFreq                               PETRA III
    24   P06        Continuous Scan - SlaveADC                                PETRA III
    25   P01        Continuous Scan                                           PETRA III
    26   FLASH 2    Shuttersteuerung                                          FLASH 2
    27   P06        Continuous Scan - SlaveADC                                PETRA III
    28   P06        1x DADC Messung (Kraftsensor)                             PETRA III
    29   P01        Continuous Scan - SlaveADC                                PETRA III
    30   P11        Continuous Scan - SlaveADC                                PETRA III          PiLC covers
    31   P11        Continuous Scan + LED Triggergenerator                    PETRA III          • Continuous scans
    31   FS-NL      2x Thermosensoren auslesen                                Geb 3
    32   P06        Continuous Scan                                           PETRA III          • Specific solutions
    33   P07        1x DADC Messung (Kraftsensor)                             PETRA III
                                                                                                 20 new modules are in preparation

                                                  Thorsten Kracht | Petra Days | Feb. 9, 2017 | Seite 4
News from FS-EC - DESY Confluence
Fast 2D Detectors

High data rates in sustained operation:
• Detector speed
• Compression: CPU time vs. transfer speed and data volumes
• Detector PC: no. of cores, RAM size, memory bandwidth
• Data transfer: NFS, SMB, HiDRA
• File writing: HDF5/NeXus, TIF, CBF

                  Thorsten Kracht | Experiment Control | July 19, 2017 | Seite 5
News from FS-EC - DESY Confluence
Fast 2D Detectors: HW Compression
                                                                                      Y. Yu
Tests done with AHA374, PCI Express, 40 Gb/s, ZLIB (HDF5 compliant) and Lambda
• 15000 images
• PC: 12 cores, 256 GB RAM
      • Data reception: 4 cores
      • Image processing: 7 cores
• Process chain:
      • raw data -> raw buffer (~180GB) -> image processing (encoding, distortion
         corrections, compression) -> encoded buffer (1GB) -> file write
• HW vs. SW image processing speed: 625 Hz / 205 Hz: Factor 3
• HW vs. SW compression speed: ~ 5
• HW vs. SW compression efficiency:
      • 15 GB (10000 noisy images) -> 4.9/5.2 GB (HW/SW): Factor: 3

                     Thorsten Kracht | Experiment Control | July 19, 2017 | Seite 6
News from FS-EC - DESY Confluence
Tango Integration of Fast 2D Detectors                                                                    Y. Yu, A. Rothkirch

                          Data Rate [GB/s]        10 GE       fps             fSize [MB]       OS     Transfer       Viewer                LaVue

Lambda (12b) HDF5         2.6                     1           2000            1.3              L      NFS,HiDRA      ATK,Taurus, Onda      TS

Lambda2M HDF5             7.8                     2           2000            1.3              L      NFS            ATK, Taurus, Onda     TS

AGIPD HDF5                0.6                     1           3520            0.131            L      HiDRA          ATK, Taurus, Onda     TS

AGIPD1M HDF5              10                      16          3520            2.1              L      n.a.           ATK, Taurus           TS

Jungfrau, HDF5, BIN       0.5                     1           500             1                L      HiDRA          P11-Onda

PCO Fr.Grabber,           0.8                     1           100             8                L      NFS, HiDRA     ATK                   TS,
TIF(PCO), HDF5                                                                                                                             HiDRA
PCO USB (P11), TIF        0.8                     1           100             8                W      SMB, (HiDRA)   Vendor SW             (HiDRA)

Eiger4M, HDF5 (LZ4,       1                       1           750             9/18             L      HDF/http       Albula (low speed)
external filter)
Pilatus2 6M, CBF/TIF      0.15/0.6                            25              6                L*     HiDRA                                HiDRA

Pilatus3 2M, CBF/TIF      0.6/ 2                  1           250             2.5 /11          L*     HiDRA                                HiDRA

Pilatus2 1M, CBF/TIF      0.05/0.2                1           50              1                L*     HiDRA                                HiDRA

Pilatus2 300k, CBF/TIF    0.05/0.2                1           200             0.3              L*     HiDRA                                HiDRA

Perkin Elmer XS 1621      0.25                    1           15              16               W      SMB, (HiDRA)   QXRD                  (HiDRA)

                                     Thorsten Kracht | Experiment Control | July 19, 2017 | Seite 7
Fast 2D Detectors

Future
• Use HiDRA for all 2D detectors
• Complete LaVue integration
• Optimize PC configuration
• HW compression

                     Thorsten Kracht | Experiment Control | July 19, 2017 | Seite 8
2D Visualisation: LaVue                               Ch. Rosemann, J. Kotanski

Tool for monitoring any 2D detector data
• Interfaces:
      • HiDRA
      • ZMQ stream (P06)
      • Eiger4M: http
      • Tango server:
           • LastImageTaken (Lambda)
           • File/Directory name
      • Lima
      • Open-file: NeXus, TIF, CBF
• ROI integration, MacroServer, ROI
   counters

Future developments:
• Support all 2D detectors used in Petra III
• Integrate motor interface
• Implement other user requests:
   projections, etc.

                             Thorsten Kracht | Petra Days | Feb. 9, 2017 | Seite 9
ADAPT                                                              Ch. Rosemann

Python-based data processing toolkit                                                   GUI           User Edits
•   General purpose application
•   Graphical user interface
•   Batch processing
                                                                                       Config File
•   Configuration file: Python dictionary syntax
•   Modular approach, sequential processing
      •    Data reader
                                                                                                              Plugins
      •    Slicing                                                                     Executor
                                                                                       •  Data Mgt
      •    Filtering                                                                                          PSIO          .cbf, .tiff,
                                                                                       •  Control
      •    Azimuthal/radial integration                                                                                     .nx, .spec
                                                                                                              Slicing
      •    Fitting
•   Data model: Blackboard                                                                                    Filtering
•   NeXus compliant
•   Use this framework to create specific applications                                                        Fitting
•   First use case: adaptation of iint, an application that fits
    several peaks, including background and de-spiking for                                                    PSIO             Results
    single and multiple scans.

                              Thorsten Kracht | Petra Days | Feb. 9, 2017 | Seite 10
GPFS Utilization                                         A. Rothkich, IT

MvL 2015-05-08 to 2018-02-22
„gap corrected“; linear regression made for periods (runs) given by DOOR and excluding
machine weeks only. Hence machine days are included, also in numbers given on the right
(range_days and numbersThorsten
                            deduced).
                                  Kracht | Petra Days | Feb. 9, 2017 | Seite 11
Central Data Management                                           A. Rothkich, IT

Core cluster utilization, Petra III data
• 2015: 200 TB (1.1 TiB/day)
• 2016: 480 TB ( 1.9 TiB/day), new detectors
• 2017: 580 TB ( 2.7 TiB/day), new detector, higher utilization
• Upgraded to 2.1 PB in Dec. 2017

Optimize: GPFS capacity vs. residence time vs. cheap storage bandwidth

Future:
• Move beamtime data automatically to dCache after X days (2 copies)
     • ‘X’ depends on the fill-state of GPFS
• Re-stage data on request

•   Extend GPFS to ‘Special Instruments’ and research groups
      • Store large data volumes, archive, fast access from compute nodes (IB), portal access,
         transfer speed, ACLs, coherent directory structure

                       Thorsten Kracht | Petra Days | Feb. 9, 2017 | Seite 12
GPFS: Special Instruments and Research Groups
                                                                              J.-P.Kurz, A. Rothkirch, IT

Special Instruments:
• Container for: Pipe, Pump-probe P08, afm@p08, mobile x-ray tube, Nanotom, large volume
   press, etc.
• For external and internal users
• Option: declaration of substances
• Beamtime allocation: user applies for beamtime, manager schedules
• GPFS path
     • /asap3/SpecialInstruments/gpfs/pipe//data/

Research groups:
• FS-LA, FS-SCS, FS-DS, etc.
• Users are DESY-group members
• Simplified beamtime allocation procedure
• GPFS path
    • /asap3/fs-la/gpfs/laser1//data/

Requires adaptations in DOOR and ASAP3

                     Thorsten Kracht | Petra Days | Feb. 9, 2017 | Seite 13
Central Compute nodes                                 A. Rothkirch, IT

Current state:
• 8 Nodes, Dual CPU with 16 cores each, 512 GB RAM
• 3 Nodes, Dual CPU with 16 cores each, 512 GB RAM, 1 NV-K40X
• Shared among FS staff and external
     • HASY: 8 CPU + 3 GPU
     • PSX: 4 CPU + 1 GPU

Soon:
• 6 Nodes, Dual CPU with 10 cores each, 512 GB RAM
• 7 Nodes, Dual CPU with 10 cores each, 512 GB RAM, 1 x P100, PCIe
• 1 Node, Dual CPU with 10 cores each, 512 GB RAM, 4 x P100, NvLink
• Into Slurm, with option to allocate nodes for beamtimes
     • Relevant Slurm partitions: PS, PSX, P99
     • Maxwell users meeting, 28.3.2018, 14:00 h, room 4a/b

2018:
• Extend compute resources
• Find the best configuration: RAM, cores, GPUs

                     Thorsten Kracht | Petra Days | Feb. 9, 2017 | Seite 14
Scientific Software Support
Under discussion

•   Strengthen the DESY LKII activities
•   Immediate action: hire additional personnel
•   Assuming 500 kEUR in 2018 and 1 MioEUR in 2019

        Year       FH     FS             M              ZN         Central (IT)   Total
        2018        2      3             1               -             2            8
        2019        2      2             1               1             1            7
       Sum till     4      5             2               1             3           15
        2020

•   The new hires will be allocated to the divisions
•   Virtual Scientific Computing Group (VSCG)
      • Communication platform: seminars, workshops, discussion, etc.
      • The communication platform is open to other colleagues working in the field
      • Coordinate with DMA, CDCS

FS
• The new hires will work on specific projects at the beamlines.
• Move to the new tasks when finished, maintaining support.
• Two job offers are open

                        Thorsten Kracht | Petra Days | Feb. 9, 2017 | Seite 15
SardanaMotorMenu

                                                    Purpose:
                                                    • Support alignment: move widget with signal
                                                    • Present selected attributes
                                                    • Support homing: encoder attributes

                                                         $ SardanaMotorMenu.py
                                                          uses online.xml

                                                         $ SardanaMotorMenu.py –tags expert
                                                          select motors by tags

                                                         With signal definition:
                                                         $ SardanaMotorMenu.py –t exp_t01 –c exp_c01

Thorsten Kracht | Petra Days | Feb. 9, 2017 | Seite 16
SardanaMotorMenu: 4 Motors

Work on selected motors only:

$ SardanaMotorMenu.py d1_mot01 d1_mot02 d1_mot03 d1_mot04

                  Thorsten Kracht | Petra Days | Feb. 9, 2017 | Seite 17
SardanaMotorMenu

                                                                   $ SardanaMotorMenu.py

MB1 – Move
MB2 – Attributes
MB3 - EncoderAttributes

          Thorsten Kracht | Petra Days | Feb. 9, 2017 | Seite 18
SardanaMotorMenu: Move Widget

Accessible from MainWidget or (to speed-up) from the command line directly:

$ SardanaMotorMenu.py d1_mot65

                   Thorsten Kracht | Petra Days | Feb. 9, 2017 | Seite 19
SardanaMotorMenu: Move Widget

Motor Selector

                 $ SardanaMotorMenu.py d1_mot65

                                  Thorsten Kracht | Petra Days | Feb. 9, 2017 | Seite 20
SardanaMotorMenu: Move Widget
                                                                                                   Motor can also be moved with
                                                                                                   Key_Left and Key_Right
                                                                                                   Key_Up, Key_Down to change slew

                                                                                                   Change incr., slew or range

• Postscript
• Write .fio

Timer/counter selector also      Invoke Attr. and EncAttr widgets                  Move right by step, incr. or to limit
possible from command line.

                              Thorsten Kracht | Petra Days | Feb. 9, 2017 | Seite 21
SardanaMotorMonitor: Cursor

 Thorsten Kracht | Petra Days | Feb. 9, 2017 | Seite 22
SardanaMotorMenu: Attributes
Selected motor attributes                                                      Encoder attributes

                              Thorsten Kracht | Petra Days | Feb. 9, 2017 | Seite 23
Summary

•   Extend the central compute resources
•   Extend GPFS to all data intense FS activities
•   Implement HiDRA for all 2D detectors
•   LaVue to become the standard visualization tool
•   Extend ADAPT to more use cases
•   First steps towards a general scientific software support

                    Thorsten Kracht | Petra Days | Feb. 9, 2017 | Seite 24
You can also read