News from FS-EC - DESY Confluence
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
News from FS-EC
Petra Days 2018
Windows/Linux Upgrades
Data collection and processing
• Fast Scans: PiLC
• 2D Detector Integration
• Visualization: LaVue
• Data Processing: ADAPT
• Central Data Mgt. and Computing
• Scientific Computing Support
Sardana: New Tool
Windows/Linux Upgrades S. Rowoldt, H. Blume
Windows 10 upgrade
• Windows-7 support ends on 14.1.2020
• Status: 50/900 under W-10
• So far, only new PC have been installed with Windows-10
• General migration process will start soon
• Hardware replacements will be necessary (150-200 PCs)
Linux: Debian-9
• 240 PCs
• Rebuild Tango servers
• VME tests have been successful
• Sardana still needs some attention
Thorsten Kracht | Experiment Control | July 19, 2017 | Seite 2
PiLC: Continuous Scans T. Spitzbart, T. Nunez
PiLC
Tango
Counts
Trigger
Encoder
Trigger modes New feature: array-defined trigger pattern
• On time, on position • P21: U. Lienert, Th. Baecker
• Start on time then trigger on position • Every trigger position is individually defined
• Start on position then trigger on time • Controls Perkin Elmer and shutter
• Trigger on position inside a sensitive region • Take care of detector ‘after-glow’
• Encoder and counters can be stored • Interface: list with trigger-channel, position and trigger-level.
Thorsten Kracht | Petra Days | Feb. 9, 2017 | Seite 3
PiLC: Applications T. Spitzbart
Nummer Beamline Projekt Titel Ort
1 P64 4x VFC und 1xPT100 Messung PETRA III EX
2 P65 4X VFC PETRA III EX
3 P06 Continuous Scan - SlaveFreq PETRA III
4 P09 XCMD PETRA III
5 P06 10x ADC Messung PETRA III
6 P21 Status: Unbekannt da die Firmware selbst erstellt wurde PETRA III
7 P07 Status: Unbekannt da die Firmware selbst erstellt wurde PETRA III
8 P03 Trigger generator für 3 Steuerdüsen PETRA III
9 P06 Continuous Scan PETRA III
10 P11 Continuous Scan PETRA III
11 P02 Continuous Scan PETRA III
12 P06 Continuous Scan PETRA III
13 P10 Shuttersteuerung PETRA III
14 FLASH Laser Triggergenerator FLASH
15 P06 4xPT1000 Messung PETRA III
16 FS-CXS LED und Kamera Triggergenerator Geb. 25B Keller
17 P23 Continuous Scan PETRA III EX
18 P03 Trigger generator für 3 Steuerdüsen Schweden
19 FS-NL Pumpprobe Geb 3
20 P07 Status: Unbekannt da die Firmware selbst erstellt wurde PETRA III
21 P03 Continuous Scan - SlaveADC PETRA III
22 P21 array-defined trigger pattern PETRA III EX
23 P06 Continuous Scan - SlaveFreq PETRA III
24 P06 Continuous Scan - SlaveADC PETRA III
25 P01 Continuous Scan PETRA III
26 FLASH 2 Shuttersteuerung FLASH 2
27 P06 Continuous Scan - SlaveADC PETRA III
28 P06 1x DADC Messung (Kraftsensor) PETRA III
29 P01 Continuous Scan - SlaveADC PETRA III
30 P11 Continuous Scan - SlaveADC PETRA III PiLC covers
31 P11 Continuous Scan + LED Triggergenerator PETRA III • Continuous scans
31 FS-NL 2x Thermosensoren auslesen Geb 3
32 P06 Continuous Scan PETRA III • Specific solutions
33 P07 1x DADC Messung (Kraftsensor) PETRA III
20 new modules are in preparation
Thorsten Kracht | Petra Days | Feb. 9, 2017 | Seite 4
Fast 2D Detectors
High data rates in sustained operation:
• Detector speed
• Compression: CPU time vs. transfer speed and data volumes
• Detector PC: no. of cores, RAM size, memory bandwidth
• Data transfer: NFS, SMB, HiDRA
• File writing: HDF5/NeXus, TIF, CBF
Thorsten Kracht | Experiment Control | July 19, 2017 | Seite 5
Fast 2D Detectors: HW Compression
Y. Yu
Tests done with AHA374, PCI Express, 40 Gb/s, ZLIB (HDF5 compliant) and Lambda
• 15000 images
• PC: 12 cores, 256 GB RAM
• Data reception: 4 cores
• Image processing: 7 cores
• Process chain:
• raw data -> raw buffer (~180GB) -> image processing (encoding, distortion
corrections, compression) -> encoded buffer (1GB) -> file write
• HW vs. SW image processing speed: 625 Hz / 205 Hz: Factor 3
• HW vs. SW compression speed: ~ 5
• HW vs. SW compression efficiency:
• 15 GB (10000 noisy images) -> 4.9/5.2 GB (HW/SW): Factor: 3
Thorsten Kracht | Experiment Control | July 19, 2017 | Seite 6
Tango Integration of Fast 2D Detectors Y. Yu, A. Rothkirch
Data Rate [GB/s] 10 GE fps fSize [MB] OS Transfer Viewer LaVue
Lambda (12b) HDF5 2.6 1 2000 1.3 L NFS,HiDRA ATK,Taurus, Onda TS
Lambda2M HDF5 7.8 2 2000 1.3 L NFS ATK, Taurus, Onda TS
AGIPD HDF5 0.6 1 3520 0.131 L HiDRA ATK, Taurus, Onda TS
AGIPD1M HDF5 10 16 3520 2.1 L n.a. ATK, Taurus TS
Jungfrau, HDF5, BIN 0.5 1 500 1 L HiDRA P11-Onda
PCO Fr.Grabber, 0.8 1 100 8 L NFS, HiDRA ATK TS,
TIF(PCO), HDF5 HiDRA
PCO USB (P11), TIF 0.8 1 100 8 W SMB, (HiDRA) Vendor SW (HiDRA)
Eiger4M, HDF5 (LZ4, 1 1 750 9/18 L HDF/http Albula (low speed)
external filter)
Pilatus2 6M, CBF/TIF 0.15/0.6 25 6 L* HiDRA HiDRA
Pilatus3 2M, CBF/TIF 0.6/ 2 1 250 2.5 /11 L* HiDRA HiDRA
Pilatus2 1M, CBF/TIF 0.05/0.2 1 50 1 L* HiDRA HiDRA
Pilatus2 300k, CBF/TIF 0.05/0.2 1 200 0.3 L* HiDRA HiDRA
Perkin Elmer XS 1621 0.25 1 15 16 W SMB, (HiDRA) QXRD (HiDRA)
Thorsten Kracht | Experiment Control | July 19, 2017 | Seite 7Fast 2D Detectors
Future
• Use HiDRA for all 2D detectors
• Complete LaVue integration
• Optimize PC configuration
• HW compression
Thorsten Kracht | Experiment Control | July 19, 2017 | Seite 82D Visualisation: LaVue Ch. Rosemann, J. Kotanski
Tool for monitoring any 2D detector data
• Interfaces:
• HiDRA
• ZMQ stream (P06)
• Eiger4M: http
• Tango server:
• LastImageTaken (Lambda)
• File/Directory name
• Lima
• Open-file: NeXus, TIF, CBF
• ROI integration, MacroServer, ROI
counters
Future developments:
• Support all 2D detectors used in Petra III
• Integrate motor interface
• Implement other user requests:
projections, etc.
Thorsten Kracht | Petra Days | Feb. 9, 2017 | Seite 9ADAPT Ch. Rosemann
Python-based data processing toolkit GUI User Edits
• General purpose application
• Graphical user interface
• Batch processing
Config File
• Configuration file: Python dictionary syntax
• Modular approach, sequential processing
• Data reader
Plugins
• Slicing Executor
• Data Mgt
• Filtering PSIO .cbf, .tiff,
• Control
• Azimuthal/radial integration .nx, .spec
Slicing
• Fitting
• Data model: Blackboard Filtering
• NeXus compliant
• Use this framework to create specific applications Fitting
• First use case: adaptation of iint, an application that fits
several peaks, including background and de-spiking for PSIO Results
single and multiple scans.
Thorsten Kracht | Petra Days | Feb. 9, 2017 | Seite 10GPFS Utilization A. Rothkich, IT
MvL 2015-05-08 to 2018-02-22
„gap corrected“; linear regression made for periods (runs) given by DOOR and excluding
machine weeks only. Hence machine days are included, also in numbers given on the right
(range_days and numbersThorsten
deduced).
Kracht | Petra Days | Feb. 9, 2017 | Seite 11Central Data Management A. Rothkich, IT
Core cluster utilization, Petra III data
• 2015: 200 TB (1.1 TiB/day)
• 2016: 480 TB ( 1.9 TiB/day), new detectors
• 2017: 580 TB ( 2.7 TiB/day), new detector, higher utilization
• Upgraded to 2.1 PB in Dec. 2017
Optimize: GPFS capacity vs. residence time vs. cheap storage bandwidth
Future:
• Move beamtime data automatically to dCache after X days (2 copies)
• ‘X’ depends on the fill-state of GPFS
• Re-stage data on request
• Extend GPFS to ‘Special Instruments’ and research groups
• Store large data volumes, archive, fast access from compute nodes (IB), portal access,
transfer speed, ACLs, coherent directory structure
Thorsten Kracht | Petra Days | Feb. 9, 2017 | Seite 12GPFS: Special Instruments and Research Groups
J.-P.Kurz, A. Rothkirch, IT
Special Instruments:
• Container for: Pipe, Pump-probe P08, afm@p08, mobile x-ray tube, Nanotom, large volume
press, etc.
• For external and internal users
• Option: declaration of substances
• Beamtime allocation: user applies for beamtime, manager schedules
• GPFS path
• /asap3/SpecialInstruments/gpfs/pipe//data/
Research groups:
• FS-LA, FS-SCS, FS-DS, etc.
• Users are DESY-group members
• Simplified beamtime allocation procedure
• GPFS path
• /asap3/fs-la/gpfs/laser1//data/
Requires adaptations in DOOR and ASAP3
Thorsten Kracht | Petra Days | Feb. 9, 2017 | Seite 13Central Compute nodes A. Rothkirch, IT
Current state:
• 8 Nodes, Dual CPU with 16 cores each, 512 GB RAM
• 3 Nodes, Dual CPU with 16 cores each, 512 GB RAM, 1 NV-K40X
• Shared among FS staff and external
• HASY: 8 CPU + 3 GPU
• PSX: 4 CPU + 1 GPU
Soon:
• 6 Nodes, Dual CPU with 10 cores each, 512 GB RAM
• 7 Nodes, Dual CPU with 10 cores each, 512 GB RAM, 1 x P100, PCIe
• 1 Node, Dual CPU with 10 cores each, 512 GB RAM, 4 x P100, NvLink
• Into Slurm, with option to allocate nodes for beamtimes
• Relevant Slurm partitions: PS, PSX, P99
• Maxwell users meeting, 28.3.2018, 14:00 h, room 4a/b
2018:
• Extend compute resources
• Find the best configuration: RAM, cores, GPUs
Thorsten Kracht | Petra Days | Feb. 9, 2017 | Seite 14Scientific Software Support
Under discussion
• Strengthen the DESY LKII activities
• Immediate action: hire additional personnel
• Assuming 500 kEUR in 2018 and 1 MioEUR in 2019
Year FH FS M ZN Central (IT) Total
2018 2 3 1 - 2 8
2019 2 2 1 1 1 7
Sum till 4 5 2 1 3 15
2020
• The new hires will be allocated to the divisions
• Virtual Scientific Computing Group (VSCG)
• Communication platform: seminars, workshops, discussion, etc.
• The communication platform is open to other colleagues working in the field
• Coordinate with DMA, CDCS
FS
• The new hires will work on specific projects at the beamlines.
• Move to the new tasks when finished, maintaining support.
• Two job offers are open
Thorsten Kracht | Petra Days | Feb. 9, 2017 | Seite 15SardanaMotorMenu
Purpose:
• Support alignment: move widget with signal
• Present selected attributes
• Support homing: encoder attributes
$ SardanaMotorMenu.py
uses online.xml
$ SardanaMotorMenu.py –tags expert
select motors by tags
With signal definition:
$ SardanaMotorMenu.py –t exp_t01 –c exp_c01
Thorsten Kracht | Petra Days | Feb. 9, 2017 | Seite 16SardanaMotorMenu: 4 Motors
Work on selected motors only:
$ SardanaMotorMenu.py d1_mot01 d1_mot02 d1_mot03 d1_mot04
Thorsten Kracht | Petra Days | Feb. 9, 2017 | Seite 17SardanaMotorMenu
$ SardanaMotorMenu.py
MB1 – Move
MB2 – Attributes
MB3 - EncoderAttributes
Thorsten Kracht | Petra Days | Feb. 9, 2017 | Seite 18SardanaMotorMenu: Move Widget
Accessible from MainWidget or (to speed-up) from the command line directly:
$ SardanaMotorMenu.py d1_mot65
Thorsten Kracht | Petra Days | Feb. 9, 2017 | Seite 19SardanaMotorMenu: Move Widget
Motor Selector
$ SardanaMotorMenu.py d1_mot65
Thorsten Kracht | Petra Days | Feb. 9, 2017 | Seite 20SardanaMotorMenu: Move Widget
Motor can also be moved with
Key_Left and Key_Right
Key_Up, Key_Down to change slew
Change incr., slew or range
• Postscript
• Write .fio
Timer/counter selector also Invoke Attr. and EncAttr widgets Move right by step, incr. or to limit
possible from command line.
Thorsten Kracht | Petra Days | Feb. 9, 2017 | Seite 21SardanaMotorMonitor: Cursor Thorsten Kracht | Petra Days | Feb. 9, 2017 | Seite 22
SardanaMotorMenu: Attributes
Selected motor attributes Encoder attributes
Thorsten Kracht | Petra Days | Feb. 9, 2017 | Seite 23Summary
• Extend the central compute resources
• Extend GPFS to all data intense FS activities
• Implement HiDRA for all 2D detectors
• LaVue to become the standard visualization tool
• Extend ADAPT to more use cases
• First steps towards a general scientific software support
Thorsten Kracht | Petra Days | Feb. 9, 2017 | Seite 24You can also read
