CO Strategy and Technology for Rapid Application and GUI Development - Vito Baggiolini CO TM 08-Nov-2018 - CERN Indico
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
CO Strategy and Technology
for Rapid Application and GUI
Development
Vito Baggiolini
CO TM 08-Nov-2018
Introduction
• This presentation is based on
• Input I gathered from 35+ users in the whole sector doing GUI and RAD
• Recommendations from the CO Controls Evolution WG (10 senior engineers)
• Discussions and decisions in the CO SLM
• Context: two simultaneous but unrelated events
• Mandate and resources for APS to provide Python and RAD
• Oracle announcing that they are not interested in Java Client technology
anymore (unlike server side Java!)
2
Outline
• Python-based RAD/GUI
• Inspector
• Java Swing and JavaFX
• Web Technologies
• Tentative timeline and summary
3
Python GUI and RAD
4
Why Python
• Motivations
• Simple to learn, easy to use (no ceremony, nice tools, e.g. Jupyter)
• Very strong ecosystem for scientific computing (SciPy, NumPy, MathPlotLib, Pandas…)
• Very good for interactive scripting (including GUIs)
• Good complementarity/integration with C/C++
• Top want-to-learn language according to Stackoverflow Developer Survey 2018
• Use cases at CERN
• Scientific computing and data analysis
• MD scripts and GUIs
• Equipment prototyping and testing
• Code generation (Encore/EDGE, Cheby, …)
• System Administration (e.g. Ansible)
• Used almost everywhere:
• BE-OP, BE-ABP, TE-ABT, BE-BI, BE-RF, TE-EPC, TE-MPE, BE-ICS, BE-CO,…
• CERN experiments (Atlas, LHCb, …)
• Most Python users are not (and don’t want to be ;-) software engineers
5
History of Python in the Acc Sector and CO
• 2015: request for Python from MD people and equipment groups to CO
• 2016 1st BE Seminar on “Python tools for machine data analysis and
equipment control” with 60 participants
• 2016: Evian presentation “Breaking the Wall between OP and Expert Tools”
Make Expert GUIs usable also in operations
• 2016: Evian: CO announces Python Focus Group (Chaired by David Cobas)
• 2017: 2nd BE Seminar on Python Tools (with participation from CO)
• 2017: CO provides a standard Python distribution based on EP/SFT distrib.
• 2017: Evian presentation “A users view on exploiting the control system in
MDs” Promote Python to make useful tools both for MD and operations
• 2019 Q1: Two new staff in CO: one for (Python) RAD and one for Python as
a Platform
6
Accepted* Principles for “Good RAD”
• Logic (e.g. written in Python) should be separated from Visual layer
• Command-line Python scripts that can be connected to thin GUIs
• GUIs should be written as “thin clients”
• As little as necessary logic in the clients
• Logic in the FESA class or the middle tier ( UCAP, next slide)
• Graphical programming of logic is an illusion
• Labview syndrome, quickly gets complex, difficult to maintain
• Logic requires some programming, which everybody can learn
• Well written Python GUIs can be promoted to OP
• Without need to re-implement them in Java
• But needs investment in coaching and tooling
* Accepted by most people I spoke to 7
Unified Controls Acquisition and Processing (UCAP)
RDA3
UCAP server
Publisher Publisher
User Transformation Space
Transformation Transformation
Event Builder Event Builder
JAPC
8
Courtesy M. Buttner
Unified Controls Acquisition and Processing (UCAP)
RDA3
UCAP Server
Publisher Publisher Users - focus only on business logic:
- Event builder configuration (input)
- Publisher declaration (output)
User Transformation Space
- Implement transformation, test and
debug
Transformation Transformation
- Deploy & monitor easily
Event Builder Event Builder
JAPC
9
Courtesy M. Buttner
Separating Logic from Visual Client Layer
Thin GUI Layer
RDA3
UCAP server
Publisher Publisher
GUI
User Transformation Space
Transformation Transformation
Transformation Transformation
Event Builder Event Builder
JAPC
Devices Devices Devices Devices
10Existing Python tools for GUI and RAD
• Python for GUI development in general
• PyQt (most popular, also at CERN)
• Python for Qt (very similar to PyQt, but provided by the Qt Company)
• There are others (TkInter, Kivy, PyGi, wxPython, …)
• Python-based RAD tools for Controls
• PyDM "Python Display Manager" (EPICS): PyQt5
• Taurus "Python Scada" (Tango): PyQt4, plans to migrate to PyQt5
11Qt - the product
• Fully featured cross-platform application development framework
implemented in C++
• Desktop, embedded and mobile
• Supported Platforms: Linux, OS X, Windows, Android, iOS, etc.
on x86 x86_64 and ARM v7/8
• Software architecture allows for separating the GUI layer from the logic
• Qt “Properties, Signals and Slots”
• Qt modules
• Networking (TCP/IP, SSL, OAuth, OPC UA, …)
• http/JSON, Websockets, WebEngine, WebGL remote rendering, WebAssembly
• State machines
• Charts, SVG, 3D
• and many 3rd party modules
1213
Evaluation Work done since April
• Team: Eric 80%, Sara 100%, Vito 35%, Felix 30% + Andres (SRC)
• Purpose: explore PyQt and Qt in general
• Evaluate features
• Evaluate learning curve
• Evaluate productivity
• Results:
• DIAMON GUI Proof-of-Concept
• Sliding Charts and tables in CPS Fixed Display PoC
• Qt integration with Controls System through an RDA3 Qt Module (PoC)
• Chart performance analysis with input from BI (Stephane BP, Jordi)
• Technical exchange with EPC on Python for Qt (Nuno)
1415
CPS Vistar
PoC for
Sliding chart
and Table
Logic in a
UCAP server
16QTableView
17Chart Performance evaluation
Random data
60 Series
1500 points
25 Hz updates
18Python access to the controls system
• Currently using existing PyJAPC and PjLSA implementations
• NxCALS will provide PySpark (and CALS Java API), but not PyTimber
• Further APIs to be discussed
RDA3
• UCAP will support transformations UCAP server
Publisher Publisher
done in Python
• First implementation in Q2/2019 User Transformation Space
• First self-service tools in Q4/2019 Transformation Transformation
Event Builder Event Builder
JAPC
19Python-based RAD for Controls
PyDM and Taurus
20Existing 3rd party Python RAD tools
• PyDM https://github.com/slaclab/pydm
• "Python Display Manager”
• Implemented by SLAC (Standford)
• Integrates with EPICS
• Started in 2015 first public release (0.3) in Aug 2017
• PyQt5
• Taurus http://taurus-scada.org/
• "Python Scada"
• Implemented at ALBA (Barcelona), Part of Tango collaboration
• Integrates with Tango and EPICS
• Started before 2011 (Version 2.0)
• Currently on PyQt4, plans to migrate to PyQt5
21PyDM Application examples
22PyDM GUI designer (Qt designer)
23PyDM Architecture
Qt Signals
& Slots
Architecture
RDA3
Implementation
24Next steps
• Continue evaluation of PyQt in collaboration with other developers
• BI mainly for charting and performance
• Developers in OP, EPC and ABT, ICS for PyQt, CO-HT (Milosz)
• Further explore PyDM and possibly Taurus
• See how suitable the GUI components are for us
• See how well it complements our controls system (overlaps, mismatches)
• How difficult/easy is it to connect it to our controls system (RDA3, CCDB)
• How easy it is to provide our own CERN-specific components
25Python GUI/RAD deliverables provided by CO
• Support for development of Python GUIs based on PyQt (also for OP)
• PyQt development tools (Qt Designer) pre-installed on all VPCs
• Tools for packaging and deployment of Python GUIs
• Support for Controls System APIs
• Support for PyJAPC, PyLSA, PySpark (other APIs to be discussed)
More
• Integration with Unified Controls Acquisition and Processing info
(UCAP)
in a Python FG
• Organization of Training in collaboration with HR
Meeting soon
• Support for a Python RAD tool (possibly PyDM)
• Integration with CCDB to browse for devices and their description
• Connection to RDA access layer (with CMW team), with support for PPM
• CERN-specific Qt components as necessary
• Pre-installed on Linux
26Inspector Based on discussions with Bertrand L., Valter C. Andy B. Luca B.
28
29
30
31
32
Many versions
of the same virtual signal
33Functionality overlapping with CO services
• Inspector has been developed over many years
• Functionality is overlapping with CO services:
• Logging CALS, NxCALS
• Server-side signals CMW Proxies
• Alarms SIS, LASER
• Equations, servers-side signals UCAP
• Script execution Sequencer
• Scripting Python
34Inspector analysis
First Impression from User perspective
Beautiful and attractive
Fast interactive development
No hassle with deployment
Nice user documentation
Encourages graphical programming
Second impression from the same User, view of software experts
Static layout, not resizable for different screen sizes
Closed in: not possible to evolve a simple RAD UI to a full GUI
Graphical development quickly gets complex and difficult to maintain and debug
Copy-Paste development, no re-use or sharing of functionality (e.g. sub-panels)
View from the perspective of a provider/maintainer of Inspector
Functionality overlaps with CO (Logging, Proxies, Alarms, UCAP)
Completely in-house development, even implemented own scripting language + interpreter
Expectations of users: “LabView and MS Visio provide feature X, could you please implement it?”
Very little JavaDoc, very few unit tests
Possible to provide exactly what the users want (as every in-house development) 35Plans for Inspector
• CO-APS will not take over the Inspector tool
• Discussed with Andy B. Luca B. and Valter C. in early June 2018
• We can help someone (B. Lefort?) debug urgent operational problems
• CO-APS will convert the LN4 source control (Inspector GUI and massive
server-side extensions) to use standard CO solutions, including Python.
• How to facilitate migration away from Inspector to Python RAD
• Inspector integrating external Python scripts => OK
• Inspector with only direct device/property access => OK
Thin Inspector Panels
• Inspector Graphical Equation Editor => Not OK => manual port to Python
•
No special features
Inspector internal Scripting => Not OK => manual port to Python
• Inspector with server-side signals => manual port to UCAP
• => Inspector users are encouraged to do thin panels
• CO-APS will investigate if it is possible to migrate thin Inspector Panels semi-
automatically
36Java Swing and JavaFX
37We have roughly
500 different operational Swing GUIs
50 operational JavaFX GUIs
developed over 18 years
38We have roughly
500 operational Swing GUIs
50 operational JavaFX GUIs
developed over 18 years
3940
41
Highly condensed
information
A lot of interactions
42High performance
43Java Applications and their Developers
• Types of Applications:
• Dashboards/Fixed displays
• Long-lived, often complex operational applications
• Rapid development for MDs (often throw-away)
• “High performance” applications (OASIS, Tune measurement, BI Apps)
• Hardware testing GUIs (a la Labview)
• Types of Developers:
• Mainly 2nd job developers (physicists or operators, hardware experts).
Experts in their areas, use software just to get a job done
=> Don’t want to learn new things if not strictly necessary
• Software developers (engineers and technicians). Software experts.
programming is their main job, they like modern technologies
=> eager to adopt cool new technology
44JavaFX and Swing – some facts
• Java SE 8 contains JavaFX and will be supported until Jan 2019 for
corporate use and with commercial support until 2022.
• Starting with Java SE 11 (Sep 2018) JavaFX will not be packaged with
the JDK, but available as a separate download
• JavaFX is in the hands of the open-source community and smaller
companies like Gluon who are offering commercial support for it
• Swing / AWT is part of JDK SE 11 and will be supported to at least
until 2025/26 (with commercial support)
• Oracle will try to also get rid of Swing for Java in 2021, but cannot
decide unilaterally (there is a formal process involving other parties)
45JavaFX - some numbers
• JavaFX development/developers:
• Gluon company: 10 JavaFX developers, who also work for client projects
• Oracle Company: 10 JavaFX developers (1000 Java in general), 2/3 on maintenance
• Github repo: 19 comitters, with 3 who did more than 5 commits in 2018
• OpenJFX mailing lists: ~150msg/month; 150 participants, 5 of which contribute 50%
• 25 Participants in JavaFX Adopters day in Munich, estimated 90% on Swing, 10% on
JavaFX
• Lines of code:
• JavaFX (without Webkit): 1.7 MLOC (half Java, half C/C++)
• Jobs on indeed.com: (7-Nov-2018)
• JavaFX+Software: USA : 73, Germany: 79, CH: 13
• Java+Swing+Software: USA: 400, Germany: 170, CH: 17
• Qt+Software: USA: 713, Germany: 716, CH: 51
• Angular+JavaScript+Software: USA: 7226; Germany: 1650, CH: 215
46What is the Future of Client Java?
47
Credits https://assets.saatchiart.com/saatchi/969578/art/3726264/2796148-MIMCYHHL-7.jpgMy interpretation => Recommendations/Plans
• The JavaFX community does not look strong enough to prosper
• => We will keep our JavaFX development on halt
• => Don’t migrate existing Swing applications to JavaFX
• => APS will support the existing code and current users until LS3
• => We will try to buy commercial support (e.g. from the Gluon company)
• Swing is likely to survive
• We have 500+ Swing GUIs which have to survive
• => We will do our best to support Swing until LS4
• => Hope that Swing remains in JDK, otherwise buy support
• => Port JDataViewer to new Java versions
• => Swing is the default recommendation for new operational Java applications
48Web Technologies
49Why Web Technologies?
• Today, Web is the de-facto standard for how people do things
(shopping, bookings, banking, etc.)
• Web applications can be built on main stream and mature
technologies
• Although there are lots of frameworks and libraries, there are only a
few that stand-out with significant company backing
• It is easy to recruit good people to work on web development, and
they can acquire / build transferrable skills for when they leave CERN
• Web applications run everywhere without pre-installation of software
50The History of Web Development until 2016 https://www.flickr.com/photos/mraible/20606289343/
51Most successful Frameworks today
Angular (Google) React (Facebook) Vue.js (Alibaba)
52Web pages from CO-DS Courtesy L. Burdzanowski 53
Courtesy C. Roderick 54
Courtesy C. Roderick 55
Web for full-fledged applications
• We use web in CO, and think that it has a future
• CO-DS have a lot of experience with this technology
• provide the accsoft-commons-web framework
• a carefully selected set of libraries and tools
• a standardized approach used amongst a number of large scale production
applications used by hundreds of users
• Plans for in 2019:
• CO-DS will help developers in CO-APS (and possibly others) assess ACW
technology
• Under active discussion
56Web fixed-display/dashboarding technology
• Thin layer of web displaying similar data as Inspector Synoptics or
Fixed Displays
• No calculations in the Web layer, but in the UCAP server
• Possible technology:
• Grafana – a highly popular third party dashboarding tool
• used e.g. in by the COSMOS (infrastructure monitoring) project
• CO-DS dashboard framework (based on angular-gridster)
• There exist other ideas/implementations in the Sector…
5758
Application architecture with web
• Client Application in the Browser
• Backend process running on a server
• Communication using REST plus
WebSockets/Server Sent Events
REST (http) WebSockets /
We will have to develop: Server Sent Events
• a CMW-to-Web gateway
• REST APIs on our Java servers Java Server
(Spring Boot)
59WebAssembly
• An execution environment for native applications (C/C++ etc) inside
the browser
• A light-weight virtual machine without garbage collection
• A standard supported by all 4 big browsers
• Not intended to replace JavaScript, but it opens possibilities
• See also https://webassembly.studio/
6061
62
CO Plans for Web
• Explore Web as an alternative to Java Desktop Applications
• Gain experience with full-fledged Web applications
• CO-APS will prototype one or two Java applications in ACW in 2019
• CO-DS will provide help with accsoft-commons-web
• Plans for Web dashboards/fixed displays
• Establish collaborations, e.g. with BE-ICS (Matthias) and EN-SMM (Alessandro)
• CO-APS will work on UCAP as a transformation server for Web Dashboards
• Work on connection from Web to the controls system
• Web-based API e.g. to LSA (REST)
• Device/property CMW for the web?
• Keep an eye on new technologies (webassembly)
• New opportunities for synergies and collaboration in the sector!
63Recommendations to our different users
• Python developer => use PyQt, PyDM
• 2nd job Java developer in OP
• Evolve existing applications in the current technology
• Java Swing by default for new GUIs
• Have a look at PyQt
• Possibly join the Web evaluation campaign
• Equipment expert/FESA class developer
• PyQt
• Java Swing
• Inspector user
• Learn Python and have a look at PyQt and UCAP
• Follow our progress on and PyDM and Web dashboards
• If you need to use Inspector -> make thin Inspector Panels on top of Python Scripts
• Software Expert
• Participate in exploration of Web technologies
64Tentative time line for Python GUI and RAD
• 2018 Q4 – Continue evaluation of PyQt with interested users
PyQt
• 2019 Q1 – Ready for PyQt development (Tools on VPCs, Training
recommendation)
• 2019 Q3 – First accelerator specific Qt components available
• 2019 Q2 – Decision about Python RAD technology (PyDM, Taurus,…)
RAD
• 2019 Q4 – First beta-version of Python RAD available
UCAP
• 2019 Q2 – UCAP server supports Python transformations
• 2019 Q4 – First self-service tools for end users
65Tentative time line for Web
• 2019 Q2: Evaluation of ACW for Operational Applications by CO-APS and
other interested/advanced users
• 2019 Q4: First conclusions on using ACW for Operational Applications
• 2019 Q1: Start collaboration (sector-wide) on Web for Dashboarding
• 2019 Q2: First version of WebSocket support from UCAP
• 2019 Q4: First conclusion of using Web for Dashboards/Fixed displays
• 2020: Web-APIs (REST) to main operational Java servers
66Summary
• CO is committed to Python and has resources
• We will support PyQt as the obvious choice
• We will investigate PyDM or Taurus for Python RAD
• CO does not take over Inspector
• Java client technology is declining (Java on the server does well)
• We will (have to) keep Java Swing alive, but recommend to abandon JavaFX
• We want to explore Web technologies as an alternative to Java
desktop
• We are actively seeking for collaborations
67Reserve Slides
68Input from the Accelerator Sector
• Input gathered from:
• BE-OP: Jorg, Verena, Kajetan, Delphine, Tibor, Raul, …
OP and Physics groups
• BE-ABP: Richard S., Gianni I., Guido S., Riccardo D.M.
• TE-ABT: Etienne C., Nicolas M., Ch. Chanavat
• BE-BI: Stephen J., Christos Z., Stephane BP
• BE-RF: Andy B., Bartek B., Valter C., Niall S. Equipment Groups
• TE-EPC: Quentin K., Stephen P., Carlos G.
• TE-MPE: Markus Z., Jean-Christophe G., Marc-Antoine G.
• BE-ICS: Matthias B., Brice C.
• EN-SMM: Alessandro M.
• GSI: Jutta F., Ralph Steinhagen Controls Groups
• CO: David C., Stephane D., Chris R., Lenaic L., Eugenia H.
+ 10 senior CO engineers
69Qt - the Company and Business model
• 1994 TrollTech, 2008 Nokia, 2011 Digia Qt Company
• Automotive, Medical, Industrial Automation, IoT, Desktop
• Fundament of the KDE Linux software
• Mercedes, Volvo, Tesla(?), Ford, Peugeot, Harman, TomTom, …
• And many more here and here
• 300 employees and hiring (currently 30 open positions)
• Several consulting companies (service partners) also in Europe
• I got an insight on a 2-day Qt Contributor Conference in Oslo
70You can also read
