ECMWF ITT COP_043 WEKEO (DIAS) DATA ACCESS SERVICES, PROCESSING SOFTWARE AND TOOLS - FUNDED BY THE EUROPEAN UNION IMPLEMENTED BY - COPERNICUS ...
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
ECMWF ITT COP_043
WEkEO (DIAS)
Data Access services,
Processing Software and
Tools
Funded by the European Union Implemented by
Welcome to WEkEO ECMWF Industry Information Session
• Purpose of today‘s information session:
• Background, objectives, scope, timing of the upcoming ITT
• Guidance to prepare the offer
• Information is preliminary, there could be changes in the final ITT
• Today‘s Focus:
• ITT on Data Access Services, Processing Software and Tools
• Questions can be raised in real time by email to:
copernicus-events@ecmwf.int
2
AGENDA
10:00 – 10:10 Welcome and introduction
10:10 – 10:40 ITT technical description, scope of work
10:40 – 10:45 Contract Management
10:45 – 10:50 Procurement rules at ECMWF, How to bid
10:50 – 11:00 Q&As
11:00 Conclusion
* UK Time
3
WEkEO
EUMETSAT, ECMWF and MERCATOR OCEAN have joined their
experience and skills to build up DIAS WEkEO:
• In a stepwise approach, minimizing the risks
• Capitalizing on user feedback
• Strongly involving industry through procurement.
4
DIAS – Background
• WEkEO responds to the Data and Information Access Services (DIAS) requirements approved in
December 2016 by the EC, building on the work of the Copernicus Integrated Ground Segment
(IGS) Task Force.
Data Distribution Services Data/Information Access Services
Land Data
(DIAS)
S1 S2 S3 S4 S5 S6 Marine Distribution
Sentinel Service Atmosphe Core Services
Data Information Emergenc
re
Security
y
Climate
Sentinels Service
Information
Sentinel Sentinel Sentinel Sentinel
Contributing In-Situ
Data Data Data Data
Missions Data
ESA EUMETSAT ESA EUMETSAT
5
What is WEkEO ?
• It is a platform which:
• Provides easy, cost efficient and reliable access to data and resources (products,
storage, tools, computing ...)
• Retains open and free access to Copernicus data and information (model data
outputs, in-situ data and satellite products)
• Allows users to apply own processing to data and bring their own data
• Makes available scientific expertise and technical user services to make best use of
the data / to develop own capabilities;
• Builds bridges from research to business;
• Promotes innovation and industry competitiveness, including SMEs;
• Enables the development of new markets benefiting from Earth Observation.
6WEkEO special features
• Builds on work done by the partners for their systems:
• Uses solutions developed by the partners (e.g CDS of ECMWF, Pathfinders of
EUMETSAT) à designs and implementations are available & verified.
• The Interfaces to the users (e.g. APIs) are here to stay:
• Interfaces for access/hosting/front offices, etc.. are those being developed for the
EUMETSAT data services:
à Operational – compatible with the needs of operational users in meteorology
à Maintained and evolved long term to guarantee continuity for our users
à Applicable/CFIs to the DIAS Contractors: secures level playing field, good
competition and “seamless“ migration of users in case of provider change
7WEkEO special features
• Integrated while being Distributed
• Federative and Open by nature:
• The solution is generic and replicable to other nodes, federating platforms for mutual
benefit.
• User-orientation – Our goal:
• Bring to our current users the benefits of DIAS;
• Secure Operational Continuity;
• Integrate the User Services with the expert knowledge of the 3 organisations;
• Facilitate the transition from research to business with an “incubator“ environment;
• To the SMEs and down-stream service providers, it provide a place to develop, grow
and sell their products under foreseeable and stable conditions.
8WekEO: Planned ITTs
ITT Scope Comments /Rationale
- Provision of operational DIAS services Includes commercial
- Provision of commercial cloud services exploitation of the DIAS
ITT-1: DIAS Services and
- Provision of network infrastructure
Operations
- Provision of Web Services, referencec front Office, Sys Admin &
first line help to users, link to user support office.
- Provision/operation of virtual environments and toolboxes
- Integration/hosting, operation of CFI elements (Tools, SW
packages maintained by the partners)
- Implementation & Test of engineering changes
- Operational reporting
ITT-2: Processing, Data - Implementation, test and maintenance of the Data Processing, Includes SW for interfacing
Access and Tools SW Access functions and the CFI tools DIAS with existing partner´s
systems and tools
ITT-3: DIAS System - DIAS System engineering & consistency/compliance Independent from the DIAS
Engineering, Validation - DIAS services Validation in support to the partners Service Provider (securing
- Defining evolution/engineering changes based on user feedback independent Validation)
ITT-4/5 User Uptake - User Uptake, identification of potential/new user communities, Interface to ITT-1 and to the
Marketing and User Support Marketing, Communication existing User Support function
(one or several ITTs)
- Integrator of the specific user Support functions @ MO, Independent from ITT#1
ECMWF, EUMETSAT Contractor.
9Overall WEkEO development, schedule
§ Overall Baseline for V2, V3 to be adapted in scope depending on
uptake and user feedback, budget/schedule
Design User feedback User feedback User feedback
inputs from V0 from V1 from V2
Procurements Managed by
EUMETSAT, ECMWF,
V1 base V2 base V3 base Mercator Ocean
Industry KOMs V1 release
V1 impl. & Test V2 release
V3 release
V2 impl. & Test
Under responsibility of
V3 impl. & Test industrial framework
...
Operations preparation, Operations, User uptake
2017 2018 2019 2020
10ITT COP-043 Scope
§ This ITT will cover:
§ The development of V1 up to its operational deployment (foreseen at the
end of Q1 2019)
§ Maintenance and support to operations of V1 release up to V2 release
(end of 2019)
§ Further upgrades are foreseen with the V2 and V3 releases.
§ V2 and V3 Data access and tools developments/upgrades will be
tendered subsequently by ECMWF in due time
§ V2 and V3 will take into account feedback from user needs
11ITT COP-043
Technical Description
12ITT COP-043, DIAS Partner Roles
EUMETSAT & Partner DIAS
User Management Mercator-
Service User mgt Web Monitor, Ocean
Marketing
validation support Portal Report
Processing & Data
ECMWF
Report to EC
Access
Processing Tools
Services
Infrastructure
EUMETSAT
Compute, System
Interfaces Storage, Elasticity Integration
Network Verification
13ITT COP-043, Technical description, Scope
• The design, development, implementation, and support of the
WEkEO Processing, Data Access, and Tools software layer which
includes:
• A Data Access API common to all the available Copernicus
satellite and services data.
• Tools and processing services based on a set of virtual
images (VMs) preloaded with software to allow users to operate
and visualize the data.
• A Big Data solution to enable Analytics and Machine Learning
on very large volumes of data.
14ITT COP-040, Technical description, Work Packages
15ITT COP-043, Technical description, WP1
Mandatory Data Sources
• Sentinel satellites
• S1, S2, S3, S5P …
• Level 1, 2
• All Copernicus services
Additional Key Data Sources
• Contributing missions
• REP, TEP …
In the future
• More satellites, more coverage, more instruments
• An explosion of data volumes
16ITT COP-043, Technical description, WP1
Satellite Estimated volume (end of 2020)
S1 4-7 PB
S2 3-4 PB
S3 3-4 PB
S5P 0.5 PB
Service Estimated Volume (end of 2020)
Climate/Atmospheric 36 PB
Marine 10 PB
Land 10 PB
Security ?
17ITT COP-043, Technical description, WP1
Data Access API (cont.)
• Different protocols and formats
• NetCDF, GRIB, PNG, SAFE/JPEG2000, HDF5, TIFF, GEOTIFF …
• Some data may be off-line (tape)
• Data not organized to fulfil all use-cases efficiently
• Data subject to licensing
• Statistics, audit trail of who access what
18ITT COP-043, Technical description, WP1
Data Access API (cont)
• A single unified REST API to access all data
• Programming language agnostic
• Data agnostic
• Object Stores will be considered
• A catalogue
• Queriable via an API
• Extensible
• Should allow easy integration of new local and remote data
sources
19ITT COP-043, Technical description, WP2
Tools and Processing services (cont)
• A catalogue of pre-defined VM images
• Pre-installed with tools relevant for use-cases
• Libraries
• Desktop applications
• VM images can be specialized
• Thematic: Satellite, Climate, Oceanography, Land …
• Targeted at specific user groups (developers, experts …)
• All images contain Data Access API clients (from WP1)
20ITT COP-043, Technical description, WP2 Tools
Remote sensing tools Data toolboxes Python / R based
Orfeo Toolbox ESA BEAT pandas
SNAP Toolbox Sentinel 1, 2, 3 PCRaster
xarray
GOCE User Toolbox
GIS Software SEAScope numpy
SAGA ecFlow scipy
QGIS matplotpy
Image Processing
ArcGIS Imagery Workflow
Efoto
GRASS GIS
Interimage
ILWIS
eCognition
Gv SIG
ITK
21ITT COP-043, Technical description, WP3
Big Data Solution
Bidder propose a solution to enable Data Analytics and optionally
Machine Learning for big data processing.
• Sample solution #1
• User can create own instance of cluster
• Multiple VMs
• Cluster automatically configured
• Example: Apache-Spark based
• Sample solution #2
• Container based framework
• Example: Docker, Kubernetes …
• Sample solution #3 for Machine Learning
• Based on TensorFlow
22ITT COP-043, Technical description, WP3
Processing
• Focus on open source Python / R based environments
• Mutual dependency on infrastructure procured by WEkEO ITT-1
• Vsphere, OpenStack ….
• File System (Ceph-FS, HSFS, Object Storage)
• Data storage must be carefully designed
• Which data on RAM / SSD / Hard_disk
• Which data should remain on the DIAS platform
• Preference for open source software but commercial solutions will
be considered
23ITT COP-043, Technical description
Notes
• Everything is a resource
• Data and products catalogue
• Data access
• Image catalogues
• VMs, virtual disks, virtual networks …
• Everything is managed by REST APIs
• When possible API must fit in existing frameworks (e.g.
OpenStack)
• Web frontends will use the same APIs
• All Software to be delivered fully tested on WEkEO federated
cloud.
24ITT COP-043
Contract Management Aspects
25What makes a contract?
Framework Agreement with Annexes:
• Terms & Conditions (published in ITT)
• Annex 1: ECMWF’s specifications (published in ITT)
• Annex 2: Contractor’s tender: based on negotiated technical proposal and
pricing table
The Framework Agreement is expected to include two Service
Contracts (SC):
• SC1: Implementation and testing of Release v1 (T0 – T0+5)
• SC2: Maintenance and support to operations of Release v1 (T0+6 – T0+14)
Each Service Contract includes:
Annexes 3A: Invitation ECMWF
Annex 3B: Technical Annex based on Annex 2 and additional service contract
negotiations
Annex 3C: Financial Annex incl. payment plan and verification process
26Price and payment
Competitive price: not specified in ITT
Agreement based on negotiated price: pricing table for human resources,
travel, licenses, hardware, etc. versus WEkEO objectives
Likely payment methods:
• SC1 Implementation and testing of Release v1: Fixed Price
• SC2 Maintenance and support to operations of Release v1: Fee based
27SC1
Key aspects contract management
Meetings and Communicatoin
Contract verification and Payment
Kick-Off Meeting: Payment against
November 2018 verification succesfull
delivery
IT development:
Contract Implentation and follow-up
Regular plan driven =
Release roadmap and
Reporting
teleconference ‘waterfall’ Quarterly
meetings with components for v1.0 Implementation
shall be alligned with ITT procedure against
ECMWF: In SC1 at pre-defined Reports:
least monthly and roadmap and achievements;
Contractor shall deliver KPIs; risks;
more frequent if requirements
fully tested Release v1.0 budget
deemed necessary
integrated into the Agile based consumption;
WEkEO environment = developments assets
Expected need to principle deliverable possible for specific
contribute to SC1
Audits
features. Audit opinion
interactions with Annual
ECMWF-EUM-MO Implementation on financial
Interim payment IPR management: information
coordination Reports and
possible agains interim Update details Implementation
meeting and other Plans
release Report by
WEkEO licences –
contractor(s) preferably open Ad-hoc requests independant
SC2 auditor
source for information:
Input based payments e.g. for
for maintenance and Tools: track communication Audits by
support phase planning, resources purposes ECMWF or
and performance, Commission
Verification OK = Notice risks: timely Financial possible
of acceptance to implementation is Reporting
proceed with invoice key!
28Review procedure- Deliverables
Contractor ECMWF
Deliverable Review
Submit Submit Submit
Deliverable 1 Deliverable 2 last Deliverable
No No
Review of
… No
Review Submit
Review of Del2 last Del
Del1 Notification of
Approved Approved Approved Milestone Completion
? ? ?
Yes Yes Yes
TIME
Details available in Framework Agreement – Annex 3CReview procedure- Deliverables
Contractor ECMWF
Submit Submit Submit
Deliverable 1 Deliverable 2 last Deliverable
No No
… No
Review Review Submit
Review Notification of
of Del2 last Del
of Del1 Milestone Completion
Approved Approved
Approved
? ?
?
Yes Yes Yes
Details available in Framework Agreement – Annex 3CContract Perfomance Review procedure- Deliverables
Subm
it
Submit
Last
Contract Performance Review Payment
…
Del1 Del
No No
Review Review Implement Send invoice AND
of Del1 last Del Submit corrections Notice of Acceptance
Approv
Approve
d ed Notification of Progress to
? ?
Milestone Completion meeting finance@ecmwf.int
Yes Yes (and Use Of
Resources) No Issue Notice
of Acceptance
Verify contract
performance Yes
All Payment
deliverables
Approved? No
Issue Notice
of Need of
Remedial
Action
21 days for ECMWF to conduct verification Max 30 days
process
Details available in Framework Agreement – Annex 3CContractual Reporting: serve as input to report to the European
Commission and EU Member States
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
Details available in Framework Agreement Clause 2.3ITT COP-043
Procurement rules and process
33ECMWF’s approach to Procurement
• Procurements (ITTs) for Copernicus Services implemented by ECMWF are done under
ECMWF procurement rules.
• These are similar to, and based on the same principles as, the most recent EU
Procurement Directive, but they are not identical.
• Dissemination of information about ECMWF Invitations to Tender (ITT):
• ECMWF publishes Prior Information Notices and Contract Notices in the Official Journal of the European
Union (OJEU).
• ECMWF’s Copernicus tenders web page (https://www.ecmwf.int/en/about/suppliers/copernicus-
procurement/update-itts), updated regularly to list ITTs that have been published.
• Tenders Sections of the Copernicus (http://copernicus.eu/main/tenders-and-grants), CAMS
(http://atmosphere.copernicus.eu/tenders) and C3S (http://climate.copernicus.eu/tenders) websites
34ECMWF’s eProcurement Portal
• ECMWF uses an outsourced eProcurement Portal to manage its procurement processes
• accessible at https://procurement.ecmwf.int
• In order to respond to an ITT, suppliers need to:
• Register on the Portal (the registration process is free and is managed by the Portal provider)
• Express an interest in an opportunity (ITT) after it has been published
• Complete the online questionnaire (together with other forms and templates) and submit bid online
35Structure of the ITT
• A standard ITT is structured in several parts:
• ITT documents:
• Volume I – Instructions for respondents and Conditions of tender
• Volume IA – Timetable for the ITT
• Volume II – Specification
• Volume III – Templates for Tenderers to Complete (technical response and pricing table templates)
• Volume IV – Forms for Tenderers to Complete (Financial Identification and Legal Identification forms)
• Volume V – Terms and Conditions for Copernicus procurements through ECMWF
• Online ITT Questionnaire – questions for all Tenderers to respond to, which are posted on the
eProcurement Portal
• OJEU Contract Notice – describes the ITT and contains important information related to the ITT
• All ITT documents are held on the Portal. In addition, Volume IA and Volume II are also
published on the ECMWF’s tenders web page.
36Guidance for Suppliers
• ECMWF has also developed a document (published at ECMWF’s Copernicus tenders
web page https://www.ecmwf.int/en/about/suppliers/copernicus-procurement/update-
itts), providing step-by-step guidance to suppliers about how to navigate the
eProcurement Portal for:
• finding an opportunity launched by ECMWF (RFPs, ITTs);
• accessing the RFPs/ITTs and the “Online questionnaire”;
• finding the RFP/ITT documents on the Portal;
• submitting a response to an ITT on the Portal;
• accessing the messaging board of an ITT/RFP.
37Procedural Guidelines for ITT COP_043
General points that apply:
• Communication is conducted exclusively via the Portal Messaging board, including
clarification questions.
• ECMWF will only enter into a contract with a single legal person/entity.
• Evaluation is done by the Tender Evaluation Board with the involvement of external
reviewers. The High Level Evaluation criteria are:
• Legal and Financial
• Price
• Track record
• Quality of resources to be deployed
• Technical solution proposed
• Management and Implementation
38Indicative Timeline & Process for ITT COP_043
Timeline 2018 Event description
Mid-July Publication of ITT and Contract Notice
until Mid-August Bidders can raise clarification questions via the messaging board on
the portal
End of August Tender closure (6 weeks after ITT release)
September TEB evaluation
September-October The TEB may request further clarifications from bidders before a final
decision of the preferred bidder is made
October ECMWF will enter into negotiations with the preferred bidder
1 November Contract award
39Q & As
Questions can be submitted until today 17pm UK time
to the following email address:
– copernicus-events@ecmwf.int
ECMWF will respond within 2 working days
40You can also read
