Operations and Release Working Group - Ricardo Wissmann-Alves - Head of IT Operations - MOSL
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Operations and Release Working Group Ricardo Wissmann-Alves – Head of IT Operations 15 December 2020
Agenda
Item Description Who Time
1. Welcome and introduction RWA 10:00–10:15
2. MOSL updates RWA 10:15-10:45
• CMOS Maintenance schedule RWA
• Release Schedule 2021 RWA
• Release 10.0 content RWA
• Tariffs 2021/2022 HA
• Settlement RWA
• G readings RWA
RWA
• Non Code Changes
3. Forthcoming changes update AH 10:45–11:15
• General AH
• CPW106 – De-registration of long-term vacant SPIDs AH/JJ
4. Bilaterals Programme update JG 11:15–11:30
5. Data Issues / Improvements 11:30–11:45
• CMOS / Remedy users RWA
• Notifications (Txxx.M) RWA
• Unplanned CMOS Outages RWA
• Observation Log (Improvement) HA
• Digital Certificates JB
6. AOB and wrap up All 11:45–12:00
ORWG December 2020 2
Welcome and Introduction
Participants
Dynamics of web conference
Hannah Allardice (IT Functional Consultant) will take the minutes
Alex Crosbie (Digital Support Analyst) will state your name, please tell us your
company name
Feel free to switch your camera on
ORWG December 2020 3
MOSL Updates
Maintenance Schedule 2020
PRE PROD = MPS/MPS2 Published on MOSL website
Outage time between 10:00 am and 04:00 pm for PRE PROD environments CMOS > CMOS Releases
Outage time between 11:00 pm and 6:00 am for PROD environment
Oracle Database compression = Outage time between 6 pm on the first day (Friday) until 12 pm on the third day (Sunday)
October/November
Oracle Database
Friday 02/10/2020 Saturday 03/10/2020 Sunday 04/10/2020
compression
MPS MPS MPS
Monday 19/10/2020 Tuesday 20/10/2020 Wednesday 21/10/2020
Cancelled (MPS) MPS MPS2
Saturday 14/11/2020
PROD
December
Monday 14/12/2020 Tuesday 15/12/2020 Wednesday 16/12/2020
Security
MPS MPS2 Mop Up
patch
NEW
Saturday 19/12/2020
PROD
On 15/12/2020 there will be a planned maintenance of 30 minutes from 6pm in both
MPS and MPS2 environments
January 2021
Cancelled
Monday 18/01/2021 Tuesday 19/01/2021 Wednesday 20/01/2021
Moved to
MPS MPS2 Mop Up
February
Saturday 06/02/2021
2021
PROD
February 2021
Monday 15/02/2021 Tuesday 16/02/2021 Wednesday 17/02/2021
NEW MPS MPS2 Mop Up
Saturday 20/02/2021
PROD
ORWG December 2020 5
Release Schedule 2021
Release Deployment date Time Environment
10.0 16.04.2021 10am-8am MPS2
30.04.2021 10am-5pm Patch drop MPS2
14.05.2021 6pm-2pm PROD
18.05.2021 10am-5pm MPS
10.5 30.07.2021 10am-8am MPS2
13.08.2021 10am-5pm Patch drop MPS2
27.08.2021 6pm-2pm PROD
31.08.2021 10am-5pm MPS
11.0 01.10.2021 10am-8am MPS2
15.10.2021 10am-5pm Patch drop MPS2
05.11.2021 6pm-2pm PROD
09.11.2021 10am-5pm MPS
Published on MOSL website CMOS > CMOS Releases, including the Release
Notes updated on a weekly basis if changes have occurred
ORWG December 2020 6Release Management – R10.0
NCC016 – Same Day Meter Readings Display
NCC017 – Meter DPID Association History
NCC019 – SPID status to be displayed on Premises screen
NCC021 – Addition of business transactions button to search market information
screen
NCC022 – Meter Network Display
CPW089 – T101 Adding Billing Address fields
Further details for NCCs on MOSL website (Market Codes > Change > Non Code Change Requests)
or MOSL Extranet - Non code change requests tracker
Further details for CPWs Market Codes > Change > Changes
ORWG December 2020 7Release Management – R10.0
NCC016 – Same Day Meter Readings Display
Display of all meter readings (included retired readings) within the meter
readings screen
Current heading of the column stating “Removed” will be changed to Retired
ORWG December 2020 8Release Management – R10.0
NCC017 – Meter DPID Association History
Expand Meter / DPID association displays to show a historic meter / DPID
association
All meter / DPID associations that have been active at the SPID will be shown
Trading Parties will be able to see when the active Meter / DPID ended and
any reactivations that may occur after.
ORWG December 2020 9Release Management – R10.0
NCC019 – SPID status to be displayed on Premises screen
SPID status to be shown on the Premises tab of the SPID
Addition to the Core Data section of Premises where it displays the details of
the Supply Points
ORWG December 2020 10Release Management – R10.0
NCC021 – Addition of business transactions button on to search market
information screen
A new button (B) for the BTD (Business Transactions Dossier) will be added on
the results for the “Search Market Information” on the CMOS Portal.
This will be a shortcut to go directly to the BTD
ORWG December 2020 11Release Management – R10.0
NCC022 – Meter Network Display
Full visualisation of the Meter Network Display, regardless of the meter
selected
If the meter selected is a sub-meter, then the Meter Network Display will
show the main meter
If the meter selected is a main-meter, then the Meter Network Display will
show the sub-meter(s)
ORWG December 2020 12Release Management – R10.0
CPW089 – T101 Adding Billing Address fields
T101.W / T102.M New Data Items
Developer/Business name* Billing Address Line 4
Customer Banner Name (if different Billing Address Line 5
from Customer Name) Billing Address Postcode*
Billing Address Building Name Contact Name
Billing Address Line 1* Contact Email
Billing Address Line 2 Contact Number
Billing Address Line 3
* Mandatory fields: if they are not known, then fill in with “Not Known”.
Remaining fields are optional and will be included in T102.M if provided in T101.W
ORWG December 2020 13System Code versions System Version as of date PROD 9.0.1 (60111) 14/12/2020 MPS 9.0.1 (59141) 14/12/2020 MPS2 9.0.1 (59141) 14/12/2020 ORWG December 2020 14
Tariffs 2021/2022
Tariffs 2021/2022 Timetable for Tariffs creation/update communicated on 1 December Tariff guidance document (CMOS > Training & Guidance > MOSL Guidance) Comms will be issued today ORWG December 2020 16
Settlement
Settlement timetable
Published on MOSL website Customer Journey > Settlement > Settlement Runs
ORWG December 2020 18Settlement timetable ORWG December 2020 19
Settlement Estimated Readings (ER run) being split into two stages to minimise runtime in advance of the RF calculation – comms always issued once the dates are confirmed January 2021 ER (RF = August 2019) will start on 28 December and the second stage will run on 4 January 2021 Reminder comms will be issued on 22 December. ORWG December 2020 20
RF Settlement runs – G Readings
G readings created
200,000
180,000
160,000
140,000
120,000
100,000
80,000
60,000
40,000
20,000
0
ORWG December 2020 21Non Code Change
Non Code Change Process
Website: Market Codes > Change > Non Code Change Requests
Guidance: CMOS > Training & Guidance > MOSL Guidance > Non Code Change
Request (NCC) TP Guidance
Allows changes to CMOS or MOSL systems to be requested and considered
Sharepoint location for submitting change requests
MOSL EXTRANET / Non Code change Requests tracker
Emailed confirmations of receipt
Tracker Form of status
Run consultations as per Code change CPs
Submit to Panel if/when funding is required
ORWG December 2020 23NCC tracker
Date Trading party Target
Request title NCC Ref Category Name Status
submitted submitted Describe the issue Release
CMOS offers users the option to select one of up to 5 buttons in the search
market information screen to enter into a SPID depending on the set up of the
SPID.
As below where a SPID is set up for water & sewerage changes with an active
meter & active DPID all 5 buttons are displayed.
You are taken into the part of the SPID you wish to view depending on the button
Addition of business transactions selected. Selecting P as an example will take you straight into the premises tab as
United Utilities CMOS
button on to search market 09/11/2020 NCC021 LVI Michael Floyd below. Review
Water Limited 10.0
information screen Note that there is an additional tab not displayed on the front screen which is for
business transactions. To navigate to the business transactions page you have to
select one of the 5 buttons above then tab into the business transactions page.
There is currently no way of going directly into the business transactions page.
We often go into this page directly to find out granular information on
transactions performed against a SPID.
Currently a meter network is only displayed from the main meter downwards,
with no information linked directly to the sub meter.
In the “Meter Network” section of the Meter tab in the CMOS Portal, it shows the
sub-meter(s) details associated with the main meter.
CMOS
Meter Network Display 22/10/2020 NCC022 LVI MOSL Ricardo Wissmann-Alves When clicking on the sub-meter, regardless of the sub meter type, the user can’t Review
10.0
see that this meter is actually a sub-meter in a meter network. So any updates
must be made with existing knowledge of the meter network that the user wishes
to amend, which will have to be found in the Business Transaction Dossier or in
Core Data.
CMOS offers users the capability to check changes to service components via the
service component history tab.
This tab allows two data states to be compared for the following fields:
Service component type
Service component history display Special agreement flag
18/08/2020 NCC020 LVI United Utilities Michael Floyd Special agreement factor Review
in CMOS
Special agreement reference
Tariff code
Unfortunately, whilst the service components history tab creates a line to indicate
that there has been a change to components with a change date, it does not
actually give an indication as to what change has been made.
ORWG December 2020 24NCC tracker
Date Trading party Target
Request title NCC Ref Category Name Status
submitted submitted Describe the issue Release
As part of investigations into splits & mergers, unpaired SPIDs and other activities,
users often perform a postcode search in CMOS. The search does seem to have
some logic however, it appears to be based on a combination of created time/date
and premises address, with any address starting with a 1 at the top of the returned
search. Although numbers beginning with 1 are found at the top of the list they
are not sequential as CMOS orders them based on the subsequent digit. For
example as per the screenshot below 147 is displayed on the list above 15 -15a.
When expanding the search to include 20 SPIDs (including some sewerage only
SPIDs) it becomes clear that where there are sewerage only premises these are
Postcode search on search market United Utilities displayed at the top of the search results. The assumption is that the search is
27/07/2020 NCC018 LVI Michael Floyd Rejected
information screen Water Limited based on water SPID and as the premises address is blank, the SPID is displayed at
the top of the search. At this point the sewerage address is then displayed in
numeric followed by alphabetic order based on the sewerage premises address for
the sewerage only address, then based on the water premises address for the
remainder of the first 20 SPIDs created.
As an example of SPIDs being returned based on creation, 143 Test Road is not
included when searching on 10 SPIDs as this was not one of the first 10 SPIDs
created however, when searching based on 20 SPIDs this appears above 147 Test
Road as expected.
This is in addition to functionality requested as part of NCC013.
When reviewing a SPID's core data in the premises tab in LVI there is a supply
SPID status to be displayed on United Utilities CMOS
27/07/2020 NCC019 LVI Michael Floyd point's section which provides a hyperlink to go into the supply point. The service Review
premises screen Water Limited 10.0
category, multiple wholesaler flag & landlord SPID flag are also displayed.
It would be useful if CMOS could provide some form of indication as to when one
or more of the SPIDs have been deregistered
In line with MOSL IT Road Map, MOSL wants to provide a unified user interface to
the market and want to establish a wider set of services to support this. These or
CMOS API Gateway 26/06/2020 NCC015 LVI MOSL John Briggs Review CMOS 9.0
similar services may, in time, be used in building the Bilaterals replacement
solution.
Within CMOS there are various scenarios that allow for different types of meter
readings to use the same date. The issue is that when this happens, both meter CMOS
Same Day Meter Readings Display 26/06/2020 NCC016 LVI MOSL Chris Dawson Review
readings are accepted but the first one is retired. Therefore the user only have 10.0
visibility of the "Active" meter readings.
CMOS
Meter DPID Association History 26/06/2020 NCC017 LVI MOSL Ricardo Wissmann-Alves When a period of Meter-DPID association is split up by a period of disassociation, Review
10.0
the portal only displays the latest record, rather than all periods of association.
ORWG December 2020 25Forthcoming Changes Update Amanda Hinde, Market Design
Implemented Code changes (6 Nov – 13 Nov 2020)
CMOS
Ref # Title Impact Update
Establishing Trade Effluent Ofwat decision: 30 April 2020
CPW077 Y
Charging Strength Methodology Implemented: 6 November 2020
Ability for Wholesalers to add Ofwat decision 6 July 2020
CPW087 Y
meter reads Implemented: 6 November 2020
Ofwat decision: 1 October 2020
CPM019 GDPR Data Security Standards N
Implemented: 6 November 2020
CPW102/ Gender Neutrality in the Market Ofwat decision: 8 October 2020
N
CPM030 Codes Implemented: 6 November 2020
CPW092/ Unified Disputes Process and N Ofwat decision: 26 October 2020
CPM029 Committee Implemented: 13 November 2020
ORWG December 2020 27Changes awaiting implementation
CMOS
Ref # Title Impact Update
T101 Adding Ofwat decision: 1 October 2020
CPW089 Y
Implementation: 14 May 2021
Billing Address fields
ORWG December 2020 28Ofwat Rejections
CPW061: Unsecured Credit Allowance (UCA) reflecting payment history (No CMOS impact)
To create a new type of UCA for retailers that would be based on strong payment histories
Ofwat considered there were insufficient rationale and evidence to support the case
Decision: 27 November 2020
CPW086: Unsecured Credit Allowance (UCA) – Rebalancing (No CMOS impact)
To increase the UCA for retailers with the highest level of credit assessment score from 20% to 40%
Ofwat considered there were insufficient evidence
Decision: 27 November 2020
CPW088: Enabling Wholesaler and Retailer to update meter location data (CMOS impacting)
To allow retailers to update GIS X and Y coordinates, and to allow wholesalers to amend the Meter Location Free
Descriptor Field
Ofwat considered this could blur responsibilities for maintaining meter location data and did not offer value for
money
Decision: 7 December 2020
ORWG December 2020 29Decisions at Panel Meeting (24 Nov 2020) (No CMOS impact for all three change proposals below) CPM036: Provision to vary term of Panel Members Allows Ofwat to grant the Panel an extension to the term of elected Panel Members Introduced due to coincidence of 2021 elections with Panel’s Governance Review recommendations Recommended for implementation 15 February 2021 CPW105: Provisions of Information Obligation Previously part of CPW059 which included the annual review of technical parameters Requirement for MOSL to prepare list of non code obligated documents remains but Panel no longer required to review their circulation Recommended for implementation 14 May 2021 CPW033: Improved process for managing Data Subject Rights Requests (GDPR) Developed by the GDPR Issues Committee Sets out timescales, responsibilities and recording of DSRR Requests Recommended for implementation 15 February 2021 ORWG December 2020 30
Decisions at Urgent Panel Meeting (2 Dec 2020) CPW108: Agreement to Unplanned Settlement Runs (USR) (No CMOS impact) Applies a backstop to requests for USR to prevent them becoming deadlocked Requests are deemed rejected after 20 business days Requesting party can then raise a dispute Recommended for implementation 5 days after Ofwat decision ORWG December 2020 31
CPW106: De-registration of long-term vacant SPIDs Amanda Hinde, Market Design Janet Judge, Market Performance
What is the change? (Overview)
CPW106 seeks to create an enduring process to identify ineligible Supply Points
and create a backstop to facilitate their deregistration from CMOS, in order to
reduce the number of long-term vacant Supply Points in the market.
This change defines ‘Long-Term Vacant Supply Points’ as Supply Points associated
with premises that have been marked as Vacant Premises for six (6) consecutive
years or more with no identifiable consumption within the last six (6) years.
MOSL will be the backstop to remove the SPIDs that have not been dealt with on
the Candidate list.
ORWG December 2020 33What are we doing? (High-level steps)
1. MOSL produces 2. Retailer 3. Wholesaler 4. MOSL is backstop to
Candidate Lists of long- reviews reviews “SPERR” SPIDs left on
term vacant SPIDs Candidate List Candidate List Candidate List
Candidate Lists Retailer has 3 MOSL uploads MOSL monitors
uploaded onto MO months to review reviewed Candidate progress using API
Portal for the the Candidate List Lists onto MO Portal
Retailers
Wholesaler has 12 2 Days after issuing
For SPIDs that comms, MOSL
months to
should not be deregisters the
investigate and
deregistered, SPIDs using SPERR
deregister SPIDs
submit T112.R with
text comment field
“K21”. For SPIDs that MOSL publishes new
should not be Candidate Lists after
deregistered, submit 3 months
T152.W with “K21”.
Key: MOSL Retailer Wholesaler
34Why are we doing this?
“De-registrations are a low priority for trading parties. They naturally focus on vacant Supply Points
with high consumption or volumetric charges given the revenue and cost impact, rather than
focusing on Supply Points in the market which may need to be considered for deregistration.” –
PwC’s review of vacancy in FY19/20
Problems with high
vacancy levels:
• Inflates size of NHH
market
• Drives additional
operational costs
(e.g. trace activity)
• Inhibits effective
management of
unplanned events
ORWG December 2020 35Next steps Consultation closed on 9 December 2020 MOSL to work with Proposer to consider the 20 responses received Panel recommendation in February 2021 Ofwat to consider change proposal Implementation in: May/August 2021, depending on Ofwat’s decision ORWG December 2020 36
Questions? ORWG December 2020 37
Bilateral Transactions Programme - Update John Gilbert, Programme Director
Agenda
Headlines
“ Our primary objective [for this business plan] is to
make the market an easier place to do business in.
The most significant market improvement that many
members and the regulator want is a faster, simpler,
Programme plan more reliable way of managing bilateral interactions
between wholesalers and retailers.
Code change process
AOB This will be the most fundamental
development since market opening.
”
MOSL 2020/21 Business Plan
ORWG December 2020 39Headlines
Programme: full steam ahead
Full Business Case approved 30 September
Trading Party CEOs updated
Expect to announce closing of £420k in-year funding gap in January
Met ‘accelerated delivery of C1’ challenge and looking at additional CGI support in Q4
Development: excellent progress - but timescales remain challenging
Process analysis by OAG/TAG: Meter verification (C1) and Meter replacement (B5)
C1.R (Retailer-initiated C1 process) user requirements complete - now with technical team
C1.W (Wholesaler) process analysis shows close matching – good news
Development: 11th two-week ‘sprint’ began today
Invaluable input by advisory groups
Demo of portal and first process on 21 October
Priority order all 67 done – output to be considered by MOSL Business Owner/OAG in January
ORWG December 2020 40Headlines (continued)
Key docs* reviewed by Pathfinders issued to trading
parties today (Tuesday 15 Dec):
Code Subsidiary Document CSD0601
XSDs (XML schemas)
Interface specification for C1 (meter verification)
Proformas to carry out system comms test (‘ping test’)
Go-live and implementation
Implementation, ‘cutover’ and training plans taking shape
Test strategy issued for review
Presenting to Contract Managers at next six-weekly planning
update 27 January *Available at www.mosl.co.uk/bilaterals_latest
ORWG December 2020 41ORWG December 2020 42
Bilateral Transactions Programme Ofwat 31 July 2021 30 Nov 2021 31 March 2022 30 June 2022 30 Nov 2022
Programme Board 14 December 2020 CPW070 Phase 2 Phase 3 Phase 4 Phase 5 Phase 6
timetable 1 process +7 processes (total: +8 processes (total: 16) +12 processes (tot: 28) +TBC
8)
2021 2022
Oct Nov Dec Jan Feb Mar Apr May Jun July Aug Sept Oct Nov Dec Jan Feb Mar Apr May Jun July Aug Sept Oct Nov
Pilot Delivery & integration
Pilot Dem Developmen Software releases
Key
o t
New activity
CPW Interfac Securit Amber: in
High level plan
070 e spec y test progress/at risk
Test Green: on track
envt
PathF
LVI
Reminders:
Activity
C1W
test
withdrawn 67 processes in scope
C1R Ping Pathfinder Activity complete
scenari
os
scenari
os
testing HVI Testing Front-end-loaded: 8 processes = “80%”
C1R
Priorit
y
Intern
al Notif Mandatory
Previous position
& direction of
Implement over 6 phases
stori ic-
es
order
revie
audit
issued ation
adoption
point
movement
First process goes live in April
Vali
w
XSD/
OSD
d-
(LVI or HVI) Available via portal immediately (optional)
atio
n Three-month integration period
B5 Rule
B5 C1 Trading Party
Revie s
Reqm HVI Integration Adoption mandated at end of July 2020
w t NOTE:
Process priority Speed of rollout will accelerate
order subject to
Other B5 started B5 HVI TP integration periodic review
Processes G2 HVI TP integration
On course for April 2020
B1 TP integration
HVI
C2 H1 TP integration
HVI
2 C5 F5 HVI TP integration
processes TP integration
C6 B3
per sprint HVI
C3 I1 HVI TP integration
G4 C4 TP integration
HVI Further processes (TBC) Further processes (TBC)
I8 [X] HVI TP integrationImplementation - Level 0 milestone plan
2021 2022
Nov Dec Jan Feb Mar Apr May Jun July Aug Sept Oct Nov Dec Jan Feb Mar Apr May Jun July Aug Sept Oct Nov
31 July 2021 30 Nov 2021 31 March 2022 30 June 2022 30 Nov 2022
Ofwat Phase 2 (P2) Phase 3 (P3) Phase 4 Phase 5 Phase 6
CPW070 1 process +7 processes (total: 8) +8 (total: 16) +12 (total: 28) +TBC
Level 0 Milestones (0-6 months)
Support Services Live P2 – Benefits Realisation P3 – Benefits Realisation
Key
On Track
C1 – Technical Specs Shared At Risk
Training Materials Live P2 – Cutover P3 – Cutover P3 – Performance Cutover
Not on Track
Completed
C1 – Go Live P2 – Performance Cutover Support Handover to BAU
C1 Meter Verification Process
LO Milestones Date Status Sign off criteria L1 Milestones
XSDs Complete
C1 – Technical Documents live on the Data Catalogue Complete
16/12/2020
Specifications Shared MOSL website
Interface Spec Complete
Support Setup Complete
Support Services All level 1 milestones Resource Training Complete
19/03/2021
Live complete
Requirements Complete
Training Materials Training materials live Training Sessions Complete
26/03/2021
Live on MOSL website
Training Videos Complete
Process Review Complete
C1 process
C1 – Go Live implemented into live User Stories Complete
01/04/2021
for all interfaces Screens/Transactions Implemented
ORWG December 2020 44Code change process
Code change process ORWG December 2020 46
47
Final thoughts
1. Bilaterals is a team effort - thank you
2. It’s ‘becoming real’
3. Q4 is going to be busy
4. So is the rest of 2021..
5. Are you ready?
6. Issues? Queries? www.mosl.co.uk/bilaterals
bilaterals@mosl.co.uk
ORWG December 2020 48Data Issues / Improvements
Data issues – CMOS/Remedy users
CMOS users – October we had 5,404 active users. Today 5,570
An increase of 166 users since October (from Aug to Oct increase of 120 users)
CSD006 – Trading Party Administration and Notification Process (Market Codes
> Codes)
Trading Parties are responsible for maintaining user accounts on the Market
Operator Systems, including the termination of accounts as part of the exit
process for leavers, but always within one (1) week of the user having left the
Trading Party.
Remedy users – maximum 10 users per Trading Party – on track
ORWG December 2020 51Data issues – Notifications (Txxx.M)
CGI is investigating some notifications that haven’t been sent
How TPs use the notifications?
Automated system?
Ignore them?
ORWG December 2020 52Data issues – Unplanned CMOS outages – 10 November 2020
Incident (Outage 2hr 1min)
The CMOS application (LVI and HVI) was unavailable to all Trading Parties
It was identified that the deployment of the API Gateway had an incorrect setting
The incorrect setting was corrected and applied, restoring the CMOS Application (LVI and HVI).
Root Cause
The API Gateway was released via the automated release tool, this included a new certificate.
An incorrect value was entered manually into the automated release tools configuration as the incorrect certificate
was selected.
Lessons Learnt and next steps
All system/server certificates used across CMOS will be revalidated, consolidated and centrally managed by Global
Technology Operations (GTO)
Review of certificates within the automated release tool for Production releases to be undertaken
All release activity to the Production environment will be reviewed by the Change Advisory Board (CAB) prior to
implementation into the Production environment - Production environment releases will only be performed outside
of core business hours
All resilient systems will be upgraded to ensure availability in the event of a failure.
Additional manual steps will be undertaken to validate all certificates have been installed and bound to the relevant
components.
ORWG December 2020 53Data issues – Unplanned CMOS outages – 26 October 2020
Incident (Outage 2hr 9min)
MOSL reported that they were unable to access the CMOS portal.
The CGI support team also advised that the server (MOSL-LVL-DB01) was not reachable.
Root Cause
The host (ccc-lpo-host-02) where MOSL-LVL-DB01 resides was restarted in error by an engineer who was managing an
incident for another client
The host and cable were correctly labelled, and the job sheet had the correct information on it
The engineer disconnected the wrong cable – human error.
Lessons Learnt and actions taken
In order to prevent a reoccurrence, the teams have reviewed the process for restarting any kit within the CGI
Commercial Cloud (CCC)
The physical labels on the host systems within the CCC have all been checked and are correct
A two-person ruling has been implemented such that each restart request will be checked by a 2 nd person before any
work is carried out.
ORWG December 2020 54Data issues – Unplanned CMOS outages – 4 June 2020
Incident (Outage 1hr 7min)
CGI lost the ability to manage the secondary firewall
To rectify this a planned multi-client firewall change was approved by GTO management to restart it
No outage was expected as the network traffic would transfer to the Primary firewall
Following the completion of the change an incident was raised as MOSL were unable to access the CMOS application
The CGI Application Team also confirmed that they were unable to access CMOS production.
Root Cause
A firewall rule that allows incoming access to the front-end web servers was corrupted from the CGI internet firewall
Following discussions with the Senior Comms Engineer he concluded that during the restart, the firewall record had
become corrupted
A problem ticket was raised but we have been unable to find any root cause.
Lessons Learnt and actions taken
Since this incident we have managed many other Multi Client Changes that involve the failover from the primary to
the secondary firewall and vice versa, all have completed successfully.
ORWG December 2020 55Data issues – Unplanned CMOS outages – 17 April 2020
Incident (Outage 7hr 15min)
The CGI Application team were unable to login to the CMOS Application and Web servers in the pre-production
environment
This caused delays to a pre-production environment deployment due for release that day.
Root Cause
A change was required to add Firewall rulesets to allow access to the Qualys - Enhanced Vulnerability Scanner
During the implementation, the engineer didn’t follow the documented process and didn’t review the underlying
rulesets that were appended to the existing access-lists
As a result, access to the Enhanced Vulnerability Scanner was allowed but all outbound traffic to the internet was
denied.
Lessons Learnt and actions taken
The CGI Team has reviewed and updated the Firewall change process to ensure that it includes a two-member review
by the 3rd Line Comms Team and the MOSL Infrastructure Design Architect before being approved and implemented.
ORWG December 2020 56Data issues – Unplanned CMOS outages – 11 February 2020
Incident (Outage 4hr 24min)
MOSL were unable to access the MPS and MPS2 services hosted in CGI Commercial Cloud (CCC)
The CGI Development Team were unable to access the Development servers
A priority 1 incident was raised – it was quickly identified that the incident was affecting multiple Trading Parties.
Root Cause
Inter-site links (10 Gig) were decommissioned by Virgin Media following a “cease request” issued by CGI
The “cease request” was sent correctly by CGI but due to a record management error at Virgin Media, the inter-site
links were registered incorrectly resulting in the outage.
Lessons Learnt and actions taken
Revised the overall “cease request” process. This now includes a requirement for the 3 rd Line Telecoms Team to check
the Impact Assessment of any cease request and sign off before the “cease request” is issued to vendors
The issue has been raised with Virgin Media who have identified a training gap - their agent failed to make the
appropriate checks as defined in the CGI Business processes. The Finance Planner role should check any “cease
requests” to ensure they have the correct circuit reference and that a change reference has been applied. This process
has now been added to the CGI internal knowledge base so the Control Bridge Comms Team can also perform checks.
CGI has also reviewed all other third-party processes
ORWG December 2020 57Data issues – Unplanned CMOS outages – 11 February 2020
Lessons Learnt and actions taken (cont.)
A vendor reference should have been received when engaging with Virgin Media. If the request is urgent and a
telephone call is made an e-mail follow up is now required to generate a vendor reference. The reference was not
received, and no follow up call was made. The correct process has been reiterated to staff to ensure a reference is
supplied
CGI has successfully completed a full audit on all 10gig links to ensure that all are recorded and labelled against the
correct customer
Proactive monitoring, alerting and priorities for critical devices and ports have been fully reviewed. All alerts are now
set at P2 priority and the circuit references have been added to the alert messages along with the multi-client name.
ORWG December 2020 58Data issues – Transaction submission
HVI Transactions
Maximum number of transactions to be submitted: 3,600/hour/TP (since the
end of March increased to 4,000/hour/TP from 8am to 6pm and, 6,000/hour
between 6pm and 12am and 3:30am to 8am). No transactions to be
submitted between 12am and 3:30am.
MVI Transactions
5,000 transactions/TP/month
We have increased this since the end of March for specific situations, but
larger volumes need to be communicated in advance for acceptance and
planning.
ORWG December 2020 59Improvements – Observation Log
Purpose: To simplify the observation/workaround log process internally and
externally.
Removed, altered or added
Defect/Observation/Design Closing date (open tab only)
Transaction 2 Workaround tab (open)
Transaction 3 Workaround tab (closed)
Transaction 4 Links added to relevant observation and
Technical layer
workaround tabs.
Columns rearranged to read easier
Example spreadsheet will be uploaded to the website (CMOS > ORWG > Documents) file: Example -
Observation Log.xlsx
ORWG December 2020 60Data issues – GlobalSign Certificates The Certificate Authority (CA) industry was alerted of compliance implications related to the inclusion of a specific extension (OCSP-signing extended key usage) in CA certificates which has, under certain conditions, unintended compliance and security implications GlobalSign PersonalSign issuing CAs "GlobalSign PersonalSign 1 CA - SHA256 - G3 and GlobalSign PersonalSign 2 CA - SHA256 - G3" are impacted by this issue No key compromise or security incident has taken place, and the change is being performed solely in the context of remediating a compliance issue and addressing any potential, unmaterialized security risk. The revocation event of the certificates will take place on 24 February 2021 ORWG December 2020 61
AOB and wrap up Slides will be published on the website this afternoon Meeting minutes to be published on the website in 2 days (CMOS > ORWG) Date of next meeting: 9 February 2021 – virtual meeting April meeting: 6 April 2021 – virtual meeting Everyone subscribed to the mailing list? ORWG December 2020 62
Merry Christmas
&
a great New Year
to all of us!!!
ORWG December 2020You can also read
