Implementing an Enterprise Order Database with DB2 pureXML at Verizon Business - Andrew Washburn, Verizon Business
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Implementing an Enterprise Order Database with
DB2 pureXML at Verizon Business
Andrew Washburn, Verizon Business
Matthias Nicola, IBM Silicon Valley Lab
Session TLU-2075
October 25–29, 2009 • Mandalay Bay • Las Vegas, Nevada
0Agenda
Verizon Business*
Brief Recap: DB2 pureXML
Enterprise Order Management at Verizon
– Requirements & Challenges
Orders as Business Object Documents (BODs)
Database Design Considerations
Implementation
Performance Tests
Benefits, Lessons Learned
*- All Verizon information is provided for informational purposes only. Verizon makes no representation or warranty of
1
the accuracy of this information, and no license in such information is granted. All rights are reserved.Verizon Business
Delivers IP, data, voice, wireless, networks, security,
mobility and other communication services
$21.1B revenues in FY2008
321 offices in 75 countries
Customers include 98% of Fortune 1000
Verizon Business connects customers in 2,700 cities
in 150 countries
Network includes 485,000 miles of terrestrial and
undersea cable, spanning 6 continents
2Overview of DB2 pureXML (1 of 2)
• XML stored in a parsed hierarchical format
• No fixed XML schema per XML column required
• XML schema validation is optional, per document
• XML indexes for specific elements/attributes
• XQuery and SQL/XML integration
create table customer (cid integer,…, info XML)
DB2 Storage
Relational Storage
page page page
pureXML Storage 3Overview of DB2 pureXML (2 of 2)
1 create table customer (cid integer, info XML)
2 insert into customer (cid, info) values (?, ?)
3 select cid, info from customer
4 select xmlquery('$INFO/customer/name')
from customer
where xmlexists('$INFO/customer/addr[zip = 95123]')
5 create index idx1 on customer(info) generate keys using
xmlpattern '/customer/addr/zip' as sql varchar(12)
6 Plus: updates, XML Schema support, utilities, etc. 4Agenda
Verizon Business
Brief Recap: DB2 pureXML
Enterprise Order Management at Verizon
– Requirements & Challenges
Orders as Business Object Documents (BODs)
Database Design Considerations
Implementation
Performance Tests
Benefits, Lessons Learned
5Enterprise Order Management Overview
Challenge
– Orders come in through different channels, stored in channel-specific
repositories
– Coherent view of all orders is required
– Reporting and tracking across all orders is critical
– Orders are XML documents (OAGIS standard)
– Medium and complex queries and updates
– Need a consistent way to figure out what changed on an order
Solution
– Single order database, hybrid storage in DB2 pureXML
– SQL, XQuery, XQuery Update in Stored Procedures
– Utilitze XQuery and XML to solve what changed from one order revision
to the next
• Many OE systems tried to solve this problem now coded in one place 6Enterprise Order Management (EOM)
“The Big Picture”
Customer Interface
Sales & Contracts
Order Entry & Management
OrderPro
Let’s zoom in…
DOMS
iOrder iOE
EODB
Enterprise Service Bus
EOM
Service Profile
Provisioning Workflow Billing
Activation / Inventory
Work Dispatch
7A typical EOM Architecture
Heterogeneous
Order Entry Systems
System1 System2 System3 System4 System5 System6
Converting XML to Converting non-XML
different data formats data to XML
XML XML
Enterprise Order Management
EOM Application
8
8Characteristics of a Heterogeneous Architecture
Each order entry system stores an order by transforming
standard OAGIS XML BODs to a relational schema
Converting XML to relational format and back can be
expensive
– Manual work to design the mapping between the data formats
– Resource consumption (CPU cycles)
Heterogeneous order repositories
– Optimized for order entry rather than for global order tracking
and reporting
– Typically use product-specific database schemas
– Provide widely varying levels of data access
9New EOM Architecture, with DB2 9.5
Single order database, common for all OE systems, product-agnostic
No shredding and construction of XML
High performance, granular data access
Order Entry Systems
System1 System2 System3 System4 System5 System6
ESB
EODB
(DB2 pureXML)
XML
Enterprise Order Management
10
EOM Application
10New EOM Architecture, with DB2 9.5
Single order database, common for all OE systems, product-agnostic
No shredding and construction of XML
High performance, granular data access
Order Entry Systems
System1 System2 System3 System4 System5 System6
ESB
pureXML SQL &
storage XQuery
DB2 9.5, Linux,
No XML Intel-based
Conversions
Enterprise Order Management server, HADR
11
EOM Application
11Agenda
Verizon Business
Brief Recap: DB2 pureXML
Enterprise Order Management at Verizon
– Requirements & Challenges
Orders as Business Object Documents (BODs)
Database Design Considerations
Implementation
Performance Tests
Benefits, Lessons Learned
12What is OAGIS, what is a BOD ?
OAGIS (Open Applications Group Integration Specification) provides a
canonical business language for information integration, in XML.
XML message format for Business Object Documents (BODs)
BODs use OO design patterns: they define a noun (object) that has
verbs to indicate the action to be performed
Verb
Noun
The name of the BOD is the verb-noun combination in the data area
13XML Schema for Business Object Documents (BODs)
ProcessServiceOrder
DataArea
ServiceOrder (ORDER)
ServiceOrderLine (LINE)
14XML Schema for Business Object Documents (BODs)
ProcessServiceOrder
DataArea
ServiceOrder (ORDER)
ServiceOrderLine (LINE)
Product (PRODUCT)
Services
…Service (SERVICE)
15XML Schema for Business Object Documents (BODs)
ProcessServiceOrder
DataArea
ServiceOrder ORDER
(ORDER) • Most BODs are 25kb to 500kb
LINE • Some in the MB range
ServiceOrderLine
(LINE)
PRODUCT
• Most BODs have 3-5 services
SERVICE • Some have dozens or more
Product (PRODUCT)
SUBSERVICE
Services
…Service (SERVICE)
…
16Agenda
Verizon Business
Brief Recap: DB2 pureXML
Enterprise Order Management at Verizon
– Requirements & Challenges
Orders as Business Object Documents (BODs)
Database Design Considerations
Implementation
Performance Tests
Benefits, Lessons Learned
17Workload & Requirements
High frequency/priority
– Update individual service Database
– Summary queries design
Medium frequency/priority
– Get individual service
– Insert new order
– Get order
– Get order summary
Application characteristics
– OK to favor update and query performance over insert
performance
– Frequent granularity of access: 1. Service 2. Order
18Database Schema – Design Options
order
(…)
oid … order
XML
(…)
…
…
(…)
19Database Schema – Design Options
Extract important XML fields into
relational tables.
order Orders are stored fully intact as
XML in the order table.
oid … order
No other table contains XML.
XML
(…)
line service
(…)
lid oid … sid oid …
…
…
(…) product subservice
pid oid … ssid sid …
20Database Schema – Design Options Extract important XML fields
into relational tables.
Also extract Service XML
fragments.
order Orders are stored as XML, but
with all Services removed.
oid … order
XML
(…)
line service
(…) lid oid … sid oid … service
XML
…
XML
…
(…)
product subservice
pid oid … ssid sid …
21
Best design for the given workload and requirementAgenda
Verizon Business
Brief Recap: DB2 pureXML
Enterprise Order Management at Verizon
– Requirements & Challenges
Orders as Business Object Documents (BODs)
Database Design Considerations
Implementation
Performance Tests
Benefits, Lessons Learned
22Insert Stored Procedure
CREATE PROCEDURE INSERT_BOD (IN BOD XML)
LANGUAGE SQL
P1: BEGIN
DECLARE orderID BIGINT;
SET orderID = NEXTVAL FOR order_id_seq;
INSERT INTO order (id, tin, tin_version, CONTRACT_ID, CONTRACT_NAME, XML_DOC,
insert_datetime, update_datetime)
SELECT
orderID, T.tin, T.tin_version, T.contract_id, T.contract_name, T.XML_DOC,
CURRENT_TIMESTAMP AS insert_datetime, CURRENT_TIMESTAMP AS update_datetime
FROM xmltable('$XML_DOC' passing BOD AS "XML_DOC"
COLUMNS
tin VARCHAR(128) PATH '.../*:ServiceOrderHeader/*:DocumentID/*:ID',
tin_version BIGINT PATH '.../*:ServiceOrderHeader/*:DocumentID/*:RevisionID',
contract_id VARCHAR(32) PATH '.../*:ServiceOrderHeader/*:Contract/*:ID',
contract_name VARCHAR(128) PATH '.../*:ServiceOrderHeader/*:Contract/@schemeName',
XML_DOC XML PATH 'copy $new := . modify do delete
$new/.../*:ServiceOrder/*:ServiceOrderLine/*:Product/*:Services/*
return $new'
) AS T;
...
blue dots … : paths are abbreviated to fit on slide 23Insert Stored Procedure (cont’d)
...
SET productID = NEXTVAL FOR product_id_seq;
INSERT INTO product (id, line_id, order_id, identifier, code, name, line_number,
insert_datetime, update_datetime)
SELECT productID, lineID, orderID,
T.IDENTIFIER, T.CODE, T.NAME, T.LINE_NUMBER,
CURRENT_TIMESTAMP AS insert_datetime,
CURRENT_TIMESTAMP AS update_datetime
FROM XMLTABLE('$XML_DOC/.../:*DataArea/*:ServiceOrder/*:ServiceOrderLine/*:Product'
passing BOD AS "XML_DOC"
COLUMNS identifier VARCHAR(32) PATH '*:ID',
code VARCHAR(32) PATH '*:Code',
name VARCHAR(32) PATH '*:Name',
line_number BIGINT PATH '*:LineNumber') AS T;
...
...
END
24Insert Stored Procedure (cont’d)
...
INSERT INTO service (ID, ORDER_ID, PRODUCT_ID, SEQUENCE_NUMBER, TYPE, IDENTIFIER,
CATEGORY_CODE, PRODUCT_OFFERING_ID, NAME, PRODUCT_IDENTIFIER,
NODE_ID, ACTIONCODE, XML_SERVICE, insert_datetime,update_datetime)
SELECT NEXTVAL FOR EODB.SERVICE_SEQ,
orderID, productID, T.SEQUENCE_NUMBER, T.TYPE, T.IDENTIFIER, T.CATEGORY_CODE,
T.PRODUCT_OFFERING_ID,T.NAME, T.PRODUCT_IDENTIFIER,T.NODE_ID,T.ACTIONCODE,
T.XML_SERVICE,
CURRENT_TIMESTAMP AS insert_datetime, CURRENT_TIMESTAMP AS update_datetime
FROM
XMLTABLE('$XML_DOC/.../*:DataArea/*:ServiceOrder/*:ServiceOrderLine/*:Product/*:Services/*'
passing BOD AS "XML_DOC"
COLUMNS
SEQUENCE_NUMBER FOR ORDINALITY,
TYPE VARCHAR(32) PATH '@type',
IDENTIFIER VARCHAR(32) PATH '*:ID[1]',
CATEGORY_CODE VARCHAR(32) PATH '*:CategoryCode',
PRODUCT_OFFERING_ID VARCHAR(32) PATH '*:ProductOfferingID',
NAME VARCHAR(64) PATH '*:Name',
PRODUCT_IDENTIFIER VARCHAR(32) PATH '*:ProductId',
NODE_ID VARCHAR(32) PATH '*:NodeID',
ACTIONCODE VARCHAR(128) PATH '*:ActionCode[1]',
XML_SERVICE XML PATH 'document{.}' ) AS T;
...
25Update an Existing Service in an Order
DB2 9.5
Order
Update BOD
update existing service in BOD Order
BOD
BOD
Order
BOD
“Update BODs” represent messages to update existing BODs
Typically smaller than the existing BOD
If an element exists in the “Update BOD”, but not in the existing BOD,
it needs to be inserted, otherwise updated
Requires “merge” operation of two XML documents
26DB2 Stored Procedure Implements Update
Update
DB2 9.5
BOD
Order
BOD
Order
BOD
Stored Procedure
UPDATE_SERVICE() Order
BOD
SQL/XML Update
Statement
The stored procedure
Reads the "Update BOD" and the "Target BOD"
Generates appropriate XQuery update operations
And applies update statement to the "Target BOD"
27
All within DB2!SP Generates and Executes Update Stmt
Stored Procedure
UPDATE_SERVICE()
SQL/XML Update
UPDATE service
SET xml_service = XMLQUERY(' Statement
copy $ORIGINAL := $XML_SERVICE
modify (
do insert
$UPDATE/*:ProcessServiceOrderRequest/*/*:DataArea/*:ServiceOrder/
*:ServiceOrderLine/*:Product/*:Services/*/*:ProvisioningStatus
after $ORIGINAL/*/*:Status
,
do replace $ORIGINAL/*/*:ServiceContact/*:ID
with
$UPDATE/*:ProcessServiceOrderRequest/*/*:DataArea/*:ServiceOrder/
*:ServiceOrderLine/*:Product/*:Services/*/*:ServiceContact/*:ID
)
return $ORIGINAL '
PASSING XMLCAST(? as XML) AS "UPDATE"
WHERE node_id = ? AND type = ? AND product_offering_id = ? AND order_id = ?
28"GET" Stored Procedure
EOM struggled with memory issues due to the size of the XML
mapped into Java memory
Needed a way to “strip down” the BOD
Solution
– Use XQuery and SQL/XML to return order summary BOD where
all non-essential data is left in the DB
– Reduces memory footprint in EOM by 75%
Also needed a more granular “component” data access
– GET stored procedure can run the gamut
• Whole order
• Order summary
• Single service
• Single service with only some subcomponents
• Could return single field if necessary
– Not limited at all by the XML structure
29Agenda
Verizon Business
Brief Recap: DB2 pureXML
Enterprise Order Management at Verizon
– Requirements & Challenges
Orders as Business Object Documents (BODs)
Database Design Considerations
Implementation
Performance Tests
Benefits, Lessons Learned
30Concurrency and Performance Tests
Using the TPoX Workload Driver (http://tpox.sourceforge.net/)
Workload:
– 10% insert order
– 50% update service
– 15% get service
– 10% get order by tin (main identifier)
– 5% get order by alternate ID
– 10% get order summary
– Each concurrent user keeps calling all of these stored procedures
randomly, but based on their relative weights
In each stored procedure call, a different BOD is inserted or retrieved
(i.e., no trivial repetition of operations).
Test system with 300,000 order BODs 31Concurrent Performance Test Results
EODB Transactions per second
180
160
140
Test system 120
100
• Old pSeries Power4
80
• 4 single-core CPUs 60
• 1500 MHz / CPU 40
20
0
1 2 3 4 5 6 7 8 9 10
Number of concurrent users (read/write)
Higher performance and throughput on modern production hardware!
32Performance Tests - Results
Scalability of 100 Inserts with 1, 2, 3 Services
140 m s The time to insert is
128.3 m s
120 m s roughly linear to the
Avg Elapsed time per insert
100 m s
80.4 m s
number of services
80 m s
60 m s
39.7 m s
40 m s
20 m s
0 ms
1 s ervic e 2 s ervic es 3 s ervic es
n u mb e r of se rv ice s in th e B O D Insert Performance for Large BODs
4 k b /m s
in s e r t t h r o u g h p u t in k b /m s
3 k b /m s
2 k b /m s
Even for very large 1 k b /m s
BODs, the time to
0 k b /m s
insert is linear in the
1 mb
2 mb
3 mb
4 mb
5 mb
6 mb
7 mb
8 mb
9 mb
10 mb
11 mb
12 mb
13 mb
14 mb
15 mb
size of the BOD Siz e Of th e Bod
33Performance Tests - Results
All critical operations perform with < 50ms response times
Elapsed Time per Stored Procedure Call
50 ms
BOD_SUMMARY_BY_TIN
BOD_BY_ALTERNATEID
40 ms
Elapsed time
30 ms
BOD_BY_ORDERID SERVICE_INCONTEXT
BOD_BY_TIN SERVICE_INCONTEXT BY_PARAMS
BY_SERVICEID
20 ms
UPDATE
SERVICEID
10 ms
0 ms
get BOD get BOD summary get Service update ServiceID
34Agenda
Verizon Business
Brief Recap: DB2 pureXML
Enterprise Order Management at Verizon
– Requirements & Challenges
Orders as Business Object Documents (BODs)
Database Design Considerations
Implementation
Performance Tests
Benefits, Lessons Learned
35Summary & Benefits of the Design (1/2)
Hybrid XML/relational storage and indexing, designed to support
specific workload and requirements
XML manipulation expressed in declarative terms (i.e., SQL/XML), not
in a procedural language
Insert, Query, Update logic in SQL/XML stored procedures instead of
Java code
• Data manipulation happens close to the data
• Number of JDBC calls greatly reduced
• Better performance
Thin Java application layer calls stored procs
– Less code, lower complexity
Leveraging the power of XQuery and SQL/XML for complex
document manipulation in DB/2
– More efficient storage so no parsing
– Leave the data in DB; don’t pull it into Java memory where the 36
footprint can be 10x or more than the raw XML footprintSummary & Benefits of the Design (2/2)
Database schema and design is product-agnostic,
hence easier to maintain
• Less maintenance to add new products
• Shorter time to market
Supports advanced operations such as
get_service and get_order_summary
• Applications can get data at a more granular level
• Less resource consumption, better performance
• Efficient updates of individual services
37Benefits of the DB2 pureXML Solution
Less application code
Lower complexity
Better performance
Single consistent view of all orders, easier and more
accurate reporting
Data access at more granular levels
Easier (less expensive) maintenance, e.g., to
introduce new products
Reduced time to market
38Lessons Learned Managing XML in the database efficiently is no longer wishful thinking Initial fears about DB2 pureXML turned out to be unfounded – pureXML performance – Resource consumption – Database and index and maintenance – Performance with large order documents – Schema changes Rapid Application Development with pureXML – Core concepts prototyped within days (POC) 39
Questions ?
40You can also read
