LEVERAGING ONLINE PRICE DATA FROM WEB CRAWLING - Mazliana Mustapa Department of Statistics Malaysia Asia-Pacific Stats Café 30 Nov 2020
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
BANCI
M A L AY S I A
LEVERAGING ONLINE PRICE
DATA FROM WEB CRAWLING
20 OCT
Mazliana Mustapa
Department of Statistics Malaysia
Asia-Pacific Stats Café 2016 - 2030
30 Nov 2020
@StatsMalaysia
@MyCensus2020 www.dosm.gov.my www.mycensus.gov.my
INTRODUCTION
Price Intelligence
leveraging the capability of Big Data in
collecting large data from various sources
and transform them into better structure
different prices of the same good can be
obtained through various websites,
providing a modernized price collection
Transform from unstructured data into
structured data to perform analysis
What is web crawler?
process of repetitively finding and fetching hyperlinks starting from a list of
starting URLs.
What is web scraping?
Web-scraping is automatically retrieving and processing information from
websites
@StatsMalaysia @MyCensus2020 www.dosm.gov.my www.mycensus.gov.my #MyCensus2020 #MakeSureYou’reCOUNTED 2
BIG DATA ANALYTICS JOURNEY
Seminar on Integrated Trade Database with
MSBR: A Big Data Initiative
17 September 2015
Inception Meeting in Colombo, Sponsored by: ADB
Sri Lanka Consultant: UNSD
30 June -2 July 2014 Malaysia Statistical Business Register rich
Organizer: Asian Development with businesses
Bank (ADB) information & become richer when it is
combined with trade database
STATSBDA Project
TEC
STATSBDA Project Embedded in STATSBDA Project
December 2016 - decide to harness the massive
STATSBDA Kickoff Structured Data amounts of trade data using big
2 December 2016 (TEC) data platform.
+
Unstructured Data
STATSBDA (PI & PMAOS)
Completed
May 2018
@StatsMalaysia @MyCensus2020 www.dosm.gov.my www.mycensus.gov.my #MyCensus2020 #MakeSureYou’reCOUNTED 3
OBJECTIVE PRICE INTELLIGENCE
Holistic View in Online and
Objective New Data Collection Offline Price
02 Methodology Allow monitoring and forecast
01 to establish new
New data collection
process introduced to
future price trend and as
valuable input for price control
to give better insight price basket analysis cover price of goods decisions by government
in consumer price which will be used as sell online Value Creation
analysis and value added to the
monitoring current Consumer
Price Index
Transform of Business
Process
New data collected enable to
create holistic landscape of
CPI monitoring process
@StatsMalaysia @MyCensus2020 www.dosm.gov.my www.mycensus.gov.my #MyCensus2020 #MakeSureYou’reCOUNTED 4
PRICE INTELLIGENCE
National Statistical Institutes are increasingly using data from the internet as a source
for their statistics Statistical Journal of the IAOS 28(2012)
Modernization of data
collection tools
Adoption of web scraping techniques
Automated processes implemented using a web crawler
Data is gathered and
copied from the web
A collection of automated and semi automated techniques
Data retrieval and analysis
Discovering;
Previously unknown patterns in data; and
Relationship that can be used for prediction of user-relevant quantiles
@StatsMalaysia @MyCensus2020 www.dosm.gov.my www.mycensus.gov.my #MyCensus2020 #MakeSureYou’reCOUNTED 5
PRICE INTELLIGENCE ARCHITECTURE
Managing massive
Various Online Source/Websites Selected Item
data data • 1Pengguna (Price Catcher) • Groceries
management • Tesco Online • Clothing
repository • Lazada • Appliances
• 11Street • Vehicle
• SenHeng • Electronics
Extracting form • MyMotor • Wearables
relevant
websites webcrawling
Automated web UNSTRUCTURED DATA
crawler Online Retail Platform
Data
ANALYTICS
consolidation on
central local
database PRICE LAKE
Price Analysis
Visualization STRUCTURED DATA
Existing System
VISUALIZATION
@StatsMalaysia @MyCensus2020 www.dosm.gov.my www.mycensus.gov.my #MyCensus2020 #MakeSureYou’reCOUNTED 6
MODULE IN PRICE INTELLIGENCE
Data Acquisition
1
• Collect & Extract data from various sources, both
internal & external
2 Price Lake Development
• Store & structure massive data
3 Data Analysis
Data
Classification • Descriptive
(COICOP • Exploration
dictionary) • Comparison
• Dependency
4 Data Visualization
• Make sense of data
Note : COICOP Classification of Individual Consumption According to Purpose
@StatsMalaysia @MyCensus2020 www.dosm.gov.my www.mycensus.gov.my #MyCensus2020 #MakeSureYou’reCOUNTED 7
DATA MANAGEMENT
PI Dictionary
Pending/Accepted to
match COICOP status
by 4D COICOP group
Partially/exact refers
to matching code with
MCOICOP
11.3% exact match
88.7% partially match
Most word appear in item description
*Information based on 231 item
specifications updates in PI
Dictionary
@StatsMalaysia @MyCensus2020 www.dosm.gov.my www.mycensus.gov.my #MyCensus2020 #MakeSureYou’reCOUNTED 8
DATA MANAGEMENT
PI Dictionary – Text matching scoring
Specification that meet requirement of items in CPI basket of goods with high scoring value
will return exact match, mapped to 7 digits item specification MCOICOP code
Meanwhile, broad specification of items only will return 6 digits item code as long the item is the
same regardless of brand name, units etc
@StatsMalaysia @MyCensus2020 www.dosm.gov.my www.mycensus.gov.my #MyCensus2020 #MakeSureYou’reCOUNTED 9
DATA MANAGEMENT
PI Dictionary
PI Dictionary has
been build in order
to get the best
match to MCOICOP
6/7 digits based on
description of the
items using text
matching scoring
Dictionary will be
updated regularly
based on basket of
goods in CPI
@StatsMalaysia @MyCensus2020 www.dosm.gov.my www.mycensus.gov.my #MyCensus2020 #MakeSureYou’reCOUNTED 10DATA SET GENERATOR
Download Data
1. Click Data Monitoring on the top left of screen
2. Click Folder Name and Click file name
3. Status data generated will display.
Click Download to download to local PC
@StatsMalaysia @MyCensus2020 www.dosm.gov.my www.mycensus.gov.my #MyCensus2020 #MakeSureYou’reCOUNTED 11PRICE INTELLIGENCE ANALYTICS
a) Trend
Trend analysis is used to evaluate data
patterns based on linear approaches. There
are various techniques used in price analysis,
such as ARIMA, Exponential Smoothing, Holt
Winter, Linear Trend, Exponential and Level
Agregation.
b) Descriptive
Descriptive analysis, more emphasis
on data exploration summaries such
as: Mean, Median, Standard
Deviation , Variants, Histogram /
Skewness, etc.
@StatsMalaysia @MyCensus2020 www.dosm.gov.my www.mycensus.gov.my #MyCensus2020 #MakeSureYou’reCOUNTED 12PRICE INTELLIGENCE ANALYTICS
c) Price Basket Enrichment
Price Basket Enrichment is the process of
adding data crawled from online sources into
available data from the SIHP-CPI System.
This process requires DOSM support
because not all prices used in CPI manual
calculation are available online.
d) Dependency
Dependency analysis looks for more connections
between data. An example is to evaluate variables by
performing factor analysis or releasing irrelevant data.
Cluster analysis can be carried out to assess data
relationships (correlations) and other analysis can be
conducted to assess such as Factor Analysis - Cause
Analysis, Correlation and Cross Category
@StatsMalaysia @MyCensus2020 www.dosm.gov.my www.mycensus.gov.my #MyCensus2020 #MakeSureYou’reCOUNTED 13APPLICATIONS OF PRICE INTELLIGENCE
Implications of the Movement Control Order (MCO), all
During the pandemic Covid-19, data set from
the price data collection at the outlet has been
airline has been used in the compilation of the
suspended. Data crawling has been done during the
CPI.
MCO for 20 main product CPI as below:
Item specification (Route)
• CAP RAMBUTAN HIJAU SST RICE 5% 10KG (RICE)
• JATI BERAS SUPER SPESIAL 10KG (RICE)
Kedah to Kuala Lumpur • JASMINE RICE SUPER SPECIAL 5% 10KG (RICE)
Johor Bahru to Kuala Lumpur • JASMINE RICE SUPER SPECIAL TEMPATAN 5KG (RICE)
• AYAM BERSIH (PELBAGAI BAHAGIAN) (CHICKEN)
Kuantan to Kuala Lumpur • IKAN BAWAL HITAM (FISH)
Penang to Kuala Lumpur •
•
IKAN CENCARU (FISH)
IKAN KEMBUNG (FISH)
Kuala Terengganu to Kuala Lumpur • KUETIAU BASAH (FLAT RICE NOODLES)
• MEE KUNING BASAH (NOODLES)
• UBI KENTANG (POTATO)
• BAWANG BESAR (ONION)
• CILI KERING KERINTING (DRIED CHILLI)
• KACANG BUNCIS (FRENCH BEAN)
• KUBIS BULAT (TEMPATAN) (CABBAGE)
• LOBAK MERAH (CARROT)
• TOMATO
• SANTAN KELAPA (FRESH COCONUT MILK)
• TELUR AYAM GRED B (HEN'S EGGS GRADE B)
• *MINYAK MASAK (PELBAGAI JENAMA) (COOKING OIL)
@StatsMalaysia @MyCensus2020 www.dosm.gov.my www.mycensus.gov.my #MyCensus2020 #MakeSureYou’reCOUNTED 14APPLICATIONS OF PRICE INTELLIGENCE
HOUSE RENTAL PRICE FOR INTERNATIONAL COMPARISON PROGRAM (ICP)
45.6% housing rental data for International Comparison Program 2017 (ICP2017) submission
were using online price data
RENTAL HOUSING DATA SOURCED USED FOR ICP 2017
100%
90% 19 types of housing specification consist of
80% • Single-detached house
70% • Attached house(row house)
60%
• Studio apartment
50%
40%
• One-bedroom apartment
30% • Two-bedroom apartment
20% • Three-bedroom apartment
10%
0%
Online Price Rent Survey/ICP Price Collection
@StatsMalaysia @MyCensus2020 www.dosm.gov.my www.mycensus.gov.my #MyCensus2020 #MakeSureYou’reCOUNTED 15LIMITATION Need to update crawler Have to build a bunch of crawlers for different sites The structure of websites change frequently Legal issues involved Storage limitation (huge amounts of data) Access and scrape data which is publicly available and avoid trying to take data which is private or protected by copyrights and other laws; Always check the website’s robot.txt file @StatsMalaysia @MyCensus2020 www.dosm.gov.my www.mycensus.gov.my #MyCensus2020 #MakeSureYou’reCOUNTED 16
RESEARCH PAPER USING WEB SCRAPING
(PRICE INTELLIGENCE)
1 Forecasting prices of fish and vegetable using web scraped price micro data
- Mazliana Mustapa, Raja Rajeswari Ponnusamy, Ho Ming Kang
2 Online and Offline Prices: Measuring selected home appliance’s products
- Mohd Saiful Husain and Norsyela Muhammad Noor Mathivanan
3 Analysis of the Mobile Phones Prices Malaysia using Web Scraped Data
- Nur Hurriyatul Huda Abdullah Sani
@StatsMalaysia @MyCensus2020 www.dosm.gov.my www.mycensus.gov.my #MyCensus2020 #MakeSureYou’reCOUNTED 17THANK YOU
BANCI MALAYSIA
7.7.2020 – 21.12.2020 (e-Census) 20.1.2021 – 6.2.2021 (Face To Face)You can also read
