PERFORM END-TO-END DATA ANALYSIS IN THE CLOUD - BUILDING AN IOT ECOSYSTEM WITH ARDUINO AND BLUEMIX - MEETUP
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Dale Mumper
Open Source Analytics Solution Engineer - Industrial
dale.mumper@ibm.com
Perform End-to-End Data Analysis in the Cloud
Building an IoT Ecosystem with Arduino and Bluemix
© 2016 IBM Corporation
Disclaimer
© Copyright IBM Corporation 2016. All rights reserved.
U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY. WHILE EFFORTS WERE
MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED
“AS IS” WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. IN ADDITION, THIS INFORMATION IS BASED ON IBM'S CURRENT
PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE. IBM SHALL NOT BE RESPONSIBLE
FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER
DOCUMENTATION. NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, NOR SHALL HAVE THE EFFECT OF, CREATING ANY
WARRANTIES OR REPRESENTATIONS FROM IBM (OR ITS SUPPLIERS OR LICENSORS), OR ALTERING THE TERMS AND CONDITIONS OF
ANY AGREEMENT OR LICENSE GOVERNING THE USE OF IBM PRODUCTS AND/OR SOFTWARE.
IBM's statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM's sole discretion. Information
regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision.
The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or
functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any
future features or functionality described for our products remains at our sole discretion.
IBM, the IBM logo, ibm.com, Information Management, DB2, DB2 Connect, DB2 OLAP Server, pureScale, System Z, Cognos, solidDB, Informix,
Optim, InfoSphere, and z/OS are trademarks or registered trademarks of International Business Machines Corporation in the United States, other
countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or
™), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks
may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at “Copyright and
trademark information” at www.ibm.com/legal/copytrade.shtml
Other company, product, or service names may be trademarks or service marks of others.
2 © 2016 IBM Corporation
Agenda
Bio
Solution Overview
Bluemix Overview
Sensor Board
NodeRED
Cloudant
dashDB
Data Science Experience
3
Watson Analytics © 2016 IBM Corporation
Bio
Dale Mumper
IBM Open Source Analytics Solution Engineer
Consultant and analytics leader for over 20 years
Background in physics and math
Certifications
- Cloudera Certified Administrator
for Apache Hadoop - CCAH
- Cloudera Certified Developer
for Apache Hadoop - CCDH
- Microsoft MCSE – Data Platform
- Microsoft MCSE – Business Intelligence
- Oracle Certified Professional - OCP
4 © 2016 IBM Corporation
IoT Defined
“The network of physical devices, vehicles, building and other items
embedded with electronics, software, sensors, actuators and network
connectivity that enables object to collect and exchange data.”
“The infrastructure of the information society.”
“Every object, device and every familiar part of the traditional home,
is being equipped with smart circuitry.”
“With a trillon sensors embedded in the environment—all connected
by computing systems, software and services—it will be possible to
hear the heartbeat of the Earth, impacting human interaction with the
globe as profoundly as the Internet has revolutionized
communications,”
5 © 2016 IBM Corporation
IoT Market Drivers
USD 157.05 Billion in 2016
USD 661.74 Billion by 2021
Compound Annual Growth Rate (CAGR) of 33%
Impacting all industries
Industry leaders admit they are lack “clear perspective” on the business opportunities
afforded in the IoT arena – the trend remains nascent
2020 could see 30 Billion devices on the global net
Supplier Attention – open source software and open source hardware,
development tool kits, major vendor support
Technological Advances – ARM Cortex (1/10 the power usage), miniaturized
sensors, declining component costs, faster bandwidth
Increasing Demand - demand for 1st gen. will increase as costs decline and
next generations become more advanced; very price sensitive
Emerging Standards – semiconductor, hardware, networking and software
companies have joined with a number of industry associations and
academics consortiums; common APIs
6 © 2016 IBM Corporation
Phone Sensor Demo
Step 1
• Take out your phone
• Go to the URL on the card
• Write down the Device ID
d:quickstart:phonesensor
Step 2
• ibm.biz/iotqstart
• Enter Device ID
Step 3
• Explore
• Move Phone
Tilt
Rotate
Slow vs. Fast
Environmental Recorder – ER1
Indoor Environmental Monitoring
• Measures and sends data
Temperature (from three different sensors)
Humidity
Air Pressure
Light Levels
LEDs provide operational feedback
• Connects to a local wifi network
Synchronizes time from an NTP source
Gets the real IP address and determines geolocation from IP address
Asks nearest weather station for local forecast
Connect to an MQTT broker and sends data
8 © 2016 IBM Corporation
Use Case for the ER1
Sleep Therapy
Room Monitoring
Remote Property
Easily add sensors and capabilities
• UV and IR Sensor
• Distance (Ultrasonics and Laser)
• Motion
• Shock
• Vibration
• Rotation
• Tension and Flex
• Soil and Moisture
• GPS Module
• LTE Cellular W-Fi
• Solar Power and Battery
9 © 2016 IBM Corporation
Bill of Materials
Arduino MKR1000 Parts
• Atmel ATSAMW25 SoC • LED
SAMD21 Cortex M0+ ARM MCU • 220ohm resistor
WINC1500 2.4GHz 801.11 b/g/n Wi-Fi • Full-sized breadboard
3.3V
• USB A/MicroB Cable
256MB Flash
• Jumper Wires, 3”, MM
32KB SRAM
• Jumper Wires, 6”, MM
Full-Speed USB w/Embedded Host
Sensors Vendors
• Adafruit DS3231 • adafruit.com
• Adafruit SHT31-D • arduino.cc
• Adafruit TSL2691 • element14.com
• Adafruit BMP183 • digikeys.com
• Adafruit Neopixels
10 © 2016 IBM CorporationIoT Analytics Ecosystem
IoT + Runtime + Cloudant + dashDB + Spark
MQTT IBM dashDB
IoT Platform Schema
Discovery
REST (HTTP/s) API Spark Connector
11 © 2016 IBM CorporationArduino MKR1000
Combines the Arduino Zero and a Wi-Fi Shield at a Great Price Point
Atmel SAMD21 Cortex-M0+
• 3.3V
• 256KB Flash
• 32KB SRAM
• Clock Speed 48MHz
8 Digital I/O Pins
• 4 with PWM (pulse width modulated)
6 Analog Input Pins
1 Analog Output Pin 1. MPC and Memory
USB connection 2. Wi-Fi
3. Small Form Factor
Reset button 4. Lower Cost
Wi-Fi
Encryption
Li-Po Battery Charger
12 © 2016 IBM CorporationSHT31-D Sensor
Sensor made by Sensiron
• 2.5 x 2.5 x 0.9 mm3
• temperature range of –40°C to 90°C
• ±2% relative humidity and ±0.3°C accuracy
PCB Board made by Adafruit
• 3V and 5V compliant
• I2C interface
Power Pins
• Vin
2.5 to 5VDC (Volts Direct Current)
• GND
Common Ground
I2C Login Pins
• SCL
I2C clock
• SDA
I2C data pin
13 © 2016 IBM CorporationTSL2561 Sensor
Sensor made by ams AG
• Light-to-digital converter
• 188ulux to 88,000lux
• Infrared and Full Spectrum diodes
PCB Board made by Adafruit
• 3V and 5V compliant
• I2C interface
Power Pins
• Vin
2.5 to 5VDC (Volts Direct Current)
• GND
Common Ground
I2C Login Pins
• SCL
I2C clock
• SDA
I2C data pin
14 © 2016 IBM CorporationAdafruit DS3231 Real-Time Clock (RTC)
Chip made by Maxim Integrated
• DS3231 Real-Time Clock (RTC)
• Temperature-compensated crystal oscillator and crystal
• Long-term accuracy
PCB Board made by Adafruit
• I2C interface
• Optional battery maintains time
Power Pins z
• Vin
• GND
I2C Login Pins
• SCL - I2C clock
• SDA - I2C data pin
15 © 2016 IBM CorporationBMP183 Sensor
Sensor made by Bosch
• 300 to 1100hPa (+9000m to -500m)
• Enhanced GPS, navigation, weather, vert. velocity
PCB Board made by Adafruit
• 3V and 5V compliant
• SPI interface
Power Pins
• Vin
2.5 to 5VDC (Volts Direct Current)
• GND
Common Ground
SPI Logic Pins
• SCK - Clock
• SDO - Serial Data OUT
• SDI - Serial Data IN
• CS - Chip Select
16 © 2016 IBM CorporationNeoPixels == TOTALLY COOL Ring Jewel Strips Stick Matrix 17 © 2016 IBM Corporation
Turning Sensors into an IoT Device (ER1)
Sensors, Clock and LEDs in Review
Wi-Fi Connectivity
NTP Client
Time and Data Handling
C/C++ Style Floating Point Operations
HTTP Client
MQTT Client
JSON Parsing
ER1 Sketch Version 3.50
• Expects to find the IBM_CLASS 2.4GHz, WPA wireless network
Already has the SSID and the password in the sketch
• Defaults to using the IBM Watson IoT Platform in Quickstart Mode
• Sketch automatically determines the Device ID from the MAC
See your laminated MKR1000 card in your student kit
18 © 2016 IBM CorporationCloud Service Models
• Virtual Servers
• Bare Metal Servers
IaaS • Network
• Storage
• Load Balancers
• Database
• Web Server
PaaS • Development Tools
• Runtime Containers
• eMail
• CRM
SaaS • Games
• Virtual Desktop
19 © 2016 IBM CorporationWho Does What?
On-Premise IaaS PaaS SaaS
Applications Applications Applications Applications
Data Data Data Data
Runtime Runtime Runtime Runtime
Middleware Middleware Middleware Middleware
OS OS OS OS
Virtualization Virtualization Virtualization Virtualization
Servers Compute Compute Compute
Storage Storage Storage Storage
Networking Networking Networking Networking
Managed by Client Managed by Provider
20 © 2016 IBM CorporationIBM Cloud 21 © 2016 IBM Corporation
IBM Bluemix
Bluemix is an open-standard, cloud-based platform for building, managing, and
running applications of all types (web, mobile, big data, new smart devices…)
Go Live in Seconds APIs and Services On-Premises Integration
Zero to running in one click. A catalog of IBM, third party, Build hybrid environments.
Development plans deploy in and open source API services Connect to on-premises
seconds. Enterprise plans allow the developer to stitch an assets plus other public and
deploy in 1-2 days. application together in minutes. private clouds.
DevOps Layered Security Flexible Pricing
Development, monitoring, IBM secures the platform and Sign up in minutes. Pay as
deployment, and logging tools infrastructure and provides you go and subscription
allow the developer to run the you with the tools to secure models offer choice and
entire application. your apps. flexibility.
22 © 2016 IBM CorporationDemo – Bluemix Overview 23 © 2016 IBM Corporation
We Are Here
MQTT
24 © 2016 IBM CorporationThis Is Our Destination
IoT + Runtime + Cloudant + dashDB + Spark
MQTT IBM dashDB
IoT Platform Schema
Discovery
REST (HTTP/s) API Spark Connector
25 © 2016 IBM CorporationIBM Watson IoT Starter Platform
1. Catalog > Boilerplates > Internet of Things Platform Starter
2. Fill in Name:
3. CREATE
Application is created and staged
• http://mybluemix.net
• Creates a Node.js SDK Container
• Creates a Cloudant NoSQL Database
26 © 2016 IBM CorporationNode-RED
A visual tool for wiring the Internet of Things
• Browser-based UI for creating flows of events
• Deploying action in a light-weight runtime
• Based upon node.js
• Event-driven, non-blocking model
• Flows stored as JSON, so super easy to share
• Large library available today
• Suitable for server, network, edge and mobile device placement
• Open source project on GitHub
• IBM is a major contributor
• Benefits
• Rapid Development
• Simple to use with JSON
• Simple REST API
• Simple MQTT messaging
• Contributor Nodes
• Simple to use other services
27 © 2016 IBM CorporationMQTT
Machine-to-Machine (M2M)/”Internet of Things” (IoT)
• Lightweight connectivity protocol for publish/subscribe messaging transport
• Small code footprint, limited bandwidth, low power usage
• Minimized packets and efficient distribution to multiple receivers
MQTT v3.1.1 now an OASIS Standard
• Invented by Dr. Andy Stanford-Clark (IBM) and Arlen Nipper (Eurotech)
• MQ Telemetry Transport (ISO/IEC PRF 20922)
MQTT Broker/Servers
• IBM Websphere MQ Telemetery, Message Sight, Integration Bus
• Mosquitto, Eclipse Paho, Europtech Everywhere Device Cloud, emqttd,
Xively, Moquette, Yunab.io, m2m.io, RabbitMQ, Apache ActiveMQ, HiveMQ
MQTT Client Methods
• Connect, Disconnect, Subscribe, Unsubscribe, Publish
28 © 2016 IBM Corporationmsg.payload
{
"topic": "iot-2/type/MKR1000/id/f8f005f5f8db/evt/status/fmt/json",
"payload": {
"d": { "IBM_IoT_Workshop": "Arduino_MKR1000", "recordType":
"sensorsRead", "DS3231_epoch": 1471003668, "DS3231_date": "08-13-
2016", "DS3231_time": "13:07:48", "DS3231_tempC": 28, "DS3231_tempF":
82.4, "SHT31_tempC": 27.72, "SHT31_tempF": 81.94, "SHT31_humidity":
45.32, "TSL2561_lux": 9, "BMP183_hPa": 1004.22, "BMP183_tempC":
28.08, "BMP183_tempF": 82.55, "BMP183_altStatic": 78.98,
"BMP183_altComputed": 68.09, "local_IP": "192.168.0.170", "mac_addr":
"f8f005f5f8db" }
},
"deviceId": "f8f005f5f8db",
"deviceType": "MKR1000",
"eventType": "status",
"format": "json",
"_msgid": "4a43bc63.b5bc44”
}
29 © 2016 IBM CorporationDemo – Node-RED 30 © 2016 IBM Corporation
ER1 Message Payloads deviceStart ipapiFetch localWeather sensorRead badJSON These are all placed into one NoSQL database 31 © 2016 IBM Corporation
deviceStart 32 © 2016 IBM Corporation
ipapiFetch 33 © 2016 IBM Corporation
localWeather 34 © 2016 IBM Corporation
sensorRead 35 © 2016 IBM Corporation
Cloudant – NoSQL Database as a Service
Cloudant delivers a fully-managed database in service to the Analytics, App, and API economy
Powerful DBaaS
Operational NoSQL JSON store
Master-less architecture for
maximum scalability & availability
A fully-managed NoSQL database layer that
can be developed & deployed in days Advanced APIs
REST (HTTPS) API
Spark Replication & synchronization
Integration Geo-load balancing
(Spark SQL)
Incremental MapReduce indexes
Military-grade Geospatial indexes
dashDB
Integration Lucene full-text search
(Analytics)
Offline access to mobile apps & data
36 © 2016 IBM CorporationDemo – Cloudant 37 © 2016 IBM Corporation
Edge to Warehouse
Cloudant sits on the Edge of Cloud
• Fast, minimal latency, scalable
• Transactional
• Not the place for long-term storage
• Not the place for analytics
Move IoT data to a warehouse
• Basic business intelligence
• Connect to other sources of data
• The start of analytics journey
dashDB on Bluemix
• Data Warehouse as a Service
38 © 2016 IBM CorporationIBM dashDB – Analytics Warehouse as a Service
In-database analytics capabilities for best performance atop a fully-managed warehouse
For apps that need: for
Analytics
BLU • Elastic scalability
Acceleration • Highdata
Fully-managed availability
warehouse on cloud
• ofData
Choice model
SoftLayer flexibility
or Amazon Web Services
Netezza • Data mobility
BLU Acceleration columnar technology +
In-Database • Text search
Analytics Netezza in-database analytics
BLU•in-memory processing, data skipping, actionable
Geospatial
compression, parallel vector processing, “Load & Go”
Cloudant NoSQL administration
Integration
Available as:
Netezza predictive analytic algorithms
• Fully managed DBaaS
Fully integrated RStudio & R language
• On-premises private cloud
Oracle compatibility
• Hybrid architecture
dashDB MPP Massively Parallel Processing (MPP)
On disk data encryption and
secure connectivity
39 © 2016 IBM CorporationDemo – dashDB 40 © 2016 IBM Corporation
Replicating Cloudant JSON Data into dashDB
Cloudant’s Schema Discovery Process (SDP) translates JSON documents into
a schema (or set of tables) that dashDB understands
SDP maintains continuous
synchronization from
Cloudant to dashDB
41 © 2016 IBM CorporationDemo – Replication and SQL 42 © 2016 IBM Corporation
Tailored Experiences For Users Collaborating Together
Data Engineer
OUTPUT INPUT Architects how data is
organized & ensures operability
Deliver Understand Data Connect
and deploy problem and
model domain
Data Scientist
Communicate Ingest Gets deep into the data to draw
results data hidden insights for the business
Data Science Experience
Create Explore and Business Analyst
and build understand Works with data to apply insights
model data to the business strategy
Watson Analytics
Transform
Evaluate App Developer
Transform: : clean
Plugs into data and models &
shape writes code to build apps
Bluemix
ANALYSIS
43 © 2016 IBM CorporationWhat is a “Notebook”?
Pen and Paper Notebooks
Pen and paper has long provided the rich Notebooks are the digital equivalent of the
experience that scientists need to document “pen and paper” lab notebook, enabling data
progress through notes and drawings: scientists to document reproducible analysis:
– Expressive Markdown and visualization
– Cumulative Iterative exploration
– Collaborative Easy to share
44 © 2016 IBM CorporationWeb-Based Notebooks…
Notebooks:
“interactive computational environment, in which you can combine
code execution, rich text, mathematics, plots and rich media”
Jupyter
• Based on Ipython
• Supports multiple interpreters
• Python, Scala, R
Zeppelin
• Apache incubator project
• Supports multiple interpreters
• Python, Scala, others
Data Scientist
&
Notebooks
45 © 2016 IBM CorporationIntroducing the Data Science Experience - DSX
Currently in Public Beta
Learn Create Collaborate
Built-in learning to The best of open source Community and
get started or go and IBM value-add to social features that
the distance with create state-of-the-art provide meaningful
advanced tutorials data products collaboration
Powered by
http://datascience.ibm.com
46 © 2016 IBM CorporationCore Attributes of the Data Science Experience
IBM Data Science Experience
Community Open Source IBM Added Value
• Find tutorials and datasets • Code in Scala/Python/R/SQL • Data Shaping/Pipeline UI *
• Connect with Data Scientists • Jupyter and Zeppelin* Notebooks • Auto-data preparation
and modeling*
• Ask questions • RStudio IDE and Shiny apps
• Advanced Visualizations*
• Read articles and papers • Apache Spark
• Model management
• Fork and share projects • Your favorite libraries and deployment*
• Documented Model APIs*
• Spark as a Service
Powered by IBM DataWorks in the Cloud
* DSX product roadmap items
47 © 2016 IBM CorporationDemo – Data Science Experience 48 © 2016 IBM Corporation
IBM Watson Analytics - Smart Data Discovery in the Cloud
All the benefits of advanced analytics without the complexity
Data prep made easy Understand outcomes
Guided exploration Share insights
Designed to support the business professional’s analytics process so it’s easy to engage
49 with and find meanings and patterns in your data in minutes. © 2016 IBM CorporationDemo – Watson Analytics 50 © 2016 IBM Corporation
IBM investment into Apache Spark "It's like Spark
just got blessed
Contribute to by the enterprise
rabbi."
the Core
Ben Horowitz
Launch Spark Andreessen Horowitz
Technology Cluster
(STC), 300 engineers
Foster Open source Infuse the
SystemML Portfolio
Community
Partner with databricks Integrate Spark
Educate 1M+ data throughout portfolio
scientists and engineers
via online courses 3,500 employees working
on Spark-related topics
Sponsor AMPLab,
creators and Spark however
evangelists of Spark customers want it –
standalone, platform or
products
51 © 2016 IBM Corporation
Source: https://www-03.ibm.com/press/us/en/pressrelease/47107.wssIBM Contributes to core Apache Spark Project
www.spark.tc
IBM has the largest investment in Spark of any company in the world
IBM Spark Technology Center
• Launched in June of 2015
• Goal to hire 300 Engineers.
• Goal to Contribute to Apache
Spark Apache community
• Contributed SystemML
technology to Apache community
• STC continues to grow...
IBM Spark
52 © 2016 IBM CorporationIBM driving SQL and Machine Learning innovation.. http://www.spark.tc/blog/ 53 © 2016 IBM Corporation
Foster Community - Free Education
Big Data University
http://bigdatauniversity.com/
54 © 2016 IBM CorporationSignup to learn more!
http://ibm.biz/datagurus
Hands-on
Webinars Meetups
Labs
Learning
Twitter:
Resources
@data_gurus
Email
55 © 2015 IBM CorporationRaffle!
Fill out the paper form
and drop it in the box.
Two books being given away!
56 © 2015 IBM Corporation Dale Mumper Open Source Analytics Solution Engineer - Industrial dale.mumper@ibm.com 57 © 2015 IBM Corporation
You can also read
