PROVISIONING SOFTWARE-DEFINED IOT CLOUD SYSTEMS
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Provisioning Software-defined IoT Cloud Systems
Stefan Nastic, Sanjin Sehic, Duc-Hung Le, Hong-Linh Truong, and Schahram Dustdar
Distributed Systems Group, Vienna University of Technology, Austria
Email: {lastname}@dsg.tuwien.ac.at
Abstract—Cloud computing is ever stronger converging with a unified manner, with a central point of management. Further,
the Internet of Things (IoT) offering novel techniques for IoT the IoT systems are envisioned to run continuously, but they
infrastructure virtualization and its management on the cloud. can be elastically scaled in/down in off-peek times, e.g., when a
However, system designers and operations managers face numer- demand for certain data sources reduces. Due to the multiplic-
ous challenges to realize IoT cloud systems in practice, mainly ity of the involved stakeholders with diverse requirements and
due to the complexity involved with provisioning large-scale IoT
cloud systems and diversity of their requirements in terms of
business models, the modern IoT cloud systems increasingly
IoT resources consumption, customization of IoT capabilities need to support different and customizable usage experiences.
and runtime governance. In this paper, we introduce the concept Therefore, to utilize the benefits of cloud computing, IoT
of software-defined IoT units – a novel approach to IoT cloud cloud systems need to support virtualization of IoT resources
computing that encapsulates fine-grained IoT resources and IoT and IoT capabilities (e.g., gateways, sensors, data streams and
capabilities in well-defined APIs in order to provide a unified communication protocols), but also enable: i) encapsulating
view on accessing, configuring and operating IoT cloud systems. them in a well-defined API, at different levels of abstraction,
Our software-defined IoT units are the fundamental building ii) centrally managing configuration models and automatically
blocks of software-defined IoT cloud systems. We present our propagating them to the edge of infrastructure, iii) automated
framework for dynamic, on-demand provisioning and deploying provisioning of IoT resources and IoT capabilities.
such software-defined IoT cloud systems. By automating provi-
sioning processes and supporting managed configuration models, In this paper, we introduce the concept of software-defined
our framework simplifies provisioning and enables flexible run- IoT units – a novel approach to IoT cloud computing that
time customizations of software-defined IoT cloud systems. We encapsulates fine-grained IoT resources and IoT capabilities
demonstrate its advantages on a real-world IoT cloud system for in a well-defined API in order to provide a unified view on
managing electric fleet vehicles. accessing, configuring and operating IoT cloud systems. Our
software-defined IoT units are the fundamental building blocks
I. I NTRODUCTION of software-defined IoT cloud systems. They enable consuming
IoT resources at a fine granularity, allow for policy-based
Cloud computing technologies have been intensively ex- configuration of IoT capabilities and runtime operation of
ploited in development and management of the large-scale IoT software-defined IoT cloud systems. We present our framework
systems, e.g., in [11], [16], [18], because theoretically, cloud for dynamic, on-demand provisioning of the software-defined
offers unlimited storage, compute and network capabilities to IoT cloud systems. By automating main aspect of provision-
integrate diverse types of IoT devices and provide an elastic ing processes and supporting centrally managed configuration
runtime infrastructure for IoT systems. Self-service, utility- models, our framework simplifies provisioning of such systems
oriented model of cloud computing can potentially offer fine- and enables flexible runtime customizations.
grained IoT resources in a pay-as-you-go manner, reducing
upfront costs and possibly creating cross-domain application The rest of the paper is structured as follows: Section II
opportunities and enabling new business and usage models of presents a motivating scenario and research challenges; Sec-
the IoT cloud systems. tion III describes main principles and our conceptual model
of software-defined IoT systems; Section IV outlines main
However, most of the contemporary approaches dealing provisioning techniques for software-defined IoT systems; Sec-
with IoT cloud systems largely focus on data and device inte- tion V introduces design and implementation of our prototype,
gration by utilizing cloud computing techniques to virtualize followed by its experimental evaluation; Section VI discusses
physical sensors and actuators. Although, there are approaches the related work; Finally, Section VII concludes the paper and
providing support for provisioning and management of the gives an outlook of the future research.
virtual IoT infrastructure (e.g, [8], [16], [18]), the convergence
of IoT and cloud computing is still at an early stage. System
designers and operations managers face numerous challenges II. M OTIVATION
to realize large-scale IoT cloud systems in practice, mainly A. Scenario
because these systems impose diverse requirements in terms Consider a scenario about fleet management (FM) sys-
of granularity and flexibility of IoT resources consumption, tem for small-wheel electric vehicles deployed worldwide,
custom provisioning of IoT capabilities such as communication on different golf courses. The FM is an IoT cloud system
protocols, elasticity concerns, and runtime governance. For comprising golf cars’ on-board gateways, network and the
example, modern large-scale IoT cloud systems heavily rely on cloud infrastructure. The main features provided by the on-
the cloud and virtualized IoT resources and capabilities (e.g., board device include: a) vehicle maintenance (fault history,
to support complex, computationally expensive analytics), thus battery health, crash history, and engine diagnostics), b) vehicle
these resources need to be accessed, configured and operated in tracking (position, driving history, and geo-fencing), c) vehicle
1info (charging status, odometer, serial number, and service match their current needs, thus minimizing resources over-
notification), d) set-up (club-specific information, maps, and provisioning and allowing for better utilization of the avail-
fleet information). Vehicles communicate with the cloud via able resources [9]. However, IoT cloud systems are usually
3G, GPRS or Wi-Fi network to exchange telematic and diag- not tailored to incorporate elasticity aspects. For example,
nostic data. On the cloud we host different FM subsystems and new types of resources, e.g., data streams, delivered by IoT
services to manage the data. For example: a) Realtime vehicle infrastructure are still not provided elastically in IoT cloud
status: location, driving direction, speed, vehicle fault alarms; systems. Opportunistic exploitation of constrained resources,
b) Remote diagnostics: equipment status, battery health and inherent to many IoT cloud systems further intensifies the
timely maintenance reminders; c) Remote control: overriding need to provision the required resources on-demand or as
on-board vehicle control system in case of emergency; d) Fleet they become available. These challenges prevent current IoT
management: service history and fleet usage patterns. systems from fully utilizing the benefits cloud’s elastic nature
In the following we highlight some of the requirements and has to offer and call for new approaches to incorporate the
features of the FM system that we need to support: elasticity capabilities in the IoT cloud systems.
• The FM subsystems and services are hosted in the cloud RC3 – Dependability is a general measure of dynamic sys-
and heavily rely on the virtualized IoT resources, e.g., tem properties, such as availability, reliability, fault resilience
vehicle gateways and their capabilities. Therefore, we and maintainability. Cloud computing supports developing and
need to enable encapsulating and accessing IoT resources operating dependable large-scale systems atop commodity in-
and IoT capabilities, via an uniform API. frastructure, by offering an abundance of virtualized resources,
• The FM system has different requirements regarding providing replicated storage, enabling distributed computa-
communication protocols. The fault alarms and events tion with different availability zones and diverse, redundant
need to be pushed to the services (e.g, via MQ Telemetry network links among the system components. However, the
Transport (MQTT) [1]), when needed vehicle’s diag- challenges to build and operate dependable large-scale IoT
nostics should be synchronously accessed via RESTfull cloud systems are significantly aggravated because in such
protocols such as CoAP [10] or sMAP [7]. The remote systems the cloud, network and embedded devices are converg-
control system requires a dedicated, secure point-to-point ing, thus creating very large-scale hyper-distributed systems,
connection. Configuring these capabilities should be de- which impose new concerns that are inherently elusive with
coupled from the underlying physical infrastructure, in traditional operations approaches.
order to allow dynamic, fine-grained customization. RC4 – Due to dynamicity, heterogeneity, geographical
• The FM system spans multiple, geographically distributed distribution and the sheer scale of IoT cloud, traditional
cloud instances and IoT devices that comprise FM’s management and provisioning approaches are hardly feasible
virtual runtime topologies. These topologies abstract a in practice. This is mostly because they implicitly make as-
portion of the IoT cloud infrastructure, e.g., needed by sumptions such as physical on-site presence, manually logging
specific subsystem, thus they should support flexible into devices, understanding device’s specifics, etc., which are
configuring to allow for on-demand provisioning. difficult, if not impossible, to achieve in IoT cloud systems.
Thus, novel techniques, which will provide an unified and
The limited support for fine-grained provisioning at higher lev-
conceptually centralized view on system’s configuration man-
els leads to tightly coupled, problem specific IoT infrastructure
agement are needed.
components, which require difficult and tedious configuration
management on multiple levels. This inherently makes provi- Therefore, we need novel models and techniques to provi-
sioning and runtime operation of IoT cloud systems a com- sion and operate the IoT cloud systems, at runtime. Some of the
plex task. Consequentially, system designers and operations obvious requirements to make this feasible in the very large-
managers face numerous challenges to provision and operate scale, geographically distributed setup are: (i) We need tools
large-scale IoT cloud systems such as our FM. which will automate development, provisioning and operations
(DevOps) processes; (ii) Supporting mechanisms need to be
B. Challenges late-bound and dynamically configurable, e.g., via policies;
RC1 – The IoT cloud services and subsystems provide (iii) Configuration models need to be centrally managed and
different functionality or analytics, but they mostly rely on automatically propagated to the edge of the infrastructure.
common physical IoT infrastructure. However, to date the IoT
infrastructure resources have been mostly provided as coarse III. P RINCIPLES AND BUILDING BLOCKS OF
SOFTWARE - DEFINED I OT SYSTEMS
grained, rigid packages, in the sense that the IoT systems,
e.g., the infrastructure components and software libraries are A. Principles of Software-Defined IoT
specifically tailored for the problem at hand and do not allow Generally, software-defined denotes a principle of abstract-
for flexible customization and provisioning of the individual ing the low-level components, e.g., hardware, and enabling
resource components or the runtime topologies. This inherently their provisioning and management through a well-defined API
hinders self-service, utility-oriented delivery and consumption [14]. This enables refactoring the underlying infrastructure into
of the IoT resources at a finer granularity level. finer-grained resource components whose functionality can be
RC2 – Elasticity, although one of the fundamental traits of defined in software after they have been deployed.
the traditional cloud computing, has not yet received enough Software-defined IoT systems comprise a set of resource
attention in IoT cloud systems. Elasticity is a principle to components, hosted in the cloud, which can be provisioned
provision the required resources dynamically and on demand, and controlled at runtime. The IoT resources (e.g., sensory
enabling applications to respond to varying load patterns data streams), their runtime environments (e.g., gateways) and
by adjusting the amount of provisioned resources to exactly capabilities (e.g., communication protocols, analytics and data
2TABLE I. S UMMARY OF MAIN PRINCIPLES AND ENABLERS OF SOFTWARE - DEFINED I OT SYSTEMS
Research challenges High-level principles Enablers
• Flexible customization • API encapsulation of IoT resources and capabilities • Software-defined IoT units
• Utility-oriented delivery and consumption • Fine-grained resources consumption • Software-defined IoT topology (complex units)
• Self-service usage model • Policy-based specification and configuration • Centrally managed configuration models and policies
• Support for elasticity concerns • Automated provisioning • Automated units composition
• Operating dependable large-scale IoT cloud systems • Cost awareness • Runtime unit control and modification
• Central point of management • Runtime elasticity governance
point controllers) are described as software-defined IoT units. allow for flexible system customization, we need to enable
Software-defined IoT units are software-defined entities that are fine-grained resource consumption, well-defined API encapsu-
hosted in an IoT cloud platform and abstract accessing and op- lation and provide support for policy-based specification and
erating underlying IoT resources and lower level functionality. configuration. These principles are enabled by our software-
Generally, software-defined IoT units are used to encapsulate defined IoT units and support for centrally managed configura-
the IoT resources and lower level functionality in the IoT cloud tion. Figure 1 gives high-level graphical overview of the main
and abstract their provisioning and governance, at runtime. To building blocks and enabling techniques, needed to support the
this end, our software-defined IoT units expose well-defined main principles of software-defined IoT systems. Subsequently,
API and they can be composed at different levels, creating we describe them in more detail.
virtual runtime topologies on which we can deploy and execute
IoT cloud systems such as our FM system. Therefore, main B. Conceptual Model of Software-defined IoT Units
principles of software-defined IoT systems include:
Figure 2 illustrates the conceptual model of our software-
• API Encapsulation – IoT resources and IoT capabilities defined IoT units. The units encapsulate functional aspects
are encapsulated in well-defined APIs, to provide a unified (e.g., communication capabilities or sensor poll frequencies)
view on accessing functionality and configurations of IoT and non-functional aspects (e.g., quality attributes, elasticity
cloud systems. capabilities, costs and ownership information) of the IoT
• Fine-grained consumption – The IoT resources and capa- resources and expose them in the IoT cloud. The functional,
bilities need to be accessible at different granularity levels provisioning and governance capabilities of the units are
to support agile utilization and self-service consumption. exposed via well-defined APIs, which enable provisioning and
• Policy-based specification and configuration Principles
– The units controlling the units at runtime, e.g., start/stop. Our conceptual
are
Principles specified declaratively and their functionality
Fine-grained is
A) Software-defined model also allows for composing and interconnecting software-
A) defined
Software-defined programmatically in software,
resources using the well-
IoT unit defined IoT units, in order to dynamically
IoT Cloud Platformdeliver the IoT
abstraction and
IoTdefined
unit API and available, familiarAPIsoftware encapsulationlibraries. resources and capabilities to the applications. The runtime
B) •UnitAutomated
configuration provisioning – Main provisioning Policy-based processes
B) Unit’s configurationprovisioning and configuration is performed by specifying
provisioning and
C) need to
Software-defined be automated in order to enable
configuration
dynamic, on-
C) Software-defined late-bound policies and configuration models. Naturally, the
runtime topology
demand
runtime configuring and operatingFlexible
topology software-defined
runtime
IoT software-defined IoT units support mechanisms to map the
D) Runtime unit modification
D) systems,
Runtime on a large-scale (e.g, hundreds
unit modification control gateways). (e.g, changing virtual resources with the underlying physical infrastructure.
•(e.g,Cost
changing comm. protocol)
awareness – We need to be able (e.g.,
toofassign
elastic and communication
control protocol)To technically realize our unit model we introduce a
E) Runtime topology capabilities) E) Runtime topology
costs of(e.g,
modification delivered IoT resources and capabilities in
add new link)
order
modification concept of unit prototypes. They can be seen as resource con-
to enable their utility-oriented consumption. (e.g, add new link)
tainers, which are used to bootstrap more complex, higher-level
• Elasticity support – They should support elasticity gov- units. Generally, they are hosted in the cloud and enriched with
ernance [9], by exposing runtime control of elastic capa- functional, provisioning and governance capabilities, which
bilities through well-defined API. are exposed via software-defined APIs. The unit prototypes
can be based on OS-level virtualization, e.g., VMs, or more
A) E) finer-grained kernel supported virtualization, e.g., Linux con-
IoT Cloud platform Comm. protocol tainers. Conceptually, virtualization choices do not pose any
Software-
B) Configuration Models defined Monitoring limitations, because by utilizing the well-defined API, our
Manager gateway
Software library unit prototypes can be dynamically configured, provisioned,
D) interconnected, deployed, and controlled at runtime.
C) Gateway Gateway Given our conceptual model (Figure 2), by utilizing the
A) Software-defined IoT unit
provisioning API, the unit prototypes can be dynamically
Communication B) Managed configuration
Broker Sensor coupled with late-bound runtime mechanisms. These can be
C) Software-defined IoT topology
Actuator (higher-level units)
any software components (custom or stock), libraries or clients
D) Runtime unit control that can be configured and whose binding with the unit proto-
and modification types is differed to the runtime. For example, the mechanisms
(e.g, changing communication
IoT Cloud Infrastructure protocol)
can be used to dynamically add communication capabilities,
Datacenters IoT Infrastructure
E) Automated unit composition new functionality or storage to our software-defined IoT
(e.g., adding a capability) units. Therefore, by specifying policies, which are bound
later during runtime, system designers or operations managers
Fig. 1. Main enablers of software-defined IoT cloud systems can flexibly manage unit configurations and customize their
capabilities, at fine granularity levels. Our conceptual model
Table I summarizes how we translate the aforementioned also allows for composing the software-defined IoT units at
high-level principles into concrete enablers. For example, to higher levels. By selecting dependency units, e.g., based on
3Non-functional aspe
Governance A
Functional aspect
Utility
Functional A
cost-function
Software-defined
IoT Unit
Runtime Runtime controllers
mechanisms (e.g., elasticity)
Provisioning API logical structure. Therefore, conceptually
IoT resource we classify them
and functionality binding
Runtime composition into: (i) atomic,
Enables (ii) composed and capabilities
Infrastructure (iii) complex software-
Dependency Late-bound
defined IoT units. Depending on their type, the units require
configuring
units policies specific runtime mechanisms and expose Non-func specific aspects:
provisioning
flexible How many cores, memory
Non-functional aspects
Governance API
API. Next we describe each unit type in more detail.
Functional API
Utility
Functional aspects cost-function pricing and
The atomic software-defined IoT units are the finest-grained
cost models
Software-defined software-defined IoT units, which are used to abstract the core
IoT Unit capabilities of an IoT resource. They provide software-defined
Software-defined IoT API
Runtime Runtime API and need to be packaged portably to include components
mechanisms controllers Attributes
and libraries, that are needed to provide desired capabilities.
FigureConfiguration
3 depicts some Governance
examples of the atomic Provisioning software-defined
Functionality
IoT resource and functionality binding units. We broadly classify them into functional and non-
Infrastructure capabilities functional atomic software-defined IoT units, based on the
Internal External
capabilities they provide. Functional units encapsulate capa-
Unit.setMem()
Fig. 2. Conceptual model of software-defined IoT units.
bilities such as communication
Unit.addLinkUnit() Unit.stop() or IoT compute andUnit.setPollFreq() storage.
Non-functional units encapsulate configuration
Unit.changeLink() Unit.addPolicy()
models and
Unit.start()
capabilities such as elasticity
Unit.addComProtocol() replicateUnit()controllers or data-quality
Unit.addDataPoint() en-
Unit.setCPU()
their costs, analytics or elasticity capabilities, and linking them forcement mechanisms. Therefore, the atomic units are used to
together, we can dynamically build more complex units. This identify fine-grained capabilities needed by an application. For
enables flexible policy-based specification and configuration example, the application might require the communication to
Software-defined
of complex relationships IoT API
between the units. Therefore, by be performed via a specific transport protocol, e.g., MQTT or
carefully choosing the granularity of our units and providing it might need a specific monitoring component, e.g., Ganglia1 .
configuration policies we can automate the units composition Classifications similar to the one presented in Figure 3 can be
process at different levels and in some cases completely defer used to guide the atomic units selection process, in order to
Configuration Governance Provisioning Functionality
it to the runtime. This makes the provisioning process flexible, easily identify the exact capabilities, needed by the application.
traceable and repeatable across different cloud instances and
Internal thus
IoT infrastructures, External
reducing time, errors and costs.
Atomic software-defined IoT units
The runtime governance API, exposed by the units,
Unit.setMem() enables
Unit.setPollFreq()
Unit.addLinkUnit()
us to perform runtime control operations such as starting
Unit.stop() Functional
Unit.addPolicy() Non-functional
or stopping
Unit.start()the unit or Unit.changeLink()
change the topological structure of capabilities capabilities
the dependency
Unit.addComProtocol() units, e.g.,
replicateUnit() dynamically adding
Unit.addDataPoint() or removing
Unit.setCPU() ...
dependencies at runtime. Therefore, one of the most important ... IoT compute IoT data Communication
consequences of having software-defined IoT unit is that the storage Config.
Elasticity Data
functionality of the virtual IoT infrastructure can be (re)defined GW Data point quality Security
and customized after it has been deployed. New features can runtime controller History Network
be added to the units and the topological structure of the Custom Volatile overlay Protocol
proc. logic Monitor.
dependency units can be customized at runtime. This enables Auto scaling
automating provisioning and governance processes, e.g., by Component In-memory Key/Value VPN group controller Sand
Outliers
-model CEP image Messaging box
utilizing the governance API and providing monitoring at unit store filter
level, we can enable elastic horizontal scaling of our units.
Therefore, most important features of software-defined IoT Fig. 3. Example classification of atomic software-defined IoT units.
units which enable the general principles of software-defined
IoT (see Section Atomic software-defined
III-A) are: IoT units The composed software-defined IoT units have multiple
functional and non-functional capabilities, i.e., they are com-
• They Functional
provide software-defined API, which can be used posed of multiple atomic units. Similarly to the atomic units
to access, configure and control Non-functional
the units, in a unified
capabilities capabilities they provide well-defined API, but require additional function-
manner. ality such as mechanisms to support declaratively composing
• They support fine-grained internal ... configurations, e.g, and binding the atomic units, at runtime (Section IV-B).
... IoT compute IoT data capabilities
adding functional Communicationlike different communica- Example of composed unit is a software-defined IoT gateway.
tion protocols,storage
at runtime. Config.
Elasticity Data The complex software-defined IoT units enable capturing
GW
• They can be composed at higher-level, via
Data point
depen-
quality Security complex relationships among the finer-grained units. Internally,
runtime dency units, creating
controller virtual topologies that can be
History Network they are represented as a topological network, which can
Custom
(re)configured
Volatile at runtime.
overlay Protocol be configured and deployed, e.g., on the cloud. They define
• They
proc. logic enable decoupled and managed configuration (via
Monitor. an API and can integrate (standalone) runtime controllers
late-bound policies) to provisionAuto thescaling
units dynamically
group controller Sand to dynamically (re)configure the internal topology, e.g., to
Component and on-demand.
In-memory Key/Value VPN Outliers
CEP box enable elastic horizontal scaling of the units. Finally, they
-model • Theyimagehave utilitystore Messaging
cost-functions that enablefilter
pricing the
rely on runtime mechanism to manage the references, e.g.,
IoT resources as utilities.
IP addresses and ports, among the dependency units.
We notice that the software-defined API and our units
C. Units Classification offer different advantages to the stakeholders involved into
Depending on their purpose and capabilities, our software-
defined IoT units have different granularity and internal topo- 1 http://ganglia.info/
4designing, provisioning and governing of software-defined IoT stock components (e.g., Sedona5 or NiagaraAX6 execution
systems. For example, IoT infrastructure providers can offer environments). Classifications similar to the one presented in
their resources at fine-granularity, on-demand. This enables Figure 3 can be used to guide the atomic units selection
specifying flexible pricing and cost models and allows for process. In case we want to perform custom builds of the
offering the IoT resources as elastic utilities in a pay-as- existing libraries and frameworks, there are many established
you-go manner. Because our units support dynamic and au- build tools which can be used, e.g., for Java-based components,
tomated composition on multiple levels, consumers of IoT Apache Ant or Maven.
cloud resources can provision the units to exactly match their On the second level, we configure the composed units, e.g.,
functional and non-functional requirements, while still taking a software-defined IoT gateway. This is performed by adding
advantage of the existing systems and libraries. Further, system the atomic units (e.g., runtime mechanisms and/or software
designers and operations managers, use late-bound policies libraries) to the composed unit. For example, we might want
to specify and configure the unit’s capabilities. Because we to enable the gateway to communicate over a specific transport
treat the functional and configuration units in a similar man- protocol, e.g., MQTT and add a monitoring component to it,
ner (see Section IV-B), configuration models can be stored, e.g., a Ganglia agent. To perform this composition seamlessly
reused, modified at runtime and even shared among differ- at runtime, additional mechanisms are required. We describe
ent stakeholders. This means that we can support managed them in Section IV-B.
configuration models, which can be centrally maintained via Third level includes defining the dependencies references
configuration management solutions for IoT cloud, e.g., based between the composed units, which ”glue together” the com-
on OpsCode Chef2 , Bosh3 or Puppet4 . plex units. These links specify the topological structure of
IV. P ROVISIONING SOFTWARE - DEFINED I OT CLOUD the desired complex units. For example, to this end we
SYSTEMS
can set up a virtual private network and provide each unit
with a list of IP addresses of the dependency units. In this
A. Automated composition of software-defined IoT units phase, we can use frameworks (e.g., TOSCA-based, OpenStack
Generally, building and deploying software-defined IoT Heat, Amazon CloudFormation, etc.) to specify the runtime
cloud systems includes creating and/or selecting suitable topological structure of our units and utilize mechanisms (e.g.,
software-defined IoT units, configuring and composing more Ubuntu CloudInit7 ) to bootstrap the composition, e.g, pass the
complex units and building custom business logic components. references to the dependency units.
The deployment phase includes deploying the software-defined
IoT units together with their dependency units and required B. Centrally managed configuration models and policies
(possibly standalone) runtime mechanisms (e.g., a message An important concept behind software-defined IoT cloud
broker). In this paper we mostly focus on provisioning reusable systems is the late-bound runtime policies. Our units are con-
stock components such as gateway runtime environments or figured declaratively, via the policies by utilizing the exposed
available communication protocols. Developing custom busi- software-defined API, without worrying about internals of the
ness logic components is out of scope of this paper and we runtime mechanisms, i.e, the atomic units. To enable seamless
address it elsewhere [15]. binding of the atomic units we provide a special unit prototype,
called bootstrap container. The bootstrap container acts as a
ACTIONS ACTIONS ACTIONS Deploy plug-in system, which provides mechanisms to define (bind)
* Pullexternal * Conf.policies
repo * Select * Select
the units based on supplied configurations or to redefine them
* Build unit prototype Compo- unit prototype when configuration policies are changed. For example, runtime
Atomic Complex
* Select
unit
* Resolve sed * Link unit
unit changes of the units are achieved by invalidating affected parts
unit prototype dependencies unit dependencies
* Configure * Exceptions * Exceptions
of the existing dependency tree and dynamically rebuilding
* Exceptions Select and errors Select and errors them, based on the new configuration directives. Therefore, the
and errors third- third- units can be simply ”droped in” and our bootstrap container
Standalone
party unit party unit
(re)binds them together,
runtime at runtime without rebooting system.
mechanism
Fig. 4. Automated composition of software-defined IoT units. We decouple the configuration models (late-bound policies)
(optional)
from the functional units. Therefore, we can treat configuration
Figure 4 illustrates most important steps to compose and policies as any software-defined IoT unit, which adheres to
deploy our IoT units. There are three levels of configuration the general principles of software-defined IoT (Section III-A).
that can
add
be performed:AtomicUnit
(i) Building/selecting atomic units; By encapsulating the configuration policies in separate units,
UnitPrototype
(ii) Configuring composed units; (iii) Linking into complex we can manage them at runtime via centralized configuration
units. Each of thelinkphases includes selecting and provisioning managements solutions for IoT cloud. Our framework provides
ComposedUnit
suitable unit prototypes.
* ComplexUnit
For example, the unit prototypes can mechanisms to specify and propagate the configuration models
be based on different resource containers such as VMs, Linux to the edge of IoT cloud infrastructure (e.g., gateways) and
Containers (e.g., Docker) or OSGi runtime. our bootstrap container enforces the provided directives. To
this end, our bootstrap container initially binds functional and
The atomic units are usually provided as stock components, configuration units and continuously listens for configuration
e.g., by a third-party, possibly in a market-like fashion. There- changes and applies them on the affected functional units, ac-
fore, this phase usually involves selecting and configuring cordingly. To enable performing runtime modifications without
2 http://opscode.com/chef 5 http://www.sedonadev.org/
3 http://docs.cloudfoundry.org/bosh/ 6 http://www.niagaraax.com/
4 http://puppetlabs.org 7 http://help.ubuntu.com/community/CloudInit/
5worrying about any side-effects we require the configuration implementation, the core of the InitalizationManager is an
actions to be idempotent. The usual approach to achieve this OpsCode Chef client, which is passed to the VMs during
is to wrap the units as OS services. Among other things the initialization via Ubuntu cloud-init. InitalizationManager also
late-bound policies and our mechanisms for managed configu- provides mechanisms for configuration management. The De-
ration enable flexible customization and dynamic configuration ploymentManager is used to deploy the software-defined IoT
changes, at runtime. units in the cloud. Our prototype relies on SALSA8 , a de-
ployment automation framework we developed. It utilizes the
V. P ROTOTYPE AND EXPERIMENTS API exposed by the CloudSystemWrapper to enable deploy-
ment across various cloud providers, currently implemented
A. Prototype implementation
for OpenStack cloud. DeploymentManager is responsible to
The main aim of our prototype is to enable developers and manage and distribute the dependency references for the
operations managers to dynamically, on-demand provision and complex units (Section III-C). Units persistence layer provides
deploy software-defined IoT systems. This includes providing functionality to store and manage our software-defined units
software-defined
Niagara gateway IoT unit prototypes, enabling automated unit and policies.
composition, at multiple levels and supporting centralized Framework > Individual u
runtime management of the configuration models. B. Experiments configuration
Sedona gateway
In Section III we introduced the conceptual model of our 1) Scenario analysis: We now show how our prototype is > Topology m
Deployment
Initialization
Coordinator
Manager
Runtime
software-defined IoT units. To technically realize our units,
Maager
used to provision a complex software-defined IoT unit, which
we utilize the concept of Repository
SensorA virtual resource containers. More provides functionality for the real-life FM location tracking
precisely, we provide different unit prototypes that can be service (Section II-A). The service reports vehicle location in
customized and/or modified during runtime by adding required near real-time on the cloud. To enable remote access, the mon-
Monitoring
runtime mechanisms encapsulated in our atomic units. The unit itored vehicles have an on-board device,Topology model
acting as a gateway
prototypes provide resources with different granularity, e.g., to its data and control points. To improve performance and
VMComm.
flavors,Protocol
group quotas, priorities, etc., and boilerplate func- reliability, the golf course provides on-site gateways, which
create/init/deploy
tionality to enable automated provisioning of custom software- topology communicate with the vehicles, provide additional processing
defined IoT units. and storage capabilities e.g,
andstart/stop
feed the data into the cloud.
Figure 5 provides a high-level overview of the framework’s Therefore, the physical software-defined
IoT infrastructure comprises network
architecture. Our framework is completely hosted in the cloud connected vehicles, on-board devices
unit and local gateways.
and follows a modular design which guarantees flexible and Typically, to provision the FM service system designers
evolvable architecture. The current prototype is implemented and operations
Runtime manager would need to directly interact with
topology
atop OpenStack [2], which is an open source Infrastructure-as- the rigid physical IoT infrastructure. Therefore, they at least
a-Service (IaaS) cloud computing platform. Presentation layer need to be aware of its topological structure and devices’
provides an user interface via Web-based UI and RESTful API. capabilities. This means that the FM service also needs to have
They allow a user to specify various configuration models understanding of the IoT infrastructure, instead of being able to
and policies, which are used by the framework to compose customize the infrastructure to its needs. Due to inherent inflex-
and deploy our units in the cloud. Cloud core services layer ibility of IoT infrastructure, its provisioning usually involves
contains the main functionality of the framework. It includes long and tedious task such as manually logging into individual
the PolicyProcessor used to read the input configurations and gateways, understanding gateway internals or even on site
transform it to the internal model defined in our framework. presence. Therefore, provisioning even a simple FM location
Units management services utilize this model for composing tracking service involves performing many complex tasks. Due
and managing the units. The InitializationManager is respon- to a large number of geographically distributed vehicles and
sible for configuring and composing more complex units. It involved stakeholders IoT infrastructure provisioning requires
translates the directives specified in configuration models into a substantial effort prolonging service delivery and increasing
concrete initialization actions on the unit level. In our current costs. Subsequently, we show the advantages our units (Sec-
tion III-B) and the provisioning techniques (Section IV) have
Presentation layer
to offer to operations managers and application designers in
Web UI RESTful API terms of: a) Simplified provisioning to reduce time, costs and
possible errors; b) Flexibility to customize and modify the IoT
Cloud core services layer units and their runtime topologies.
Units management services Policy
Processor To enable the FM system we developed a number of atomic
Deployment
IoT units9 such as: a software-defined sensor
Coordin.
Runtime
Manager
software-define
Configuration
Management
that reports vehicle location in realtime, messaging infrastruc-
CloudSystem
Initialization
Wrapper
ture based on Apache ActiveMQ10 , software-defined protocol
Manager
based on MQTT and JSON, the bootstrap container based
Repository on the Spring framework11 , and corresponding configuration
Services units. The experiments are simulated on our OpenStack (Fol-
som) cloud and we used Ubuntu 12.10 cloud image (Memory:
Units persistence layer
8 https://github.com/tuwiendsg/SALSA/
Policies repo. SD IoT units repo. 9 https://github.com/tuwiendsg/SDM
10 http://activemq.apache.org/
Fig. 5. Framework architecture overview. 11 http://projects.spring.io/spring-framework/
62GB, VCPUS: 1, Storage: 20GB). To display location changes the dependencies between the units, the user continues com-
we develop a Web application which displays changes of posing on the finer granularity level. By applying the top-down
vehicles’ location on Google Maps. approach we enable differing design decisions and enable early
2) Simplified provisioning: To demonstrate how our ap- automation of known functionality, to avoid over-engineering
proach simplifies provisioning of the virtual IoT infrastructure, and provisioning redundant resources.
we show how a user composes the FM complex software- In the next phase, the user provisions individual unit proto-
defined IoT unit, using our framework. Figure 6 shows the types. To this end, he/she provides policies specifying desired
custom deployment of the topological structure of the FM finer-grained capabilities. Listing 2 shows example capabili-
vehicle tracking unit, deployed in the cloud. The unit contains ties, that can be added to the gateway. To enable asynchronous
two gateways for the vehicles it tracks, a web server for the pushing of the location changes it should communicate over
Web application and a message broker that connects them. the MQTT protocol. Listing 3 shows a part of Chef recipe used
to add MQTT client to the gateway. Our framework fetches the
SD UNIT atomic units, that encapsulate the required capabilities, from
UNIT PROTOTYPE Id: mqtt_broker the repository and composes them automatically, relying on
state: RUNNING
Id: VM container
state: RUNNING
the software-defined API and our bootstrap container.
SD UNIT
Id: SD gateway_1278 {"run_list":
SALSA CENTER UNIT PROTOTYPE state: RUNNING ["recipe[bootstrap_container]",
Id: sd IoT system Id: VM container "recipe[mqtt-client]",
state: RUNNING state: RUNNING SD UNIT "recipe[protocol-config-unit]",
Id: SD gateway_1280 "recipe[sd-sensor]"]
state: RUNNING }
UNIT PROTOTYPE
Id: VM container
state: RUNNING SD UNIT
Listing 2. Run list for software-defined gateway.
Id: Web server
state: RUNNING
include_recipe ’bootstrap_container::default’
Fig. 6. Topological structure of FM vehicle tracking unit (a screen shot). remote_file "mqtt-client-0.0.1-SNAPSHOT.jar" do
source "http://128.130.172.215/salsa/upload/files/..."
group "root"
In order to start provisioning the complex unit, system mode 00644
action :create_if_missing
designer only needs to provide a policy describing the re- end
quired high-level resources and capabilities required by the
FM service. For example, Listing 1 shows a snippet from Listing 3. Chef recipe for adding the MQTT protocol.
the configuration policy for FM location tracking unit, that
illustrates specifying a software-defined gateway, for the on- Therefore, compared to the traditional approaches, which
board device. require gateway-specific knowledge, using proprietary API,
manually logging in the gateways to set data points, our
...
ative unit configuration policies, simplifies the provisioning
process and makes it traceable and repeatable. Our units can
easily be shared among the stakeholders and composed to
m1.small provide custom functionality. This enables system designers
openstack@dsg and operations managers to rely on the existing, established
ami-00000163
systems, thus reducing provisioning time, potential errors and
costs.
3) Flexible customization: To exemplify the flexibility of
figuration of the FM unit to use CoAP instead of MQTT.
This can be due to requirements change (Section II-A),
for a golf course with different networking capabilities. To
customize the existing unit, an operations manager only needs
... to change the "recipe[protocol-config-unit]"unit
Listing 1. Partial TOSCA-like complex unit description. (Listing 2) and provide an atomic unit for the CoAP client.
This is a nice consequence of our late-bound runtime mecha-
The policy describes gateway’s initial configuration and the nisms and support for managed configuration models, provided
cloud instance where it should be deployed. Additionally, it by our framework. We treat both functional and configuration
defines a dependency unit, i.e. the MQTT broker and specifies units in the same manner and our bootstrap container manages
vehicle’s Id that can be used to map it on the underlying device, their runtime binding (Section IV-B). Compared to traditional
as shown in [15]. Our framework takes the provided policy, approaches that require addressing each gateway individually,
spawns the required unit prototypes and provides them with firmware updates or even modifications on the hardware level,
references to the dependency units. At this stage the virtual our framework enables flexible runtime customization of our
infrastructure comprises solely of unit prototypes (VM-based). units and supports operation managers to seamlessly enforce
After performing the high-level unit composition and establish configuration baseline and its modifications on a large-scale.
7VI. R ELATED W ORK for provisioning software-defined IoT systems. The initial
Recently, many interesting approaches enabling conver- results are promising in the sense that software-defined IoT
gence of IoT and cloud computing appeared. For example, system enable sharing of the common IoT infrastructure among
in [5], [8], [11], [16], [18] the authors mostly deal with multiple stakeholders and offer advantages to IoT cloud system
IoT infrastructure virtualization and its management on cloud designers and operations managers in terms of simplified, on-
platforms, [6], [13] utilize the cloud for additional computation demand provisioning and flexible customization. Therefore,
resources, in [17], [19] the authors mostly focus on utilizing we believe that software-defined IoT systems can significantly
cloud’s storage resources for Big-IoT-Data and [12], [16] contribute the evolution of the IoT cloud systems.
integrating IoT devices with enterprise applications based on In the future we plan to continue developing the prototype
SOA paradigm. Due to limited space in the following we only and extend it in several directions: a) Providing techniques
mention related work exemplifying approaches that deal with and mechanisms to support runtime governance of software-
IoT infrastructure virtualization and management. defined IoT systems; b) Enabling our software-defined IoT
In [11] the authors develop an infrastructure virtualization systems to better utilize the edge of infrastructure, e.g., by
framework for wireless sensor networks. It is based on a providing code distribution techniques between cloud and IoT
content-based pub/sub model for asynchronous event exchange devices; c) Enabling policy-based automation of data-quality,
and utilizes a custom event matching algorithm to enable security and safety aspects of software-defined IoT systems.
delivery of sensory events to subscribed cloud users. In [18] the
authors introduce sensor-cloud infrastructure that virtualizes ACKNOWLEDGMENT
physical sensors on the cloud and provides management and This work is sponsored by Pacific Controls Cloud Com-
monitoring mechanisms for the virtual sensors. However, their puting Lab (PC3L12 )
support for sensor provisioning is based on static templates
that, contrary to our approach, do not support dynamic provi- R EFERENCES
sioning of IoT capabilities such as communication protocols. [1] Mq telemetry transport. http://mqtt.org/. March 2014.
SenaaS [5] mostly focuses on providing a cloud semantic [2] Openstack. http://www.openstack.org/. March 2014.
overlay atop physical infrastructure. They define an IoT on- [3] Thingworks. http://thingworx.com. Last accessed: March 2014.
tology to mediate interaction with heterogeneous devices and [4] Xively. https://xively.com. Last accessed: March 2014.
data formats, exposing them as event stream services to the [5] S. Alam, M. Chowdhury, and J. Noll. Senaas: An event-driven sensor
upper layers. OpenIoT framework [16] utilizes semantic web virtualization approach for internet of things cloud. In NESEA, 2010.
technologies and CoAP [10] to enable web of things and [6] B.-G. Chun, S. Ihm, P. Maniatis, M. Naik, and A. Patti. Clonecloud:
elastic execution between mobile device and cloud. In Proceedings of
linked sensory data. They mostly focus on discovering, linking the sixth conference on Computer systems. ACM, 2011.
and orchestrating internet connected objects, thus conceptually [7] S. Dawson-Haggerty, X. Jiang, G. Tolle, J. Ortiz, and D. Culler. smap:
complementing our approach. In [8] the authors focus on a simple measurement and actuation profile for physical information.
developing a virtualization infrastructure to enable sensing In SenSys, pages 197–210, 2010.
and actuating as a service on the cloud. They propose a [8] S. Distefano, G. Merlino, and A. Puliafito. Sensing and actuation as a
software stack which includes support for management of service: a new development for clouds. In NCA, pages 272–275, 2012.
device identification, selection and aggregation, all of which [9] S. Dustdar, Y. Guo, B. Satzger, and H.-L. Truong. Principles of elastic
can be seen as enablers of our approach. Also there are various processes. Internet Computing, IEEE, 15(5):66–71, 2011.
commercial solutions such as Xively [4] and ThingWorx [3], [10] B. Frank, Z. Shelby, K. Hartke, and C. Bormann. Constrained applica-
tion protocol (coap). IETF draft, Jul, 2011.
which allow users to connect their sensors to the cloud and
enable remote access and management. [11] M. M. Hassan, B. Song, and E.-N. Huh. A framework of sensor-cloud
integration opportunities and challenges. In ICUIMC, 2009.
Most of these approaches focus on different virtualization [12] M. Kovatsch, M. Lanter, and S. Duquennoy. Actinium: A restful runtime
techniques for IoT devices and data format mediation. They container for scriptable internet of things applications. In Internet of
also enable some form of configuration, e.g., setting sensor Things, pages 135–142, 2012.
poll rates. These approaches can be seen as complementary [13] K. Kumar and Y.-H. Lu. Cloud computing for mobile users: Can
to our own, as device virtualization sets the corner stone offloading computation save energy? Computer, 43(4):51–56, 2010.
for achieving software-defined IoT systems. We rely on the [14] B. Lantz, B. Heller, and N. McKeown. A network in a laptop: rapid
advances in convergence of IoT and cloud, introduced by the prototyping for software-defined networks. In Proceedings of the 9th
ACM SIGCOMM Workshop on Hot Topics in Networks. ACM, 2010.
previous work and extend it with novel concepts for abstracting
[15] S. Nastic, S. Sehic, M. Voegler, H.-L. Truong, and S. Dustdar. Patricia
and encapsulating IoT resources and capabilities, exposing - a novel programing model for iot applications on cloud platforms. In
them via software-defined API on the cloud and enabling fine- SOCA, 2013.
grained provisioning. Therefore, our approach can be seen as [16] J. Soldatos, M. Serrano, and M. Hauswirth. Convergence of utility
a natural step in evolution of IoT cloud systems. computing with the internet-of-things. In IMIS, pages 874–879, 2012.
[17] P. Stuedi, I. Mohomed, and D. Terry. Wherestore: Location-based data
VII. C ONCLUSION storage for mobile devices interacting with the cloud. In ACM Workshop
on Mobile Cloud Computing & Services. ACM, 2010.
In this paper, we introduced the conceptual model of [18] M. Yuriyama and T. Kushida. Sensor-cloud infrastructure-physical
software-defined IoT units. To our best knowledge this is sensor management with virtualized sensors on cloud computing. In
the first attempt to apply software-defined principles on IoT NBiS, 2010.
systems. We showed how they are used to abstract IoT [19] A. Zaslavsky, C. Perera, and D. Georgakopoulos. Sensing as a service
resources and capabilities in the cloud, by encapsulating and big data. arXiv preprint arXiv:1301.0159, 2013.
them in software-defined API. We presented automated unit
composition and managed configuration, the main techniques 12 http://pcccl.infosys.tuwien.ac.at/
8You can also read
