Provisioning Software-defined IoT Cloud Systems

                Stefan Nastic, Sanjin Sehic, Duc-Hung Le, Hong-Linh Truong, and Schahram Dustdar
                               Distributed Systems Group, Vienna University of Technology, Austria
                                               Email: {lastname}

    Abstract—Cloud computing is ever stronger converging with            a unified manner, with a central point of management. Further,
the Internet of Things (IoT) offering novel techniques for IoT           the IoT systems are envisioned to run continuously, but they
infrastructure virtualization and its management on the cloud.           can be elastically scaled in/down in off-peek times, e.g., when a
However, system designers and operations managers face numer-            demand for certain data sources reduces. Due to the multiplic-
ous challenges to realize IoT cloud systems in practice, mainly          ity of the involved stakeholders with diverse requirements and
due to the complexity involved with provisioning large-scale IoT
cloud systems and diversity of their requirements in terms of
                                                                         business models, the modern IoT cloud systems increasingly
IoT resources consumption, customization of IoT capabilities             need to support different and customizable usage experiences.
and runtime governance. In this paper, we introduce the concept          Therefore, to utilize the benefits of cloud computing, IoT
of software-defined IoT units – a novel approach to IoT cloud            cloud systems need to support virtualization of IoT resources
computing that encapsulates fine-grained IoT resources and IoT           and IoT capabilities (e.g., gateways, sensors, data streams and
capabilities in well-defined APIs in order to provide a unified          communication protocols), but also enable: i) encapsulating
view on accessing, configuring and operating IoT cloud systems.          them in a well-defined API, at different levels of abstraction,
Our software-defined IoT units are the fundamental building              ii) centrally managing configuration models and automatically
blocks of software-defined IoT cloud systems. We present our             propagating them to the edge of infrastructure, iii) automated
framework for dynamic, on-demand provisioning and deploying              provisioning of IoT resources and IoT capabilities.
such software-defined IoT cloud systems. By automating provi-
sioning processes and supporting managed configuration models,                In this paper, we introduce the concept of software-defined
our framework simplifies provisioning and enables flexible run-          IoT units – a novel approach to IoT cloud computing that
time customizations of software-defined IoT cloud systems. We            encapsulates fine-grained IoT resources and IoT capabilities
demonstrate its advantages on a real-world IoT cloud system for          in a well-defined API in order to provide a unified view on
managing electric fleet vehicles.                                        accessing, configuring and operating IoT cloud systems. Our
                                                                         software-defined IoT units are the fundamental building blocks
                      I. I NTRODUCTION                                   of software-defined IoT cloud systems. They enable consuming
                                                                         IoT resources at a fine granularity, allow for policy-based
    Cloud computing technologies have been intensively ex-               configuration of IoT capabilities and runtime operation of
ploited in development and management of the large-scale IoT             software-defined IoT cloud systems. We present our framework
systems, e.g., in [11], [16], [18], because theoretically, cloud         for dynamic, on-demand provisioning of the software-defined
offers unlimited storage, compute and network capabilities to            IoT cloud systems. By automating main aspect of provision-
integrate diverse types of IoT devices and provide an elastic            ing processes and supporting centrally managed configuration
runtime infrastructure for IoT systems. Self-service, utility-           models, our framework simplifies provisioning of such systems
oriented model of cloud computing can potentially offer fine-            and enables flexible runtime customizations.
grained IoT resources in a pay-as-you-go manner, reducing
upfront costs and possibly creating cross-domain application                  The rest of the paper is structured as follows: Section II
opportunities and enabling new business and usage models of              presents a motivating scenario and research challenges; Sec-
the IoT cloud systems.                                                   tion III describes main principles and our conceptual model
                                                                         of software-defined IoT systems; Section IV outlines main
    However, most of the contemporary approaches dealing                 provisioning techniques for software-defined IoT systems; Sec-
with IoT cloud systems largely focus on data and device inte-            tion V introduces design and implementation of our prototype,
gration by utilizing cloud computing techniques to virtualize            followed by its experimental evaluation; Section VI discusses
physical sensors and actuators. Although, there are approaches           the related work; Finally, Section VII concludes the paper and
providing support for provisioning and management of the                 gives an outlook of the future research.
virtual IoT infrastructure (e.g, [8], [16], [18]), the convergence
of IoT and cloud computing is still at an early stage. System
designers and operations managers face numerous challenges                                      II. M OTIVATION
to realize large-scale IoT cloud systems in practice, mainly             A. Scenario
because these systems impose diverse requirements in terms                   Consider a scenario about fleet management (FM) sys-
of granularity and flexibility of IoT resources consumption,             tem for small-wheel electric vehicles deployed worldwide,
custom provisioning of IoT capabilities such as communication            on different golf courses. The FM is an IoT cloud system
protocols, elasticity concerns, and runtime governance. For              comprising golf cars’ on-board gateways, network and the
example, modern large-scale IoT cloud systems heavily rely on            cloud infrastructure. The main features provided by the on-
the cloud and virtualized IoT resources and capabilities (e.g.,          board device include: a) vehicle maintenance (fault history,
to support complex, computationally expensive analytics), thus           battery health, crash history, and engine diagnostics), b) vehicle
these resources need to be accessed, configured and operated in          tracking (position, driving history, and geo-fencing), c) vehicle

info (charging status, odometer, serial number, and service             match their current needs, thus minimizing resources over-
notification), d) set-up (club-specific information, maps, and          provisioning and allowing for better utilization of the avail-
fleet information). Vehicles communicate with the cloud via             able resources [9]. However, IoT cloud systems are usually
3G, GPRS or Wi-Fi network to exchange telematic and diag-               not tailored to incorporate elasticity aspects. For example,
nostic data. On the cloud we host different FM subsystems and           new types of resources, e.g., data streams, delivered by IoT
services to manage the data. For example: a) Realtime vehicle           infrastructure are still not provided elastically in IoT cloud
status: location, driving direction, speed, vehicle fault alarms;       systems. Opportunistic exploitation of constrained resources,
b) Remote diagnostics: equipment status, battery health and             inherent to many IoT cloud systems further intensifies the
timely maintenance reminders; c) Remote control: overriding             need to provision the required resources on-demand or as
on-board vehicle control system in case of emergency; d) Fleet          they become available. These challenges prevent current IoT
management: service history and fleet usage patterns.                   systems from fully utilizing the benefits cloud’s elastic nature
    In the following we highlight some of the requirements and          has to offer and call for new approaches to incorporate the
features of the FM system that we need to support:                      elasticity capabilities in the IoT cloud systems.
   • The FM subsystems and services are hosted in the cloud                  RC3 – Dependability is a general measure of dynamic sys-
      and heavily rely on the virtualized IoT resources, e.g.,          tem properties, such as availability, reliability, fault resilience
      vehicle gateways and their capabilities. Therefore, we            and maintainability. Cloud computing supports developing and
      need to enable encapsulating and accessing IoT resources          operating dependable large-scale systems atop commodity in-
      and IoT capabilities, via an uniform API.                         frastructure, by offering an abundance of virtualized resources,
   • The FM system has different requirements regarding                 providing replicated storage, enabling distributed computa-
      communication protocols. The fault alarms and events              tion with different availability zones and diverse, redundant
      need to be pushed to the services (e.g, via MQ Telemetry          network links among the system components. However, the
      Transport (MQTT) [1]), when needed vehicle’s diag-                challenges to build and operate dependable large-scale IoT
      nostics should be synchronously accessed via RESTfull             cloud systems are significantly aggravated because in such
      protocols such as CoAP [10] or sMAP [7]. The remote               systems the cloud, network and embedded devices are converg-
      control system requires a dedicated, secure point-to-point        ing, thus creating very large-scale hyper-distributed systems,
      connection. Configuring these capabilities should be de-          which impose new concerns that are inherently elusive with
      coupled from the underlying physical infrastructure, in           traditional operations approaches.
      order to allow dynamic, fine-grained customization.                    RC4 – Due to dynamicity, heterogeneity, geographical
   • The FM system spans multiple, geographically distributed           distribution and the sheer scale of IoT cloud, traditional
      cloud instances and IoT devices that comprise FM’s                management and provisioning approaches are hardly feasible
      virtual runtime topologies. These topologies abstract a           in practice. This is mostly because they implicitly make as-
      portion of the IoT cloud infrastructure, e.g., needed by          sumptions such as physical on-site presence, manually logging
      specific subsystem, thus they should support flexible             into devices, understanding device’s specifics, etc., which are
      configuring to allow for on-demand provisioning.                  difficult, if not impossible, to achieve in IoT cloud systems.
                                                                        Thus, novel techniques, which will provide an unified and
The limited support for fine-grained provisioning at higher lev-
                                                                        conceptually centralized view on system’s configuration man-
els leads to tightly coupled, problem specific IoT infrastructure
                                                                        agement are needed.
components, which require difficult and tedious configuration
management on multiple levels. This inherently makes provi-                  Therefore, we need novel models and techniques to provi-
sioning and runtime operation of IoT cloud systems a com-               sion and operate the IoT cloud systems, at runtime. Some of the
plex task. Consequentially, system designers and operations             obvious requirements to make this feasible in the very large-
managers face numerous challenges to provision and operate              scale, geographically distributed setup are: (i) We need tools
large-scale IoT cloud systems such as our FM.                           which will automate development, provisioning and operations
                                                                        (DevOps) processes; (ii) Supporting mechanisms need to be
B. Challenges                                                           late-bound and dynamically configurable, e.g., via policies;
    RC1 – The IoT cloud services and subsystems provide                 (iii) Configuration models need to be centrally managed and
different functionality or analytics, but they mostly rely on           automatically propagated to the edge of the infrastructure.
common physical IoT infrastructure. However, to date the IoT
infrastructure resources have been mostly provided as coarse                    III.   P RINCIPLES AND BUILDING BLOCKS OF
                                                                                       SOFTWARE - DEFINED I OT SYSTEMS
grained, rigid packages, in the sense that the IoT systems,
e.g., the infrastructure components and software libraries are          A. Principles of Software-Defined IoT
specifically tailored for the problem at hand and do not allow              Generally, software-defined denotes a principle of abstract-
for flexible customization and provisioning of the individual           ing the low-level components, e.g., hardware, and enabling
resource components or the runtime topologies. This inherently          their provisioning and management through a well-defined API
hinders self-service, utility-oriented delivery and consumption         [14]. This enables refactoring the underlying infrastructure into
of the IoT resources at a finer granularity level.                      finer-grained resource components whose functionality can be
    RC2 – Elasticity, although one of the fundamental traits of         defined in software after they have been deployed.
the traditional cloud computing, has not yet received enough                Software-defined IoT systems comprise a set of resource
attention in IoT cloud systems. Elasticity is a principle to            components, hosted in the cloud, which can be provisioned
provision the required resources dynamically and on demand,             and controlled at runtime. The IoT resources (e.g., sensory
enabling applications to respond to varying load patterns               data streams), their runtime environments (e.g., gateways) and
by adjusting the amount of provisioned resources to exactly             capabilities (e.g., communication protocols, analytics and data

   Research challenges                                      High-level principles                                    Enablers
   • Flexible customization                                 • API encapsulation of IoT resources and capabilities    • Software-defined IoT units
   • Utility-oriented delivery and consumption              • Fine-grained resources consumption                     • Software-defined IoT topology (complex units)
   • Self-service usage model                               • Policy-based specification and configuration           • Centrally managed configuration models and policies
   • Support for elasticity concerns                        • Automated provisioning                                 • Automated units composition
   • Operating dependable large-scale IoT cloud systems     • Cost awareness                                         • Runtime unit control and modification
   • Central point of management                            • Runtime elasticity governance

 point controllers) are described as software-defined IoT units.                          allow for flexible system customization, we need to enable
 Software-defined IoT units are software-defined entities that are                        fine-grained resource consumption, well-defined API encapsu-
 hosted in an IoT cloud platform and abstract accessing and op-                           lation and provide support for policy-based specification and
 erating underlying IoT resources and lower level functionality.                          configuration. These principles are enabled by our software-
 Generally, software-defined IoT units are used to encapsulate                            defined IoT units and support for centrally managed configura-
 the IoT resources and lower level functionality in the IoT cloud                         tion. Figure 1 gives high-level graphical overview of the main
 and abstract their provisioning and governance, at runtime. To                           building blocks and enabling techniques, needed to support the
 this end, our software-defined IoT units expose well-defined                             main principles of software-defined IoT systems. Subsequently,
 API and they can be composed at different levels, creating                               we describe them in more detail.
 virtual runtime topologies on which we can deploy and execute
 IoT cloud systems such as our FM system. Therefore, main                                               B. Conceptual Model of Software-defined IoT Units
 principles of software-defined IoT systems include:
                                                                                                            Figure 2 illustrates the conceptual model of our software-
    • API Encapsulation – IoT resources and IoT capabilities                                            defined IoT units. The units encapsulate functional aspects
        are encapsulated in well-defined APIs, to provide a unified                                     (e.g., communication capabilities or sensor poll frequencies)
        view on accessing functionality and configurations of IoT                                       and non-functional aspects (e.g., quality attributes, elasticity
        cloud systems.                                                                                  capabilities, costs and ownership information) of the IoT
    • Fine-grained consumption – The IoT resources and capa-                                            resources and expose them in the IoT cloud. The functional,
        bilities need to be accessible at different granularity levels                                  provisioning and governance capabilities of the units are
        to support agile utilization and self-service consumption.                                      exposed via well-defined APIs, which enable provisioning and
    • Policy-based specification and configuration          Principles
                                                                               – The units              controlling the units at runtime, e.g., start/stop. Our conceptual
Principles       specified       declaratively       and    their      functionality
                                                            Fine-grained                      is
                                                                                 A) Software-defined model also allows for composing and interconnecting software-

A)      defined
     Software-defined  programmatically           in  software,
                                                            resources using the         well-
                                                                                    IoT unit            defined IoT units, in order to dynamically
                                                                                                                                            IoT Cloud Platformdeliver the IoT
                                                            abstraction and
           unit        API and available, familiarAPIsoftware    encapsulationlibraries.                resources and capabilities to the applications. The runtime
B) •UnitAutomated
            configuration provisioning – Main provisioning  Policy-based          processes
                                                                                 B) Unit’s configurationprovisioning and configuration is performed by specifying
                                                            provisioning and
C)      need       to
     Software-defined   be  automated         in order   to   enable
                                                                            dynamic,        on-
                                                                                 C) Software-defined    late-bound policies and configuration models. Naturally, the
                                                                                    runtime topology
     runtime           configuring and operatingFlexible
                 topology                                     software-defined
                                                                                           IoT          software-defined IoT units support mechanisms to map the
                                                                                 D) Runtime unit modification
D)      systems,
     Runtime            on a large-scale (e.g, hundreds
                  unit modification                         control gateways).      (e.g, changing      virtual resources with the underlying physical infrastructure.
           changing comm. protocol)
                  awareness – We need to be able            (e.g.,
                                                                     elastic   and communication
                                                                                      control protocol)To technically realize our unit model we introduce a
E)   Runtime topology                                       capabilities)        E) Runtime topology
        costs of(e.g,
     modification      delivered      IoT resources and capabilities in
                          add new link)
                                                                                    modification        concept of unit prototypes. They can be seen as resource con-
        to enable their utility-oriented consumption.                               (e.g, add new link)
                                                                                                        tainers, which are used to bootstrap more complex, higher-level
    • Elasticity support – They should support elasticity gov-                                          units. Generally, they are hosted in the cloud and enriched with
        ernance [9], by exposing runtime control of elastic capa-                                       functional, provisioning and governance capabilities, which
        bilities through well-defined API.                                                              are exposed via software-defined APIs. The unit prototypes
                                                                                                        can be based on OS-level virtualization, e.g., VMs, or more
                                                      A)               E)                               finer-grained kernel supported virtualization, e.g., Linux con-
                  IoT Cloud platform                                         Comm. protocol             tainers. Conceptually, virtualization choices do not pose any
            B) Configuration Models                     defined              Monitoring                 limitations, because by utilizing the well-defined API, our
                         Manager                       gateway
                                                                             Software library           unit prototypes can be dynamically configured, provisioned,
                            D)                                                                          interconnected, deployed, and controlled at runtime.
      C) Gateway                Gateway                                                                     Given our conceptual model (Figure 2), by utilizing the
                                                       A) Software-defined IoT unit
                                                                                                        provisioning API, the unit prototypes can be dynamically
          Communication                                B) Managed configuration
                Broker                Sensor                                                            coupled with late-bound runtime mechanisms. These can be
                                                       C) Software-defined IoT topology
                         Actuator                         (higher-level units)
                                                                                                        any software components (custom or stock), libraries or clients
                                                       D) Runtime unit control                          that can be configured and whose binding with the unit proto-
                                                          and modification                              types is differed to the runtime. For example, the mechanisms
                                                          (e.g, changing communication
   IoT Cloud Infrastructure                               protocol)
                                                                                                        can be used to dynamically add communication capabilities,
       Datacenters            IoT Infrastructure
                                                       E) Automated unit composition                    new functionality or storage to our software-defined IoT
                                                          (e.g., adding a capability)                   units. Therefore, by specifying policies, which are bound
                                                                                                        later during runtime, system designers or operations managers
 Fig. 1. Main enablers of software-defined IoT cloud systems                                            can flexibly manage unit configurations and customize their
                                                                                                        capabilities, at fine granularity levels. Our conceptual model
      Table I summarizes how we translate the aforementioned                                            also allows for composing the software-defined IoT units at
 high-level principles into concrete enablers. For example, to                                          higher levels. By selecting dependency units, e.g., based on

Non-functional aspe

                                                                                                                                                                                                                                                                                             Governance A
                                                                                                                                                                                                         Functional aspect

                                                                                                                                                                                          Functional A
                                                                                                                                                                                                                                        IoT Unit

                                                                                                                                                                                                                              Runtime           Runtime controllers
                                                                                                                                                                                                                             mechanisms           (e.g., elasticity)

                                                                           Provisioning API                                                                 logical structure. Therefore,                conceptually
                                                                                                                                                                                                   IoT resource                       we classify them
                                                                                                                                                                                                                and functionality binding

                                                                            Runtime composition                                                             into: (i) atomic,
                                                                                                                                                                          Enables   (ii)   composed             and capabilities
                                                                                                                                                                                                      Infrastructure      (iii) complex software-
                                                                      Dependency            Late-bound
                                                                                                                                                            defined IoT     units. Depending on their type, the units require
                                                                         units                policies                                                      specific runtime mechanisms and expose                    Non-func   specific  aspects:
                                                                                                                                                                          flexible                            How       many          cores,   memory

                                                                                                              Non-functional aspects

                                                                                                                                       Governance API
                                                                                                                                                            API. Next we describe each unit type in more detail.
                               Functional API
                                                Functional aspects              cost-function                                                                           pricing and
                                                                                                                                                                 The atomic software-defined IoT units are the finest-grained
                                                                                                                                                                        cost models
                                                                              Software-defined                                                              software-defined IoT units, which are used to abstract the core
                                                                                  IoT Unit                                                                  capabilities of an IoT resource.                They provide                  software-defined
                                                                                                                                                                                               Software-defined         IoT API
                                                                        Runtime               Runtime                                                       API and need to be packaged portably to include components
                                                                       mechanisms            controllers                                                                 Attributes
                                                                                                                                                            and libraries,     that are needed to provide desired capabilities.
                                                                                                                                                                     3 depicts some Governance
                                                                                                                                                                                           examples of the atomic           Provisioning  software-defined
                                                                     IoT resource and functionality binding                                                 units. We broadly classify them into functional and non-
                                                                         Infrastructure capabilities                                                        functional atomic software-defined                         IoT units, based on the
                                                                                                                                                                                       Internal External
                                                                                                                                                            capabilities they provide. Functional                         units encapsulate capa-
                Fig. 2.   Conceptual model of software-defined IoT units.
                                                                                                                                                            bilities such as communication
                                                                                                                                                            Unit.addLinkUnit()         Unit.stop()            or IoT compute andUnit.setPollFreq()  storage.
                                                                                                                                                            Non-functional        units encapsulate                 configuration
                                                                                                                                                                                                       Unit.changeLink()              Unit.addPolicy()
                                                                                                                                                                                                                                              models and
                                                                                                                                                            capabilities such as elasticity
                                                                                                                                                            Unit.addComProtocol()         replicateUnit()controllers                or data-quality
                                                                                                                                                                                                                     Unit.addDataPoint()                  en-
               their costs, analytics or elasticity capabilities, and linking them                                                                          forcement mechanisms. Therefore, the atomic units are used to
               together, we can dynamically build more complex units. This                                                                                  identify fine-grained capabilities needed by an application. For
               enables flexible policy-based specification and configuration                                                                                example, the application might require the communication to
               of complex relationships                  IoT API
                                                   between     the units. Therefore, by                                                                     be performed via a specific transport protocol, e.g., MQTT or
               carefully choosing the granularity of our units and providing                                                                                it might need a specific monitoring component, e.g., Ganglia1 .
               configuration policies we can automate the units composition                                                                                 Classifications similar to the one presented in Figure 3 can be
               process at different levels and in some cases completely defer                                                                               used to guide the atomic units selection process, in order to
         Configuration            Governance                Provisioning Functionality
               it to the runtime.      This makes the provisioning          process flexible,                                                               easily identify the exact capabilities, needed by the application.
               traceable and repeatable across different cloud instances and
                               Internal thus
               IoT infrastructures,         External
                                                reducing time, errors and costs.
                                                                                                                                    Atomic software-defined IoT units
                    The runtime governance API,           exposed by the units,
                                                      Unit.setMem()                   enables
               us to perform         runtime control operations such as starting
                              Unit.stop()                                                                                        Functional
                                                                    Unit.addPolicy()                                                                         Non-functional
               or stopping
                    Unit.start()the unit or  Unit.changeLink()
                                                 change the topological          structure of                                   capabilities                     capabilities
               the dependency
Unit.addComProtocol()                 units,  e.g.,
                                 replicateUnit()     dynamically      adding
                                                       Unit.addDataPoint()      or  removing
                                                                                  Unit.setCPU()                                                              ...
               dependencies at runtime. Therefore, one of the most important                              ... IoT compute             IoT data Communication
               consequences of having software-defined IoT unit is that the                                                            storage                                        Config.
                                                                                                                                                               Elasticity Data
               functionality of the virtual IoT infrastructure can be (re)defined                          GW            Data point                                       quality Security
               and customized after it has been deployed. New features can                              runtime           controller      History Network
               be added to the units and the topological structure of the                                      Custom              Volatile        overlay Protocol
                                                                                                              proc. logic Monitor.
               dependency units can be customized at runtime. This enables                                                                                        Auto scaling
               automating provisioning and governance processes, e.g., by                            Component              In-memory Key/Value VPN           group controller          Sand
                                                                                                       -model     CEP          image                      Messaging                      box
               utilizing the governance API and providing monitoring at unit                                                                 store                             filter
               level, we can enable elastic horizontal scaling of our units.
                    Therefore, most important features of software-defined IoT                      Fig. 3. Example classification of atomic software-defined IoT units.
               units which enable the general principles of software-defined
               IoT (see Section   Atomic   software-defined
                                       III-A)  are:           IoT units                                  The composed software-defined IoT units have multiple
                                                                                                    functional and non-functional capabilities, i.e., they are com-
                   • They Functional
                              provide software-defined API, which can be used                       posed of multiple atomic units. Similarly to the atomic units
                      to access,     configure and control   Non-functional
                                                                   the units, in a unified
                             capabilities                        capabilities                       they provide well-defined API, but require additional function-
                      manner.                                                                       ality such as mechanisms to support declaratively composing
                   • They support fine-grained internal      ...       configurations, e.g,         and binding the atomic units, at runtime (Section IV-B).
      ... IoT compute               IoT data capabilities
                      adding functional         Communicationlike different communica-              Example of composed unit is a software-defined IoT gateway.
                      tion protocols,storage
                                          at runtime.                                Config.
                                                               Elasticity Data                              The complex software-defined IoT units enable capturing
                   • They      can be composed at higher-level, via
                       Data point
                                                                          quality Security          complex relationships among the finer-grained units. Internally,
    runtime           dency     units, creating
                       controller                    virtual topologies that can be
                                         History Network                                            they are represented as a topological network, which can
                                 Volatile at runtime.
                                                  overlay Protocol                                  be configured and deployed, e.g., on the cloud. They define
                   • They
          proc. logic         enable decoupled and managed configuration (via
                        Monitor.                                                                    an API and can integrate (standalone) runtime controllers
                      late-bound policies) to provisionAuto         thescaling
                                                                         units dynamically
                                                               group  controller        Sand        to dynamically (re)configure the internal topology, e.g., to
 Component            and   on-demand.
                         In-memory       Key/Value VPN                        Outliers
             CEP                                                                         box        enable elastic horizontal scaling of the units. Finally, they
   -model          • Theyimagehave utilitystore           Messaging
                                               cost-functions     that enablefilter
                                                                                  pricing the
                                                                                                    rely on runtime mechanism to manage the references, e.g.,
                      IoT resources as utilities.
                                                                                                    IP addresses and ports, among the dependency units.
                                                                                                         We notice that the software-defined API and our units
               C. Units Classification                                                              offer different advantages to the stakeholders involved into
                    Depending on their purpose and capabilities, our software-
               defined IoT units have different granularity and internal topo-                           1

designing, provisioning and governing of software-defined IoT                                 stock components (e.g., Sedona5 or NiagaraAX6 execution
systems. For example, IoT infrastructure providers can offer                                  environments). Classifications similar to the one presented in
their resources at fine-granularity, on-demand. This enables                                  Figure 3 can be used to guide the atomic units selection
specifying flexible pricing and cost models and allows for                                    process. In case we want to perform custom builds of the
offering the IoT resources as elastic utilities in a pay-as-                                  existing libraries and frameworks, there are many established
you-go manner. Because our units support dynamic and au-                                      build tools which can be used, e.g., for Java-based components,
tomated composition on multiple levels, consumers of IoT                                      Apache Ant or Maven.
cloud resources can provision the units to exactly match their                                    On the second level, we configure the composed units, e.g.,
functional and non-functional requirements, while still taking                                a software-defined IoT gateway. This is performed by adding
advantage of the existing systems and libraries. Further, system                              the atomic units (e.g., runtime mechanisms and/or software
designers and operations managers, use late-bound policies                                    libraries) to the composed unit. For example, we might want
to specify and configure the unit’s capabilities. Because we                                  to enable the gateway to communicate over a specific transport
treat the functional and configuration units in a similar man-                                protocol, e.g., MQTT and add a monitoring component to it,
ner (see Section IV-B), configuration models can be stored,                                   e.g., a Ganglia agent. To perform this composition seamlessly
reused, modified at runtime and even shared among differ-                                     at runtime, additional mechanisms are required. We describe
ent stakeholders. This means that we can support managed                                      them in Section IV-B.
configuration models, which can be centrally maintained via                                       Third level includes defining the dependencies references
configuration management solutions for IoT cloud, e.g., based                                 between the composed units, which ”glue together” the com-
on OpsCode Chef2 , Bosh3 or Puppet4 .                                                         plex units. These links specify the topological structure of
    IV.     P ROVISIONING SOFTWARE - DEFINED I OT CLOUD                                       the desired complex units. For example, to this end we
                                                                                              can set up a virtual private network and provide each unit
                                                                                              with a list of IP addresses of the dependency units. In this
A. Automated composition of software-defined IoT units                                        phase, we can use frameworks (e.g., TOSCA-based, OpenStack
    Generally, building and deploying software-defined IoT                                    Heat, Amazon CloudFormation, etc.) to specify the runtime
cloud systems includes creating and/or selecting suitable                                     topological structure of our units and utilize mechanisms (e.g.,
software-defined IoT units, configuring and composing more                                    Ubuntu CloudInit7 ) to bootstrap the composition, e.g, pass the
complex units and building custom business logic components.                                  references to the dependency units.
The deployment phase includes deploying the software-defined
IoT units together with their dependency units and required                                   B. Centrally managed configuration models and policies
(possibly standalone) runtime mechanisms (e.g., a message                                         An important concept behind software-defined IoT cloud
broker). In this paper we mostly focus on provisioning reusable                               systems is the late-bound runtime policies. Our units are con-
stock components such as gateway runtime environments or                                      figured declaratively, via the policies by utilizing the exposed
available communication protocols. Developing custom busi-                                    software-defined API, without worrying about internals of the
ness logic components is out of scope of this paper and we                                    runtime mechanisms, i.e, the atomic units. To enable seamless
address it elsewhere [15].                                                                    binding of the atomic units we provide a special unit prototype,
                                                                                              called bootstrap container. The bootstrap container acts as a
     ACTIONS                       ACTIONS                         ACTIONS      Deploy        plug-in system, which provides mechanisms to define (bind)
  * Pullexternal                * Conf.policies
  repo                          * Select                       * Select
                                                                                              the units based on supplied configurations or to redefine them
  * Build                       unit prototype    Compo-       unit prototype                 when configuration policies are changed. For example, runtime
                    Atomic                                                      Complex
  * Select
                                * Resolve           sed        * Link unit
                                                                                  unit        changes of the units are achieved by invalidating affected parts
  unit prototype                dependencies        unit       dependencies
  * Configure                   * Exceptions                   * Exceptions
                                                                                              of the existing dependency tree and dynamically rebuilding
  * Exceptions       Select     and errors          Select     and errors                     them, based on the new configuration directives. Therefore, the
  and errors         third-                         third-                                    units can be simply    ”droped in” and our bootstrap container
                   party unit                     party unit
                                                                                              (re)binds them together,
                                                                                                                  runtime at runtime without rebooting system.
Fig. 4.    Automated composition of software-defined IoT units.                                   We decouple the   configuration models (late-bound policies)
                                                                                              from the functional units. Therefore, we can treat configuration
    Figure 4 illustrates most important steps to compose and                                  policies as any software-defined IoT unit, which adheres to
deploy our IoT units. There are three levels of configuration                                 the general principles of software-defined IoT (Section III-A).
that can
             be performed:AtomicUnit
                            (i) Building/selecting atomic units;                              By encapsulating the configuration policies in separate units,
(ii) Configuring composed units; (iii) Linking into complex                                   we can manage them at runtime via centralized configuration
units. Each of thelinkphases includes selecting and provisioning                              managements solutions for IoT cloud. Our framework provides
suitable unit prototypes.
                      *   ComplexUnit
                            For example, the unit prototypes can                              mechanisms to specify and propagate the configuration models
be based on different resource containers such as VMs, Linux                                  to the edge of IoT cloud infrastructure (e.g., gateways) and
Containers (e.g., Docker) or OSGi runtime.                                                    our bootstrap container enforces the provided directives. To
                                                                                              this end, our bootstrap container initially binds functional and
    The atomic units are usually provided as stock components,                                configuration units and continuously listens for configuration
e.g., by a third-party, possibly in a market-like fashion. There-                             changes and applies them on the affected functional units, ac-
fore, this phase usually involves selecting and configuring                                   cordingly. To enable performing runtime modifications without
     2                                                                   5
     3                                                        6
     4                                                                     7

worrying about any side-effects we require the configuration                implementation, the core of the InitalizationManager is an
actions to be idempotent. The usual approach to achieve this                OpsCode Chef client, which is passed to the VMs during
is to wrap the units as OS services. Among other things the                 initialization via Ubuntu cloud-init. InitalizationManager also
late-bound policies and our mechanisms for managed configu-                 provides mechanisms for configuration management. The De-
ration enable flexible customization and dynamic configuration              ploymentManager is used to deploy the software-defined IoT
changes, at runtime.                                                        units in the cloud. Our prototype relies on SALSA8 , a de-
                                                                            ployment automation framework we developed. It utilizes the
              V. P ROTOTYPE AND EXPERIMENTS                                 API exposed by the CloudSystemWrapper to enable deploy-
                                                                            ment across various cloud providers, currently implemented
A. Prototype implementation
                                                                            for OpenStack cloud. DeploymentManager is responsible to
    The main aim of our prototype is to enable developers and               manage and distribute the dependency references for the
operations managers to dynamically, on-demand provision and                 complex units (Section III-C). Units persistence layer provides
deploy software-defined IoT systems. This includes providing                functionality to store and manage our software-defined units
   Niagara gateway  IoT unit prototypes, enabling automated unit            and policies.
composition, at multiple levels and supporting centralized                      Framework                                                  > Individual u
runtime management of the configuration models.                             B. Experiments                                                  configuration
    Sedona gateway
    In Section III we introduced the conceptual model of our                    1) Scenario analysis: We now show how our prototype is     > Topology m



software-defined IoT units. To technically realize our units,

                                                                            used to provision a complex software-defined IoT unit, which
we utilize the   concept of Repository
             SensorA              virtual resource containers. More         provides functionality for the real-life FM location tracking
precisely, we provide different unit prototypes that can be                 service (Section II-A). The service reports vehicle location in
customized and/or modified during runtime by adding required                near real-time on the cloud. To enable remote access, the mon-
runtime mechanisms encapsulated in our atomic units. The unit               itored vehicles have an on-board device,Topology     model
                                                                                                                         acting as a gateway
prototypes provide resources with different granularity, e.g.,              to its data and control points. To improve performance and
              group quotas, priorities, etc., and boilerplate      func-    reliability, the golf course provides on-site gateways, which
tionality to enable automated provisioning of custom software-    topology communicate with the vehicles, provide additional processing
defined IoT units.                                                          and storage capabilities e.g,
                                                                                                              feed the data into the cloud.
    Figure 5 provides a high-level overview of the framework’s              Therefore, the physical software-defined
                                                                                                      IoT infrastructure comprises network
architecture. Our framework is completely hosted in the cloud               connected vehicles, on-board devices
                                                                                                             unit    and local gateways.
and follows a modular design which guarantees flexible and                      Typically, to provision the FM service system designers
evolvable architecture. The current prototype is implemented                and operations
                                                                           Runtime            manager would need to directly interact with
atop OpenStack [2], which is an open source Infrastructure-as-              the rigid physical IoT infrastructure. Therefore, they at least
a-Service (IaaS) cloud computing platform. Presentation layer               need to be aware of its topological structure and devices’
provides an user interface via Web-based UI and RESTful API.                capabilities. This means that the FM service also needs to have
They allow a user to specify various configuration models                   understanding of the IoT infrastructure, instead of being able to
and policies, which are used by the framework to compose                    customize the infrastructure to its needs. Due to inherent inflex-
and deploy our units in the cloud. Cloud core services layer                ibility of IoT infrastructure, its provisioning usually involves
contains the main functionality of the framework. It includes               long and tedious task such as manually logging into individual
the PolicyProcessor used to read the input configurations and               gateways, understanding gateway internals or even on site
transform it to the internal model defined in our framework.                presence. Therefore, provisioning even a simple FM location
Units management services utilize this model for composing                  tracking service involves performing many complex tasks. Due
and managing the units. The InitializationManager is respon-                to a large number of geographically distributed vehicles and
sible for configuring and composing more complex units. It                  involved stakeholders IoT infrastructure provisioning requires
translates the directives specified in configuration models into            a substantial effort prolonging service delivery and increasing
concrete initialization actions on the unit level. In our current           costs. Subsequently, we show the advantages our units (Sec-
                                                                            tion III-B) and the provisioning techniques (Section IV) have
                             Presentation layer
                                                                            to offer to operations managers and application designers in
                     Web UI                RESTful API                      terms of: a) Simplified provisioning to reduce time, costs and
                                                                            possible errors; b) Flexibility to customize and modify the IoT
                         Cloud core services layer                          units and their runtime topologies.
               Units management services             Policy
                                                   Processor                    To enable the FM system we developed a number of atomic
                                                                                              IoT units9 such as: a software-defined sensor


                                                                            that reports vehicle location in realtime, messaging infrastruc-


                                                                            ture based on Apache ActiveMQ10 , software-defined protocol
                                                                            based on MQTT and JSON, the bootstrap container based
                      Repository                                            on the Spring framework11 , and corresponding configuration
                       Services                                             units.  The experiments are simulated on our OpenStack (Fol-
                                                                            som) cloud and we used Ubuntu 12.10 cloud image (Memory:
                            Units persistence layer
                   Policies repo.          SD IoT units repo.                                   9
Fig. 5.   Framework architecture overview.                                                      11

2GB, VCPUS: 1, Storage: 20GB). To display location changes                         the dependencies between the units, the user continues com-
we develop a Web application which displays changes of                             posing on the finer granularity level. By applying the top-down
vehicles’ location on Google Maps.                                                 approach we enable differing design decisions and enable early
    2) Simplified provisioning: To demonstrate how our ap-                         automation of known functionality, to avoid over-engineering
proach simplifies provisioning of the virtual IoT infrastructure,                  and provisioning redundant resources.
we show how a user composes the FM complex software-                                   In the next phase, the user provisions individual unit proto-
defined IoT unit, using our framework. Figure 6 shows the                          types. To this end, he/she provides policies specifying desired
custom deployment of the topological structure of the FM                           finer-grained capabilities. Listing 2 shows example capabili-
vehicle tracking unit, deployed in the cloud. The unit contains                    ties, that can be added to the gateway. To enable asynchronous
two gateways for the vehicles it tracks, a web server for the                      pushing of the location changes it should communicate over
Web application and a message broker that connects them.                           the MQTT protocol. Listing 3 shows a part of Chef recipe used
                                                                                   to add MQTT client to the gateway. Our framework fetches the
                                                         SD UNIT                   atomic units, that encapsulate the required capabilities, from
                               UNIT PROTOTYPE       Id: mqtt_broker                the repository and composes them automatically, relying on
                                                   state: RUNNING
                             Id: VM container
                             state: RUNNING
                                                                                   the software-defined API and our bootstrap container.
                                                         SD UNIT
                                                   Id: SD gateway_1278             {"run_list":
         SALSA CENTER          UNIT PROTOTYPE      state: RUNNING                    ["recipe[bootstrap_container]",
      Id: sd IoT system      Id: VM container                                        "recipe[mqtt-client]",
      state: RUNNING         state: RUNNING               SD UNIT                    "recipe[protocol-config-unit]",
                                                   Id: SD gateway_1280               "recipe[sd-sensor]"]
                                                    state: RUNNING                 }
                                UNIT PROTOTYPE
                              Id: VM container
                              state: RUNNING             SD UNIT
                                                                                   Listing 2.   Run list for software-defined gateway.
                                                   Id: Web server
                                                   state: RUNNING

                                                                                   include_recipe ’bootstrap_container::default’
Fig. 6.   Topological structure of FM vehicle tracking unit (a screen shot).       remote_file "mqtt-client-0.0.1-SNAPSHOT.jar" do
                                                                                    source ""
                                                                                    group "root"
    In order to start provisioning the complex unit, system                         mode 00644
                                                                                    action :create_if_missing
designer only needs to provide a policy describing the re-                         end
quired high-level resources and capabilities required by the
FM service. For example, Listing 1 shows a snippet from                            Listing 3.   Chef recipe for adding the MQTT protocol.
the configuration policy for FM location tracking unit, that
illustrates specifying a software-defined gateway, for the on-                     Therefore, compared to the traditional approaches, which
board device.                                                                      require gateway-specific knowledge, using proprietary API,
                                                                                   manually logging in the gateways to set data points, our
                                                       ative unit configuration policies, simplifies the provisioning
                                                                 process and makes it traceable and repeatable. Our units can
                                                        easily be shared among the stakeholders and composed to
      m1.small                            provide custom functionality. This enables system designers
      openstack@dsg                           and operations managers to rely on the existing, established
                                                                 systems, thus reducing provisioning time, potential errors and
                                                                   3) Flexible customization: To exemplify the flexibility of
                                                                                   figuration of the FM unit to use CoAP instead of MQTT.
                                                        This can be due to requirements change (Section II-A),
                                                       for a golf course with different networking capabilities. To
                                                              customize the existing unit, an operations manager only needs
...                                                                                to change the "recipe[protocol-config-unit]"unit
Listing 1.    Partial TOSCA-like complex unit description.                         (Listing 2) and provide an atomic unit for the CoAP client.
                                                                                   This is a nice consequence of our late-bound runtime mecha-
The policy describes gateway’s initial configuration and the                       nisms and support for managed configuration models, provided
cloud instance where it should be deployed. Additionally, it                       by our framework. We treat both functional and configuration
defines a dependency unit, i.e. the MQTT broker and specifies                      units in the same manner and our bootstrap container manages
vehicle’s Id that can be used to map it on the underlying device,                  their runtime binding (Section IV-B). Compared to traditional
as shown in [15]. Our framework takes the provided policy,                         approaches that require addressing each gateway individually,
spawns the required unit prototypes and provides them with                         firmware updates or even modifications on the hardware level,
references to the dependency units. At this stage the virtual                      our framework enables flexible runtime customization of our
infrastructure comprises solely of unit prototypes (VM-based).                     units and supports operation managers to seamlessly enforce
After performing the high-level unit composition and establish                     configuration baseline and its modifications on a large-scale.

VI. R ELATED W ORK                                 for provisioning software-defined IoT systems. The initial
    Recently, many interesting approaches enabling conver-              results are promising in the sense that software-defined IoT
gence of IoT and cloud computing appeared. For example,                 system enable sharing of the common IoT infrastructure among
in [5], [8], [11], [16], [18] the authors mostly deal with              multiple stakeholders and offer advantages to IoT cloud system
IoT infrastructure virtualization and its management on cloud           designers and operations managers in terms of simplified, on-
platforms, [6], [13] utilize the cloud for additional computation       demand provisioning and flexible customization. Therefore,
resources, in [17], [19] the authors mostly focus on utilizing          we believe that software-defined IoT systems can significantly
cloud’s storage resources for Big-IoT-Data and [12], [16]               contribute the evolution of the IoT cloud systems.
integrating IoT devices with enterprise applications based on               In the future we plan to continue developing the prototype
SOA paradigm. Due to limited space in the following we only             and extend it in several directions: a) Providing techniques
mention related work exemplifying approaches that deal with             and mechanisms to support runtime governance of software-
IoT infrastructure virtualization and management.                       defined IoT systems; b) Enabling our software-defined IoT
    In [11] the authors develop an infrastructure virtualization        systems to better utilize the edge of infrastructure, e.g., by
framework for wireless sensor networks. It is based on a                providing code distribution techniques between cloud and IoT
content-based pub/sub model for asynchronous event exchange             devices; c) Enabling policy-based automation of data-quality,
and utilizes a custom event matching algorithm to enable                security and safety aspects of software-defined IoT systems.
delivery of sensory events to subscribed cloud users. In [18] the
authors introduce sensor-cloud infrastructure that virtualizes                             ACKNOWLEDGMENT
physical sensors on the cloud and provides management and                   This work is sponsored by Pacific Controls Cloud Com-
monitoring mechanisms for the virtual sensors. However, their           puting Lab (PC3L12 )
support for sensor provisioning is based on static templates
that, contrary to our approach, do not support dynamic provi-                                           R EFERENCES
sioning of IoT capabilities such as communication protocols.             [1]   Mq telemetry transport. March 2014.
SenaaS [5] mostly focuses on providing a cloud semantic                  [2]   Openstack. March 2014.
overlay atop physical infrastructure. They define an IoT on-             [3]   Thingworks. Last accessed: March 2014.
tology to mediate interaction with heterogeneous devices and             [4]   Xively. Last accessed: March 2014.
data formats, exposing them as event stream services to the              [5]   S. Alam, M. Chowdhury, and J. Noll. Senaas: An event-driven sensor
upper layers. OpenIoT framework [16] utilizes semantic web                     virtualization approach for internet of things cloud. In NESEA, 2010.
technologies and CoAP [10] to enable web of things and                   [6]   B.-G. Chun, S. Ihm, P. Maniatis, M. Naik, and A. Patti. Clonecloud:
                                                                               elastic execution between mobile device and cloud. In Proceedings of
linked sensory data. They mostly focus on discovering, linking                 the sixth conference on Computer systems. ACM, 2011.
and orchestrating internet connected objects, thus conceptually          [7]   S. Dawson-Haggerty, X. Jiang, G. Tolle, J. Ortiz, and D. Culler. smap:
complementing our approach. In [8] the authors focus on                        a simple measurement and actuation profile for physical information.
developing a virtualization infrastructure to enable sensing                   In SenSys, pages 197–210, 2010.
and actuating as a service on the cloud. They propose a                  [8]   S. Distefano, G. Merlino, and A. Puliafito. Sensing and actuation as a
software stack which includes support for management of                        service: a new development for clouds. In NCA, pages 272–275, 2012.
device identification, selection and aggregation, all of which           [9]   S. Dustdar, Y. Guo, B. Satzger, and H.-L. Truong. Principles of elastic
can be seen as enablers of our approach. Also there are various                processes. Internet Computing, IEEE, 15(5):66–71, 2011.
commercial solutions such as Xively [4] and ThingWorx [3],              [10]   B. Frank, Z. Shelby, K. Hartke, and C. Bormann. Constrained applica-
                                                                               tion protocol (coap). IETF draft, Jul, 2011.
which allow users to connect their sensors to the cloud and
enable remote access and management.                                    [11]   M. M. Hassan, B. Song, and E.-N. Huh. A framework of sensor-cloud
                                                                               integration opportunities and challenges. In ICUIMC, 2009.
    Most of these approaches focus on different virtualization          [12]   M. Kovatsch, M. Lanter, and S. Duquennoy. Actinium: A restful runtime
techniques for IoT devices and data format mediation. They                     container for scriptable internet of things applications. In Internet of
also enable some form of configuration, e.g., setting sensor                   Things, pages 135–142, 2012.
poll rates. These approaches can be seen as complementary               [13]   K. Kumar and Y.-H. Lu. Cloud computing for mobile users: Can
to our own, as device virtualization sets the corner stone                     offloading computation save energy? Computer, 43(4):51–56, 2010.
for achieving software-defined IoT systems. We rely on the              [14]   B. Lantz, B. Heller, and N. McKeown. A network in a laptop: rapid
advances in convergence of IoT and cloud, introduced by the                    prototyping for software-defined networks. In Proceedings of the 9th
                                                                               ACM SIGCOMM Workshop on Hot Topics in Networks. ACM, 2010.
previous work and extend it with novel concepts for abstracting
                                                                        [15]   S. Nastic, S. Sehic, M. Voegler, H.-L. Truong, and S. Dustdar. Patricia
and encapsulating IoT resources and capabilities, exposing                     - a novel programing model for iot applications on cloud platforms. In
them via software-defined API on the cloud and enabling fine-                  SOCA, 2013.
grained provisioning. Therefore, our approach can be seen as            [16]   J. Soldatos, M. Serrano, and M. Hauswirth. Convergence of utility
a natural step in evolution of IoT cloud systems.                              computing with the internet-of-things. In IMIS, pages 874–879, 2012.
                                                                        [17]   P. Stuedi, I. Mohomed, and D. Terry. Wherestore: Location-based data
                     VII. C ONCLUSION                                          storage for mobile devices interacting with the cloud. In ACM Workshop
                                                                               on Mobile Cloud Computing & Services. ACM, 2010.
    In this paper, we introduced the conceptual model of                [18]   M. Yuriyama and T. Kushida. Sensor-cloud infrastructure-physical
software-defined IoT units. To our best knowledge this is                      sensor management with virtualized sensors on cloud computing. In
the first attempt to apply software-defined principles on IoT                  NBiS, 2010.
systems. We showed how they are used to abstract IoT                    [19]   A. Zaslavsky, C. Perera, and D. Georgakopoulos. Sensing as a service
resources and capabilities in the cloud, by encapsulating                      and big data. arXiv preprint arXiv:1301.0159, 2013.
them in software-defined API. We presented automated unit
composition and managed configuration, the main techniques                 12

You can also read
NEXT SLIDES ... Cancel