Communication Protocols of an Industrial Internet of Things Environment: A Comparative Study - MDPI
Page content transcription
If your browser does not render page correctly, please read the page content below
Future Internet 2019, 11, 66 2 of 18 In the industry context, Industrial Internet of Things (IIoT) [22,23], Industry 4.0 [24,25], Smart Manufacturing [26,27], and Smart Factory [28,29] are all terminologies for the emerging industrial environments that employ information and communication technologies (ICTs), including IoT platforms, while maintaining industry requirements. The term IIoT refers to the use of IoT technologies in many fields of application such as manufacturing, factories, transportation, gas, oil, and electric grids. However, most industrial and SCADA-like (supervisory control and data acquisition) systems of these fields employ proprietary communication protocols and ICTs, which lead to closed industrial systems. Hence, customers are stuck to a single vendor, costs are high, and interoperability is not fulfilled. In this paper, standard IoT protocols are introduced and compared. Then, MQTT is chosen for machine-to-machine (M2M) communications to complement the MODBUS TCP [30–32] operations in an IIoT environment. This environment integrates an event-based message-oriented protocol, i.e., MQTT, with a polling-based request–response protocol, intended for industrial applications, i.e., MODBUS TCP. The study shows that the MODBUS TCP and MQTT can coexist together, and in parallel, within the same IIoT environment. While industrial requirements of control and monitoring can be met via the MODBUS TCP using the request–response model, the MQTT protocol complements its operation by fulfilling the IoT requirements, using the publish–subscribe pattern for M2M communications. The IoT protocol works in parallel with the MODBUS TCP and relays industrial data to an Internet-based server for remote monitoring, analysis, and archiving. To solve the interoperability problem, industrial data are formatted using the MODBUS message format and transferred in the payload of the MQTT publish messages. In fact, the IoT protocols cannot replace the MODBUS protocol in most industrial applications, especially when an optimized message structure is required in the application layer. A second scenario is proposed here, which is to consider the MODBUS TCP as an IoT protocol, and hence build the IIoT environment with the MODBUS TCP only. This solution is efficient and complies with most industrial applications; however, it only depends on the request–response model. The choice of solution is totally dependent on the requirements of the industrial application. The developed environment can be employed in many industrial applications, including Internet SCADA, Internet-based monitoring, and industrial control systems. Hence, customers are not stuck to a single vendor, costs are reduced, and interoperability is maintained. The remainder of the paper is organized as follows. Section 2 presents a related literature survey, Section 3 presents a theoretical background of the MODBUS protocol, Section 4 compares IoT protocols, and Section 5 compares MQTT and the MODBUS TCP. Section 6 compares the latencies and resource usage of both protocols. Section 7 presents a discussion and Section 8 concludes the paper. 2. Related Work In the context of industry, researchers are proposing and examining protocols, networks, and middleware architectures for industrial ICT infrastructure and integration. For integration purposes, the authors of  proposed a data-oriented machine-to-machine (M2M) communication middleware based on the ZeroMQ platform for IIoT applications. The researchers of  presented a case study for controlling industrial robots and monitoring energy consumption remotely based on Ebbits middleware, which transforms data into web services. Another service-oriented IIoT middleware is proposed in  for balancing task allocation between a mobile terminal and a utility cloud service. In , a platform based on SystemJ for IIoT is proposed using an FPGA (field programmable gate array), and then tested in an automation system. Reference  describes a collaboration-oriented M2M (CoM2M) messaging mechanism for IIoT, based on the platform PicknPack food packaging line. In , legacy flexible manufacturing systems are integrated with the SCADA system of an Industry 4.0 framework via Ethernet. However, these research papers investigated middleware architectures for integration purposes. The infrastructure of an IIoT was examined in , where the IoT vision in industrial wireless sensor networks was implemented using the IPv6 over low-power wireless personal area network
Future Internet 2019, 11, 66 3 of 18 (6LoWPAN) and CoAP. Furthermore, the author of  proposed a software-defined network for an IIoT based on new networking technologies. However, these research papers investigated network solutions and architectures. Infrastructure protocols were examined in many studies. For example, the authors of  developed an edge IoT gateway to extend the connectivity of MODBUS devices to IoT by storing the scanned data from MODBUS devices locally, and then transferring the changes via an MQTT publisher to MQTT clients via a broker. The researchers of  designed and implemented a web-based real-time data monitoring system that uses MODBUS TCP communications, following which all data are displayed in a real-time chart in an Internet browser, which is refreshed at regular intervals using hypertext transfer protocol (HTTP) polling communications. In , a MODBUS serial protocol was reported that collects data serially via a RS-232 protocol and transfers the collected data over the ZigBee protocol. In , measurements of field devices collected by the MODBUS serial protocol were transferred over HTTP using a Wi-Fi network. However, these papers proposed a gateway as a bridge between MODBUS and the Internet, Intranet, or wireless network. In this study, the MODBUS TCP is proposed to be implemented in two scenarios—either alone, or in parallel with an MQTT publisher without a gateway. The first solution proposes the MODBUS TCP as an IoT protocol to build the industrial environment on a standalone basis. Using the second arrangement, the MODBUS TCP executes its operations while maintaining the requirements of industrial applications and MQTT achieves the M2M communications for IoT functions. 3. MODBUS Theory The MODBUS TCP is a byte-oriented, industrial communication protocol, open de facto standard, used for data exchange between embedded systems, devices, and industrial applications. Devices, reacting as clients, may benefit from the inexpensive implementation of such a lightweight protocol for polling industrial devices that react as servers. Polling communications follow the request–response mechanism, where a client queries the server for specific data or executes commands in the server using a frame of bytes arranged in a specific way, called a frame format. The server replies to the client queries via a frame of bytes either holding measurement data from sensors or confirming the execution of commands. Sixteen-bit data registers store measurement values, and coils hold the status of ON and OFF switches. Therefore, MODBUS TCP uses the polling mechanism, as opposed to the event-based mechanism, explained in the next section. As listed in Table 1, the protocol specifications  define three categories of function codes for the access of data in remote devices. These data are stored in coils or registers as status values for measurements or transferred as setpoints for control. Coils perform one-bit read and write operations for switching the attached devices ON and OFF or reading and writing one-bit internal configuration values. Discrete inputs perform one-bit read operations for reading the status of the attached devices, whether they are switched ON or OFF. The 16-bit input registers are responsible for measurements from physical devices, and the 16-bit holding registers perform read and write operations related to internal reconfigurable values. Table 1. Function codes for data access in MODBUS. Data Access Type Function Code Meaning 1 bit physical discrete input 0x02 read discrete inputs 1 bit internal bits, physical coils 0x01 read coils 1 bit internal bits, physical coils 0x05 write single coil 1 bit internal bits, physical coils 0x0F write multiple coils 16 bit physical input registers 0x04 read input registers 16 bit internal and physical output registers 0x03 read holding registers 16 bit internal and physical output registers 0x06 write single register 16 bit internal and physical output registers 0x10 write multiple registers 16 bit internal and physical output registers 0x17 read/write registers 16 bit internal and physical output registers 0x16 mask write register 16 bit internal and physical output registers 0x18 read first in first out (FIFO) queue
internal and physical output 16 bit 0x17 read/write registers registers internal and physical output 16 bit 0x16 mask write register registers internal and physical output Future Internet 2019, 11, 66 read first in first out (FIFO) 4 of 18 16 bit 0x18 registers queue A message structure of a MODBUS TCP client query for reading input registers is shown in Figure 1. The slave replies to the master query in the same format with the read registers using the function code (FC) “read input registers” (FC = 0x04), or as a confirmation to executing commands in case of other function codes such as “write single coil” coil” (FC(FC == 0x05). The header of the MODBUS frame consists of four fields: a two-byte two-byte transaction transaction identifier identifier (ID); (ID); a two-byte protocol type (MODBUS over TCP); a two-byte length, which counts the number of bytes for the therest restfields; fields;and and a one-byte a one-byteunitunit identifier (Unit). identifier However, (Unit). the protocol However, data unit the protocol (PDU) data unitconsists (PDU) of a one-byte consists function function of a one-byte code (FC), which code (FC),is,which here, ais,code here,toaread codetheto registers read the and a data registers and field that field a data may contain that may other fieldsother contain depending fields on the FC itself. depending Both on the FCtheitself. header Bothandthe theheader PDU form andan theapplication PDU formdata an unit (ADU),data application which is the unit complete (ADU), whichframe is theof the query. complete frame of the query. The following followingillustrative illustrativeexample exampleexplains explainsthetheprinciple principleof the of theMODBUS MODBUS frame format frame that uses format that a function uses codecode a function (0x04) to read (0x04) threethree to read continuous input continuous registers input in ainremote registers a remotedevice. device.The function The function is able to read from 1 to 125 contiguous is able to read from 1 to 125 contiguous input registers. Here, a client query asks a server registers. Here, a client query asks a server to read the to read the values values of three of three continuous continuous inputinput registers—register registers—register address address “14” “14” (0x000E), (0x000E), register register addressaddress “15” “15” (0x000F), (0x000F), and and register register address address “16” “16” (0x0010). (0x0010). Therefore, Therefore, thetheclient clientsends sendsaa single single message “0001000000060104000E0003” “0001000000060104000E0003”andand the serverthe replies serverby sending repliesone frame by “000100000009010406FE20 sending one frame 6666A63F” that contains three values of that “000100000009010406FE206666A63F” continuous containsregisters. The first three values ofregister contains continuous the hexadecimal registers. The first value register“0xFE20,” containswhich corresponds the hexadecimal to the value sixteen-bit “0xFE20,” signed which short integer corresponds value to the “11111110 sixteen-bit 00100000” signed short or the decimal integer value “–480”. value “11111110 00100000”Theorlastthetwo registers decimal valuehold the The “–480”. IEEE last754 twoshort floating-point registers hold the IEEE representation “0x3FA66666” 754 short floating-point or the decimal“0x3FA66666”  representation value “1.3”. or the decimal value “1.3”. Figure 1. MODBUS query and response, an illustrative example. Using the MODBUS MODBUS specifications specifications, ,the theinformation informationfromfromTable Table1, 1, andandthethe MODBUS MODBUS frame of frame Figure 1, the of Figure total 1, the consumed total consumedbytes (size) bytes (size)can canbebecalculated calculatedviaviaEquation Equation(1). (1).The The formula formula is is derived for the function code of “read input registers” (0x04) and is plotted in Figure 2, where N refers to the number of registers, which is doubled because because each each register register contains contains two two bytes: bytes: Size = request_bytes + response _bytes = (7 + 5) + (9 + 2 × N ) . (1) = 2 × N + 21 For the request case, the header occupies 7 bytes, the function code occupies 1 byte, the start register-address occupies 2 bytes, and the quantity occupies 2 bytes. For the response case, the header occupies 7 bytes, the function code 1 byte, and the length 1 byte.
2 * N 21 For the request case, the header occupies 7 bytes, the function code occupies 1 byte, the start register-address occupies 2 bytes, and the quantity occupies 2 bytes. For the response case, the header Future occupies Internet 2019, 11,766 bytes, the function code 1 byte, and the length 1 byte. 5 of 18 Figure Relationship Figure2. 2. between Relationship betweenthethe number of bytes number and and of bytes the number of registers the number of function of registers codes (FC) of function codes “read (FC)input “readregisters” (0x04), “write input registers” multiple (0x04), “writecoils” (0x0F), multiple and (0x0F), coils” “write multiple and “writeregisters” (0x10). multiple registers” (0x10). The same principle is applied to other function codes, such as “read holding registers” (0x03), which The has the samesame linear equation; principle is applied“write multiple to other registers” function codes, (0x10), such aswhich “readcan be represented holding registers”by the (0x03), linear equation (2 ∗ N + 25); and “write multiple coils” (0x0F), which can be represented which has the same linear equation; “write multiple registers” (0x10), which can be represented by by the linear equation the linear(N equation + 25). As (2*N an example, if the + 25); and number “write of registers multiple coils” (N) is 4, which (0x0F), the sizecan is 29 befor the “read input represented by the registers” (0x04) and 33 for the “write multiple registers” (0x10) function codes. linear equation (N + 25). As an example, if the number of registers (N) is 4, the size is 29 for the “read If the input client application registers” (0x04) and 33requires for thethe execution “write multipleof both operations—read registers” (0x10) function andcodes. write—within the same message for application If the client a remote device, the the requires function codeof“read/write execution multiple registers” both operations—read (0x17) shows and write—within the higher performance than both the “read input registers” (0x04) and the “write same message for a remote device, the function code “read/write multiple registers” (0x17) multiple registers” shows (0x10) higherfunction codes ifthan performance usedboth separately. the “read Equation (2) illustrates input registers” twoand (0x04) linear the formulae for the function “write multiple registers” code “read/write multiple registers” (0x17), which sends both commands within the same (0x10) function codes if used separately. Equation (2) illustrates two linear formulae for the function message: code “read/write multiple registers” (0x17), which sends both commands within the same message: Bytes_request = 17 + 2 × NRead Bytes _ request 17 2 * NRead Bytes_response = 9 + 2 × NWrite . (2) Bytes _ response 9 2 * N Write . (2) ∴ Total_size = 26 + 2 × ( NRead + NWrite ) Total _ size 26 2 * NRead N Write Both equations of the function code “read/write multiple registers” (0x17) request and response Both equations of the function code “read/write multiple registers” (0x17) request and response are plotted in Figure 3. Both equations are linear and depend on the number of registers to be written are plotted in Figure 3. Both equations are linear and depend on the number of registers to be written (NWrite ) and read (NRead ). (NWrite) and read (NRead). As a result, the MODBUS TCP has an optimized frame structure suitable for SCADA-like systems As a result, the MODBUS TCP has an optimized frame structure suitable for SCADA-like and has a communication mechanism that fulfills the industrial requirements. In addition, it has a systems and has a communication mechanism that fulfills the industrial requirements. In addition, it communication model and pattern that are compatible with industrial applications. As shown in has a communication model and pattern that are compatible with industrial applications. As shown Figures 1–3, the protocol is lightweight. Moreover, it has an open specification, and uses TCP/IP networks. Accordingly, it can be considered as an IoT protocol.
Future Internet 2019, 11, x FOR PEER REVIEW 6 of 18 in Figures 1–3, the protocol is lightweight. Moreover, it has an open specification, and uses TCP/IP networks. Future Internet Accordingly, 2019, 11, 66 it can be considered as an IoT protocol. 6 of 18 Figure3.3.Relationship Figure Relationshipbetween betweenthe thenumber numberofofbytes bytesand andthe thenumber numberofofregisters registersofofthe theMODBUS MODBUS function code (FC) “read/write multiple registers” (0x17). function code (FC) “read/write multiple registers” (0x17). 4. Comparison between IoT Protocols 4. Comparison between IoT Protocols A performance comparison between HTTP and MQTT is conducted in  on required network A performance comparison between HTTP and MQTT is conducted in  on required resources for IoT, while the payload was fixed to zero bytes and the topic names were varied. network resources for IoT, while the payload was fixed to zero bytes and the topic names were In addition, a performance analysis of MQTT, HTTP, and CoAP was performed in  for IoT-based varied. In addition, a performance analysis of MQTT, HTTP, and CoAP was performed in  for monitoring of a smart home. The authors of  discussed and analyzed the efficiency, usage, and IoT-based monitoring of a smart home. The authors of  discussed and analyzed the efficiency, requirements of MQTT and CoAP. Moreover, the authors of  compared AMQP and MQTT over usage, and requirements of MQTT and CoAP. Moreover, the authors of  compared AMQP and mobile networks, and the authors of  emulated a quantitative performance assessment of CoAP in MQTT over mobile networks, and the authors of  emulated a quantitative performance comparison with HTTP. assessment of CoAP in comparison with HTTP. In this section, the main differences between HTTP, CoAP, MQTT, AMQP, XMPP, and MODBUS In this section, the main differences between HTTP, CoAP, MQTT, AMQP, XMPP, and TCP protocols are discussed from various telecommunication aspects. Then, a protocol is chosen that MODBUS TCP protocols are discussed from various telecommunication aspects. Then, a protocol is fulfills the requirements of the IIoT environment. chosen that fulfills the requirements of the IIoT environment. Accordingly, Table 2 summarizes these differences from different communication aspects Accordingly, Table 2 summarizes these differences from different communication aspects including infrastructure, architecture, mechanism, model, messaging pattern, methodology, and including infrastructure, architecture, mechanism, model, messaging pattern, methodology, and transmission paradigm. These protocols use the client–server communication architecture. HTTP transmission paradigm. These protocols use the client–server communication architecture. HTTP uses the request–response model and is a document-oriented protocol, whereas MQTT uses the uses the request–response model and is a document-oriented protocol, whereas MQTT uses the publish–subscribe model and is message-oriented. Thus, MQTT is one-to-many, and HTTP publish–subscribe model and is message-oriented. Thus, MQTT is one-to-many, and HTTP is is one-to-one (peer-to-peer). CoAP uses a specific infrastructure—namely, 6LoWPAN (IEEE one-to-one (peer-to-peer). CoAP uses a specific infrastructure—namely, 6LoWPAN (IEEE 802.15.4)—which employs IPv6 in the network layer. Both MQTT and HTTP use an inexpensive and 802.15.4)—which employs IPv6 in the network layer. Both MQTT and HTTP use an inexpensive and available communication infrastructure, which is Internet or Intranet in wire mode (Ethernet—IEEE available communication infrastructure, which is Internet or Intranet in wire mode (Ethernet—IEEE 802.3) or wireless mode (Wi-Fi—IEEE 802.11)—which may employ either IPv4 or IPv6 in the network 802.3) or wireless mode (Wi-Fi—IEEE 802.11)—which may employ either IPv4 or IPv6 in the layer. In the transport layer, MQTT and HTTP protocols use TCP port numbers 1883 and 80, respectively. network layer. In the transport layer, MQTT and HTTP protocols use TCP port numbers 1883 and 80, However, CoAP uses UDP port number 5683. Given that MQTT is event-based, it is a message-oriented respectively. However, CoAP uses UDP port number 5683. Given that MQTT is event-based, it is a protocol. Thus, CoAP mimics HTTP in using polling-based messaging, but in a shorter time and message-oriented protocol. Thus, CoAP mimics HTTP in using polling-based messaging, but in a smaller frame-size. shorter time and smaller frame-size.
Future Internet 2019, 11, x FOR PEER REVIEW 7 of 18 Future Internet 2019, 11, 66 7 of 18 Table 2. Comparison of Internet of Things (IoT) protocols. Table 2. Comparison of Internet of Things (IoT) protocols. Feature HTTP CoAP MQTT MODBUS TCP Feature infrastructure HTTP Ethernet, Wi-Fi CoAP 6LoWPAN MQTT Ethernet, Wi-Fi MODBUS Ethernet, TCP Wi-Fi network layer infrastructure IPv4 or IPv6 Ethernet, Wi-Fi IPv6 6LoWPAN IPv4 or IPv6 Ethernet, Wi-Fi IPv4 or IPv6 Ethernet, Wi-Fi network transportlayer layer IPv4 TCP or IPv6 IPv6 UDP IPv4 TCPor IPv6 IPv4 TCPor IPv6 transport transportlayer port 80, TCP 443 UDP 5683 1883,TCP 8883 TCP 502, 802 transport modelport 80, 443 synchronous 5683 asynchronous 1883, asynchronous8883 502, synchronous 802 model pattern synchronous request—response asynchronous both asynchronous synchronous publish—subscribe request—response pattern request—response both publish—subscribe request—response mechanism one-to-one one-to-one one-to-many one-to-one mechanism one-to-one one-to-one one-to-many one-to-one methodology document-oriented document-oriented message-oriented byte-oriented methodology document-oriented document-oriented message-oriented byte-oriented paradigm paradigm long longpolling-based polling-based polling-based polling-based event-based event-based polling-based polling-based quality level quality level one level one level two: CON or NON two: CON or NON three: QoS 0, 1, 2 three: QoS 0, 1, 2 oneone level level standard standard IETF IETF(RFC7230) (RFC7230) IETF IETF(RFC7252) (RFC7252) ISO/IEC, ISO/IEC, OASIS OASIS modbus.org modbus.org encoding encoding ASCII ASCII text text RESTful RESTful (Binary) (Binary) UTF-8 UTF-8(Binary) (Binary) Binary Binary security security SSL, SSL, TLS TLS DTLS DTLS SSL, SSL, TLSTLS TLSTLS CoAP is an application layer protocol, dedicated to communication with constrained devices in CoAP is IoT IPv6-based an application layer protocol, infrastructures. dedicated to communication Two communication patterns are withused constrained by CoAP, devicesi.e., in IPv6-based IoT infrastructures. Two communication patterns are used by publish–subscribe and request–response . The CoAP messaging pattern is based on the exchange CoAP, i.e., publish–subscribe and of request–response messages . The CoAP between endpoints messaging and uses a shortpattern is based fixed-length on the binary exchange header thatof maymessages between be followed by endpoints and uses a short fixed-length binary header that may be followed a compact binary option and a payload. Compared to HTTP, as shown in Figure 4, CoAP runs over by a compact binary option andconnectionless the a payload. Compared UDP in to theHTTP, as shown transport layer,inwhereas Figure 4,inCoAP runs over the network the CoAP layer, connectionless uses eitherUDPIPv6in the6LoWPAN. or transport layer,When whereas CoAP uses in theIPv6, network layer, CoAP it is necessary foruses it to either IPv6 oror use Ethernet 6LoWPAN. Wi-Fi for the When CoAP data link uses IPv6, it is necessary for it to use Ethernet or Wi-Fi for the data link and physical layers, respectively. When CoAP uses 6LoWPAN, it employs IEEE 802.15.4e for the and physical layers, respectively. When CoAP data link anduses 6LoWPAN, physical layers. it employs IEEE 802.15.4e for the data link and physical layers. The content The content (payload) (payload) of of HTTP HTTP may may vary vary according according to to the the type type of of transferred transferred data, called data, called content-type, which could be plain text, HTML, XML, GIF image, content-type, which could be plain text, HTML, XML, GIF image, PDF application, or audio. For thePDF application, or audio. For the exchange exchange of dataofusing data using HTTP,HTTP, XMLXML is used, is used, whichwhich handles handles verboseplain verbose plaintext text for for solving solving interoperability issues. However, for CoAP, the efficient XML interchange (EXI)  is used, which interoperability issues. However, for CoAP, the efficient XML interchange (EXI)  is used, which encodes verbose encodes verbose XML XML documents documents in in binary binary format, format, if if interoperability interoperability is is considered. considered. This This is is normally normally used for used for constrained constrained devices devices to to increase increase thethe performance performance and and decrease decrease thethe consumed consumed power. power. Hence, Hence, CoAP is suitable for constrained devices in IoT-based wireless sensor CoAP is suitable for constrained devices in IoT-based wireless sensor networks that employ networks that employ IPv6-based infrastructure. IPv6-based However, itHowever, infrastructure. needs a gateway it needstoa exchange gateway to data over the exchange Internet. data over the Internet. (a) (b) (c) Figure 4. Communication protocols in the IEEE model (a); the HTTP (b); the CoAP (c). Figure 4. Communication protocols in the IEEE model (a); the HTTP (b); the CoAP (c).
Future Internet 2019, 11, 66 8 of 18 There are two common features and two main differences between MQTT and CoAP. Both target constrained devices and networks, and both have low data overhead. Importantly, MQTT is one-to-many and TCP-based message-oriented, whereas CoAP is one-to-one and UDP-based. Since TCP is connection-oriented, and UDP is connectionless, MQTT is more reliable. Moreover, CoAP needs a dedicated communication infrastructure based on IPv6 networks in addition to a dedicated gateway to pass content over the Internet. Hence, CoAP was not used for this IIoT environment. In the same manner, HTTP, which is polling-based, was not considered for this study because of its philosophy that uses the synchronous communication model for peer-to-peer and request–response exchange of data. XMPP is an XML-based messaging protocol that is able to transfer verbose messages, audio and video signals in chat conversations. In addition, it supports request–response as well as an event-based communication pattern. However, the XML-based verbose messages of XMPP increase the message size of SCADA-like applications, which have byte-oriented messages, and hence, cannot be used for IIoT efficiently. As a comparison, MODBUS has small-sized data units, with a maximum of 255 bytes, which are suitable for automation, telecontrol, and monitoring, whereas XMPP messages need more than 400 bytes of overhead. To summarize, many IoT protocols exist, and event-based protocols are of considerable interest for transferring data as notifications to complement the MODBUS TCP. This MODBUS protocol is polling-based, synchronous, request–response, and optimized for control and monitoring in industrial applications. It can establish an IIoT environment, either on a standalone basis or in conjunction with an event-based protocol to cover the publish–subscribe mechanism if needed. MQTT is able to complement the MODBUS TCP via its asynchronous model, event-based paradigm, and publish–subscribe pattern. Alternatively, HTTP uses a request–response mechanism and, hence, was not considered for this study. CoAP was also found to be not suitable for this scenario because it needs a specific infrastructure, and hence, a gateway to pass data over the Internet, which adds more costs and causes complications to the environment. In addition, XMPP was not considered here, because it is XML-based verbose protocol that requires a large overhead for small-sized industrial packets. Moreover, AMQP was eliminated because it is dedicated to the exchange of business messages between two entities. This protocol is used normally for application-to-application integration at the enterprise level, which is higher than the level of both MODBUS TCP and MQTT. Transport Layer Security (TLS) and its predecessor, Secure Sockets Layer (SSL), have been designed to provide security over a TCP/IP network. If HTTP uses SSL or TLS, it employs port 443. This is also applicable to MODBUS TCP and MQTT, which employ ports 802 and 8883, respectively. 5. Comparison between MODBUS TCP and MQTT In this section, a comparison between the MODBUS TCP and the MQTT protocol is conducted from three aspects. These aspects are the communication model of the protocol in the original IEEE model, the message exchange philosophy, and the required number of bytes for some message types that demonstrate the overhead of each type. However, the other protocols are not compared for the reasons listed above. As shown in Figure 5, the MQTT and the MODBUS TCP protocols are both in the same level in the IEEE model. While the MQTT protocol encodes the user data in UTF-8, the MODBUS TCP uses a byte-encoded frame format for the user data, which is intended for industrial applications. Figure 6 illustrates a comparison between MQTT philosophy and MODBUS philosophy, from the perspective of the exchange of messages. The request of the MODBUS query uses a TCP-based connection and employs a frame-format based on an application-layer message structure, which is optimized and dedicated for telecontrol and monitoring. The case is different with MQTT; while the first client (publisher) produces an event using four messages, the second client (subscriber) consumes that event in six messages.
Future Internet 2019, 11, 66 9 of 18 Future Internet 2019, 11, x FOR PEER REVIEW 9 of 18 Future Internet 2019, 11, x FOR PEER REVIEW 9 of 18 (a)(a) (b) (c)(c) Figure 5. The IEEE model (a); compared to the MODBUS TCP (b); and the MQTT (c). Figure Figure 5.5.The TheIEEE IEEEmodel model(a); (a);compared compared to the the MODBUS MODBUSTCP TCP(b); (b);and andthe MQTT the (c).(c). MQTT The publisher sends a connect packet (CONNECT), with username (un) and password (pwd) if TheThe publishersends publisher sendsaaconnect connectpacket packet (CONNECT), (CONNECT), with username (un) and andpassword (pwd) if if required, to the server (broker) trying to establish a TCPwith username connection. The (un) server password acknowledges(pwd) the required, required,with to to thethe server (broker) server (broker) trying trying to establish to the establish a TCP connection. a TCP connection. The server acknowledges the attempt a CONNACK packet, telling client (publisher) whetherThe server acknowledges the connection is successfullythe attemptwith attempt witha aCONNACK CONNACK packet, packet, telling telling the the client client (publisher) (publisher) whether whether thetheconnection connection is is established. Then, the client publishes the temperature value, for example, via a PUBLISH packet, successfullyestablished. successfully established. Then,Then, the the client client publishes publishes the the temperature temperature value, value, for forexample, example, viaviaa a with temp topic PUBLISH andwith packet, a value tempoftopic 22.7.and Theaclient valueends the of 22.7. publish The clienteventends with the server the publish bywith event sending the a PUBLISH packet, DISCONNECT with temp topic and a value of 22.7. The client ends the publish event with the packet. server by sending a DISCONNECT packet. server by sending Meanwhile, a DISCONNECT topacket. Meanwhile,and andinin addition addition to theaforementioned the aforementioned steps, steps, thethe subscriber subscriber mustmust SUBSCRIBE SUBSCRIBE to the to Meanwhile, thesame same and in to topic(temp) (temp) addition receiveto thepublished the aforementioned messages steps,ofthe subscriber interest. The must SUBSCRIBE subscription to the packet topic to receive the published messages of interest. The subscription packet is is same topic acknowledged (temp) to receive the published messages of interest. The subscription packet is acknowledged with the SUBACK packet. When the broker receives the message that handles thethe with the SUBACK packet. When the broker receives the message that handles acknowledged temperature with thethe SUBACKPUBLISHpacket. When the broker receives the message that handles the temperaturevalue, value,via via thesamesame PUBLISH packet,packet, itit forwards forwards itittotothe the subscriber, subscriber, which which may may then then temperature terminate terminatethe value, the via connection the connectionusing same PUBLISH usingthe theDISCONNECT packet, DISCONNECT message.it forwards message. it to the subscriber, which may then terminate the connection using the DISCONNECT message. (a) (b) Figure 6. Comparison (a)ofof protocols for the exchange of messages: (a) MQTT; (b)(b)MODBUS TCP. Figure 6. Comparison protocols for the exchange of messages: (a) MQTT; (b) MODBUS TCP. As a comparison Figure between MODBUS TCP andofMQTT messages, Figure 7 showsTCP. the required As a6.comparison Comparison of protocols between for the exchange MODBUS messages: TCP and MQTT (a) MQTT; messages, (b) MODBUS Figure 7 shows the required number of bytes of the MODBUS messages, compared to those of MQTT. The message of the MODBUS number of bytes of the MODBUS messages, compared to those of MQTT. The message of the TCP requires 27 bytes tobetween As a comparison read the MODBUS values of three TCP registers and of MQTTusing the function messages, code7“read Figure shows input the registers” required MODBUS TCP requires 27 bytes to read the values three registers using the function code “read (FC = number0x04), of as shown bytes of in theFigures MODBUS 1 and 2. However, messages, the message compared to of the those MODBUS of MQTT. TCP The input registers” (FC = 0x04), as shown in Figures 1 and 2. However, the message of the MODBUSrequires message 29ofbytes the MODBUS TCP requires 27 bytes to read the values of three registers using the function code “read input registers” (FC = 0x04), as shown in Figures 1 and 2. However, the message of the MODBUS
Future Internet 2019, 11, 66 10 of 18 Future Internet 2019, 11, x FOR PEER REVIEW 10 of 18 Future Internet 2019, 11, x FOR PEER REVIEW 10 of 18 to write the values of two registers using the function code “write multiple registers” (FC = 0x10) in a TCP requires TCP requires 2929 bytes bytes to to write write the the values values of of two two registers registers using using the the function function code code “write “write multiple multiple remote device. In both cases, the MODBUS TCP issues a query from the client side and a response registers” (FC registers” (FC == 0x10) 0x10) in in aa remote remote device. device. In In both both cases, cases, the the MODBUS MODBUS TCP TCP issues issues aa query query from from the the from theside client server and side. a Meanwhile, response from if server the the function side. code “read/write Meanwhile, if the multiple function coderegisters” “read/write(0x17) is used multiple client side and a response from the server side. Meanwhile, if the function code “read/write multiple forregisters” both operations, (0x17) is the total is used used consumed for both both bytes operations, thefor reading total consumedthreebytes registers and writing for reading reading two registers three registers registers and registers” (0x17) for operations, the total consumed bytes for three and within the writing same two connection registers within is 36 the bytes. same For MQTT, connection is in 36 this example, bytes. For publishing MQTT, in this a 10-byte example, payload by publishing writing two registers within the same connection is 36 bytes. For MQTT, in this example, publishing thea producer and consuming 10-byte payload payload by the it by the subscriber the producer producer requires and consuming consuming 115the it by by bytes. It is important subscriber requiresto115 mention 115 bytes.that isthe It is a 10-byte by and it the subscriber requires bytes. It total consumed important to bytes to mention of MQTT mention thatthat the publish the total and total consumedsubscribe consumed bytes messages bytes of of MQTT depends MQTT publish on publish and the topic and subscribe length messagesthe and subscribe messages important length of the depends depends onuser on the data.length the topic topic length and and the the length length of of the the user user data. data. Figure7.7. Figure Figure 7.Required Requiredbytes Required bytes of bytes of MQTT of MQTT messages MQTT messages compared messages compared totothose comparedto those ofofMODBUS. thoseof MODBUS. MODBUS. Figure Figure8 8shows Figure showsthe 8 shows theoverhead the overheadportion overhead portion of portion the three of the three messages three messagesfor messages foraa aconstant for constant constant payload payload payload of of of 10 10 10 bytes. bytes. bytes. The function The functioncode code“read/write “read/write multiple multiple registers” registers” (0x17) (0x17) has has an an overhead overhead ofof 2626 bytes bytes out The function code “read/write multiple registers” (0x17) has an overhead of 26 bytes out of 36 bytes. out of of 36 36 bytes. bytes. The The function function The codes functioncodes “read codes“read input “readinput registers” inputregisters” (0x04) registers” (0x04) and “write and “write (0x04) and multiple “writemultiple registers” multipleregisters” (0x10) registers”(0x10) (0x10)have have have a total a total a total overhead overhead ofof 46 46 bytes bytes out out ofof 56-bytes 56-bytes (the (the total total size). size). In In comparison, comparison, the the MQTT MQTT overhead of 46 bytes out of 56-bytes (the total size). In comparison, the MQTT publish–subscribepublish–subscribe publish–subscribe message message message has has an anan has overheadofof overhead overhead of105 105bytes 105 bytesout bytes outof out of 115 of 115 bytes. 115 bytes. bytes. Figure 8. Figure 8. MQTT MQTT and and MODBUS MODBUS overheads overheads for for a payload of of 10 bytes. Figure 8. MQTT and MODBUS overheads foraapayload payload of1010bytes. bytes.
Future Internet 2019, 11, x FOR PEER REVIEW 11 of 18 Future Internet 2019, 11, 66 11 of 18 The result of Figure 8 is illustrated in Figure 9 for values between 2 bytes and 106 bytes. The figure Thedemonstrates result of Figurethe 8 isoverhead illustratedpercentage in Figure 9with respect for values to the2total between bytesmessage size according and 106 bytes. The figureto Equation (3):the overhead percentage with respect to the total message size according to Equation (3): demonstrates Overhead Overhead(%) Overhead 100% . (3) Overhead(%) = Overhead Payload× 100%. (3) Overhead + Payload The percentage overheads of the aforementioned function codes are calculated and plotted in The9.percentage Figure The figureoverheads shows valuesof thefor aforementioned fewer than 106function codes are bytes because the calculated message size andinplotted in industrial Figure 9. The and applications figure shows values SCADA-like for fewer systems thanMQTT is small. 106 byteshas because the highestthe overhead; message size the in industrial function code applications “read inputand SCADA-like registers” systems (0x04) has is small. the least MQTT overhead (21has the highest bytes), and the overhead; function codethe “write function code multiple “read input (0x10), registers” registers” (0x04) which has the least is covered by theoverhead curve of(21 bytes), the andcode function the function codemultiple “read/write “write multiple registers registers” (0x17), has(0x10), which 25 bytes is coveredHowever, of overhead. by the curve of the if the functionclient application code reads “read/write multiple and writes registers registers in the (0x17), same has 25 bytes message, theof overhead. function codeHowever, “read/writeif the application multiple client(0x17) registers” reads shows and writes lowerregisters overheadin the than same message,codes the function the function “read inputcoderegisters” “read/write(0x04)multiple registers” and “write (0x17) multiple shows lower registers” (0x10)overhead if both arethan used the function for separately codes the“read same input registers” (0x04) and “write multiple registers” (0x10) if both are used purpose. separately for the same Accordingly, purpose. the MQTT protocol is suitable for IoT-based publish–subscribe applications, but it Accordingly, is unable the MQTT to replace protocol isTCP, the MODBUS suitable for IoT-based which fulfills thepublish–subscribe applications, industrial requirements and but it isan uses unable to replace optimized the MODBUS frame-format for TCP, which fulfills the request–response industrial requirements communications between and uses an clients industrial optimizedand frame-format servers. These forproperties request–response make it communications suitable for SCADA-likebetween systems, industrialautomation, clients and monitoring, servers. These and properties make it suitable for SCADA-like systems, automation, monitoring, control. In Section 6, performance of MQTT and the MODBUS TCP are tested and compared based and control. In Section 6, performance of MQTT on the Round-Trip and(RTT) time the MODBUS TCP are measurements tested and and compared the central processingbased on(CPU) unit the Round-Trip usage. time (RTT) measurements and the central processing unit (CPU) usage. Figure9.9.The Figure Theoverhead overheadofofananMQTT MQTTmessage messagecompared comparedtotothat thatofofMODBUS. MODBUS. 6. Latency and CPU Usage 6. Latency and CPU Usage In order to support the results of Section 5, the performance of MQTT and the MODBUS TCP are In order to support the results of Section 5, the performance of MQTT and the MODBUS TCP compared in this section. This performance is based on the Round-Trip Time (RTT) measurements and are compared in this section. This performance is based on the Round-Trip Time (RTT) the CPU usage. A Java-based MQTT publisher was developed and installed on a laptop that runs the measurements and the CPU usage. A Java-based MQTT publisher was developed and installed on a operating system Linux Ubuntu 16.04. The specifications of the laptop were: Dell-Inspiron 3537, Intel® laptop that runs the operating system Linux Ubuntu 16.04. The specifications of the laptop were: CoreTM i5-4200U CPU at 1.6 GHz × 4, 5.85 GiB RAM. The server (broker) was the Apache ActiveMQ Dell-Inspiron 3537, Intel® CoreTM i5-4200U CPU at 1.6 GHz × 4, 5.85 GiB RAM. The server (broker) software, which was installed on a desktop PC that runs the operating system Microsoft Windows XP. was the Apache ActiveMQ software, which was installed on a desktop PC that runs the operating The desktop PC featured Pentium Dual Core CPU at 3.2 GHz and 2 GB RAM. The consumer was the system Microsoft Windows XP. The desktop PC featured Pentium Dual Core CPU at 3.2 GHz and 2 MQTT.FX software, which was installed on the desktop PC as well. The RTT values measured from the GB RAM. The consumer was the MQTT.FX software, which was installed on the desktop PC as well.
Future Future Internet Internet 2019, 2019, 11, 11, x66FOR PEER REVIEW 1212of of 18 18 The RTT values measured from the publisher and the CPU usage measured from the desktop PC are publisher shown and the10. in Figure CPU usage The measured publisher that from the desktop connected PC are (broker) to the server shown instarted Figurecounting 10. The publisher the RTT that connected to the server (broker) started counting the RTT from the time of creating the from the time of creating the socket to closing it, which includes the time needed for formatting the socket to closing it,in message which includes a publish the timeconnecting message, needed for to formatting the serverthewithout messageainlogon a publish message, process, connecting publishing the to the server message, andwithout a logonfrom disconnecting process, publishing the server, the message, in addition to theand disconnecting network fromCPU latency. The the server, usage in addition values to the are the networkregistered maximum latency. The CPU usage values, and thevalues tests are weretheconducted maximumforregistered payloadvalues, and sizes of 5, the 10, tests 25, 50,were and conducted 100 bytes. for payload sizes of 5, 10, 25, 50, and 100 bytes. (a) (b) Figure 10. MQTT-related measurements: (a) Round-Trip Time (RTT); (b) CPU usage. Figure 10. MQTT-related measurements: (a) Round-Trip Time (RTT); (b) CPU usage. The RTT measurements of the MODBUS TCP were also conducted using the same network. The RTT measurements of the MODBUS TCP were also conducted using the same network. A A MODBUS TCP client was developed and installed on the laptop, and a MODBUS TCP server was MODBUS TCP client was developed and installed on the laptop, and a MODBUS TCP server was developed and installed on the desktop PC. Many tests of function codes were carried out for “read developed and installed on the desktop PC. Many tests of function codes were carried out for “read holding registers” (FC = 0x03), “read input registers” (FC = 0x04), and “write multiple registers” holding registers” (FC = 0x03), “read input registers” (FC = 0x04), and “write multiple registers” (FC (FC = 0x10). The total transmitted bytes in each transaction were 27 bytes, 33 bytes, and 35 bytes = 0x10). The total transmitted bytes in each transaction were 27 bytes, 33 bytes, and 35 bytes respectively; the total was 95 bytes when all messages were transmitted within one transaction. respectively; the total was 95 bytes when all messages were transmitted within one transaction. The The RTT values, which were measured on the client side, were between 6 ms and 8 ms, and the CPU RTT values, which were measured on the client side, were between 6 ms and 8 ms, and the CPU usage, which was measured on the server side, had a maximum value of 3%. usage, which was measured on the server side, had a maximum value of 3%. These results confirm the theoretical outcomes of Section 5, which show that the MODBUS TCP is These results confirm the theoretical outcomes of Section 5, which show that the MODBUS TCP a lightweight protocol suitable for SCADA-like systems and industrial applications. The MODBUS is a lightweight protocol suitable for SCADA-like systems and industrial applications. The MODBUS TCP client connects to the MODBUS TCP server using polling communications (direct one-to-one TCP client connects to the MODBUS TCP server using polling communications (direct one-to-one communications). However, the MQTT client connects to another client via a server, called a broker. communications). However, the MQTT client connects to another client via a server, called a broker. This fact explains the high CPU consumption while the MQTT server was busy in receiving and This fact explains the high CPU consumption while the MQTT server was busy in receiving and redirecting the publish messages. redirecting the publish messages. 7. Discussion 7. Discussion Based on the comparisons of Sections 4–6 as well as the theoretical study of the MODBUS TCP in Based Section 3, twoon the comparisons scenarios of Sections are discussed here 4–6 as well to build theas the theoretical industrial study of the MODBUS TCP IoT environment. in Section 3, two The first scenarios scenario is toare discussed employ hereMODBUS only the to build the TCPindustrial to build theIoTIIoT environment. environment. The study The first scenario is to employ only the MODBUS TCP to build of Section 3 proved that MODBUS TCP is able to react as an IoT protocol. Security the IIoT environment. The study is a fundamental of Section issue 3 proved in IoT, thatsolved which was MODBUS TCP in  forisMODBUS able to react TCPasusing an IoT protocol. TLS. Securityabove As mentioned is a fundamental in Section 5, issue in IoT, which was solved in  for MODBUS TCP using TLS. As the MODBUS TCP is able to build the IIoT environment using the synchronous request–response mentioned above in Section 5, the MODBUS pattern communication TCP is able on a to build the basis, standalone IIoT environment using the which is a scenario thatsynchronous complies with request–response most industrial communication pattern on a standalone basis, which is a scenario applications. However, this solution totally relies on the polling-based mechanism. that complies with most industrial applications. However, this solution totally relies on the polling-based In the second scenario, the event-based mechanism is fulfilled by MQTT, where M2M mechanism. In the second communications scenario, using are required, the event-based the asynchronous mechanism is fulfilledcommunication publish–subscribe by MQTT, where M2M pattern. For communications industrial functions,are the required, MODBUS using TCP the asynchronous is employed, which publish–subscribe communication fulfills the synchronous pattern. request–response For industrial functions, communication pattern. In the this MODBUS scenario, MQTT TCP works is employed, in parallel which fulfills with the MODBUS the synchronous TCP within request–response communication pattern. In this scenario, MQTT the same platform, as shown in Figure 11. This figure illustrates the simulation environmentworks in parallel withof the the MODBUS TCP within the same platform, as shown in Figure 11. This second scenario. The environment consisted of a desktop PC running Microsoft Windows XP, a laptop figure illustrates the simulation running Linux environment Ubuntu 16.04,of the and secondan scenario. LAN/InternetThe environment network. The consisted desktopofPC a desktop PC running is equipped with a Microsoft Windows XP, a laptop running Linux Ubuntu 16.04, and an LAN/Internet network. The
Future Internet Future Future Internet 2019, Internet 2019,11, 2019, 11,66 11, xx FOR FOR PEER PEER REVIEW REVIEW 13 13 of 18 13 of 18 18 desktop PC desktop PC is is equipped equipped with with aa java-based java-based MODBUS MODBUS TCP TCP client, client, the the ActiveMQ ActiveMQ MQTT MQTT server, server, and and java-based the MQTT.FXMODBUS MQTT.FX consumer. TCP client, consumer. The the The laptop ActiveMQ laptop was MQTT was equipped server, equipped with and the MQTT.FX with aa multithreaded multithreaded Java consumer. Java program; The program; one laptop one thread thread the was ran equipped the MQTT with a multithreaded publisher, and Java another program; thread ran oneMODBUS the thread ranTCPthe MQTT server.publisher, In andananother addition, online ran the MQTT publisher, and another thread ran the MODBUS TCP server. In addition, an online thread ran MQTT brokerthe MODBUS broker and TCP and consumer, server. consumer, providedIn addition, provided by an online by HiveMQ HiveMQ for MQTT broker for testing, and testing, were consumer, were employed provided employed here here as by as an an MQTT HiveMQ for testing, Internet-based were server. employed here as an Internet-based server. Internet-based server. Figure 11. Figure Figure 11. Simulation environment 11. environment of of the the second second scenario. scenario. The The RTT The RTTmeasurements RTT measurements measurements andand andthe the CPUCPU the usageusage CPU of thisof usage ofenvironment were measured this environment this environment for the MODBUS were measured were measured for the for the TCP MODBUS MODBUS and then TCPcompared TCP and thentocompared and then the previous compared to results to the the of Section previous previous 6. The results results RTT values of Section of Section 6. The 6. of the The RTTMODBUS RTT values of values ofTCP, the the which MODBUS were measured TCP, which on the were desktop measured PC, had on thea maximum desktop value PC, had MODBUS TCP, which were measured on the desktop PC, had a maximum value of 9 ms. This ofa9 ms. This maximum represents value of an 9 increase ms. This of maximum represents represents an12.5% an from increase increase of the of previous maximum maximum RTT from 12.5% 12.5% values from presented the the previousinRTT previous Section RTT 6. As values values illustrated presented presented in Figure in Section in Section 12, 6. As 6. As the CPU history of the laptop, which contains an MQTT publisher and a MODBUS illustrated in Figure 12, the CPU history of the laptop, which contains an MQTT publisher and aa illustrated in Figure 12, the CPU history of the laptop, which contains an MQTTTCP server, publisher showed and that MODBUS MODBUSthe CPU TCP TCPusage is always server, server, showedless showed thatthan that the 20%. the CPU Theseis CPU usage usage isresults always always indicate that less than less than the These 20%. 20%. concurrent These results results execution indicate indicate of that thatMQTT the in parallelexecution the concurrent concurrent with the of execution MODBUS of MQTT MQTT in inTCP within parallel parallel thethe with with same the platform MODBUS MODBUS does TCP TCP not severely within within sameinfluence the same the platform platform the does performance not of severely the MODBUS influence the TCP. Nevertheless, performance of while the MQTT MODBUS does not severely influence the performance of the MODBUS TCP. Nevertheless, while MQTT fulfills TCP. the event-based Nevertheless, paradigm, while MQTT it lacks fulfills interoperability. the event-based paradigm, it lacks interoperability. fulfills the event-based paradigm, it lacks interoperability. Figure 12. Figure 12. CPU CPU history history of of the the laptop laptop that that ran ran an an MQTT MQTT publisher publisher and and MODBUS MODBUS TCP TCP server. server. server. The MQTT The MQTT protocol protocol formats formats thethe data data using using the the UTF-8 UTF-8 standard, standard, and and hence hence interoperability interoperability is is not fulfilled. not fulfilled. To To maintain maintain a homogeneous homogeneous message for the the entire entire environment, environment, industrial industrial datadata are are organized using organized using the the structure structure of of MODBUS MODBUS messages, messages, formatted formatted in in the the UTF-8 UTF-8 andand then then transferred transferred in in the payload the payload of of an an MQTT MQTT publish publish message, message, as as shown shown inin Figure Figure 13a. 13a. The hexadecimal representations representations of Figure of Figure 13a13a were were obtained obtained basedbased onon the the information information of information of Table Table 3. Table 3. Digit 3. Digit numbers numbers zero zero toto nine nine are are formatted in formatted in UTF-8 UTF-8 using using thethe hexadecimal hexadecimal representation. Digit number representation. Digit number number “0”“0” is represented “0” is represented represented by by “30” “30” and digit and digit number number “9”“9” is is represented represented by by “39”. “39”. Additionally, Additionally, letters letters “A” represented by “A” to “F” are represented “41”to “41” “41” to“46”, to “46”,respectively. “46”, respectively.Each respectively. Eachbyte Each byteof byte ofthe of theMODBUS the MODBUS message MODBUS message isis represented represented by by two two bytes, bytes, asas shown shown in Figure in Figure 13. 13. For For example, Forexample, example,the the ID theID part IDpart ofofthe partof the MODBUS theMODBUS message MODBUSmessage message is is is “00 “00 01”, “00 which 01”, 01”, which which leads leads leads to “30 to “30“30 to 30 30 30 30 31” 30 representation, 31” representation, andand doubles doubles thethepayload payload of ofthe the MQTT MQTT 31” representation, and doubles the payload of the MQTT publish message. The publish publish message. message. The MQTT publish publish message was message was subsequently subsequently transmitted transmitted to to the the online online hivemq.com hivemq.com server, server, and and the the result result isis illustrated illustrated in Figure in Figure 13b. 13b.
You can also read
NEXT SLIDES ... Cancel