UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

An industrial internet architecture based on SDN and fog computing He, Junxiong 2018

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata

Download

Media
24-ubc_2018_february_he_junxiong.pdf [ 3.76MB ]
Metadata
JSON: 24-1.0363154.json
JSON-LD: 24-1.0363154-ld.json
RDF/XML (Pretty): 24-1.0363154-rdf.xml
RDF/JSON: 24-1.0363154-rdf.json
Turtle: 24-1.0363154-turtle.txt
N-Triples: 24-1.0363154-rdf-ntriples.txt
Original Record: 24-1.0363154-source.json
Full Text
24-1.0363154-fulltext.txt
Citation
24-1.0363154.ris

Full Text

AN INDUSTRIAL INTERNETARCHITECTURE BASED ON SDNAND FOG COMPUTINGbyJunxiong HeMSc., The University of York, United Kingdom, 2014A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFMASTER OF APPLIED SCIENCEinTHE COLLEGE OF GRADUATE STUDIES(Electrical Engineering)THE UNIVERSITY OF BRITISH COLUMBIA(Okanagan)January 2018c© Junxiong He, 2018The following individuals certify that they have read, and recommend to the College of GraduateStudies for acceptance, a thesis/dissertation entitled: AN INDUSTRIAL INTERNET ARCHI-TECTURE BASED ON SDN AND FOG COMPUTING submitted by Junxiong He inpartial fulfillment of the requirements of the degree of Master of Applied ScienceDr. Julian Cheng, School of EngineeringSupervisorDr. Chen Feng, School of EngineeringSupervisory Committee MemberDr. Yang Cao, School of EngineeringSupervisory Committee MemberDr. Zheng Liu, School of EngineeringUniversity ExamineriiAbstractIn traditional industrial networks, conventional distributed switches adopt traditional optimiza-tion algorithms to equally share bandwidth among contending flows, such as transmission controlprotocol (TCP) congestion control algorithm. However, a large number of machines, goods, con-trol systems and information systems will be interconnected in industrial Internet. As a result,when an emergency occurs, traffic will increase considerably and network congestion will occur.In the worst-case scenario, the machines may not function properly (e.g. some robot-arms cannotrespond fast enough to complete the intended tasks). In this thesis, based on the features of soft-ware defined network (SDN), network status information of each SDN switch is extracted by anSDN controller through a centralized control architecture. The information is also processed by afog computing server through network utility maximization fabric (NUMFabric) using the weightmax-min fairness algorithm. Our testbed results show that when an emergency occurs, the newarchitecture can reduce the bandwidth allocation recovery time by more than 350 times that of thetraditional TCP. Moreover, by identifying the requirement of industrial Internet and the features ofInternet of Things (IoT), an integrated architecture can be designed based on SDN, fog computing,cloud computing, object linking and embedding for process control unified architecture (OPC UA)standard. This thesis includes key aspects of analysis, design and implementation for industrialInternet architecture with short recovery time. It is demonstrated that network entities such as thefog computing server, the SDN Ryu controller, SDN OpenFlow switches and optical sensors canfunction together. The centralized parameter acquiring method for NUMFabric using the weightmax-min fairness algorithm can also perform bandwidth allocation through OPC UA standard andOpenFlow protocol on our testbed.iiLay SummaryA novel centralized parameter acquiring method for NUMFabric using the weight max-minfairness algorithm based on OPC UA TCP/IP was studied. The proposed method solves theproblem of slow bandwidth allocation recovery in industrial Internet. The new architecture usingSDN and fog computing to obtain the applied algorithm parameters and proved to be effective inreducing bandwidth allocation recovery time. Based on SDN, fog computing, cloud computing, IoTdevices and OPC UA TCP/IP, we proposed a detailed architecture to realize industrial Internet.Testbed analysis shows that the proposed method effectively can reduce bandwidth allocationrecovery time, which makes centralized parameter acquiring method for NUMFabric using theweight max-min fairness algorithm more applicable in industrial Internet.iiiTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiiAcknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiChapter 1: Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Background and Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2.1 Intelligent Industrial Internet . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2.2 SDN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61.2.3 Fog Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71.3 Contributions and Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9Chapter 2: Industrial Internet with New Technology and Architecture . . . . . . 112.1 Industrial Internet with SDN and Centralized Architecture . . . . . . . . . . . . . . 112.2 Industrial Internet with Fog Computing . . . . . . . . . . . . . . . . . . . . . . . . . 132.3 Overall Architecture Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152.4 Protocol and Standard Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172.4.1 OpenFlow Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17ivTABLE OF CONTENTS2.4.2 OPC UA Standard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18Chapter 3: Fast Convergent Bandwidth Allocation . . . . . . . . . . . . . . . . . . 193.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193.1.1 NUMFabric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193.1.2 Weight Max-Min Fairness Algorithm . . . . . . . . . . . . . . . . . . . . . . . 203.2 The Design of Fast Convergent Bandwidth Allocation . . . . . . . . . . . . . . . . . 223.2.1 Obtaining the Flow Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223.2.2 The Design of Optimal Algorithm . . . . . . . . . . . . . . . . . . . . . . . . 233.2.3 The SDN Packet Process Pipeline . . . . . . . . . . . . . . . . . . . . . . . . 243.2.4 The Design of Meter-QoS Rule . . . . . . . . . . . . . . . . . . . . . . . . . . 243.3 The Design of OPC Server to OPC Client Transmission Process . . . . . . . . . . . . 253.3.1 The Design of Sensor Collector . . . . . . . . . . . . . . . . . . . . . . . . . . 253.3.2 The Design of OPC Server Flow Chart . . . . . . . . . . . . . . . . . . . . . . 263.3.3 The Design of OPC Client Flow Chart . . . . . . . . . . . . . . . . . . . . . . 273.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28Chapter 4: Testbed Implementation and Testing . . . . . . . . . . . . . . . . . . . . 294.1 Testbed Topology Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294.2 Testbed Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314.3 The Connectivity of the Network Equipments . . . . . . . . . . . . . . . . . . . . . . 324.4 The OPC UA TCP Data Packet Analysis . . . . . . . . . . . . . . . . . . . . . . . . 334.5 The Emergency Bandwidth Allocation Testing . . . . . . . . . . . . . . . . . . . . . 354.6 Testbed Numerical Results and Discussions . . . . . . . . . . . . . . . . . . . . . . . 364.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39Chapter 5: Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415.1 Summary of Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42vTABLE OF CONTENTSBibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43viList of TablesTable 1.1 Comparison of the different intelligent industrial Internet among of America,Germany and China . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5Table 2.1 Comparison of Fog Computing and Cloud Computing in Industrial Internet . 15Table 3.1 The First-step Bandwidth Allocation . . . . . . . . . . . . . . . . . . . . . . . 21Table 3.2 The Second-step Bandwidth Allocation . . . . . . . . . . . . . . . . . . . . . . 21Table 3.3 The Third-step Bandwidth Allocation . . . . . . . . . . . . . . . . . . . . . . 21Table 3.4 The Fourth-step Bandwidth Allocation . . . . . . . . . . . . . . . . . . . . . . 22Table 4.1 The Names and Number of Testbed Equipments . . . . . . . . . . . . . . . . 30Table 4.2 Network Equipment Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . 31viiList of FiguresFigure 1.1 Four Development Stages of German Industry 4.0 . . . . . . . . . . . . . . . 6Figure 1.2 SDN Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6Figure 1.3 Cloud-Fog-Device Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8Figure 2.1 Industrial Internet Architecture . . . . . . . . . . . . . . . . . . . . . . . . . 15Figure 3.1 The Fast Convergent Bandwidth Allocation Flow Chart . . . . . . . . . . . . 23Figure 3.2 The Flow Rate Deriving Flow Chart . . . . . . . . . . . . . . . . . . . . . . . 23Figure 3.3 The Algorithm Flow Chart . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24Figure 3.4 SDN Packet Process Pipeline Flow Chart . . . . . . . . . . . . . . . . . . . . 24Figure 3.5 Fog Computing Flow Chart . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25Figure 3.6 The Transmission Process Flow Chart . . . . . . . . . . . . . . . . . . . . . . 25Figure 3.7 Sensor Collector Flow Chart . . . . . . . . . . . . . . . . . . . . . . . . . . . 25Figure 3.8 OPC Server Flow Chart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26Figure 3.9 OPC Client Flow Chart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27Figure 4.1 The Testbed Topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30Figure 4.2 The Connection of Overall Network Equipments . . . . . . . . . . . . . . . . 32Figure 4.3 The Connection of Network Equipment Results . . . . . . . . . . . . . . . . 32Figure 4.4 The Data Packet of TCP and OPC UA . . . . . . . . . . . . . . . . . . . . . 33Figure 4.5 The Process of OPC Server to Client . . . . . . . . . . . . . . . . . . . . . . 34Figure 4.6 The Sensor Data in OPC UA TCP Packet . . . . . . . . . . . . . . . . . . . 34Figure 4.7 The Alert Message of Node 0 . . . . . . . . . . . . . . . . . . . . . . . . . . . 35Figure 4.8 The New Bandwidth Allocation . . . . . . . . . . . . . . . . . . . . . . . . . 36viiiLIST OF FIGURESFigure 4.9 Recovery Time Comparison Between the Traditional TCP and the ProposedArchitecture: the Results of 100 Test . . . . . . . . . . . . . . . . . . . . . . 37Figure 4.10 Recovery Time Comparison Between the Traditional TCP and the ProposedArchitecture: the Results of One Test . . . . . . . . . . . . . . . . . . . . . . 38ixLIST OF FIGURESAcronyms DefinitionsAPI Application Programming InterfaceADC Analog-to-Digital ConverterBGP Border Gateway ProtocolCPS Cyber-Physical SystemsDPID OpenFlow Datapath IdentifierGPIO General Purpose Input OutputICT Information Communications TechnologyID IdentificationIoT Internet of Things (IoT)IP Internet ProtocolIPv4 Internet Protocol Version 4IPv6 Internet Protocol Version 6MaaS Machine-as-a-ServiceM2M Machines to MachinesNAT Network Address TranslatorsNUMFabric Network Utility Maximization FabricONF Open Networking FoundationOPC-UA Object Linking and Embedding for Process Control Unified ArchitectureOSPF Open Shortest Path FirstPWM Pulse Width ModulationQoS Quality of ServiceRTT Round-Trip TimexLIST OF FIGURESSDN Software Defined NetworkSIGINT Signal InterruptTCP Transmission Control ProtocolT2T Things to Things5G Fifth GenerationxiAcknowledgmentsI am deeply grateful to my thesis supervisor Dr. Julian Cheng for his enthusiasm, guidance,advice, encouragement, support, and friendship. I will continue to be influenced by their rigorousscholarship, clarity in thinking, and professional integrity.I would like to thank Dr. Zheng Liu for his willingness to serve as my university examiner.I would also like to thank Dr. Chen Feng and Dr. Yang Cao for their willingness to serve onthe committee. I really appreciate their valuable time and constructive comments on my thesis.I would also like to express my thanks to Dr. Chen Feng and Mr. Junyuan Leng from TheUniversity of British Columbia Okanagan campus for their feedback, constructive comments, andvaluable suggestions on my research work.I owe many people for their generosity and support during my study at The University of BritishColumbia. I would like to thank my dear colleagues for sharing their academic experiences andconstructive viewpoints generously with me during our discussions. I would also like to thank mydear friends for sharing in my excitement and encouraging me when I was in frustration during thisjourney.Finally, I would like to thank my parents for their patience, understanding, support, and loveover all these years. All my achievements would not have been possible without their constantencouragement and support.xiiChapter 1Introduction1.1 Background and MotivationWith the rapid development of microprocessor technology for the past 40 years and popular useof Internet in the past 20 years [1, 2], the industrial Internet, which is based on microprocessor andInternet, was first proposed by the General Electric Company in 2012. The industrial Internet isdefined as a network interconnected of machines, goods, control systems, information systems andpeople [3]. It has two important functions: data analysis and device connection. On the one hand,the data analysis is divided into two levels: primary analysis level and advanced analysis level. Theprimary analysis refers to the industrial production analytics. For example, the primary analysiscan perform a timely analysis and monitor for manufacturing lines. The advanced analysis refersto the predictive analytics. For example, the advanced analysis can forecast the developing trend ofthe market based on Big data [4]. On the other hand, the device connection means that all kinds ofdevices can connect to the objectives, and it relies on the traditional industrial network transmissionprotocols such as Ethernet/industrial protocol and transmission control protocol/Internet protocol(TCP/IP).The Ethernet/industrial protocol uses the Ethernet infrastructure to manage the connectionamong various automation devices such as robots, sensors and other industrial machines. TheTCP/IP is responsible for routing the data to a specific IP address [5]. In traditional industrialnetworks, although both Ethernet/industrial protocol and TCP/IP can offer data packets trans-mission, TCP/IP is more suitable for industrial Internet. The reason is that the industrial Internetneeds to offer the fast service for customers through connecting a large number of devices bothin interior and exterior networks [6]. Correspondingly, TCP/IP can connect all network devicesto Internet, while Ethernet/industrial protocol cannot enable the exterior network devices to com-11.1. Background and Motivationmunicate with interior network devices. For instance, object linking and embedding for processcontrol unified architecture (OPC UA) can be based on TCP/IP to transmit data packets in thetraditional industrial network and the industrial Internet.The OPC UA TCP/IP is an industrial communication standard. It can quickly achieve au-tomatic communication among various devices. Compared with Ethernet/industrial protocol, theobvious benefit of OPC UA TCP/IP is its compatibility. For example, sensors, machines, networkdevices, servers and computers can all utilize the OPC UA TCP/IP to transmit data packets. Inaddition, using the OPC UA TCP/IP in industrial Internet, thousands of industrial devices canhave better access to Internet, and the production efficiency can be improved [7].However, the problem of OPC UA TCP/IP based on the traditional industrial distributionnetwork is that it cannot obtain the global network information. Correspondingly, the bandwidthallocation recovery time for congestion control can be long. Conventional switches use the tra-ditional TCP optimization algorithm, such as TCP congestion control algorithms, to decide howto allocate bandwidth. The TCP congestion control algorithms are reactive bandwidth allocationalgorithms. The reactive bandwidth allocation means that when network congestion occurs, theavailable bandwidth will reduce to half of the original maximum bandwidth [8] and gradually in-crease for each user. Nevertheless, the drawback of the reactive bandwidth is that the bandwidthallocation is not fast enough [9]. In the worst-case scenario, the machines may not function properly.For example, some robot-arms cannot respond fast enough to complete their intended tasks.In order to address the above-mentioned issues of reactive bandwidth allocation, TCP needsproactive bandwidth allocation in the industrial Internet, which means that different bandwidthsshould be allocated for different objectives. Among the prior arts, the NUMFabric has been studiedin [9], and it allows per-flow resource allocation to be expressed using utility functions [9]. At thesame time, it adopts the weight max-min fairness algorithm to allocate bandwidth for each user[10]. Due to the fact that the NUMFabric is deployed on conventional switches and is based on adistributed network architecture, the NUMFabric’s global parameters cannot be directly obtained,and the bandwidth allocation recovery time will become long.Motivated by these considerations, this thesis builds upon the previous work on NUMFabricusing the weight max-min fairness algorithm, and deploys OPC UA TCP/IP on the centralized21.2. Related Workcontrol network architecture. Moreover, fog computing and software defined network (SDN) arealso used in this thesis. Fog computing is defined as an open platform for computing and storingapplications at data source devices. Using a fog computing server, an algorithm can be quicklydeployed to calculate bandwidth allocation. The SDN is defined as a network in which the networkcontrol plane is separated from the network forwarding plane. An SDN controller can centrallycontrol multiple SDN switches [7]. Using SDN, the centralized control architecture easily replacesthe traditional one in NUMFabric to obtain the flow rate fast, and the flow rate acquisition speedwill directly affect the bandwidth allocation recovery time. To this end, we propose an industrialInternet characterized by fast bandwidth allocation. To the best of our knowledge, this is the firstwork that deploys SDN and fog computing to address the issue of reducing bandwidth allocationrecovery time in industrial Internet.In this thesis, the following important questions will be answered: Q1) What kind of networkarchitecture can achieve fast bandwidth allocation recovery for industrial Internet? Q2) How canSDN and fog computing be used to realize bandwidth allocation based on OPC UA TCP/IP?Q3) How does the industrial Internet architecture with short bandwidth allocation recovery timeperform on a testbed?The answers to the above questions will be explored by both design description and testbednumerical results. On the experimental platform, when traffic at some nodes rapidly increases,the bandwidth allocation recovery time will be compared between the traditional TCP and theproposed architecture.1.2 Related WorkThe state-of-arts of the intelligent industrial Internet, SDN and fog computing will be describedin the following subsections.1.2.1 Intelligent Industrial InternetThe intelligent industrial Internet will provide intelligent manufacturing information sensing,transmission, analysis, feedback and control support. A large number of industrial internal datapackets will be exchanged through connected devices and server platforms. Using Internet of things31.2. Related Work(IoT) technologies, the manufacturing line will be filled with a large number of sensors to performintelligent production. Various information sensing devices can timely acquire information for anymonitoring, connection and interactive objects.To easily control all kinds of objects, IoT technologies can establish the network connection ofobjects to objects and objects to people. The traditional industrial production uses the machinesto machines (M2M) mode communication. However, the intelligent industrial Internet performsthings to things (T2T) mode communication to provide intelligent control and analysis. In an intel-ligent industrial Internet, the intelligent devices, systems and people will be connected intelligently,interactively and seamlessly [11, 12].To achieve the production efficiency optimization, intelligent management and quality monitor-ing, the intelligent industrial Internet will use server platforms to monitor and control the wholeproduction processes. At the same time, intelligent industrial Internet will seamlessly cooperatewith different factories through exterior data networks [13–15].The intelligent industrial Internet can connect to factories, suppliers, products and customers.Ultimately, a deep integration of computing, control and communication networks will be formed[3]. For the intelligent industrial Internet, different countries have established their own standardssuch as American Industrial Internet, German Industry 4.0 and Made in China 2025. Table 1.1compares the differences these standards [16–21].The American Industrial Internet was proposed by the General Electric Company in 2012. Theobjective is to use information data chain to perform intelligent decisions and executions throughoutproduct lifecycle on the industrial systems. During the whole processes of design, manufacturingand service, intelligent industrial Internet will break the boundaries between industrial interior andexterior networks. The industrial Internet can optimize both the industrial assets and operation,and create industrial service values. The industrial Internet will promote interactive intellectualcapabilities between M2M and machine-to-control platforms. The industrial Internet is able tohave timely connections using software. Ultimately, an open and intelligent industrial system canbe formed [22, 23].The German Industry 4.0 standard, was originated in the Germany Hannover Industrial Fairin 2011, and has been used as the German national integrated strategy. German Industry 4.0 will41.2. Related WorkTable 1.1: Comparison of the different intelligent industrial Internet among of America, Germanyand ChinaProgram Industrial Internet(American)Industry 4.0(German)Made in China 2025(China)ProductIntelligenceProduct intelligenceInterconnection Person-to-person,person-to-machine,machine-to-machineand machine-to-business interconnec-tionCyber-PhysicalSystems (vertical, hor-izontal and end-to-endintegration)Integration of thecompositive andspecial Internet ofthingsManufacturingIntelligenceIntelligent manufac-turing, intelligentfactory and intelligentmanufacturing systemSpecial IntelligentmanufacturingBusinessIntelligenceThe complex manage-ment of the wholevalue chainCollaborative innova-tion in supply chainServiceFunctionCustomer service’sLifecycle ManagementUsing new services tocreate valueTransition tomanufacturing ser-vicesBig DataAnalysisEmphasis on big dataanalysisOptimize decisionwith big data supportIntelligence decisionAdvancedManufacturing3D print New material and newprocesseliminate information asymmetry, and will also provide data packets transmission capabilities toindustrial operaters [22, 24]. Fig. 1.1 shows the four development stages of German industry 4.0.Encouraged by the advantages of traditional technologies and new information technologies,German Industry 4.0 will create a Cyber-Physical System (CPS) including resources, information,physical objects and people, to perform intelligent manufacturing [25].The Made in China 2025 was proposed by the Chinese government in 2015. It is based oninformation data chains, and the core is product lifecycle and intelligent manufacturing sectors.The objective is to shorten the research and development cycles, improve production efficiencyand product quality, reduce energy consumption and optimizing the use of space. Made in China2025 will solve some problems such as weak industrial innovation ability, low degree of intelligence,serious pollution and massive consumption of resources and energy in manufacturing processes [26].51.2. Related WorkFigure 1.1: Four Development Stages of German Industry 4.01.2.2 SDNThe Open Networking Foundation (ONF) released the white paper “Software-Defined Network-ing: The New Norm for Networks” in April 2012. SDN is defined as follows: In SDN architecture,the control and data planes are decoupled, network intelligence and state are logically centralized,and the underlying network infrastructure is abstracted from the applications [27].Figure 1.2: SDN ArchitectureFig. 1.2 shows the SDN architecture, which is described by the ONF. It is a three-layer archi-tecture including infrastructure layer, control layer, and application layer.In the infrastructure layer, a large number of data switching devices are deployed, e.g., Open-Flow switches. These data switching devices have two functions. The first one is collecting networkstatus in local devices. Then, send these information to the controller. The second function is to61.2. Related Workprocess the data packets based on the specified rules from the SDN controller [27].In the control layer, there are two important application programming interfaces (API) toconnect the application layer and infrastructure layer, namely north-bound API and south-boundAPI. The north-bound API is responsible for providing different service access points. It can providesome status information reports of the access network to SDN applications. According to theseinformation, the system will make a specific rule to forward data packets to the data switchingdevices through the north-bound API. The south-bound API is responsible for providing mutualinformation among different data switching devices of infrastructure layer [27–29].In the application layer, SDN applications fulfill a large number of user requirements. Besides,SDN applications manage the data switching devices in infrastructure layer. Furthermore, SDNapplications also support some functions such as dynamic access control, network virtualization,server load balancing and seamless mobility/migration [27].1.2.3 Fog ComputingFog computing is also called edge computing. It is an open platform for computing and storingapplications on the side of data source devices. It provides the key requirements for intelligentservices, intelligent businesses, data aggregation, interoperability, security and privacy protection.Figure 1.3 shows that the fog computing layer is deployed between cloud computing layer anddevice layer. Usually, for the execution of the functions of fog computing, the devices on this layerwill be deployed closely to the place of the data source devices, namely the device layer. By thismeans, the network latency and cost of network bandwidth can be reduced, the security concernscan be addressed, the service levels can be improved. The fog computing layer will periodicallysend the aggregated data packets from the device layer to the cloud layer. After the devices of thecloud computing layer finish computing and analyzing these aggregated data, they will send newapplication rules to the fog computing layer and the device layer [4, 30].The functions of this three-layer architecture are listed as follows:− In the device layer, thousands of intelligent network devices will generate Gigabytes of trafficto meet industrial production needs;− In the fog computing layer, fog computing is used for real-time and short-period data analysis.71.2. Related WorkFigure 1.3: Cloud-Fog-Device LayerThe original data generated in the devices layer will be processed by fog computing layerfirstly. Then, only the processing results need to be sent to the cloud computing layer forfurther processing. In this fashion, not only the network transmission delay can be reduced,but also the demand for bandwidth resources can be effectively reduced in the process ofdata transmission. By this means, the real-time intelligent processing of local services can bebetter supported;− In the cloud computing layer, cloud computing provides powerful computing and storageservices. These services not only support large data analysis in non-real-time and long-periodfashion, but also play an important role in periodical maintenances and business decision.The important applications contain enterprise big data processing and artificial intelligenceapplications.81.3. Contributions and Organization1.3 Contributions and OrganizationThe contributions of this thesis are listed as follows:− We propose a detailed architecture for realizing the requirements and functions of industrialInternet. The architecture integrates SDN, fog computing, cloud computing, IoT devices andthe OPC UA TCP/IP standard;− We propose a centralized parameter acquiring method for NUMFabric using the weight max-min fairness algorithm, which is based on server to client transmission process of the OPCUA TCP/IP standard;− We implement a sensor testbed for proving the feasibility of the proposed architecture andthe centralized parameter acquiring method. On our experimental platform, the bandwidthallocation recovery time is short.This work is organized as follows:Chapter 1: Introduction. The background/motivation and thesis contributions/organizationare described in this chapter. The motivation summarizes the shortcomings of the existing tradi-tional industrial networks. The contributions take into account the solutions to these shortcomingscorresponding.Chapter 2: Industrial Internet with New Technology and Architecture. The industrial Internetintegrates with SDN, fog computing and centralized architecture. The “integration of interior-exterior network” and “integration of Cloud-Fog-SDN-IoT” concepts are presented in this chapter.A four-layer architecture and the functions of industrial Internet are described. Each IoT devicecan run the OPC UA TCP/IP to transmit data, and the compatibility among different devices willbe better than traditional standards. Using SDN and fog computing, the centralized architecturereplaces the traditional distribution network to transmit and process local data packets. Thecentralized architecture enables more flexible industrial and reduces bandwidth allocation recoverytime.Chapter 3: Fast Convergent Bandwidth Allocation. When an emergency occurs, the bandwidthallocation uses SDN and fog computing to compute the parameters in a centralized manner, in which91.3. Contributions and Organizationthe NUMFabric and the weight max-min fairness algorithm are utilized. On the other hand, theprogramming of the whole OPC UA transmission processes includes the design of sensor collectors,OPC servers and the OPC client.Chapter 4: Testbed Implementation and Testing. The testbed topology and parameters aredesigned. Some devices are deployed and implemented on the testbed. The connectivity of allnetwork devices are tested. When an emergency occurs, SDN switches can obtain and send flowchanging information to the fog computing server. After the fog computing server finishes thebandwidth allocation calculation, a new bandwidth allocation schedule will be sent to an SDNcontroller. Then the SDN controller will send this new schedule to each SDN switch to changebandwidth. The bandwidth allocation convergence process will be tested. On our experimentalplatform, the bandwidth allocation recovery time will be compared between the traditional TCPand the proposed architecture when an emergency occurs.Chapter 5: Conclusions. The contributions and future work are summarized.10Chapter 2Industrial Internet with NewTechnology and ArchitectureThis chapter describes the industrial Internet with SDN, fog computing and centralized architec-ture first. Then, an overall and detailed network architecture will be presented. Lastly, OpenFlowprotocol and the OPC UA standard will be introduced.2.1 Industrial Internet with SDN and Centralized ArchitectureIndustrial Internet is one of the key technologies to achieve ubiquitous network connectivity.It is used to connect the distributed industrial sensors, controllers, network transmission devices,gateways and other subsystems. At the same time, for large-scale industrial Internet connections,the requirements of bandwidth, low delay and timely control also need to be met. The current net-work control architecture can be generally divided into distributed and centralized network controlarchitectures. Currently, a large number of distributed industrial sensors and machines are widelyconnected by conventional switches in the distributed network control architecture. Based on thefeatures of traditional industrial networks, the network designers believe that combining indus-trial Internet with a distributed network control architecture is suitable for large-scale connections[31–33].However, using a centralized network control architecture based on SDN is favorable in termsof industrial Internet. Since the SDN differs from traditional network in that the control planeand forward plane are separated, and the SDN controller can be directly programmed to controlglobal network functions [7]. Industrial Internet with SDN is a timely centralized control networkarchitecture. Based on the user defined policies and the instantaneous network status, industrial112.1. Industrial Internet with SDN and Centralized ArchitectureInternet with SDN will break the barrier of layering, thus the industrial operators can design andoperate the network easily. In the case of increasing factories’ demand, the industrial Internetwith SDN is not only more suitable for large-scale sensors and machine connections through SDNswitches, but also able to smartly guarantee high bandwidth service quality and reduce bandwidthusage during off work hours [34].Comparing the traditional industrial networks to the industrial Internet with SDN, the otheradvantages in performance, configuration and innovation will be described in the following threeaspects:− Improved Performance: In traditional industrial network operations, maximum utilizationof the network devices is one of the important objectives. Industrial Internet with SDNcan reduce the cost and use reliable physical network devices to carry out the routing func-tions through software programming. The system will be simplified, and the security will bestrengthened at the same time. Three advantages arise in the industrial Internet with SDNcontroller by adopting a centralized routing scheme. First, routing management can integratethe existing traditional industrial network devices and administration, which is more benefi-cial to industrial Internet maintenance and security management. To be specific, quality ofservice (QoS), data traffic scheduling, load balanced packet routing, end-to-end congestioncontrol and energy efficient operation can be easily achieved. Second, resource managementwill integrate computation and storage into a unified and automatic management system.Third, industrial Internet with SDN can also be dynamically optimized and configured basedon network status [35–41].− Enhanced Configuration: In the traditional industrial network management, one of the mostimportant functions is configuration. To attain the traditional industrial network integrity,when a new device is added to the existing traditional industrial networks, some basic manualconfigurations are necessary. In these manual configuration processes, errors are inevitable.Nevertheless in industrial Internet with SDN, the unified control layer can meet differentrequirements from the infrastructure layer by using software programmings, which can carryout the functions of routers, switches, firewalls, load balancers and network address translators(NAT). Thus, industrial Internet with SDN can effectively prevent unnecessary errors with122.2. Industrial Internet with Fog Computingthe aids of automated controlling [42].− Encouraged Innovation: for the evolution of traditional industrial network applications, theindustrial Internet innovation should be promoted to meet the requirements of future theindustrial control. Unfortunately, proprietary hardware components are prevented from beingmodified in the traditional industrial networks. Thus, the traditional industrial network willgreatly impede the innovation and development. In contrast, industrial Internet with SDN canprovide a programmable network platform to encourage the innovation and the development.The industrial Internet with SDN will not only provide a clear separation between virtualnetworks and current traditional industrial networks, but also put experiment platforms in areal environment. The industrial Internet with SDN will gradually and seamlessly implementinnovate from experiments to deployments stage [43–45].2.2 Industrial Internet with Fog ComputingCloud computing is the brain of Internet applications. It has been widely deployed in variousfields. In industrial field, cloud computing is integrated into traditional industrial networks for theaim of enhancing the level of intelligent factories. However, with the rapid development of IoT,50 billion “things” will be connected to Internet by 2020. With a large number of data packetsin various kinds will generate, and the traditional cloud computing architectures can not meet thefollowing requirements [30]:− High Latency: The cloud computing servers are usually far away from factories, and theservice response and transmission latency will be in minutes level. Due to the above men-tioned, when manufacturing line encounters an emergency, the cloud computing server cannot be able to collect data immediately and send the corresponding control instructions tothe manufacturing line then, so the system will fail and a disaster will occur.− Low Bandwidth: Limited by the network bandwidth, it is impossible for the airplane toimmediately transmit these data packets to cloud computing server for monitoring, analyzing,computing and storing.132.3. Overall Architecture Description− Low Security: During the manufacturing processes, the sensitive IoT data will be transmitted,analyzed, computed and stored on exterior cloud computing servers. These sensitive IoT datapackets should be protected to avoid leakage.To relieve the above disadvantages, the industrial Internet with fog computing is presented.The benefits of industrial Internet with fog Computing are listed below [4]:− Lower Latency: In comparison to cloud computing server, the fog computing server is closerto data source devices. It will timely collect the IoT data packets and send the correspondingcontrol instructions to the manufacturing line to avoid a system failure and disaster.− Lower Public Network Traffic and Cost: The analysis and computation of industrial data canuse local fog computing servers rather than exterior cloud computing servers. The exteriornetwork bandwidth and cost can be reduced.− Better Security: By using a local fog computing server to analyze and compute sensitive IoTdata, the data packets can avoid from leakage.− Greater Business Agility: Fog applications can be quickly deployed into industrial environ-ments and the manufacturer can offer machine as a service (MaaS) to customers. Accordingto the customers’ requirements, new products will improve the service level through fog ap-plications.A comparison of different aspects of fog computing and cloud computing in industrial Internetis presented in Table 2.1. The existing traditional industrial network with cloud computing andindustrial Internet with fog computing are applied to different industrial scenarios. As shown inTable 2.1, the traditional industrial network with cloud computing focuses on the nonreal-time andlong-period data. It is mainly applied to the areas of periodic maintenance and business decisionsupport. However, the industrial Internet with fog computing focuses on timely analysis for rapidlygenerated data. It will process local data timely and intelligently. For future interior networks,industrial Internet with fog computing will become indispensable.142.3. Overall Architecture DescriptionTable 2.1: Comparison of Fog Computing and Cloud Computing in Industrial InternetItem Fog Computing Cloud ComputingResponse Time Milliseconds to sub-second Minutes, days, weeksStorage Data Time Transient storage, often 1 or 2hoursMonths or yearsGeographic Coverage Local GlobalSecurity High, suitable for sensitiveIoT dataLow, suitable for general dataCostly of Public NetworkBandwidthLess MoreTransition protocol Any protocol IP protocolApplication Examples M2M communication Big data analyticsEnvironment Airplanes, ships, vehicles,roadways, railways andfactoryCompany2.3 Overall Architecture DescriptionThe detailed network architecture is shown in Fig. 2.1. The design challenges are depicted inthe following paragraphs. Compared with the existing traditional industrial network architecture,INTERIOR CONTROL LAYERINTERIOR DATA LAYEREXTERIOR ACCESS LAYERSDN ControllerFog  ServerINTERIOR BUSINESS LAYERSDN SwitchesIoT devicesCloud ServerRemote Private Computer Office Computer OPC UA StandardFigure 2.1: Industrial Internet Architecture152.3. Overall Architecture Descriptionthe industrial Internet architecture can not only improve service efficiency, compatibility, flexibilityand security, but also reduce transmission latency between servers and IoT devices. As shown inFig. 2.1, the industrial Internet architecture includes four layers: interior data layer, interior controllayer, interior business layer and exterior access layer.− In the interior data layer, thousands of sensors will collect data packets, and then encapsulatethese data packets using the OPC UA standard based on TCP. Using the OPC UA standardwill improve the compatibility among different devices. Then, the data packets are forwardedto the interior business layer through SDN switches. SDN switches can identify the networkstatus to transmit data packets according to a flow table and actions from an SDN controller.− In the interior control layer, the SDN controller performs different functions under differentsituations. When the manufacture line is normal, the SDN controller will automatically per-form the network control layer functions according to the timely network status. For example,QoS functions, switching and routing data packets can be realized. When an emergency oc-curs on the manufacture line, the SDN controller can allocate bandwidth according to theanalyzing results performed by the fog computing server. Then the SDN controller will trans-mit the corresponding bandwidth allocation schedule to the SDN switches using OpenFlowprotocol. The SDN controller can centrally manage all SDN switches. In this manner, theindustrial Internet can improve performance, enhance configuration and realize innovation.− In the interior business layer, fog computing is used to store, compute and analyze datafrom a large number of sensors through a local server. When an emergency occurs, the fogcomputing server can analyze and allocate bandwidth, and transmit the bandwidth allocationschedule to the SDN controller through the proxy. Compared with cloud computing, fogcomputing will reduce the transmission latency and traffic load in the industrial Internetto avoid a system failure. On the other hand, by using of fog computing programs, thecustomers’ requirements can be met. The collaborative manufacturing, industrial innovationand production maintenance services will be largely enhanced and the production design cycleis shortened.− In the exterior access layer, office computers, private computers and cloud computing servers162.4. Protocol and Standard Descriptionwill be allowed to access a fog computing server in the interior business layer through networksecurity authentication. Any person can timely read and monitor the latest information andthe storage data in the fog computing server. The interior data layer and interior control layercan also communicate with the exterior access layer under some network security strategies.In summary, through integrating Cloud-Fog-SDN-IoT, interior network and exterior network, adetailed industrial Internet architecture is formed and can realize M2M, machine-to-platform andupper-to-lower mechanical connections.2.4 Protocol and Standard DescriptionTo establish connections between the SDN controller and switches, and between IoT sensorsand the fog computing server, the OpenFlow protocol and the OPC UA standard are proposed asthe transmission protocol and standard respectively.2.4.1 OpenFlow ProtocolThe OpenFlow protocol was proposed by Nick McKeown from Stanford University in [46].This protocol is used to describe the way in which information is exchanged, and the standardinteractions between SDN controllers and SDN switches. The idea of the OpenFlow protocol isstraightforward: the SDN controller is responsible for generation, maintenance and transmissionof the FlowTable, and SDN switches only forward data packets according to this table. TheOpenFlow protocol was written and developed by the ONF organization. So far, the versions havebeen released from 1.0 to 1.5.2.4.2 OPC UA StandardThe industrial Internet needs to improve compatibility among IoT devices. The compatibilitymainly refers to the data format compatibility and the communication protocol compatibility. Aunified standard is of key importance to realize compatibility. Factories will build the industrialInternet based on the existing traditional industrial networks. To improve the compatibility betweenoriginal industrial devices and industrial Internet devices, several communication standards for172.5. Summaryindustrial devices have been proposed, e.g. the OPC UA standard. The OPC UA standard notonly features standard uniform access, but also improves communication performance. The OPCUA is the most popular standard among the European industrial devices communication standards[47].The OPC is an industrial telecommunication standard that was proposed in 1996. The coresof the OPC communication standard are interoperability and standardization. The OPC standardis suitable for solving the interoperability problem among hardware devices in the control level.To tackle the new challenges exist in data modeling and data security, the OPC UA standardwas released in 2008. The OPC UA is used to provide a feature-rich technology open-platformarchitecture that is extendable and scalable [48].In the intelligent industrial Internet, the OPC UA standard helps to quickly realize automaticcommunication among different software product providers from the physical network to the virtualnetwork. According to the Europe automation director Jonathan Wilkins said, with the rapiddevelopment of industrial intelligence, the OPC UA standard will be developed for a long time ratherthan slow down. Ultimately, the OPC UA standard will provide a plug and play communicationsolution for users. It will comply to the development trend of industrial Internet, big data and IoT[49].2.5 SummaryIn this chapter, we first introduced the industrial Internet with SDN and a centralized archi-tecture. In comparison to the traditional industrial network, SDN and centralized architectureenhance configuration, improve performance and encourage innovation for the industrial Internet.Then, we compared the existing industrial Internet with cloud computing and the industrial Inter-net with fog computing. The fog computing is necessary in the industrial Internet. Furthermore,we presented an overall description of industrial Internet architecture. The roles and functions ofthe four-layer architecture were described in detail. Lastly, the concepts of the OpenFlow protocoland the OPC UA standard were introduced.18Chapter 3Fast Convergent BandwidthAllocationIn this chapter, a fast-convergent bandwidth allocation method is designed. The proposedmethod uses SDN and fog computing to centrally obtain and compute the flow rate parameter forNUMFabric, which uses the weight max-min fairness algorithm based on the OPC UA TCP/IPtransmission standard. In addition, the whole server to client transmission process of OPC UATCP/IP will be designed, including the designs of the sensor collector, the OPC server and theOPC client.3.1 BackgroundIn industrial Internet, TCP needs proactive bandwidth allocation. Thus, the traditional TCPbandwidth allocation algorithm should be redesigned for different bandwidth allocation objectives.The NUMFabric allows per-flow bandwidth allocation to be expressed using utility functions [9]and adopts the weight max-min fairness algorithm [10]. In the following section, the NUMFabricand the weight max-min fairness algorithm will be introduced.3.1.1 NUMFabricIn [9], a novel, flexible and fast control transport design of bandwidth allocation based on thedistribution network was presented. Under the constraints of link capacity, the maximum overallsystem rates can be used as the optimization goal, which can be expressed as [50]max∑iUi(xi) (1)193.1. Backgrounds.t.Rx ≤ cwhere Ui(·) is the utility function, and xi is the flow rate of flow i. The x is the vector of flowrates and term R is the 0-1 routing matrix, i.e., R(i,l) = 1 if and only if flow i traverses link l. Thec is the vector of link capacities. In [9], the α-fair class of utility function, which is one of the fourpopular and broad bandwidth allocation policies, will be used as the bandwidth allocation policyin this work. The allocation objective is weighted α-fairness, and the network utility maximization(NUM) objective [10] is given by∑iwαi x1−αi /(1− α) (2)where wi is bandwidth allocation weights. The α-fairness can use different bandwidth allocationweights wi to express relative priorities for different flows. The non-negative constant α expressesthe fairness and efficiency trade-off. When α is 0, it is an extreme corresponding the maximaloverall throughput without concerns of fairness. As α increases, the throughput becomes fairer,eventually converging to the egalitarian max-min fair allocation. Especially, when α is 1, it isanother extreme illustrating no utilization of the network. There are two constraints, which areexpressed asflow rate xi ≥ minimum capacityflow rate xi ≤ maximum capacity.3.1.2 Weight Max-Min Fairness AlgorithmBandwidth allocation has generally been at the mercy of TCP. TCPs model of allocation as-sumes that bandwidth should be shared equally among different devices. However, for an emergencysender of industrial Internet such an allocation is not a good fit. For example, some robot-arms can-not respond fast enough to complete the intended tasks. The weighted max-min fairness algorithmis a fast-convergent bandwidth allocation algorithm and can improve the bandwidth allocation ef-ficiency. The max-min fairness algorithm achieves the weighted max-min allocation rapidly as thedata/traffic flows or their weights change. The link weights can quickly converge and aggressivelyupdate towards the optimal point, and therefore, bandwidth allocation duration is reduced.203.1. BackgroundThe detailed bandwidth allocation steps are explained as follows. Assume that there are fourusers and the total bandwidth is 100 Mbps. The users bandwidth demands are 20 Mbps, 50 Mbps,10 Mbps and 60 Mbps. The corresponding bandwidth weights are 0.4, 0.3, 0.2 and 0.1 respectively.The bandwidth allocation process can be designed as a four-step process.According to the total available bandwidth, users bandwidth demands and bandwidth weights,Table 3.1 shows the first step bandwidth allocation process.Table 3.1: The First-step Bandwidth AllocationItem User 1 User 2 User 3 User 4Demand 20 Mbps 50 Mbps 10 Mbps 60 MbpsBandwidth Allocation 40 Mbps 30 Mbps 20 Mbps 10 MbpsSince User 1 and User 3 only need 20 Mbps and 10 Mbps bandwidth rather than 40 Mbps and20 Mbps, the fairness algorithm will only allocate 20 Mbps and 10 Mbps bandwidth to User 1 andUser 3. Table 3.2 shows the second bandwidth allocation step.Table 3.2: The Second-step Bandwidth AllocationItem User 1 User 2 User 3 User 4Demand 20 Mbps 50 Mbps 10 Mbps 60 MbpsBandwidth Allocation 20 Mbps 30 Mbps 10 Mbps 10 MbpsAfter the second-step bandwidth allocation process, the rest 30 Mbps bandwidth remains, thusthe bandwidth requirements of User 2 and 4 still can not be satisfied. According to the bandwidthweights, the remaining 30 Mbps bandwidth will be allocated to User 2 and User 4, and they willget 22.5 Mbps and 7.5 Mbps respectively. Then, the following bandwidth allocation repeats step 1and step 2. The third bandwidth allocation step is shown in Table 3.3. The fairness algorithm willallocate only bandwidth 50 Mbps, rather than bandwidth 52.5 Mbps to User 2.Table 3.3: The Third-step Bandwidth AllocationItem User 1 User 2 User 3 User 4Demand 20 Mbps 50 Mbps 10 Mbps 60 MbpsBandwidth Allocation 20 Mbps 50 Mbps 10 Mbps 17.5 MbpsAfter the third bandwidth allocation step, the rest 2.5 Mbps bandwidth still remains, andthe bandwidth requirements of User 4 are still not satisfied. The remaining 2.5 Mbps bandwidth213.2. The Design of Fast Convergent Bandwidth Allocationtogether with the existing 17.5 Mbps can be allocated to User 4. The fourth bandwidth allocationstep is shown in Table 3.4.Table 3.4: The Fourth-step Bandwidth AllocationItem User 1 User 2 User 3 User 4Demand 20 Mbps 50 Mbps 10 Mbps 60 MbpsBandwidth Allocation 20 Mbps 50 Mbps 10 Mbps 20 Mbps3.2 The Design of Fast Convergent Bandwidth AllocationThe NUMFabric uses the weight max-min fairness algorithm, which is based on distributed net-work architecture and is deployed on conventional switches. The NUMFabric’s flow rate parameterneeds to be accumulated and repeatedly computed on different conventional switches in distributednetwork. In this case, when network congestion occurs, the bandwidth allocation recovery processwill become slow. To address this issue, a fast-convergent bandwidth allocation method will bedesigned. By using the SDN and fog computing based on a centralized industrial Internet archi-tecture, the NUMFabric’s global parameters can be directly obtained at OpenFlow counter in theSDN controller. When network congestion occurs, the bandwidth allocation recovery time is short.The fast-convergent bandwidth allocation flow chart is shown in Fig. 3.1. The flow rate isfirst obtained by OpenFlow counter. Second, take this flow rate and initialized weight into theoptimization formula (2), a new weight can be created for each user. Third, the weight Max-Minfairness algorithm utilizes the new weight to obtain a new bandwidth allocation schedule. Next, theRyu controller of SDN receives the new bandwidth allocation schedule. Lastly, the Ryu controllerwill send the new bandwidth allocation schedule to OpenFlow switches. The above process achievesfast-convergent bandwidth allocation.3.2.1 Obtaining the Flow RateThe flow rate obtaining flow chart is shown in Fig. 3.2. First, the OpenFlow datapath identifier(DPID) number or switch identify number needs to be derived to switch identification. Then basedon this number, the port state needs to be read. The port state is stored in a table which includesthe number of transmission packets/bytes, the number of receive packets and duration time. The223.2. The Design of Fast Convergent Bandwidth AllocationWeight Max-Min Fairness Bandwidth AllcationRyu QoS  RedistributionFlow RateOpenFlow Counter1iw / (1 )i iX   OpenFlow switch QoS  RedistributionWeight Max-Min Fairness Bandwidth AllocationFlow Rate1iw / (1 )i iX   Ryu Controller QoS  RedistributionOpenFlow Switch QoS  RedistributionFigure 3.1: The Fast Convergent Bandwidth Allocation Flow Chartnumber of transmit and receive packets can be read from OpenFlow counter. Lastly, by dividingthe number of packets by the duration time, the flow rate can be derived.Get DPID Get Port State Flow RateFigure 3.2: The Flow Rate Deriving Flow Chart3.2.2 The Design of Optimal AlgorithmThe optimal algorithm flow chart is shown in Fig. 3.3. First, initialize the weight value. Forexample, assign an array with eight elements, and the eight weights correspond to eight nodes.Then, change the weight of the emergency node. To make emergency node weight greater thanthat of the other nodes, the original default weight values are multiplied with ten. Lastly, theoptimal problem is constructed and solved.The parameter α takes 0.5 to reflect the general case that the trade-off between the two extremesof 0 and 1 is achieved. The optimization objective becomes∑i2√wixi. (3)There are two constraints, which are expressed asflow rate xi ≥ minimum capacityflow rate xi ≤ maximum capacity.233.2. The Design of Fast Convergent Bandwidth AllocationInitialize Bandwidth Allocation Weight Change Weight of theEmergency NodeConstruct the Optimization ProblemSolve the Optimization ProblemFigure 3.3: The Algorithm Flow Chart3.2.3 The SDN Packet Process PipelineThe flow chart of SDN packet process pipeline is shown in Fig. 3.4. When a data packet istransmitted into OpenFlow switch, the data packet will be checked whether it matches the QoSrules, including matching of TCP protocol and IPv4 address. If the data packet is matched, itwill be forwarded to a meter. Similar as a counter, the meter is used to count data packets andlimit the number of data packets. Before the meter reaches a predetermined value, the OpenFlowswitch forwards data packets. Once the predetermined value is reached, the OpenFlow switch willdrop data packets according to the QoS requirements. The meter value and QoS rules from the fogcomputing server write the switching application and QoS application respectively. Initialize PIN GPIO Initialize PWM GPIO Initialize Board Open File From PCF8591 FileRead Sensor Value From PCF8591 File Initialize ClientInitialize UA VariantBuild Node IDRead Value From SensorOutputCleanConnect ServerConnect FailedState == Good ?YESNORegister SIGINT Handler Initialize BoardLoad ConfigurationLoad Network LayerInitialize Sensor Initialize ServerAdd Sensor VariableRegister Callback and runSDN (OpenFlow Switch)OPC Server(ARM NEO Board)Sensor Collector(ARM NEO Board)ADCOptical Sensor(Matrix Photoresistor)OPC Client (Fog Computing server)1000 timesSDN (OpenFlow Switch)OPC Server(ARM NEO Board)Sensor Collector(ARM NEO Board)PacketOptical Sensor(Matrix Photoresistor)OPC Client (Fog Computing server)Read Configuration File Set Meter ValueMatch QoS RuleMatch QoS Rules  Set Meter ValueForward PacketDrop PacketFigure 3.4: SDN Packet Process Pipeline Flow Chart3.2.4 The Design of Meter-QoS RuleThe flow chart of meter-QoS rule design is shown in Fig. 3.5. First, the configuration fileincluding QoS value needs to be read. When the manufacture line is normal, the QoS value can beset as a default value. When an emergency occurs, the value should be changed according to theoptimization algorithm, which has been presented in Section 3.2.2. Second, the meter value needsto be set. Lastly, the QoS rules need to be matched. Due to the fact that matching the QoS rulesrequires the meter ID, the meter value should be set before matching the QoS rules. The functionsof meter and QoS rules have been described in Section 3.2.3.243.3. The Design of OPC Server to OPC Client Transmission ProcessRead  the Configuration File Set Meter Value Set QoS RulesFigure 3.5: Fog Computing Flow Chart3.3 The Design of OPC Server to OPC Client TransmissionProcessOPC UA can utilize TCP/IP, which is suitable for industrial Internet, to transmit data packetsin both traditional industrial networks and industrial Internet. Using OPC UA TCP/IP standardto transmit and receive data packets, not only the compatibility issue can be solved, but alsothousands of industrial devices can gain easily the Internet access to improve industrial productionefficiency. The detailed flow charts of programming design are described as follows.The flow chart of OPC server to OPC client transmission process is shown in Fig. 3.6. It includessix parts: optical sensor module, analog-to-digital converter (ADC) module, sensor collector, OPCserver, SDN packet process and OPC client. The optical sensor and ADC module can directlyrun on computers. When connected to the OPC server through dupont wires. The flow charts ofsensor collector, OPC server and OPC client functions will be sequentially presented in the nextsubsection. Initialize PIN GPIO Initialize PWM GPIO Initialize Board Open File From PCF8591 FileRead Sensor Value From PCF8591 File Initialize ClientInitialize UA VariantBuild Node IDRead Value From SensorOutputCleanConnect ServerConnect FailedState == Good ?YESNORegister SIGINT Handler Initialize BoardLoad ConfigurationLoad Network LayerInitialize Sensor Initialize ServerAdd Sensor VariableRegister Callback and runSDN (OpenFlow Switch)OPC Server(ARM NEO Board)Sensor Collector(ARM NEO Board)ADCOptical Sensor(Matrix Photoresistor)OPC Client (Fog Computing server)1000 timesSDN (OpenFlow Switch)OPC Server(ARM NEO Board)Sensor Collector(ARM NEO Board)PacketOptical Sensor(Matrix Photoresistor)OPC Client (Fog Computing server)Read Configuration File Set Meter ValueMatch QoS RuleMatch QoS Rules  Set Meter ValueForward PacketDrop PacketOptical SensorADCSensor CollectorOPC ServerSDN OPC Client  Initialize PIN GPIO Initialize PWM GPIO Initialize the BoardRead Sensor Value From the PCF8591 File Open File From the PCF8591 FileRegister SIGINT Handler Initialize the BoardLoad ConfigurationLoad Network Layer ParametersInitialize Sensors Initialize the ServerAdd Sensor VariableRegister Callback and running Initialize ClientInitialize UA VariantBuild Node IDRead Value From SensorOutputCleanConnect ServerConnected?YESNO1000 CyclesFigure 3.6: The Transmission Process Flow Chart3.3.1 The Design of Sensor Collector Initialize PIN GPIO Initialize PWM GPIO Initialize Board Open File From PCF8591 FileRead Sensor Value From PCF8591 File Initialize ClientInitialize UA VariantBuild Node IDRead Value From SensorOutputCleanConnect ServerConnect FailedState == Good ?YESNORegister SIGINT Handler Initialize BoardLoad ConfigurationLoad Network LayerInitialize Sensor Initialize ServerAdd Sensor VariableRegister Callback and runSDN (OpenFlow Switch)OPC Server(ARM NEO Board)Sensor Collector(ARM NEO Board)ADCOptical Sensor(Matrix Photoresistor)OPC Client (Fog Computing server)1000 timesSDN (OpenFlow Switch)OPC Server(ARM NEO Board)Sensor Collector(ARM NEO Board)PacketOptical Sensor(Matrix Photoresistor)OPC Client (Fog Computing server)Read Configuration File Set Meter ValueMatch QoS RuleMatch QoS Rules  Set Meter ValueForward PacketDrop PacketOptical SensorADCSensor CollectorOPC ServerSDN OPC Client  Initialize PIN GPIO Initialize PWM GPIO Initialize the BoardRead Sensor Value From the PCF8591 File Open File From the PCF8591 FileRegister SIGINT Handler Initialize the BoardLoad ConfigurationLoad Network Layer ParametersInitialize Sensors Initialize the ServerAdd Sensor VariableRegister Callback and running Initialize ClientInitialize UA VariantBuild Node IDRead Value From SensorOutputCleanConnect ServerConnected?YESNO1000 CyclesFigure 3.7: Sensor Collector Flow ChartThe sensor collector is used to collect and read the data from the sensors, and can be imple-253.3. The Design of OPC Server to OPC Client Transmission Processmented on the ARM NEO board. The sensor collector flow chart is shown in Fig. 3.7. First,the pins general purpose input output (GPIO), pulse width modulation (PWM) GPIO and boardsystem are initialized. Then, the sensor collector will open the PCF8591 file which is automaticallygenerated by the optical sensors, and the sensor data will be stored in this file. Lastly, the “readsensor data” function will read the PCF8591 file to wait the OPC server connection.3.3.2 The Design of OPC Server Flow Chart Initialize PIN GPIO Initialize PWM GPIO Initialize Board Open File From PCF8591 FileRead Sensor Value From PCF8591 File Initialize ClientInitialize UA VariantBuild Node IDRead Value From SensorOutputCleanConnect ServerConnect FailedState == Good ?YESNORegister SIGINT Handler Initialize BoardLoad ConfigurationLoad Network LayerInitialize Sensor Initialize ServerAdd Sensor VariableRegister Callback and runSDN (OpenFlow Switch)OPC Server(ARM NEO Board)Sensor Collector(ARM NEO Board)ADCOptical Sensor(Matrix Photoresistor)OPC Client (Fog Computing server)1000 timesSDN (OpenFlow Switch)OPC Server(ARM NEO Board)Sensor Collector(ARM NEO Board)PacketOptical Sensor(Matrix Photoresistor)OPC Client (Fog Computing server)Read Configuration File Set Meter ValueMatch QoS RuleMatch QoS Rules  Set Meter ValueForward PacketDrop PacketOptical SensorADCSensor CollectorOPC ServerSDN OPC Client  Initialize PIN GPIO Initialize PWM GPIO Initialize the BoardRead Sensor Value From the PCF8591 File Open File From the PCF8591 FileRegister SIGINT Handler Initialize the BoardLoad ConfigurationLoad Network Layer ParametersInitialize Sensors Initialize the ServerAdd Sensor VariableRegister Callback and running Initialize ClientInitialize UA VariantBuild Node IDRead Value From SensorOutputCleanConnect ServerConnected?YESNO1000 Cycles Initialize the BoardLoad Network Layer ParametersInitialize Sensors Initialize the ServerAdd Sensor VariableLoad ConfigurationRegister Callback and RunningFigure 3.8: OPC Server Flow ChartThe OPC server is used to receive data from the sensor collector, and it is also implementedon the ARM NEO board. The flow chart of OPC server is shown in Fig. 3.8. First, the signalinterrupt (SIGINT) handler is registered. The function of SIGINT is to clean up the temporaryvariable after running OPC server program. Second, the ARM NEO board is initialized, includingGPIO initialization, loaded configuration initialization and network layer functions initialization.The configuration information includes server description, security login, network information suchas port number and buffer size, and limit condition such as secure channel and session. Afterfinishing the above-mentioned configuration and loading, the OPC server loads OPC configurationfile and listens. Third, the sensor variable can be created to store the data from sensors. Fourth, theregister callback and running contain “before read” and “after write”. The “before read” includescollecting sensor data and writing sensor data to sensor variable function. It is used to update thesensor data from a PCF8591 file in the sensor collector. The “after write” is used to output loginformation which will show that the sensor data has been updated. Lastly, the OPC server waits263.3. The Design of OPC Server to OPC Client Transmission Processfor OPC client connection.3.3.3 The Design of OPC Client Flow ChartThe OPC client is used to receive data from OPC server through SDN, and can be implementedusing the fog computing server. The flow chart of OPC client is shown in Fig. 3.9. First, the clientis initialized and connected to the OPC server. If the server connection fails, go back to initializethe client. If the server is connected, the UA-Variant will be initialized. The UA-Variant stores thesensor data from OPC server. Second, build node identification (ID) and read the value from theOPC server. Third, the value will be displayed as an output. Fourth, clean the terms such as theconnection, buffer and temporary variable. Lastly, the 1000 cycles will be used to realize receiving1000 times OPC server data, namely sensor data. Initialize PIN GPIO Initialize PWM GPIO Initialize Board Open File From PCF8591 FileRead Sensor Value From PCF8591 File Initialize ClientInitialize UA VariantBuild Node IDRead Value From SensorOutputCleanConnect ServerConnect FailedState == Good ?YESNORegister SIGINT Handler Initialize BoardLoad ConfigurationLoad Network LayerInitialize Sensor Initialize ServerAdd Sensor VariableRegister Callback and runSDN (OpenFlow Switch)OPC Server(ARM NEO Board)Sensor Collector(ARM NEO Board)ADCOptical Sensor(Matrix Photoresistor)OPC Client (Fog Computing server)1000 timesSDN (OpenFlow Switch)OPC Server(ARM NEO Board)Sensor Collector(ARM NEO Board)PacketOptical Sensor(Matrix Photoresistor)OPC Client (Fog Computing server)Read Configuration File Set Meter ValueMatch QoS RuleMatch QoS Rules  Set Meter ValueForward PacketDrop PacketOptical SensorADCSensor CollectorOPC ServerSDN OPC Client  Initialize PIN GPIO Initialize PWM GPIO Initialize the BoardRead Sensor Value From the PCF8591 File Open File From the PCF8591 FileRegister SIGINT Handler Initialize the BoardLoad ConfigurationLoad Network Layer ParametersInitialize Sensors Initialize the ServerAdd Sensor VariableRegister Callback and running Initialize ClientInitialize UA VariantBuild Node IDRead Value From SensorOutputCleanConnect ServerConnected?YESNO1000 CyclesFigure 3.9: OPC Client Flow Chart273.4. Summary3.4 SummaryIn this chapter, we designed a centralized parameter acquiring method for NUMFabric using theweight max-min fairness algorithm. The flow chart of each component is presented to illustrate thefast-convergent bandwidth allocation. Moreover, we designed the flow charts of sensor collector,OPC sever and OPC client to realize OPC UA standard for transmitting TCP/IP data packet.The flow chart of each component is also illustrated in the OPC server to OPC client transmissionprocess.28Chapter 4Testbed Implementation and TestingIn this chapter, the connectivity of a controller, a fog computing server, two traditional switches,four OpenFlow switches, eight optical sensors and ARM boards will be implemented and tested atfirst. Then, the Wireshark data packet monitoring software will be used in the fog computing serverto catch and analyze the OPC UA TCP and traditional TCP data packets. Further, bandwidthallocation under emergency will be tested. Lastly, the bandwidth allocation recovery time will betested under the traditional TCP architecture and the proposed architecture.4.1 Testbed Topology DesignThe testbed of the interior network topology is shown in Fig. 4.1. In the testbed, there is athree-layer interior industrial Internet architecture, including interior data layer, interior controllayer and interior business layer.− For the interior control layer, a computer will act as an SDN Ryu controller. It will controlall network SDN OpenFlow switches through an 8-port traditional switch.− For the interior data layer, it contains two traditional switches, four SDN OpenFlow switchesand eight optical seniors. They will collect, transmit and receive data packets which are basedon OPC UA standard.− For the interior business layer, another computer will act as a fog computing server. It isused to analyze and compute data packets. When an emergency occurs, the fog computingserver can analyze data packets, run optimization algorithm and transmit new bandwidthallocation schedule to the SDN Ryu controller.The names and number of testbed equipments are listed in Table 4.1,294.1. Testbed Topology DesignINTERIOR CONTROL LAYERINTERIOR DATA LAYERINTERIOR BUSINESS LAYERPORT 1 PORT 1 PORT 1 PORT 1PORT 2 PORT 2 PORT 2 PORT 2PORT 4 PORT 4 PORT 4 PORT 4PORT 8PORT 1 PORT 2 PORT 3 PORT 4CONTROLLERFOG COMPUTING SERVERTRADITIONAL SWITCH 1OPENFLOW SWITCH 1 OPENFLOW SWITCH 2 OPENFLOW SWITCH 3 OPENFLOW SWITCH 4NODE 0 NODE 2 NODE 4 NODE 6NODE 1 NODE 3 NODE 5 NODE 7TRADITIONAL SWITCH 2PORT 3 PORT 3 PORT 3 PORT 3PORT 1 PORT 2 PORT 3 PORT 4PORT 8Figure 4.1: The Testbed TopologyTable 4.1: The Names and Number of Testbed EquipmentsNames of Equipments Number of EquipmentsComputer 2Traditional Switch (TL-SG1008D) 2OpenFlow Switch (Zodiac FX switch) 4Optical Sensor (Matrix Photoresistor) 8ARM Board (ARM NEO board) 8ADC Module 8Power Cable 16Network Cable 18Dupont Line 56304.2. Testbed Parameters4.2 Testbed ParametersThe network equipment parameters are listed in Table 4.2,Table 4.2: Network Equipment ParametersAttribute IP address andsubnet maskConnection OpenFlowSwitch PortConnection Tradi-tional Switch PortControllerServer192.168.0.250/24 None Traditional Switch 1Port 8Fog Comput-ing Server192.168.0.251/24 None Traditional Switch 2Port 8TraditionalSwitch 1192.168.0.1/24 OpenFlow Switch 1/2/3/4Port 4NoneTraditionalSwitch 2192.168.0.1/24 OpenFlow Switch 1/2/3/4Port 3NoneNode 0 192.168.0.11/24 OpenFlow Switch 1Port 1NoneNode 1 192.168.0.22/24 OpenFlow Switch 1Port 2NoneOpenFlowSwitch 1192.168.0.211/24 None Traditional Switch 1/2Port 1Node 2 192.168.0.33/24 OpenFlow Switch 2Port 1NoneNode 3 192.168.0.44/24 OpenFlow Switch 2Port 2NoneOpenFlowSwitch 2192.168.0.222/24 None Traditional Switch 1/2Port 2Node 4 192.168.0.55/24 OpenFlow Switch 3Port 1NoneNode 5 192.168.0.66/24 OpenFlow Switch 3Port 2NoneOpenFlowSwitch 3192.168.0.233/24 None Traditional Switch 1/2Port 3Node 6 192.168.0.77/24 OpenFlow Switch 4Port 1NoneNode 7 192.168.0.88/24 OpenFlow Switch 4Port 2NoneOpenFlowSwitch 4192.168.0.244/24 None Traditional Switch 1/2Port 4314.3. The Connectivity of the Network Equipments4.3 The Connectivity of the Network EquipmentsFig. 4.2 shows the connection of overall network devices. The ARM board and the fog computingserver are used as OPC servers and an OPC client respectively. Fig. 4.3 shows that the OPC clientcan receive the OPC server data packets, namely the data packets of node 0 to node 7, throughSDN. For instance, the first optical sensor value 1820 is from the OPC server with IP address192.168.0.11:4840, which is also the IP address of node 0.  Figure 4.2: The Connection of Overall Network Equipments   Figure 4.3: The Connection of Network Equipment Results324.4. The OPC UA TCP Data Packet Analysis4.4 The OPC UA TCP Data Packet AnalysisFig. 4.4 shows that the data packets of OPC UA TCP and traditional TCP read from theWireshark data packet monitoring software. These data packets are obtained from OPC server onnode 0 to OPC client on fog computing computer. As shown in Fig. 4.4, the total number of datapackets is 26, which consists 15 OPC UA TCP data packets and 11 traditional TCP data packets.          Figure 4.4: The Data Packet of TCP and OPC UAThe 11 data packets of traditional TCP are TCP three-way handshake connection packets (1-3), TCP timestamps connection packets (5,7,17,20 and 26) and TCP three-way handshake closeconnection packets (23-25).The 15 data packets (4, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 21 and 22) of OPC UA TCPinclude all OPC UA communication processes from OPC server to OPC client, which shows inFig. 4.5. The packets include hello message, open secure channel, get end point, create end point,activate end point, read, close session and close secure channel connection. In the ReadResponseconnection, the sensor collected data can be taken. For example, the optical value is 1820 from theOPC server 192.168.0.11:4840, which is highlighted in Fig. 4.6.334.4. The OPC UA TCP Data Packet AnalysisOPC Client(192.168.0.11)OPC Server(192.168.0.251)Hello messageAcknowledge messageGetEndpointsRequestGetEndpointsResponseCreateEndpointsRequestCreateEndpointsResponseActivateEndpointsRequestActivateEndpointsResponseReadRequestReadResponseCloseSessionRequestCloseSessionReqsponseOpenSecureChannelRequestOpenSecureChannelResponseCloseSecureChannelRequestFigure 4.5: The Process of OPC Server to Client          Figure 4.6: The Sensor Data in OPC UA TCP Packet344.5. The Emergency Bandwidth Allocation Testing4.5 The Emergency Bandwidth Allocation TestingWhen luminous intensity becomes strong, the luminous intensity of the node becomes greaterthan the security value, and the fog computing server will receive an alert message that the nodeis abnormal. As shown in Fig. 4.7, Node 0 is an abnormal node.The Emergency QoS Traffic Redistribution Testing When the luminous intensity becomes strong, the luminous intensity value of node is more than security value, the Fog computing server can receive an alert message that the node is abnormal. Figure 1 is shown that Node 0 is abnormal.  After the Fog computing server receives the alert message, it will start to compute bandwidth allocation. When Fog computing server finishes the calculation, a new bandwidth allocation schedule will be formed. Figure 2 is shown that Node 0 to 7 have been allocated new bandwidth. The Node 0 bandwidth is 30M, and Node 1 to 7 are 9M respectively. Due to the error of the floating data structure, the total bandwidth is 100M, but actual total bandwidth is 93M. Figure 4.7: The Alert Message of Node 0After the fog computing server receives the alert message, it will start to execute bandwidth354.6. Testbed Numerical Results and Discussionsallocation. When the fog computing server finishes the calculation, a new bandwidth allocationschedule will be formed. Fig. 4.8 shows that node 0 to node 7 have been allocated new bandwidth.The bandwidth for node 0 is 30Mbps, and the bandwidth for node 1 to node 7 is 9Mbps. Due to theerror of the floating data structure, although the total bandwidth is 100Mbps, a total bandwidthof 100Mbps is in fact reduced to 93Mbps.   Figure 4.8: The New Bandwidth Allocation4.6 Testbed Numerical Results and DiscussionsWe investigate the recovery time of bandwidth allocation for the traditional TCP and theproposed architecture. The test uses iPerf, which is a network bandwidth measurement tool. Bydirectly reading the kernel parameters in the computer, the bandwidth allocation recovery timecan be derived. A bandwidth of 10Mbps will be set as the maximum network bandwidth, and theexperiment will be repeated 100 times. Fig. 4.9 shows 100 test results of the bandwidth allocationrecovery time for the proposed architecture through running the algorithm for the data of traditional364.6. Testbed Numerical Results and DiscussionsTCP (blue dots) and the proposed architecture (red dots).Comparing the experimental data for the traditional TCP and that of the proposed architecture,the measurement results of traditional TCP are dispersed, and the measurement results of proposedarchitecture are converged to a line. The reason is that the mechanisms of adjusting window sizeare different. Since the bandwidth is used too much and the congestion is too heavy, the recoverytime of traditional TCP is affected by these external network conditions, and the measurementresults of traditional TCP are dispersed. However, the recovery time of the proposed architectureis only related to the running time of optimization algorithm. Comparing to the traditional TCP,whose running time varies in a wide region, the running time of the proposed algorithm convergedto a relatively constant value.Since the average recovery time can reflect the differences of mechanism performance mostdirectly, by comparing the average time between the two kinds of mechanisms, the performanceadvantage of proposed method will be reflected. The recovery time for the proposed architectureis obviously reduced comparing to that of traditional TCP. The statistical data shows that theaverage bandwidth allocation recovering time for traditional TCP is 6409.53 ms, while it is 17.97ms for the proposed architecture. The proposed architecture reduces the bandwidth allocationrecovery time by 356.68 times of that for the traditional TCP. Figure 4.9: Recovery Time Comparison Between the Traditional TCP and the Proposed Architec-ture: the Results of 100 Test374.6. Testbed Numerical Results and DiscussionsFig. 4.10 shows the bandwidth allocation recovery time of the traditional TCP and the proposedarchitecture, and this result is one of the 100 results as shown in Fig. 4.9. In the traditional TCP,when network congestion occurs, one or more data packets will be lost. According to the TCPcongestion control protocol, the new bandwidth will become half of the original maximum one,which is shown in the blue line of Fig. 4.10.IEEE NETWORK MAGAZINE, VOL. 14, NO. 8, NOVEMBER 2017 2A. IoT Monitoring ModuleThis module resides on Fog computing nodes. communicat-ing with IoT sensors through OPC protocol and retrieve data.when this module detects the data variance exceeds a certainthreshold, it will trigger an "emergency" mechanism andgenerate an alarm message containing the corresponding nodenumber and send it to the bandwidth allocation optimizationmodule.B. Bandwidth Allocation Optimization ModuleIn order to meet bandwidth allocation fairness requirementof bandwidth allocation, we use NUM algorithm to utilizethe network bandwidth. Also, we used weighted max-minalgorithm to allocate network bandwidth between contendingflows.C. TCP Window Size Notification ModuleWe chose ordinary IP packet for carrying the window sizedata to the target IoT device. In order to make the packet asshort as possible, we used the commonly used 2 ECN bitsin IP header and set these 2 bits to 11 to tag our packet.The window size value is attached as the payload. The packetformat is shown in figure 2.We wrote a netfilter kernel module to capture our windowsize packet and extract the window size value, then by modi-fying the TCP stack logic we can pass the new window sizeto TCP congestion control algorithm.IV. EVALUATIONWe build a physical testbed for our performance evaluation.We used eight NanoPi Neo boards from FriendlyARM asthe IoT nodes, each connected with one sensor to collectenvironmental luminous intensity data. For the network part,four ZodiacFX SDN switches together with two traditionalTP-Link TL-SG1008D switches provide SDN connectivity.Additionally, our testbed consists of two x86 servers, one asthe Fog computing host while the other one running Ryu asthe SDN controller.V. CONCLUSIONThe conclusion goes here.ACKNOWLEDGMENTThe authors would like to thank...REFERENCES[1] H. Kopka and P. W. Daly, A Guide to LATEX, 3rd ed. Harlow, England:Addison-Wesley, 1999.Figure 4.10: Recovery Time Comparison Between the Traditional TCP and the Proposed Archi-tecture: the Results of One TestIn the traditional TCP bandwidth allocation recovery process, the round-trip delay is commonto all the flows in the network. The relationships between bandwidth recovery time and round-tripdelay are expressed asTold = RTT ∗NrttNrtt = Nwin + 1where Told is the traditional TCP bandwidth allocation recovery time. RTT is each round tripdelay, which is about 4.5 ms. Nrtt is the number of RTT. Nwin is the number of window size,which is 1348. According to the two formulas above, all bandwidth allocation recovery processeswill result in a recovery time of 6066 ms. In the experiment, the average bandwidth allocationrecovering time for the traditional TCP is 6409.53 ms.384.7. SummaryIn the proposed architecture, the bandwidth allocation process will start-up once the emergencysender sends an alert message to the fog computing server. After the fog computing server com-pletes the bandwidth allocation calculation, the new bandwidth will be directly sent back to theemergency sender, which is shown in the red line of Fig. 4.10. In this bandwidth recovery process,the relationship among bandwidth recovery time, propagation delay and calculation delay is shownasTnew = Tpro + TcalTpro = RTTwhere Tnew is bandwidth allocation recovery time for the proposed architecture. Tpro is propagationdelay, which is about one round-trip delay, namely 4.5 ms. Tcal is calculation delay, which is thetime requiring to run the optimization algorithm in the fog computing server, and is about 10 ms.In the experiment, the average bandwidth allocation recovery time for the proposed architecture is17.97 ms.The proposed promising architecture can be used in industrial Internet. Due to the use ofinterior industrial Internet, physical devices are deployed in a limited space. Hence, the propagationdelay is small and has little effect on the bandwidth allocation recovery time. Second, the interiorindustrial Internet topology is relatively stable, and the network global information can be quicklyachieved. Though the propagation delay and calculation delay exist, the round-trip delay will nolonger exist, and the propagation delay and calculation delay will become small. Moreover, whenmore bandwidth of bottleneck link is available, the recovery time of the proposed architecture willbe smaller. Thus, the proposed architecture is better than the traditional one in industrial Internet.4.7 SummaryIn this chapter, we implemented a testbed for connectivity of IoT devices, the SDN and the fogcomputing server. The topology, parameters and equipment of the testbed are described at first.Second, we tested the connectivity of the overall testbed equipments under a normal condition.Third, the OPC UA TCP data packets are analyzed. Through analyzing these packets, we identified394.7. Summarythe OPC UA connection processes. Fourth, under the emergency condition, the fast-convergentbandwidth allocation is tested. According to the proposed architecture, an emergency bandwidthallocation processes can be realized. Lastly, using the proposed industrial Internet architecture,testbed numerical results show that bandwidth allocation recovery time becomes smaller than thatof the traditional TCP industrial network architecture.40Chapter 5ConclusionsThis chapter concludes the thesis with some general comments on a proposed architecture basedon OPC UA TCP/IP in industrial Internet. Following the general comments, the possible futurework for increasing industrial Internet functions and performance is discussed.5.1 Summary of ContributionsA novel centralized parameter acquiring method for NUMFabric using the weight max-minfairness algorithm based on OPC UA TCP/IP was studied. The proposed method solves theproblem of slow bandwidth allocation recovery in industrial Internet. The new architecture usingSDN and fog computing to obtain the applied algorithm parameters and proved to be effective inreducing bandwidth allocation recovery time. Based on SDN, fog computing, cloud computing, IoTdevices and OPC UA TCP/IP, we proposed a detailed architecture to realize industrial Internet.Testbed analysis shows that the proposed method effectively can reduce bandwidth allocationrecovery time, which makes centralized parameter acquiring method for NUMFabric using theweight max-min fairness algorithm more applicable in industrial Internet.The contributions of this thesis are listed below:− We introduced the industrial Internet, and analyzed the industrial Internet with SDN and fogcomputing characteristics. Based on the analysis, we proposed a detailed industrial Internetarchitecture for realizing the functions of industrial Internet. The architecture integratesSDN, fog computing, cloud computing, IoT devices and OPC UA TCP/IP standard.− We proposed a centralized parameter acquiring method for NUMFabric using the weight max-min fairness algorithm, with which the flow charts of the centralized method were presented.415.2. Future WorkMoreover, the flow charts of server to client transmission processes of OPC UA standard wasrealized to transmit TCP/IP data packets.− We implemented a sensor testbed to prove the feasibility of the proposed architecture andcentralized parameter acquiring method. On our experimental platform, the bandwidth allo-cation recovery time was compared between the traditional TCP and our new architectureswhen an emergency occurs. The testbed numerical results indicate that the proposed ar-chitecture reduces the bandwidth allocation recovery time by 356.68 times to that of thetraditional TCP.5.2 Future WorkFirst, in terms of the future industrial Internet technologies, the machine learning and deeplearning may be deployed on the fog computing server to realize intelligent management. Second,the development of security authentication may allow exterior access devices to access interiorbusiness layer devices remotely. Third, the wireless devices may be deployed on the interior layer,and how to present interference among wireless devices deserves investigation.This is the first stage of our study on industrial Internet. With the continuous improvementand development of network technologies, as well as the studies on industrial Internet, we will beencouraged to implement industrial Internet deployment.42Bibliography[1] “Why software is eating the world,” The wall street journal, 2011. [Online]. Available:https://www.wsj.com/articles/SB10001424053111903480904576512250915629460[2] “Forty years of the internet: how the world changed for ever,” The-guardian, 2009. [Online]. Available: https://www.theguardian.com/technology/2009/oct/23/internet-40-history-arpanet[3] P. Evans and M. Annunziata, “Industrial internet: Pushing the boundaries of minds andmachines, 2012,” Available(accessed on 4.10. 2016): https://www. ge. com/docs/chapters/In-dustrial Internet. pdf, 2015.[4] S. Kitanov, E. Monteiro, and T. Janevski, “5G and the fogsurvey of related technologies andresearch directions,” in Electrotechnical Conference (MELECON), 2016 18th Mediterranean.IEEE, 2016, pp. 1–6.[5] “What is the difference between Ethernet/IP and TCP/IP?”Robotiq, 2014. [Online]. Available: http://blog.robotiq.com/bid/52756/what-is-the-difference-between-Ethernet-IP-and-TCP-IP[6] N. corporation, “White paper: network evolution toward 2020 and beyond,” 2015.[7] “Software-defined networking definition,” Open networking foundation, 2017. [Online].Available: https://www.opennetworking.org/sdn-resources/sdn-definition[8] “TCP congestion control,” Jason Puzzle, 2017. [Online]. Available: https://blog.jason.party/28/TCP-congestion-control[9] K. Nagaraj, D. Bharadia, H. Mao, S. Chinchali, M. Alizadeh, and S. Katti, “Numfabric: Fast43Bibliographyand flexible bandwidth allocation in datacenters,” in Proceedings of the 2016 conference onACM SIGCOMM 2016 Conference. ACM, 2016, pp. 188–201.[10] J. Mo and J. Walrand, “Fair end-to-end window-based congestion control,” IEEE/ACM Trans-actions on Networking (ToN), vol. 8, no. 5, pp. 556–567, 2000.[11] M. Hermann, T. Pentek, and B. Otto, “Design principles for industry 4.0 scenarios,” in SystemSciences (HICSS), 2016 49th Hawaii International Conference on. IEEE, 2016, pp. 3928–3937.[12] X. Wu, X. Zhu, G.-Q. Wu, and W. Ding, “Data mining with big data,” IEEE transactions onknowledge and data engineering, vol. 26, no. 1, pp. 97–107, 2014.[13] “TCP/IP,” History of computers, 2017. [Online]. Available: http://history-computer.com/Internet/Maturing/TCPIP.html[14] “A framework for multiprotocol label switching,” Network working group internet draft,1998. [Online]. Available: https://tools.ietf.org/html/draft-ietf-mpls-framework-02[15] A. Sa´nchez-Monge and K. G. Szarkowicz, MPLS in the SDN Era. O’Reilly Media, 2015.[16] M. Armbrust, A. Fox, R. Griffith, A. D. Joseph, R. H. Katz, A. Konwinski, G. Lee, D. A.Patterson, A. Rabkin, I. Stoica et al., “Above the clouds: A berkeley view of cloud computing,”Technical Report UCB/EECS-2009-28, EECS Department, University of California, Berkeley,Tech. Rep., 2009.[17] J.-q. Li, F. R. Yu, G. Deng, C. Luo, Z. Ming, and Q. Yan, “Industrial internet: A survey onthe enabling technologies, applications, and challenges,” IEEE Communications Surveys andTutorials, 2017.[18] J. C. Chen and V. S. Gabriel, “Revolution of 3d printing technology and application of sixsigma methodologies to optimize the output quality characteristics,” in Industrial Technology(ICIT), 2016 IEEE International Conference on. IEEE, 2016, pp. 904–909.44Bibliography[19] R. R. Rajkumar, I. Lee, L. Sha, and J. Stankovic, “Cyber-physical systems: the next computingrevolution,” in Proceedings of the 47th Design Automation Conference. ACM, 2010, pp. 731–736.[20] F.-J. Wu, Y.-F. Kao, and Y.-C. Tseng, “From wireless sensor networks towards cyber physicalsystems,” Pervasive and Mobile Computing, vol. 7, no. 4, pp. 397–413, 2011.[21] R. H. Rawung and A. G. Putrada, “Cyber physical system: Paper survey,” in ICT For SmartSociety (ICISS), 2014 International Conference on. IEEE, 2014, pp. 273–278.[22] J. Posada, C. Toro, I. Barandiaran, D. Oyarzun, D. Stricker, R. de Amicis, E. B. Pinto,P. Eisert, J. Do¨llner, and I. Vallarino, “Visual computing as a key enabling technology forindustry 4.0 and industrial internet,” IEEE computer graphics and applications, vol. 35, no. 2,pp. 26–40, 2015.[23] F. Jian-zhong, “Development status and trend of intelligent manufacturing equipment.” Jour-nal of Mechanical and Electrical Engineering, vol. 31, no. 8, 2014.[24] L. Wang, M. To¨rngren, and M. Onori, “Current status and advancement of cyber-physicalsystems in manufacturing,” Journal of Manufacturing Systems, vol. 37, no. Part 2, pp. 517–527, 2015.[25] J. Lee, H.-A. Kao, and S. Yang, “Service innovation and smart analytics for industry 4.0 andbig data environment,” Procedia Cirp, vol. 16, pp. 3–8, 2014.[26] X. Tu, M. Wang, K. Sun, C. Zhang, and L. Zhang, “China internet industry state analysisand prosperity indexes,” China Communications, vol. 13, no. 10, pp. 245–252, 2016.[27] O. networking foundation, “Sdn architecture overview version 1.1,” 2014.[28] H. Yin, H. Xie, T. Tsou, D. Lopez, P. Aranda, and R. Sidi, “Sdni: A message exchangeprotocol for software defined networks (sdns) across multiple domains,” IETF draft, work inprogress, 2012.[29] H. Xie, T. Tsou, D. Lopez, R. Sidi, H. Yin, and P. Aranda, “Software-defined networkingefforts debuted at ietf 84,” IETF J, 2012.45Bibliography[30] C. Syst, “Fog computing and the internet of things: extend the cloud to where the things are,”2016.[31] K. Gray and T. D. Nadeau, Network function virtualization. Morgan Kaufmann, 2016.[32] “The future of customer experience: Why nfv/sdn can bringmuch more than just operational savings to operators,” Delta part-ners, 2017. [Online]. Available: https://www.deltapartnersgroup.com/future-customer-experience-why-nfvsdn-can-bring-much-more-just-operational-savings-operators[33] “Benefits of software-defined networking,” Cbtnuets, 2017. [Online]. Available: https://www.cbtnuggets.com/blog/2017/05/3-benefits-of-software-defined-networking/[34] Y. Li and M. Chen, “Software-defined network function virtualization: A survey,” IEEE Access,vol. 3, pp. 2542–2553, 2015.[35] W. Xia, Y. Wen, C. H. Foh, D. Niyato, and H. Xie, “A survey on software-defined networking,”IEEE Communications Surveys and Tutorials, vol. 17, no. 1, pp. 27–51, 2015.[36] M. Al-Fares, S. Radhakrishnan, B. Raghavan, N. Huang, and A. Vahdat, “Hedera: Dynamicflow scheduling for data center networks.” in NSDI, vol. 10, 2010, pp. 19–19.[37] M. Ghobadi, S. H. Yeganeh, and Y. Ganjali, “Rethinking end-to-end congestion control insoftware-defined networks,” in Proceedings of the 11th ACM Workshop on Hot Topics in net-works. ACM, 2012, pp. 61–66.[38] N. Handigol, M. Flajslik, S. Seetharaman, N. McKeown, and R. Johari, “Aster* x: Load-balancing as a network primitive,” in 9th GENI Engineering Conference (Plenary), 2010, pp.1–2.[39] B. Heller, S. Seetharaman, P. Mahadevan, Y. Yiakoumis, P. Sharma, S. Banerjee, and N. McK-eown, “Elastictree: Saving energy in data center networks.” in Nsdi, vol. 10, 2010, pp. 249–264.[40] A. D. Ferguson, A. Guha, J. Place, R. Fonseca, and S. Krishnamurthi, “Participatory net-working.” in Hot-ICE, 2012.46Bibliography[41] K. Jeong, J. Kim, and Y.-T. Kim, “Qos-aware network operating system for software definednetworking with generalized openflows,” in Network Operations and Management Symposium(NOMS), 2012 IEEE. IEEE, 2012, pp. 1167–1174.[42] A. Gember, P. Prabhu, Z. Ghadiyali, and A. Akella, “Toward software-defined middleboxnetworking,” in Proceedings of the 11th ACM Workshop on Hot Topics in Networks. ACM,2012, pp. 7–12.[43] T. Koponen, S. Shenker, H. Balakrishnan, N. Feamster, I. Ganichev, A. Ghodsi, P. Godfrey,N. McKeown, G. Parulkar, B. Raghavan et al., “Architecting for innovation,” ACM SIGCOMMComputer Communication Review, vol. 41, no. 3, pp. 24–36, 2011.[44] A. R. Sharafat, S. Das, G. Parulkar, and N. McKeown, “Mpls-te and mpls vpns with openflow,”ACM SIGCOMM Computer Communication Review, vol. 41, no. 4, pp. 452–453, 2011.[45] N. B. Melazzi, A. Detti, G. Mazza, G. Morabito, S. Salsano, and L. Veltri, “An openflow-based testbed for information centric networking,” in Future Network and Mobile Summit(FutureNetw), 2012. IEEE, 2012, pp. 1–9.[46] N. McKeown, T. Anderson, H. Balakrishnan, G. Parulkar, L. Peterson, J. Rexford, S. Shenker,and J. Turner, “Openflow: enabling innovation in campus networks,” ACM SIGCOMM Com-puter Communication Review, vol. 38, no. 2, pp. 69–74, 2008.[47] “Vdma robotics partners with opc foundation to achieve standardized information integrationacross robots/machines to realize requirements of industry4.0 machine-to-machine,” OPCFOUNDATION, 2017. [Online]. Available: https://opcfoundation.org/news/press-releases/vdma-robotics-partners-opc-foundation-achieve-standardized-information-integration[48] S.-H. Leitner and W. Mahnke, “Opc ua service-oriented architecture for industrial applica-tions,” Pervasive and Mobile Computing.[49] “Opc ua and the industrial internet of things,” Engineering Live, 2017. [Online]. Available:http://www.engineerlive.com/content/opc-ua-and-industrial-internet-things47Bibliography[50] F. P. Kelly, A. K. Maulloo, and D. K. Tan, “Rate control for communication networks: shadowprices, proportional fairness and stability,” Journal of the Operational Research society, vol. 49,no. 3, pp. 237–252, 1998.48

Cite

Citation Scheme:

        

Citations by CSL (citeproc-js)

Usage Statistics

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
http://iiif.library.ubc.ca/presentation/dsp.24.1-0363154/manifest

Comment

Related Items