Enhancing Reliability via Checkpointing in Cloud Computing Systems

Ao Zhou, Qibo Sun, Jinglin Li

State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China

* The corresponding author, email： hellozhouao@gmail.com

I. INTRODUCTION

Due to the feature to share and multiplex resources across multiple tenants, cloud computing has gained widespread concern recently[1]. More and more applications are deployed in the cloud computing environment [2, 3].There are a large number of host servers in the cloud computing system. Therefore, the probability of failures is nontrivial based on the statistical analysis [4]. It is a critical challenge to guarantee the service reliability [5, 6].

To tolerate the node failure, the replication technique and the checkpoint technique are extensively adopted in the cloud computing system to enhance the service reliability [7, 8].

[9, 10] propose replication-based methods to tolerate failures. In [9], there are is one replica for each primary virtual machine. The primary virtual machine and its replica are grouped together to form a survival group.The method guarantees robustness of service when at least one virtual machine in a survival group works well. An efficient heuristic optimization algorithm is presented to solve the virtual machine placement problem. In [10],the data replication technique is employed to tolerate the data corruption. The problem is formulated as an integer linear programming problem. To solve the problem in polynomial time, the problem is transformed to a minimum-cost maximum- flow problem.

Checkpoint mechanism saves the execution state of the virtual machine as a checkpoint image periodically during failure-free execution [11, 12]. When a failure event occurs,the previously saved checkpoint image can be used to restart the virtual machine from an intermediate state. Therefore, the amount of lost computation is reduced [13]. When the checkpoint mechanism is adopted, the checkpoint image should be transferred to the storage server periodically. The checkpoint traffic may congest the data center network. To address this problem, delta-checkpoint method is proposed to reduce the datacenter network resource consumption [14]. However, the root layer is the bottleneck of the datacenter network. When the checkpoint image is stored in the central database, the root layer network would become even more congested. Therefore, a distributed delta-checkpoint method is proposed by [15, 16]. The delta-checkpoint method stores the checkpoint images on a neighboring host server of the service providing host server. The checkpoint image transfer only consumes the edge layer network resource and the aggregation layer network resource. However, the switches are also unreliable [17]. Although network redundancy in the root layer and the aggregation layer helps to mask the switches failures from the service,there is no redundancy in the edge layer. Each host server only connects to one edge switch.The checkpoint image may become inaccessible because of the failure of the edge switches.

Therefore, this paper addresses the problem and proposes an optimal checkpoint-based method with edge switch failure-aware (EDCKP). The contribution of this paper is twofold：

(1) We propose a checkpoint-based method to tolerate the host server failure and the edge switch failure. Considering the two types of failures and fat-tree data center topology, the main goal of EDCKP is to ensure the service reliability and minimize the data center network resource consumption.

(2) We provide extensive experiments to evaluate the effectiveness of EDCKP and study the impact of parameters on the reliability of cloud services. The experiment results demonstrate the effectiveness of our EDCKP.

The remainder of this paper is organized as follows： Section 2 introduces the related work.Section 3 presents the system model and gives some de finitions that will be used throughout this paper. Section 4 presents our optimal checkpoint-based method. Section 5 evaluates the performance of the proposed method. Finally, Section 6 concludes the paper.

In this paper, the authors propose EDCKP,a checkpoint-based fault tolerance method for service reliability enhancement in cloud computing system.

II. RELATED WORK

The techniques of checkpoint and replication have been extensively employed to tolerate failures in cloud computing systems [18].

[12, 19] employ checkpoint or replication to tolerate out-of-bid failure. The application provider should bid for the virtual machines in dynamic virtual machine pricing model. Out of bid event occurs when the bid of an application provider is lower than the final price of a virtual machine [20, 21]. Then, the virtual machine is stopped and the physical resource is released. The application providing is interrupted, and the service reliability cannot be ensured. To address the problem, checkpoint interface is provided by the virtual resource provider. The application provider can decide when to take a checkpoint for the obtained virtual machine. To eliminate the effects of outof-bid and reduce the lost computation, [12]takes a checkpoint image when the price rises.To minimize the overhead of checkpoint, [19]attempts to skip unnecessary checkpoint taking. The proposed method skips a checkpoint taking when there is a high chance that the application provider can win current bid. Otherwise, a checkpoint is taken immediately.

In cloud data center, the hardware or software failures can results in the unavailability of virtual machines. Fault-tolerance mechanism should be employed to enhance the service reliability.

[22, 23] propose replication-based methods to tolerate failures. To improve the system reliability and reduce the communication cost,a structure constraint-aware virtual machine placement algorithm is presented in [24]. The problem is formulated as an optimization problem and a four approximation algorithms is proposed to solve the problem. In [9], there is one replica for each primary virtual machine. The primary virtual machine and its replica are grouped together to form a survival group. The method guarantees robustness of service when at least one virtual machine in a survival group works well. An efficient heuristic optimization algorithm is presented to solve the virtual machine placement problem. In [10], the data replication technique is employed to tolerate the data corruption. The problem is formulated as an integer linear programming problem. To solve the problem in polynomial time, the problem is transformed to a minimum-cost maximum-flow problem.[22] proposes a k-fault tolerance virtual machine placement method. Two problems are considered in virtual machine placement. The first problem is how to minimize the required physical servers for satisfying the required response time of the applications. The second problem is how to ensure that the required response time is still satis fied at any k physical server failures. First-fit strategy is extended to solve the k fault tolerance virtual machine placement problem. [23] attempts to study how to improve the reliability of multi-tier applications in cloud computing environment.Both the overhead of reliability enhancement strategy and the data transfer cost are consideration. The problem is formulated as a virtual network embedding and a heuristic algorithm is presented to solve the problem. Similar to[23], [25] attempts to achieve a trade-off between service reliability and network resource consumption for multi-tenant cloud computing.

[14, 16] propose checkpoint-based methods to tolerate failures. To avoid the checkpoint traffic congesting the datacenter network,delta-checkpoint mechanism is proposed by[14] to save the datacenter network resource consumption. The datacenter network adopts a fat-tree topology. When the checkpoint image is stored in the central database, the root layer switches, which are the bottleneck of the datacenter network, may become congested. To address the problem, the distributed delta-checkpoint method is proposed [15].However, the switches are also unreliable. Different from the root layer and the aggregation layer, there is no redundancy in the edge layer.In the distributed delta-checkpoint method, the checkpoint image may become inaccessible when an edge switch fails. We will attack the challenge in this paper.

III. SYSTEM MODEL AND MOTIVATION

3.1 System model

We consider a data center adopting a fat-tree topology [26]. Fig. 1 illustrates the fat-tree data center network. A fat-tree data center network consists of three layers： the root layer,the aggregation layer, and the edge layer. The host servers that share the same edge switch are referred to as “in the same subnet”. The host servers that share the same aggregation switches are referred to as “in the same pod”.Different from the root layer and the aggregation layer, there is no redundancy in the edge layer. The host servers connect to the network through the edge switches.

Similar to the method proposed in [15],distributed delta-checkpoint is employed to tolerate the failures. The checkpoint image is stored in the neighboring server to avoid congesting the upper layer switches. The current state (e.g., the base system, and the RAM disk content) of the virtual machine is saved as a checkpoint image periodically. There are two types of checkpoint image： system checkpoint image and delta checkpoint image. The first checkpoint image taken for a virtual machine is named system checkpoint image. Once the system checkpoint is done, the remaining checkpoint images only contain the modi fied page compared with the last checkpoint. Fig. 2 illustrates the delta-checkpoint method.

Fig. 1 Fat-tree data center network

3.2 Motivation

We begin by using a motivating example to show the research problem of this paper.

Application providers [27, 28] implement and deploy their own applications on cloud.There are a large number of host servers in the cloud computing system. Therefore, the probability of failures is nontrivial based on the statistical analysis. To enhance the service reliability, checkpoint is employed to tolerate the node failure in a cloud data center. Traditionally, the checkpoint image is stored in the central database. As the bottleneck of the datacenter network, the root layer switches may get congested. Therefore, a distributed delta-checkpoint method is proposed. In the distributed delta-checkpoint method, the checkpoint image is stored in the neighboring servers. The method attempts to store the checkpoint image a server which is in the same subnet with the checkpoint image generation server. Hence, the method can avoid using the root layer data center network resource.However, the switches are also unreliable.Although network redundancy helps to mask the switches failures from the service, there is no redundancy in the edge layer. As shown in Fig.1, each host server only connect to one edge switch [29]. When the service providing server and its corresponding checkpoint image storage server are in the same subnet, current execution task should be restarted from the beginning once the connected edge switch fails.We would address the problem and select the optimal checkpoint image storage server for each service providing virtual machine.

There is another problem we should confront in this paper： how to select the recovery server when a failure event has occurred.When a host server fails or an edge switch that connects to a service providing server fails, the system checkpoint image and the delta checkpoint image are merged to a complete checkpoint image before restarting the interrupted virtual machine. We need to select an optimal host server and restart the interrupted virtual machine based on the merged checkpoint image. Therefore, the problem is divided into two sub-problems： edge switch failure-aware storage server selection, and recovery server selection. Different from [15], we will take both the server failure and the edge failure into consideration in checkpoint image storage server selection.

Fig. 2 Delta-checkpoint method

Table I Notations

IV. PROPOSED OPTIMAL CHECKPOINTBASED METHOD

4.1 Edge switch failure-aware storage server selection for checkpoint image

The notations in Table 1 will be used throughout the paper. The storage server selection problem can be formulated as a multi-object optimization problem by the following：

where the first objective is to maximize the service reliability, the second objective is to minimize the network resource consumption,1 when thei-thserver is the checkpoint image storage server,Ris the service reliability.

Algorithm 1: Storage server selection Input: service providing server Ps, the subnet subs that Ps belongs to, the pod pods that Ps belongs to Output: checkpoint image storage server Tserver 1. initialize pmList;2. add all subnets in pods to subList;3. For{ each subi in subList}4. {5. If{ subi is subs}6. continue;7. If{ subi contains an image storage server for the service}8. continue;9. add all servers in subi to pmList;10. sort all servers in pmList by reliability dsc;11. For{ each pmi in pmList }12. {13. If{cdisk pm＞=rdisk}14. {15. assign pmi to Tserver ;16. return Tserver ;17. }18. }19. }20. assign a central storage server to Tserver;21. return Tserver ;

We design a heuristic algorithm to solve the storage server selection. The algorithm is shown in Algorithm 1. We try to select a server that is in the same pod with the service providing server. Because the delta checkpoint image should be generated and transferred periodically, we attempt to select the storage server from the same pod with the source virtual machine. Therefore, we can avoid the packet transferring through the core layer of the data center network. For all servers in the target pod, we try to select the server with the highest reliability.

4.2 Recovery server selection

When a host server fails or an edge switch to which a service providing server connects fails, we need to select a recovery server. The system checkpoint image and the delta checkpoint image are transferred to the recovery server and merged to create a complete checkpoint image. The virtual machine is restarted based on the complete checkpoint image. The problem can be formulated as an optimization problem by the following：

Algorithm 2: Recovery server selection Input: subsckp that Psckp belongs to, podsckp that Psckp belongs to, subdckp that Pdckp belongs to, poddckp that Pdckp belongs to Output: recovery server Rserver 1. If{ podsckp is poddckp }2. {3. add all subnets in podsckp to subList;4. For{ each subi in subList }5. {6. If{ subi is subsckp || subi is subdckp }7. continue;8. If{there is at least one available server in subi }9. {10. select an available server with the highest reliability from subi and assign it to Rserver;11. return Rserver;12. }13. }14. }15. Else 16. {17. If{the size of the system checkpoint image is larger than the size of the delta checkpoint image}18. {19. subα= subsckp, podα= podsckp;20. subβ= subdckp, podβ= poddckp;21. }22. Else 23. {24. subα= subdckp, podα= poddckp;25. subβ= subsckp, podβ= podsckp;26. }27. add all subnets in podα to subList;28. For{ each subi in subList }29. {30. If{ subi is subα}31. continue;32. If{there is at least one available server in subi }33. {34. select an available server with the highest reliability from subi and assign it to Rserver;35. return Rserver;36. }37. }38. clear subList and add all subnets in podβ to subList;39. For{ each subi in subList }40. {41. If{ subi is subβ}42. continue;43. If{there is at least one available server in subi }44. {45. select an available server with the highest reliability from subi and assign it to Rserver;46. return Rserver ;47. }48. }49. }50. select an available server with the highest reliability from other pods and assign it to Rserver;51. return Rserver;

where the objective is to minimize the network recovery server.

We design a heuristic algorithm to solve the problem. The algorithm is shown in Algorithm 2. “Available server” in Algorithm 2 denotes(10)-(14) are satis fied. Because the upper layer link is easier to get congested in fat-tree data center network, we try to avoid the packet with the larger size transferring through the upper layer link. The algorithm firstly decide whether the system checkpoint image and the delta checkpoint image are stored in the same pod. If so, a server that has enough remaining resources is selected from the pod as a recovery server. Secondly, if the size of system checkpoint image is larger than the size of delta checkpoint image, a recovery server is selected from servers in the same pod with the system checkpoint image storage server.Otherwise, a recovery server is selected from servers in the same pod with the delta checkpoint image storage server.

V. PERFORMANCE EVALUATION

In this section, in order to study the performance of EDCKP, we conduct several experiments to compare our EDCKP with several other checkpoint-based methods.

5.1 Simulation setup

In our simulation experiment, a 16-port fattree data center network is constructed. The capacity of the link on the root layer and the aggregation layer is 10 Gbps. The capacity of the link on the edge layer is 1 Gbps [15]. 1000 tasks are generated. The task size is uniformly distributed in [10 hour, 20 hour]. All tasks are randomly allocated to the virtual machines that provide the given service. The checkpoint interval is 20 min. The time to take a system checkpoint image is 60s, and the time to take a delta checkpoint image is 30s. The size of the virtual machine setup image is 1.28 GB.The size of system checkpoint image is 1.78 GB. The size of the delta checkpoint image is(143*log10T-254) MB [15], and T is the checkpoint interval. The failure rate of host server is 0.05. We vary the failure rate of edge switch from 0.01 to 0.05 with a step of 0.01. To study the performance of EDCKP, we compare EDCKP with other two methods, which are as follows：

·NOCKP.No checkpoint-based method is employed. When a failure event occurs, the interrupted task is restarted from the beginning.

·NDCKP.Network topology-aware distributed delta checkpoint-based method [15]. The checkpoint image storage server is selected based on the data center network topology.Total execution time and three types of network resource consumption are employed to measure the performance of different methods.The total execution time and the three types of network resource consumption are de fined by the following：

·The total execution time (TET).The total time the virtual machines take to complete all the submitted tasks.TETis defined by the following：

whereTiis a task,tstart(Ti) is the timeTiis submitted to the cloud computing system, andtend(Ti) is the timeTiis completed by the virtual machine.

·Root layer packet. (RLP)The size of all packets that have transferred through root layer.RLPis de fined by the following：

wherepktiis a packet for transferring the checkpoint image or the setup image,Xiis equal to 1 ifpktihas transferred through the root layer. Otherwise,Xiis equal to 0.

·Aggregation layer packet. (ALP).The size of all packets that have transferred through aggregation layer.ALPis defined by the following：

wherepktiis a packet for transferring the checkpoint image or the setup image,Yiis equal to 1 ifpktihas transferred through the aggregation layer. Otherwise,Yiis equal to 0.

·Edge layer packet. (ELP).The size of all packets that have transferred through edge layer.ELPis de fined by the following：

Fig. 3 Total execution time of all methods

Fig. 4 The size of all packets that have transferred through root layer

wherepktiis a packet for transferring the checkpoint image or the setup image,Ziis equal to 1 ifpktihas transferred through the edge layer. Otherwise,Ziis equal to 0.

5.1 Performance Comparison

The simulation results are plotted in Figs. 3-6.We have the following observations：

· EDCKP considerably reduces the total execution time with the increase of failure rate.The performance of NDCKP is even worse than NOCKP in total execution time. The reason is that NDCKP does not take the edge switch failure into consideration. The service providing server and the checkpoint image storage server are likely in the same subnet. Therefore, the network resource consumption is lower than EDCKP. However, there is no redundancy in the edge layer. Each physical server only connects to one edge switch. When the service providing server and the checkpoint image storage server are in the same subnet, the edge switch failure interrupts the service providing and results in the inaccessibility of the checkpoint image. The interrupted task needs to be restarted from the beginning.The total execution time increases quickly.In addition, much time is taken to re-generate the checkpoint images.

· EDCKP and NDCKP consume less root layer network resource than NOCKP. Taking fat-tree topology into consideration,EDCKP and NDCKP tend to avoid storing the checkpoint image in different pods with the source virtual machine. Therefore, the checkpoint image transfer does not consume the root layer network resource. In addition, NOCKP needs to fetch the virtual machine setup image when a failure occurs.

· EDCKP and NDCKP consume more aggregation and edge layer network resources than NOCKP. The network resource is consumed to transfer the checkpoint image.EDCKP consumes more aggregation layer network resources than NDCKP. Taking the edge switch failure into consideration, EDCKP tends to store the checkpoint image in different subnets with the service providing server. Therefore, more aggregation layer network resources are consumed.

Therefore, among all methods achieving the highest service reliability, EDCKP consumes the least network resource.

VI. CONCLUSION

In this paper, we propose EDCKP, a checkpoint-based fault tolerance method for service reliability enhancement in cloud computing system. The problem is divided into two sub-problems： edge switch failure-aware storage server selection for checkpoint image and recovery server selection. Two optimal server selection algorithms have been proposed to solve the problem. The simulation results show that the proposed checkpoint-based method can efficiently ensure the service reliability and reduce the root layer network resource consumption in cloud computing systems. Our on-going research includes investigating the performance of the methods in different experimental settings and implementing the proposed EDCKP method in a real cloud computing platform.

ACKNOWLEDGMENT

The work presented in this study is supported by Beijing Natural Science Foundation (4174100),NSFC (61602054), and the Fundamental Research Funds for the Central Universities.

[1] M. D. Dikaiakos, D. Katsaros, P. Mehra, G. Pallis,and A. Vakali, “Cloud computing: distributed internet computing for IT and scienti fic research,”Internet Computing, IEEE,vol. 13, no. 5, pp. 10-13, 2009.

[2] X. He, P. Shenoy, R. Sitaraman, and D. Irwin,“Cutting the Cost of Hosting Online Services Using Cloud Spot Markets,” presented at theThe 25th International ACM Symposium on High-Performance Parallel and Distributed Computing(HPDC), 2015.

[3] Y. Ma, S. Wang, P. C. K. Hung, C. H. Hsu, Q. Sun,and F. Yang, “A Highly Accurate Prediction Algorithm for Unknown Web Service QoS Values,”IEEE Transactions on Services Computing,vol. 9,no. 4, pp. 511-523, 2016.

[4] E. Bauer and R. Adams,Reliability and availability of cloud computing: John Wiley & Sons, 2012.

[5] Z. Zheng, T. C. Zhou, M. R. Lyu, and I. King,“Component ranking for fault-tolerant cloud applications,”Services Computing, IEEE Transactions on,vol. 5, no. 4, pp. 540-550, 2012.

[6] Y. Zhang, Z. Zheng, and M. R. Lyu, “BFTCloud:A byzantine fault tolerance framework for voluntary-resource cloud computing,” inCloud Computing (CLOUD), 2011 IEEE International Conference on, 2011, pp. 444-451.

[7] J. Liu, S. Wang, A. Zhou, and F. Yang, “PFT-CCKP:A proactive fault tolerance mechanism for data center network,” in2015 IEEE 23rd International Symposium on Quality of Service (IWQoS), 2015,pp. 79-80.

[8] S. Wang, T. Lei, L. Zhang, C.-H. Hsu, and F. Yang,“Offloading mobile data traffic for QoS-aware service provision in vehicular cyber-physical systems,”Future Generation Computer Systems,vol. 61, pp. 118-127, 2016.

[9] J. Xu, J. Tang, K. Kwiat, W. Zhang, and G. Xue,“Survivable virtual infrastructure mapping in virtualized data centers,” inCloud Computing(CLOUD), 2012 IEEE 5th International Conference on, 2012, pp. 196-203.

[10] J. Lin, C. Chen, and J. Chang, “QoS-aware data replication for data intensive applications in cloud computing systems,” 2013.

[11] T. Knauth and C. Fetzer, “VeCycle: Recycling VM Checkpoints for Faster Migrations,” presented at theProceedings of the 16th Annual Middleware Conference, Vancouver, BC, Canada, 2015.

[12] D. Jung, S. Chin, K. Chung, H. Yu, and J. Gil, “An eき cient checkpointing scheme using price history of spot instances in cloud computing environment,” inNetwork and Parallel Computing,ed: Springer, 2011, pp. 185-200.

[13] B. Nicolae and F. Cappello, “BlobCR: efficient checkpoint-restart for HPC applications on IaaS clouds using virtual disk image snapshots,” presented at theProceedings of 2011 InternationalConference for High Performance Computing,Networking, Storage and Analysis, Seattle,Washington, 2011.

Fig. 5 The size of all packets that have transferred through aggregation layer

Fig. 6 The size of all packets that have transferred through edge layer

[14] í. Goiri, F. Julia, J. Guitart, and J. Torres, “Checkpoint-based fault-tolerant infrastructure for virtualized service providers,” presented at theNetwork Operations and Management Symposium (NOMS), 2010 IEEE, 2010.

[15] N. Limrungsi, J. Zhao, Y. Xiang, T. Lan, H. H.Huang, and S. Subramaniam, “Providing reliability as an elastic service in cloud computing,”inCommunications (ICC), 2012 IEEE International Conference on, 2012, pp. 2912-2917.

[16] A. Zhou, S. Wang, Z. Zheng, C. H. Hsu, M. R. Lyu,and F. Yang, “On Cloud Service Reliability Enhancement with Optimal Resource Usage,”IEEE Transactions on Cloud Computing,vol. 4, no. 4,pp. 452-466, 2016.

[17] P. Bod, I. Menache, M. Chowdhury, P. Mani,D. A. Maltz, et al., “Surviving failures in bandwidth-constrained datacenters,” presented at theProceedings of the ACM SIGCOMM 2012 conference on Applications, technologies, architectures, and protocols for computer communication, Helsinki, Finland, 2012.

[18] S. Wang, A. Zhou, C. H. Hsu, X. Xiao, and F.Yang, “Provision of Data-Intensive Services Through Energy- and QoS-Aware Virtual Machine Placement in National Cloud Data Centers,”IEEE Transactions on Emerging Topics in Computing,vol. 4, no. 2, pp. 290-300, 2016.

[19] S. Yi, A. Andrzejak, and D. Kondo, “Monetary cost-aware checkpointing and migration on Amazon cloud spot instances,”Services Computing, IEEE Transactions on,vol. 5, no. 4, pp.512-524, 2012.

[20] S. Yi, D. Kondo, and A. Andrzejak, “Reducing costs of spot instances via checkpointing in the amazon elastic compute cloud,” presented at theCloud Computing (CLOUD), 2010 IEEE 3rd International Conference on, 2010.

[21] Y. Song, M. Zafer, and K.-W. Lee, “Optimal bidding in spot instance market,” presented at theINFOCOM, 2012 Proceedings IEEE, 2012.

[22] F. Machida, M. Kawato, and Y. Maeno, “Redundant virtual machine placement for fault-tolerant consolidated server clusters,” inNetwork Operations and Management Symposium(NOMS), 2010 IEEE, 2010, pp. 32-39.

[23] M. Shen, X. Ke, F. Li, F. Li, L. Zhu, and L. Guan,“Availability-Aware Virtual Network Embedding for Multi-tier Applications in Cloud Networks,”presented at theProceedings of the 2015 IEEE 17th International Conference on High Performance Computing and Communications, 2015 IEEE 7th International Symposium on Cyberspace Safety and Security, and 2015 IEEE 12th International Conf on Embedded Software and Systems, 2015.

[24] D. Jayasinghe, C. Pu, T. Eilam, M. Steinder, I. Whally, and E. Snible, “Improving Performance and Availability of Services Hosted on IaaS Clouds with Structural Constraint-Aware Virtual Machine Placement,” in2011 IEEE International Conference on Services Computing, 2011, pp. 72-79.

[25] X. Li and C. Qian, “Traきc and failure aware VM placement for multi-tenant cloud computing,”in2015 IEEE 23rd International Symposium on Quality of Service (IWQoS), 2015, pp. 41-50.

[26] A. Zhou, S. Wang, B. Cheng, Z. Zheng, F. Yang,R. Chang, et al., “Cloud Service Reliability Enhancement via Virtual Machine Placement Optimization,”IEEE Transactions on Services Computing,vol. PP, no. 99, pp. 1-1, 2016.

[27] S. Wang, Z. Zheng, Z. Wu, M. R. Lyu, and F.Yang, “Reputation Measurement and Malicious Feedback Rating Prevention in Web Service Recommendation Systems,”IEEE Transactions on Services Computing,vol. 8, no. 5, pp. 755-767, 2015.

[28] S. Wang, L. Huang, C.-H. Hsu, and F. Yang,“Collaboration reputation for trustworthy Web service selection in social networks,”Journal of Computer and System Sciences,vol. 82, no. 1,Part B, pp. 130-143, 2016.

[29] R. N. Mysore, A. Pamboris, N. Farrington, N.Huang, P. Miri, S. Radhakrishnan, et al., “Port-Land: a scalable fault-tolerant layer 2 data center network fabric,”SIGCOMM Comput. Commun. Rev.,vol. 39, no. 4, pp. 39-50, 2009.