• <tr id="yyy80"></tr>
  • <sup id="yyy80"></sup>
  • <tfoot id="yyy80"><noscript id="yyy80"></noscript></tfoot>
  • 99热精品在线国产_美女午夜性视频免费_国产精品国产高清国产av_av欧美777_自拍偷自拍亚洲精品老妇_亚洲熟女精品中文字幕_www日本黄色视频网_国产精品野战在线观看 ?

    Deep Learning Based Attack Detection for Cyber-Physical System Cybersecurity: A Survey

    2022-01-26 00:35:22JunZhangLeiPanQingLongHanChaoChenShengWenandYangXiang
    IEEE/CAA Journal of Automatica Sinica 2022年3期

    Jun Zhang,,Lei Pan,,Qing-Long Han,,Chao Chen,,Sheng Wen,,and Yang Xiang,

    Abstract—With the booming of cyber attacks and cyber criminals against cyber-physical systems (CPSs),detecting these attacks remains challenging.It might be the worst of times,but it might be the best of times because of opportunities brought by machine learning (ML),in particular deep learning (DL).In general,DL delivers superior performance to ML because of its layered setting and its effective algorithm for extract useful information from training data.DL models are adopted quickly to cyber attacks against CPS systems.In this survey,a holistic view of recently proposed DL solutions is provided to cyber attack detection in the CPS context.A six-step DL driven methodology is provided to summarize and analyze the surveyed literature for applying DL methods to detect cyber attacks against CPS systems.The methodology includes CPS scenario analysis,cyber attack identification,ML problem formulation,DL model customization,data acquisition for training,and performance evaluation.The reviewed works indicate great potential to detect cyber attacks against CPS through DL modules.Moreover,excellent performance is achieved partly because of several highquality datasets that are readily available for public use.Furthermore,challenges,opportunities,and research trends are pointed out for future research.

    I.INTRODUCTION

    CYBER-physical systems (CPSs) suffer from cyber attacks when they are increasingly connected to the cyber space.According to [1] published in 2017,more than 30 surveys were published to cover the cybersecurity issue in the CPSs.Cyber attacks have become increasingly sophisticated and prevalent as automated attacking tools,and professional hacking groups have started to get involved.A successful cyber attack against a CPS may be disastrous,catastrophic,or even fatal [2]–[6].However,it is a challenge to defend against cyber attacks on CPSs.Many CPS systems lack cybersecurity mechanisms like message authentication,resulting in challenges to detect false data injection attacks.A lack of universal encryption,especially on the systems employing dated technologies,makes it challenging to defend against eavesdropping attacks.System states need to be referred to detect replay attacks.In addition,the use of dated technology in operation limits the choices of defenses to network traffic in most cases [7].

    Deep learning (DL) [8],[9] delivers superior performance to traditional machine learning (ML) solutions.Whenever there is adequate data,DL models almost deliver excellent results.However,DL models have been slowly applied to solve the CPS cybersecurity issue compared with other fields such as NLP,image processing,software vulnerability [10],[11],and many more [12]–[17].It is also observed that many DL models have been proposed in recent publications to detect CPS cyber attacks.A widely accepted view to explain the difficulty of detecting cyber attacks on CPSs was accredited to the degree of complexity when superposing cybersecurity over CPSs [2].

    There exist a few short-length survey papers on CPS cybersecurity [1],[2],[18],[19].Some papers investigated data-driven methods for detecting cyber attacks against CPS systems [18],[20].However,there is no detailed discussion on applying DL methods to detect CPS cyber attacks.A short survey was provided in [18] with a four-step framework to apply DL methods on CPS issues,including cybersecurity,adaptability,recoverability,and many more,without a specific focus on cybersecurity.Furthermore,most of the cited works in [18] were published between 2012 and 2016,but this survey includes most papers between 2017 and 2021.A survey of surveys was presented in [1] without relevance to DL models.A comprehensive survey on the cyber attacks against CPSs was presented in [20] without investigating the DL models.Various methods of detecting cyber attacks in the CPSs were summarized in [2] without using DL methods.A comprehensive list of CPS attacks and challenges were provided in [19] but overlooking ML,or DL approaches.A cybersecurity analysis framework was proposed in [21]without utilizing the rich sources of available data.A recently published survey in [22] presents cybersecurity control and state estimation from active and passive defence perspectives.

    Fig.1.The DL driven methodology for CPS cybersecurity considers the essential needs for training robust and usable DL models in the context of cyber attacks against the CPS systems.

    We aim to review current research works on the advances of DL driven solutions for detecting cyber attacks in the CPS domain.It provides an overview for readers to quickly understand and step into the field by following our six-step DL driven methodology.Our six-step methodology considers the complete cycle of DL application from broad scenarios to performance evaluation.This paper caters to researchers,practitioners,and students interested in building DL-based cybersecurity applications in CPSs.The key contributions of this survey are three-fold:

    1) We conduct an up-to-date review of detecting cyber attacks in CPSs using DL models and propose a six-step methodology to position and analyze the surveyed works.

    2) We provide an overview for the state-of-the-art solutions with preservation of technical details.

    3) Based on the methodology,we discuss the challenges and future research directions.

    The rest of this survey is organized as follows: Section II proposes a research methodology for deep learning driven CPS cybersecurity.Section III presents the reviews on stateof-the-art research.Section IV discusses the research challenges and future work.Finally,Section V concludes this survey.

    II.RESEARCH METHODOLOGY

    Our methodology represents a deep understanding of the surveyed papers.The process consists of six steps,including CPS scenario analysis,cyber attack identification,DL problem formulation,DL model construction,data acquisition,and performance evaluation.Fig.1 shows a process of detecting cyber attacks in the context of a CPS by using DL models.For example,a smart grid may suffer from erroneous controls derived by electric load forecasts [20],[23].Falsely injected messages containing maliciously crafted information need to be identified and eliminated before committing the prediction process.A stacked AutoEncoder(AE) proposed in [24] may serve as a reliable regressor to predict the energy load on the system.The chosen AutoEncoder was subsequently trained with sufficient simulation data.At last,the DL model delivered excellent prediction results with the mean absolute percentage error of 3.51% on annual predictions.

    A.Step I: CPS Scenario Analysis

    The normal operations of CPSs rely on several important factors,including dependability,real-time operation,fault tolerance,cybersecurity,and many more.We must consider these requirements holistically.Dependability consists of service availability and reliability to minimize the system downtime; real-time operation is a critical factor for maintaining the system operation when the inputs and environment rapidly change; fault tolerance requires that the critical components of the system have sufficient backups to prevent the system from shutting down; and cybersecurity requirements are becoming more and more prominent when many CPSs are connected to the cyber space to improve the quality of system control and the overall level of quality of service.According to Mitchellet al.[2],there are four primary categories of characteristics of CPS intrusion detection,including physical process monitoring,closed control loops,attack sophistication,and legacy technology.

    Physical process monitoring:Physical properties of a CPS should be constantly monitored to identify any anomalies of the system because many physical processes of the CPS follow the laws of physics.

    Closed control loops:CPS events are significantly more regular and predictable than user-triggered events because many CPS events are driven by the preset feedback-based controllers.

    Attack sophistication:Sophisticated cyber attacks are increasingly popular in the CPS context because the potentially huge payoff for a successful cyber attack may bring sensitive information,valuable intelligence for military or finance operations,and many more.

    Legacy technology:Legacy hardware commonly used in the CPSs cannot interact with software-defined control because of the existing mechanical and hydraulic control.

    Analyzing the characteristics of a CPS scenario will help craft an appropriate cybersecurity problem.The involvement of physical signals enriches the input variables and complicates the design of any security solutions for CPSs.Although the behaviors of simplified proof-of-concept systems are relatively regular and predictable,real-world systems often operate in a noisy environment with unprecedented cyber threats.

    B.Step II: Cyber Attack Identification

    Upon completion of identifying the CPS scenario,we need to define a set of appropriate cyber attacks associated with CPS characteristics.For example,we will have more confidence to detect the falsely injected network packets if physical processes of the CPS components are properly monitored; cyber attacks like replay attacks may be detected on a CPS with a closed control loop; unknown attacks and sophisticated attacks like web attacks need to be considered if there is any concern of attack sophistication; denial of service(DoS) attacks and replay attacks are more prevalent in the presence of legacy technology.

    Based on the surveyed articles,we identify many common cyber attacks.Some frequent cyber attacks against the industrial control network include false data injection attacks,DoS attacks,replay attacks,and alike; and some frequent cyber attacks against the software-based controllers with a centralized server include brute force attacks,botnets,web attacks,heartbleed attacks,infiltration attacks and many more.Effective and efficient detection of these cyber attacks can be leveraged by using DL models,so we will need to translate the cybersecurity problem to the ML domain.

    C.Step III: ML Problem Formulation

    After aligning the cyber attacks to the CPS characteristics,the research problem can be translated to the ML/DL domain.ML is defined in [25] as “A computer program is said to learn from experienceEwith respect to some class of tasksTand performance measureP,if its performance at tasks inT,as measured byP,improves with experienceE.” DL is referred to in [8] as solving a complex problem by using a hierarchy of more straightforward concepts without too much human intervention.The definition of ML is general,and we will implement an ML solution in multiple steps.In this step,we need to define the taskT,including classification,clustering,regression,etc.A classification task requires that the trained model allocates its output to a pre-defined set of “classes”which could be the specific cyber attack categories; a clustering task often requires that the trained model allocates its output to a few “clusters” which could indicate normal traffic or attack traffic; a regression task is also known as a prediction task which requires the trained model to predict some numerical values.For example,a classification problem was found in [26] to differentiate cyber attack types; a clustering problem was found in [27] to separate covert messages from the normal messages; and a regression problem was set in [24] to predict the electric load in a smart grid.The choice of the ML tasks will impact the construction of the DL models.

    D.Step IV: DL Model Customization

    The DL model is constructed by selecting an architecture suitable for the research problem and optimizing parameters.The choice of DL models should be made according to actual needs.For example,autoencoders are good at translating the input data so that they are suitable for learning the representations of the data often required in prediction or regression tasks [24]; convolution networks (CNNs) and other models are usually used in classification tasks [28].

    The configuration of the chosen DL model also depends on the available data.A DL model with a large number of neurons per layer will almost always require more data than a DL model with the same design but a few neurons per layer.Some trade-offs can also be made by stacking more hidden layers inside the DL model instead of expanding the layer size.The ways and insights of the customizing model can be explored based on a thorough understanding of DL algorithms and CPS cybersecurity data.Furthermore,we can achieve improvement at various levels by combining the choice of DL models with a specific research problem.

    E.Step V: Data Acquisition for Training

    Data acquisition is a critical step for training DL models.The quality and quantity of data determine the effectiveness of solving the research problem.Also,data can serve as the source for setting up ground truth and affect the prediction model’s performance.One of the simplest methods to collect data is through simulation.This method is often used to generate datasets for power grids such as IEEE 9-bus,14-bus,30-bus,and 118-bus systems in Matlab.The other method relies on several existing datasets harvested by other researchers.These datasets include the SWaT dataset1http://itrust.sutd.edu.sg/dataset/SWaT,the SCADA IDS dataset2https://sites.google.com/a/uah.edu/tommy-morris-uah/ics-data-sets,the CICIDS2017 dataset3https://sites.google.com/a/uah.edu/tommy-morris-uah/ics-data-sets,the UNSWNB15 dataset4https://www.unsw.adfa.edu.au/unsw-canberra-cyber/cybersecurity/ADFANB15-Datasets/,and the KDD99 Cup dataset5http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html.

    Different cyber attacks were included in the datasets:

    1) The SWaT dataset contained eleven days of network traffic collected from a scaled-down water treatment plant.And there were no attacks during the first seven days.It includes 36 types of cyber attacks that are most commonly seen in today’s CPS systems.

    2) The SCADA IDS dataset contained network traffic logs of a SCADA IDS system.It includes seven types of cyber attacks — injection random response packets,hide the real state of the controlled process,inject malicious state commands,inject malicious parameter commands,inject malicious function code commands,DoS attack,and recon attack.

    3) The CICIDS2017 dataset [29] contained network traffic logs collected from an industrial control system.It includes six types of cyber attacks — brute force attacks,botnet,DoS attack,web attack,heartbleed attack,and infiltration attack.

    4) The Bot-IoT dataset [30] contained network traffic logs collected from an IoT setup.It includes three types of cyber attacks — infiltration,DoS attack,and information theft.

    5) The power system dataset [31] contained the network traffic collected from a power grid.It includes three cyber attacks — data injection,remote command injection,and replay attacks.

    6) The UNSW-NB15 dataset [32] contained the network traffic information extracted form a 100 GB packet capture dump.It includes nine cyber attacks — DoS attacks,exploits,recon attacks,worms,fuzzers,web penetration attacks,backdoors,shellcode,and generic attacks against block ciphers.

    7) The KDD99 Cup dataset [33] contained the network traffic information presented at the ACM SIGKDD conference in 1999.It includes four cyber attacks — DoS attacks,unauthorized accesses,privilege escalation,and probing attacks.

    F.Step VI: Performance Evaluation

    The last step is used to determine whether the DL model meets our expected objectives through performance evaluation.The performance is usually measured according to various metrics.We divide the performance metrics into two categories according to the tasks: 1) For prediction or regression tasks,a number of error metrics are used to measure the performance,including mean absolute error(MAE),mean relative error (MRE),root of mean squared error (RMSE),and mean absolute percentage error (MAPE).2) For classification or clustering tasks,there are a few standard metrics,including accuracy,recall,precision,false positive rate (FPR),F1 score.And occasionally,graphical plots like receive operating characteristic (ROC) curves are used by plotting TPR asy-axis and FPR asx-axis to depict the trade-offs between benefits and costs.Finally,area under ROC curve (auROC) is used to indicate the cumulative strength of a particular ROC curve.

    In many cases,FPR poses challenges for the DL models because the false alarms almost always result in excessive costs associated with manual verification.And it is always challenging to detect the rare or even unknown attacks as proven in [34] so that most of the surveyed literature aimed to maximize the TPR while minimizing the FPR.On the other hand,the error rate can be tolerated more generously in the regression task than in the classification task.By leveraging comprehensive evaluation metrics,we can decide whether the outputs of a specific DL model are satisfactory.Whenever there are unsatisfactory results,the process should be repeated with proper adjustments.

    III.CPS CYBERSECURITY WITH DEEP NEURAL MODELS

    This Section surveys the relevant literature of detecting cyber attacks in the context of CPSs by following the research methodology described in Fig.1.In particular,the body of the literature is divided into two parts according to the DL architectures,which will be elaborated below.

    A.Representation Learning for Attack Detection

    An AutoEncoder-based (AE) model was proposed in [26] to preserve privacy information in the context of smart power networks.Data privacy violations are becoming more and more popular in smart power networks.It is challenging to defend against inference attacks,because the smart power networks represent the CPS characteristics of physical process monitoring,closed control loops,attack sophistication,and legacy technology.The research problem of defending against inference attacks was translated into a classification problem in the ML domain.A Variational AutoEncoder (VAE) was proposed to provide transformed features for the ultimate classification task and transform raw data into an encoded format for preventing inference attacks.A VAE is a feedforward model used for encoding an input into new data codes using a set of weighted parameters.The VAE consisted of one input layer,four hidden layers,and one output layer.The transformed data from the output layer were written to the database for publication.Two datasets were used to evaluate the VAE,i.e.,the power system dataset [31] and the UNSWNB15 dataset [32].The Power system dataset is a multi-class dataset involving 37 scenarios that include 8 natural events,28 intrusive events,and 1 no event; and the UNSW-NB15 dataset includes a combination of current normal and attack records.300,000 random samples of legitimate and attack observations were chosen from each dataset for assessing the performance of the proposed framework.Although the VAE was only employed as a part of the intrusion detection system,its strength was demonstrated while transforming complex data into a simple form.The VAE achieved 0.921 for accuracy and 0.005 for loss on the power system dataset,and 0.998 for accuracy and 0.0001 for loss on the UNSW-NB15 dataset.

    An AE-based solution was proposed in [35] to detect various cyber attacks in the context of industrial control networks.There exist many kinds of cyber attacks when control networks are connected to the internet.The research problem generally reflects the CPS characteristics of attack sophistication.And it was translated to a classification problem in the ML domain.Hence,a 7-layer AE consisted of an input layer,four hidden layers,and an output layer.The input layer had 41 units corresponding to the feature space’s dimension,and the output layer had five units corresponding to the five types of network traffic.In particular,the last hidden layer was a softmax layer to provide the stability of the model.The AE was trained using the NSL-KDD dataset [33].As an early study,the proposed AE suffered low performance in detecting small classes like probe attack and remote attack.The stacked AE achieved 0.978 for accuracy over the five categories.The model achieved an F1 score of 0.9683.

    An AE-based model was proposed in [24] to detect cyber attacks in the context of smart grids.One big challenge is a large number of control parameters.The smart power networks represent the CPS characteristics of physical process monitoring and legacy technology.The smart grid’s essential controller is based on state estimation,so the lower and upper bounds of each state variable need to be predicted as accurately as possible.Hence,this research problem was translated into a regression problem in the ML domain.A stacked AE (SAE) was proposed to process the smart grid data.The SAE consisted of an input layer,three vanilla AEs,and a logistic regressor as the output layer.The SAE was trained with simulated data representing IEEE 9-bus,14-bus,30-bus,and 118-bus systems.Overall,the SAE in this study achieved excellent results in predicting the electric load forecast.The mean absolute percentage error (MAPE) was used to evaluate the SAE’s accuracy.And the SAE achieved a MAPE of 3.51% on an annual prediction and outperformed the baseline models like SVM and BP.Despite the SAE model’s simplicity,the empirical studies showed its applicability and consistency in performing load forecasts.

    Another AE-based model was proposed in [36] to detect Phasor measurement unit data manipulation attacks (PDMAs)in smart grids.PDMAs are challenging to be detected because of the similarity between PDMAs and man-in-the-middle attacks with infiltration of communication networks.This problem represents the CPS characteristics of physical process monitoring and attack sophistication.The main idea was to detect anomalies based on the normal operation patterns from the data collected from the PMUs with PDMA-free measurements in a distributed manner.Hence,the research problem was translated to a regression problem in the ML domain.A deep AE (DAE) was constructed by stacking four RBM models.Training the deep AE required multiple stages by fine-tuning the intermediate RBMs.The input layer took 108 numerical features,and the output layer is a regressor.The dataset was collected from a simulated IEEE 9-bus system and had 250 000 records.The deep AE was trained by using 200 000 benign records,20 000 records were reserved as the validation dataset,and the testing dataset consisted of 30 000 samples with half from attack records.The studies showed that the deep AE outperformed the baseline models like OCSVM,C4.5,MLP,SVM,and kNN.The DAE achieved 0.941 for accuracy,0.996 for precision,0.886 for recall,and 0.9038 for F1 score.Despite the success of using the deep AE in this study,it is challenging to obtain benign data from real-world power networks,and new methods may need to be explored.

    An AE-based model was proposed in [37] to detect attacks against physical measurements in the context of smart grids.This problem represents the CPS characteristics of physical process monitoring and legacy technology.It is challenging to derive useful features for intrusion detection in a noisy environment in a real-world factory.Hence,the research problem was translated into a classification problem in the ML domain.A stacked denoising AE (SDAE) was proposed to learn the advanced features from the input data.And the learned features were fed to an ELM for classification.Simulated data from gas turbines were collected and used to train the model.The proposed model achieved excellent results with an FPR of 0.000006,which was significantly below the required FPR of 0.01.The SDAE was used as a part of the IDS model but demonstrated its strength in extracting useful features to represent physical measurements from a noisy environment.

    An AE-based model was proposed in [28] to detect cyber attacks in the context of the industrial control systems.This problem represents the CPS characteristics of physical process monitoring,attack sophistication,and legacy technology.Hence,the problem was translated into a classification problem in the ML domain.An AE was proposed to extract features for a 1D CNN classifier.The AE consisted of five layers,including an input layer,a corruption layer applying Gaussian noise to the input,a fully connected layer with an activation function,an encoder layer,a decoding layer as the output layer to generate the extracted feature.The SWaT dataset was used to train the model.In particular,the training time for the AE was less than half-second,which was significantly faster than the 1D CNN model.The AE achieved 0.890 for precision,0.827 for recall,and 0.844 for F1 score.In summary,the AE model was validated as a powerful and efficient method to extract useful features.

    An LSTM autoencoder architecture was proposed to detect cyber attacks in the context of the autonomous vehicles (AVs)[38].AVs are linked together by using communication technologies,and thus are vulnerable to network attacks,such as Denial of Service,replay ans spoofing attacks.Such attacks can be inferred from network traffic.Authors designed an LSTM autoencoder to detect these cyber attacks.Statistical features from network traffic were extract to represent the activities of AVs.The designed neural network architecture was consisted of two types layers,LSTM and fully connected layer.A number of LSTM layers were used to encode the representation of the transformed likelihood stream.Then the reconstructed output was produced by the fully connected layer.Two datasets,i.e.,Car Hacking dataset and UNSWNB15 were used to evaluate the proposed scheme.In particular,the proposed LSTM based autoencoder achieved 0.99 for precision,1.0 for recall,and 0.99 for F1 score in the Car Hacking dataset.While on UNSW-NB15 dataset,the proposed scheme achieved 0.1 for precision,0.97 for recall,and 0.98 for F1 score.In a word,this work can successfully detect multiple types of attack vectors.

    Remark 1:Research works employing AE-based architecture were summarized in Table I.Most of them focused on smart grids or power network systems.Due to the difficulties in smart grids’ control systems,most AE models were used to learn the useful features of an intrusion detection system or predict the electric load as an indicator of cyber attacks.Moreover,the AE models were relatively small in size,so that they could be trained in a short amount of time.

    B.Cyber Recognition with Deep Learning Methods

    1) Cybersecurity Pattern Recognition with Deep Neural Networks (DNNs):A DNN-based model was proposed in [39]to learn the communication patterns between electronics control units (ECUs) in the context of in-vehicular network security.The security of communication messages among ECUs is vital because a group of ECUs can control and monitor a vehicle’s status during a maneuver.It is challenging to ensure cybersecurity because most communications between ECUs are through the controller area network protocol,which has no support for authentication or integrity check.Specifically,fake packets injected into the open communication channel through the controller area network protocol pose severe cybersecurity risks.Detecting the fabricated or modified packets in the vehicular setup needs to meet the requirements of physical process monitoring and legacy technologies.This intrusion detection problem was translated into a binary classification problem in the ML domain.That is,statistical features were extracted from highdimensional CAN packet data through a dimension reduction process to represent the normal and attack packets.A 5-layered DNN model was constructed based on a standard DBN model by adding a binary classification layer as the final output layer.The DBN’s coefficient weights were determined through an unsupervised pre-training process,but the final DNN model was trained with a bottom-up supervised manner.During each simulation round,a total of 200 000 packets were generated by the Open Car Test-bed and Network Experiments(OCTANE) generator.A 70:30 split was made to divide training and testing sets.Many experiments were conducted by varying the layers of the DNN model from 5 to 11 to investigate the trade-offs between performance and efficiency.The empirical results demonstrated the effectiveness of the proposed DNN model while comparing it with ANN and SVM.The best performance was achieved as 0.978 foraccuracy,0.016 for false positive rate,and 0.028 for false negative rate.Given the detection ratio of over 99%,the proposed DNN model showed good potentials to detect fake packets on vehicular networks despite that the DNN models’efficiency with more than five layers needed to be improved to meet the real-time requirements.

    TABLE IRESEARCH WORKS EMPLOYING AUTOENCODERS (AES)

    Another DNN-based model was proposed in [40] to learn the network traffic patterns in the electric power grid context.The cybersecurity of an electric power grid largely depends on state estimation underpinning critical control processes for the grid.It is challenging to detect false data injection attacks against the state estimation because a skilled cyber attacker may disguise the injected data stealthily with the inside knowledge of system topology.Such successful attacks may blackout an entire region due to the falsely impacted state estimation because the injected data value is progressively added to the legitimate signal and the Gaussian noise values.Detecting the injected data fed to the state estimation model needs to meet the requirements of physical process monitoring and closed control loops in a power grid.Hence,this intrusion detection problem was translated into a binary classification problem with the objective function of simultaneously minimizing the number of false positives and false negatives.A series of measurement vectors were created for a specific time slot so that a compromised vector contains any injected component corresponding to a false data inject attack.Four variations of DNN models were constructed with different settings — 1 or 3 hidden layers,100 or 150 neurons per hidden layer,whether to use L1 regularization.The DNN models were trained with the standard stochastic gradient descent approach using the back-propagation method.The activation function was tanh.The most accurate DNN model was also the most complex among all the four DNN models.In terms of accuracy,the DNN model of 3 hidden layers with 150 neurons on each layer without L1 regularization outperformed generalized linear models,gradient boosting machines,a distributed random forest classifier,and the other three DNN models.The best performance was 0.9802 for precision,0.9895 for recall,0.9852 for F1 score,and a low false alarm rate of 0.1840.The proposed DNN model demonstrated the effectiveness of a simulated IEEE 14-bus power grid without testing realistic datasets generated by Real-time Digital Simulation (RTDS) and physical testbeds.

    A DNN-based model was proposed in [41],[42] to detect the anomalies in the context of secure water treatment(SWaT).The SWaT system represents many challenges of securing CPSs,including physical process monitoring,closed control loops,attack sophistication,and legacy technologies.A divide-and-conquer strategy was employed in this scenario by separating different sets of sensors and actuators into groups according to their functionalities.Thereafter,the high dimension and complexity of detecting anomalies were mitigated.Data points with a low probability were regarded as outliers because common processes in the system usually generated the normal data points.The proposed DNN model included an LSTM layer and 100 intermediate layers.The LSTM layer predicted the actuator’s position based on its historical positions.Each hidden layer was fully connected to an output layer with a bi-linear function.The cost function was defined by the cross-entropy of the real probability distribution and the predicted probability distribution.The dataset was the log entries collected from 51 sensors and actuators of a testbed for 11 days.According to [41],[42],the DNN detected 13 of the total 36 scenarios,and the DNN model achieved 0.98295 for precision,0.67847 for recall,and 0.80281 for F1 score.However,it took two weeks to train the DNN model and 8 hours to complete testing the data.Due to its inefficiency,DNN has limited use for real-world applications.

    A DNN-based model was proposed in [27] to detect covert message transmission in the context of a chemical process plant.The covert messages containing critical control information exfiltrated from the actuators can be used to detect anomaly operations on hardware devices.Because the covert channel was established without any modification to the system,conducting data analytics on the covert channel had to involve CPS characteristics,including physical process monitoring and closed control loops.It is challenging for simplistic solutions to detect the messages transmitted via a covert channel when analog emission from the physical instrument was used to disguise the existence of the messages.Hence,the problem of detecting covert messages was translated to a clustering problem.A 10-layers DNN received the inputs from the digitized audio samples and produced binary outputs to indicate whether there was a covert message transmission or not.The ten layers included two dropout layers,four linear layers,three ReLU activation functions,and a tanh activation function.The DNN model was trained using Adam optimizer to maintain the learning quality.The dataset consisted of 9 minutes of audio recordings recorded by a Hardware-In-The-Loop (HITL) simulator.Due to the nature of analog signals,the performance of the DNN model was measured by an accuracy of 0.95 on testing data in online operation.The successful application of the DNN model opened many opportunities to develop and deploy real-time monitoring mechanisms over critical actuators using audio sampling and covert channels.

    2) Cybersecurity Pattern Recognition with Convolutional Neural Networks (CNNs):A CNN-based model was proposed in [43] to detect cyber attacks in the context of an industrial water treatment plant.Because the SWaT dataset was used in this study,it inherited many CPS characteristics,including physical process monitoring,closed control loops,attack sophistication,and legacy technologies.An anomaly detection approach was used for improving the detection rate of the cyber attacks on the SWaT dataset.A 1D CNN model was constructed by stacking a set of 1D convolution layer,a ReLU function,and a 1D max-pooling layer before applying to flatten,dropout,and a final fully connected layer.Among all the 36 different attack scenarios,the CNN model successfully detected 31.The CNN model achieved 0.912 for precision,0.861 for recall,and 0.886 for F1 score.When using 8 layers,the 1D CNN model reached 0.967 as the AUC value with only 15 535 parameters.This model’s training time and testing time for one epoch were 88 seconds and 47 seconds,respectively.Based on the positive results of this empirical study,the proposed CNN model achieved excellent results in detecting cyber attacks on the SWaT dataset and demonstrated good potentials in real-world deployment.

    A CNN-based model was proposed in [44] to detect message injection attacks in the vehicular networks.Due to the lack of security protections to the controller area network(CAN),network packets with malicious contents can be easily injected to the CAN bus resulting in gaining control of the vehicle.The detection of malicious packets on a physical vehicle needs to be conducted according to the CPS requirements.Two Raspberry Pi devices acting as a listener and an attacker were connected to an operational passenger vehicle via the OBD-II port to validate the model’s effectiveness.The attacker had four attacking modes — DoS attack,fuzzy attack,drive gear spoofing attack,and engine RPM gauge spoofing attack.It is challenging to effectively and efficiently detect all of the attacks in real-time.A deep CNN model named Inception-ResNet was employed with two blocks of convolution layers,pooling layers,a fully connected layer,and a softmax layer before generating the final output.The activation function was set to the ReLU function.The whole dataset was split into four sub-datasets according to the attack scenarios.The CNN model outperformed other classifiers,including LSTM,ANN,SVM,kNN,NB,and Decision Trees.Each attack lasted for 3 to 5 seconds during the four recording sessions of forty minutes each.Thus,each dataset contained 300 injection attack instances.The number of messages in the four attacking scenarios are 3 078 250 (DoS normal),587 521 (DoS attack),3 347 013 (fuzzy normal),491 847 (fuzzy attack),2 766 522 (gear normal),597 252 (gear spoofing),2 290 185 (RPM normal),and 654 897 (RPM spoofing).False negative rate (FNR) and error rate (ER) were used to measure the CNN model’s performance.In terms of FNR,the CNN model achieved 0.06,0.07,0.10,and 0.24 for gear spoofing,RPM spoofing,DoS attack,and fuzzy attack,respectively; in terms of ER,the CNN model achieved 0.03,0.04,0.05,and 0.18 for DoS attack,RPM spoofing,gear spoofing,and fuzzy attack,respectively.The empirical studies showed that the CNN model was a promising technique for detecting false message injection attacks to vehicular networks.

    A federated deep learning scheme (DeepFed) in [45],to detect cyber threats targeting industrial CPSs.Li et al.firstly designed a CNN-GRU based intrusion detection model.Second,they took account into the federated learning scenario,and built a framework to allow multiple industrial CPSs to build an intrusion detection model together.Lastly,they applied a secure communication protocol based on Paillier cryptosystem to preserve the privacy of model parameters.

    Industrial CPSs are not only targeted by traditional cyber threats,such as DoS attacks,but also by some cyber threats which are customized to industrial systems,such as response injection attacks.Furthermore,in the federation based CPS cyber threats detection framework,eavesdropping attacks on data sources and model parameters are emerging.Thus,the authors proposed DeepFed,which was based on CNN-GRU to detect such threats.The model architecture was consisted of a CNN module,a GRU module,an MLP module,and a softmax layer.To evaluate the performance of DeepFed,they ran experiments on a real-world dataset collected from a gas pipelining system,with 80% splited for training and the rest 20% for testing.The performance varied when different number of local agents (K) were in consideration.If K = 3,DeepFed could achieve an accuracy,precision,recall,F-score of 99.20%,98.86%,97.34%,and 98.08%.Overall,all metrics could reach over 97%.The experiments demonstrated the effectiveness of DeepFed to detect different types of cyber threats to industrial CPSs.

    3) Cybersecurity Pattern Recognition with Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM)Models:An LSTM-based model was proposed in [46] to identify anomaly sensor behaviors in the water treatment plant context.Detecting the cyber attacks in a real plant is challenging because of the complex setups of the sensors and actuators representing many CPS characteristics,including physical process monitoring,closed control loops,attack sophistication,and legacy technologies.Specifically,the anomaly behaviors and the sensors under attack needed to be identified at the same time.The LSTM model using a cumulative sum was proposed to learn the sensors’ behaviors to achieve a low false positive rate.The LSTM model consisted of three LSTM stacks with 100 hidden units each.The cumulative sum of the sequence predictions was introduced to reduce false positives because of their capabilities of indicating small deviations over time.And its loss function was a mean-square loss function.Ten attack scenarios of the SWaT dataset was used for training the LSTM model.The attack scenarios were made up of six single-stage-single-point attacks,two single-stage-multi-point attacks,one multi-stage-single-point attack,and one multistage-multi-point attack.It took about 24 hours to train the LSTM model,which is a big drawback.Nine out of ten attack scenarios were detected without listing detailed results for specific performance metrics.This work was an early study of applying DL models on the SWaT dataset with only limited success.Nevertheless,it showed good potentials for applying DL models for detecting cyber attacks against CPSs.

    An LSTM-based model was proposed in [47] to learn the traffic pattern generated by unknown or zero-day attacks in the context of a control system for a gas pipeline.Detecting unknown or zero-day attacks is challenging because most intrusion detectors on industrial control systems are manually configured for specific protocols and systems.A hybrid strategy of combining packet contents and temporal dependencies and a fast signature-based Bloom filter were used to triage anomaly traffic based on the network packets’contents.A subsequent and slow LSTM model was dedicated to capturing the unknown or zero-day attacks based on the time-series information.The proposed LSTM model consisted of two LSTM layers and a softmax layer.Each LSTM layer had 256 fully connected neuron cells,and the output softmax layer had 613 bits to match the length of signatures used by the Bloom filter.A dataset of network logs collected from the gas pipeline was used to train the model,where there were seven types of cyber attacks with a total of 214 580 normal network packets and 60 048 attack packets.Combining a fast Bloom filter and a slow but powerful LSTM model demonstrated its success in detecting cyber attacks in a small SCADA system.Regarding the detection ratio (recall),the hybrid model outperformed the other six classifiers for six out of seven attacks,including Bloom filter,Bayesian network,SVDD,isolation forest,PCA-SVD,and Gaussian mixture model.The proposed model also achieved 0.92 for accuracy,0.94 for precision,0.78 for recall,and 0.85 for F1 score.This study showed the promising application of the LSTM model to detect time-series anomalies supplemented by an efficient and lean model like Bloom filter.

    An RNN-based model was proposed in [48] to detect various cyber attacks in the context of smart grids.There are various cyber attacks against smart grids,such as DoS attacks,data infiltration,and so on.Detecting all the attacks on a large scale smart grid network is challenging because of the CPS characteristics in terms of attack sophistication and legacy technology.Hence,a vanilla RNN model was trained by using the truncated backpropagation through time (BPTT)algorithm.Three datasets were fed to the RNN model to cover a wide range of cyber attacks.In particular,the CICIDS2017 dataset [29] included brute force attacks,botnets,DoS attacks,web attacks,heartbleed attacks,and infiltration attacks; the Bot-IoT dataset [30] consisted of infiltration,DoS attack,and information theft; and the power system dataset [31] included data injection,remote command injection,were used to replay attacks.The CICIDS2017 dataset contains 2 830 743 records,the Bot-IoT dataset 73 360 900 records,and the power system dataset 78 404 records.Experiments were conducted on the three datasets separately,and the results showed that the RNN model outperformed the benchmark classifiers like SVM,random forest,and NB in terms of mean false positive rate —0.00986 for the CICIDS2017 dataset,0.01281 for the Bot-IoT dataset,and 0.03986 for the power system dataset.In terms of accuracy,the RNN model achieved 0.98941 for the CICIDS2017 dataset,0.99912 for the Bot-IoT dataset,and 0.96882 for the power system dataset,respectively.Although the proposed vanilla RNN performed well across the dataset,its integration with the blockchain component of the DeepCoin system for detecting fraudulent transactions remains unclear but may inspire further research works to use blockchain technologies together with DL models.

    An LSTM-based model was proposed in [49] to detect anomalous automatic dependent surveillance-broadcast (ADSB) messages transmitted between airplanes and control towers.A typical ADS-B message includes an aircraft’s flight information,such as flight number,speed,GPS coordinates,altitude,and many more.Due to historical design and implementation,the ADS-B system is an open messaging system without authentication or encryption.Therefore,it is challenging to defend ADS-B systems against malicious message injection attacks in the aviation industry because the ADS-B messages are transmitted very frequently at an average rate of 4.2 messages per second.Detecting the spoofed ADS-B messages represents the CPS characteristics of legacy technology.Hence,the problem was translated into an unsupervised anomaly detection problem in the ML domain.An LSTM model was proposed to capture the message sequences to detect anomalies according to estimated credibility scores.The LSTM model consisted of two vanilla RNN models stacked as an encoder-decoder paradigm.The LSTM was trained to reconstruct sequences of benign messages with minimal errors,where the reconstruction error score was calculated by using the cosine similarity.A largescale flight tracking dataset namedFlightradar24was used to train the LSTM model using flight data collected from 13 international airports.The LSTM achieved 1.00 as recall,0.03 as FPR,and 0.99 as TPR.The LSTM outperformed the baseline anomaly detection methods,including HMM-GMM,one-class SVM (OCSVM),local outlier factor (LOF),isolation forest (IF),and DBSTREAM.In particular,the spoof messages through RND and ROUTE attacks were detected almost instantaneously,but the SHIFT Down and SHIFT Up attacks were detected with some delay.The LSTM model achieved excellent results in detecting the spoofing messages.

    An LSTM-based model was proposed in [50] to predict multimedia data requests transmitted in the cyber-physical industrial networks.Multimedia data as the carrier for many cyber attacks are manifold,including image files,audio files,and many more.The prediction of multimedia data through a cache resource allocation system became a new proposal to defend the underlying network,representing the CPS characteristics of attack sophistication.Hence,this research problem was translated as a regression problem in the ML domain.A vanilla LSTM model was proposed to capture the spatio-temporal relations of the multimedia contents.Specifically,the LSTM consisted of 3 layers where the input layer’s size was equal to the number of multimedia types,and the hidden layer’s size was half of the input size.A two-yearlong network traffic logs collected from a factory-intensive area in Tianjin,China,were considered the normal traffic of the dataset; all cyber attacks were manually added to the dataset.And various ratios of training data and testing data were used to train the LSTM model.The empirical studies showed that the LSTM accurately predicted the multimedia contents.The LSTM outperformed three baseline models:SVM,ANN,and RNN.The performance was measured in MAE,MRE,and RMSE,where the LSTM achieved 3.9857 for MAE,0.0553 for MRE,and 4.6166 for RMSE.Despite the LSTM’s excellent performance,predicting multimedia contents in the industrial network remains open.

    4) Cybersecurity Pattern Recognition with Deep Belief Network (DBN):A DBN-based model was proposed in [51] to detect false data injection attacks in the context of energy internet.It is challenging to detect stealthy cyber attacks due to many control signals and meter reading data on the energy internet.And this represents the CPS characteristics of physical process monitoring,closed control loops,and legacy technology.In this study,the detection problem was formulated as a dual bi-level programming problem with the upper and lower bounds because the variations of the electric loads can be predicted.And the prediction problem is translated as a regression problem in the ML domain.A DBN model was trained to forecast electric load.The DBN was made of three stacked RBM models forming six layers — an input layer,four hidden layers,and a logistic regression layer as the output layer.The DBN was trained with the data collected from simulated IEEE 14- and 118-bus systems.An overall error rate was used to measure the DBN model’s performance,and the DBN achieved a 2.73% error rate that was almost 3% lower than the benchmark model SVM.In this study,the DBN used only as a forecasting component of the intrusion detection model is a good example of trading off between the DL model and other programming solutions.

    A DBN-based model was proposed in [52] to detect false data injection attacks in electric power networks.The detection of false data injection attacks represents the CPS characteristics of physical process monitoring,closed control loops,and legacy technology.The research problem was translated into a classification problem in the ML domain.The detection accuracy was investigated through the features automatically generated by DL models.Specifically,a DBN model was proposed as a baseline approach to compare with an RNN model and a graph neural network (GNN) model.The DBN was created by using the default settings of Tensorflow package without optimization.The data was collected from two simulated systems,IEEE 30-bus and 118-bus.The DBN achieved 0.9939 for precision and 0.98231 for recall on the IEEE 30-bus,which was almost identical to the GNN model and slightly higher than the RNN model.However,on the IEEE 118-bus,the DBN model’s performance was consistently better than the RNN model and comparable to the graph neuron network model.Overall,the DBN model shows consistency,and the GNN model shows some promising future.

    Remark 2:Table II lists thirteen different DL models for detecting various cyber attacks across various CPS scenarios.Among the models,four were DNN-based,two CNN-based,five RNN-based,and two DBN-based.It took longer time and more computational resources to train some models than others because of the nature of the architectures and parameter settings.In general,CNNs,DNNs,and DBNs were faster to train than RNNs and LSTMs.But the slower models like LSTMs or even an embedded LSTM layer had their merits in forecasting electric power load soon.On the contrary,DNNs,CNNs,and DBNs provided more powerful capabilities to detect anomalies.Last but not least,DNNs earned their popularity partly because of their simple setup and various hyperparameter fine-tuning techniques.

    IV.CHALLENGES AND FUTURE OPPORTUNITIES

    Six potential areas are depicted in Fig.2 where challenges and new research directions may arise.These six areas correspond to the six steps of our research methodology,as shown in Fig.1.Our research methodology helps provide an overview of the research literature and extract important elements for comparative analysis.The analysis results underpin research challenges and opportunities in the near future.

    A.New CPS Cybersecurity Scenarios

    The papers studied the communication networks in CPS scenarios.The majority of the surveyed publications investigated the CPS scenarios in water treatment plants or smart grids,which accounted for thirteen out of twenty surveyed papers.Among the remaining six papers,two studied vehicle networks,one on generic industrial control networks,one on chemical process plant,one on aviation communication networks,and one on gas pipeline controllers.The imbalanced topics suggest that CPS scenarios were underrepresented.

    Applying DL to the 21st-century manufacturing industry is an emerging topic.DL has been used to detect flaws anddefects during the manufacturing process of complicated items such as semiconductor [53],and 3D printing [54].They were excluded because no cybersecurity issues were covered.A hybrid model was proposed in [53] to detect wafer defect patterns during the semiconductor fabrication process,and an AE-based model was proposed in [54] to detect defects in the 3D printed objects.These two works primarily considered the causes of defects as a production issue or a modeling issue without the presence of cyber attacks.However,cyber attacks and threats may exist in the manufacturer’s network or in the cloud server where the design models are stored.Moreover,blockchain technologies have been studied in two of the surveyed papers [26],[48] in smart grids for providing additional cybersecurity and privacy guarantee.We anticipate that blockchain will be studied much more widely together with CPS.And the diversity and development of CPS scenarios will require intensive studies on cybersecurity.

    TABLE IICYBERSECURITY PATTERN RECOGNITION WITH DEEP LEARNING

    B.Identification of New Cyber Threats

    Almost all the surveyed papers studied false data injection attacks.The detection of stealthy false data injection attacks is challenging because of the large amount of noise produced in the CPS and the lack of cybersecurity mechanisms to authenticate devices and messages transmitted across the network.There are a few types of false injection attacks,depending on the attacker’s information and goals.For example,no advanced information is required for launching DoS attacks; only recorded packets are required for replay attacks; scanning tools are required for probing attacks;automated tools are required for fuzzy attacks.The effective and efficient analysis of the network traffic is crucial for defending against these attacks [55]–[57].

    Furthermore,CPS cybersecurity is much broader than cyber attacks against CPS systems.Detecting the cyber attacks that originated in the cyber space and penetrated the physical domain remains a great challenge.For example,the Stuxnet attack [58] utilized many cyber attack elements like system vulnerabilities,network sniffing,and many more,to exploit the kinetic system.Such attacks may cause significant damage and loss to the physical infrastructure.There are proposals to mitigate these sophisticated attacks by monitoring the CPS system’s cyber component and the physics component.However,it remains an open problem to correlate the interdependent monitoring mechanism in a distributed and realtime manner.

    Fig.2.Research directions for DL driven CPS cybersecurity.

    Considering the characteristics of CPS cybersecurity,it needs to explore three topics — advanced persistent threats,insider attacks,and cyber incident forecasts.Advanced persistent threats (APTs) quickly appear due to the availability of attack tools and techniques and are difficult to defend against because of their frequently updated signatures and evolving behaviors based on reconnaissance results [59].Insiders are very challenging to defend against because their specialized knowledge can be utilized to circumvent security checkpoints deployed on the perimeters of the critical infrastructure [60].Despite its potential high false positives,cyber incident forecasting is valuable to prepare early for fortifying strong defense and improve the overall cyber resilience [61].Due to available big data,many aspects of these three topics can be significantly advanced and improved by adopting ML and DL models.In particular,new insights and knowledge can be obtained via visualization,and deep analysis.We expect that emerging cyber attacks will precede the defense mechanisms,but the risk can be mitigated through the data-driven approach.

    C.Adopting New ML/DL Paradigms

    All the surveyed papers followed traditional ML paradigms,including supervised and unsupervised learning.Apart from four papers that examined regression problems and three papers on clustering problems,the remaining papers studied classification problems.The dominating use of the supervised learning paradigm reflected the importance of the well-labeled data.In particular,network packets were labeled as normal or attack traffic,and the attack types often were differentiated.Such reliance on labeled data restricted the wide adoption of ML or DL methods.We advocate for researchers and practitioners to try new ML/DL paradigms.These new paradigms include reinforcement learning and self-supervised learning,together with improving the model’s explainability.Instead of relying on learning the records,reinforcement learning focuses on experience [62].It is very suitable in isolated environments where many CPSs are currently deployed.For example,a reinforcement learning model was constructed in [63] to predict the normal driving behaviors.Deep reinforcement learning was applied to build human-level intelligence [64].Deep reinforcement learning may have huge potentials in fighting against cyber attacks on CPSs.Selfsupervised learning is a relatively new paradigm to generate data labels automatically.Self-supervised learning is a special kind of supervised learning without the need for the manual labeling process.In theory,self-supervised learning is ideal for changing environments where zero-day attacks and unknown attacks exist because the relationship between input data is deeply analyzed.A popular and successful model is BERT [65] for deriving deep bidirectional representations from unlabeled data.Because of BERT’s huge success in NLP and many other domains,we anticipate that self-supervised learning will prosper in the CPS domain because DL models generally suffer from poor explainability [66].We know how well the DL model works but do not know why the model works.Explainability is important for many CPS problems when we need to improve the system.Fortunately,tools like LEMNA [67] are available to explain DL-based cybersecurity applications.LEMNA can identify the critical features for a trained DL model like RNN and LSTM.We are optimistic to predict that the DL models will be more and more explainable when new tools and techniques are invented and used.

    D.Defending the Trained DL Models

    No surveyed works are considered to defend the trained DL models against various attacks.However,we would like to highlight the importance of defending the trained DL models because of the computational expenses to train the DL models,the important roles of the trained models,and potential dangers if the DL models are compromised.Due to the hunger for training samples,the DL models are sometimes trained with data from untrustworthy sources.Thus,adversarial attacks are prevalent because of linear behavior in highdimensional space [68].For example,Android HIV was proposed in [69] to automatically generate adversarial Android malware that the existing detectors failed to detect.

    Adversarial attacks are categorized into two types,including evasion attacks and poisoning attacks [70].Evasion attacks aim to evade a trained classifier by altering input data,and poisoning attacks aim to affect the trained classifier by injecting poisoning samples to the training dataset.These attacks differ by their time of occurrence.Poisoning attacks usually occur during the training phase,but evasion attacks occur during the test phase.Much research efforts have been devoted to fighting against data poisoning attacks,including robust optimization,adding some adversarial samples to training data,blind-spot removal,examining decision boundaries,and many more.There are many strategies to counter poisoning attacks,such as outlier removal,data sensitization,frequent model retraining,classifier ensembles,randomizing the classifier’s output,and many more.Last but not least,model stealing attacks were proven in [71] as a big threat to the trained model.The attackers can obtain sufficient information to mimic an ML or DL model by launching many queries and aggregating the results.The extracted information can be used to build a mimicking model for the attacker to find possible evasion attacks.We strongly advocate that cyber defense should be conducted as soon as possible due to the unawareness of adversarial attacks in the CPS scenarios.

    E.Enriching CPS Cybersecurity Datasets

    Among the surveyed papers,datasets collected in the field dominated the simulation with a 14:6 ratio.Simulated data were investigated in the two CPS scenarios — smart grids and vehicular networks.In smart grids,Matlab was the only choice for simulating electric load in five papers; and the OCTANE simulator was used in one paper on vehicular networks.However,there is a significant risk of solely relying on proprietary products like Matlab because the availability of such products may be discontinued unprecedently.On the other hand,field data is independent of the simulation platform and offers researchers good flexibility.Among the papers using field data,five papers chose the SWaT dataset,two papers the CICIDS2017 dataset,and the rest seven papers different datasets.The SWaT dataset dominated the field data category for a few reasons: 1) The network traffic data were continuously collected for 11 days from the control networks and from the sensors of a physical testbed,2) the traffic with and without attack were chronologically separated for easy use,and 3) there were 36 attack scenarios against different components of the testbed.To our surprise,the NSL-KDD dataset [33],also known as KDD99-cup,was studied in one surveyed paper despite its age of 20+ years.Using such a dated dataset may cause people to draw biased conclusions because many cyber attacks were not included in the NSLKDD dataset.Therefore,we recommend the researchers to use datasets like the UNSW-NB15 dataset [32] where recent cyber attacks were present.

    Moreover,new datasets will always be valuable and appreciated.Ideally,the new datasets are open-sourced field data collected from physical testbeds.Several CPS testbeds are proposed to facilitate detecting cyber attacks,such as[72]–[74].The recent trend of increasing interest in building CPS testbeds may benefit researchers to collect high-quality attack and defense data.The new datasets should be large enough to exploit DL models’ power,and both new and old cyber attacks should be included because cyber attacks evolve quickly.If labeling data is challenging,then chronologically separating the attacks from the normal traffic is a feasible idea.Artificially blending the data entries representing attacks into a set of normal traffic records should be avoided because the simple data augmentation method does not consider feasibility,attacking sequences,and possible changes of correlations.To benefit the advancement of research and knowledge,we strongly encourage more and more highquality datasets to be made available to the community.

    F.Improving the Model Evaluation

    Standard performance metrics were used in most of the surveyed papers.False positives were investigated,along with accuracy and error rates.It is proven in [34] that it is significantly more difficult to detect the rarely occurred attacks than the common ones derived by the Bayesian laws.In real-world CPSs,cyber attacks may rarely occur,so the DL models trained in the lab settings may be invalid.Since most papers did not investigate the impact of the imbalanced data between normal and attack traffic,the empirical results may be substantially biased or inflated.Cross comparisons in [75]showed that the precision-recall curve (PRC) and the area under precision-recall curve (auPRC) were more resilient to imbalanced data than ROC and auROC.Therefore,new studies should consider reporting PRC and auPRC.

    Furthermore,time decay should be considered in future studies because each trained ML or DL model’s performance will inevitably degrade over time.When the cyber attacks rapidly evolve,the models trained with old data will struggle with detecting new attacks.A time decay metric was proposed in [75] to evaluate a trained model’s performance loss.By studying the time decay,we will be able to decide when the model needs to be retrained.We strongly hope to see future work similar to [75] in the context of CPS and cyber attacks.Once the in-depth knowledge is developed and gained,we may expect to mitigate the risk of CPS cyber attacks.

    V.CONCLUSION

    This survey provides a current view of detecting cyber attacks in the CPSs.A six-step DL driven methodology is proposed to summarize and analyze the twenty recently published papers in this survey.Specifically,a panoramic view is obtained through inspecting the CPS scenarios,identifying cybersecurity problems,translating the research problem to the ML/DL domain,constructing the DL model,preparing datasets,and finally evaluating the model.Cyber attacks persist as an ongoing and prominent threat to the security and safety of the CPSs.The reviewed works show great potential to exploit CPS cyber data through DL models because of their promising performances.The excellent performance is achieved partly because of several high-quality datasets that are readily available for public use.In addition to following the success of current research,we also identified promising research topics,including integration with blockchain,detection of advanced persistent threats,adopting new ML and DL paradigms,prevention of adversarial and model extraction attacks,enriching datasets,and applications of additional performance metrics.We are optimistic and confident that the research in this field will flourish.

    少妇被粗大猛烈的视频| 成人毛片60女人毛片免费| 免费观看av网站的网址| 亚洲av日韩在线播放| 亚洲精品自拍成人| 国产熟女午夜一区二区三区 | 国产视频首页在线观看| 国产综合精华液| 啦啦啦视频在线资源免费观看| 国模一区二区三区四区视频| 欧美日韩av久久| 国产一级毛片在线| 久久久a久久爽久久v久久| 丁香六月天网| 国产日韩欧美在线精品| 久热这里只有精品99| 亚洲欧美成人精品一区二区| 黄色一级大片看看| 亚洲欧洲国产日韩| 午夜老司机福利剧场| 免费黄网站久久成人精品| 欧美人与善性xxx| 天堂中文最新版在线下载| 男人操女人黄网站| 国产精品偷伦视频观看了| 少妇 在线观看| 51国产日韩欧美| 国产深夜福利视频在线观看| 两个人免费观看高清视频| 亚洲欧美成人精品一区二区| 人体艺术视频欧美日本| 日韩av不卡免费在线播放| 日本色播在线视频| 各种免费的搞黄视频| 欧美少妇被猛烈插入视频| 国产视频首页在线观看| 91精品国产国语对白视频| 国产精品熟女久久久久浪| 秋霞在线观看毛片| 亚洲av在线观看美女高潮| 精品一区二区三卡| 免费看不卡的av| 欧美激情极品国产一区二区三区 | 欧美激情国产日韩精品一区| 国产有黄有色有爽视频| 各种免费的搞黄视频| 免费观看的影片在线观看| 老女人水多毛片| 国产av码专区亚洲av| 人人妻人人澡人人看| 中文字幕av电影在线播放| 美女视频免费永久观看网站| 在线精品无人区一区二区三| 极品人妻少妇av视频| 热re99久久国产66热| 久久久久久久大尺度免费视频| 欧美激情国产日韩精品一区| 一区在线观看完整版| 精品久久蜜臀av无| 校园人妻丝袜中文字幕| 久久精品久久精品一区二区三区| 国产av国产精品国产| 97精品久久久久久久久久精品| 国产欧美日韩一区二区三区在线 | 国产伦精品一区二区三区视频9| 日韩中文字幕视频在线看片| 欧美日韩视频高清一区二区三区二| 国产免费一区二区三区四区乱码| av国产久精品久网站免费入址| 一级黄片播放器| 黑人巨大精品欧美一区二区蜜桃 | 国产日韩欧美亚洲二区| 精品一区二区三卡| 久久鲁丝午夜福利片| 国产精品一区二区三区四区免费观看| 日本黄大片高清| 日本-黄色视频高清免费观看| 一区在线观看完整版| 久久久国产一区二区| 校园人妻丝袜中文字幕| 一本一本综合久久| 狂野欧美白嫩少妇大欣赏| 999精品在线视频| 又黄又爽又刺激的免费视频.| 91精品一卡2卡3卡4卡| 精品国产露脸久久av麻豆| 亚洲美女黄色视频免费看| 日日撸夜夜添| 国产成人av激情在线播放 | 特大巨黑吊av在线直播| 在线观看免费视频网站a站| videosex国产| 亚洲精品aⅴ在线观看| 欧美bdsm另类| 2018国产大陆天天弄谢| 大片免费播放器 马上看| 日韩亚洲欧美综合| 久久人人爽人人片av| 日日啪夜夜爽| 中文字幕精品免费在线观看视频 | 国产精品人妻久久久影院| 亚洲精品色激情综合| 午夜91福利影院| 免费黄频网站在线观看国产| 亚洲av欧美aⅴ国产| 国产成人精品婷婷| 国产亚洲一区二区精品| 国产成人精品在线电影| xxx大片免费视频| 欧美精品亚洲一区二区| 日日摸夜夜添夜夜添av毛片| 亚洲欧美成人综合另类久久久| 亚洲国产最新在线播放| 午夜视频国产福利| 欧美日韩精品成人综合77777| 久久国产亚洲av麻豆专区| 蜜桃久久精品国产亚洲av| 老司机影院毛片| 亚洲av福利一区| 日韩 亚洲 欧美在线| 欧美日韩成人在线一区二区| 一区二区三区四区激情视频| 久久久久久久久久久丰满| 你懂的网址亚洲精品在线观看| 精品午夜福利在线看| 国产黄色视频一区二区在线观看| 女人久久www免费人成看片| 亚洲av成人精品一二三区| 国内精品宾馆在线| 人人妻人人爽人人添夜夜欢视频| 成人国产av品久久久| 欧美日本中文国产一区发布| 爱豆传媒免费全集在线观看| 日本黄色片子视频| 99精国产麻豆久久婷婷| 高清av免费在线| 国产乱来视频区| 国产精品一区二区三区四区免费观看| 久久人人爽人人爽人人片va| 免费看av在线观看网站| 夜夜看夜夜爽夜夜摸| 国产亚洲午夜精品一区二区久久| 亚洲国产成人一精品久久久| 精品熟女少妇av免费看| 日韩成人av中文字幕在线观看| 精品99又大又爽又粗少妇毛片| 美女脱内裤让男人舔精品视频| 自拍欧美九色日韩亚洲蝌蚪91| 最近的中文字幕免费完整| 99re6热这里在线精品视频| 欧美bdsm另类| 国产黄色视频一区二区在线观看| 色94色欧美一区二区| 大码成人一级视频| 亚洲精品456在线播放app| 亚洲久久久国产精品| 日韩av在线免费看完整版不卡| 色婷婷av一区二区三区视频| 久久人人爽av亚洲精品天堂| 下体分泌物呈黄色| 久久久a久久爽久久v久久| 亚洲中文av在线| 亚洲天堂av无毛| 色哟哟·www| 午夜老司机福利剧场| 久久久久久人妻| 国产不卡av网站在线观看| 欧美精品一区二区大全| 中国三级夫妇交换| 亚洲婷婷狠狠爱综合网| 妹子高潮喷水视频| 国产精品不卡视频一区二区| 纵有疾风起免费观看全集完整版| 精品久久久久久电影网| 国产成人a∨麻豆精品| 妹子高潮喷水视频| 91精品一卡2卡3卡4卡| 国产永久视频网站| xxxhd国产人妻xxx| 国产免费又黄又爽又色| 人人妻人人澡人人看| 亚洲四区av| 亚洲欧美成人精品一区二区| 精品久久蜜臀av无| 蜜桃久久精品国产亚洲av| 交换朋友夫妻互换小说| 午夜91福利影院| 亚洲人成网站在线观看播放| 欧美日韩精品成人综合77777| 亚洲av免费高清在线观看| 午夜福利,免费看| 国产成人免费观看mmmm| 国产精品久久久久久久久免| 男女边摸边吃奶| 一级毛片电影观看| 秋霞在线观看毛片| 亚洲精品久久成人aⅴ小说 | 精品国产乱码久久久久久小说| 日本爱情动作片www.在线观看| 蜜桃国产av成人99| 免费看光身美女| 国产欧美日韩综合在线一区二区| 亚洲欧洲精品一区二区精品久久久 | 国产精品一区二区三区四区免费观看| 亚洲精品自拍成人| 夫妻午夜视频| 亚洲欧美一区二区三区国产| 日韩一区二区视频免费看| 精品人妻偷拍中文字幕| 欧美少妇被猛烈插入视频| 在线免费观看不下载黄p国产| 国产精品蜜桃在线观看| 欧美性感艳星| 高清欧美精品videossex| 又黄又爽又刺激的免费视频.| 国产免费现黄频在线看| 草草在线视频免费看| 亚洲色图综合在线观看| 熟女人妻精品中文字幕| 黄色欧美视频在线观看| 一级,二级,三级黄色视频| 成人18禁高潮啪啪吃奶动态图 | 久久精品国产亚洲网站| 男人添女人高潮全过程视频| 大话2 男鬼变身卡| 最黄视频免费看| 久久久久人妻精品一区果冻| 亚洲人与动物交配视频| 又黄又爽又刺激的免费视频.| 满18在线观看网站| 丝袜美足系列| 在线精品无人区一区二区三| 男女国产视频网站| 亚洲精品一区蜜桃| 成年美女黄网站色视频大全免费 | 99热这里只有精品一区| 亚洲中文av在线| 69精品国产乱码久久久| 免费观看无遮挡的男女| 国产精品久久久久久精品古装| 在线精品无人区一区二区三| 日韩av免费高清视频| 国产欧美日韩一区二区三区在线 | 在线观看www视频免费| 国产一区二区三区综合在线观看 | 极品人妻少妇av视频| 久久 成人 亚洲| 妹子高潮喷水视频| 日韩一本色道免费dvd| 午夜老司机福利剧场| 亚洲,欧美,日韩| 亚洲综合色网址| 熟妇人妻不卡中文字幕| 成人手机av| 老司机亚洲免费影院| www.av在线官网国产| 亚洲国产欧美在线一区| 在线免费观看不下载黄p国产| 亚洲,欧美,日韩| 日产精品乱码卡一卡2卡三| 国产亚洲最大av| 在线观看www视频免费| 99re6热这里在线精品视频| 欧美日韩视频精品一区| 午夜激情福利司机影院| 高清黄色对白视频在线免费看| 91精品伊人久久大香线蕉| 熟女av电影| 国产欧美另类精品又又久久亚洲欧美| 99热这里只有是精品在线观看| 色94色欧美一区二区| 美女国产视频在线观看| 精品卡一卡二卡四卡免费| 亚洲精品乱码久久久v下载方式| 成人免费观看视频高清| 精品一区二区免费观看| 王馨瑶露胸无遮挡在线观看| 亚洲不卡免费看| 欧美三级亚洲精品| 亚洲欧洲日产国产| 欧美精品高潮呻吟av久久| 又大又黄又爽视频免费| 国产亚洲一区二区精品| 久久久久久久久久久久大奶| 大香蕉97超碰在线| 一级毛片黄色毛片免费观看视频| 久久午夜福利片| 伦精品一区二区三区| 有码 亚洲区| 久久久国产欧美日韩av| 日韩制服骚丝袜av| 91精品国产九色| 婷婷色麻豆天堂久久| 在线 av 中文字幕| 日日摸夜夜添夜夜添av毛片| 日韩制服骚丝袜av| 高清欧美精品videossex| 午夜免费观看性视频| 国产精品秋霞免费鲁丝片| 国产亚洲一区二区精品| 亚洲第一区二区三区不卡| 欧美国产精品一级二级三级| 秋霞在线观看毛片| 国产探花极品一区二区| 久久久久久伊人网av| a级毛片黄视频| a级片在线免费高清观看视频| 精品久久蜜臀av无| 一级爰片在线观看| av又黄又爽大尺度在线免费看| 亚洲国产精品国产精品| 亚洲av成人精品一二三区| 999精品在线视频| 国产精品国产三级专区第一集| 国产片特级美女逼逼视频| 国产精品久久久久久久电影| 精品卡一卡二卡四卡免费| 成年人免费黄色播放视频| 国产精品女同一区二区软件| 国产精品国产av在线观看| 2018国产大陆天天弄谢| 日韩电影二区| 日韩av免费高清视频| 久久久久久久亚洲中文字幕| 一本色道久久久久久精品综合| 午夜精品国产一区二区电影| 美女主播在线视频| 午夜精品国产一区二区电影| 又大又黄又爽视频免费| 国产深夜福利视频在线观看| 极品人妻少妇av视频| 久久久久久久国产电影| 日韩精品有码人妻一区| 国产深夜福利视频在线观看| 又大又黄又爽视频免费| 久久精品国产亚洲av天美| 国产精品偷伦视频观看了| 97精品久久久久久久久久精品| 91久久精品电影网| 狂野欧美激情性bbbbbb| 一级,二级,三级黄色视频| 精品一区二区三区视频在线| 欧美精品人与动牲交sv欧美| 国产精品一区www在线观看| 午夜福利在线观看免费完整高清在| 久久午夜福利片| 久久久久久久久久久免费av| 热re99久久国产66热| 国产精品一区二区三区四区免费观看| 91精品伊人久久大香线蕉| www.av在线官网国产| 免费观看性生交大片5| 色5月婷婷丁香| 国产视频首页在线观看| 午夜福利,免费看| 中国国产av一级| 亚洲精华国产精华液的使用体验| 国产高清国产精品国产三级| 亚洲性久久影院| 男女边吃奶边做爰视频| 少妇的逼水好多| 亚洲欧洲日产国产| 国产不卡av网站在线观看| 五月玫瑰六月丁香| 纯流量卡能插随身wifi吗| 亚洲中文av在线| 伦精品一区二区三区| 欧美精品国产亚洲| 少妇 在线观看| av有码第一页| 日本wwww免费看| 国产精品一区www在线观看| 久久久久国产网址| 亚洲国产欧美在线一区| 亚洲精品456在线播放app| 久久热精品热| 91久久精品国产一区二区成人| 18+在线观看网站| 黑人高潮一二区| 18禁动态无遮挡网站| 18在线观看网站| 国产亚洲精品久久久com| 久久精品国产亚洲av涩爱| 精品久久国产蜜桃| 高清毛片免费看| 久久免费观看电影| 五月伊人婷婷丁香| 能在线免费看毛片的网站| 91精品三级在线观看| 国产精品三级大全| 久久久久久久久久久丰满| 欧美精品一区二区免费开放| 一本一本综合久久| 人妻制服诱惑在线中文字幕| 国国产精品蜜臀av免费| 精品亚洲乱码少妇综合久久| 亚洲精品自拍成人| 日韩免费高清中文字幕av| 欧美老熟妇乱子伦牲交| 七月丁香在线播放| 嘟嘟电影网在线观看| 寂寞人妻少妇视频99o| 久久国产亚洲av麻豆专区| 久久久欧美国产精品| 十分钟在线观看高清视频www| 欧美丝袜亚洲另类| 欧美日韩综合久久久久久| 欧美少妇被猛烈插入视频| 在线亚洲精品国产二区图片欧美 | 国产视频首页在线观看| 日日啪夜夜爽| 视频在线观看一区二区三区| 一本久久精品| 国产精品人妻久久久久久| 日本wwww免费看| 天堂俺去俺来也www色官网| 肉色欧美久久久久久久蜜桃| 久久综合国产亚洲精品| 日韩视频在线欧美| av网站免费在线观看视频| 免费大片18禁| 欧美一级a爱片免费观看看| 三级国产精品欧美在线观看| 国产精品女同一区二区软件| 亚洲精品乱码久久久v下载方式| 中文字幕人妻丝袜制服| 国国产精品蜜臀av免费| 纯流量卡能插随身wifi吗| 欧美精品人与动牲交sv欧美| 天美传媒精品一区二区| 精品一品国产午夜福利视频| 人人妻人人澡人人看| 国产精品国产三级国产专区5o| 中文字幕人妻丝袜制服| 国产精品女同一区二区软件| 日韩视频在线欧美| 婷婷色综合大香蕉| 欧美 亚洲 国产 日韩一| 91精品一卡2卡3卡4卡| 啦啦啦在线观看免费高清www| 晚上一个人看的免费电影| 天天躁夜夜躁狠狠久久av| 国产成人免费无遮挡视频| 两个人免费观看高清视频| 国产一区二区在线观看av| 性色av一级| 国产黄色免费在线视频| 久久国内精品自在自线图片| 免费av不卡在线播放| 91在线精品国自产拍蜜月| 亚洲第一av免费看| 一级,二级,三级黄色视频| 一级毛片我不卡| kizo精华| 十分钟在线观看高清视频www| 韩国高清视频一区二区三区| 99国产精品免费福利视频| 亚洲成人一二三区av| 女的被弄到高潮叫床怎么办| 肉色欧美久久久久久久蜜桃| 日本猛色少妇xxxxx猛交久久| 国产视频内射| 午夜福利在线观看免费完整高清在| 最近中文字幕高清免费大全6| 久久久久久久亚洲中文字幕| 91aial.com中文字幕在线观看| 美女福利国产在线| 99久久精品一区二区三区| 成人国产麻豆网| 韩国av在线不卡| 精品国产露脸久久av麻豆| 国产精品一区二区在线不卡| 中文字幕人妻熟人妻熟丝袜美| 亚洲av国产av综合av卡| 午夜老司机福利剧场| 18+在线观看网站| xxx大片免费视频| 妹子高潮喷水视频| 人妻夜夜爽99麻豆av| 国产免费视频播放在线视频| 国产成人午夜福利电影在线观看| 26uuu在线亚洲综合色| 欧美精品亚洲一区二区| 人妻制服诱惑在线中文字幕| 少妇的逼好多水| 特大巨黑吊av在线直播| 日韩一本色道免费dvd| 一级毛片我不卡| 一个人看视频在线观看www免费| 国产国拍精品亚洲av在线观看| 九九爱精品视频在线观看| 2022亚洲国产成人精品| videossex国产| 欧美激情 高清一区二区三区| 爱豆传媒免费全集在线观看| 香蕉精品网在线| 精品人妻偷拍中文字幕| 午夜老司机福利剧场| 亚洲久久久国产精品| 少妇的逼好多水| 伊人久久国产一区二区| 亚洲美女黄色视频免费看| 欧美日韩av久久| 精品国产一区二区三区久久久樱花| 观看美女的网站| 一区二区日韩欧美中文字幕 | 久久久久国产网址| 黑人猛操日本美女一级片| 丁香六月天网| 少妇丰满av| 在线观看www视频免费| 男女免费视频国产| 91精品国产国语对白视频| 熟女av电影| 精品久久国产蜜桃| 久久久国产欧美日韩av| 美女主播在线视频| 免费黄色在线免费观看| 久久人妻熟女aⅴ| 美女福利国产在线| 久久久久久久久久成人| 九色成人免费人妻av| 亚洲美女搞黄在线观看| av播播在线观看一区| 下体分泌物呈黄色| 五月开心婷婷网| a级毛片免费高清观看在线播放| 涩涩av久久男人的天堂| 亚洲av.av天堂| 久久精品国产鲁丝片午夜精品| 欧美bdsm另类| 免费看av在线观看网站| 亚洲一区二区三区欧美精品| 五月天丁香电影| 免费观看av网站的网址| 一区二区三区乱码不卡18| videosex国产| 国精品久久久久久国模美| 99re6热这里在线精品视频| 日韩强制内射视频| 亚洲丝袜综合中文字幕| 一本色道久久久久久精品综合| 久久久久久久久久久丰满| 日韩中字成人| 日本黄色片子视频| 精品人妻偷拍中文字幕| 男人爽女人下面视频在线观看| 日本免费在线观看一区| 国产精品久久久久久av不卡| 欧美人与善性xxx| 中文字幕最新亚洲高清| 考比视频在线观看| 一级a做视频免费观看| 美女中出高潮动态图| 五月玫瑰六月丁香| 夜夜爽夜夜爽视频| 97超视频在线观看视频| 黄色欧美视频在线观看| 亚洲综合色网址| 老女人水多毛片| 日本wwww免费看| 乱人伦中国视频| 国产精品秋霞免费鲁丝片| 午夜精品国产一区二区电影| 在线观看免费日韩欧美大片 | 高清午夜精品一区二区三区| 99九九在线精品视频| 波野结衣二区三区在线| videossex国产| 欧美最新免费一区二区三区| 午夜福利视频精品| 国产免费现黄频在线看| av卡一久久| 成人免费观看视频高清| 91久久精品国产一区二区成人| www.av在线官网国产| 免费观看性生交大片5| 国产一区二区三区综合在线观看 | 色哟哟·www| 精品人妻在线不人妻| 多毛熟女@视频| 少妇被粗大的猛进出69影院 | 母亲3免费完整高清在线观看 | 日韩成人伦理影院| 久久影院123| 午夜福利,免费看| 观看美女的网站| 欧美激情极品国产一区二区三区 | 一区二区三区四区激情视频| 一个人看视频在线观看www免费| 一区二区三区四区激情视频| 成人免费观看视频高清| 99九九在线精品视频| 国产av精品麻豆| 最近手机中文字幕大全| 午夜福利在线观看免费完整高清在| 女性被躁到高潮视频| 男的添女的下面高潮视频| 下体分泌物呈黄色| 日韩一本色道免费dvd| 少妇熟女欧美另类| 91aial.com中文字幕在线观看| 如何舔出高潮| 国产一区二区在线观看日韩| 亚洲av在线观看美女高潮| 成人国语在线视频| 一级毛片aaaaaa免费看小| 黑人猛操日本美女一级片| 日日摸夜夜添夜夜添av毛片| 综合色丁香网| 久久久国产精品麻豆| 我要看黄色一级片免费的| 亚洲精品aⅴ在线观看| 人妻系列 视频| 美女主播在线视频|